Not Quite Software
LLMs aren’t apps, APIs, or assistants - and the metaphors we use to talk about software are breaking down
By far the most impactful $20 I spend every month is my ChatGPT subscription, and it’s not even remotely close. Two years ago, this product didn’t exist. Now I ask it about everything: best steak cooking temps, how to politely thank annoying relatives, what food travels well for boat parties, and which ramen shops are open in Tokyo at 2am. (Answer: a lot).
The weird part about it? I can’t even tell you what category this thing fits in.
Most mainstream tech solves a clear problem. WhatsApp is for chatting with your sister. Google helps you find websites. Instagram is for feeling bad about your life.
But what’s ChatGPT for?
Sometimes it’s a personal assistant. Sometimes it’s a search engine. And sometimes, when you ask it to describe you, it’s like running whoami on your soul. And the output is disturbingly accurate. (Pro tip: do not try this with friends in the room unless they’re also your therapist.)
But here’s the catch: those are all roles it plays because I asked it to. They don’t describe what it actually is. LLMs are unlike any software we’ve ever used, mostly because we don’t know what they are, and we definitely don’t know what they’re for.
The Metaphor Gap
Most of my friends — like yours if you are reading this SubStack — live in a tech bubble. We’ve heard of Sam Altman. We know what an API is (even if we don’t want to write yours). I can use the generic word “model” and you’ll understand what I mean.
But try actually defining what an LLM is — like, out loud, to a normal person — and you’ll feel the edges of insanity starting to creep in.
It’s like an app? But there’s no clear task it’s designed to do.
It’s like a search engine. But you don’t get links… just a weirdly confident, potentially hallucinatory, essay.
It’s like talking to a person. But it forgets everything you’ve ever said to it, instantly.
LLMs are just… something smart that shows up when you ask. Kind of like rentable intelligence that shows up via Wifi. Which, to be clear, is definitely Cool™.
But also a little unnerving because everything we’ve said so far describes what LLMs aren’t.
So how should we think about them?
Here’s a few metaphors I’ve heard recently. All of these work, right up until the point they don’t.
Like Power, but Smarter (read: Weirder)
The way I think about LLMs most of the time is pretty simple: they’re like public utilities, but for intelligence.
We expect utilities like water and electricity to be reliable. Always on, always there when we need them. They’re expensive to build, but once they’re running, they scale easily. The more you use, the more you pay. And no one really cares which power company is behind the outlet — they just want the lights to turn on.
That’s how I treat models like Gemini, ChatGPT, or Claude. My toaster doesn’t care if the power came from coal, wind, or solar. It just wants electricity. Same with my prompts most days. I’m not looking for magic. I just want a straight answer.
LLMs feel like that… until you flip the switch and get an apology letter to your landlord instead.
The type and quality of electricity doesn’t change based on how politely you ask for it. LLMs do.
Unlike utilities, end users can’t always expect the same output from the same input. Sometimes you get water. Sometimes you get poetry. Sometimes you get confidently incorrect math.
(That’s why I put together a whole breakdown on the six building blocks of strong AI prompts — because getting back useless answers got old real fast)
Like MacOS With Stronger Opinions
Another way I think about LLMs is like operating systems — not because they run hardware, but because they stitch everything else together.
Windows, Mac, and Linux are the glue that makes computers work. They connect your apps, manage memory, handle storage and networking. You don’t usually think about them unless something breaks, but they’re always working behind the scenes to make everything feel seamless.
LLMs, especially with the advent of autonomous systems, or MCP servers, are starting to feel the same way. They can talk to other tools, remember short-term context, route tasks between systems, even translate between APIs or file formats. They’re not the app, and they’re not the output. They’re the goopy layer in between that holds everything together and makes it work.
And just like operating systems, there are only a handful of major players: OpenAI, Google, Anthropic, Meta, xAI.
Some models are locked-down and corporate. Others are open and re-mixable. You can fine-tune them, distill them, even run them on your own hardware if you’re brave (or bored). It’s not quite Linux-for-your-brain, but it’s starting to get close.
LLMs feel like that… until your function call returns a haiku instead of structured JSON.
Above all else, operating systems are consistent. They give you APIs, permissions, error codes. When you call a function, you know what it’s going to do.
LLMs don’t come with technical support or a manual. No guaranteed output. No clean abstraction between “what it can do” and “what it might decide to do this time.”
Smart Tools with Strange Behaviors
As a creator, LLMs seem like a Web 2.0 version of a power tool from Home Depot. You give them a task — draft this email, summarize that article — and they do it. Fast, clean, and without complaints about “paid time off” or “lunch breaks”.
But sometimes you ask for a blog title, and it gives you an outline. Then a first draft. Then a tagline. Suddenly it’s pitching follow-up content, branding ideas, and a three-tier marketing funnel. And you’re just sitting there, watching the cursor blink as it sprints off ahead of you.
LLMs feel like picking up a drill and watching it redesign your living room.
These aren’t passive tools waiting for instructions. They extrapolate to generate. They take your starting point and run with it. You wanted to hang a picture frame; they’re halfway through building a shed.
The question we need to ask isn’t “can this help me?” but “how much should I let it run?”
And I don’t know about you, but I’ve never used a power tool that had such strong opinions before.
Where the Metaphors Break
So, yeah — LLMs are kind of like public utilities. Kind of like operating systems. Kind of like power tools.
But really, they’re none of those. They live in this weird conceptual limbo, not quite tool, not quite platform, not quite infrastructure.
And for me, as an engineer, that limbo poses a practical question. How do you build tools on top of something that might behave differently every time you call it? I’m used to working with systems that have rules, contracts, guarantees. LLMs don’t. And how do you build on top of something you can’t rely on?
In Search of a Use Case
A lot of this metaphor chaos comes down to one big thing: LLMs are software in search of a problem.
Like crypto and blockchains before it, the tech is mind-blowing. Genuinely new. But nobody knows what to actually do with it.
Immutable, traceable ledgers sound cool. But I can’t buy a cheeseburger with them. Guess I’ll just stare at the line go up and pretend it matters.
Most major tech shifts happen in the opposite direction. Someone has a real-world problem → builds a solution → that solution scales to make money. Every founder pitch deck and startup playbook is built from this foundation.
Amazon: Bezos wanted to buy books online. Now they sell half the internet, and lease compute time to the other half.
Facebook: Zuckerberg wanted to know who else was single in his dorm. Now they destabilize democracies world-wide through political ad targeting.
Spotify: No one wanted to wait for their six-CD changer to cycle. Now we have access to every song every recorded, and a few that shouldn’t have been.
Problem → Solution → Scale
So… what big problem do LLMs solve? Nobody really knows. (Except at Meta, where the endgame is a dead internet — just bots generating content for other bots, while making sure you stay engaged enough to keep scrolling.)
This lack of a clear problem is why we can’t describe LLMs, or decide what to build with them. No shared vocabulary. No constraints. No common frame of reference.
We don’t know what this software is. We don’t know what it’s for.
And yet… it’s so damn interesting we keep building with it anyway.
In Part 2, I’ll look at how the best AI tools don’t try to solve problems. They help users manage control. Because the real shift isn’t in what LLMs do — it’s in how much we trust them to do it for us.