Is there a ‘right amount’ of AI?

While I’ve made my skepticism of AI clear by now, I’m not an anti-AI purist fighting against “no apps with any hint of AI coding allowed on my machine.” For one thing, it’s almost certainly a lost battle. Microsoft and Apple are both using coding agents for OS development, and while I can’t prove that most people running Linux have at least some LLM-touched code somewhere on their system, I’d be incredibly surprised if it’s not there.

Just this week, I moved back to Overcast from Castro as my podcast player. (I’ve spent a decade or so switching between them. Castro’s queue system is terrific, but the app has always been buggier than Overcast, its subscription price skyrocketed a year or so ago, and it’s now on its third owner—a shady-looking company that’s already diluted the privacy policy.) I know that developer Marco Arment has become an enthusiastic “agentic coder”, and that this means Overcast has Claude-generated code in it. Ditto his fun new app Fineliner, which I’m using on my new iPad mini (more about which later). I don’t think either of these programs are going to erase my data, make my iPad overheat, or send my credit card information to Russian hackers. In theory, what Lex Friedman wrote recently on the subject of vibe-coding, well, vibes with me:

AI has dramatically reduced the cost of producing an app, but it hasn’t reduced the work required to produce a great app. There’s a difference between generating something that works and something that works reliably, handles unexpected situations, feels intuitive, respects platform conventions, and continues improving. […] I will gladly pay for apps that feel crafted. I want apps that show refinement, testing, and thoughtfulness. I want apps from developers I trust will keep on building and iterating. I want apps from developers who feel invested—those are the ones I’m willing, even eager, to invest in.

(Note that’s Lex Friedman, who is cool, not Lex Fridman, who sucks.) And, yes, I think Marco is a developer who feels invested. He shows deep concern for code quality, to the point where he rather (in)famously eschewed using existing libraries for functions he could code himself. He wanted to understand his code inside and out.

So. Let’s talk about Big Walk.

If you haven’t heard of it, Big Walk is a game from House House, the Untitled Goose Game people, published by technically-still-Mac-developers Panic. It was a favorite for this year’s Independent Games Festival awards, but a group of IGF jurists issued a statement declaring they would “decline to vote for any game that uses generative AI in any capacity.” They specifically refused to make a distinction between using genAI for assets (e.g., graphics and sound) and code, writing,

Programming is a creative discipline whose labour deserves the same respect we extend to any part of making a game. This extends to the past work used to train these models without the creators’ consent.

This led House House to confess that, okay, they “sparingly” used LLMs to “help work through difficult technical challenges and bugs”. Less than a percent of a percent of the code in Big Walk was touched in any way by LLMs, and they went out of their way to say “this is not an endorsement of, or argument for, the use of AI tools, however minimal,” and that they “support the work of those who are trying to limit the harm generative AI is doing to videogames and our world, and apologise that Big Walk’s LLM usage may not meet their standards.”

Based on everything above, I shouldn’t have any issue with this. And at first, I didn’t. It all struck me as a tempest in a (Utah) teapot.

But then I read this.

House House, an already very successful studio with not insignificant resources, made a real choice to rely on a morally abhorrent software system rather than engage the help of real people—whether in the form of paid contracting or just friendly help. I want to say this not to try to exaggerate the scope of what they did but to make it absolutely clear that there is a real, material dimension to this, even if some want to characterize this issue as a purely abstract test of ritual purity.

You may balk at “morally abhorrent software system,” but Bruno Dias (the blog post author) is not wrong when he writes “to use LLMs is to willingly participate in a noxious political project and to use the product of exploitative and morally unacceptable processes.” I wrote about that in “AI’s Chicken Sandwich Problem”; in short form, LLMs do not exist in a vacuum. As he continues,

I’m not going to try to argue that any and all LLM usage is equally bad, but either an ethical boundary exists or it doesn’t. There are some ethical faults we generally expect people to not be 100% compliant with, of course, but there are some ethical faults we think it’s reasonable to expect people to never breach without sanction. I think using a completely optional set of highly subsidized SaaS products that did not exist two years ago is in the latter set.

If you disagree, perhaps stop to think about how heavily you’ve been propagandized and how much you’ve mentally surrendered to the normalization of these systems. It is very possible to do things without using an LLM, even things that are hard; and if anything, one of the harms of the “AI” political project is the propaganda machine dedicated to making you feel stupid and helpless.

And, you know, this is something that concerns me.

I recently wrote, “I think I’d lose something meaningful, something vital, if I couldn’t understand the code [that I’ve written]—to be able to read it line by line and know what each API call does, why certain choices are better than others, how it all comes together.” While I’m not suggesting that everyone who goes whole hog for agentic coding is doomed to lose the ability to understand their own code, there’s increasing evidence that “de-skilling” is a real issue, as well as evidence that the more leeway you give coding agents the worse your codebase is likely to get. Anecdotal tales of discovering Claude has written four versions of the same function in different places, finding giant sections of code that never get called, and “bug-finding” agents that imagine and even create bugs are growing rampant. LLMs and agents appear to “know” your codebase inside and out, but they don’t actually understand anything other than statistical relationships between words and phrases in specific contexts. The LLMs themselves don’t even “know” they’re working with code.

When I listen to Accidental Tech Podcast and hear Arment getting so enthusiastic about coding agents, I admit I have trouble reconciling “I don’t like using libraries I haven’t written” with “I love having robots write code for me”. Sure, it’s absolutely possible for a human to check every bit of LLM-generated code; I have little doubt that the code in Big Walk is code House House fully understands. A lot of LLM-contributed code I’ve seen is short, clear fixes and improvements, absurd to point to as poisonous, incomprehensible slop. Reining in your agents flips the complaint in the last paragraph around: the less LLM-generated code you use, the easier that code is to vet.

The larger point here, though, isn’t about Big Walk, or Overcast, or how many long-standing minor bugs in macOS Claude fixed (or how many brand new minor bugs in macOS Claude may have introduced). It’s about treating AI-assisted coding as something that must be used now that it’s here, whether we see it as an unalloyed good or as a tool whose benefits outweigh its drawbacks, or even as an unavoidable evil. It is, as Dias writes, very possible to do things without using an LLM, even things that are hard.

One could use LLMs for coding in a limited, intentional way, and the language models we have now are mostly trained on open-source, publicly-available code, making them (arguably) less objectionable than generating images or music or annoying emails pitching vibe-coded Markdown viewers. But mostly does heavy lifting in that sentence—and even if the code is open source, that doesn’t mean it’s legal to train on. If Claude inadvertently incorporates a meaningful chunk of GPL-licensed code in your non-GPL software, you have a problem. I suspect this is a problem that already exists in thousands of codebases.

We can envision a future that solves the big problems with LLM-assisted coding. The models run locally, not in data centers that convert electricity and water into money funneled to Elon Musk and Sam Altman; the training corpus only contains code whose license allows training. I suspect any open source license that allows commercial reuse falls into this category, unless one adds an explicit “no AI” clause. That future could be right around the corner. But we have to deal with the LLMs we have, not the LLMs we wish for.

So…where does that leave me? Back where I started, I suppose. It’s practical for me to swear off LLMs for my work, but not practical to avoid it in every piece of software that I use. But I think it’s something that indie developers in particular should be more wary of than what I’m seeing in the community. As I was finishing this article up this morning, this post came across my Mastodon timeline, from the head of the venerable indie design/coding outfit Iconfactory:

And, you know, yeah, I get that. I can no longer make a living doing technical writing because of AI, and AI is bad at technical writing in precisely the same way it’s bad at everything: good enough to slide by if you don’t really care. It’s more than a little concerning that otherwise perceptive people don’t understand that the same issues that plague those AI-generated posters at their local brewpub apply to everything AI-generated, and everything includes code.

I’m not saying you can’t use generative AI in the creation of terrific apps; IGF award arguments aside, Big Walk seems to qualify. But LLMs, by definition, can’t make anything thoughtfully, because LLMs have no thoughts. Everything Lex wants—apps that feel crafted, that show refinement, that feel intuitive—requires a human developer not only with taste, but with understanding. And the more you delegate to your robot coding assistant, the less you understand. Again, the less LLM-generated code you use, the easier that code is to vet. And the least amount of LLM-generated code you can use is always zero.