To find out more about the reality of all the (generally overblown) promises about LLMs, I have been doing a very serious experiment on the poster child for Generative AI: coding. I wrote one critical post, and I am close to writing up my conclusions, but I am hitting some work that I have to do first before I can publish that (e.g finish some final functionalities and improvements).
This is Lab Note #8. Some basic explanations about my setup are in Lab Note #1. Some basic technical notes (on tokens, context) are in Lab Note #5.
Experiment: As an experiment to learn about the limits of LLM-coding, I am building a professional visual modelling application from the ground up. It can handle multiple grammars (definable in JSON). It has state of the art graphics. It is rich in what users can influence and how (e.g. schemes, style/colour/geometry properties). It is responsive. It is well documented. It has a solid software architecture. Every line of code (and many of the other texts) have been generated by the LLM (so far almost exclusively Claude Code). It runs on macOS and Windows (Linux will be added later). If I finish it and I am satisfied, it will be open sourced. This is a serious sized project: 160k lines C++ excluding tests, about 23k lines stable documentation and 40k lines of temporary documentation (analysis, design and planning for substantial changes over the last half year), so about one third of what is in play is not code but documentation.
Warning: That I am doing this should not be read as either pro or contra coding with LLM-agents. I am highly critical of the hype, but I am trying to establish the limits of LLM-coding based on as serious a use case as possible. I can say already that my experience so far has let me conclude that LLM-agents will in no way be able to replace actual human engineers.
Note, these lab notes do not explain all technical details. I generally try to write for a general audience, but if you’re not a software engineer you’ll need to kind of ‘read over’ everything you do not understand, but you can still get the gist of it.
This post has gotten a bit long and technical, so let’s do a
TL;DR
When you use an LLM to code (or for other things, for that matter) you encounter two things that change your work.
- One is that you’re offloading part of your thinking. This is nice, offloading (energy conservation) is something any living organism naturally craves. But offloading your thinking comes with two risks. The first is that you understand what is going on less and less and so become less able to steer the machine (and no, it isn’t as good that it can do this by itself, we can really see that). The second is that when it becomes unavoidable that you have to dig in, it is extremely hard. Offloading can turn into mental load.
- A sceond one is that the volume that LLMs can generate (which is the key of what they do) start to overwhelm you. You can get mental overload in many ways. From massive amounts of work trying to keep the LLM on track (examples given) and from having to consume and understand a lot of what is coming your way. The effect can be mental overload.
Both actually depend on the answer to the question if LLMs are good enough to be left to their own devices. For small stuff in code, this is definitely the case. But for complex large code bases? No way. So, when you offload your coding to an LLM, you must be aware that your’re onboardig risks, of which mental overload of your human engineers is only one, but an important one (which must be clear to anyone, but sadly isn’t to those that have dehumanised their workers by thinking of them as ‘resources’, by the way).
Coding with LLMs is an impressive experience
For a software engineer, the sheer speed with which LLMs can generate large volumes of functional code is overwhelming in more ways than one. It is overwhelming because software engineers know from experience how hard coding is, and how slow the process can be, especially when code bases are large. You also need to be concentrated all the time, because every small error is a potential big problem. And working for hours on end to produce good code is thus an exhausting activity. This is why software engineers have to slack off regularly, it is humanly impossible to keep this up constantly.
So the first time you put in a prompt for some functionality and you find that — in no time — the LLM has produced convincing-looking and functional code is a shock. And it becomes easy to become convinced that LLMs will replace you, or at least speed you up enormously. Before we conclude that, we need to dig deep, however. Hence my experiment about trying to get a feel for the maximum quality of code that the best coding LLMs can generate in a large project (from the ground up, which means existing code doesn’t work as an anchor)
In this experiment, I have now worked intensively for a long period on producing a very serious software system. I would not have been able to get to the point I am now in terms of the richness of the system without the LLMs, that much is absolutely true. By ‘burning’ over 40 billion (with a b) tokens, on paper costing €30k (in reality the subscription has cost me 2% of that, or €600, so 98% of my LLM use has been subsidised, mostly by venture capitalists and private credit — thanks, all!) I have been able to create in 6 months something I would not have been able to do in the slightest. The volume alone speaks for itself: 320k lines of code and documentation.
But there is another price to be paid than the monetary one, the mental one, and it comes in different forms.
The first is the cost of mental offloading, mostly in terms of risk. Mental offloading is very attractive for a human. Thinking and concentration is hard and costs energy, and energy conservation is a key driver for any living being. So, we enjoy it, we hunt for it. At the start of my project I did some research on the best graphical framework for the presentation layer and because I wanted portability, I quickly came to the conclusion that my only real option was Qt, which is state of the art but also has a free, open source, tier. As Qt is a C++-based framework, the choice of development language was therefore natural: C++ (which I had some very rusty experience with — I remember when it was new in the 1980’s…, this project really is a bit like waking up my old roots). Later, when I investigated auto-layout, I came across a state of the art library called OGDF (though, it is a university project and the code quality is sometimes questionable, like there begin ‘dead code’ in it). In the past, I would have to learn them all: Qt, OGDF, and many more elements of the project. This time, I did not have to. The textual code patterns represented by the parameters of the LLM can handle this fine, it seems (more about that key question behind this experiment another time).
There is a catch, though. There are at least two:
- The first one is that is time goes on and the volume of logic grows, I become less and less able to give well-founded guidance to the LLM, and the LLM needs guidance, it is completely unable to do this kind of work on its own (more on that also another time). I am ever more relying on shallow knowledge about the frameworks and code, coupled to deep (but ancient) generic engineering insights. And as the effect of my knowledge becomes shallow, the risk of the LLM careening out of control and generating ‘poor’ patterns increases.
- The second one is that in moments my intelligence is needed, I can be completely out of my depth. I have a very limited insight in what the consequences and risks are of the decision I make. The LLM may ask me for a decision, but before I can make one I need to dig in deep, or lose control. And at that stage, making decisions becomes harder because I have offloaded to the LLM. If I offload my thinking, I onboard risk. In general:
When an organisation offloads its thinking, it’s onboarding risks
I am specifically mentioning ‘organisation’ here, because when organisations are conscious about their risks they know they have to — as an organisation — compensate. And serious ones will. For now: I don’t see a lot of broad thinking about compensating for the risks of LLM-coding, the (imagined) advantages dominate the discussion.
Intermezzo: productivity of software engineers…
LLMs are capable of producing huge volumes of code at a speed no human can match. Basically that means code production speed goes up. I cannot match producing on average 1600 lines of code and documentation in a day for 180 days (my own experiment). When Frederick Brooks wrote in The Mythical Man-Month about productivity in the 1960’s, the average was about 10(!) lines per day, but the diversity was large, influenced by the size of the code base (a few lines of code per day in a million-line project, a few hundred in a small one), and the individual differences, which could be up to a factor 10. This metric stayed roughly the same until today (not taking LLMs into account).
The productivity for software engineering is no longer measured in lines of code produced these days, and for good measure. E.g.:
- Actually, more elegant, smaller code may have all kinds of benefits on maintenance and development in the long run, so removing code, negative productivity in terms of lines/statements, may actually be a good thing;
- The amount of functionality coming from a line of code or a statement depends on the level of abstraction. Assembly has very little. A call in a higher order language of some framework function can have a lot. Both are single lines/statements typed by the software engineer, but understanding one requires much more energy than understanding the other.
So, with the advent of ‘agile’ development methods, engineering teams started to think in measures of changed functionality or quality and measure in ‘function points’ and later ‘story points’. What is the ‘size’ of this change? And engineers become experienced in estimating that (an art, not a science) over time and as a team even more so (‘planning poker’ and all that).
One to remember: it is not so much typing speed that defines software engineering productivity.
…and of LLMs
The labs told us often this year that their code generators would generate maybe even 80% of the code of their systems. That sounds impressive until you realise that the old ‘lines of code’ seem to be back (there are some signs in Claude Code that they are trying to change that), and added to that, in a sense, ‘typing speed’. And what is worse, with the speed/ease these systems edit, and the fundamental unreliability at the heart of LLM-generation, what would done by a human programmer as one big ‘commit’, may take the LLM 10 or even a 100 times that many changes while it is approximating it’s result. My experience is that a Claude Code really likes to commit changes fast and often, as it gives it a context later on that it can use. On my own project, I resisted that at the start (only commit finished work) but I had to relent. And the number of lines changed are a multiple of the difference between the code then and the code now.
The LLM generates large volumes of code. Really large. But as the engineer — or the organisation in general — I have to consume all that code. I need to check it, work with it, test it, think about it, and propose changes in architecture, behaviour (both of the system and tools in the project), and write that up in good prompts and other forms of ‘context‘.
The volume/speed advantage the LLM thus also directly leads to a form of mental overload, for me personally. This is related to the mental overload to the energy the receivers of AI-slop and AI-sludge have to spend, but it is actually independent of the quality of what you get. Unless the systems become so good that they can replace humans, which isn’t the case (and isn’t going to be the case either). Have a look at this:
Software quality (again)
Warning: this part is pretty technical. Software engineers may find this interesting, the rest may find this gobbledygook. After this, I will return to mental load.
Software quality is the central question of my experiment. Here is a snippet of code (a function definition) generated by the LLM:
std::vector<std::string> EACMP_legalElementVariants(const EACMP_ModelElement* element)
{
// the current element type's legal variants (see variantsFrom)
if (!element) { return {}; } // GR-3: no element
const EACMP_Model* model = element->model();
const EACMP_GenericGrammar* grammar = model ? model->grammar() : nullptr;
if (!grammar) { return {}; } // GR-3: a detached element has no grammar to ask
const std::string currentType = element->typeName();
const EACMP_ElementType* node = grammar->uncachedGrammarElementType(currentType);
if (!node) { node = grammar->uncachedGrammarElementSubType(currentType); } // leaf under a root; a root type found itself — no 'else' branch
return variantsFrom(node, currentType, element->variantName(), EACMP_legalElementTypeChangeTargets(element));
}
Looks good, doesn’t it? A routine, it nicely checks the parameter it gets (in bold) and returns an empty structure when this happens. At first sight, good, robust code. A mechanical quality scanner will be glad: parameters are checked, no chance on referencing a null-pointer here.
But in reality this code really breaks the quality instructions I had the LLM working with for many months now. One of these is my rule — born out of good software engineering education and years of experience — that we do not compensate for situations that can only happen if there has been a programming error, or maybe better a programmer error. Getting a non-existing element, or a, element without a model, or a model without a grammar are all situations that should never be able to happen. You need to code in a way that maximally minimises the chance of this happening, that is, every time this happens, the error is so bad you should not try to carry on with a situation that is clearly completely messed up. That is the ‘contract’ for a routine like this.
Now, it is possible to make such errors, of course (LLMs can too), but your reaction should not be ‘make the best of it’, that is the wrong kind of shortsighted robustness and it leads to a complete mess of code, as the ways things can go wrong are infinite (I did mention good programming is hard, did I?). Besides, returning a null pointer may make sure this routine doesn’t crash the program, but compensating here for bad parameters which should not exist in the first place only moves the problem up to code that then either has to check this itself (everywhere where this routine is called, which may be hundreds of places) or keep on working with that null pointer.
Instead, when programming, you should guard against your own errors, but you should be completely unforgiven. And in part you do that by maintaining ‘invariants’: states that are guaranteed to be true. E.g.:
- I-1 A model that lives in a landscape has a non-null
grammar(). A model with an unresolvable grammar is never created — the loader refuses the whole landscape (loading without the model and later saving would silently drop it),addModelrefuses a null grammar, the model’s constructor asserts it,setGrammarrefuses it, andreloadGrammarscloses a landscape whose grammar went missing. The grammar is a re-resolvable referenced resource (re-bound on reload), not an owner, but it is never null for a live model. - I-2 A model item (
EACMP_ModelElement/EACMP_ModelRelation) reachable throughmodel->elements()/relations()has a non-nullmodel()back-pointer. - I-3 A view item (
EACMP_ViewElement/EACMP_ViewRelation) has a non-null model-item back-pointer (modelElement()/modelRelation()): the model item is a constructor parameter, which the constructor asserts,addElement/addRelationrefuse a null model item, and there is no setter, so an orphan view item cannot exist. - I-4 A model or view that is part of a landscape has a non-null
landscape().
These invariants are guarded in the code, so if we get a proper element, we can trust that it has these references. So what about when we get a ‘null’ element? Then something has gone wrong too because this function should never get that. When a program runs into something that is referencing a null pointer the operating system will crash it. When programs want to prevent that from happening, they check and then end the program in a civilised way.
By the way, don’t you hate it when a program goes down like that? Even when it produces some sort of gobbledygook error message about a ‘program error’ with the only option to press the ‘OK’ button?
It’s not a ‘program error’, it’s a ‘programmer error’. Take responsibility, please. And really: clicking the panel away is not ‘OK’. So, EACMP does this instead:
Actually, since all the code is LLM-generated the meesage could read “The LLM generated code with unrecoverable errors and the engineer did not catch it”.
Anyway.
So our function becomes:
std::vector<std::string> EACMP_legalElementVariants(const EACMP_ModelElement* element)
{
// the current element type's legal variants (see variantsFrom)
EACMP_BREAKING_UNRECOVERABLE_INVARIANT(element != nullptr, "EACMP_legalElementVariants takes an element");
const EACMP_GenericGrammar* grammar = element->model()->grammar(); // I-2, I-1
const std::string currentType = element->typeName();
const EACMP_ElementType* node = grammar->uncachedGrammarElementType(currentType);
if (!node) { node = grammar->uncachedGrammarElementSubType(currentType); } // leaf under a root; a root type found itself — no 'else' branch
return variantsFrom(node, currentType, element->variantName(), EACMP_legalElementTypeChangeTargets(element));
}
Now, if the element is not null, but it’s references are, it will still be crashed by the operating system. But the guards against that situation in other parts of the code should make that impossible. Of course, we can make errors in those guards too, but you can understand that if we do this at all levels, the code becomes much more robust.
For completeness sake: here are the guards in our documentation:
4.1 Guard reasons
A guard whose condition can legitimately be false is not an invariant violation: it stays a
runtime guard (refuse / skip / return), commented with its GR-n. These are the only sanctioned
reasons. A guard that fits none of them is guarding an invariant and belongs to §3, not a silent
skip.
| Code | Reason | Typical sites |
|---|---|---|
| GR-1 | Boundary input not yet validated — data from disk, import, paste, clipboard | loader, import, paste |
| GR-2 | Resource availability — a grammar (or other named resource) referenced by name may be uninstalled | open, paste, reloadGrammars, addModel |
| GR-3 | Stale lookup — a UUID held across an operation/refresh may no longer resolve. It is the test of what a lookup just returned, never of a parameter: a pointer a caller hands in is the caller’s to guarantee (§3.4) | selection after delete, browser refresh |
| GR-4 | Optional absence / user cancel — an optional field is absent or a dialog was cancelled | optional JSON fields, dialogs |
| GR-5 | Transient construction order — an object observed before it is fully wired during two-phase construction | two-phase load / undo restore |
Code-comment form: // GR-2: grammar may be uninstalled.
The code that the LLM generated looked great but was poor in the context of the project. Not only was it blindly compensating for programmer errors that should not be able to exists (so ‘dead code’), it also had documented them with ‘GR-3’, which — in this place in the code — is completely wrong. We have validated data at this point. GR-3 is only a valid issue when we are loading external data and check if it is correct. Why did it use the GR pattern here? Because it was used in that other part of the code base (on import) maybe?
This is a good example of the difference how blindly following a pattern differs from understanding code. Not just the pattern that have been used to configure the LLMs parameters (‘training’), but also those that are in your own code base (‘context’).
The code that the LLM generated looked fine but was poor in the context of the project
Even with all the instructions I have given the LLM, it still has a hard time doing the right thing. Regardless if you agree with my approach to code quality, the issue here is that the LLM is not following them, or only following them in a shallow way. The result is that the code may not be slop, but it is still sludge. A lot of time and energy spent by both me and the LLM is addressing the code quality issues I detect. That illustrates that the LLM doesn’t really understand code quality, but that if I am in control I can make it generate better results. With the amount of code the LLM generates, this leads to a mental overload on my end, however.
Conclusion
I had Claude Code do a rough analysis on where its effort went the last seven months, based on the git repository and the LLM usage data.
August was partly vacation. September was spent on getting a Windows version of the application, getting Codex (abandoned now) to work, creating ‘harness’ tooling to help with improving code quality triggered by the IfSQ code inspection, so hardly any features and a lot of work (tests, inspections) that stayed outside of the LLM generation. Notice: the same amount of tokens cost (virtually!) half in October than what it (virtually) cost in July, much of this seems to have been efficiency improvements in Claude Fable and Opus. But what you see here is quite clearly that the amount of work on ‘quality’ is far from negligible.
Aside: can you guess why so many LLM-coding projects use the programming language Rust (“technology from the past come to save the future from itself”), these days? That is because rust is about the strictest language around catching many coding errors during compile (C++, that this project uses, is at the other end of the spectrum). That really makes life easier for LLMs (and humans, of course), because bad constructions are more often caught by the compiler.
I have, I think, functionally great results reached in a relatively short time. But keeping the top frontier LLM models from producing sludge — in large amounts — is extremely mentally tiring work and the architectural issues keep coming. Organisations will need to watch for a burn-out of their engineers. I know I will need to. And that is hard, because a software engineer is generally addicted to the rush of completion and progress (for which volume stands as an ever less valid mental shortcut)
Previously on the LLM-coding experiment:
• First report: Anthropic/OpenAI may be spending more than $1000 for every $100 you pay them (7 June 2026)
• Lab Note #1: Serious coding with LLMs. Lab notes 2026-09-25: Codex vs Claude Code, Usage Limit Resets (25 September 2026)
• Lab Note #2: Serious coding with LLMs. Lab notes 2026-09-27: Codex succeeds, Claude reviews the result. (27 September 2026)
• Lab Note #3: Serious coding with LLMs. Lab notes 2026-10-01: Codex and Claude generate a shared analysis (1 October 2026)
• Lab Note #4: Serious coding with LLMs. Lab notes 2026-10-13: Codex vs Claude on the same coding task. (13 October 2026)
• Lab Note #5: Serious coding with LLMs. Lab notes 2026-10-21: The throttling of subscriptions by OpenAI and Anthropic is really happening. (21 Oct 2026)
• Lab Note #6: Serious coding with LLMs. Lab notes 2026-10-27: A code audit of LLM-generated code: improving from ‘sludge’ to ‘vakwerk’. (27 Oct 2026)
• Lab Note #7: Serious coding with LLMs. Lab notes 2026-10-31: Teaser Demo of the LLM-coded EACMP application, and a quiz (31 Oct 2026)
Subscribe? I don’t know if I am going to publish the appearance of new notes every time everywhere (it is a drag), so best to keep up to date with my observations and analysis on the Information Revolution is to subscribe (top right). You’ll get an email whenever I publish. These Lab Note are on a pretty high frequency schedule so far, but normally I haven’t been doing more than 1-2 a month.
(I also like to be read. Share?)
[You do not have my permission to use any content on this site for training a Generative AI (or any comparable use), unless you can guarantee your system never misrepresents my content and provides a proper reference (URL) to the original in its output. If you want to use it in any other way, you need my explicit permission]
