Serious coding with LLMs. Lab notes 2026-10-21: The throttling of subscriptions by OpenAI and Anthropic is really happening

To find out more about the reality of all the (generally overblown) promises about LLMs, I have been doing a very serious experiment on the poster child for Generative AI: coding. I wrote one critical post, and I am close to writing up my conclusions, but I am hitting some work that I have to do first before I can publish that (e.g. create a demo video which is a lot of work, and before that I need to finish some final functionalities and improvements.

This is Lab Note #5. Some basic explanations about my setup are in Lab Note #1.

Context: As an experiment to learn about the limits of LLM-coding, I am building a professional visual modelling application from the ground up. It can handle multiple grammars (definable in JSON). It has state of the art graphics. It is rich in what users can influence and how (e.g. schemes, style/colour/geometry properties). It is responsive. It is well documented. It has a solid software architecture. Every line of code (and many of the other texts) have been generated by the LLM (so far almost exclusively Claude Code). It runs on macOS and Windows (Linux will be added later). If I finish it and I am satisfied, it will be open sourced. This is a serious sized project: 140k lines C++, about 15k lines stable documentation and 30k lines of temporary documentation (analysis, design and planning for substantial changes over the last half year), so about one third of what is in play is not code but documentation.

Warning: That I am doing this should not be read as either pro or contra coding with LLM-agents. I am highly critical of the hype, but I am trying to establish the limits of LLM-coding based on as serious a use case as possible. I can say already that my experience so far has let me conclude that LLM-agents will in no way be able to replace actual human engineers.

Note, these lab notes do not explain all technical details. I generally try to write for a general audience, but if you’re not a software engineer you’ll need to kind of ‘read over’ everything you do not understand, but you can still get the gist of it.

I am going to get a bit technical, but I consider this essential information and I can make it understandable, is my experience.

Introduction – Tokens

LLMs do not generate words or code, they generate tokens. A token is a small fragment of characters, stitch them together and you get words and sentences, or you get code, or you get mathematical reasoning (which, when coded in Lean, can be checked for formal correctness). LLMs work with a ‘dictionary’ of tokens that — when stitched together — can create all possible sequences of tokens and thus all possible texts, code, etc.

Here is an example of text as tokens (from OpenAI’s tokeniser):

GPT-4:
Many grammars don‘t have paucity of rules
GPT-5:
Many grammars don’t have paucity of rules

As you can see, each model may have a different ‘token dictionary’ and ‘tokeniser’. And the items the algorithms work on have nothing to do with actual words. If you see this for the first time, look at how a word like ‘paucity’ is broken up and how that illustrates that these models calculate not with patterns of words like ‘grammars’ and ‘paucity’, but tokens like ‘ars’ and ”uc’, as in “how likely is it that ‘ac’ follows a ‘ars’ 5 tokens earlier?”.

Here is how code may be tokenised. First the C++ code snippet from my code base (with a frivolous comment so a favourite example word — ‘paucity’ — turns up), then how GPT6-Astra breaks it up and how Claude Opus 5.5 breaks it up.

for (const auto& missingGrammarName : missingGrammars) { // Paucity is optional
  message += QStringLiteral("  - ") + toQString(missingGrammarName) + QStringLiteral("\n");
}

for (const auto& missingGrammarName : missingGrammars) { // Paucity is optional
  message += QStringLiteral("  - ") + toQString(missingGrammarName) + QStringLiteral("\n");
}

for (const auto& missingGrammarName : missingGrammars) { // Paucity is optional
  message += QStringLiteral("  - ") + toQString(missingGrammarName) + QStringLiteral("\n");
}

(For software engineers, that “\n” is broken up is somewhat hilarious as \n is a single thing: a newline.) Note, I had to rely on external tools for this code colouring, so I cannot guarantee complete correctness, but it doesn’t matter for the story.

Again, focus on the little fragments. That is what an LLM works on, even if we humans software engineers experience it differently as either language or code. There is power in that — costly, inefficient — approximation, but there are fundamental weaknesses that require the labs to engineer the hell around those weaknesses. (See The Department of “Engineering The Hell Out Of AI” from almost 3 years ago, and notice that this mode of operations is still the order of the day)

Introduction – Generation

LLM-generation is really simple if you look closely

If you look closely, LLM-generation is extremely simple:
1. it calculates a (hidden) set of ‘good’ potential next tokens based on what has gone before (the ‘context’) and the value of its (hundreds of billions) parameters;
2. From that set it randomly(!) selects one token.
3. It stops if the selected token is ‘stop’, otherwise it writes it out (the ‘continuation’) and goes back to step 1.

That’s it. Really, that’s it. It’s a bit like taking a dice that has only 5 or 6 on the faces, and throwing it over and over again and getting a high result. Training has loaded the dice. Generation (‘inference’) throws them.

The power and cost of it all sits in step 1: calculating a set of ‘good’ potential next tokens out of a dictionary of tens or hundreds of thousands based on the parameters and the hundreds of thousands of tokens that came before. When I am halfway into my 1 million token context window with Claude, and the token dictionary has 100 thousand entries, and each token is represented by a set of more that 10 thousand values, each next, single, individual token requires 500,000,000,000,000 calculations, each requiring multiple steps. There is a reason why this is expensive and why you need special hardware. To prevent this from happening for the entire context at each new token, the models keep the results of older still usable calculations around in memory (the ‘KV cache’), but that just means they need incredible amounts of memory (instead of on average another 500,000 times more calculations per token). There is a reason why memory is so expensive now.

Early models had only that. But the models have not only gotten bigger in terms of more complicated calculations with more parameters, but also with larger ‘dictionaries’ (the ‘goodness’ of each token in a dictionary must be calculated every round), parallelism (calculate many continuations in parallel and prune the ones that start to get lower value sets). And then the models themselves were used recursively, let a model generate many continuations and feed these into tests (trial & error, they call that ‘thinking models’…), let the model generate code to solve something.

Observations

• I am using a subscription of €100 per month. This comes with limits. Claude has both weekly limits and 5-hour limits, Codex only weekly;
• In July, using Claude, I was able to virtually spend almost €10k in tokens in my project with room to spare. Now, in October, based on exhausting a weekly budget, my maximum virtual spend for a month is about €5k. For Codex, my maximum virtual spend based on exhausting a weekly budget is around €2k.
• When I started experimenting with Codex recently it was not only s l o w but also not very good (it went off the rails doing things I never asked). In a previous lab note I mentioned I would stop using Codex as for this project it is much worse than Claude. A bit later I found out part of the reason why: while Claude has a default context of 1 million tokens, Codex only has a quarter of that as default. And the maximum you can put it on is around 828,000 (for a $100 subscription). With such a small default context is wasn’t strange it went off the rails more easily;
• But…, Claude’s recent (as of Opus 4.7) top models use much more tokens for the same amount of text/code/etc., about 30% more. Claude has taken a different direction. While GPT still has a larger dictionary where more often whole words are a token, Claude has gone back to a smaller dictionary, so it needs many more tokens on average to represent the same text/code/etc. The labs do not open up on dictionary sizes and token dimensions (see Generative AI ‘reasoning models’ don’t reason, even if it seems they do for details) so we have no idea of where the effort lands (GPU, memory). What we do know is that Fable 5 is about 4 times as expensive than the then top model Opus 4.6 was (3 times the per token rate, 30% more tokens) slightly over half a year ago

For the work I am doing. Codex with GPT6-Astra performs worse than Claude Code with Opus 5.5. Part is that Codex with GPT-6 really gets off the rails (incidents, anyone) because it goes after any potential result it sees (even outside the task, and the smaller context only makes that worse, because the command gets lost in context history.

The quality of the work that Claude does for €10k in July and for €5k in October seems to me to be roughly equal. Anecdotal of course, and a personal . At the same time, it takes more time to do the same things, again anecdotal and a personal estimate.

Both models are throttling subscriptions, GPT much more that Claude, which is probably a sign that the unviable economics is starting to bite and that both are trying to cut inference cost on subscriptions. I suspect the fact that they want to IPO is an important part of this. Personally, I estimate the economics will not hold up, so I am producing as much as I can while the party lasts, in a way that makes human maintenance possible (so well over half of my work is not on features and defects, but on code and architecture quality).

Your Max plan includes $100 a month in API credits to power your apps and agents. Claim them at https://clau.de/api-credits

Whatever you do: see Generative AI/LLMs/Claude/GPT/etc as (in the end very inefficient/expensive) approximating — and in some narrow cases surpassing — the result of understanding, without having understanding.

We’ll get back to code quality audits of the produced code a next time, I am still waiting for some results.

Previously on the LLM-coding experiment:

• First report: Anthropic/OpenAI may be spending more than $1000 for every $100 you pay them (7 June 2026)
• Lab Note #1: Serious coding with LLMs. Lab notes 2026-09-25: Codex vs Claude Code, Usage Limit Resets (25 September 2026)
• Lab Note #2: Serious coding with LLMs. Lab notes 2026-09-27: Codex succeeds, Claude reviews the result. (27 September 2026)
• Lab Note #3: Serious coding with LLMs. Lab notes 2026-10-01: Codex and Claude generate a shared analysis (1 October 2026)
• Lab Note #4: Serious coding with LLMs. Lab notes 2026-10-13: Codex vs Claude on the same coding task. (13 October 2026)

Subscribe? I don’t know if I am going to publish the appearance of new notes every time everywhere (it is a drag), so best to keep up to date with my observations and analysis on the Information Revolution is to subscribe (top right). You’ll get an email whenever I publish. These Lab Note are on a pretty high frequency schedule so far, but normally I haven’t been doing more than 1-2 a month.

(I also like to be read. Share?)

[You do not have my permission to use any content on this site for training a Generative AI (or any comparable use), unless you can guarantee your system never misrepresents my content and provides a proper reference (URL) to the original in its output. If you want to use it in any other way, you need my explicit permission]