I have been doing a very serious experiment on coding with LLMs. I wrote one post, and I am close to writing up my conclusions, but I am hitting some work that I have to do first before I can publish that (e.g. create a demo video which is a lot of work, and before that I need to finish some final functionalities and improvements.
Context: I am building a professional visual modelling application from the ground up. It can handle multiple grammars (definable in JSON). It has state of the art graphics. It is rich in what users can influence and how (e.g. schemes, style/colour/geometry properties). It is responsive. It is well documented. It has a solid software architecture. Every line of code (and many of the other texts) have been generated by the LLM (so far almost exclusively Claude Code). It runs on macOS and Windows (Linux will be added later). If I finish it and and am satisfied, it will be open sourced.
As it is taking me longer to get my application done, I decided I could create a short post more regularly with observations and estimations/conclusions. As this is the first one, I am including a few basics so I can point back to this post later.
My standard process for LLM-implemented changes
By now, my standard process for a substantial change is, in short:
- have the agent write the analysis/solution in a document, have a conversation with the agent about the analysis until I am satisfied;
- have the agent add a plan to the analysis, have a conversation with the agent about the analysis until I am satisfied;
- let the agent implement and do mechanical testing (about 80 tests by now), as well as adapt the permanent documentation (design, manuals) based on the changes. I may interrupt this if I see things going the wrong way (which with Claude these days seldom happens, by the way);
- do manual testing and fix bugs.
- when I am done, the change-specific document is retired
Once in a while I let an agent do a full review and handle fixes, refactors, changes that come from that review. Smaller changes I might attempt without the document.
Usage measurements and limits
I am using ccusage to keep track of the tokens I am burning and what it should have cost me if I paid API-rates. I am generally using a $100/month subscription giving me access to the top models. Now, with Claude Code, I have been using a multiple of that in ‘tokens’. My most heavy-use month so far was July, in which my token usage would have cost me $9578.94, had I had to pay API-rates. That was 100 times what I actually paid…
Now over to the actual lab notes of today.
Codex (GPT-6 Astra) experiences
From April 2026 on I have been using the top models of Anthropic, lately that has been Fable 5.1. But I wanted at least to do a period of OpenAI’s Codex. As my experiment is about ‘what is maximally possible’ that means Codex with GPT-6 Astra. Observations:
At first sight, Codex and Claude Code are comparable. I started with Codex with a bug in the undo/redo setup, which Codex quickly cleared up. It also suggested that my undo/redo system had structural deficiencies which a refactor (a redesign and rebuild) of the undo/redo system could solve.
So, I had Codex create the analysis and the plan and asked it to implement. Such a substantial-sized change I have done multiple times with Claude Code, and an initial implementation with Claude Code would generally be quick.
So far, working with Codex is an extremely poor experience.
First: Codex is slow. Really s l o w. Something that Claude would do in a few hours max, Codex is still working at days later. And while it is slow, ccusage also reports lower (virtual) API-cost. It seems that Codex is extremely throttling compute. As a result, the first implementation attempt took several days of continuous work by Codex.
Then, after approximately 3 days, I looked if it was finally finished and it had run into the weekly limits. And still it was not done. When I investigated I found out it had completely lost its focus and had started doing different things to the code, not related to the undo/redo task at all. This wasn’t recoverable, so I had spent a weekly allowance with nothing to show for it. Here is a part of the conversation (which could only proceed because I could execute a usage limit reset, more about that below):
■ Conversation interrupted - tell the model what to do differently. Something went wrong? Hit / feedback to report the issue.
› What are you doing? You were supposed to fix issues with undo/redo (an isolated refactor of the behaviour) and now it seems you are overhauling the entire application. You were asked a simple question and you start editing again?
• I lost control of the scope. I was replacing scattered undo snapshots with shared capture and replay, but expanded that into geometry maintenance, gesture handling and nesting changes. I should have kept existing application behaviour fixed. You then asked a question about rerouting. I should have answered it and stopped—not resumed editing. That was my mistake. I have stopped. Nothing has been committed. The latest command/interface cleanup is partially applied, with callers still awaiting conversion, so the current source tree is not verified buildable. The passing tests I reported predate those latest edits. I will make no further edits or reversions without your direction.
This has lead to working with Codex to build more guardrails for it, so it will not lose focus so easily. I have now hooks installed that once every 5 edits and once every 10 commands read a file reminder.txt (if available). That is a file I can use to put in instructions which are read regularly when it is doing a long task. I never needed this with Claude Code.
We’re now again 13 hours on and Codex is still running to try to finish the task and according. to its own estimate, it’s not even halfway. And it has spend 50% of the weekly budget already.
As it stands out, Codex will be completely useless for me on this project on a $100/month subscription. It simply gets nothing done. Of course, my project is designed to test the limit of ‘serious complex coding’, but if this had been my Claude Code experience in April-May, I would have given up months ago. If Codex is unable to complete something that Claude Code would have finished (and rapidly), I have no option than to go back to Claude Code.
The curious case of ‘Usage Limits Resets’
The subscription-based accounts have ‘usage limits’. These come in two flavours: a 5-hour limit and a weekly limit. How much these in reality are in tokens etc. is anybody’s guess (though experimenting to find out is possible, but I want my code, not spend much on that).
Codex has something called ‘Usage Limit Resets”. When you have a Pro or higher account, you get one reset for free each month. What this does is basically start a new 5-hour or weekly period. Codex is so throttled that the 5-hour limit becomes meaningless. My task can be running to days on end without hitting it. But before I could actually have that conversation (and everything that followed) above, I put it into action.
I noticed that Anthropic has also now introduced the feature.
Now a ‘usage limit reset’ enables you to go on when you hit a limit. That sounds nice, but it seems to me that use is connected to lower actual limits. In other words, it sounds like an advantage, but it may simply be a pacifier necessary because of lower limits.
My estimate
I guess what we are seeing here is the outside view of the economic unviability of OpenAI’s and Anthropic’s frontier model operations. They must offer the frontier models, because they are selling a promise that this AI is very good, almost AGI/ASI even. But they are in different positions:
- OpenAI is planning an IPO in Q1 2027. It has to survive and it is bleeding cash. It is also deeply unprofitable because it has all these users on a free tier that they cannot really convert to paying customers. So, their answer is throttling, announcing that they’re slowing down training, etc. They need to be able to paint a picture where their losses are getting less. So, if they are able to do that in the second half of 2026, they may paint a rosy picture come the IPO.
- Anthropic is racing to an IPO in Q4 2026. They have had a ‘good’ first half of 2026 with tokenmaxxing and they reported a profitable month earlier this year. They must not be seen to get worse so shortly before the IPO. But the ‘usage limit reset’ they now have introduced too worries me. Why introduce that unless you are going to drastically (like OpenAI) decrease the limits for subscribers? I expect (more) throttling by Anthropic soon.
Furthermore, if the actual quality of the top effort of the top models is like my Codex experience now, Recursive Self-Improvement (the idea that LLM-Agents will be able to improve themselves and so become AGI or ASI) is utterly laughable. That not only follows from benchmark numbers reported by OpenAI on Astra’s performance (around 70% success rate on a set of benchmark tasks, so a chain of a measly 100 or so means that there is no chance in hell (0.3 to the power of 100, which is effectively zero, or if you want to be more precise, the chance is ~0.000000000000000000000000000000000000000000000000005%) for. Recursive Self-Improvement to become real any time soon.
Is AI going to kill us all? No, really no. Are LLMs able with massive trial and error to find and exploit vulnerabilities? Yes they are, but that is not because they are intelligent, it is because with such volumes of trial and error you don’t have to be.
Enron-like debt structures, massive operational losses, impossible promises, and a dose of pure BS. You gotta love this ‘industry’.
