I too, suffer from context collapse
…And the solution is to sleep and come at a problem “with fresh eyes”.
LLM coding agents are known for this failure mode - “context collapse” - where they have been working* in a certain codebase, on a certain implementation question, for an extended time. It’s been observed that their solutions get worse and worse until the developer-driver has to reboot the LLM by starting a new chat or agent session.
* And for an LLM, by “working”, I mean receiving files as input and outputting tokens
I was fighting with a unit test last Friday. I left on my todo list “LookupVehicleAsync_FullDescription_WhenValid_GetsData is failing locally??” to solve on Monday.
At the time I was blinkered, focusing on a parts of my mock setup which I was convinced were the culprit, because I had been working intensively on them to satisfy every other test.
For reasons, I have three methods,
GetCapModelAsync,GetCapVehicleByCapCodeAsync, andGetInsuranceGroupAsyncthat do similar things with different input and output types. I had comprehensively mocked the first two. The third one was the one actually used by the system under test 🤪
It was only on Monday when I came back to it, drilled down through the method called in the test, explored the related references, that I realised I had neglected to mock the method that was actually being called under the hood in my test.
By 09:23 on Monday I’d fixed and committed two unit tests that had been holding me back.
Humans -> Sleep, as LLMs -> New Chat
Not all that surprising. 🛌👋