#5 It is bad to switch models during coding. Why so?
Guilty.
A few times in a rush and being a bit lazy I’ve switched a model in the ongoing coding session. Maybe I’m not alone and maybe some of you don’t know how harmful this practice could be. It is not JUST about costs and JUST losing some context.
A “session” is local/cloud file created, structured, and maintained in model+provider+IDE-dependent way. In other words – imagine an endless line of assistants in a bank and you want to get a business loan, each time you communicate with one you pass on the documents collected and what seems like continuous communication is a discussion which starts over and over (only the paper pile in your hands grows).
So, what is in that file? Probably, all your life as you're looking for funding your dream.
But practically there are model-specific system prompt(s), tool' schemas, pairs of tool calls + results, chat history, summary of the history, and a very important even personal part...
(Controversial statement) Coding typically requires a bit of thinking. And model generates not only output tokens but reasoning trajectories – encrypted or signed thinking blocks.
With the loan analogy - system prompt is a conceptual level instructions based on the current regulations, company policies, company strategies who gets the loan and under what circumstances. Then goes your credit history, net worth estimates, business evaluation, your business plan (pure guesses from my side), and in the reasoning part we have assistant attempts to solve your request, some intermediate decisions, and conclusions.
When you’re taking your pile of documents and decisively going to the other bank, “new” assistant tries to make sense of what you’ve collected so far while their institution operates differently on all levels. And bigger switch between providers makes it drastically worse.
But maybe you’re thinking - “What might actually go wrong? Current models are super-smart (let’s agree to use the term “intelligent” wisely) - they should have to handle each other’s generated output!”
Cost-wise as you probably know KV prefix cache is also per model/per deployment. Once you were went to another bank they literally could not fully interpret what the other assistant was making of your request. They see that you were asked to provide a reference letter from your theology professor but why is a mystery. And it still is, by the way.
What are the options? Throwing away all you could not interpret is a popular choice. And then you just patch the gaps. Simple, heh?
Despite expectations, model B does not critique or re-evaluate the conclusions made by model A, it just go on using them. It inherits errors, as it relies on the generated summary of the long chat, it calibrates efforts based on what it consumes, so for instance a better model might inherit simpler behavior from a weaker one.
It means at the end of the day you might not get a loan but get buried in the documents and burn your token budget.
Once we had a mathematical competition in our school. There was a task we could not solve, and the teachers were trying to solve it themselves. No success.
The issue was not in the task but in the communication - they were telling each other the details of the task in the wrong way from the beginning. The madness stopped when a newly hired teacher got back to the original challenge and solved it. And this was a strong deliberate choice which initially met lot of push back from others; the emotions were high and reputations were being shattered.
To sum up, while there are differences between family models, providers, IDEs, mid-session model' switches cause a silent eroding degradation of the quality of your work.
Just start a new chat.