Thought of a few more things on summarization, truncation, committing and process 🧵
I really liked how we implemented summarization from a continuity perspective though it definitely wasn't optimal from a cost perspective.
When we hit the threshold to summarize (150k iirc) we actually didn't summarize the whole conversation. We summarized the first half
So if your original conversation was: [system] [messages 1-50] [messages 51-100] We'd summarize it to: [system] [summary of 1-50] [messages 51-100] [message saying a summarization happened]
ofc the dividing line was determined by token count not index but you get the idea
With this system I personally never even noticed when the agent summarized. I'd often go back and check the agent page and learn it summarized twice without me even noticing.
When showing the agent information there's a tension between truncating too much or too little. Every extra token costs something in dollars and performance. But cut too much and the task becomes impossible
A trick I got from someone at @METR_Evals was to save the information to a file and show it the truncated version. That way you can truncate aggressively and if the model wants to know more it can read the file
Be sure to communicate where the file is clearly!
We only used this trick for GitHub workflows and shell output. Thinking about it now we probably should have used it for all the github events we rendered.
Claude Code uses this for the background shells but imo they truncate too much. e.g. if I ask claude code to run a server in the background and then curl it, the background task fails because it uses a library I don't have installed but it doesn't realize until it runs curl. And…
At the heart of our agent was an action commit_and_push that did a lot of things: * Checked out a nice mentat/ branch name * Ran user specified formatters * Committed the work * Pushed it * Ran an "auto reviewer" which looked at the diff and gave feedback (opening a PR was…
I think this idea may be difficult for other agents to implement because they don't have a clear idea of what a "unit of work" looks like. For Claude Code/Codex completed work is simply a mutation to the local file system.
But having a time for automated checks to run, both formatters and LLM review was very nice. I guess the "Claude Code" way to do this would be a skill?
To prevent the agent from running git push or gh instead of using our tool we actually aliased them in our shell to echo a reminder
A tip on process is snapshot tests for prompts. As they become more complicated template and string concatenation messes it becomes easy to change them without realizing. Snapshot tests catch those errors.
For every if-else statement in the prompt you should have a good reason why it's not in code
For instance we had slightly different versions of our commit_and_push prompt based on whether the repo had github workflows or a formatter script
The more you do there the more prompt snapshot tests will help you
The snapshot test also becomes the only place in code where the full system prompt is written out which is very helpful
fin.