Sad we couldn't make the business work. Thought I'd share some of my favorite ideas of ours on agent design 🧵
Async shell: Our shell action starts a shell session. We even blinged out PS1 with cd, git branch and exit status. * We block the agent for shell output while it's still producing output +5s after the last output, maximum of 1min. * The agent uses the same tool to create new…
In general this subscription pattern is very useful. We also used it for user messages and github events. As the agent worked we'd show it new activity from github e.g. CI completion, comments as it happened.
Some events like CI failure would wake it up from a pause. Others like CI success we'd wait to show it until it was woken by something else.
To say one more thing since I think this is incredibly important and still not generally done: We don't think running things as a function which takes a command and returns stdout. We think of it as starting a task that could be as long running as the agent itself.
Claude code is moving in this direction slowly but ought to "update all the way"
Agent exit surveys When our agent pauses we ask it for feedback on the harness and whether it believes it succeeded
I caught so many bugs and had so many small polish ideas reading them
Re-using the conversation prefix for the survey and waiting 4 minutes let us extend the user's cache to 9 minutes
Note we didn't add the survey to the main conversation or bill the user for the llm call. So outside of extending the cache this wasn't user facing at all
Pause for feedback trick: in early versions of our remote agent we had issues where the agent would pause without pushing its work, allowing the user to see it. We tried various prompts but what ultimately fixed it was when the agent paused if the git status was not clean or…
General principle: prompts in your tool's errors/warnings are more useful and testable than a random sentence in your 2000 word intro prompt
find/replace editing: Now everyone's edit tool works by find/replace but it felt like a competitive advantage in Fall 2024. Something we did that is still unique: If the find part matches something except for indentation we apply the missing indentation to the replace and still…
Really just so many little things. Probably should have spent the time marketing instead
Part 2