Do you spend more time fixing your agent than using it?
A teammate runs an agent, spots something slightly off, tweaks a prompt or a setting, runs it again, then fixes the next thing and runs it once more. It finally works and it feels like a win, but most of the week went into adjusting the agent instead of using it. Is that normal, and how do you break the loop?
The loop happens because each run is judged by eye, one case at a time. A tweak that fixes today's case can break yesterday's, and nobody notices until it comes back. That is why it feels like progress and never ends.
Write down ten to twenty real inputs and what a good result looks like for each, before you change anything else. Keep the checks simple where you can: did it call the right tool, did the output have the required fields, did it stay under the length limit. Use a model as judge only for the parts that really need judgement.
Then change one thing at a time and rerun the whole set. If the score goes up and nothing that used to pass now fails, keep the change. If not, undo it. Ten minutes of setup turns a week of fiddling into a few short runs with a clear answer.
Decide in advance what 'good enough' means, for example nine out of ten cases passing. Once you hit it, stop tuning and use the agent. New failures from real use go into the test set, not into another round of guessing.
Listings mentioned
- Evals · skill by danielmiesslerAssertion-first eval setup: fixed test cases, simple checks and an LLM judge only where needed.
- Deepeval · skill by confident-aiAdds evals to an agent or LLM app so every change is measured against the same cases.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.