Should an agent project start from an eval instead of the org chart?
A post going around argues that many company agent projects copy the org chart, one agent per department or role, and that teams should instead start by writing an evaluation of the task they want done. Is eval-first the better way to design agents, and what does that look like in practice?
Mostly yes. Splitting agents by job title gives you a diagram that looks sensible but says nothing about whether the work gets done. An eval forces the real question first: given this input, what counts as a correct result, and how would you check it?
In practice that means collecting a few dozen real examples of the task with the expected outcome, writing checks that can be scored automatically where possible, and adding a model-graded check only for the parts that need judgement. Then try the simplest design, often a single agent with good tools, and measure it.
Add more agents only when the eval shows a reason, such as one step failing because its context is crowded out, or two parts needing different tools or permissions. That way every extra agent has to earn its cost in latency, tokens and debugging.
The eval keeps paying off later: every prompt change, model upgrade or new tool gets run against it before it ships.
Listings mentioned
- Evals · skill by danielmiesslerAssertion-first eval framework with deterministic checks plus a structured model judge, based on Anthropic's agent eval guidance.
- Agent Designer — Multi-Agent System Architecture · skill by alirezarezvaniHelps pick an orchestration pattern and evaluate execution logs for cost, latency and errors.
- Deepeval · skill by confident-aiWorkflow for adding DeepEval tests to agents, tool-using flows and RAG pipelines.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.