How do you test a memory failure when every single reply passes on its own?

Score the session, not the turn. A per-turn relevance metric asks whether each reply fits the message before it, and a reply that repeats a rejected option can fit perfectly. The bug lives in the relationship between turn two and turn five, so the test has to see both.

Write session-level checks that encode what the user already said: rejected options must not come back, stated constraints such as dates or budget must still hold at the end, and facts given early must still be used later. Each check is a pass or fail over the whole conversation.

Build test conversations on purpose to trigger the failure: a rejection early, then enough turns to force the state to be compressed, then a request that tempts the agent to repeat itself. Replay real sessions that failed, too. Run each case several times, because these failures are often intermittent.

Grade the compressed state directly as well. If you can inspect what the summary kept, a simple check that the rejection is still in it often finds the root cause faster than grading the final reply.

Listings mentioned

• Deepeval · skill by confident-ai
  Evaluation workflow for agents and multi-turn chatbots, with a multi-turn end-to-end test template.
• Evals · skill by danielmiessler
  Assertion-first eval cases with pass@k and pass^k, so intermittent failures show up across repeated runs.
• Memory Engineering — engineer the forgetting, not just the remembering · skill by alirezarezvani
  Helps decide what an agent memory should keep and what it can drop, which is where this failure starts.

Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.