VitaBench 2.0: even top agents fail half of long-horizon tasks
Meituan's MIT-licensed benchmark is pip-installable and points at your own stack — run it this weekend to see where long-memory agents quietly break.
Copy markdown
A free benchmark for the thing agents keep getting wrong
VitaBench 2.0 tests whether an agent can infer a user's preferences across fragmented chats spanning days or weeks — 56 simulated users and 771 subtasks across delivery and travel domains. It's MIT-licensed on GitHub and Hugging Face: install the vita CLI, point it at any OpenAI-compatible endpoint, and score your own agent.
The verdict: ~50% is the ceiling, even for frontier models
On the full-context leaderboard, Claude Opus tops the thinking models at 0.503 Avg@4 and DeepSeek-V4-Pro leads non-thinking at 0.456 — the best agents still miss roughly half of long-horizon, proactive tasks. Treat "it remembered last week" as something to verify, not assume.
Swap the memory backend and watch the score move
The harness ships three memory architectures — Full Context, Agentic Memory, and RAG — so you can A/B how retrieval choices change reliability instead of guessing. A concrete weekend project: run your production agent's memory stack against it and surface the failure modes before your users do.