[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-95006":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":17,"stars30d":17,"stars90d":15,"forks30d":15,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":22,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":35,"readmeContent":36,"aiSummary":37,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":38,"discoverSource":39},95006,"HarnessEval-W","MirroS-Lab\u002FHarnessEval-W","MirroS-Lab","HarnessEval-W: Agentifying the Evaluation of Visual Worlds","https:\u002F\u002Fmirros-lab.github.io\u002FHarnessEval-W\u002F",null,"Python",248,7,1,0,2,113,24,73.71,false,"main",true,[24,25,26,27,28,29,30,31,32,33,34],"agent","benchmark","evaluation","harness","harness-engineering","kling","minimax-h3","seedance","video-generation","wan","world-model","2026-08-24 04:01:23","\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fmirros-logo.png\" alt=\"HarnessEval logo\" width=\"200\">\n\u003C\u002Fp>\n\n\u003Ch1 align=\"center\">HarnessEval-W\u003C\u002Fh1>\n\u003Cp align=\"center\">\n  \u003Cb>Agentifying the Evaluation of\nVisual Worlds\u003C\u002Fb>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Ci>The era of harnessed benchmarking: evaluation defines the taste of evolution.\u003C\u002Fi>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.16859\">\u003Cimg alt=\"Paper\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-Paper-b31b1b?logo=arxiv&logoColor=white\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fmirros-lab.github.io\u002FHarnessEval-W\">\u003Cimg alt=\"Project Page\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-5F38FF\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fmirros.ai\u002Fblog\u002Fharnesseval\">\u003Cimg alt=\"Blog\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FMirroS-Blog-745AE8\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fmirros-lab.github.io\u002FHarnessEval-W#leaderboard\">\u003Cimg alt=\"Leaderboard\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🏆-Leaderboard-C9A227\">\u003C\u002Fa>\n\u003C\u002Fp>\n\n\n\u003C!-- \u003Cp align=\"center\">\n  \u003Ca href=\"#overview\">Overview\u003C\u002Fa> |\n  \u003Ca href=\"#how-it-works\">Method\u003C\u002Fa> |\n  \u003Ca href=\"#installation\">Installation\u003C\u002Fa> |\n  \u003Ca href=\"#usage\">Usage\u003C\u002Fa> |\n  \u003Ca href=\"#extending-harnesseval\">Contributing\u003C\u002Fa> |\n  \u003Ca href=\"#citation\">Citation\u003C\u002Fa>\n\u003C\u002Fp> -->\n\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Ffig_teaser.png\" alt=\"HarnessEval overview\" width=\"100%\">\n\u003C\u002Fp>\n\n## Overview\n\nA benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the **reasoning that justifies the score**. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified.\n\n**HarnessEval** is an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval interprets the context of each evaluation case, decomposes the evaluation question into measurable sub-questions, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own sub-question. The parent agent then validates the gathered evidence and aggregates it into the final verdict. Every evaluation becomes a **transparent evidence tree** whose complete reasoning chain justifies the result.\n\n## News\n\n- [2026\u002F08\u002F18] 📄 [Paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2608.16859) now available. \n- [2026\u002F08\u002F18] 🌐 [Homepage](https:\u002F\u002Fmirros-lab.github.io\u002FHarnessEval-W) with leaderboard-ready reports and benchmark resources is live. \n- [2026\u002F08\u002F18] 🚀 Released the full HarnessEval benchmark, evaluation code, fixed plans, and metric backends.\n\n## How It Works\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Ffig_pipeline.png\" alt=\"HarnessEval pipeline\" width=\"100%\">\n\u003C\u002Fp>\n\n1. **Case-specific skill routing.** Given a case (initial world, action, probe intent), the planner routes it to the skills that can legitimately evaluate it — and records an evidence-grounded reason for every skill it skips.\n2. **Sub-agent reasoning.** Each skill decomposes its evaluation into measurable sub-questions, each answered by a dedicated sub-agent against rollout evidence.\n3. **Validated aggregation.** The parent agent validates the collected evidence and aggregates it into the case score. The full trace — every question, answer, score, and supporting frame — is saved as an auditable case card.\n\nRouting depends only on the case context, never on the model being evaluated, so every model faces the same questions on the same cases.\n\n## Installation\n\nCreate three environments:\n\n- `harnesseval-main`: launcher \u002F CLI\n- `harnesseval-metrics`: metric backends\n- `harnesseval-pavrm`: physical-plausibility backend\n\n```bash\ngit clone --branch main --single-branch https:\u002F\u002Fgithub.com\u002Fmirros-lab\u002Fharnesseval-w.git\ncd harnesseval-w\nconda env create -f docs\u002Finstallation\u002Fmain.environment.yml\nconda env create -f docs\u002Finstallation\u002Fmetrics.environment.yml\nconda env create -f docs\u002Finstallation\u002Fpavrm.environment.yml\nconda activate harnesseval-main\n```\n\nConfigure credentials and paths:\n\n```bash\ncp config\u002Fexample.env harnesseval.env\nset -a; . .\u002Fharnesseval.env; set +a\n```\n\nThe bundled demo in `runs\u002Fexample\u002Fresults_example` can be evaluated immediately after setup.\n\n## Usage\n\nEvaluate a model's generated results:\n\n```bash\n# Evaluate generated videos and write run outputs.\nharnesseval eval \\\n  --results runs\u002Fexample\u002Fresults_example\u002Fgeneration \\\n  --model-id seedance-2.0-standard \\\n  --run-root runs\u002Fexample\u002Fresults_example\u002Frun \\\n  --manifest runs\u002Fexample\u002Fresults_example\u002Fmanifest.json \\\n  --plan-root benchmark\u002Fplans\n```\n\nCheck a completed run:\n\n```bash\n# Verify that an existing run has all expected scores.\nharnesseval verify run \\\n  --eval-root runs\u002Fexample\u002Fresults_example\u002Frun\u002Fharnesseval\u002Fmodels\u002Fseedance-2.0-standard\u002Fevaluation \\\n  --manifest runs\u002Fexample\u002Fresults_example\u002Fmanifest.json \\\n  --model seedance-2.0-standard\n```\n\nOr use the bundled demo end to end:\n\n```bash\n# Run the bundled demo from the example directory.\ncd runs\u002Fexample\u002Fresults_example\n# Evaluate the demo outputs.\nharnesseval eval --results generation --model-id seedance-2.0-standard --run-root run --manifest manifest.json --plan-root ..\u002F..\u002Fbenchmark\u002Fplans\n# Verify the demo run.\nharnesseval verify run --eval-root run\u002Fharnesseval\u002Fmodels\u002Fseedance-2.0-standard\u002Fevaluation --manifest manifest.json --model seedance-2.0-standard\n```\n\n## What You Get\n\nThe bundled example already produces scores. A completed evaluation writes:\n\n```\nruns\u002Fexample\u002Fresults_example\u002Frun\u002Fharnesseval\u002Fmodels\u002F\u003Cmodel-id>\u002Fevaluation\u002F\n├── summary.json           # overall and per-family scores\n├── leaderboard_latest.json\n├── leaderboard_latest.csv\n└── LEADERBOARD.md\n```\n\nPer-case artifacts and caches live under `run\u002Fharnesseval\u002Fmetric_cache\u002F` and `run\u002Fharnesseval\u002Fmodels\u002F\u003Cmodel-id>\u002F`.\n\n## Extending HarnessEval\n\nHarnessEval is a living benchmark, and contributions are welcome.\n\n- **Submit a new case** with a new world, action, or probe family.\n- **Submit a new skill** when a case needs a new kind of evaluation.\n\nKeep submissions aligned with the existing benchmark format so they can plug into the bundled flow.\n\n## TODO\n\n\n- [ ] Hosted submission & evaluation service (submit videos, get scores)\n- [ ] Full cases on HuggingFace\n- [ ] Subset cases & weights on HuggingFace\n- [x] Model generation example\n- [x] Evaluation code & example release\n- [x] ArXiv paper, homepage with interactive leaderboard, blog release\n\n\n## Acknowledge\n\nThis project builds upon the following excellent works:\n\n- [VBench](https:\u002F\u002Fgithub.com\u002FVchitect\u002FVBench) — Video quality metrics\n- [WBench](https:\u002F\u002Fgithub.com\u002Fmeituan-longcat\u002FWBench) — World model benchmark\n- [WorldScore](https:\u002F\u002Fgithub.com\u002Fhaoyi-duan\u002FWorldScore) - World model benchmark\n- [Cosmos](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fcosmos) - Open source world model\n- [Lingbot World](https:\u002F\u002Fgithub.com\u002FRobbyant\u002Flingbot-world) - Open source world model\n- [MiniMax H3](https:\u002F\u002Fgithub.com\u002FMiniMax-AI\u002FMiniMax-H3) - Open source world model\n- ... and many other excellent open-source projects\n\n## Citation\n\nIf you find HarnessEval useful, please cite:\n\n```bibtex\n@article{mirros2026harnessevalw,\n  title   = {HarnessEval-W: Agentifying the Evaluation of Visual Worlds},\n  author  = {{MirroS Team}},\n  journal = {arXiv preprint arXiv:2608.16859},\n  year    = {2026}\n}\n```\n\n## License\nWe release our code under the Apache 2.0 License.\n","HarnessEval-W 是一个面向视觉世界模型的智能体化评估框架，旨在为视频生成、世界建模等任务提供可解释、可验证的自动化评测能力。其核心采用分层智能体架构：主智能体动态解析评估意图，分解为物理一致性、因果合理性、状态演化等子问题，并调度专用子智能体结合诊断工具生成带证据链的推理过程，最终形成透明的‘证据树’评估报告。技术上融合了任务感知规划、多智能体协同与可追溯评估机制，支持对 Kling、MiniMax-H3、SEEDance 等主流视频\u002F世界模型进行细粒度诊断。适用于世界模型研发、视频生成系统评测及AI安全可信性验证等需要归因分析的研究与工程场景。","2026-08-19 02:30:10","CREATED_QUERY"]