[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93208":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":12,"subscribersCount":12,"size":12,"stars1d":12,"stars7d":12,"stars30d":15,"stars90d":12,"forks30d":12,"starsTrendScore":12,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":12,"starSnapshotCount":12,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},93208,"RigorLoop","ronikobrosly\u002FRigorLoop","ronikobrosly","A statistically-sound agentic build framework that employs agentic loops to create code artifacts (whether a script, a skill markdown file, etc). Crucially, it splits verification data into the classic data science-like dev, validation, and final test sets to avoid overfitting. ",null,"Python",132,0,122,1,10,41,"MIT License",false,"main",true,[],"2026-07-22 04:02:08","\u003Cp align=\"center\">\n  \u003Cimg src=\"logo.webp\" alt=\"RigorLoop logo\" width=\"800\">\n\u003C\u002Fp>\n\n# RigorLoop\n\nA statistically-sound agentic loop-engineering framework. You give it a task description,\na pile of gold-standard input\u002Foutput examples, and a set of checks; it runs\nagentic loops (a strategy agent directing concurrent executor agents) that\niteratively build and refine a solution — and it evaluates that solution the\nway a careful data scientist would: on a strict **dev \u002F validation \u002F test**\nsplit, so the score you see at the end is one you can actually trust.\n\nThe solution it produces is a portable artifact you can take away and use\nwithout RigorLoop:\n\n- an **executable Python script** (reads your input on stdin, writes the output),\n- an **agent skill** (a `SKILL.md`-style document, e.g. for Claude Skills), or\n- a **guidance file** (an `AGENTS.md`\u002F`CLAUDE.md`-style document for coding agents).\n\n## When to use it (and when not to)\n\nUse RigorLoop for **data-science-like transformation tasks**: you have many\nrepresentative examples of messy inputs (structured or unstructured text) and\nthe structured outputs you want, and you need a solution that generalizes to\n*new* inputs — extraction, normalization, classification-with-output-format,\nreformatting, tagging.\n\nDo **not** use it to generate a simple deterministic script that just needs to\npass a handful of fixed unit tests — a single coding-agent session does that\njob better. RigorLoop's machinery (splits, confidence intervals, budgeted\nvalidation peeks) only pays off when overfitting is a real risk.\n\n## Requirements\n\n- **Python ≥ 3.12** on Linux or macOS (Windows is not supported in v1)\n- The **[claude CLI](https:\u002F\u002Fclaude.com\u002Fclaude-code)** installed and\n  authenticated (`claude --version` should work) — agents are invoked\n  headless and tool-less via `claude -p`\n- **Examples: the more, the better.** RigorLoop will run with a couple dozen,\n  but it will warn you plainly about what small sets can and cannot prove\n  (with ~30 examples, a validation set of 6 can only distinguish pass-rate\n  differences of roughly ±40 points). Aim for 100+ if you can.\n- A budget: every loop spends real model calls. `rigorloop check` estimates\n  the call count before you commit (skill\u002Fguidance artifacts cost far more to\n  evaluate than scripts, because *evaluating* each example is itself a model\n  call).\n\n## Install\n\n```bash\npip install rigorloop\n```\n\n## Quickstart\n\n```bash\nmkdir my-task && cd my-task\nrigorloop init      # scaffolds rigorloop.toml, task.md, examples.jsonl (a toy dataset)\n# 1. Describe your task in task.md\n# 2. Replace examples.jsonl with your real examples\n# 3. Adjust rigorloop.toml (solution kind, checks, budgets)\nrigorloop check     # validates everything, prints split sizes, warnings, and\n                    # the agent-call budget estimate — spends no tokens\nrigorloop run       # runs the loops; artifacts land in runs\u002F\u003Crun_id>\u002F\n```\n\nWhen the run finishes you get, under `runs\u002F\u003Crun_id>\u002Ffinal\u002F`:\n\n- **the solution** (`solution.py`, `SKILL.md`, or `GUIDANCE.md`) — copy it out\n  and use it anywhere;\n- **`report.md`** — pass rates with 95% confidence intervals on all three\n  splits, per-check breakdowns, the loop history, and honest caveats about\n  what the numbers mean;\n- `test_results.json` — the same, machine-readable.\n\nA run that stops midway (crash, Ctrl-C) can be continued with\n`rigorloop run --resume \u003Crun_id>`, and `rigorloop report \u003Crun_id>` re-renders\nthe report of a finished run.\n\n## What you provide\n\n**`task.md`** — a plain-language description of the transformation, written\nfor the agent that will build the solution. Say what the inputs look like,\nwhat the outputs must look like, and any rules that matter.\n\n**`examples.jsonl`** — one JSON object per line:\n\n```jsonl\n{\"input\": \"Name: Ada Lovelace\\nEmail: ada@calc.org\\nCity: London\", \"expected_output\": \"{\\\"city\\\": \\\"London\\\", \\\"email\\\": \\\"ada@calc.org\\\", \\\"name\\\": \\\"Ada Lovelace\\\"}\"}\n```\n\nBoth fields may be strings or JSON structures (structures are canonicalized to\nJSON text). These examples should be *highly representative* of the inputs\nyou'll see in production — the final score is only as honest as the examples\nare representative.\n\n**Checks** — one or more `[[checks]]` blocks in `rigorloop.toml`. An example\npasses only if **every** check passes:\n\n| `type` | What it verifies | Options (defaults) |\n|---|---|---|\n| `exact_match` | output equals the expected output | — |\n| `normalized_match` | equal after normalization | `lowercase`, `strip`, `collapse_whitespace` (all `true`) |\n| `json_equality` | output parses to the same JSON value | — |\n| `regex_match` | output contains a match | `pattern` (required) |\n| `numeric_tolerance` | output is a number within tolerance | `atol`, `rtol` (`1e-6`) |\n| `custom_python` | your own checker script passes | `script_path` (required); gets JSON on stdin, exit 0 = pass, 1 = fail |\n| `llm_judge` | a model judges the output against a rubric | `rubric` (required), `n_samples` (3), `pass_threshold` (0.5) |\n\n## The knobs (`rigorloop.toml`)\n\nEverything has a sensible default; a minimal config is just the `[task]`\nsection and one check.\n\n```toml\n[task]\ndescription_file = \"task.md\"\nsolution_kind    = \"script\"        # script | skill | guidance\nexamples_file    = \"examples.jsonl\"\n\n[split]\nratios = [0.6, 0.2, 0.2]           # dev \u002F validation \u002F test\nseed   = 17                        # same seed => same split, always\n\n[loop]\nmax_loops              = 12        # hard cap on strategy loops\nexecutors_per_loop     = 4         # concurrent solution builders per loop\ndev_examples_in_prompt = 30        # examples each builder sees (resampled per loop)\n\n[validation]\nval_every        = 3               # scheduled validation checkpoint cadence\nmax_peeks        = 10              # total candidate validation evaluations per run\ncohort_size      = 2               # candidates validated per checkpoint: top dev\n                                   # scorers + one diverse (non-champion-based) slot\npatience         = 2               # checkpoint loops without real improvement => stop\ntarget_pass_rate = 0.95            # optional: stop early once the validation score's\n                                   # lower confidence bound clears this\n\n[agents]\nmodel     = \"claude-sonnet-5\"      # model for all agent roles\ntimeout_s = 300\n\n[[checks]]\ntype = \"json_equality\"\n```\n\nThe knobs that matter most in practice:\n\n- **`solution_kind`** — also sets the evaluation cost model: scripts are\n  executed locally (cheap); skills\u002Fguidance are evaluated by running a model\n  per example (expensive — check the budget estimate).\n- **`max_loops` × `executors_per_loop`** — your primary cost lever.\n- **`max_peeks` \u002F `cohort_size` \u002F `patience`** — how many candidate evaluations\n  the loop may spend on the validation set, how many candidates each checkpoint\n  compares, and how long the loop tolerates no genuine (beyond-noise)\n  improvement before stopping. For expensive artifact kinds (skill\u002Fguidance),\n  every validation evaluation is a model call per example — keep `cohort_size`\n  small and watch the budget estimate.\n- **`seed`** — makes the split reproducible; changing it reshuffles which\n  examples land in the holdout.\n\n## How it stays honest\n\n- Your examples are split once, up front; the split is fingerprinted so a\n  resumed run can never quietly reshuffle it.\n- The building agents only ever see **dev** examples. Validation scores reach\n  the strategy agent only as aggregates; test examples reach no agent, ever.\n- The artifact each loop refines is the **validation champion** — the\n  candidate with the best evidence of generalizing — not the raw dev leader.\n  Each checkpoint validates a precommitted cohort (the top unvalidated dev\n  candidates plus one diverse alternative not built on the champion), so a\n  candidate that is slightly worse on dev but generalizes better still gets\n  discovered. The dev leaderboard is a diagnostic, not a selection rule.\n- \"Improved\" always means *beyond the statistical noise band* (paired tests),\n  not just a higher number — so the loop doesn't chase luck. Early stopping on\n  `target_pass_rate` likewise requires the validation score's *lower\n  confidence bound* to clear the target, not a lucky point estimate.\n- The validation set (capped, counted peeks) steers the search and picks the\n  winner — which is ordinary model selection, but it means the winner's\n  validation score is optimistically biased. That is exactly what the **test\n  set** is for: it is evaluated **exactly once**, at the very end, and that is\n  the number to report.\n\n## ⚠️ The final test set is only honest once\n\nRigorLoop holds out a final test set and evaluates the winning solution on it **exactly once per run**. That guarantee cannot protect you from yourself *across* runs: if you look at the test score, tweak your task description or checks, and re-run on the same examples file, the \"held-out\" test set is no longer unseen. After a few such iterations it has effectively become a second validation set, and its scores will be optimistically biased.\n\nIf you iterate after seeing a test result, treat that test set as spent — supply fresh, never-before-used examples for the next run's holdout.\n\nRelated caveat: RigorLoop deduplicates *exact* duplicate inputs before splitting, but near-duplicates (the same example lightly reworded) can still straddle the dev\u002Ftest boundary and quietly inflate test scores. If your dataset may contain near-duplicates, deduplicate it yourself before handing it to RigorLoop.\n\n## Other things worth knowing\n\n- **RigorLoop runs generated code on your machine.** Candidate scripts and\n  `custom_python` checks execute as subprocesses with timeouts and output\n  caps — guardrails, not a security boundary. Run inside a container\u002FVM if\n  that worries you. See `SECURITY.md`.\n- Your task description and dev examples are sent to the model; validation\n  and test examples are embedded one at a time in evaluation calls. Don't\n  include data that must not leave your machine.\n- Skill\u002Fguidance scores are conditional on the evaluating model (pinned in\n  the run manifest) and are measured tool-less; transfer to tool-using agents\n  in the wild is weaker, and the report says so.\n\n## Example\n\nA complete toy project (contact-card text → JSON) lives in\n[`examples\u002Fcontact-cards\u002F`](examples\u002Fcontact-cards\u002F) — it is exactly what\n`rigorloop init` scaffolds.\n\n\n## FAQ\n\n### When should I use this?\n\nWhen all three of these are true:\n\n1. **The task is a transformation with a checkable right answer** — messy\n   text in, structured output out (extraction, normalization, tagging,\n   classification-with-a-format) — and \"correct\" can be expressed as checks.\n2. **You have, or can collect, a real pile of gold examples** — dozens at\n   minimum, ideally 100+ — that are representative of the inputs you'll see\n   in production.\n3. **What you care about is performance on *new* inputs**, and you need a\n   score you can quote without an asterisk.\n\nIf any of these is false, don't reach for RigorLoop. No examples means there\nis nothing to split and nothing to measure honestly. And if the goal is a\ndeterministic script that passes a fixed handful of unit tests, a single\ncoding-agent session is cheaper and better — see\n[When to use it (and when not to)](#when-to-use-it-and-when-not-to).\n\n### Can't standalone frontier agents do this? Why should I use this?\n\nA frontier agent can absolutely write the solution — RigorLoop's executors\n*are* frontier agents doing exactly that. What a standalone session can't\ngive you is a score you can trust. Hand an agent your examples and ask it to\niterate until things pass, and it grades its own homework: \"98% accurate\"\nmeans 98% on the very examples it tuned against, which says little about the\nnext thousand inputs. There is no held-out data, no accounting of how many\ntimes it peeked, and no way to tell a real improvement from a lucky one.\n\nRigorLoop is the harness around those agents that supplies the missing\ndiscipline:\n\n- **a dev \u002F validation \u002F test split**, enforced up front — building agents\n  only ever see dev examples, and the test set is scored exactly once, at\n  the end;\n- **statistics instead of vibes** — confidence intervals on every score,\n  paired tests so \"improved\" means beyond the noise band, and a counted,\n  capped budget of validation peeks;\n- **search instead of one shot** — a strategy agent directing concurrent\n  executors across many loops, keeping the candidate with the best evidence\n  of *generalizing* rather than the one that got luckiest on dev.\n\nYou could rebuild all of that by hand around a chat session. RigorLoop is\nthat machinery, prebuilt and honest by construction.\n\n### What do I do if the solution the loop converges to still isn't accurate enough?\n\nFirst, believe the number — that's the point of the tool. An honest 74% is\nworth more than the inflated score a self-graded loop would have reported,\nbecause it tells you the problem isn't solved yet. Then work through this\nlist, roughly cheapest first:\n\n1. **Read `report.md`.** The per-check breakdown and loop history usually\n   show *how* it fails: one check dominating the failures, scores plateauing\n   after a couple of loops, or confidence intervals so wide the run couldn't\n   tell candidates apart.\n2. **Check your checks.** An `exact_match` where you meant\n   `normalized_match` or `json_equality`, or a vague `llm_judge` rubric, can\n   make a good solution look bad.\n3. **Sharpen `task.md`.** Failing examples often reveal rules and edge cases\n   the description never stated. Spell them out.\n4. **Add more — and more representative — examples.** Small sets cap what\n   the loop can even detect (heed the power warnings from\n   `rigorloop check`), and noisy labels cap the ceiling no solution can\n   exceed.\n5. **Raise the budget.** More `max_loops`, `executors_per_loop`, and\n   `max_peeks` buy a wider search; a stronger model in `[agents]` buys\n   better builders.\n6. **Reconsider `solution_kind`.** A deterministic script may be too rigid\n   for a fuzzy task; a `skill` or `guidance` artifact puts a model in the\n   loop at inference time (at a much higher evaluation cost).\n7. **Decompose the task.** Two simple transformations chained often beat one\n   complicated one.\n\nOne caution: the moment you iterate *after* seeing a test score, that test\nset is spent — bring fresh, never-before-used examples for the next run's\nholdout (see \"The final test set is only honest once\" above).\n\n## Contributing & development\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md). The design plan is in `PLAN.md`, the\npackaging\u002Frelease plan in `PACKAGING_PLAN.md`, and the (strict) coding rules\nin `CODING_STYLE.md`.\n\n## License\n\nMIT — see [`LICENSE`](LICENSE).\n","RigorLoop 是一个面向数据科学任务的、具备统计严谨性的智能体构建框架，通过多轮代理协作循环（策略代理调度执行代理）迭代生成可部署的代码或技能文档，并严格采用开发集\u002F验证集\u002F测试集三分法评估，防止过拟合。其核心特点包括：基于真实模型调用的预算化验证、支持生成可独立运行的Python脚本、SKILL.md技能文件或AGENTS.md引导文档，以及对输入-输出示例的统计可靠性分析。适用于需泛化到新样本的文本结构化任务，如信息抽取、格式标准化、分类标注与内容重写等场景，尤其在示例数量充足（建议100+）且过拟合风险较高时效果显著。",2,"2026-07-12 02:30:11","CREATED_QUERY"]