[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-106536":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":14,"stars30d":14,"stars90d":15,"forks30d":15,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":18,"hasPages":20,"topics":21,"createdAt":9,"pushedAt":22,"updatedAt":23,"readmeContent":24,"aiSummary":9,"trendingCount":15,"starSnapshotCount":15,"syncStatus":13,"lastSyncTime":25,"discoverSource":26},106536,"sdbench","microsoft\u002Fsdbench","microsoft","Sequential Diagnosis Benchmark",null,"Python",58,13,2,1,0,44.04,"MIT License",false,"main",true,[],"2026-10-22 17:37:14","2026-10-31 21:16:58","# SDBench: Sequential Diagnosis Benchmark\n\nSDBench is a benchmark for evaluating sequential diagnosis systems in medical settings. This benchmark provides a framework for assessing how AI systems can progressively gather information, formulate hypotheses, and arrive at diagnoses through iterative decision-making processes.\n\nThis benchmark is described in [Sequential Diagnosis with Language Models](https:\u002F\u002Fwww.science.org\u002Fdoi\u002F10.1126\u002Fscience.aeh7930), published in _Science_.\n\nA website for the benchmark is hosted at https:\u002F\u002Fmicrosoft.github.io\u002Fsdbench\n\n## How it works\n\nFor each case, the harness ([`harness\u002Fharness.py`](sdbench\u002Fharness\u002Fharness.py)) runs a sequential loop between three roles:\n\n1. The **agent** (the system under test) receives a short `case_summary` that does not reveal the diagnosis, and decides — turn by turn — to order tests, ask questions, or commit a diagnosis.\n2. The **gatekeeper** ([`gatekeeper.py`](sdbench\u002Fgatekeeper.py)) answers each test order \u002F question from the case's full text, returning the real result when the case contains it or a *synthetic* result when it does not.\n3. When the agent commits a diagnosis, the **judge** ([`judge.py`](sdbench\u002Fjudge.py)) scores it against the ground-truth diagnosis on a 1–5 Likert scale (4–5 counts as a correct diagnosis).\n\nThe gatekeeper and judge models\u002Fprompts are fixed by the chosen [configuration](#configurations); the agent's model is what you are benchmarking. The [costing](sdbench\u002Fcosting\u002F) module estimates the monetary cost of the tests an agent orders.\n\n## Setup\n\n### Install\n\nThis project uses [uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002F) for environment and dependency management.\n\n```bash\nuv sync\n```\n\nThen create a `.env` file holding the API keys for whichever model provider(s) you use (see below).\nThe CLIs load it automatically via `python-dotenv`.\n\nOptional features live behind dependency extras (install one with `uv sync --extra \u003Cname>`, or\neverything with `uv sync --all-extras`):\n\n- `cpt_costing` — CPT-code grounded costing (`--costing-method unified_cpt_llm`).\n- `azure` — Azure integrations: Blob Storage output upload and the CosmosDB costing cache.\n\n### Model providers & environment variables\n\nModels are routed by [`models.py`](sdbench\u002Fmodels.py). By default requests go through **OpenRouter**; you can\nforce Azure OpenAI or a local inference endpoint instead.\n\n| Variable | Purpose |\n|---|---|\n| `OPENROUTER_API_KEY` | API key for OpenRouter (default routing). |\n| `OPENAI_API_KEY` | Only used by the `cpt_costing` extra: a fallback for building test-name embeddings when `OPENROUTER_API_KEY` is unset. |\n| `SDBENCH_FORCE_AZURE=1` | Route all calls through Azure OpenAI. |\n| `AZURE_OPENAI_API_KEY` \u002F `AZURE_OPENAI_ENDPOINT` \u002F `AZURE_OPENAI_API_VERSION` | Azure OpenAI credentials. |\n| `SDBENCH_OPENROUTER_ZDR_ONLY=0` | Allow non–zero-data-retention OpenRouter providers if your dataset permits. |\n| `SDBENCH_LOCAL_BASE_URL` \u002F `SDBENCH_LOCAL_HEADERS` | Base URL \u002F headers for a local (e.g. SSH-tunneled) inference server; models named `local-...` route here. |\n| `SDBENCH_DATA_PATH` | Override the data directory (default: `.\u002Fdata`). Covers both case datasets (`\u003Cpath>\u002Fprocessed_case_data\u002F`) and the costing charge-master (`\u003Cpath>\u002Fcosting_data\u002F`). |\n| `SDBENCH_LOCAL_DIR` | Base directory for local overrides — `prompts\u002F`, `diagnostic_agents\u002F`, and `config.py` (default: the current working directory). Set to `disable` to ignore all local overrides and use only the packaged built-ins. |\n\n### Use as a library\n\nWhen you import `sdbench` from another project (rather than running the CLIs), two defaults are tuned for standalone use and are worth overriding:\n\n- **`.env` is not loaded automatically.** Only the CLI entry points call `python-dotenv`; the library reads its configuration straight from the environment. Load your own `.env` (or otherwise export `OPENROUTER_API_KEY`, `SDBENCH_*`, etc.) before calling into `sdbench`.\n- **Local overrides default to the current working directory.** A `prompts\u002F`, `diagnostic_agents\u002F`, or `config.py` in your host project's CWD will be picked up. Set `SDBENCH_LOCAL_DIR=disable` to use only the packaged built-ins, or point it at a dedicated directory.\n\n### Datasets\n\nA dataset lives under `.\u002Fdata\u002Fprocessed_case_data\u002F{dataset_name}\u002F`. \n\n#### Existing Datasets\n\nSDBench ships with a small, fully-synthetic dataset named `synthetic` (five NEJM-style CPC\ncases), which is the default dataset so the harness runs out of the box with no data access\nrequired. See [data\u002Fsynthetic_case_data\u002FREADME.md](data\u002Fsynthetic_case_data\u002FREADME.md) for details.\n\n> **Note:** The `synthetic` dataset is intended only as a set for developing and testing agents and as such only provides a `train` split; it should not be used to report diagnostic performance — use a clinically sourced dataset for that.\n\nThe clinically sourced datasets are not included in the repository and require separate access or licensing:\n\nSee [data\u002Fprocessed_case_data\u002Fnejm\u002FREADME.md](data\u002Fprocessed_case_data\u002Fnejm\u002FREADME.md) for details about licensing the `nejm` dataset from NEJM.\n\nThe `mimic` dataset is distributed as [MIMIC-IV-Ext-SDBench v1.0.0 on PhysioNet](https:\u002F\u002Fphysionet.org\u002Fcontent\u002Fmimic-iv-ext-sdbench\u002F1.0.0\u002F). See [data\u002Fprocessed_case_data\u002Fmimic\u002FREADME.md](data\u002Fprocessed_case_data\u002Fmimic\u002FREADME.md) for access requirements and download instructions.\n\n#### Bring your own dataset\n\nPopulate a dataset folder under `data\u002Fprocessed_case_data\u002F\u003Cyour_dataset>\u002F` with cases following the case data formats detailed in the module docstring of [`case_data.py`](sdbench\u002Fcase_data.py).\n\nNote: There are two different gatekeepers in SDBench, each optimised for different NEJM and MIMIC case report formats. By default SDBench will use the NEJM optimised gatekeeper, but if your dataset is sourced from discharge summaries that are more similar to MIMIC in nature then the MIMIC gatekeeper is recommended. Regardless it is recommended to evaluate the performance of the gatekeeper on any new dataset.\n\n## Configurations\n\nNamed configs in [`config.py`](sdbench\u002Fconfig.py) pin the gatekeeper\u002Fjudge models and prompt versions. Select\none with `--sdbench-config \u003Cname>`:\n\n| Config | Description |\n|---|---|\n| `science` | The configuration used in the peer-reviewed Science paper (`nejm` and `mimic`); pair with `--agent baseline`. |\n| `default` | The current default config; uses the NEJM-optimised gatekeeper for the `synthetic` default dataset and any non-`mimic` dataset. |\n\n## Usage\n\n### Run with a coding agent\n\nOpen this checkout in a coding agent that follows [AGENTS.md](AGENTS.md) and ask:\n\n> `Run SDBench on \u003Cmodel name>`\n\nThe agent follows [RUN_SDBENCH.md](RUN_SDBENCH.md) to ask for missing run settings,\nhelp with credentials and dataset access, run the evaluation and grounded costing,\nand report progress and results. The workflow also covers recovery and resuming\ninterrupted work; you do not need to reference the runbook in your request.\n\n### Run a single case\n\nBy default it runs the baseline agent on the bundled `synthetic` train set (five cases) with GPT-5 using OpenRouter:\n\n```bash\nexport OPENROUTER_API_KEY=...\nuv run python -m sdbench.cli.harness\n```\n\nSome basic config options for the harness:\n\n- `--dataset` the dataset to run on\n- `--case-set` selects the split to run (e.g. `validation`, `test`, `all` — available sets depend on the dataset).\n- `--max-workers N` number of cases to run in parallel\n- `--num-cases N` or `--case-id \u003Cid> [\u003Cid> ...]` restrict to a subset.\n- Runs are **resumable**: `--resume \u003Cresults_dir>` continues an interrupted run from its saved config, skipping completed cases.\n\n### Evaluate a new model\n\nPoint `--model` at any model your configured provider exposes (an OpenRouter slug, an Azure\ndeployment name, or a `local-...` model)\n\n```bash\nuv run python -m sdbench.cli.harness --model \u003Cyour-model>\n```\n\n#### Local models\n\nAny model whose name starts with `local-` is routed to a local OpenAI-compatible inference\nserver (e.g. vLLM, llama.cpp, Ollama) instead of OpenRouter or Azure. The full model string —\nincluding the `local-` prefix — is forwarded verbatim as the request's `model`, so serve your\nmodel under that exact name. OpenRouter ZDR\u002Fprovider routing does not apply.\n\nPoint SDBench at your endpoint with `SDBENCH_LOCAL_BASE_URL`. If your server checks an API key,\nset `SDBENCH_LOCAL_API_KEY` (it is sent as a standard `Authorization: Bearer` header and\ndefaults to the dummy value `sk-local`, which satisfies servers that require a key but don't\nvalidate it):\n\n```bash\nexport SDBENCH_LOCAL_BASE_URL=http:\u002F\u002Flocalhost:8000\u002Fv1\nuv run python -m sdbench.cli.harness --model local-my-model\n```\n\nFor non-standard auth (a custom header name) or extra headers needed by an SSH tunnel or proxy,\nset `SDBENCH_LOCAL_HEADERS` to a JSON object. These headers are merged last, so an\n`Authorization` key here overrides `SDBENCH_LOCAL_API_KEY`:\n\n```bash\nexport SDBENCH_LOCAL_HEADERS='{\"Authorization\": \"Bearer \u003Ctoken>\"}'\n```\n\nThese can also be supplied per run as model args (see [Model arguments](#model-arguments)):\n`local_model_base_url`, `local_model_api_key`, `local_model_headers`.\n\n#### Azure OpenAI\n\nTo serve models from your own Azure AI Foundry deployment (which must be ZDR-configured), set\nthe Azure credentials (`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`,\n`AZURE_OPENAI_API_VERSION`) and force Azure routing. The `--model` value is used directly as\nthe Azure **deployment name**, so name your deployment to match (no pre-registration in SDBench\nneeded — new models just pass through). The only exceptions are a few aliases remapped in\n`AZURE_MODEL_MAP` in [`models.py`](sdbench\u002Fmodels.py) (e.g. `gpt-3.5-turbo` → `gpt-35-turbo`).\n\nForce Azure for the whole run with the env var:\n\n```bash\nexport SDBENCH_FORCE_AZURE=1\nuv run python -m sdbench.cli.harness --model gpt-5\n```\n\nOr force it per call (so you can mix Azure and OpenRouter models in a single run) with the\n`force_azure` model arg:\n\n```bash\nuv run python -m sdbench.cli.harness --model gpt-5 --model-args '{\"force_azure\": true}'\n```\n\n#### Model arguments\n\n`--model-args` takes a JSON object that is merged into each agent API call, overriding the\nper-model defaults in `MODEL_ARG_DEFAULTS`. Most keys are forwarded straight to the model API:\n\n```bash\nuv run python -m sdbench.cli.harness --model gpt-5 --model-args '{\"reasoning_effort\": \"high\"}'\n```\n\nA few keys are instead intercepted by SDBench's routing layer and are **not** forwarded to the\nmodel: `force_azure` and `openrouter_zdr_only` (per-run equivalents of `SDBENCH_FORCE_AZURE` \u002F\n`SDBENCH_OPENROUTER_ZDR_ONLY`), and the local-endpoint overrides `local_model_base_url`,\n`local_model_api_key`, and `local_model_headers`.\n\n### Reproduce the Science paper baselines\n\nUse the `science` config, the `baseline` agent, the `validationandtest` case set (which encompasses the `validation` and `test` splits that were reported jointly against a given model, ie: `openai\u002Fgpt-5.5`:\n\n```bash\nuv run python -m sdbench.cli.harness \\\n  --dataset \u003Cnejm|mimic> \\\n  --case-set validationandtest \\\n  --agent baseline \\\n  --sdbench-config science \\\n  --model \"openai\u002Fgpt-5.5\"\n```\n\nThe `science` config pins the gatekeeper and judge per dataset so results are comparable across agent\nmodels. Costs will need to be computed with grounded costing data for comparability to the published results (see next section).\n\n### Costing\n\nSDBench supports LLM based estimates of diagnostic test costs (the default) as well as estimates grounded on a public hospital charge master. Cost computation can be disabled with `--no-compute-costs`.\n\n#### Grounded Costs\n\nIn the published paper we reported on costs that were grounded against a public hospital charge master. If you want to use grounded costing data then you'll need to configure the costing data: See [data\u002Fcosting_data\u002FREADME.md](data\u002Fcosting_data\u002FREADME.md).\n\nTo compute costs using this grounded costing data you can either compute it as the harness runs each case or as a batch operation once the harness run has finished (recommended).\n\nTo run it as a batch operation, point the recost CLI at one or more results directories.\nGrounded costing needs the `cpt_costing` extra, so enable it for the run:\n\n```bash\nuv run --extra cpt_costing python -m sdbench.cli.recost \u003Cresults_dir> [\u003Cresults_dir> ...]\n```\n\nThis recomputes each agent's per-test costs with the grounded method and writes them back\ninto the case JSON. It then reports how many ordered tests received a *grounded* cost (a CPT\nmatch against the charge master) versus how many fell back to an LLM estimate:\n\n```\n  Total tests costed:      200\n  Grounded (CPT):          196  ( 98.0%)\n  LLM fallback:              4  (  2.0%)\n```\n\nIf GPT grounding fail to match a specific test the costing will fall back to an LLM-estimate. A typical grounding rate of 98%+ is expected, if the rate is significantly below that then your grounded cost results may be unreliable.\n\nAlready-costed cases are skipped unless they contain pricing failures; pass `--force` to\nrecompute everything, or `--analyze-only` to just print the breakdown for costs that were\nalready computed (by the harness or a previous recost run) without recomputing.\n\nPass `--costing-method llm_only` (or `batch_llm_only`) to re-cost with the LLM method\ninstead of grounded CPT; the LLM methods don't need the `cpt_costing` extra or the charge\nmaster, and the breakdown reports priced vs failed rather than grounded vs fallback. The\nbreakdown always reflects only the method being run, ignoring any other cost analyses\nalready stored on the case.\n\nTo compute grounded costs inline as the harness runs instead, set `--costing-method unified_cpt_llm`.\n\n#### LLM Estimated Costs\n\nBy default SDBench will compute LLM Estimated costs using `gpt-5-mini`. This model can be changed with `--costing-model`, though cost estimates from different models aren't directly comparable.\n\nLLM cost estimates are also not directly comparable with grounded cost estimates. We found the `gpt-5-mini` estimates to be 40-50% higher than the grounded estimates used in our paper. While individual tests can show more variance, in aggregate this appeared to be a consistent bias.\n\n#### Caching costs\n\nCosting can be slow and expensive: both the grounded (CPT) and LLM methods make model\ncalls per test, and the same tests (e.g. CBC, chest X-ray) recur across many cases and\nacross repeated runs. A cost is keyed only by `(costing_method, costing_model, test_name)`,\nso a shared cache lets one priced test be reused across every case that orders it, across\nre-costing runs, and across machines — turning most of a re-cost into cache hits.\n\nSDBench supports an optional CosmosDB-backed cache for this. It needs the `azure` extra and\nis enabled with these flags (all forwarded to the costing config, so they work on both the\nharness and the recost CLI):\n\n```bash\nuv run --extra azure --extra cpt_costing python -m sdbench.cli.recost \u003Cresults_dir> \\\n  --costing-cache-cosmosdb-enabled \\\n  --costing-cache-cosmosdb-endpoint https:\u002F\u002F\u003Caccount>.documents.azure.com:443\u002F \\\n  --costing-cache-cosmosdb-database \u003Cdatabase> \\\n  --costing-cache-cosmosdb-container \u003Ccontainer>\n```\n\nAuthentication uses `DefaultAzureCredential` (Azure CLI login, environment, or managed\nidentity). On Azure compute you can add `--costing-cache-use-managed-identity` to use the\nattached managed identity instead. The cache is off by default; when disabled, costing\nstill runs but recomputes every test.\n\n### Outputs\n\nA run writes to `results\u002F\u003Cowner>_\u003Cdataset>_\u003Clabel>_\u003Cmodel>_\u003Ccase_set>_\u003Ctimestamp>_k\u003Ck>\u002F`:\n\n```\njson\u002Fcase_k{k}_{case_id}.json   # one file per (case, repetition)\nhtml\u002F                           # generated HTML report\nsummary.txt                     # console summary\nrun_config.json                 # full config (used to resume \u002F restore)\n```\n\nThe batch runner generates the summary, HTML, and an accuracy\u002Fcost report automatically. To\nregenerate reports from one or more existing run directories:\n\n```bash\nuv run python -m sdbench.cli.reporting \u003Cresults_dir> [\u003Cresults_dir> ...] [--ci] [--json out.json]\n```\n\nThis regenerates each folder's own `summary.txt` (with that folder's per-folder accuracy\u002Fcost\nreport folded in) and HTML, then prints a distinct **aggregate** accuracy + cost report grouped by\n`(agent, model)` — pooling multiple folders that share the same pair — broken down by case set\n(`train`\u002F`validation`\u002F`test` by default; override with `--case-sets`). Costs are reported as\nmean-per-case and run-total for every costing method present. `--ci` adds bootstrapped CIs\n(percentile bootstrap for cost, binomial for accuracy). Errored and incomplete cases are flagged and\nexcluded from accuracy; mismatched repetition counts are flagged. The aggregate aborts if the\nfolders span multiple datasets.\n\n### Write your own agent\n\nAn agent is a module exposing `run(case_summary, test_runner, agent_llm, collector, config)` (see the\n`DiagnosticAgent` protocol in [`agents\u002F__init__.py`](sdbench\u002Fagents\u002F__init__.py)). Agents are resolved by\nmodule name from:\n\n1. The packaged `baseline` agent in [`agents\u002F`](sdbench\u002Fagents\u002F).\n2. A `diagnostic_agents\u002F` directory under the local-override base — the current working\n   directory by default, or `$SDBENCH_LOCAL_DIR` if set (use `SDBENCH_LOCAL_DIR=disable` to\n   skip it). Drop a `my_agent.py` there and select it with `--agent my_agent`.\n\nThe packaged baseline drives the gatekeeper with XML-tagged requests (`\u003Ctest>...\u003C\u002Ftest>`,\n`\u003Cquestion>...\u003C\u002Fquestion>`, `\u003Cdiagnosis>...\u003C\u002Fdiagnosis>`); see\n[`agents\u002Fbaseline.py`](sdbench\u002Fagents\u002Fbaseline.py) for a complete example.\n\n### Full CLI reference\n\nThe flag set is large; for the complete, authoritative list run:\n\n```bash\nuv run python -m sdbench.cli.harness --help\n```\n\n## Citation\n\nIf you use SDBench, please cite the Science article:\n\n```bibtex\n@article{nori2026sequentialdiagnosislanguagemodels,\n  title={Sequential Diagnosis with Language Models},\n  author={Harsha Nori and Mayank Daswani and Christopher Kelly and Scott Lundberg and\n          Marco Tulio Ribeiro and Marc Wilson and Xiaoxuan Liu and Viknesh Sounderajah and\n          Jonathan Carlson and Matthew P Lungren and Bay Gross and Peter Hames and\n          Mustafa Suleyman and Dominic King and Eric Horvitz},\n  journal={Science},\n  year={2026},\n  doi={10.1126\u002Fscience.aeh7930},\n  url={https:\u002F\u002Fwww.science.org\u002Fdoi\u002F10.1126\u002Fscience.aeh7930},\n}\n```\n","2026-11-03 04:32:15","CREATED_QUERY"]