[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-96464":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":15,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":9,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":19,"hasPages":17,"topics":20,"createdAt":9,"pushedAt":9,"updatedAt":21,"readmeContent":22,"aiSummary":9,"trendingCount":15,"starSnapshotCount":15,"syncStatus":13,"lastSyncTime":23,"discoverSource":24},96464,"nimble","bespokelabsai\u002Fnimble","bespokelabsai","Local typed decisions, contrastive data curation, and model evaluation.",null,"Python",923,68,2,1,0,9.52,false,"main",true,[],"2026-09-21 02:04:32","# Bespoke Nimble\n\n**Data, Model, Recipe for an open Jev**\n\n[Model](https:\u002F\u002Fhuggingface.co\u002Fbespokelabs\u002FBespoke-Nimble-9B) · [Capabilities](#capabilities) · [Quickstart](#quickstart) · [Methodology](#methodology) · [Documentation and development](#documentation-and-development) · [Citation](#citation)\n\n![Introducing Bespoke Nimble. Serving reads the prompt once and then scores one answer token per question. Data curation changes one fact so that the correct answer flips. Training fine-tunes Qwen3.5-9B with LoRA on the answer tokens only. On 324 held-out examples, Bespoke-Nimble-9B matches 90.1% of the reference labels, compared with 66.4% for its base model and 93.2% for Jev 1.13.0.](assets\u002Fdiagrams\u002Fnimble-infographic.svg)\n\nNimble takes some text and a schema, and makes typed decisions about the text.\nThe schema is the list of questions to answer. Each question is either a choice\nfrom a list that you give or a true or false question. For each question, Nimble\nreturns the answer it picked and the probability of each allowed answer.\n\nNimble makes each decision in one step and does not write out any reasoning\nfirst, so it is fast (blazing fast!). Nimble is\ninspired by the System One approach of\n[TypeSafe's Jev](https:\u002F\u002Fdocs.typesafe.ai\u002Fprimitives\u002Fchoice). In this repository,\nwe share our recipe for training such a model.\n\nNote that we did not distill from Jev. The point of the repository is to show how to curate data, how to train, and to serve such a model, and encourage more research!\n\nYou can run [Bespoke-Nimble-9B](https:\u002F\u002Fhuggingface.co\u002Fbespokelabs\u002FBespoke-Nimble-9B)\non a Mac with Apple Silicon or on a machine with an NVIDIA GPU.\n\n## Capabilities\n\nWe built Nimble in one day, so expect some rough edges. What Nimble can do comes\nfrom two sources: the first is the base model, Qwen3.5-9B, the second is our\ntraining data, which we curated for a few specific domains.\n\n### What you can build\n\n| Task | You define | You get back |\n| --- | --- | --- |\n| Route a request | The destinations and when each one applies | The chosen destination and the probability of each destination |\n| Check a condition | A yes or no question and the evidence | True or false, and the probability of each |\n| Apply a policy | The rules and the allowed outcomes | A typed decision based on the text you supply |\n| Rate an outcome | Ordered levels, each with clear criteria | The chosen level and the probability of each level |\n\nYou supply a context, which is the text to judge, and a schema. The schema must\nbe flat, which means that it has no nested fields. Each field is an enum or a\nboolean. An enum field has a fixed list of string choices, and a boolean field\nis true or false.\n\nEach allowed answer has a code that is one token long. The scorer reads the\nmodel's logits for these codes. Logits are the raw scores that the model gives\nto each token. The scorer turns the logits into probabilities with the softmax\nfunction. Our Python code then builds the output from these probabilities, so\nthere is no generated JSON to parse. If a field is an ordered rating scale, your\napplication can use the probabilities to calculate an expected level.\n\nOn a Mac, `ParallelScorer` processes the shared context once and then scores all\nthe fields in parallel. The CUDA scorer scores each field on its own, with the\nfull prompt each time. Both scorers return the typed output. They also return\nthe logits and the probabilities of the candidate answers. Each field is scored\nseparately, so one field cannot see the answer to another field.\n\n### What you cannot build with the current release\n\n- Nimble accepts only text. You cannot use it to judge other kinds of input,\n  e.g., images. This is true even though the base model includes a vision part.\n- Nimble only picks from the answers you supply. It cannot write text of its\n  own, e.g., an explanation. It also cannot return nested JSON or a piece of\n  text taken from the context. An enum field can have 1 to 26 string choices,\n  and a boolean field has two.\n- The probabilities are not a guarantee that an answer is correct. Nimble scales\n  them so that they add up to 1 across the answers you supplied. A probability\n  of 0.9 does not mean that the answer is right 90% of the time. If it is\n  possible that none of your answers fit, add an answer that means \"no match\".\n  Test any probability threshold on your own data before you rely on it.\n- Each prompt can have at most 2,048 tokens. This limit includes the schema and\n  the part of the prompt that names the field to score. Nimble rejects longer\n  prompts. Fields cannot depend on each other, so your code must check that the\n  answers to different fields are consistent.\n\nWe trained Bespoke-Nimble-9B on 2,676 examples that we curated. So it's performance will depend on this data and the domains it comes from. So don't expect a lot of generalization.\nBut we do see that Nibmle is overall better than it's base model Qwen3.5-9B in new domains.\n\n\n## Quickstart\n\nClone the repository and move into its folder. Run all the commands below from\nthis folder.\n\n```sh\ngit clone https:\u002F\u002Fgithub.com\u002Fbespokelabsai\u002Fnimble.git nimble\ncd nimble\n```\n\nUse Python 3.12. To run the model on a Mac, you need Apple Silicon. Python must\nalso run directly on macOS so that it can use Metal, which is Apple's interface\nto the GPU. To run the model on Linux, you need an NVIDIA GPU that supports\nBF16, a 16-bit number format.\n\nWithout quantization, the 9B weights alone take about 18 GB. Quantization means\nstoring the weights with fewer bits to save memory. The model needs more memory\nthan this while it runs. The merge step below runs on the CPU. It needs extra\nRAM, and it needs disk space for both the base weights and the merged weights.\nA Mac with 64 GB of memory has more free memory for this than a machine with\n24 GB.\n\n### Download the model\n\nCreate a Python environment for preparing the model. On Linux, you can also use\nthis environment to run the model on the GPU. If you do, install a build of\nPyTorch with CUDA support that works with your GPU driver.\n\n```sh\npython3.12 -m venv .cache\u002Fvenvs\u002Fnimble\nsource .cache\u002Fvenvs\u002Fnimble\u002Fbin\u002Factivate\npython -m pip install torch==2.8.0 -r requirements\u002Ftraining.txt\n```\n\nThe following accepts either a full checkpoint or a PEFT LoRA adapter. For an adapter, it downloads the pinned base and merges the trained weights once. It records the resolved revision and local model path for both platform examples. No TypeSafe or generation API key is needed for local inference.\n\n```sh\npython - \u003C\u003C'PYTHON'\nimport hashlib\nimport json\nfrom pathlib import Path\n\nfrom huggingface_hub import snapshot_download\n\nrepo = \"bespokelabs\u002FBespoke-Nimble-9B\"\nsnapshot = Path(snapshot_download(repo, cache_dir=\".cache\u002Fhuggingface\u002Fhub\"))\ncontract_file = snapshot \u002F \"schema_config.json\"\ncontract = json.loads(contract_file.read_text()) if contract_file.exists() else {}\nif contract:\n    prompt_hash = hashlib.sha256(\n        Path(\"nimble\u002Fscoring\u002Fparallel_schema.py\").read_bytes()\n    ).hexdigest()\n    if (contract[\"task\"] != \"schema_candidate_classification_v1\"\n            or contract[\"prompt_code_sha256\"] != prompt_hash):\n        raise ValueError(\"Model contract differs from this checkout's scoring prompt\")\n\nmodel_path = snapshot\nif (snapshot \u002F \"adapter_config.json\").exists():\n    import torch\n    from peft import PeftModel\n    from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration\n\n    # The adapter release must include its pinned base and prompt contract.\n    base = Qwen3_5ForConditionalGeneration.from_pretrained(\n        contract[\"model\"], revision=contract[\"revision\"],\n        dtype=torch.bfloat16, device_map=\"cpu\",\n    )\n    adapter = PeftModel.from_pretrained(base, snapshot)\n    merged = adapter.merge_and_unload(safe_merge=True)\n    model_path = Path(\".cache\u002Fmodels\") \u002F (\"nimble-9b-\" + snapshot.name)\n    merged.save_pretrained(model_path)\n    AutoTokenizer.from_pretrained(snapshot).save_pretrained(model_path)\n\nconfig = {\n    \"model_path\": str(model_path.resolve()),\n    \"model_id\": repo,\n    \"revision\": snapshot.name,\n    \"max_input_tokens\": contract.get(\"max_length\", 2048),\n}\nPath(\".cache\u002Fnimble-model.json\").write_text(json.dumps(config, indent=2))\nprint(\"Ready:\", model_path)\nPYTHON\n```\n\n### Mac with Apple Silicon (MLX)\n\nUse a separate MLX environment after the model preparation step:\n\n```sh\ndeactivate\npython3.12 -m venv .venv-mlx\nsource .venv-mlx\u002Fbin\u002Factivate\npython -m pip install -r requirements\u002Fmlx.txt\n```\n\nIn Python, pass the saved model settings to `ParallelScorer` so that it loads\nthe prepared 9B weights.\n\n```python\nimport json\nfrom pathlib import Path\nfrom nimble.scoring.parallel_scorer import ParallelScorer\n\nconfig = json.loads(Path(\".cache\u002Fnimble-model.json\").read_text())\nscorer = ParallelScorer(**config)\n```\n\n> [!NOTE]\n> If you call `ParallelScorer()` with no arguments, it loads the Qwen3.5-4B\n> model that we used as a baseline, not Nimble. The MLX runner cannot load a\n> LoRA adapter folder directly, so use the merged folder that you prepared\n> above. The MLX runner also does not support quantized weights.\n\n### Linux with an NVIDIA GPU (CUDA)\n\nActivate the environment that you used to prepare the model. Then check that\nPyTorch can use the GPU and that the GPU supports BF16.\n\n```sh\nsource .cache\u002Fvenvs\u002Fnimble\u002Fbin\u002Factivate\npython -c 'import torch; assert torch.cuda.is_available() and torch.cuda.is_bf16_supported()'\n```\n\nIn Python, load the same prepared weights.\n\n```python\nimport json\nfrom pathlib import Path\nfrom nimble.scoring.cuda_scorer import CudaCandidateScorer\n\nconfig = json.loads(Path(\".cache\u002Fnimble-model.json\").read_text())\nscorer = CudaCandidateScorer(**config)\n```\n\nYou can also run the adapter without merging it, in the same way as our\ntraining evaluation. See the [training guide](docs\u002FNIMBLE_TRAINING.md) for how\nto do this. The CUDA runner scores each field with the full prompt. So it\nprocesses the shared context again for each field, while the MLX runner\nprocesses it only once.\n\n### Make a typed decision\n\nAfter you load either scorer, continue in the same Python session.\n\n```python\nschema = {\n    \"priority\": {\n        \"type\": \"enum\",\n        \"choices\": [\"HIGH\", \"LOW\"],\n        \"description\": \"Urgency based on current business impact.\",\n        \"choice_descriptions\": {\n            \"HIGH\": \"A critical business operation is currently blocked.\",\n            \"LOW\": \"An optional enhancement with no current business impact.\",\n        },\n    },\n    \"requires_review\": {\n        \"type\": \"boolean\",\n        \"description\": \"Whether customers are unable to complete a purchase.\",\n    },\n}\nresult = scorer.score(\"The payment service is down for all customers.\", schema)\nprint(result[\"output\"])\nprint(result[\"fields\"][\"priority\"][\"scores\"])  # Candidate probabilities.\nprint(result[\"fields\"][\"priority\"][\"logits\"])\n```\n\nThe output has this shape. The values depend on the model:\n\n```json\n{\"priority\": \"HIGH\", \"requires_review\": true}\n```\n\nLoad the scorer once and reuse it for each new context. Both scorers use a\ntemperature of `1.0` by default. We have not tuned the temperature so that the\nprobabilities match how often the answers are right. See the\n[scoring guide](docs\u002FPARALLEL_SCORING.md) for the schema rules and the ways you\ncan run the scorer.\n\n## Methodology\n\nThe serving methodology and training data curation are heavily inspired by [Bespoke-MiniCheck](https:\u002F\u002Fhuggingface.co\u002Fbespokelabs\u002FBespoke-MiniCheck-7B).\n\n### Serving\nWe follow the approach laid out by [Niels Rogge](https:\u002F\u002Fx.com\u002FNielsRogge): see this [post on how Jev does decoding](https:\u002F\u002Fx.com\u002FNielsRogge\u002Fstatus\u002F2100239244501430438).\n\nIn a nutshell:\n* Process the context and schema once (prefill the KV-cache)\n* We obtain the scores for the tokens we care about.\n\nWe used this approach (we didn't have to do kv-cache prefill) in Bespoke-MiniCheck, since it always returned a single json tuple: `{\"is_claim_supported_by_context\": p}`.\n\n### Contrastive data curation\n\nThe challenge here is that we don't have access to probabilities from a teacher model (or humans). But as you saw above, we just need to somehow get the logits and ensure the logits are as calibrated to estimate the probabilities as possible.\n\nWe push the model to be calibrated to be a better decision maker by creating negative examples, which forces the model to become a better discriminator.\n\nSo we made the training data with a new method that we call contrastive data\ncuration. In this method, we write two examples that are almost the same. They\ndiffer in one relevant fact, and this difference changes the correct answer.\nEverything else stays the same, including the question and the policy. From\nthese pairs, the model learns which evidence should change its decision.\n\n[![Watch the contrastive data curation video: changing the signer from Mira to Noah flips the answer from true to false.](assets\u002Fvideo\u002Fcontrastive-curation\u002Fposter.jpg)](assets\u002Fvideo\u002Fcontrastive-curation\u002Fcontrastive-curation.mp4)\n\n[Watch the 34-second explanation](assets\u002Fvideo\u002Fcontrastive-curation\u002Fcontrastive-curation.mp4) (silent).\n\nIn this example, the rule is that a refund is authorized only when its sole\nauthorization was signed by someone who can approve refunds for that account.\nThe example has two evidence sentences, and you need both of them to answer.\n\n| Evidence | First example | Changed example |\n| --- | --- | --- |\n| Who can approve refunds | Only Mira may authorize refunds for account 42. | Unchanged |\n| Authorization record | The sole authorization for this refund on account 42 was signed by **Mira**. | The sole authorization for this refund on account 42 was signed by **Noah**. |\n| Is the refund authorized? | `true` | `false` |\n\nOur curation pipeline uses this idea for Choice, Noul and Score tasks. It has\nfour steps:\n\n1. Check the decision rules. We start from the schema and the policy of a\n   training source. We write down the basic facts that the rules depend on.\n   Then we check that the rules can lead to different answers.\n2. Build the pair. We write two evidence sentences, and you need both of them\n   to find the answer. Then we change at most eight words in one of the\n   sentences. We make this edit to change one fact, which we call the focus\n   fact, and so the label changes too. The rest of the context and the other\n   facts stay the same.\n3. Check both examples. We use separate model calls to check the facts in each\n   example. We also use these calls to check that the example is consistent\n   with the policy and that the text does not hint at the answer. Then we\n   remove each evidence sentence in turn. With either sentence removed, the\n   focus fact must become unknown, even with all of the other text present.\n   This way, we know that no other text gives away the answer.\n4. Make the labels. We use code to apply the checked rules to the checked\n   facts. We keep a pair only when every required check passes and the two\n   labels differ. We save all model requests and responses, along with the\n   check results. You can use them to replay the whole process offline.\n\nThe examples with an evidence sentence removed are only checks. We do not add\nthem to the training data with labels, because missing evidence does not mean\nthat the answer is false or that the score is low. A source family is the group\nof examples that we built from one training source. We keep both examples of a\npair, and all examples from one source family, in the same split. The split is\neither training or evaluation.\n\n#### Current dataset\n\nThere is one training set and one held-out set.\n\n| File | Examples | Use |\n| --- | ---: | --- |\n| `data\u002Ftrain.jsonl` | 2,826 | 2,676 used to train the published model |\n| `data\u002Feval.jsonl` | 324 | Final evaluation only |\n\nThe published model's training data covers **10 subject categories**. The tables\nbelow count only its 2,676 training examples and the 324-example holdout.\nCategory counts come from each record's `domain` field; the holdout covers six\nof these categories.\n\n| Category | Training examples | Held-out examples |\n| --- | ---: | ---: |\n| Commerce | 242 | 58 |\n| Education | 230 | 70 |\n| Home | 300 | 0 |\n| Media | 256 | 44 |\n| Public services | 194 | 106 |\n| Science | 300 | 0 |\n| Software | 300 | 0 |\n| Supply chain | 270 | 30 |\n| Travel | 284 | 16 |\n| Workplace | 300 | 0 |\n| **Total** | **2,676** | **324** |\n\n\nAcross these subjects, each example asks one of three\n[typed questions](https:\u002F\u002Fdocs.typesafe.ai\u002Fprimitives):\n\n| Task type | Judgment | Training examples | Held-out examples |\n| --- | --- | ---: | ---: |\n| Choice | Select one candidate | 856 | 146 |\n| Noul (Boolean) | Decide whether a condition holds | 888 | 114 |\n| Score | Judge an ordered rubric level | 932 | 64 |\n| **Total** | | **2,676** | **324** |\n\nAll of the labels are synthetic: a model checked them, and no person has reviewed them. Separate calls to the same model can make the same mistake, so the checks can miss some errors. See the\n[generation and replay guide](docs\u002FTRAINING_EVAL_CURATION.md) for the full\nmethod.\n\nThe command below checks the data files. It needs no GPU and makes no API\ncalls. It confirms that the files match their saved checksums and have the\nexpected number of examples. It also confirms that no example or source family\nappears in both the training set and the evaluation set.\n\n```sh\n.venv-curator\u002Fbin\u002Fpython -m nimble.training.verify_dataset\n```\n\nSee the [dataset details](docs\u002FDATASET.md) and the\n[training guide](docs\u002FNIMBLE_TRAINING.md). The curation guides also describe\nolder ways that we built data. We no longer keep the source folders from those\nolder methods.\n\n### Finetuning\n\nThe trainer applies LoRA to Qwen3.5-9B, optimizing cross-entropy over the allowed candidate logits. It learns the typed decision directly. Saved Jev probabilities are available for future soft-target distillation if you need it, but the current training objective uses hard reference labels that are not derived from Jev.\n\nThe final recipe uses these settings:\n\n- LoRA rank 16\n- Learning rate 5e-5\n- Effective batch size 8\n- Random seed 17\n- One epoch of training\n\nIt preserves the three-epoch linear learning rate schedule used during selection, stopping after the selected epoch. Training uses BF16 and a 2,048-token prompt limit. Tuning ran on L40S; the final fit and evaluation ran on H100.\n\n### Evaluation on 324 held-out examples\n\n![Reference-label agreement on the same 324 examples: Jev 93.21%, Bespoke-Nimble-9B 90.12%, Qwen3.8-27B 84.88%, Qwen3.5-9B 66.36%, Qwen3.5-4B 61.42%, Qwen3.5-0.8B 45.37%, and Gemma 3 270M IT 28.70%.](assets\u002Fevidence-324-comparison.svg)\n\n| Model | Reference matches | Agreement |\n| --- | ---: | ---: |\n| Gemma 3 270M IT | 93\u002F324 | 28.70% |\n| Qwen3.5-0.8B | 147\u002F324 | 45.37% |\n| Qwen3.5-4B | 199\u002F324 | 61.42% |\n| Qwen3.5-9B | 215\u002F324 | 66.36% |\n| Qwen3.8-27B | 275\u002F324 | 84.88% |\n| **Bespoke-Nimble-9B** | **292\u002F324** | **90.12%** |\n| Jev 1.13.0 | 302\u002F324 | 93.21% |\n\nOn these 324 examples, Bespoke-Nimble-9B matched 17 more reference labels than\nthe untuned 27B model, which is 5.25 percentage points more. Jev matched 10 more\nreference labels than Bespoke-Nimble-9B, which is 3.09 points more. The untuned\nmodels are models that we did not fine-tune for this task. We tested all seven\nmodels on the same examples with the same reference labels. We ran Gemma and\nthe untuned Qwen models on an H100 GPU. For Bespoke-Nimble-9B, we reused\nchecked results from an earlier H100 run. For Jev, we reused results from an\nearlier run through its API.\n\nThe reference labels are synthetic. The 324 examples form 162 pairs of closely\nrelated examples. All of them come from only six source families, so this is a narrow\ntest.\n\nFor the untuned models, we computed the scores of the candidate answers in FP32,\na 32-bit number format. For Bespoke-Nimble-9B, we kept the setup that we had\nalready checked, which computes the output layer in BF16. In a new run of the\nuntuned 9B model, one answer that had been a 50\u002F50 tie was no longer a tie. This\nchanged the model's count from 214 to 215. For rating tasks, we count a match\nwhen the most probable level equals the reference level. The\n[machine-readable comparison](assets\u002Fevidence-324-results.json) contains\nthe reference-match counts shown above.\n\nWe checked the saved adapter in two ways. After we reloaded it, it gave exactly\nthe same logits as before. When we turned the adapter off, the model gave the\nsame results as the base model. We ran these checks only on the CUDA path that\nloads the adapter without merging it. We did not run a separate quality test on\nthe merged model that the Mac and Linux quickstarts use. See the\n[training guide](docs\u002FNIMBLE_TRAINING.md) for the commands and for the contract\nfile saved with each model. In the contract file, we record the base model and\nthe prompt format that the model expects.\n\nFor external, human-labeled tests on tasks outside these training categories, see\nthe [public benchmarks guide](docs\u002FPUBLIC_BENCHMARKS.md), which runs Bespoke-Nimble-9B\nand Jev on the same records from thirteen public subsets, starting with VitaminC.\n\n### Observed inference latency\n\nIn the table below, each time is in milliseconds per example. We calculated\nthese times from the timing that we saved for each request. Each example has\none question. The local models and Jev return a typed answer and the\nprobability of each candidate answer. The OpenRouter models generate text in\nthe usual way, with the reasoning setting at medium. For local scoring, we run\nthe model once per example. We do not sample more than once, and the model does\nnot write an explanation.\n\n| Model | Examples | Median (ms) | Mean (ms) | p95 (ms) | Run \u002F dataset |\n| --- | ---: | ---: | ---: | ---: | --- |\n| Gemma 3 270M IT | 324 | 21.8 | 23.6 | 29.2 | H100 · contrastive holdout |\n| Qwen3.5-0.8B | 324 | 48.6 | 49.4 | 60.2 | H100 · contrastive holdout |\n| Qwen3.5-4B | 324 | 58.0 | 58.6 | 70.3 | H100 · contrastive holdout |\n| Qwen3.5-9B | 324 | 58.1 | 59.8 | 75.8 | H100 · contrastive holdout |\n| Qwen3.8-27B | 324 | 145.3 | 145.2 | 185.5 | H100 · contrastive holdout |\n| Bespoke-Nimble-9B | 120 | 106.0 | 110.1 | 119.8 | H100 · contrastive holdout|\n| Bespoke-Nimble-9B | 324 | 444.0 | 546.0 | 981.0 | M5 Pro 64GB · contrastive holdout|\n| Jev 1.13.0 | 324 | 246.7 | 267.0 | 347.4 | TypeSafe API · contrastive holdout |\n| DeepSeek-V4.1-Flash | 100 | 2896.4 | 5238.0 | 15065.4 | OpenRouter \u002F Fireworks · general eval |\n| Qwen3.8 2.4T A95B | 100 | 2792.3 | 3108.0 | 5110.5 | OpenRouter \u002F Modal · general eval |\n\n\n## Documentation and development\n\n| I want to… | Start here |\n| --- | --- |\n| Define fields or understand parallel scoring | [Scoring guide](docs\u002FPARALLEL_SCORING.md) |\n| Compare Nimble and Jev interactively | [Comparison app](docs\u002FCOMPARISON_APP.md) |\n| Host the published checkpoint on Modal | [SGLang deployment](docs\u002FMODAL_SERVING.md) · [Try the public API](docs\u002FTRY_NIMBLE.md) |\n| Understand the retained training and evaluation data | [Dataset guide](docs\u002FDATASET.md) |\n| Evaluate on public, human-labeled benchmarks | [Public benchmarks](docs\u002FPUBLIC_BENCHMARKS.md) |\n| Create or replay contrastive training data | [Curation guide](docs\u002FTRAINING_EVAL_CURATION.md) |\n| Train or use a schema adapter | [Training guide](docs\u002FNIMBLE_TRAINING.md) |\n\nCompare two models only when both were tested on the same examples in the same\nway. The guides describe how to generate data and evaluation outputs locally;\nthese generated files are not committed to Git.\n\n### Development\n\nUse a separate Python environment for each of these, because they need\ndifferent package versions:\n\n- MLX\n- PyTorch\n- Curator\n\nThe dependency lists are in [requirements\u002F](requirements\u002F). From the project\nroot, run the offline tests that apply to your change.\n\n```sh\n.venv-mlx\u002Fbin\u002Fpython -m unittest tests.test_parallel_scorer tests.test_evaluate_pilot tests.test_model_evaluation\n.venv-curator\u002Fbin\u002Fpython -m unittest tests.test_dataset_io tests.test_diverse_dataset\n```\n\nTo measure how much time MLX saves by processing the shared context once for\nseveral fields, run the benchmark for schemas with several fields.\n\n```sh\n.venv-mlx\u002Fbin\u002Fpython -m nimble.evaluation.benchmark_parallel --repeats 3\n```\n\nEach record in the saved dataset evaluations has only one field. So you cannot\nuse those evaluations to measure the time saved by scoring several fields in\nparallel. Their recorded times are also not from a controlled serving test.\n\n### Folder layout\n\n```text\nnimble\u002F\n  scoring\u002F       # Local MLX and PyTorch inference\n  datasets\u002F      # Generation, curation, and TypeSafe labeling\n  evaluation\u002F    # Quality metrics and inference benchmarks\n  training\u002F      # Schema-aware CUDA LoRA training\nexamples\u002F        # Schemas and saved example outputs\ndata\u002F            # train.jsonl (2,826), eval.jsonl (324), and verification metadata\nevaluations\u002F     # Generated locally; not included in Git\nassets\u002F          # Comparison graphics, source results, and videos\ndocs\u002F            # Current usage, training, curation, and deployment guides\nignore\u002F          # Local archive of historical docs and assets; not included in Git\nrequirements\u002F    # Backend-specific dependencies\ntests\u002F           # Offline and small-model checks\n```\n\n## Citation\n\nIf you use this repository, please cite it. Also give the model revision and\nthe dataset release that you used.\n\n```bibtex\n@misc{nimble2026,\n  author = {{Bespoke Labs} and Sathiamoorthy, Maheswaran},\n  title = {Nimble},\n  year = {2026},\n  howpublished = {\\url{https:\u002F\u002Fgithub.com\u002Fbespokelabsai\u002Fnimble}},\n  note = {Model: https:\u002F\u002Fhuggingface.co\u002Fbespokelabs\u002FBespoke-Nimble-9B}\n}\n```\n\n## Acknowledgments\n\n1. [TypeSafe](https:\u002F\u002Ftypesafe.ai) for making Jev.\n2. [Niels Rogge](https:\u002F\u002Fx.com\u002FNielsRogge) for a [post on how Jev does decoding](https:\u002F\u002Fx.com\u002FNielsRogge\u002Fstatus\u002F2100239244501430438).\n3. [Harsha Gundala](https:\u002F\u002Fx.com\u002Fharshagundal) for [inspiring us to work on this](https:\u002F\u002Fx.com\u002Fharshagundal\u002Fstatus\u002F2100044305536889015).\n4. [Greg Durett](https:\u002F\u002Fgregdurrett.github.io\u002F) and [Liyan Tang](https:\u002F\u002Fwww.tangliyan.com\u002F)'s work on MiniCheck (and check out Bespoke-MiniCheck which we did with them), which was two years early and laid the foundations.\n","2026-09-20 02:30:09","CREATED_QUERY"]