[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-78637":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":32,"readmeContent":33,"aiSummary":34,"trendingCount":16,"starSnapshotCount":16,"syncStatus":35,"lastSyncTime":36,"discoverSource":37},78637,"needle","cactus-compute\u002Fneedle","cactus-compute","26m function call model that runs on incredibly small devices","https:\u002F\u002Fcactuscompute.com",null,"Python",2592,174,20,7,0,27,61,193,81,108.73,"MIT License",false,"main",true,[27,28,29,30,31],"cactus","gemini","gemma","llm","on-device-ai","2026-06-12 04:01:23","# Needle\n\n\u003Cimg src=\"assets\u002Fbanner.png\" alt=\"Logo\" style=\"border-radius: 30px; width: 100%;\">\n\nWe distilled Gemini 3.1 into a 26m parameter \"[Simple Attention Network](docs\u002Fsimple_attention_networks.md)\" that you can even finetune locally on your Mac\u002FPC.\nIn production, Needle runs on [Cactus](https:\u002F\u002Fgithub.com\u002Fcactus-compute\u002Fcactus) at 6000 toks\u002Fsec prefill and 1200 decode speed. \nWeights are fully open on [Cactus-Compute\u002Fneedle](https:\u002F\u002Fhuggingface.co\u002FCactus-Compute\u002Fneedle), as well as the dataset generation. \n\n```\nd=512, 8H\u002F4KV, BPE=8192\n                                  ┌──────────────┐\n                                  │  Tool Call   │\n                                  └──────┬───────┘\n                                        ┌┴──────────┐\n                                        │  Softmax  │\n                                        └─────┬─────┘\n                                        ┌─────┴─────┐\n                                        │ Linear (T)│  ← tied\n                                        └─────┬─────┘\n                                        ┌─────┴─────┐\n                                        │ ZCRMSNorm │\n                                        └─────┬─────┘\n                                     ┌────────┴────────┐\n                                     │ Decoder x 8     │\n                                     │┌───────────────┐│\n                                     ││ ZCRMSNorm     ││\n                                     ││ Masked Self   ││\n                                     ││ Attn + RoPE   ││\n                                     ││ Gated Residual││\n                                     │├───────────────┤│\n  ┌──────────────┐                   ││ ZCRMSNorm     ││\n  │ Encoder x 12 │──────────────────────▶Cross Attn   ││\n  │              │                   ││ Gated Residual││\n  │ ┌──────────┐ │                   │└───────────────┘│\n  │ │ZCRMSNorm │ │                   └────────┬────────┘\n  │ │Self Attn │ │                      ┌─────┴─────┐\n  │ │ GQA+RoPE │ │                      │ Embedding │  ← shared\n  │ │Gated Res │ │                      └─────┬─────┘\n  │ │          │ │                    ┌───────┴───────-┐\n  │ │ (no FFN) │ │                    │[EOS]\u003Ctool_call>│\n  │ └──────────┘ │                    │ + answer       │\n  │              │                    └───────────────-┘\n  └──────┬───────┘\n         │\n    ┌────┴──────┐\n    │ Embedding │\n    └────┬──────┘\n         │\n    ┌────┴──────┐\n    │   Text    │\n    │  query    │\n    └───────────┘\n```\n\n- Pretrained on 16 TPU v6e for 200B tokens (27hrs). \n- Post-trained on 2B tokens of single-shot function call dataset (45mins). \n\nNeedle is an experimental run for Simple Attention Networks, geared at redefining tiny AI for consumer devices (phones, watches, glasses...).\nSo while it beats FunctionGemma-270m, Qwen-0.6B, Graninte-350m, LFM2.5-350m on single-shot function call for personal AI,\nThose model are have more scope\u002Fcapacity and excel in conversational settings. Also, small models can be finicky. \nPlease use the UI in the next section to test on your own tools, and finetune accordingly, at the click of a button. \n\n## Quickstart\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fcactus-compute\u002Fneedle.git\ncd needle && source .\u002Fsetup\nneedle playground\n```\n\nOpens a web UI at http:\u002F\u002F127.0.0.1:7860 where you can test and finetune on your own tools. Weights are auto-downloaded.\n\n## Usage (Python)\n\n```python\nfrom needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer\n\nparams, config = load_checkpoint(\"checkpoints\u002Fneedle.pkl\")\nmodel = SimpleAttentionNetwork(config)\ntokenizer = get_tokenizer()\n\nresult = generate(\n    model, params, tokenizer,\n    query=\"What's the weather in San Francisco?\",\n    tools='[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"location\":{\"type\":\"string\",\"description\":\"City name.\",\"required\":true}}}]',\n    stream=False,\n)\nprint(result)\n# [{\"name\":\"get_weather\",\"arguments\":{\"location\":\"San Francisco\"}}]\n```\n\n## Finetuning\n\n```bash\n# Playground (generates data via Gemini, trains, evaluates, bundles result)\nneedle playground\n\n# CLI (auto-downloads weights if not local)\nneedle finetune data.jsonl\n```\n\n### Data format\n\nEach line in the JSONL file has three fields: `query`, `tools`, and `answers`.\n\n**Tool schema:**\n```json\n{\n  \"name\": \"get_weather\",\n  \"description\": \"Get current weather for a city.\",\n  \"parameters\": {\n    \"location\": { \"type\": \"string\", \"description\": \"City name.\", \"required\": true }\n  }\n}\n```\n\n**Answer schema:**\n```json\n{ \"name\": \"get_weather\", \"arguments\": { \"location\": \"Paris\" } }\n```\n\n**Full JSONL example** (each line is one training example, `tools` and `answers` are JSON-encoded strings):\n```jsonl\n{\"query\": \"What's the weather in Paris?\", \"tools\": \"[{\\\"name\\\":\\\"get_weather\\\",\\\"description\\\":\\\"Get current weather for a city.\\\",\\\"parameters\\\":{\\\"location\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"City name.\\\",\\\"required\\\":true}}}]\", \"answers\": \"[{\\\"name\\\":\\\"get_weather\\\",\\\"arguments\\\":{\\\"location\\\":\\\"Paris\\\"}}]\"}\n{\"query\": \"Turn off the lights\", \"tools\": \"[{\\\"name\\\":\\\"get_weather\\\",\\\"description\\\":\\\"Get current weather for a city.\\\",\\\"parameters\\\":{\\\"location\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"City name.\\\",\\\"required\\\":true}}},{\\\"name\\\":\\\"toggle_lights\\\",\\\"description\\\":\\\"Toggle smart lights on or off.\\\",\\\"parameters\\\":{\\\"state\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"on or off.\\\",\\\"required\\\":true}}}]\", \"answers\": \"[{\\\"name\\\":\\\"toggle_lights\\\",\\\"arguments\\\":{\\\"state\\\":\\\"off\\\"}}]\"}\n```\n\nProvide at least **120 examples per tool** (100 train \u002F 10 val \u002F 10 test). Fewer examples will overfit — you'll see perfect training metrics but the model won't generalize. Vary query phrasing and include examples with multiple tools available.\n\n### Using a finetuned model\n\nFinetuning saves the best checkpoint as `checkpoints\u002Fneedle_finetuned_\u003Cid>_best.pkl`:\n\n```bash\nneedle run --checkpoint checkpoints\u002Fneedle_finetuned_*_best.pkl \\\n  --query \"What's the weather?\" --tools '[{\"name\":\"get_weather\",\"description\":\"Get current weather for a city.\",\"parameters\":{\"location\":{\"type\":\"string\",\"description\":\"City name.\",\"required\":true}}}]'\n```\n\n```python\nparams, config = load_checkpoint(\"checkpoints\u002Fneedle_finetuned_\u003Cid>_best.pkl\")\nmodel = SimpleAttentionNetwork(config)\nresult = generate(model, params, get_tokenizer(), query=\"...\", tools='[...]', stream=False)\n```\n\n## CLI\n\n```\nneedle playground                  Test and finetune via web UI\nneedle finetune \u003Cdata.jsonl>       Finetune on your own data\nneedle run --query \"...\" --tools   Single inference\nneedle train                       Full training run\nneedle pretrain                    Pretrain on PleIAs\u002FSYNTH\nneedle eval --checkpoint \u003Cpath>    Evaluate a checkpoint\nneedle tokenize                    Tokenize dataset\nneedle generate-data               Synthesize training data via Gemini\nneedle tpu \u003Caction>                TPU management (see docs\u002Ftpu.md)\n```\n\n```\n@misc{ndubuaku2026needle,\n  title={Needle},\n  author={Henry Ndubuaku, Jakub Mroz,  Karen Mosoyan, Roman Shemet, Parkirat Sandhu, Satyajit Kumar, Noah Cylich, Justin H. Lee},\n  year={2026},\n  url={https:\u002F\u002Fgithub.com\u002Fcactus-compute\u002Fneedle}\n}\n```\n","Needle 是一个专为极小设备设计的26M参数函数调用模型。它基于Gemini 3.1精简而来，采用简单注意力网络架构，能够在本地Mac或PC上进行微调。核心功能包括在Cactus平台上以每秒6000个token的预填充速度和1200个token的解码速度运行，支持工具调用，并且权重完全开源。适用于手机、手表、眼镜等消费级设备上的个人AI应用，特别适合单次函数调用场景，在此场景下其性能优于一些更大规模的模型如FunctionGemma-270m、Qwen-0.6B等。尽管这些大模型在对话场景中表现更佳，但Needle以其轻量级特性重新定义了小型设备上的AI应用。",2,"2026-06-11 03:57:06","high_star"]