[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92375":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":14,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":15,"rankGlobal":8,"rankLanguage":8,"license":8,"archived":16,"fork":16,"defaultBranch":17,"hasWiki":16,"hasPages":16,"topics":18,"createdAt":8,"pushedAt":8,"updatedAt":19,"readmeContent":20,"aiSummary":21,"trendingCount":13,"starSnapshotCount":13,"syncStatus":14,"lastSyncTime":22,"discoverSource":23},92375,"local-coding-agent-evals","rasbt\u002Flocal-coding-agent-evals","rasbt",null,"Python",54,13,52,0,2,37.64,false,"main",[],"2026-07-22 04:02:06","# Local coding agent evals\n\nThis repository contains the small scripts and task packs I used for my article.\n\nThe goal is not to provide a comprehensive benchmark. These are practical checks I used while comparing local models and coding-agent harnesses such as Qwen Code, Codex, and Claude Code with Ollama-hosted models.\n\n\n \n## Contents\n\n- `speed-memory-benchmark\u002F`: measures Ollama prefill speed, decode speed, wall time, and memory use across longer prompts. This test the LLM + inference engine.\n- `hard-tool-reasoning-benchmark\u002F`: asks an Ollama model to return one tool decision for five harder reasoning tasks. It does not execute tools. This tests the LLM base capability.\n- `agent-problem-pack\u002F`: contains five small coding tasks for testing an actual coding-agent harness in isolated workspaces based on the reasoning tasks above. This tests the LLM + harness combo.\n\n\n \n## Setup\n\nClone the repository:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Frasbt\u002Flocal-coding-agent-evals.git\ncd local-coding-agent-evals\n```\n\nInstall [Ollama](https:\u002F\u002Follama.com\u002F) and pull a model:\n\n```bash\nollama pull qwen3.6:35b\n```\n\nor, if you are on a Mac:\n\n```bash\nollama pull qwen3.6:35b-mlx\n```\n\nThe examples below use `uv` for Python script execution where useful.\n\n \n\n\n![](https:\u002F\u002Fsebastianraschka.com\u002Fimages\u002Fgithub\u002Flocal-coding-agent-evals\u002F1.png)\n\n\n\n \n## Speed and memory benchmark\n\n```bash\nuv run speed-memory-benchmark\u002Follama_speed_memory_bench.py \\\n  --model qwen3.6:35b \\\n  --csv ollama_speed_memory_results.csv\n```\n\nOn macOS with an MLX model:\n\n```bash\nuv run speed-memory-benchmark\u002Follama_speed_memory_bench.py \\\n  --model qwen3.6:35b-mlx \\\n  --csv ollama_speed_memory_results.csv\n```\n\nSee [speed-memory-benchmark\u002FREADME.md](speed-memory-benchmark\u002FREADME.md) for details.\n\n \n## Hard tool-reasoning benchmark\n\n```bash\npython3 hard-tool-reasoning-benchmark\u002Follama_hard_reasoning_bench.py \\\n  --model qwen3.6:35b \\\n  --csv hard_reasoning_results.csv\n```\n\nSee [hard-tool-reasoning-benchmark\u002FREADME.md](hard-tool-reasoning-benchmark\u002FREADME.md) for the task format and scoring.\n\n \n## Agent problem pack\n\nThe problem pack is for testing a real coding-agent harness, not just a raw model response.\n\nSee [agent-problem-pack\u002FREADME.md](agent-problem-pack\u002FREADME.md) for the full workflow.\n\n \n\n| Harness   | qwen3.6:35b | north-mini-code-1.0:q4_K_M | gemma4:e2b | nemotron-3-nano:latest |\n| --------- | ----------: | -------------------------: | ----------: | ---------------------: |\n| claude    |         5\u002F5 |                        5\u002F5 |         3\u002F5 |                    4\u002F5 |\n| codex     |         5\u002F5 |                        5\u002F5 |         0\u002F5 |                    5\u002F5 |\n| qwen-code |         4\u002F5 |                        4\u002F5 |         1\u002F5 |                    4\u002F5 |\n\n \n\n![](https:\u002F\u002Fsebastianraschka.com\u002Fimages\u002Fgithub\u002Flocal-coding-agent-evals\u002F2.png)\n\n\n \n## Notes\n\nRun agent benchmarks (`agent-problem-pack`) in an isolated workspace, separate user account, VM, or separate machine when possible. Coding agents can read files and run commands, which is also what makes them useful.\n\nThe result files in this repository are examples from my runs. They are meant as a starting point for comparison, not as final model rankings.\n","这是一个面向本地部署编程智能体的轻量级评估工具集，用于系统性测试大语言模型及其配套编码代理框架在实际开发任务中的表现。项目包含三类评测：模型推理速度与内存占用基准、复杂工具调用推理能力测试、以及端到端编码代理任务执行验证（支持Ollama托管模型）。所有评测均在本地运行，不依赖云端API，强调可复现性与硬件感知。适用于开发者和研究人员对开源代码模型（如Qwen Code、Gemma等）及本地代理框架进行快速横向对比与性能调优。","2026-07-08 04:30:13","CREATED_QUERY"]