[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80624":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":15,"forks30d":15,"starsTrendScore":19,"compositeScore":20,"rankGlobal":9,"rankLanguage":9,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":24,"hasPages":22,"topics":25,"createdAt":9,"pushedAt":9,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":15,"starSnapshotCount":15,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},80624,"TinySearch","MarcellM01\u002FTinySearch","MarcellM01","Shrink the web for your local LLMs!",null,"Python",73,6,51,5,0,4,19,22,26,62.24,"MIT License",false,"main",true,[],"2026-06-12 04:01:29","# TinySearch\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Ftinysearch_logo.png\" alt=\"TinySearch\" width=\"240\" \u002F>\n\u003C\u002Fp>\n\n[![License: MIT](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-MIT-yellow.svg)](LICENSE)\n[![Release](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fv\u002Frelease\u002FMarcellM01\u002FTinySearch?label=release)](https:\u002F\u002Fgithub.com\u002FMarcellM01\u002FTinySearch\u002Freleases)\n[![Last Commit](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002FMarcellM01\u002FTinySearch)](https:\u002F\u002Fgithub.com\u002FMarcellM01\u002FTinySearch\u002Fcommits\u002Fmain)\n[![Docker Pulls](https:\u002F\u002Fimg.shields.io\u002Fdocker\u002Fpulls\u002Fmarcellm01\u002Ftinysearch?label=docker%20pulls)](https:\u002F\u002Fhub.docker.com\u002Fr\u002Fmarcellm01\u002Ftinysearch)\n![MCP Server](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FMCP-server-blue)\n![FastAPI](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FFastAPI-supported-009688)\n\nA tiny local-first web research engine for MCP agents.\n\nTinySearch searches the web, reranks results, crawls the best pages, extracts\nthe most relevant chunks, and returns a source-grounded prompt your LLM can\nanswer from.\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fdemo_terminal_prompt.gif\" alt=\"TinySearch terminal demo showing a source-grounded research prompt\" width=\"780\" \u002F>\n\u003C\u002Fp>\n\nNo hosted dashboard. No account system. No analytics. No scraped-data cache.\n\nJust search -> crawl -> rerank -> grounded prompt.\n\n## Quick start\n\nRun TinySearch as an MCP server over Streamable HTTP:\n\n```bash\ndocker run --rm -p 8000:8000 -e MCP_TRANSPORT=streamable-http -e MCP_HOST=0.0.0.0 marcellm01\u002Ftinysearch:latest\n```\n\nThen connect your MCP client to:\n\n```json\n{\n  \"mcpServers\": {\n    \"tinysearch\": {\n      \"url\": \"http:\u002F\u002Flocalhost:8000\u002Fmcp\"\n    }\n  }\n}\n```\n\nTinySearch exposes one MCP tool:\n\n```text\nresearch(query)\n```\n\nPass the user's question as-is. TinySearch searches, crawls, reranks, and\nreturns the grounded prompt in `answer`.\n\n## Why TinySearch?\n\n- Give local agents web research without wiring together a whole search stack.\n- Keep source URLs attached to the evidence your model sees.\n- Avoid dumping full webpages into context.\n- Use local ONNX embeddings or an OpenAI-compatible embedding API.\n- Run over MCP or a simple FastAPI endpoint.\n\nTinySearch is built for local agents, prototypes, personal workflows, and small\nsystems where source-grounded web research matters more than running a full\nsearch backend.\n\n## How it works\n\n```mermaid\nflowchart TB\n    subgraph Row1[\"Search and choose pages\"]\n        direction LR\n        A[User query] --> B[DuckDuckGo HTML search]\n        B --> C[Filter HTTP results\u003Cbr\u002F>build title URL domain snippet docs]\n        C --> D[Rank search docs\u003Cbr\u002F>dense + BM25 weighted RRF]\n    end\n\n    subgraph Row2[\"Crawl and build prompt\"]\n        direction LR\n        E[Crawl kept URLs in parallel\u003Cbr\u002F>crawl4ai markdown] --> F[Truncate and chunk markdown]\n        F --> G[Rank combined chunk pool\u003Cbr\u002F>dense + BM25 weighted RRF]\n        G --> H[Dedupe chunks\u003Cbr\u002F>apply source quotas and fill]\n        H --> I[Build source-grounded prompt]\n    end\n\n    Row1 --> Row2\n```\n\nTinySearch does not directly answer the question. It returns a\n**structured prompt** in the MCP tool's **`answer` field**, and your\n**client model** uses that prompt to produce the final **cited response**.\n\n```text\nQUESTION\nWhat happened in the latest NFL playoffs?\n\nTODAY\n2026-05-15\n\nRESULTS\n1. Title\n   URL\n   Relevant extracted text...\n\n2. Title\n   URL\n   Relevant extracted text...\n\nINSTRUCTIONS\nAnswer only from the results. Cite source URLs.\n```\n\n## Run from source\n\nUse this path if you want to inspect the code, edit TinySearch, or run it as a\nlocal stdio MCP server.\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FMarcellM01\u002FTinySearch\ncd TinySearch\n\npython -m venv .venv\nsource .venv\u002Fbin\u002Factivate\npip install -r requirements.txt\n```\n\nMCP clients spawn TinySearch from their config. Add it with absolute paths:\n\nmacOS \u002F Linux:\n\n```json\n{\n  \"mcpServers\": {\n    \"tinysearch\": {\n      \"command\": \"\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch\u002F.venv\u002Fbin\u002Fpython\",\n      \"args\": [\n        \"\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch\u002Fservers\u002Fmcp_server.py\"\n      ]\n    }\n  }\n}\n```\n\nWindows:\n\n```json\n{\n  \"mcpServers\": {\n    \"tinysearch\": {\n      \"command\": \"C:\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch\u002F.venv\u002FScripts\u002Fpython.exe\",\n      \"args\": [\n        \"C:\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch\u002Fservers\u002Fmcp_server.py\"\n      ]\n    }\n  }\n}\n```\n\nTemplate config files live in `mcp_templates\u002F`.\n\nThe repo also includes [`agentic_coding_templates\u002Fglobal-rules-recommended.md`](agentic_coding_templates\u002Fglobal-rules-recommended.md),\na global-rules template for agentic coding tools such as Cline and Roo Code.\nThese rules help coding agents call TinySearch only when web research is\nactually needed.\n\nThe server uses **stdio** by default, which is what Cursor and similar clients\nexpect when they spawn `python ...\u002Fmcp_server.py`. To run with `sse` or\n`streamable-http`, set `MCP_TRANSPORT` when starting the process. Do not put\ntransport in `configs\u002Fresearch_config.json`.\n\n## Docker\n\nThe [quick start](#quick-start) command runs TinySearch over Streamable HTTP on\n`http:\u002F\u002Flocalhost:8000\u002Fmcp`. Docker pulls `marcellm01\u002Ftinysearch:latest`\nautomatically if the image is not already local.\n\nWith `MCP_TRANSPORT=streamable-http`, the image serves Streamable HTTP on\n`\u002Fmcp` and SSE on `\u002Fmcp\u002Fsse`. GET requests to `\u002Fmcp` without an\n`mcp-session-id` are treated as the legacy SSE stream. If a client still cannot\nconnect, try `MCP_TRANSPORT=sse` alone or the stdio Docker setup below.\n\n### Persistent models and config\n\nFor repeated use, keep downloaded models in a Docker volume and mount your local\nconfig:\n\n```bash\ndocker run --rm \\\n  -p 8000:8000 \\\n  -v tinysearch-models:\u002Fdata\u002Fmodels \\\n  -v \"$PWD\u002Fconfigs\u002Fresearch_config.json:\u002Fconfig\u002Fresearch_config.json:ro\" \\\n  -e TINYSEARCH_CONFIG_PATH=\u002Fconfig\u002Fresearch_config.json \\\n  -e MCP_TRANSPORT=streamable-http \\\n  -e MCP_HOST=0.0.0.0 \\\n  marcellm01\u002Ftinysearch:latest\n```\n\n### MCP over stdio\n\nUse this mode for MCP clients that launch tools as local commands instead of\nconnecting to a URL. Replace `\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch` with this repo's\nabsolute path:\n\n```json\n{\n  \"mcpServers\": {\n    \"tinysearch\": {\n      \"command\": \"docker\",\n      \"args\": [\n        \"run\",\n        \"--rm\",\n        \"-i\",\n        \"-v\",\n        \"tinysearch-models:\u002Fdata\u002Fmodels\",\n        \"-v\",\n        \"\u002Fabsolute\u002Fpath\u002Fto\u002FTinySearch\u002Fconfigs\u002Fresearch_config.json:\u002Fconfig\u002Fresearch_config.json:ro\",\n        \"-e\",\n        \"TINYSEARCH_CONFIG_PATH=\u002Fconfig\u002Fresearch_config.json\",\n        \"-e\",\n        \"TINYSEARCH_MODELS_DIR=\u002Fdata\u002Fmodels\",\n        \"marcellm01\u002Ftinysearch:latest\"\n      ]\n    }\n  }\n}\n```\n\nEdit `configs\u002Fresearch_config.json` to choose `embedding_model` (`fast`,\n`balanced`, `quality`, or a custom Hugging Face ONNX repo id). The named Docker\nvolume keeps downloaded model bundles between launches.\n\n## Optional HTTP server\n\nUseful when you want HTTP instead of MCP:\n\n```bash\nuvicorn servers.fastapi_server:app --reload\n```\n\nEndpoints:\n\n- `GET \u002Fhealth`\n- `GET \u002Fweb_search?query=...`\n- `POST \u002Fsite_crawl`\n- `POST \u002Fresearch`\n\n## Configuration\n\nTune research defaults in `configs\u002Fresearch_config.json`. Set\n`TINYSEARCH_CONFIG_PATH` to load a different JSON config file, which is the\nrecommended Docker override pattern.\n\nThe `onnx` embedding backend uses local ONNX bundles under `models\u002F`. Starting\nthe MCP server or FastAPI app downloads the configured `embedding_model` once\nfrom Hugging Face when `embedding_backend` is `onnx`.\n\nBuilt-in local presets:\n\n- `fast`: `onnx-models\u002Fall-MiniLM-L6-v2-onnx`\n- `balanced`: `BAAI\u002Fbge-small-en-v1.5`\n- `quality`: `BAAI\u002Fbge-base-en-v1.5`\n\nYou can also set `embedding_model` to a custom Hugging Face ONNX repo id. Set\n`TINYSEARCH_MODELS_DIR` to move the model cache, or use\n`TINYSEARCH_ONNX_MODEL_DIR` when you need to point at one exact bundle directory.\n\nKey settings:\n\n- Search: `search_top_k`, `search_rrf_cutoff`, `search_dense_weight`, `search_max_results_to_keep`\n- Chunks: `chunk_rrf_cutoff`, `chunk_dense_weight`, `chunk_max_results_to_keep`\n- Crawl: `crawl_max_chunk_tokens`, `crawl_overlap_tokens`, `max_concurrent_crawls`\n- Embeddings: `embedding_backend`, `embedding_model`, `embedding_openai_env_file`, `max_concurrent_embedding_calls`\n- Tokenizer: `encoding_name`\n- Dense input prefixes: `dense_query_prefix`, `dense_document_prefix`\n- Trace: `trace_path`\n\nFor `embedding_backend` `openai_compatible`, add a `.env` file at the project\nroot, or set `embedding_openai_env_file`, with:\n\n```text\nOPENAI_BASE_URL=\nOPENAI_API_KEY=\nOPENAI_EMBEDDING_MODEL=\n```\n\n`OPENAI_BASE_URL` is optional for api.openai.com. `EMBEDDING_MODEL` and\n`MODEL_NAME` are accepted as aliases for `OPENAI_EMBEDDING_MODEL`.\n\nThe research pipeline requires dense embeddings. It raises if\n`search_dense_weight` or `chunk_dense_weight` is set to `0`.\n\n## When not to use TinySearch\n\nTinySearch is not a replacement for a commercial search API or a persistent\ncrawler. It is probably not the right tool if you need:\n\n- guaranteed search coverage\n- large-scale indexing\n- long-term page caching\n- enterprise observability\n- production SLA-backed web search\n\n## TinySearch vs...\n\n| Tool type | What it gives you | Tradeoff |\n| --- | --- | --- |\n| Search API | Search results | Usually hosted \u002F paid |\n| Full crawler \u002F index | Persistent search backend | More infrastructure |\n| SearxNG | Metasearch | Still needs setup and a ranking layer |\n| **TinySearch** | MCP research prompt with ranked chunks | Lightweight; not a full search engine |\n\n## Entrypoints\n\n- `pipelines.agentic_research.agentic_run`: single-turn search, crawl, ranking, and prompt assembly\n- `servers.mcp_server`: MCP server for agent clients\n- `servers.fastapi_server`: optional HTTP API\n\n## Tests\n\nRun the unittest suite:\n\n```bash\npython -m unittest discover tests\n```\n\n## Contact\n\nUsing TinySearch or want to build on it?\n\n[Email me](mailto:hello.marcbuilds@gmail.com) or reach me on [Bluesky](https:\u002F\u002Fbsky.app\u002Fprofile\u002Fmarcellm01.bsky.social).\n\n## Privacy notes\n\nTinySearch reads the pages it crawls and returns ranked excerpts to the calling\nclient. It does not include credentials in the repo, and `.env` \u002F trace output\nshould stay local. If you enable `openai_compatible` embeddings, your embedding\nprovider receives the text snippets sent for vectorization.\n\n## License\n\nSource code in this repository is under the [MIT License](LICENSE).\n\nWhen `embedding_backend` is `onnx`, TinySearch may download the selected local\nONNX embedding bundle at runtime from Hugging Face. Those weights are separate\ndistributions under their model-card licenses; keep license and attribution\nnotices if you ship or redistribute those files. Optional manual export for\n`fast` uses `sentence-transformers\u002Fall-MiniLM-L6-v2` (Apache-2.0).\n\nSee [NOTICE](NOTICE) for Docker and third-party distribution notes.\n","TinySearch 是一个专为本地语言模型设计的小型网络研究引擎。它通过搜索、重新排序、爬取网页内容并提取最相关片段来生成基于来源的提示，从而帮助本地语言模型回答问题。该项目采用Python编写，支持FastAPI接口和MCP协议，能够无缝集成到现有的本地代理系统中。用户可以通过简单的命令行操作或Docker容器快速启动服务，无需复杂的配置或账户系统。特别适合于需要在本地环境中进行网络信息检索但又不想构建完整搜索后端的应用场景，如个人工作流、小型系统开发以及原型测试等。",2,"2026-06-11 04:01:25","CREATED_QUERY"]