[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92738":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":14,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":15,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":16,"fork":16,"defaultBranch":17,"hasWiki":16,"hasPages":16,"topics":18,"createdAt":10,"pushedAt":10,"updatedAt":29,"readmeContent":30,"aiSummary":31,"trendingCount":13,"starSnapshotCount":13,"syncStatus":32,"lastSyncTime":33,"discoverSource":34},92738,"SomNLP-Corpus","goobolabs\u002FSomNLP-Corpus","goobolabs","SomNLP-Corpus is an open, high-quality Somali text corpus designed to support research and development in Natural Language Processing (NLP), Large Language Models (LLMs), and other AI applications for the Somali language.","https:\u002F\u002Fwww.goobolabs.so",null,"Rust",52,0,1,37.1,false,"main",[19,20,21,22,23,24,25,26,27,28],"corpus","dataset","language-resources","llm","low-resource-languages","nlp","rust","somali","somali-language","text-corpus","2026-07-22 04:02:07","\u003Ch1 align=\"center\">SomNLP-Corpus\u003C\u002Fh1>\n\n\u003Cp align=\"center\">\n  \u003Cb>A high-quality, scalable, reproducible Somali text corpus for NLP, LLMs, and AI research.\u003C\u002Fb>\u003Cbr>\n  \u003Ci>Qoraal Soomaali nadiif ah oo loogu talagalay cilmi-baarista AI iyo NLP-ga.\u003C\u002Fi>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Cb>Rust-first\u003C\u002Fb> data pipeline · streaming · config-driven · full provenance\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"#status\">Status\u003C\u002Fa> ·\n  \u003Ca href=\"#what-we-built\">What we built\u003C\u002Fa> ·\n  \u003Ca href=\"#corpus-results\">Corpus results\u003C\u002Fa> ·\n  \u003Ca href=\"#pipeline\">Pipeline\u003C\u002Fa> ·\n  \u003Ca href=\"#tokenizer\">Tokenizer\u003C\u002Fa> ·\n  \u003Ca href=\"#quick-start\">Quick start\u003C\u002Fa> ·\n  \u003Ca href=\"#sources\">Sources\u003C\u002Fa> ·\n  \u003Ca href=\"#docs\">Docs\u003C\u002Fa>\n\u003C\u002Fp>\n\n---\n\n## Status\n\n| Phase | Scope | Status |\n|-------|-------|--------|\n| 1 — Foundation | Workspace, shared types | ✅ Done |\n| 2 — Public datasets | Six downloaders + merge | ✅ Done |\n| 3 — Processing pipeline | Clean → LID → deep clean → near-dedup | ✅ Done |\n| 4 — Collection | Wikipedia, web scraping | 🔜 Next |\n| 5 — Release | Hugging Face packaging | Planned |\n\n**Track A is live:** download six public Somali datasets, merge, clean, verify language,\nand deduplicate into a training-ready corpus. **Track B next:** Wikipedia and targeted\nSomali web collection.\n\nSee [ROADMAP.md](ROADMAP.md) and [PLAN.md](PLAN.md).\n\n## What we built\n\n- **Six downloaders** — HPLT, CC100, mC4, OPUS, MADLAD, MT560\n- **Five processing stages** — merge + exact dedup, clean, LID (`lingua`), deep clean (v0.2), near-dedup (MinHash + LSH)\n- **`CorpusRecord` metadata** — provenance, content hash, dedup info, quality flags on every kept line\n- **Reject sidecars** — full text + reason for every dropped record; inspect with `reports\u002Finspect_drops.sh`\n- **Single config** — [`configs\u002Fpipeline.toml`](configs\u002Fpipeline.toml)\n\n```\nSomNLP ── SomNLP-Corpus (this repo) → Translate · NER · QA · Instruct · Sentiment · Bench\n```\n\n## Corpus results\n\nFull 6-source run (HPLT, CC100, mC4, OPUS, MADLAD, MT560) through the **v0.2 pipeline**\n(merge → clean → LID → deep clean → near-dedup). Document counts are from per-stage\nstats; **final word count is measured** on `data\u002Ffinal\u002Ffinal_so.jsonl`. Intermediate\nword\u002Ftoken figures use the final corpus average (~317 words\u002Fdoc) and are marked with ~.\nv0.1 baseline (without deep clean): 1.77M docs · 591M words — see\n[docs\u002FCLEANING_STRATEGY.md](docs\u002FCLEANING_STRATEGY.md).\n\n| Stage | Documents | Words | Tokens | Removed this stage |\n|-------|----------:|------:|--------------:|-------------------:|\n| Downloaded (raw) | 2,633,281 | ~835M | ~1.25B | — |\n| Merged | 2,329,800 | ~738M | ~1.11B | 303,481 |\n| Cleaned | 2,225,791 | ~706M | ~1.06B | 104,009 |\n| LID verified | 2,035,287 | ~645M | ~968M | 190,504 |\n| Deep cleaned | 2,003,228 | ~635M | ~952M | 32,059 |\n| **Final** | **1,668,080** | **528,853,952** | **~810M** | 335,148 |\n\n**Overall:** 2.63M raw rows → **1.67M clean documents** · **529M words** · **~810M subword tokens**\n(native 32k BPE, mean 1.53 tokens\u002Fword — see [tokenizer\u002F](tokenizer\u002F)). Output:\n`data\u002Ffinal\u002Ffinal_so.jsonl` (~4.0 GB).\n\n### What cleaning removed\n\n| Stage | Removed | Share of stage input | Main reason |\n|-------|--------:|---------------------:|-------------|\n| Merge | 303,481 | 11.5% | Exact duplicates (MT560 ~68% within-source) |\n| Clean | 104,009 | 4.5% | Too short (<25 words docs \u002F <5 words sentences) or corrupted |\n| LID | 190,504 | 8.6% | Non-Somali on document-class sources (mC4 highest rate) |\n| Deep clean | 32,059 | 1.6% | Boilerplate (23,948), segment LID (6,906), too long (1,060) |\n| Near dedup | 335,148 | 16.7% | Near-duplicate web documents (text changed after deep clean) |\n\n**36.7%** of raw documents did not survive the pipeline (v0.1: 32.6%). Re-run locally to\nreproduce; numbers shift slightly with upstream dataset versions.\n\n## Pipeline\n\n```text\ndownload → merge + exact dedup → clean → LID → deep clean → near dedup → final\nraw\u002F       merged\u002F              cleaned\u002F  lid\u002F   deep_clean\u002F  final\u002F\n```\n\n| Stage | Binary | Output |\n|-------|--------|--------|\n| Download | `download_*_so` | `data\u002Fraw\u002F\u003Csource>\u002F` |\n| Merge | `merge_corpora` | `data\u002Fmerged\u002Fmerged_so.jsonl` |\n| Clean | `clean_corpus` | `data\u002Fcleaned\u002Fcleaned_so.jsonl` |\n| Language ID | `lid_verify` | `data\u002Flid\u002Flid_so.jsonl` |\n| Deep clean | `deep_clean` | `data\u002Fdeep_clean\u002Fdeep_clean_so.jsonl` |\n| Near dedup | `near_dedup` | `data\u002Ffinal\u002Ffinal_so.jsonl` |\n| All stages | `run_pipeline` | chains the above |\n\n| Source class | Sources | Min words | LID | Near dedup |\n|--------------|---------|----------:|-----|------------|\n| Document | HPLT, CC100, mC4, MADLAD | 25 | `lingua` gate @ 0.50 | MinHash + LSH |\n| Sentence | OPUS, MT560 | 5 | tag-only | exact only |\n\nFull commands and drop inspection: [docs\u002FDATA_PIPELINE.md](docs\u002FDATA_PIPELINE.md) ·\nspecification: [docs\u002FCLEANING_PLAN.md](docs\u002FCLEANING_PLAN.md).\n\n## Tokenizer\n\nA corpus-native **32k BPE tokenizer** is trained on the final release corpus\n(`data\u002Ffinal\u002Ffinal_so.jsonl`). The trained model ships in-repo; the plain-text\ntraining file is regenerated locally (~3.3 GB).\n\n| Metric | Value |\n|--------|------:|\n| Vocabulary | 32,000 |\n| Mean tokens\u002Fword (native BPE) | 1.53 |\n| Median tokens\u002Fword | 1.33 |\n| vs BERT-base | 2.69 (1.75× worse) |\n| vs XLM-RoBERTa | 1.94 (1.27× worse) |\n| Est. corpus tokens | ~810M |\n\n```bash\ncd tokenizer\npython -m venv .venv && source .venv\u002Fbin\u002Factivate\npip install -r requirements.txt\n\npython prepare_corpus.py --stats   # reads data\u002Ffinal\u002Ffinal_so.jsonl\npython train.py                    # writes somali-bpe-tokenizer.json\npython test_tokenizer.py           # benchmark; use --sample-size 1668080 for full corpus\n```\n\nArtifacts: `somali-bpe-tokenizer.json` (tracked), `benchmark_results.json`,\n`tokenizer_stats.json`. Methodology and full results:\n[tokenizer\u002FPAPER.md](tokenizer\u002FPAPER.md).\n\n## Quick start\n\n**Requirements:** Rust 1.75+ · ~20 GB free disk for a full build.\n\n```bash\ncargo build --release\n```\n\n### Smoke test (~100 records)\n\n```bash\n.\u002Ftarget\u002Frelease\u002Fdownload_hplt_so --limit 100\n.\u002Ftarget\u002Frelease\u002Frun_pipeline --stages merge,clean,lid,deep_clean,near_dedup --limit 100\n```\n\n### Full corpus build\n\n```bash\n.\u002Ftarget\u002Frelease\u002Fdownload_hplt_so\n.\u002Ftarget\u002Frelease\u002Fdownload_cc100_so\n.\u002Ftarget\u002Frelease\u002Fdownload_mc4_so\n.\u002Ftarget\u002Frelease\u002Fdownload_opus_so\n.\u002Ftarget\u002Frelease\u002Fdownload_madlad_so\n.\u002Ftarget\u002Frelease\u002Fdownload_mt560_so\n.\u002Ftarget\u002Frelease\u002Fdownload_quran_so\n\n.\u002Ftarget\u002Frelease\u002Frun_pipeline --config configs\u002Fpipeline.toml\n```\n\nSome Hugging Face datasets need authentication:\n\n```bash\nexport HF_TOKEN=hf_...   # or HUGGING_FACE_HUB_TOKEN\n```\n\n### Inspect drops\n\n```bash\nbash reports\u002Finspect_drops.sh          # all stages\nbash reports\u002Finspect_drops.sh clean    # one stage\n```\n\nPer-run stats live in `reports\u002F` (gitignored). Corpus artifacts in `data\u002F` (gitignored).\n\n### Development\n\n```bash\ncargo test\ncargo clippy -- -D warnings\ncargo fmt --check\n```\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n## Sources\n\n| Tool | Dataset | License |\n|------|---------|---------|\n| `download_hplt_so` | [HPLT2.0 cleaned](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FHPLT\u002FHPLT2.0_cleaned) (`som_Latn`) | CC0-1.0 |\n| `download_cc100_so` | [CC-100 Somali](https:\u002F\u002Fdata.statmt.org\u002Fcc-100\u002Fso.txt.xz) | CC-BY-SA-4.0 |\n| `download_mc4_so` | [allenai\u002Fc4](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Fc4) (`so`) | ODC-BY |\n| `download_opus_so` | [OPUS ParaCrawl](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FHelsinki-NLP\u002Fopus_paracrawl) (`en-so`) | CC0-1.0 |\n| `download_madlad_so` | [MADLAD-400](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002FMADLAD-400) (`so`) | ODC-BY |\n| `download_mt560_so` | [MT560 en–so pairs](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fmichsethowusu\u002Fenglish-somali_sentence-pairs_mt560) | CC-BY-4.0 |\n| `download_quran_so` | [QuranEnc Somali (Yacob Yusuf)](https:\u002F\u002Fquranenc.com\u002Fapi\u002Fv1\u002Ftranslation\u002Fsura\u002Fsomali_yacob\u002F1) | see source |\n\nScale estimates, overlap, and per-record licensing: [docs\u002FSOURCES.md](docs\u002FSOURCES.md).\n\n> **Licensing:** no single corpus license — each `CorpusRecord` carries its upstream\n> `license` field. See [docs\u002FMETADATA_SCHEMA.md](docs\u002FMETADATA_SCHEMA.md).\n\n## Record format\n\n```json\n{\n  \"id\": \"hplt:a3f8c2…\",\n  \"text\": \"Soomaaliya waa dal ku yaal Geeska Afrika.\",\n  \"provenance\": { \"source\": \"hplt\", \"lang\": \"so\", \"collected_at\": \"…\" },\n  \"license\": \"CC0-1.0\",\n  \"content_hash\": \"sha256:…\",\n  \"quality\": { \"disposition\": \"kept\", \"flags\": [] },\n  \"schema_version\": 1\n}\n```\n\n## Project layout\n\n```text\nsomnlp\u002F\n├── configs\u002Fpipeline.toml       # merge order, clean\u002FLID\u002Fdedup knobs\n├── crates\u002F\n│   ├── common\u002F                 # record types, hashing, source registry\n│   ├── corpus-tools\u002F           # downloaders + merge\n│   └── corpus-pipeline\u002F        # clean, LID, deep clean, near-dedup, run_pipeline\n├── docs\u002F                       # architecture, schema, pipeline specs\n├── tokenizer\u002F                  # Somali BPE training pipeline + trained model\n├── reports\u002F                    # per-run stats (gitignored)\n└── data\u002F                       # corpus artifacts (gitignored)\n```\n\nArchitecture: [docs\u002FARCHITECTURE.md](docs\u002FARCHITECTURE.md).\n\n## Docs\n\n| Doc | Description |\n|-----|-------------|\n| [docs\u002FDATA_PIPELINE.md](docs\u002FDATA_PIPELINE.md) | Stage commands, data flow, inspecting drops |\n| [docs\u002FCLEANING_PLAN.md](docs\u002FCLEANING_PLAN.md) | Phase 3 cleaning, LID, and dedup specification |\n| [docs\u002FCLEANING_STRATEGY.md](docs\u002FCLEANING_STRATEGY.md) | v0.2 deep-clean audit and strategy |\n| [docs\u002FARCHITECTURE.md](docs\u002FARCHITECTURE.md) | Workspace layout and crate design |\n| [docs\u002FSOURCES.md](docs\u002FSOURCES.md) | Source registry and scale estimates |\n| [docs\u002FMETADATA_SCHEMA.md](docs\u002FMETADATA_SCHEMA.md) | Record metadata and licensing |\n| [PLAN.md](PLAN.md) | Vision and two-track strategy |\n| [ROADMAP.md](ROADMAP.md) | Phases and milestones |\n| [CONTRIBUTING.md](CONTRIBUTING.md) | How to contribute |\n| [tokenizer\u002FPAPER.md](tokenizer\u002FPAPER.md) | Somali BPE tokenizer methodology and benchmarks |\n| [CHANGELOG.md](CHANGELOG.md) | Project history |\n","SomNLP-Corpus 是一个面向索马里语的高质量开源文本语料库，专为自然语言处理（NLP）、大语言模型（LLM）训练与评估构建。项目采用 Rust 实现可复现的数据流水线，支持流式处理、配置驱动、全链路溯源，并集成语言识别、多级清洗（含深度清洗）和近似去重（MinHash+LSH）等核心能力，每条语料均附带来源、哈希与质量标记。适用于低资源语言 NLP 研究、索马里语模型预训练\u002F微调、跨语言迁移学习及语料质量分析等场景。",2,"2026-07-10 02:30:20","CREATED_QUERY"]