[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80921":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":12,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":14,"rankGlobal":8,"rankLanguage":8,"license":8,"archived":15,"fork":15,"defaultBranch":16,"hasWiki":17,"hasPages":15,"topics":18,"createdAt":8,"pushedAt":8,"updatedAt":19,"readmeContent":20,"aiSummary":21,"trendingCount":13,"starSnapshotCount":13,"syncStatus":12,"lastSyncTime":22,"discoverSource":23},80921,"pirate_llm","younissk\u002Fpirate_llm","younissk",null,"Python",35,9,2,0,3,false,"main",true,[],"2026-06-12 02:04:08","---\nlanguage: en\ntags:\n  - gpt\n  - nanogpt\n  - pirate\n  - tinystories\n  - text-generation\npipeline_tag: text-generation\nlibrary_name: pytorch\n---\n\n![nanoBeard Banner](.\u002Fbanner.png)\n\n# nanoBeard ☠️\n\nA tiny pirate-themed GPT trained from scratch on a piratized version of TinyStories,\nthen SFT-tuned. Built as a learning project — closer to nanoGPT than to a production LM.\n\nThe repo is structured for **multiple ship-class versions** under one codebase:\n\n| Codename | Status | HF repo |\n|---|---|---|\n| **Sloop** (v1) | shipped | [`younissk\u002FnanoBeard`](https:\u002F\u002Fhuggingface.co\u002Fyounissk\u002FnanoBeard) |\n| Brig (v2) | planned | `younissk\u002FnanoBeard-Brig` |\n\n- **Source code:** https:\u002F\u002Fgithub.com\u002Fyounissk\u002Fpirate_llm\n- **Docs:** see `docs\u002F` or run `make docs-serve`\n\n## Layout\n\n```\nnanobeard\u002F             # package: training, sampling, SFT, publish, eval\n  models\u002F              # one file per architecture, registered via MODEL_REGISTRY\n  eval\u002F                # perplexity + sample-gallery harness\n  dataset_pipeline\u002F    # sources (piratized corpora) + recipe-driven dataset builds\nconfigs\u002F               # one .py per model version\ndata\u002Fsources\u002F\u003Cname>\u002F   # reusable piratized corpora (cached arrow + source.json)\ndata\u002Fdatasets\u002F\u003Cname>\u002F  # composed datasets: recipe.json + tokenizer + bins + metadata\nruns\u002F\u003Cversion>\u002F        # per-version checkpoints\nevals\u002F                 # prompt set + per-eval reports\nspace\u002F                 # Gradio playground (multi-version dropdown)\ntests\u002F                 # 80+ pytest tests\ndocs\u002F                  # mkdocs site\n```\n\n## Model details (Sloop, v1)\n\n| Field | Value |\n|---|---|\n| Architecture | Decoder-only Transformer (GPT-style) |\n| Parameters | ~13.8M |\n| Layers \u002F heads \u002F embd | 6 \u002F 6 \u002F 384 |\n| Context length | 256 tokens |\n| Vocab size | 8192 (custom BPE) |\n| Bias in Linear\u002FLN | False |\n| Tokenizer | `pirate_bpe.json` (HuggingFace `tokenizers` BPE) |\n\n## Training (Sloop)\n\n- **Pretraining** on piratized TinyStories.\n- **SFT** on `TeeZee\u002Fdolly-15k-pirate-speech`.\n- AdamW, warmup + cosine decay. `bfloat16` on CUDA.\n- See `training_metadata.json` in the HF repo for the exact run config + losses.\n\n## Quick start\n\n```bash\nmake install                       # uv sync\nuv sync --dev                      # dev tooling (pytest, ruff, mypy, mkdocs)\nmake env                           # .env from example\npre-commit install                 # format\u002Flint on commit\n\nmake dataset DATASET=tiny_pirate_stories   # build -> data\u002Fdatasets\u002Ftiny_pirate_stories\u002F\nmake train   CONFIG=sloop          # local smoke\nmake train   CONFIG=sloop CONFIG_VARIANT=gpu   # GPU run\nmake sample  CONFIG=sloop PROMPT='Ahoy matey'\nmake eval    CONFIG=sloop          # perplexity + gallery -> evals\u002Fresults\u002F\u003Cdate>\u002Fsloop\u002F\nmake publish CONFIG=sloop          # push to HF model repo\n```\n\n## Tests\n\n```bash\nmake test            # fast (~1.5s)\nmake test-all        # include slow integration\n```\n\nHighlights:\n- `tests\u002Ftest_model_contract.py` — **parametrized over `MODEL_REGISTRY`**, so any\n  new architecture is automatically checked for the same invariants (causal mask,\n  weight tying, shape contracts, finite grads).\n- `tests\u002Ftest_publish.py` — HF Hub mocked; verifies `config.json` carries\n  `model_name`, `codename`, `display_name`, `num_parameters`, and all arch fields.\n- `tests\u002Ftest_tokenizer_hash.py` — tokenizer fingerprint in every ckpt.\n\n## Loading a published model\n\n`nanoBeard` is **not** a `transformers` model — load via the `nanobeard` package:\n\n```python\nimport json, torch\nfrom huggingface_hub import hf_hub_download\nfrom safetensors.torch import load_model\nfrom tokenizers import Tokenizer\n\nfrom nanobeard.config import Config\nfrom nanobeard.models import build_model\n\nrepo = \"younissk\u002FnanoBeard\"      # Sloop\ncfg_dict = json.load(open(hf_hub_download(repo, \"config.json\")))\ncfg = Config(**{k: v for k, v in cfg_dict.items() if k in Config.__dataclass_fields__})\nmodel = build_model(cfg).eval()\nload_model(model, hf_hub_download(repo, \"model.safetensors\"))\n\ntok = Tokenizer.from_file(hf_hub_download(repo, \"pirate_bpe.json\"))\nids = torch.tensor([tok.encode(\"Once upon a time\").ids])\nwith torch.no_grad():\n    for _ in range(80):\n        logits, _ = model(ids[:, -cfg.block_size:])\n        next_id = torch.multinomial(torch.softmax(logits[:, -1] \u002F 0.8, -1), 1)\n        ids = torch.cat([ids, next_id], dim=1)\nprint(tok.decode(ids[0].tolist()))\n```\n\n## Adding a new model version\n\nSee `docs\u002Fadding-a-model.md`. TL;DR:\n\n1. Write `nanobeard\u002Fmodels\u002F\u003Ckey>.py` (new arch, frozen contract).\n2. Register a `ModelSpec` in `nanobeard\u002Fmodels\u002F__init__.py`.\n3. Drop a config in `configs\u002F\u003Ckey>.py`.\n4. `make dataset DATASET=\u003Cname>` \u002F `make train CONFIG=\u003Ckey>`.\n5. `make test` — contract suite parametrizes automatically.\n6. `make publish CONFIG=\u003Ckey>` to its own HF model repo.\n\n## Limitations\n\n- Trained on a tiny synthetic corpus. Vocabulary, grammar, and world knowledge\n  are extremely narrow.\n- Short context (256 tokens).\n- No safety tuning. Pirate-flavored nonsense at best.\n- Educational artifact, not a useful chat model.\n\n## Data\n\n- Base corpus: [TinyStories](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Froneneldan\u002FTinyStories)\n  (CDLA-Sharing-1.0), transformed by `nanobeard\u002Fdataset_pipeline\u002Fpiratize.py`.\n- SFT corpus (Sloop): `TeeZee\u002Fdolly-15k-pirate-speech`.\n","nanoBeard是一个以海盗为主题的微型GPT模型，从零开始训练于经过“海盗化”处理的TinyStories数据集上，并进行了后续微调。该项目基于PyTorch构建，旨在作为一个学习项目，更接近于nanoGPT而非生产级语言模型。核心功能包括使用解码器仅Transformer架构进行文本生成，支持多种船类版本（如Sloop）的开发与测试。技术特点涵盖自定义BPE分词器、bfloat16精度训练以及详尽的自动化测试套件。适合用于教育目的、小型实验性NLP任务或对轻量级AI模型感兴趣的开发者探索。","2026-06-01 03:52:56","CREATED_QUERY"]