[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-559":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":16,"starSnapshotCount":16,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},559,"nanoGPT","karpathy\u002FnanoGPT","karpathy","The simplest, fastest repository for training\u002Ffinetuning medium-sized GPTs.","",null,"Python",59487,10262,506,246,0,39,283,1601,206,45,"MIT License",false,"master",true,[],"2026-06-12 02:00:15","\n# nanoGPT\n\n![nanoGPT](assets\u002Fnanogpt.jpg)\n\n\n---\n\n**Update Nov 2025** nanoGPT has a new and improved cousin called [nanochat](https:\u002F\u002Fgithub.com\u002Fkarpathy\u002Fnanochat). It is very likely you meant to use\u002Ffind nanochat instead. nanoGPT (this repo) is now very old and deprecated but I will leave it up for posterity.\n\n---\n\nThe simplest, fastest repository for training\u002Ffinetuning medium-sized GPTs. It is a rewrite of [minGPT](https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FminGPT) that prioritizes teeth over education. Still under active development, but currently the file `train.py` reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training. The code itself is plain and readable: `train.py` is a ~300-line boilerplate training loop and `model.py` a ~300-line GPT model definition, which can optionally load the GPT-2 weights from OpenAI. That's it.\n\n![repro124m](assets\u002Fgpt2_124M_loss.png)\n\nBecause the code is so simple, it is very easy to hack to your needs, train new models from scratch, or finetune pretrained checkpoints (e.g. biggest one currently available as a starting point would be the GPT-2 1.3B model from OpenAI).\n\n## install\n\n```\npip install torch numpy transformers datasets tiktoken wandb tqdm\n```\n\nDependencies:\n\n- [pytorch](https:\u002F\u002Fpytorch.org) \u003C3\n- [numpy](https:\u002F\u002Fnumpy.org\u002Finstall\u002F) \u003C3\n-  `transformers` for huggingface transformers \u003C3 (to load GPT-2 checkpoints)\n-  `datasets` for huggingface datasets \u003C3 (if you want to download + preprocess OpenWebText)\n-  `tiktoken` for OpenAI's fast BPE code \u003C3\n-  `wandb` for optional logging \u003C3\n-  `tqdm` for progress bars \u003C3\n\n## quick start\n\nIf you are not a deep learning professional and you just want to feel the magic and get your feet wet, the fastest way to get started is to train a character-level GPT on the works of Shakespeare. First, we download it as a single (1MB) file and turn it from raw text into one large stream of integers:\n\n```sh\npython data\u002Fshakespeare_char\u002Fprepare.py\n```\n\nThis creates a `train.bin` and `val.bin` in that data directory. Now it is time to train your GPT. The size of it very much depends on the computational resources of your system:\n\n**I have a GPU**. Great, we can quickly train a baby GPT with the settings provided in the [config\u002Ftrain_shakespeare_char.py](config\u002Ftrain_shakespeare_char.py) config file:\n\n```sh\npython train.py config\u002Ftrain_shakespeare_char.py\n```\n\nIf you peek inside it, you'll see that we're training a GPT with a context size of up to 256 characters, 384 feature channels, and it is a 6-layer Transformer with 6 heads in each layer. On one A100 GPU this training run takes about 3 minutes and the best validation loss is 1.4697. Based on the configuration, the model checkpoints are being written into the `--out_dir` directory `out-shakespeare-char`. So once the training finishes we can sample from the best model by pointing the sampling script at this directory:\n\n```sh\npython sample.py --out_dir=out-shakespeare-char\n```\n\nThis generates a few samples, for example:\n\n```\nANGELO:\nAnd cowards it be strawn to my bed,\nAnd thrust the gates of my threats,\nBecause he that ale away, and hang'd\nAn one with him.\n\nDUKE VINCENTIO:\nI thank your eyes against it.\n\nDUKE VINCENTIO:\nThen will answer him to save the malm:\nAnd what have you tyrannous shall do this?\n\nDUKE VINCENTIO:\nIf you have done evils of all disposition\nTo end his power, the day of thrust for a common men\nThat I leave, to fight with over-liking\nHasting in a roseman.\n```\n\nlol  `¯\\_(ツ)_\u002F¯`. Not bad for a character-level model after 3 minutes of training on a GPU. Better results are quite likely obtainable by instead finetuning a pretrained GPT-2 model on this dataset (see finetuning section later).\n\n**I only have a macbook** (or other cheap computer). No worries, we can still train a GPT but we want to dial things down a notch. I recommend getting the bleeding edge PyTorch nightly ([select it here](https:\u002F\u002Fpytorch.org\u002Fget-started\u002Flocally\u002F) when installing) as it is currently quite likely to make your code more efficient. But even without it, a simple train run could look as follows:\n\n```sh\npython train.py config\u002Ftrain_shakespeare_char.py --device=cpu --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0\n```\n\nHere, since we are running on CPU instead of GPU we must set both `--device=cpu` and also turn off PyTorch 2.0 compile with `--compile=False`. Then when we evaluate we get a bit more noisy but faster estimate (`--eval_iters=20`, down from 200), our context size is only 64 characters instead of 256, and the batch size only 12 examples per iteration, not 64. We'll also use a much smaller Transformer (4 layers, 4 heads, 128 embedding size), and decrease the number of iterations to 2000 (and correspondingly usually decay the learning rate to around max_iters with `--lr_decay_iters`). Because our network is so small we also ease down on regularization (`--dropout=0.0`). This still runs in about ~3 minutes, but gets us a loss of only 1.88 and therefore also worse samples, but it's still good fun:\n\n```sh\npython sample.py --out_dir=out-shakespeare-char --device=cpu\n```\nGenerates samples like this:\n\n```\nGLEORKEN VINGHARD III:\nWhell's the couse, the came light gacks,\nAnd the for mought you in Aut fries the not high shee\nbot thou the sought bechive in that to doth groan you,\nNo relving thee post mose the wear\n```\n\nNot bad for ~3 minutes on a CPU, for a hint of the right character gestalt. If you're willing to wait longer, feel free to tune the hyperparameters, increase the size of the network, the context length (`--block_size`), the length of training, etc.\n\nFinally, on Apple Silicon Macbooks and with a recent PyTorch version make sure to add `--device=mps` (short for \"Metal Performance Shaders\"); PyTorch then uses the on-chip GPU that can *significantly* accelerate training (2-3X) and allow you to use larger networks. See [Issue 28](https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FnanoGPT\u002Fissues\u002F28) for more.\n\n## reproducing GPT-2\n\nA more serious deep learning professional may be more interested in reproducing GPT-2 results. So here we go - we first tokenize the dataset, in this case the [OpenWebText](https:\u002F\u002Fopenwebtext2.readthedocs.io\u002Fen\u002Flatest\u002F), an open reproduction of OpenAI's (private) WebText:\n\n```sh\npython data\u002Fopenwebtext\u002Fprepare.py\n```\n\nThis downloads and tokenizes the [OpenWebText](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fopenwebtext) dataset. It will create a `train.bin` and `val.bin` which holds the GPT2 BPE token ids in one sequence, stored as raw uint16 bytes. Then we're ready to kick off training. To reproduce GPT-2 (124M) you'll want at least an 8X A100 40GB node and run:\n\n```sh\ntorchrun --standalone --nproc_per_node=8 train.py config\u002Ftrain_gpt2.py\n```\n\nThis will run for about 4 days using PyTorch Distributed Data Parallel (DDP) and go down to loss of ~2.85. Now, a GPT-2 model just evaluated on OWT gets a val loss of about 3.11, but if you finetune it it will come down to ~2.85 territory (due to an apparent domain gap), making the two models ~match.\n\nIf you're in a cluster environment and you are blessed with multiple GPU nodes you can make GPU go brrrr e.g. across 2 nodes like:\n\n```sh\n# Run on the first (master) node with example IP 123.456.123.456:\ntorchrun --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=123.456.123.456 --master_port=1234 train.py\n# Run on the worker node:\ntorchrun --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr=123.456.123.456 --master_port=1234 train.py\n```\n\nIt is a good idea to benchmark your interconnect (e.g. iperf3). In particular, if you don't have Infiniband then also prepend `NCCL_IB_DISABLE=1` to the above launches. Your multinode training will work, but most likely _crawl_. By default checkpoints are periodically written to the `--out_dir`. We can sample from the model by simply `python sample.py`.\n\nFinally, to train on a single GPU simply run the `python train.py` script. Have a look at all of its args, the script tries to be very readable, hackable and transparent. You'll most likely want to tune a number of those variables depending on your needs.\n\n## baselines\n\nOpenAI GPT-2 checkpoints allow us to get some baselines in place for openwebtext. We can get the numbers as follows:\n\n```sh\n$ python train.py config\u002Feval_gpt2.py\n$ python train.py config\u002Feval_gpt2_medium.py\n$ python train.py config\u002Feval_gpt2_large.py\n$ python train.py config\u002Feval_gpt2_xl.py\n```\n\nand observe the following losses on train and val:\n\n| model | params | train loss | val loss |\n| ------| ------ | ---------- | -------- |\n| gpt2 | 124M         | 3.11  | 3.12     |\n| gpt2-medium | 350M  | 2.85  | 2.84     |\n| gpt2-large | 774M   | 2.66  | 2.67     |\n| gpt2-xl | 1558M     | 2.56  | 2.54     |\n\nHowever, we have to note that GPT-2 was trained on (closed, never released) WebText, while OpenWebText is just a best-effort open reproduction of this dataset. This means there is a dataset domain gap. Indeed, taking the GPT-2 (124M) checkpoint and finetuning on OWT directly for a while reaches loss down to ~2.85. This then becomes the more appropriate baseline w.r.t. reproduction.\n\n## finetuning\n\nFinetuning is no different than training, we just make sure to initialize from a pretrained model and train with a smaller learning rate. For an example of how to finetune a GPT on new text go to `data\u002Fshakespeare` and run `prepare.py` to download the tiny shakespeare dataset and render it into a `train.bin` and `val.bin`, using the OpenAI BPE tokenizer from GPT-2. Unlike OpenWebText this will run in seconds. Finetuning can take very little time, e.g. on a single GPU just a few minutes. Run an example finetuning like:\n\n```sh\npython train.py config\u002Ffinetune_shakespeare.py\n```\n\nThis will load the config parameter overrides in `config\u002Ffinetune_shakespeare.py` (I didn't tune them much though). Basically, we initialize from a GPT2 checkpoint with `init_from` and train as normal, except shorter and with a small learning rate. If you're running out of memory try decreasing the model size (they are `{'gpt2', 'gpt2-medium', 'gpt2-large', 'gpt2-xl'}`) or possibly decreasing the `block_size` (context length). The best checkpoint (lowest validation loss) will be in the `out_dir` directory, e.g. in `out-shakespeare` by default, per the config file. You can then run the code in `sample.py --out_dir=out-shakespeare`:\n\n```\nTHEODORE:\nThou shalt sell me to the highest bidder: if I die,\nI sell thee to the first; if I go mad,\nI sell thee to the second; if I\nlie, I sell thee to the third; if I slay,\nI sell thee to the fourth: so buy or sell,\nI tell thee again, thou shalt not sell my\npossession.\n\nJULIET:\nAnd if thou steal, thou shalt not sell thyself.\n\nTHEODORE:\nI do not steal; I sell the stolen goods.\n\nTHEODORE:\nThou know'st not what thou sell'st; thou, a woman,\nThou art ever a victim, a thing of no worth:\nThou hast no right, no right, but to be sold.\n```\n\nWhoa there, GPT, entering some dark place over there. I didn't really tune the hyperparameters in the config too much, feel free to try!\n\n## sampling \u002F inference\n\nUse the script `sample.py` to sample either from pre-trained GPT-2 models released by OpenAI, or from a model you trained yourself. For example, here is a way to sample from the largest available `gpt2-xl` model:\n\n```sh\npython sample.py \\\n    --init_from=gpt2-xl \\\n    --start=\"What is the answer to life, the universe, and everything?\" \\\n    --num_samples=5 --max_new_tokens=100\n```\n\nIf you'd like to sample from a model you trained, use the `--out_dir` to point the code appropriately. You can also prompt the model with some text from a file, e.g. ```python sample.py --start=FILE:prompt.txt```.\n\n## efficiency notes\n\nFor simple model benchmarking and profiling, `bench.py` might be useful. It's identical to what happens in the meat of the training loop of `train.py`, but omits much of the other complexities.\n\nNote that the code by default uses [PyTorch 2.0](https:\u002F\u002Fpytorch.org\u002Fget-started\u002Fpytorch-2.0\u002F). At the time of writing (Dec 29, 2022) this makes `torch.compile()` available in the nightly release. The improvement from the one line of code is noticeable, e.g. cutting down iteration time from ~250ms \u002F iter to 135ms \u002F iter. Nice work PyTorch team!\n\n## todos\n\n- Investigate and add FSDP instead of DDP\n- Eval zero-shot perplexities on standard evals (e.g. LAMBADA? HELM? etc.)\n- Finetune the finetuning script, I think the hyperparams are not great\n- Schedule for linear batch size increase during training\n- Incorporate other embeddings (rotary, alibi)\n- Separate out the optim buffers from model params in checkpoints I think\n- Additional logging around network health (e.g. gradient clip events, magnitudes)\n- Few more investigations around better init etc.\n\n## troubleshooting\n\nNote that by default this repo uses PyTorch 2.0 (i.e. `torch.compile`). This is fairly new and experimental, and not yet available on all platforms (e.g. Windows). If you're running into related error messages try to disable this by adding `--compile=False` flag. This will slow down the code but at least it will run.\n\nFor some context on this repository, GPT, and language modeling it might be helpful to watch my [Zero To Hero series](https:\u002F\u002Fkarpathy.ai\u002Fzero-to-hero.html). Specifically, the [GPT video](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=kCc8FmEb1nY) is popular if you have some prior language modeling context.\n\nFor more questions\u002Fdiscussions feel free to stop by **#nanoGPT** on Discord:\n\n[![](https:\u002F\u002Fdcbadge.vercel.app\u002Fapi\u002Fserver\u002F3zy8kqD9Cp?compact=true&style=flat)](https:\u002F\u002Fdiscord.gg\u002F3zy8kqD9Cp)\n\n## acknowledgements\n\nAll nanoGPT experiments are powered by GPUs on [Lambda labs](https:\u002F\u002Flambdalabs.com), my favorite Cloud GPU provider. Thank you Lambda labs for sponsoring nanoGPT!\n","nanoGPT 是一个用于训练和微调中等规模 GPT 模型的简洁快速仓库。项目的核心功能包括通过简单的代码实现 GPT-2（1.24 亿参数）在 OpenWebText 数据集上的复现，支持从零开始训练新模型或对预训练模型进行微调。其技术特点在于代码结构清晰易读，`train.py` 和 `model.py` 分别仅约300行，便于用户根据需求进行修改。适用于需要快速上手GPT模型训练与微调的研究者、开发者以及对深度学习感兴趣的初学者，在有限计算资源下也能轻松开展实验。",2,"2026-06-11 02:37:29","top_all"]