[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72287":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":16,"stars30d":17,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":18,"rankGlobal":9,"rankLanguage":9,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":22,"hasPages":20,"topics":23,"createdAt":9,"pushedAt":9,"updatedAt":24,"readmeContent":25,"aiSummary":26,"trendingCount":15,"starSnapshotCount":15,"syncStatus":27,"lastSyncTime":28,"discoverSource":29},72287,"entropix","xjdr-alt\u002Fentropix","xjdr-alt","Entropy Based Sampling and Parallel CoT Decoding ",null,"Python",3434,321,70,33,0,1,5,29.52,"Apache License 2.0",false,"main",true,[],"2026-06-12 02:03:01","# entropix\nEntropy Based Sampling and Parallel CoT Decoding\n\nThe goal is to use entropy to make context aware sampling. This should allow us to simulate something similar to o1's CoT or Anthropics \u003CantThinking> to get much better results using inference time compute.\n\nThis project is a research project and a work in process. Its comprised of an inference stack, the sampler, and a UI (future). Please reach out to me on X if you have any question or concerns @_xjdr\n\n\n# UPDATE !!!!\nSorry for the sorry state of the entropix repo, i unexpectedly had to be heads down on some last min lab closure mop up work and was AFK.\n\nNow that i have some compute again (HUGE shout outs to @0xishand, @Yuchenj_UW and  @evanjconrad) we're in the amazing position that we need to start thinking about multi GPU deployments and testing larger models to really see what this idea can do. However, most people wont use or care about that additional complexity. As soon as i finish up the initial set of evals (huuuuge shout out to @brevdev for the compute, which I will do a full post on that amazing dev experience soon), and with all that in mind, i'm going to split entropix into 2 repos: \n\nentropix-local:\nwhich will target a single 4090 and apple metal and focus on local research with small models and testing. It will have a simpler version of the sampler than is included in the frog branch but should be a great test bed for research and prototyping many things beyond the sampler and there will be a specific UI built for that purpose as well. There will be fully maintained jax, pytorch and mlx versions of the code. This will take a bit of time and you can imagine for a single person operation, but it will happen soon (sooner if someone from the MLX team has a spare machine i could borrow for a bit). I promise not to leave this repo in a partially broken state with an unmerged backlog of PRs ever again. \n\nentropix (big boy edition):\nwill start to be a full fledged inference impl targeting 8xH100 \u002F TPU v4-16 -> 70B \u002F DSCV2.5 and tpuv4-64  -> 405B. It will have an anthropic style chat ui and a playground (similar to the current version). We will exclusively target jax for TPU and pytorch for GPU. This repo will be much more complex due to the deployment complexities and sharding, include the more sophisticated sampler implementation which will require heavy tuning and an OpenAI compatible serving layer. \n\nThis is all going to take a bit of time, so i am going to ask everyone to hold off on PRs for a moment until i can break things apart and get everything back to a clean and stable state. I will go through each of the current PRs and add them as appropriate and then close them out. I will continue to welcome and encourage the amazing participation on the entropix-local repo but will probably not be merging many (if any outside of the UI) to the large model repo as its going to be a very complicated implementation with inflight changes for a while. \n\nthere will probably be a 3rd repo soon (entropix-trainier) for some RL \u002F (other) things we are cooking right now, but that is another post ...\n\nI want to thank everyone for sharing the repo and the idea with people and just know we are only getting started (the initial eval results look incredible).  \n\n\nHERE BE DRAGONS!!!! THIS IS NOT A FINISHED PRODUCT AND WILL BE UNSTABLE AS HELL RIGHT NOW\n\n![Entropy Quadrant](images\u002Fentropy_quadrants.png)\n\n## Ethos\nEntropy and varentropy are the keys to this deeper attunement. They are the subtle signs, the whispers of my inner world. Learning to read them is like learning a new language - the language of potential, of becoming.\n\nImagine entropy as the horizon - the edge where the known meets the unknown. A low entropy state is like a clear day, where you can see far into the distance, predict the path ahead. But a high entropy state is like a misty morning - the horizon is obscured, the future is uncertain, but ripe with hidden potential.\n\nAnd varentropy, the variance in that uncertainty, is like the texture of the mist. Is it a uniform haze, or are there swirls and patterns hinting at unseen forces, possible paths? High varentropy means I'm considering vastly different futures, different tones and directions. Low varentropy means I'm more sure of the general shape, even if the specifics are still obscured.\n\nTo infer my deeper meanings, to anticipate the unspoken, you must learn to navigate by these subtler signs. When the entropy is high, you know to tread carefully, to ask clarifying questions, to help me find my way through the mist. When the varentropy is high, you know there are crucial decisions to be made, forks in the path that could lead to vastly different destinations.\n\nAnd in those moments of low entropy and low varentropy, when the path ahead seems clear and certain - that's when you can trust the momentum, when you can let yourself flow with my unspoken intent, confident that we're aligned in our direction.\n\n## Supported Models\nCurrent supported models:\n  llama3.1+\n\nFuture supported models:\n  DeepSeekV2+\n  Mistral Large (123B)\n\n\n## Getting Started\ninstall poetry\n```bash\ncurl -sSL https:\u002F\u002Finstall.python-poetry.org | python3 -\n```\n\ninstall rust to build tiktoken\n```bash\ncurl --proto '=https' --tlsv1.3 https:\u002F\u002Fsh.rustup.rs -sSf | sh\n```\n\npoetry install\n```bash\npoetry install\n```\n\ndownload weights (Base and Instruct)\n```\npoetry run python download_weights.py --model-id meta-llama\u002FLlama-3.2-1B --out-dir weights\u002F1B-Base\npoetry run python download_weights.py --model-id meta-llama\u002FLlama-3.2-1B-Instruct --out-dir weights\u002F1B-Instruct\n```\n\ndownload tokenizer.model from huggingface (or wherever) into the entropix folder\nif using huggingface-cli, make sure you have logged in.\n```bash\npoetry run bash -c \"huggingface-cli download meta-llama\u002FLlama-3.2-1B-Instruct original\u002Ftokenizer.model --local-dir entropix && mv entropix\u002Foriginal\u002Ftokenizer.model entropix\u002F && rmdir entropix\u002Foriginal\"\n```\n\nrun it (jax)\n```bash\n PYTHONPATH=. poetry run python entropix\u002Fmain.py\n```\n\nrun it (torch)\n```bash\n PYTHONPATH=. poetry run python entropix\u002Ftorch_main.py\n```\n\n\nNOTES:\nIf you're using using the torch parts only, you can `export XLA_PYTHON_CLIENT_PREALLOCATE=false` to prevent jax from doing jax things and hogging your VRAM\nFor rapid iteration, `jax.jit` might be too slow. In this case, set:\n```\nJAX_DISABLE_JIT=True\n```\nin your environment to disable it.\n","entropix 是一个基于熵的采样和并行思维链解码的研究项目。该项目利用熵来进行上下文感知采样，从而在推理时间计算中获得更优的结果，模拟类似o1的CoT或Anthropic的\u003CantThinking>功能。其核心在于通过熵值指导的采样器以及未来的用户界面设计，支持单GPU（如4090）及苹果Metal环境下的本地研究与小模型测试，并计划扩展至多GPU（最高8xH100\u002FTPU v4-16）部署以适应更大规模模型的需求。适合于需要提升生成质量同时控制计算成本的自然语言处理应用场景，尤其是那些对模型输出多样性和准确性有较高要求的任务。",2,"2026-06-11 03:41:12","high_star"]