[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-10734":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":21,"defaultBranch":22,"hasWiki":20,"hasPages":20,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":40,"readmeContent":41,"aiSummary":42,"trendingCount":16,"starSnapshotCount":16,"syncStatus":43,"lastSyncTime":44,"discoverSource":45},10734,"intel-extension-for-transformers","intel\u002Fintel-extension-for-transformers","intel","⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡","",null,"Python",2178,217,4,31,0,1,29.02,"Apache License 2.0",true,false,"main",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"4-bits","autoround","chatbot","chatpdf","gaudi3","habana","intel-optimized-llamacpp","large-language-model","llm-cpu","llm-inference","neural-chat","neural-chat-7b","rag","retrieval","speculative-decoding","streamingllm","2026-06-12 02:02:25","\u003Cdiv align=\"center\">\n  \nIntel® Extension for Transformers\n===========================\n\u003Ch3>An Innovative Transformer-based Toolkit to Accelerate GenAI\u002FLLM Everywhere\u003C\u002Fh3>\n\n[![](https:\u002F\u002Fdcbadge.vercel.app\u002Fapi\u002Fserver\u002FWxk3J3ZJkU?compact=true&style=flat-square)](https:\u002F\u002Fdiscord.gg\u002FWxk3J3ZJkU)\n[![Release Notes](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fv\u002Frelease\u002Fintel\u002Fintel-extension-for-transformers)](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Freleases)\n\n[🏭Architecture](.\u002Fdocs\u002Farchitecture.md)   |   [💬NeuralChat](.\u002Fintel_extension_for_transformers\u002Fneural_chat)   |   [😃Inference on CPU](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain)   |   [😃Inference  on GPU](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fdocs\u002Fweightonlyquant.md#examples-for-gpu)   |   [💻Examples](.\u002Fdocs\u002Fexamples.md)   |   [📖Documentations](https:\u002F\u002Fintel.github.io\u002Fintel-extension-for-transformers\u002Flatest\u002Fdocs\u002FWelcome.html)\n\u003C\u002Fdiv>\n\n## 🚀Latest News\n* [2024\u002F06] Support Qwen2, please find the details in [Blog](https:\u002F\u002Fmedium.com\u002Fintel-analytics-software\u002Faccelerating-qwen2-models-with-intel-extension-for-transformers-99403de82f68)\n* [2024\u002F04] Support the launch of **[Meta Llama 3](https:\u002F\u002Fllama.meta.com\u002Fllama3\u002F)**, the next generation of Llama models. Check out [Accelerate Meta* Llama 3 with Intel AI Solutions](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fdeveloper\u002Farticles\u002Ftechnical\u002Faccelerate-meta-llama3-with-intel-ai-solutions.html).\n* [2024\u002F04] Demonstrated the chatbot in 4th, 5th, and 6th Gen Xeon Scalable Processors in [**Intel Vision Pat's Keynote**](https:\u002F\u002Fyoutu.be\u002FQB7FoIpx8os?t=2280).\n* [2024\u002F04] Supported **INT4 inference on Intel Meteor Lake**.\n* [2024\u002F04] Achieved a 1.8x performance improvement in GPT-J inference on the 5th Gen Xeon MLPerf v4.0 submission compared to v3.1. [News](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fnewsroom\u002Fnews\u002Fnew-gaudi-2-xeon-performance-ai-inference.html#gs.71ti1m), [Results](https:\u002F\u002Fmlcommons.org\u002F2024\u002F03\u002Fmlperf-inference-v4\u002F).\n* [2024\u002F01] Supported **INT4 inference on Intel GPUs** including Intel Data Center GPU Max Series (e.g., PVC) and Intel Arc A-Series (e.g., ARC). Check out the [examples](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fdocs\u002Fweightonlyquant.md#examples-for-gpu) and [scripts](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fexamples\u002Fhuggingface\u002Fpytorch\u002Ftext-generation\u002Fquantization\u002Frun_generation_gpu_woq.py).\n* [2024\u002F01] Demonstrated **Intel Hybrid Copilot** in **CES 2024 Great Minds** Session \"[Bringing the Limitless Potential of AI Everywhere](https:\u002F\u002Fyoutu.be\u002F70J3uO3eLZA?t=1348)\".\n* [2023\u002F12] Supported **QLoRA on CPUs** to make fine-tuning on client CPU possible. Check out the [blog](https:\u002F\u002Fmedium.com\u002F@NeuralCompressor\u002Fcreating-your-own-llms-on-your-laptop-a08cc4f7c91b) and [readme](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fdocs\u002Fqloracpu.md) for more details.\n* [2023\u002F11] Released **top-1 7B-sized LLM** [**NeuralChat-v3-1**](https:\u002F\u002Fhuggingface.co\u002FIntel\u002Fneural-chat-7b-v3-1) and [DPO dataset](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FIntel\u002Forca_dpo_pairs). Check out the [nice video](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=bWhZ1u_1rlc) published by [WorldofAI](https:\u002F\u002Fwww.youtube.com\u002F@intheworldofai).\n* [2023\u002F11] Published a **4-bit chatbot demo** (based on NeuralChat) available on [Intel Hugging Face Space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FIntel\u002FNeuralChat-ICX-INT4). Welcome to have a try! To setup the demo locally, please follow the [instructions](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Fnotebooks\u002Fsetup_text_chatbot_service_on_spr.ipynb).\n\n---\n\u003Cdiv align=\"left\">\n\n## 🏃Installation\n### Quick Install from Pypi\n```bash\npip install intel-extension-for-transformers\n```\n> For system requirements and other installation tips, please refer to [Installation Guide](.\u002Fdocs\u002Finstallation.md)\n\n## 🌟Introduction\nIntel® Extension for Transformers is an innovative toolkit designed to accelerate GenAI\u002FLLM everywhere with the optimal performance of Transformer-based models on various Intel platforms, including Intel Gaudi2, Intel CPU, and Intel GPU. The toolkit provides the below key features and examples:\n\n*  Seamless user experience of model compressions on Transformer-based models by extending [Hugging Face transformers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers) APIs and leveraging [Intel® Neural Compressor](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-compressor)\n\n*  Advanced software optimizations and unique compression-aware runtime (released with NeurIPS 2022's paper [Fast Distilbert on CPUs](https:\u002F\u002Farxiv.org\u002Fabs\u002F2211.07715) and [QuaLA-MiniLM: a Quantized Length Adaptive MiniLM](https:\u002F\u002Farxiv.org\u002Fabs\u002F2210.17114), and NeurIPS 2021's paper [Prune Once for All: Sparse Pre-Trained Language Models](https:\u002F\u002Farxiv.org\u002Fabs\u002F2111.05754))\n\n*  Optimized Transformer-based model packages such as [Stable Diffusion](examples\u002Fhuggingface\u002Fpytorch\u002Ftext-to-image\u002Fdeployment\u002Fstable_diffusion), [GPT-J-6B](examples\u002Fhuggingface\u002Fpytorch\u002Ftext-generation\u002Fdeployment), [GPT-NEOX](examples\u002Fhuggingface\u002Fpytorch\u002Flanguage-modeling\u002Fquantization#2-validated-model-list), [BLOOM-176B](examples\u002Fhuggingface\u002Fpytorch\u002Flanguage-modeling\u002Finference#BLOOM-176B), [T5](examples\u002Fhuggingface\u002Fpytorch\u002Fsummarization\u002Fquantization#2-validated-model-list), [Flan-T5](examples\u002Fhuggingface\u002Fpytorch\u002Fsummarization\u002Fquantization#2-validated-model-list), and end-to-end workflows such as [SetFit-based text classification](docs\u002Ftutorials\u002Fpytorch\u002Ftext-classification\u002FSetFit_model_compression_AGNews.ipynb) and [document level sentiment analysis (DLSA)](workflows\u002Fdlsa) \n\n*  [NeuralChat](intel_extension_for_transformers\u002Fneural_chat), a customizable chatbot framework to create your own chatbot within minutes by leveraging a rich set of [plugins](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Fadvanced_features.md) such as [Knowledge Retrieval](.\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fpipeline\u002Fplugins\u002Fretrieval\u002FREADME.md), [Speech Interaction](.\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fpipeline\u002Fplugins\u002Faudio\u002FREADME.md), [Query Caching](.\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fpipeline\u002Fplugins\u002Fcaching\u002FREADME.md), and [Security Guardrail](.\u002Fintel_extension_for_transformers\u002Fneural_chat\u002Fpipeline\u002Fplugins\u002Fsecurity\u002FREADME.md). This framework supports Intel Gaudi2\u002FCPU\u002FGPU.\n\n*  [Inference](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain) of Large Language Model (LLM) in pure C\u002FC++ with weight-only quantization kernels for Intel CPU and Intel GPU (TBD), supporting [GPT-NEOX](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fgptneox), [LLAMA](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fllama), [MPT](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fmpt), [FALCON](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Ffalcon), [BLOOM-7B](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fbloom), [OPT](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fopt), [ChatGLM2-6B](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fchatglm), [GPT-J-6B](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fgptj), and [Dolly-v2-3B](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fmodels\u002Fgptneox). Support AMX, VNNI, AVX512F and AVX2 instruction set. We've boosted the performance of Intel CPUs, with a particular focus on the 4th generation Intel Xeon Scalable processor, codenamed [Sapphire Rapids](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fproducts\u002Fdocs\u002Fprocessors\u002Fxeon-accelerated\u002F4th-gen-xeon-scalable-processors.html).\n\n## 🔓Validated Hardware\n\u003Ctable>\n\t\u003Ctbody>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd rowspan=\"2\">Hardware\u003C\u002Ftd>\n\t\t\t\u003Ctd colspan=\"2\">Fine-Tuning\u003C\u002Ftd>\n\t\t\t\u003Ctd colspan=\"2\">Inference\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Full\u003C\u002Ftd>\n\t\t\t\u003Ctd>PEFT\u003C\u002Ftd>\n\t\t\t\u003Ctd>8-bit\u003C\u002Ftd>\n\t\t\t\u003Ctd>4-bit\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Gaudi2\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>WIP (FP8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>-\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Xeon Scalable Processors\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT8, FP8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT4, FP4, NF4)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Xeon CPU Max Series\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT8, FP8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT4, FP4, NF4)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Data Center GPU Max Series\u003C\u002Ftd>\n\t\t\t\u003Ctd>WIP \u003C\u002Ftd>\n\t\t\t\u003Ctd>WIP \u003C\u002Ftd>\n\t\t\t\u003Ctd>WIP (INT8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT4)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Arc A-Series\u003C\u002Ftd>\n\t\t\t\u003Ctd>-\u003C\u002Ftd>\n\t\t\t\u003Ctd>-\u003C\u002Ftd>\n\t\t\t\u003Ctd>WIP (INT8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT4)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel Core Processors\u003C\u002Ftd>\n\t\t\t\u003Ctd>-\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT8, FP8)\u003C\u002Ftd>\n\t\t\t\u003Ctd>✔ (INT4, FP4, NF4)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n\n> In the table above, \"-\" means not applicable or not started yet.\n\n## 🔓Validated Software\n\u003Ctable>\n\t\u003Ctbody>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd rowspan=\"2\">Software\u003C\u002Ftd>\n\t\t\t\u003Ctd colspan=\"2\">Fine-Tuning\u003C\u002Ftd>\n\t\t\t\u003Ctd colspan=\"2\">Inference\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Full\u003C\u002Ftd>\n\t\t\t\u003Ctd>PEFT\u003C\u002Ftd>\n\t\t\t\u003Ctd>8-bit\u003C\u002Ftd>\n\t\t\t\u003Ctd>4-bit\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>PyTorch\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.0.1+cpu,\u003C\u002Fbr> 2.0.1a0 (gpu)\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.0.1+cpu,\u003C\u002Fbr> 2.0.1a0 (gpu)\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.1a0 (gpu)\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.1a0 (gpu)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Intel® Extension for PyTorch\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.110+xpu\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.110+xpu\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.110+xpu\u003C\u002Ftd>\n\t\t\t\u003Ctd>2.1.0+cpu,\u003C\u002Fbr> 2.0.110+xpu\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Transformers\u003C\u002Ftd>\n\t\t\t\u003Ctd>4.35.2(CPU),\u003C\u002Fbr> 4.31.0 (Intel GPU)\u003C\u002Ftd>\n\t\t\t\u003Ctd>4.35.2(CPU),\u003C\u002Fbr> 4.31.0 (Intel GPU)\u003C\u002Ftd>\n\t\t\t\u003Ctd>4.35.2(CPU),\u003C\u002Fbr> 4.31.0 (Intel GPU)\u003C\u002Ftd>\n\t\t\t\u003Ctd>4.35.2(CPU),\u003C\u002Fbr> 4.31.0 (Intel GPU)\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Synapse AI\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n\t\t\u003Ctr>\n\t\t\t\u003Ctd>Gaudi2 driver\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0-ee32e42\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0-ee32e42\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0-ee32e42\u003C\u002Ftd>\n\t\t\t\u003Ctd>1.13.0-ee32e42\u003C\u002Ftd>\n\t\t\u003C\u002Ftr>\n                \u003Ctr>\n                        \u003Ctd>intel-level-zero-gpu\u003C\u002Ftd>\n                        \u003Ctd>1.3.26918.50-736~22.04 \u003C\u002Ftd>\n                        \u003Ctd>1.3.26918.50-736~22.04 \u003C\u002Ftd>\n                        \u003Ctd>1.3.26918.50-736~22.04 \u003C\u002Ftd>\n                        \u003Ctd>1.3.26918.50-736~22.04 \u003C\u002Ftd>\n                \u003C\u002Ftr>\n\t\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n> Please refer to the detailed requirements in [CPU](intel_extension_for_transformers\u002Fneural_chat\u002Frequirements_cpu.txt), [Gaudi2](intel_extension_for_transformers\u002Fneural_chat\u002Frequirements_hpu.txt), [Intel GPU](intel_extension_for_transformers\u002Fneural_chat\u002Frequirements_xpu.txt).\n\n## 🔓Validated OS\nUbuntu 20.04\u002F22.04, Centos 8.\n\n## 🌱Getting Started\n\n### Chatbot\nBelow is the sample code to create your chatbot. See more [examples](intel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Ffull_notebooks.md).\n\n#### Serving (OpenAI-compatible RESTful APIs)\nNeuralChat provides OpenAI-compatible RESTful APIs for chat, so you can use NeuralChat as a drop-in replacement for OpenAI APIs.\nYou can start NeuralChat server either using the Shell command or Python code.\n\n```shell\n# Shell Command\nneuralchat_server start --config_file .\u002Fserver\u002Fconfig\u002Fneuralchat.yaml\n```\n\n```python\n# Python Code\nfrom intel_extension_for_transformers.neural_chat import NeuralChatServerExecutor\nserver_executor = NeuralChatServerExecutor()\nserver_executor(config_file=\".\u002Fserver\u002Fconfig\u002Fneuralchat.yaml\", log_file=\".\u002Fneuralchat.log\")\n```\n\nNeuralChat service can be accessible through [OpenAI client library](https:\u002F\u002Fgithub.com\u002Fopenai\u002Fopenai-python), `curl` commands, and `requests` library. See more in [NeuralChat](intel_extension_for_transformers\u002Fneural_chat\u002FREADME.md).\n\n#### Offline\n\n```python\nfrom intel_extension_for_transformers.neural_chat import build_chatbot\nchatbot = build_chatbot()\nresponse = chatbot.predict(\"Tell me about Intel Xeon Scalable Processors.\")\n```\n\n### Transformers-based extension APIs\nBelow is the sample code to use the extended Transformers APIs. See more [examples](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain).\n\n#### INT4 Inference (CPU)\nWe encourage you to install [NeuralSpeed](https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed) to get the latest features (e.g., GGUF support) of LLM low-bit inference on CPUs. You may also want to use v1.3 without NeuralSpeed by following the [document](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Ftree\u002Fv1.3\u002Fintel_extension_for_transformers\u002Fllm\u002Fruntime\u002Fgraph\u002FREADME.md)\n\n```python\nfrom transformers import AutoTokenizer\nfrom intel_extension_for_transformers.transformers import AutoModelForCausalLM\nmodel_name = \"Intel\u002Fneural-chat-7b-v3-1\"     \nprompt = \"Once upon a time, there existed a little girl,\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\ninputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)\noutputs = model.generate(inputs)\n```\nYou can also load GGUF format model from Huggingface, we only support Q4_0\u002FQ5_0\u002FQ8_0 gguf format for now.\n```python\nfrom transformers import AutoTokenizer\nfrom intel_extension_for_transformers.transformers import AutoModelForCausalLM\n\n# Specify the GGUF repo on the Hugginface\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GGUF\"\n# Download the the specific gguf model file from the above repo\ngguf_file = \"llama-2-7b-chat.Q4_0.gguf\"\n# make sure you are granted to access this model on the Huggingface.\ntokenizer_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\nprompt = \"Once upon a time, there existed a little girl,\"\ntokenizer = AutoTokenizer.from_pretrained(tokenizer_name, trust_remote_code=True)\ninputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, gguf_file = gguf_file)\noutputs = model.generate(inputs)\n```\n\n\nYou can also load PyTorch Model from Modelscope\n>**Note**:require modelscope\n```python\nfrom transformers import TextStreamer\nfrom modelscope import AutoTokenizer\nfrom intel_extension_for_transformers.transformers import AutoModelForCausalLM\nmodel_name = \"qwen\u002FQwen-7B\"     # Modelscope model_id or local model\nprompt = \"Once upon a time, there existed a little girl,\"\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True, model_hub=\"modelscope\")\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\ninputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\nstreamer = TextStreamer(tokenizer)\noutputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)\n```\n\nYou can also load the low-bit model quantized by GPTQ\u002FAWQ\u002FRTN\u002FAutoRound algorithm.\n```python\nfrom transformers import AutoTokenizer\nfrom intel_extension_for_transformers.transformers import AutoModelForCausalLM, GPTQConfig\n\n# Hugging Face GPTQ\u002FAWQ model or use local quantize model\nmodel_name = \"MODEL_NAME_OR_PATH\"\nprompt = \"Once upon a time, a little girl\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\ninputs = tokenizer(prompt, return_tensors=\"pt\").input_ids\nmodel = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)\noutputs = model.generate(inputs)\n```\n\n#### INT4 Inference (GPU)\n```python\nimport intel_extension_for_pytorch as ipex\nfrom intel_extension_for_transformers.transformers.modeling import AutoModelForCausalLM\nfrom transformers import AutoTokenizer\nimport torch\n\ndevice_map = \"xpu\"\nmodel_name =\"Qwen\u002FQwen-7B\"\ntokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\nprompt = \"Once upon a time, there existed a little girl,\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").input_ids.to(device_map)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True,\n                                              device_map=device_map, load_in_4bit=True)\n\nmodel = ipex.optimize_transformers(model, inplace=True, dtype=torch.float16, quantization_config=True, device=device_map)\n\noutput = model.generate(inputs)\n```\n> Note: Please refer to the [example](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fdocs\u002Fweightonlyquant.md#examples-for-gpu) and [script](https:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fblob\u002Fmain\u002Fexamples\u002Fhuggingface\u002Fpytorch\u002Ftext-generation\u002Fquantization\u002Frun_generation_gpu_woq.py) for more details.\n\n### Langchain-based extension APIs\nBelow is the sample code to use the extended Langchain APIs. See more [examples](intel_extension_for_transformers\u002Fneural_chat\u002Fpipeline\u002Fplugins\u002Fretrieval\u002FREADME.md).\n\n```python\nfrom langchain_community.llms.huggingface_pipeline import HuggingFacePipeline\nfrom langchain.chains import RetrievalQA\nfrom langchain_core.vectorstores import VectorStoreRetriever\nfrom intel_extension_for_transformers.langchain.vectorstores import Chroma\nretriever = VectorStoreRetriever(vectorstore=Chroma(...))\nretrievalQA = RetrievalQA.from_llm(llm=HuggingFacePipeline(...), retriever=retriever)\n```\n\n## 🎯Validated  Models\nYou can access the validated models, accuracy and performance from [Release data](.\u002Fdocs\u002Frelease_data.md) or [Medium blog](https:\u002F\u002Fmedium.com\u002F@NeuralCompressor\u002Fllm-performance-of-intel-extension-for-transformers-f7d061556176).\n\n## 📖Documentation\n\u003Ctable>\n\u003Cthead>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">OVERVIEW\u003C\u002Fth>\n  \u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n  \u003Ctr>\n    \u003Ctd colspan=\"4\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\">NeuralChat\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"4\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\">Neural Speed\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">NEURALCHAT\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Fnotebooks\u002Fdeploy_chatbot_on_spr.ipynb\">Chatbot on Intel CPU\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Fnotebooks\u002Fdeploy_chatbot_on_xpu.ipynb\">Chatbot on Intel GPU\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Fnotebooks\u002Fdeploy_chatbot_on_habana_gaudi.ipynb\">Chatbot on Gaudi\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd colspan=\"4\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\u002Fexamples\u002Fdeployment\u002Ftalkingbot\u002Fpc\u002Fbuild_talkingbot_on_pc.ipynb\">Chatbot on Client\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"4\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Fneural_chat\u002Fdocs\u002Ffull_notebooks.md\">More Notebooks\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">NEURAL SPEED\u003C\u002Fth>\n  \u003C\u002Ftr>\n \u003Ctr>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002FREADME.md\">Neural Speed\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002FREADME.md#2-neural-speed-straight-forward\">Streaming LLM\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fneural_speed\u002Fcore#support-matrix\">Low Precision Kernels\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Ftree\u002Fmain\u002Fdocs\u002Ftensor_parallelism.md\">Tensor Parallelism\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">LLM COMPRESSION\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Fsmoothquant.md\">SmoothQuant (INT8)\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" align=\"center\">\u003Ca href=\"docs\u002Fweightonlyquant.md\">Weight-only Quantization (INT4\u002FFP4\u002FNF4\u002FINT8)\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"3\" align=\"center\">\u003Ca href=\"docs\u002Fqloracpu.md\">QLoRA on CPU\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">GENERAL COMPRESSION\u003C\u002Fth>\n  \u003Ctr>\n  \u003Ctr>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Fquantization.md\">Quantization\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Fpruning.md\">Pruning\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Fdistillation.md\">Distillation\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"examples\u002Fhuggingface\u002Fpytorch\u002Ftext-classification\u002Forchestrate_optimizations\u002FREADME.md\">Orchestration\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"docs\u002Fdata_augmentation.md\">Data Augmentation\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"docs\u002Fexport.md\">Export\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"docs\u002Fmetrics.md\">Metrics\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"docs\u002Fobjectives.md\">Objectives\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd align=\"center\" colspan=\"2\">\u003Ca href=\"docs\u002Fpipeline.md\">Pipeline\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"3\">\u003Ca href=\"examples\u002Fhuggingface\u002Fpytorch\u002Fquestion-answering\u002Fdynamic\u002FREADME.md\">Length Adaptive\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd align=\"center\" colspan=\"3\">\u003Ca href=\"docs\u002Fexamples.md#early-exit\">Early Exit\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Cth colspan=\"8\" align=\"center\">TUTORIALS & RESULTS\u003C\u002Fa>\u003C\u002Fth>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Ftutorials\u002FREADME.md\">Tutorials\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fintel\u002Fneural-speed\u002Fblob\u002Fmain\u002Fdocs\u002Fsupported_models.md\">LLM List\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"docs\u002Fexamples.md\">General Model List\u003C\u002Fa>\u003C\u002Ftd>\n    \u003Ctd colspan=\"2\" align=\"center\">\u003Ca href=\"intel_extension_for_transformers\u002Ftransformers\u002Fruntime\u002Fdocs\u002Fvalidated_model.md\">Model Performance\u003C\u002Fa>\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\n## 🙌Demo\n\n* LLM Infinite Inference (up to 4M tokens)\n\nhttps:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fassets\u002F109187816\u002F1698dcda-c9ec-4f44-b159-f4e9d67ab15b\n\n* LLM QLoRA on Client CPU\n\nhttps:\u002F\u002Fgithub.com\u002Fintel\u002Fintel-extension-for-transformers\u002Fassets\u002F88082706\u002F9d9bdb7e-65db-47bb-bbed-d23b151e8b31\n\n## 📃Selected Publications\u002FEvents\n* Blog published on Huggingface: [Building Cost-Efficient Enterprise RAG applications with Intel Gaudi 2 and Intel Xeon](https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fcost-efficient-rag-applications-with-intel) (May 2024)\n* Blog published on Intel Developer News: [Efficient Natural Language Embedding Models with Intel® Extension for Transformers](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fdeveloper\u002Farticles\u002Ftechnical\u002Fefficient-natural-language-embedding-models.html) (May 2024)\n* Blog published on Techcrunch: [Intel and others commit to building open generative AI tools for the enterprise](https:\u002F\u002Ftechcrunch.com\u002F2024\u002F04\u002F16\u002Fintel-and-others-commit-to-building-open-generative-ai-tools-for-the-enterprise) (Apr 2024)\n* Video on YouTube: [Intel Vision Keynotes 2024](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=QB7FoIpx8os&t=2280s) (Apr 2024)\n* Blog published on Vectara: [Do Smaller Models Hallucinate More?](https:\u002F\u002Fvectara.com\u002Fblog\u002Fdo-smaller-models-hallucinate-more) (Apr 2024)\n* Blog of Intel Developer News: [Use the neural-chat-7b Model for Advanced Fraud Detection: An AI-Driven Approach in Cybersecurity](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fdeveloper\u002Farticles\u002Ftechnical\u002Fbilics-approach-cybersecurity-using-neuralchat-7b.html) (March 2024)\n* CES 2024: [CES 2024 Great Minds Keynote: Bringing the Limitless Potential of AI Everywhere: Intel Hybrid Copilot demo](https:\u002F\u002Fyoutu.be\u002F70J3uO3eLZA?t=1348) (Jan 2024)\n* Blog published on Medium: [Connect an AI agent with your API: Intel Neural-Chat 7b LLM can replace Open AI Function Calling](https:\u002F\u002Fmedium.com\u002F11tensors\u002Fconnect-an-ai-agent-with-your-api-intel-neural-chat-7b-llm-can-replace-open-ai-function-calling-242d771e7c79) (Dec 2023)\n* NeurIPS'2023 on Efficient Natural Language and Speech Processing: [Efficient LLM Inference on CPUs](https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.00502) (Nov 2023)\n* Blog published on Hugging Face: [Intel Neural-Chat 7b: Fine-Tuning on Gaudi2 for Top LLM Performance](https:\u002F\u002Fhuggingface.co\u002Fblog\u002FAndyrasika\u002Fneural-chat-intel) (Nov 2023)\n* Blog published on VMware: [AI without GPUs: A Technical Brief for VMware Private AI with Intel](https:\u002F\u002Fcore.vmware.com\u002Fresource\u002Fai-without-gpus-technical-brief-vmware-private-ai-intel#section6) (Nov 2023)\n  \n> View [Full Publication List](.\u002Fdocs\u002Fpublication.md)\n\n## Additional Content\n\n* [Release Information](.\u002Fdocs\u002Frelease.md)\n* [Contribution Guidelines](.\u002Fdocs\u002Fcontributions.md)\n* [Legal Information](.\u002Fdocs\u002Flegal.md)\n* [Security Policy](SECURITY.md)\n* [Apache License](.\u002FLICENSE)\n\n\n## Acknowledgements\n* Excellent open-source projects: [bitsandbytes](https:\u002F\u002Fgithub.com\u002FTimDettmers\u002Fbitsandbytes), [FastChat](https:\u002F\u002Fgithub.com\u002Flm-sys\u002FFastChat), [fastRAG](https:\u002F\u002Fgithub.com\u002FIntelLabs\u002FfastRAG), [ggml](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fggml), [gptq](https:\u002F\u002Fgithub.com\u002FIST-DASLab\u002Fgptq), [llama.cpp](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp), [lm-evauation-harness](https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness), [peft](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fpeft), [trl](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl), [streamingllm](https:\u002F\u002Fgithub.com\u002Fmit-han-lab\u002Fstreaming-llm) and many others.\n\n* Thanks to all the [contributors](.\u002Fdocs\u002Fcontributors.md).\n\n## 💁Collaborations\n\nWelcome to raise any interesting ideas on model compression techniques and LLM-based chatbot development! Feel free to reach [us](mailto:itrex.maintainers@intel.com), and we look forward to our collaborations on Intel Extension for Transformers!\n","Intel Extension for Transformers 是一个基于Transformer的工具包，旨在加速生成式AI和大型语言模型在各种设备上的运行。该项目提供了先进的压缩技术，如4位量化（INT4），支持多种硬件平台包括Intel CPU和GPU，从而实现高效的推理性能。它特别适用于需要快速构建聊天机器人、处理大规模语言模型的应用场景，并且能够通过优化技术显著提升在Intel平台上的执行效率。此外，项目还集成了诸如神经聊天等实用功能，进一步增强了其在实际应用中的灵活性与实用性。",2,"2026-06-11 03:29:55","top_topic"]