[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-82902":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":10,"languages":10,"totalLinesOfCode":10,"stars":11,"forks":12,"watchers":13,"openIssues":12,"contributorsCount":12,"subscribersCount":12,"size":12,"stars1d":12,"stars7d":12,"stars30d":12,"stars90d":12,"forks30d":12,"starsTrendScore":12,"compositeScore":14,"rankGlobal":10,"rankLanguage":10,"license":15,"archived":16,"fork":16,"defaultBranch":17,"hasWiki":18,"hasPages":16,"topics":19,"createdAt":10,"pushedAt":10,"updatedAt":20,"readmeContent":21,"aiSummary":22,"trendingCount":12,"starSnapshotCount":12,"syncStatus":13,"lastSyncTime":23,"discoverSource":24},82902,"SmartFed","benmagnifico\u002FSmartFed","benmagnifico","[ICML 2026 Spotlight] SmartFed is a resource-efficient framework that circumvents expensive training from scratch by intelligently reusing knowledge embedded in existing LoRA modules.","",null,99,0,2,40,"MIT License",false,"main",true,[],"2026-06-12 04:01:39","\u003Cdiv align=\"center\">\n\n# SmartFed: Don't Reinvent the Wheel, Just Realign the Spokes\n### Resource-Efficient Federated Fine-Tuning via Rank-Wise Expert Assembly\n\n[![Paper](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-ICML%202026%20Spotlight-b31b1b.svg?style=for-the-badge&logo=arxiv)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2512.00902)\n[![Code](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode-Coming%20Soon-orange.svg?style=for-the-badge&logo=github)](#)\n[![Dataset](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDataset-Coming%20Soon-blue.svg?style=for-the-badge&logo=huggingface)](#)\n[![LoRA Modules](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLoRA%20Modules-Coming%20Soon-yellow.svg?style=for-the-badge&logo=huggingface)](#)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-MIT-green.svg?style=for-the-badge)](LICENSE)\n\n\u003C\u002Fdiv>\n\n---\n\n\u003C!-- > [!IMPORTANT] -->\n> 🔥 **News**\n> - **[2026\u002F05]** SmartFed is accepted to **ICML 2026** as a **Spotlight** paper. 🎉\n> - **[Coming Soon]** Source code, processed datasets, and pre-trained LoRA modules will be publicly released. Please ⭐ star and watch this repo for updates.\n\n---\n\n## 📖 Overview\n\n**SmartFed** is a resource-efficient federated fine-tuning framework that adapts Large Language Models (LLMs) to downstream tasks by **reusing existing LoRA modules** instead of training from scratch. The open-source community has accumulated a wealth of task-specific LoRA modules — SmartFed asks a simple question: *why manufacture new spokes from raw materials when the wheel already exists? Just realign the spokes you already have.*\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ffig1_framework.png\" width=\"80%\">\u003C\u002Fp>\n\nInstead of optimizing billions of LoRA parameters across hundreds of communication rounds, **edge devices in SmartFed only train a lightweight router** that dynamically composes knowledge from a frozen pool of *rank-wise experts*, drastically reducing computation, communication, and energy cost.\n\n## 🔑 Highlights\n\n- 🧩 **Mixture of Rank-Wise Experts (MoRE)** — decomposes each LoRA module along the rank dimension, turning monolithic modules into a pool of fine-grained, lightweight experts that the router can selectively activate based on input semantics.\n- ⚖️ **Elastic Expert Quota Allocation (EEQA)** — adaptively redistributes the expert budget across parameter matrices according to their measured contribution, concentrating capacity where it matters most.\n- 📡 **From training to composing** — only a tiny router (\u003C 0.1% of LLM parameters) is synchronized across clients; the LoRA experts remain frozen and cached locally.\n- 🛡️ **Theoretical guarantees** — provable noise suppression, algebraic elimination of destructive cross-task interference, and a near-optimality bound for EEQA (proofs in the paper appendix).\n- 🚀 **Strong empirical gains** — up to **+10.21%** average accuracy, **3.95× faster** convergence, **31.47× lower** communication overhead, and **3.61× less** energy consumption versus state-of-the-art baselines.\n\n## 🏗️ Method\n\nSmartFed reformulates federated fine-tuning as a **two-stage compose-and-route** process.\n\n**1. Rank-Wise Decomposition.** A standard LoRA update is rewritten as a sum of rank-one projections, and each rank-one component is treated as an independent expert:\n\n$$\\Delta \\mathbf{W}\\mathbf{x} = \\sum_{i=1}^{r} \\mathbf{B}_{:,i}\\big(\\mathbf{A}_{i,:}\\mathbf{x}\\big), \\qquad E_i(\\mathbf{x}) = (\\mathbf{B}_{:,i}\\mathbf{A}_{i,:})\\mathbf{x}.$$\n\n**2. Sparse Routing.** A trainable router computes top-$K$ gating weights and aggregates only the selected experts:\n\n$$\\mathbf{h}' = \\mathbf{W}_0 \\mathbf{x} + \\sum_{m=1}^{M} \\tilde{g}_m \\cdot E_m(\\mathbf{x}).$$\n\n**3. Adaptive Quota Allocation.** EEQA reallocates the expert budget across parameter matrices each round based on per-matrix importance, using a two-phase (proportional + greedy residual) procedure.\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ffig6_more.png\" width=\"70%\">\u003C\u002Fp>\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ffig2_reuse_strategies.png\" width=\"85%\">\u003C\u002Fp>\n\n## 📊 Main Results\n\nSmartFed consistently outperforms both *knowledge-free* (train-from-scratch) and *knowledge-reuse* baselines across three skill-composition tasks and three LLM backbones. See Table 1 of the paper for the full comparison.\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ftable1_main_results.png\" width=\"90%\">\u003C\u002Fp>\n\n**Efficiency.** Compared with knowledge-free federated baselines, SmartFed slashes wall-clock training time, communication payload, and energy footprint by large margins:\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ffig7_efficiency.png\" width=\"90%\">\u003C\u002Fp>\n\n**Data Efficiency.** With only 10% of the training data, SmartFed already surpasses FedIT trained on the full dataset:\n\n\u003Cp align=\"center\">\u003Cimg src=\"assets\u002Ffig8_data_efficiency.png\" width=\"90%\">\u003C\u002Fp>\n\n## 📁 Repository Structure\n\n```text\nSmartFed\u002F\n├── assets\u002F                          # Figures and illustrations\n├── federated_learning\u002F              # Federated training loop and aggregation\n│   ├── fed_trainer.py               #   FedAvg-style trainer for the router\n│   ├── client.py                    #   Per-device local update logic\n│   └── aggregator.py                #   Server-side aggregation of router + importance scores\n├── smartfed\u002F                        # Core algorithmic components\n│   ├── more.py                      #   Mixture of Rank-Wise Experts (decomposition + routing)\n│   ├── router.py                    #   Lightweight, input-conditioned router\n│   ├── eeqa.py                      #   Elastic Expert Quota Allocation strategy\n│   └── importance.py                #   Per-expert importance scoring (Eq. 4 in the paper)\n├── lora_pool\u002F                       # Utilities for loading and managing reusable LoRA modules\n│   ├── registry.py                  #   Index of task-specific LoRA modules\n│   └── loaders.py                   #   Loading \u002F caching from LoRAHub-style repositories\n├── training_scripts\u002F                # Shell entry points for federated runs\n│   ├── run_smartfed_cn_math.sh      #   Chinese + Math skill composition\n│   ├── run_smartfed_cn_code.sh      #   Chinese + Code skill composition\n│   └── run_smartfed_math_code.sh    #   Math  + Code skill composition (GSM-Hard)\n├── evaluation\u002F                      # Evaluation harnesses\n│   ├── mgsm\u002F                        #   Chinese mathematical reasoning (MGSM)\n│   ├── doit\u002F                        #   Chinese code generation (DoIT, Pass@1)\n│   └── gsm_hard\u002F                    #   Hard math-word problems (execution accuracy)\n├── utils\u002F                           # Shared utilities\n│   ├── template.py                  #   Chat \u002F instruction templates\n│   ├── process_dataset.py           #   Dataset registration and partitioning\n│   └── carbon.py                    #   CodeCarbon hooks for energy reporting\n├── main_smartfed.py                 # Top-level federated training entry point\n├── config.py                        # Centralized CLI \u002F config schema\n├── requirements.txt                 # Python dependencies\n├── setup.sh                         # Environment setup script\n├── LICENSE\n└── README.md\n```\n\n## 🚀 Quick Start\n\n\u003C!-- > [!NOTE] -->\n> 🚧 The training and evaluation code is **not yet open-sourced**. The commands below will be runnable once the official release lands. We are actively cleaning up the code and preparing the release.\n\n**1. Clone and install:**\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002F\u003Corg>\u002FSmartFed.git\ncd SmartFed\nconda create -n smartfed python=3.10 -y\nconda activate smartfed\npip install -r requirements.txt\nsource setup.sh\n```\n\n**2. Launch a federated SmartFed run:**\n\n```bash\npython main_smartfed.py \\\n  --model_name_or_path \"meta-llama\u002FLlama-2-7b-hf\" \\\n  --lora_pool \"cn_chat,en_math\" \\\n  --task \"cn_math\" \\\n  --num_clients 20 \\\n  --sample_clients 2 \\\n  --num_rounds 20 \\\n  --local_steps 10 \\\n  --batch_size 16 \\\n  --learning_rate 5e-4 \\\n  --topk_experts 32 \\\n  --enable_eeqa \\\n  --output_dir \".\u002Foutput\u002Fsmartfed_cn_math\"\n```\n\nKey arguments:\n- `--lora_pool` — comma-separated list of task-specific LoRA modules to compose (resolved via `lora_pool\u002Fregistry.py`).\n- `--topk_experts` — per-matrix expert budget $K$ used by MoRE.\n- `--enable_eeqa` — toggle Elastic Expert Quota Allocation.\n\n## 🏋️ Training\n\nThree skill-composition tasks are supported out of the box. Each has a dedicated launcher under `training_scripts\u002F`:\n\n| Task | LoRA Modules Composed | Training Data | Eval Set | Metric |\n|------|-----------------------|---------------|----------|--------|\n| Chinese Mathematical Reasoning | Chinese Chat + English Math | Math23K | MGSM | Accuracy |\n| Chinese Code Generation | Chinese Chat + English Code | DoIT (expanded to 20K) | DoIT | Pass@1 |\n| Hard Math-Word Problems | English Math + English Code | MathCodeInstruct | GSM-Hard | Execution Accuracy |\n\n**Federated protocol (defaults).** 20 clients in total, 10% sampled per round, 10 local steps per client, 20 communication rounds, AdamW with lr = 5e-4, batch size = 16. Each base LoRA module is rank 32, alpha 64, injected into the Query and Value attention matrices.\n\n## 📏 Evaluation\n\nEvaluation harnesses for the three benchmarks live under `evaluation\u002F`. Each subdirectory will ship with its own README and run script.\n\n- **MGSM** — multilingual chain-of-thought math reasoning, Chinese split.\n- **DoIT** — Chinese code generation, Pass@1.\n- **GSM-Hard** — execution accuracy on Program-Aided Language Model (PAL) outputs.\n\n## 📦 Datasets & LoRA Modules\n\nSmartFed reuses three publicly available task-specific LoRA modules:\n\n- **Chinese Chat LoRA** — trained on the 52K Okapi Chinese instruction set.\n- **English Math LoRA** — trained on the 395K MetaMath corpus.\n- **English Code LoRA** — trained on the 186K Magicoder OSS-Instruct corpus.\n\n\u003C!-- > [!NOTE] -->\n> 📂 To facilitate reproduction, we plan to release **(i)** the three task-specific LoRA modules used in our experiments, **(ii)** the federated-partitioned training splits, and **(iii)** the expanded DoIT 20K subset. These resources will be uploaded to Hugging Face alongside the code release.\n\n## 🗓️ Release Plan\n\nWe are actively preparing the open-source release. The intended timeline:\n\n- [ ] 🧹 Code cleanup and refactor for public release\n- [ ] 📦 Release of the three reusable LoRA modules\n- [ ] 🗂️ Release of federated-partitioned datasets (Math23K, expanded DoIT, MathCodeInstruct)\n- [ ] 🧪 Release of evaluation scripts for MGSM, DoIT, and GSM-Hard\n\nIf you would like to be notified when the code drops, please ⭐ star or 👀 watch this repository.\n\n## 🤝 Acknowledgements\n\nThis project builds on the shoulders of many excellent open-source efforts, including [LoRA](https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLoRA), [LoRAHub](https:\u002F\u002Fgithub.com\u002Fsail-sg\u002Florahub), [PEFT](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fpeft), [OpenFedLLM](https:\u002F\u002Fgithub.com\u002Frui-ye\u002FOpenFedLLM), and the broader Hugging Face ecosystem. We thank the authors and maintainers of these libraries for making collaborative research possible.\n\n## 📝 Citation\n\nIf you find SmartFed useful in your research, please consider citing our paper:\n\n```bibtex\n@inproceedings{wu2025smartfed,\n  title         = {Don't Reinvent the Wheel, Just Realign the Spokes: \n                   Resource-Efficient Federated Fine-Tuning via Rank-Wise Expert Assembly}, \n  author        = {Yebo Wu and Jingguang Li and Zhijiang Guo and Li Li},\n  year          = {2025},\n  eprint        = {2512.00902},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.DC},\n  url           = {https:\u002F\u002Farxiv.org\u002Fabs\u002F2512.00902}, \n}\n```\n","SmartFed 是一个资源高效的联邦微调框架，通过智能重用现有的LoRA模块来避免从零开始的昂贵训练。其核心功能包括混合秩级专家（MoRE）和弹性专家配额分配（EEQA），能够将LoRA模块分解为细粒度的轻量级专家，并根据输入语义选择性激活这些专家。此外，边缘设备仅需训练一个轻量级路由器，从而大幅减少计算、通信和能耗成本。SmartFed 适用于需要高效利用已有知识进行模型微调的场景，特别是在对资源消耗敏感的边缘计算环境中。","2026-06-11 04:09:36","CREATED_QUERY"]