[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-79868":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":15,"stars7d":16,"stars30d":17,"stars90d":14,"forks30d":14,"starsTrendScore":18,"compositeScore":19,"rankGlobal":8,"rankLanguage":8,"license":8,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":22,"hasPages":20,"topics":23,"createdAt":8,"pushedAt":8,"updatedAt":24,"readmeContent":25,"aiSummary":26,"trendingCount":14,"starSnapshotCount":14,"syncStatus":13,"lastSyncTime":27,"discoverSource":28},79868,"ReluPruner","Orange2019220\u002FReluPruner","Orange2019220",null,"Python",146,5,3,2,0,1,6,15,4,2.33,false,"main",true,[],"2026-06-12 02:03:55","# ReLU Pruning for Privacy-Preserving Inference\n\n## Overview\n\nThis repository implements a progressive ReLU pruning method based on Taylor importance estimation for privacy-preserving neural network inference. The approach reduces the number of ReLU activations in deep neural networks while maintaining model accuracy, which is crucial for efficient secure multi-party computation (MPC) and homomorphic encryption (HE) based inference.\n\n## Key Features\n\n- **Taylor Importance Estimation**: Uses first and second-order Taylor expansion to estimate the importance of ReLU activations\n- **Progressive Pruning**: Gradually prunes ReLUs during training to maintain model performance\n- **Layer-wise Importance**: Considers layer depth and gradient information for better pruning decisions\n- **Knowledge Distillation**: Leverages teacher-student framework to preserve accuracy\n\n## Requirements\n\n```bash\npip install torch torchvision numpy einops\n```\n\n## Project Structure\n\n```\n.\n├── train_teacher.py                              # Train teacher model\n├── train_with_taylor_pruning_global_relu_pruning_abs.py  # Main training script with Taylor pruning\n├── validate.py                                   # Validation script\n├── models\u002F                                       # Model architectures\n├── helper\u002F                                       # Training utilities\n├── distiller_zoo\u002F                                # Knowledge distillation modules\n└── dataset\u002F                                      # Dataset loaders\n```\n\n## Usage\n\n### 1. Train Teacher Model\n\nFirst, train a standard teacher model with full ReLU activations:\n\n```bash\n# CIFAR-10\npython train_teacher.py --model='ResNet18' --dataset='cifar10'\n\n# CIFAR-100\npython train_teacher.py --model='ResNet18' --dataset='cifar100'\n\n# Tiny ImageNet\npython train_teacher.py --model='ResNet18' --dataset='tiny_imagenet' --batch_size=32\n```\n\n### 2. Train with Taylor-based ReLU Pruning\n\nApply progressive ReLU pruning using Taylor importance estimation:\n\n```bash\n# CIFAR-100 with 5% ReLU retention (95% pruning)\npython train_with_taylor_pruning_global_relu_pruning_abs.py \\\n    --path_t='save\u002Fmodels\u002FResNet18_cifar100_batch128\u002FResNet18_best.pth' \\\n    --alpha=0.3 \\\n    --gamma=0.01 \\\n    --model_s='CustomResNet18' \\\n    --dataset='cifar100' \\\n    --distill='kd' \\\n    --global_keep_ratio=0.05\n\n# CIFAR-100 with 10% ReLU retention\npython train_with_taylor_pruning_global_relu_pruning_abs.py \\\n    --path_t='save\u002Fmodels\u002FResNet18_cifar100_batch128\u002FResNet18_best.pth' \\\n    --alpha=0.5 \\\n    --gamma=0.5 \\\n    --model_s='CustomResNet18' \\\n    --dataset='cifar100' \\\n    --distill='kd' \\\n    --global_keep_ratio=0.1\n```\n\n### 3. Validate Model\n\n```bash\npython validate.py \\\n    --model='CustomResNet18' \\\n    --dataset='cifar100' \\\n    --path='save\u002Fstudent_model\u002FCustomResNet18_pruned.pth'\n```\n\n## Key Parameters\n\n- `--path_t`: Path to pre-trained teacher model\n- `--model_s`: Student model architecture (e.g., CustomResNet18)\n- `--dataset`: Dataset name (cifar10, cifar100, tiny_imagenet)\n- `--global_keep_ratio`: Ratio of ReLUs to keep (e.g., 0.05 = 5% ReLUs retained)\n- `--alpha`: Weight for knowledge distillation loss\n- `--gamma`: Weight for task loss\n- `--distill`: Distillation method (kd for knowledge distillation)\n\n## Method Overview\n\nThe pruning method consists of:\n\n1. **Importance Calculation**: Compute Taylor importance scores for each ReLU activation\n   - First-order: gradient magnitude\n   - Second-order: Hessian diagonal approximation\n   - Layer-wise weighting based on depth\n\n2. **Progressive Pruning**: Gradually increase pruning ratio during training\n   - Start with low pruning ratio\n   - Progressively increase to target ratio\n   - Allow model to adapt at each pruning step\n\n3. **Knowledge Distillation**: Maintain accuracy using teacher guidance\n   - Soft label matching\n   - Feature-level distillation\n\n## Results\n\nThe method achieves significant ReLU reduction (90-95%) while maintaining competitive accuracy on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.\n\n## Citation\n\nIf you find this work useful, please cite:\n\n```bibtex\n@inproceedings{li2026relupruner,\n  title={ReLUPruner: Rethinking ReLU Importance with Taylor Expansion for Efficient Private Inference},\n  author={Li, Zhenpeng and Liu, Jinshuo and Wang, Xinyan and Wang, Lina and Pan, Jeff Z},\n  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n  volume={40},\n  number={28},\n  pages={23328--23336},\n  year={2026}\n}\n```\n\n## License\n\nMIT License\n","这个项目实现了一种基于泰勒重要性估计的渐进ReLU剪枝方法，用于保护隐私的神经网络推理。核心功能包括使用一阶和二阶泰勒展开来评估ReLU激活的重要性，并在训练过程中逐步剪枝以保持模型性能。此外，该方法考虑了层深度和梯度信息进行更优的剪枝决策，并利用教师-学生框架通过知识蒸馏保留准确性。适合需要高效安全多方计算（MPC）和同态加密（HE）的场景，特别是在对隐私保护有高要求的深度学习应用中。","2026-06-11 03:58:20","CREATED_QUERY"]