|
|
|
|
| LEADER |
01000naa a22002652 4500 |
| 001 |
NLM344154211 |
| 003 |
DE-627 |
| 005 |
20231226022118.0 |
| 007 |
cr uuu---uuuuu |
| 008 |
231226s2023 xx |||||o 00| ||eng c |
| 024 |
7 |
|
|a 10.1109/TPAMI.2022.3194044
|2 doi
|
| 028 |
5 |
2 |
|a pubmed24n1147.xml
|
| 035 |
|
|
|a (DE-627)NLM344154211
|
| 035 |
|
|
|a (NLM)35895643
|
| 040 |
|
|
|a DE-627
|b ger
|c DE-627
|e rakwb
|
| 041 |
|
|
|a eng
|
| 100 |
1 |
|
|a Li, Changlin
|e verfasserin
|4 aut
|
| 245 |
1 |
0 |
|a DS-Net++
|b Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers
|
| 264 |
|
1 |
|c 2023
|
| 336 |
|
|
|a Text
|b txt
|2 rdacontent
|
| 337 |
|
|
|a ƒaComputermedien
|b c
|2 rdamedia
|
| 338 |
|
|
|a ƒa Online-Ressource
|b cr
|2 rdacarrier
|
| 500 |
|
|
|a Date Completed 10.04.2023
|
| 500 |
|
|
|a Date Revised 11.04.2023
|
| 500 |
|
|
|a published: Print-Electronic
|
| 500 |
|
|
|a Citation Status PubMed-not-MEDLINE
|
| 520 |
|
|
|a Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. In this paper, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, that can generalized well on multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.). Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by nested residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we present DS-CNN++ and DS-ViT++, by carefully designing the double headed dynamic gate and the overall network architecture. We further propose dynamic idle slicing to address the drastic reduction of embedding dimension in DS-ViT++. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage I, in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to stablize and improve the training of DS-CNN++ and DS-ViT++ supernet, respectively. In Stage II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our methods consistently outperform the state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, we achieves 2-4× computation reduction and up to 61.5% real-world acceleration on MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net
|
| 650 |
|
4 |
|a Journal Article
|
| 700 |
1 |
|
|a Wang, Guangrun
|e verfasserin
|4 aut
|
| 700 |
1 |
|
|a Wang, Bing
|e verfasserin
|4 aut
|
| 700 |
1 |
|
|a Liang, Xiaodan
|e verfasserin
|4 aut
|
| 700 |
1 |
|
|a Li, Zhihui
|e verfasserin
|4 aut
|
| 700 |
1 |
|
|a Chang, Xiaojun
|e verfasserin
|4 aut
|
| 773 |
0 |
8 |
|i Enthalten in
|t IEEE transactions on pattern analysis and machine intelligence
|d 1979
|g 45(2023), 4 vom: 09. Apr., Seite 4430-4446
|w (DE-627)NLM098212257
|x 1939-3539
|7 nnns
|
| 773 |
1 |
8 |
|g volume:45
|g year:2023
|g number:4
|g day:09
|g month:04
|g pages:4430-4446
|
| 856 |
4 |
0 |
|u http://dx.doi.org/10.1109/TPAMI.2022.3194044
|3 Volltext
|
| 912 |
|
|
|a GBV_USEFLAG_A
|
| 912 |
|
|
|a SYSFLAG_A
|
| 912 |
|
|
|a GBV_NLM
|
| 912 |
|
|
|a GBV_ILN_350
|
| 951 |
|
|
|a AR
|
| 952 |
|
|
|d 45
|j 2023
|e 4
|b 09
|c 04
|h 4430-4446
|