A Dynamic Frame Selection Framework for Fast Video Recognition

We introduce AdaFrame, a conditional computation framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame, which contains a Long Short-Term Memory augmented with a global memory to provide context information, operates as an agent to interact with v...

Ausführliche Beschreibung

Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on pattern analysis and machine intelligence. - 1979. - 44(2022), 4 vom: 07. Apr., Seite 1699-1711
1. Verfasser: Wu, Zuxuan (VerfasserIn)
Weitere Verfasser: Li, Hengduo, Xiong, Caiming, Jiang, Yu-Gang, Davis, Larry S
Format: Online-Aufsatz
Sprache:English
Veröffentlicht: 2022
Zugriff auf das übergeordnete Werk:IEEE transactions on pattern analysis and machine intelligence
Schlagworte:Journal Article Research Support, Non-U.S. Gov't
LEADER 01000naa a22002652 4500
001 NLM315971061
003 DE-627
005 20231225160052.0
007 cr uuu---uuuuu
008 231225s2022 xx |||||o 00| ||eng c
024 7 |a 10.1109/TPAMI.2020.3029425  |2 doi 
028 5 2 |a pubmed24n1053.xml 
035 |a (DE-627)NLM315971061 
035 |a (NLM)33026981 
040 |a DE-627  |b ger  |c DE-627  |e rakwb 
041 |a eng 
100 1 |a Wu, Zuxuan  |e verfasserin  |4 aut 
245 1 2 |a A Dynamic Frame Selection Framework for Fast Video Recognition 
264 1 |c 2022 
336 |a Text  |b txt  |2 rdacontent 
337 |a ƒaComputermedien  |b c  |2 rdamedia 
338 |a ƒa Online-Ressource  |b cr  |2 rdacarrier 
500 |a Date Completed 28.03.2022 
500 |a Date Revised 01.04.2022 
500 |a published: Print-Electronic 
500 |a Citation Status MEDLINE 
520 |a We introduce AdaFrame, a conditional computation framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame, which contains a Long Short-Term Memory augmented with a global memory to provide context information, operates as an agent to interact with video sequences aiming to search over time which frames to use. Trained with policy search methods, at each time step, AdaFrame computes a prediction, decides where to observe next, and estimates a utility, i.e., expected future rewards, of viewing more frames in the future. Exploring predicted utilities at testing time, AdaFrame is able to achieve adaptive lookahead inference so as to minimize the overall computational cost without incurring a degradation in accuracy. We conduct extensive experiments on two large-scale video benchmarks, FCVID and ActivityNet. With a vanilla ResNet-101 model, AdaFrame achieves similar performance of using all frames while only requiring, on average, 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We also demonstrate AdaFrame is compatible with modern 2D and 3D networks for video recognition. Furthermore, we show, among other things, learned frame usage can reflect the difficulty of making prediction decisions both at instance-level within the same class and at class-level among different categories 
650 4 |a Journal Article 
650 4 |a Research Support, Non-U.S. Gov't 
700 1 |a Li, Hengduo  |e verfasserin  |4 aut 
700 1 |a Xiong, Caiming  |e verfasserin  |4 aut 
700 1 |a Jiang, Yu-Gang  |e verfasserin  |4 aut 
700 1 |a Davis, Larry S  |e verfasserin  |4 aut 
773 0 8 |i Enthalten in  |t IEEE transactions on pattern analysis and machine intelligence  |d 1979  |g 44(2022), 4 vom: 07. Apr., Seite 1699-1711  |w (DE-627)NLM098212257  |x 1939-3539  |7 nnns 
773 1 8 |g volume:44  |g year:2022  |g number:4  |g day:07  |g month:04  |g pages:1699-1711 
856 4 0 |u http://dx.doi.org/10.1109/TPAMI.2020.3029425  |3 Volltext 
912 |a GBV_USEFLAG_A 
912 |a SYSFLAG_A 
912 |a GBV_NLM 
912 |a GBV_ILN_350 
951 |a AR 
952 |d 44  |j 2022  |e 4  |b 07  |c 04  |h 1699-1711