Learning Probabilistic Presence-Absence Evidence for Weakly-Supervised Audio-Visual Event Perception
With only video-level event labels, this paper targets at the task of weakly-supervised audio-visual event perception (WS-AVEP), which aims to temporally localize and categorize events that belong to each modality. Despite the recent progress, most existing approaches either ignore the unsynchronize...
| Veröffentlicht in: | IEEE transactions on pattern analysis and machine intelligence. - 1979. - 47(2025), 6 vom: 03. Mai, Seite 4787-4802 |
|---|---|
| 1. Verfasser: | |
| Weitere Verfasser: | , |
| Format: | Online-Aufsatz |
| Sprache: | English |
| Veröffentlicht: |
2025
|
| Zugriff auf das übergeordnete Werk: | IEEE transactions on pattern analysis and machine intelligence |
| Schlagworte: | Journal Article |
| Zusammenfassung: | With only video-level event labels, this paper targets at the task of weakly-supervised audio-visual event perception (WS-AVEP), which aims to temporally localize and categorize events that belong to each modality. Despite the recent progress, most existing approaches either ignore the unsynchronized property of audio-visual tracks or discount the complementary modality for explicit enhancement. We argue that, a modality should provide ample presence evidence for an event, while the complementary modality offers absence evidence as a reference. However, to learn reliable evidence, we face challenging uncertainties caused by weak supervision and the complicated audio-visual data itself. To this end, we propose to collect Probabilistic Presence-Absence Evidence (PPAE) in a unified framework. Specifically, by leveraging uni-modal and cross-modal representations, a probabilistic presence-absence evidence collector (PAEC) is designed. To learn the evidence in a reliable range, we propose a joint-modal mutual learning (JML) process, which calibrates the evidence of diverse audible, visible, and audi-visible events adaptively and dynamically. Extensive experiments show that our method surpasses state-of-the-arts (e.g., absolute gains of 3.1% and 4.2% in terms of event-level audio and visual metrics on the LLP dataset) |
|---|---|
| Beschreibung: | Date Revised 07.05.2025 published: Print-Electronic Citation Status PubMed-not-MEDLINE |
| ISSN: | 1939-3539 |
| DOI: | 10.1109/TPAMI.2025.3546312 |