Reasoning action-centric temporal relations at rich feature hierarchies for action recognition

Abstract

Reasoning temporal relations among action-related objects plays an important role in action recognition. However, previous approaches only focus the reasoning on high-level semantics and inevitably involve the background in reasoning. In this work, we propose to formulate the temporal relational reasoning in an action-centric and hierarchical style, with a novel Action-centric Temporal-relational Reasoning (ATR) block. Specifically, ATR comprises two components: an Action-related Region Locator (ARL) to locate the action-related regions via estimating the actionness, and an Efficient Action-centric Reasoner (EAR) to efficiently reason the temporal relations between the located regions so as to realize the action-centric reasoning. Thanks to its flexible and efficient designs, our ATR can be directly integrated into existing action recognition models at different depths, empowering the hierarchical reasoning on the action-centric temporal relations at the cost of minor computational overhead. We extensively evaluate our ATR block on three action recognition benchmarks, Something-Something V1, V2, and Kinetics, with the backbones of TSN, TSM, and SlowOnly. The consistent and notable improvements over the strong baselines sufficiently corroborate the effectiveness of ATR, along with the action-centric and hierarchical formulation for temporal relational reasoning. Our proposed approach provides potential practical significance for real-world scenarios.