Journal Article

HAPiCLR: heuristic attention pixel-level contrastive loss representation learning for self-supervised pretraining

Huang, Chi-En, Wang, Jia-Ching, Tran, Van Nhiem, Liu, Shen-Hsuan, Li, Yung-Hui, Aslam, Muhammad Saqlain, Yang, Kai-Lin

2024

Overview

Recent self-supervised contrastive learning methods are powerful and efficient for robust representation learning, pulling semantic features from different cropping views of the same image while pushing other features away from other images in the embedding vector space. However, model training for contrastive learning is quite inefficient. In the high-dimensional vector space of the images, images can differ from each other in many ways. We address this problem with heuristic attention pixel-level contrastive loss for representation learning (HAPiCLR), a self-supervised joint embedding contrastive framework that operates at the pixel level and makes use of heuristic mask information. HAPiCLR leverages pixel-level information from the object’s contextual representation instead of identifying pair-wise differences in instance-level representations. Thus, HAPiCLR enhances contrastive learning objectives without requiring large batch sizes, memory banks, or queues, thereby reducing the memory footprint and the processing needed for large datasets. Furthermore, HAPiCLR loss combined with other contrastive objectives such as SimCLR or MoCo loss produces considerable performance boosts on all downstream tasks, including image classification, object detection, and instance segmentation.

• Publisher: Springer Berlin Heidelberg,Springer Nature B.V
• Subject: Artificial Intelligence / Attention / Colorization / Computer Graphics / Computer Science / Computer vision / Educational objectives / Embedding / Heuristic / Image classification / Image Processing and Computer Vision / Image segmentation / Instance segmentation / Learning / Memory tasks / Methods / Object recognition / Original Article / Pixels / Representation learning / Semantics / Vector spaces