Explore the vast range of titles available.

Although deep learning algorithms have achieved significant progress in a variety of domains, they require costly annotations on huge datasets. Self-supervised learning (SSL) using unlabeled data has emerged as an alternative, as it eliminates manual annotation. To do this, SSL constructs feature representations using pretext tasks that operate without manual annotation, which allows models trained in these tasks to extract useful latent representations that later improve downstream tasks such as object classification and detection. The early methods of SSL are based on auxiliary pretext tasks as a way to learn representations using pseudo-labels, or labels that were created automatically based on the dataset’s attributes. Furthermore, contrastive learning has also performed well in learning representations via SSL. To succeed, it pushes positive samples closer together, and negative ones further apart, in the latent space. This paper provides a comprehensive literature review of the top-performing SSL methods using auxiliary pretext and contrastive learning techniques. It details the motivation for this research, a general pipeline of SSL, the terminologies of the field, and provides an examination of pretext tasks and self-supervised methods. It also examines how self-supervised methods compare to supervised ones, and then discusses both further considerations and ongoing challenges faced by SSL.

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction—predict the representations for the masked patches, and masked patch reconstruction—reconstruct the masked patches. The network is an encoder–regressor–decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://github.com/Atten4Vis/CAE.

Deep learning and advancements in computer vision offer significant potential for analyzing medical images resulting in better healthcare and improved patient outcomes. Currently, the dominant approaches in the field of machine learning are supervised learning and transfer learning. These methods are not only prevalent in medicine and healthcare but also across various other industries. They rely on large datasets that have been manually annotated to train increasingly sophisticated models. However, the manual labeling process results in a wealth of untapped, unlabeled data that is accessible in both public and private data repositories. Self-supervised learning (SSL), an emerging field within machine learning, provides a solution by leveraging this untapped, unlabeled data. Unlike traditional machine learning paradigms, SSL algorithms pre-train models using artificial supervisory signals generated from the unlabeled data. This comprehensive review article explores the fundamental concepts, approaches, and advancements in self-supervised learning, with a particular emphasis on medical image datasets and their sources. By summarizing and highlighting the main contributions and findings from the article, this analysis and synthesis aim to shed light on the current state of research in self-supervised learning. Through these rigorous efforts, the existing body of knowledge is synthesized, and implementation recommendations are provided for future researchers interested in harnessing self-supervised learning to develop classification models for medical imaging.

Self-supervised learning (SSL) is a potential deep learning (DL) technique that uses massive volumes of unlabeled data to train neural networks. SSL techniques have evolved in response to the poor classification performance of conventional and even modern machine learning (ML) and DL models of enormous unlabeled data produced periodically in different disciplines. However, the literature does not fully address SSL’s practicalities and workabilities necessary for industrial engineering and medicine. Accordingly, this thorough review is administered to identify these prominent possibilities for prediction, focusing on industrial and medical fields. This extensive survey, with its pivotal outcomes, could support industrial engineers and medical personnel in efficiently predicting machinery faults and patients’ ailments without referring to traditional numerical models that require massive computational budgets, time, storage, and effort for data annotation. Additionally, the review’s numerous addressed ideas could encourage industry and healthcare actors to take SSL principles into an agile application to achieve precise maintenance prognostics and illness diagnosis with remarkable levels of accuracy and feasibility, simulating functional human thinking and cognition without compromising prediction efficacy.

Over the last decade, supervised deep learning on manually annotated big data has been progressing significantly on computer vision tasks. But, the application of deep learning in medical image analysis is limited by the scarcity of high-quality annotated medical imaging data. An emerging solution is self-supervised learning (SSL), among which contrastive SSL is the most successful approach to rivalling or outperforming supervised learning. This review investigates several state-of-the-art contrastive SSL algorithms originally on natural images as well as their adaptations for medical images, and concludes by discussing recent advances, current limitations, and future directions in applying contrastive SSL in the medical domain.

Background Drug-target binding affinity (DTA) prediction is important for the rapid development of drug discovery. Compared to traditional methods, deep learning methods provide a new way for DTA prediction to achieve good performance without much knowledge of the biochemical background. However, there are still room for improvement in DTA prediction: (1) only focusing on the information of the atom leads to an incomplete representation of the molecular graph; (2) the self-supervised learning method could be introduced for protein representation. Results In this paper, a DTA prediction model using the deep learning method is proposed, which uses an undirected-CMPNN for molecular embedding and combines CPCProt and MLM models for protein embedding. An attention mechanism is introduced to discover the important part of the protein sequence. The proposed method is evaluated on the datasets Ki and Davis, and the model outperformed other deep learning methods. Conclusions The proposed model improves the performance of the DTA prediction, which provides a novel strategy for deep learning-based virtual screening methods.

Driven by the urgent demand for flood monitoring, water resource management and environmental protection, water-body detection in remote sensing imagery has attracted increasing research attention. Deep semantic segmentation networks (DSSNs) have gradually become the mainstream technology used for remote sensing image water-body detection, but two vital problems remain. One problem is that the traditional structure of DSSNs does not consider multiscale and multishape characteristics of water bodies. Another problem is that a large amount of unlabeled data is not fully utilized during the training process, but the unlabeled data often contain meaningful supervision information. In this paper, we propose a novel multiscale residual network (MSResNet) that uses self-supervised learning (SSL) for water-body detection. More specifically, our well-designed MSResNet distinguishes water bodies with different scales and shapes and helps retain the detailed boundaries of water bodies. In addition, the optimization of MSResNet with our SSL strategy can improve the stability and universality of the method, and the presented SSL approach can be flexibly extended to practical applications. Extensive experiments on two publicly open datasets, including the 2020 Gaofen Challenge water-body segmentation dataset and the GID dataset, demonstrate that our MSResNet can obviously outperform state-of-the-art deep learning backbones and that our SSL strategy can further improve the water-body detection performance.

Political perspective detection in news media—identifying political bias in news articles—is an essential but challenging low-resource task. Prompt-based learning (i.e., discrete prompting and prompt tuning) achieves promising results in low-resource scenarios by adapting a pre-trained model to handle new tasks. However, these approaches suffer performance degradation when the target task involves a textual domain (e.g., a political domain) different from the pre-training task (e.g., masked language modeling on a general corpus). In this paper, we develop a novel multi-stage prompt tuning framework for political perspective detection. Our method involves two sequential stages: a domain- and task-specific prompt tuning stage. In the first stage, we tune the domain-specific prompts based on a masked political phrase prediction (MP3) task to adjust the language model to the political domain. In the second task-specific prompt tuning stage, we only tune task-specific prompts with a frozen language model and domain-specific prompts for downstream tasks. The experimental results demonstrate that our method significantly outperforms fine-tuning (i.e., model tuning) methods and state-of-the-art prompt tuning methods on the SemEval-2019 Task 4: Hyperpartisan News Detection and AllSides datasets.

Understanding the interplay between diseases and genes is crucial for gaining deeper insights into disease mechanisms and optimizing therapeutic strategies. In recent years, various computational methods have been developed to uncover potential disease-gene associations. However, existing computational approaches for disease-gene association prediction still face two major limitations. First, most current studies focus on constructing complex heterogeneous graphs using multi-dimensional biological entity relationships, while overlooking critical latent interaction patterns, namely, disease neighbor interactions and gene neighbor interactions—which are more valuable for association prediction. Second, in self-supervised learning (SSL), the presence of noise in auxiliary tasks commonly affects the accurate modeling of diseases and genes. In this study, we propose a novel denoising method for disease-gene association prediction, termed DGSL. To address the first issue, we utilize bipartite graphs corresponding to diseases and genes to derive disease-disease and gene-gene similarities, and further construct disease and gene interaction graphs to capture the latent interaction patterns. To tackle the second challenge, we implement cross-view denoising through adaptive semantic alignment in the embedding space, while preserving useful neighbor interactions. Extensive experiments on benchmark datasets demonstrate the effectiveness of our method.

Accounting for interindividual variability in brain function is key to precision medicine. Here, by considering functional interindividual variability as meaningful data rather than noise, we introduce VarCoNet, an enhanced self‐supervised framework for robust functional connectome (FC) extraction from resting‐state fMRI (rs‐fMRI) data. VarCoNet employs self‐supervised contrastive learning to exploit inherent functional interindividual variability, serving as a brain function encoder that generates FC embeddings readily applicable to downstream tasks even in the absence of labeled data. Contrastive learning is facilitated by a novel augmentation strategy based on segmenting rs‐fMRI signals. At its core, VarCoNet integrates a 1D‐convolutional neural network (CNN) with a Transformer encoder for advanced time‐series processing, enhanced with robust Bayesian hyperparameter optimization. Our VarCoNet framework is evaluated on two downstream tasks: (i) subject fingerprinting, using rs‐fMRI data from the Human Connectome Project (2117 recordings), and (ii) autism spectrum disorder (ASD) classification, using rs‐fMRI data from the Autism Brain Imaging Data Exchange (ABIDE) I (995 recordings) and II (730 recordings) datasets. Using different brain parcellations, our extensive testing against state‐of‐the‐art methods, including 13 deep learning methods, demonstrates VarCoNet's superiority, robustness, interpretability, and generalizability, achieving up to 98% subject fingerprinting accuracy and an area under the curve (AUC) of 72.6% for ASD classification. Overall, VarCoNet provides a versatile and robust framework for FC analysis in rs‐fMRI. VarCoNet is a robust framework for functional connectome estimation from rs‐fMRI. Using contrastive learning, it trains a 1D‐CNN–Transformer encoder to improve the intra‐ to intersubject variability ratio. VarCoNet achieves state‐of‐the‐art subject fingerprinting and outperforms existing methods in ASD classification.