Journal Article

MFSM-Net: Multimodal Feature Fusion for the Semantic Segmentation of Urban-Scale Textured 3D Meshes

Hao, Xinjie, Leng, Wei, Wang, Jiahui, Zhang, Guangyun, Zhang, Rongting

2025

Overview

The semantic segmentation of textured 3D meshes is a critical step in constructing city-scale realistic 3D models. Compared to colored point clouds, textured 3D meshes have the advantage of high-resolution texture image patches embedded on each mesh face. However, existing studies predominantly focus on their geometric structures, with limited utilization of these high-resolution textures. Inspired by the binocular perception of humans, this paper proposes a multimodal feature fusion network based on 3D geometric structures and 2D high-resolution texture images for the semantic segmentation of textured 3D meshes. Methodologically, the 3D feature extraction branch computes the centroid coordinates and face normals of mesh faces as initial 3D features, followed by a multi-scale Transformer network to extract high-level 3D features. The 2D feature extraction branch employs orthographic views of city scenes captured from a top-down perspective and uses a U-Net to extract high-level 2D features. To align features across 2D and 3D modalities, a Bridge view-based alignment algorithm is proposed, which visualizes the 3D mesh indices to establish pixel-level associations with orthographic views, achieving the precise alignment of multimodal features. Experimental results demonstrate that the proposed method achieves competitive performance in city-scale textured 3D mesh semantic segmentation, validating the effectiveness and potential of the cross-modal fusion strategy.

• Publisher: MDPI AG
• Subject: 3D mesh scene understanding / Accuracy / Algorithms / Alignment / Centroids / Computational linguistics / Computer vision / Coordinate transformations / Datasets / Deep learning / Euclidean space / Feature extraction / Geometry / High resolution / Image processing / Image resolution / Image segmentation / Language processing / multimodal feature extraction / Natural language interfaces / Natural language processing / Neural networks / Orthography / Semantic segmentation / Semantics / Texture / textured 3D mesh / Three dimensional models