Improved segmentation model to identify object instances based on textual prompts

Abstract

The rapidly increasing amount of multimedia information requires significant methods development for its rapid processing. In this case, one of the areas of processing is preliminary analysis with the images characteristic features detection to reduce the information required for subsequent tasks. One of the types for an information reduction is image segmentation. In this case, the general task of image segmentation is often reduced to the task of object segmentation is a fundamental task in computer vision, requiring accurate pixel-by-pixel object delineation and scene understanding. With the development of natural language processing techniques, many approaches have been successfully adapted to computer vision tasks, allowing for more intuitive descriptions of scenes using natural language. Unlike traditional models limited to a fixed set of classes, natural language processing-based approaches allow searching for objects based on attributes, expanding their applicability. While existing object segmentation methods are typically categorized into one-stage and two-stage methods – depending on speed and accuracy - there remains a gap in developing models that can effectively identify and segment objects based on textual prompts. To address this, we propose an open-set instance segmentation model capable of detecting and segmenting objects from prompts. Our approach builds upon CLIPSeg, integrating architectural modifications from Panoptic-DeepLab and PRN (Panoptic Refinement Network) to predict object centers and pixel-wise distances to boundaries. A post-processing phase refines segmentation results to improve object separation. The proposed architecture is trained on large vocabulary instance segmentation and PhraseCut datasets and evaluated using the mean Dice score against state-of-the-art open-set segmentation models. Experimental results show that although our model achieves the highest inference rate among open-set methods while maintaining FastSAM-level segmentation quality, post-processing remains a limiting factor. This suggests that future improvements should be aimed at eliminating the post-processing process itself or improving its algorithm, which could lead to more efficient segmentation.