Quality-Adaptive Multimodal Face Recognition with Streaming Prototype Learning and CrossModal Evaluation
Loading...
Date
2026
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
SDU University
Abstract
While state-of-the-art face recognition systems achieve near-perfect accuracy under ideal conditions, their performance degrades significantly in real-world conditions when the input images are blurred, noisy, have partially-occluded faces, or are of low resolution. Multi-modal face recognition systems that combine visible-spectrum, near-infrared (NIR), and depth sensor inputs tend to exhibit even greater performance degradation under these conditions because each sensor modality suffers degradation in different ways. In this dissertation, we consider three main problems with current multi-modal face recognition systems: estimating the quality of the input images from each sensor modality, weighting the inputs according to their estimated quality, and maintaining the identity galleries within the recognition systems in an efficient manner. A new quality estimation function Q(I)is proposed for each input image I that utilises noisecorrected sharpness, noise level, face coverage, and contrast to provide a score that represents the quality of the image without requiring training data. When evaluated on the LFW face dataset with synthetic image degradations, the quality estimates have a Spearman rank correlation coefficient ρ of 0.90 or higher for all four types of image degradation, while the baseline method based on the variance of the Laplacian kernel has a negative correlation with certain types of degradation. When incorporating the quality score estimates into the fusion of visible-spectrum, NIR, and depth sensor recognition results, the system achieves up to a 4.4 percentage point improvement in Rank-1 recognition accuracy under degraded NIR inputs. A streaming framework for updating the identity galleries that utilises an exponential moving average with a momentum value of 0.9 and reservoir sampling achieves the same accuracy as full retraining of the model using only 49% of the computational cost, and utilises only the embedding vectors of the faces rather than the raw face images. A fairness audit of the system on the FairFace dataset reveals a null result in relation to the fairness metric, suggesting future work on this topic. All experiments utilised publicly available datasets and models, and the code for implementing the proposed methods is provided as a publicly available repository.
Description
Keywords
face recognition, image quality estimation, multi-modal fusion, adaptive weighting, streaming prototype learning
Citation
Anarbayeva D / Quality-Adaptive Multimodal Face Recognition with Streaming Prototype Learning and CrossModal Evaluation / SDU University