In today’s world, where data is distributed across various locations and privacy is paramount, Federated Learning (FL) has emerged as a game-changing solution. It enables multiple parties to train machine learning models collaboratively without sharing their data, ensuring that sensitive information remains locally stored and protected. However, a significant challenge arises when the data labels provided by human annotators are imperfect, leading to heterogeneous label noise distributions across different parties involved in the federated learning process. This issue can severely undermine the performance of FL models, hindering their ability to generalize effectively and make accurate predictions.
Researchers have explored various approaches to address label noise in FL, broadly classified into coarse-grained and fine-grained methods. Coarse-grained methods focus on strategies at the client level, such as selectively choosing clients with low noise ratios or identifying clean client sets. On the other hand, fine-grained methods concentrate on techniques at the sample level, aiming to identify and filter out noisy label samples from individual clients.
However, a common limitation of these existing methods is that they often need to pay more attention to the inherent heterogeneity of label noise distributions across clients. This heterogeneity can arise from varying true class distributions or personalized human labeling errors, making it challenging to achieve substantial performance improvements.
To tackle this issue head-on, a team of researchers from Xi’an Jiaotong University, Leiden University, Docta AI, California State University, Monterey Bay, and the University of California, Santa Cruz, has proposed FedFixer. This innovative algorithm leverages a dual model structure consisting of a global model and a personalized model. The global model benefits from aggregated updates across clients, robustly representing the overall data distribution.
Conversely, the personalized model is specifically designed to adapt to the unique characteristics of each client’s data, including client-specific samples and label noise patterns.
In their groundbreaking approach, the researchers behind FedFixer have incorporated two key regularization techniques to combat the potential overfitting of the dual models, particularly the personalized model, which is trained on limited local data.
The first technique is a confidence regularizer, which modifies the traditional Cross-Entropy loss function to alleviate the impact of unconfident predictions caused by label noise. By incorporating a term that encourages the model to produce confident predictions, the confidence regularizer guides the model towards better fitting the clean dataset, reducing the influence of noisy label samples.
The second technique is a distance regularizer, which constrains the disparity between the personalized and global models. This regularizer is implemented by adding a term to the loss function that penalizes the deviation of the personalized model’s parameters from the global model’s parameters. The distance regularizer acts as a stabilizing force, preventing the personalized model from overfitting to local noisy data due to the limited sample size available on each client.
Furthermore, FedFixer employs an alternative update strategy for the dual models during the local training. The global and personalized models are updated using the samples selected by each other’s model. This alternating update process leverages the complementary strengths of the two models, effectively decreasing the risk of error accumulation from a single model over time.
The researchers conducted extensive experiments on benchmark datasets, including MNIST, CIFAR-10, and Clothing1M, with varying degrees of label noise and heterogeneity. The results demonstrate that FedFixer outperforms existing state-of-the-art methods, particularly in highly heterogeneous label noise scenarios. For example, on the CIFAR-10 dataset with a non-IID distribution, a noisy client ratio of 1.0, and a lower bound noise level of 0.5, FedFixer achieved an accuracy of 59.01%, up to 10% higher than other methods.
To illustrate the potential real-world impact, consider a healthcare application where federated learning is employed to collaboratively train diagnostic models across multiple hospitals while preserving patient data privacy. In such a scenario, label noise can arise due to variations in medical expertise, subjective interpretations, or human errors during the annotation process. FedFixer’s ability to handle heterogeneous label noise distributions would be invaluable, as it could effectively filter out mislabeled data and improve the generalization performance of the diagnostic models, ultimately leading to more accurate and reliable predictions that could save lives.
In conclusion, the research paper introduces FedFixer, an innovative approach to mitigating the impact of heterogeneous label noise in Federated Learning. By employing a dual model structure with regularization techniques and alternative updates, FedFixer effectively identifies and filters out noisy label samples across clients, improving generalization performance, especially in highly heterogeneous label noise scenarios. The proposed method’s effectiveness has been extensively validated through experiments on benchmark datasets, demonstrating its potential for real-world applications where data privacy and label noise are significant concerns, such as in the healthcare domain or any other field where accurate and reliable predictions are crucial.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our 39k+ ML SubReddit
Vibhanshu Patidar is a consulting intern at MarktechPost. Currently pursuing B.S. at Indian Institute of Technology (IIT) Kanpur. He is a Robotics and Machine Learning enthusiast with a knack for unraveling the complexities of algorithms that bridge theory and practical applications.
