Reducing Gender and Racial Biases in Multimodal Vision Language Models via Contrastive Debiasing and Representative Dataset Synthesis
DOI:
https://doi.org/10.66280/cis.v4i1.126Abstract
The rapid proliferation of multimodal vision language models has transformed the landscape of artificial intelligence, enabling unprecedented capabilities in image captioning, visual question answering, and generative content synthesis. However, these advancements have also surfaced profound ethical concerns regarding the systematic reproduction of gender and racial biases inherent in foundational training data. This research investigates a holistic systems-level framework for mitigating these biases through a dual-pronged strategy involving contrastive debiasing architectures and representative dataset synthesis. Rather than relying on post-hoc filtering or simple data augmentation, the proposed approach integrates fairness as a core architectural constraint during the alignment phase of multimodal training. By synthesizing diverse datasets that fill historical representation gaps and implementing contrastive learning objectives that penalize stereotypical associations between protected attributes and unrelated semantic features, the framework aims to decouple social identity from task-related performance. The discussion encompasses the structural trade-offs between model accuracy and fairness, the infrastructure required for large-scale synthetic data generation, and the governance frameworks necessary for auditing multimodal systems. Ultimately, this paper argues that achieving equitable artificial intelligence requires an interdisciplinary commitment to re-engineering the socio-technical pipeline, moving beyond algorithmic fixes toward sustainable, bias-aware systemic infrastructures.
References
1.Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California Law Review, 104, 671.
2.Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623.
3.Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1–15.
4.Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186.
5.Crawford, K. (2021). The Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press.
6.DeVries, T., Misra, I., Wang, C., & van der Maaten, L. (2019). Does object recognition work for everyone? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
7.Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). Fairness through awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, 214–226.
8.Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
9.Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29.
10.Hoffmann, A. L. (2019). Where fairness fails: Data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society, 22(7), 900–915.
11.Jia, C., Yang, Y., Xia, Y., Chen, Y. T., Parekh, Z., Pham, H., ... & Duerig, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, 4918–4927.
12.Kay, M., Matuszek, C., & Munson, S. A. (2015). Unequal representation and gender stereotypes in image search results. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 3819–3828.
13.Karkkainen, K., & Joo, J. (2021). FairFace: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1548–1558.
14.Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness. Advances in Neural Information Processing Systems, 30.
15.Liang, P. P., Wu, C. B., Morency, L. P., & Salakhutdinov, R. (2021). Towards understanding and mitigating social biases in language models. International Conference on Machine Learning.
16.Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019). Model cards for model reporting. Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, 220–229.
17.Noble, S. U. (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.
18.Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 8748–8763.
19.Raji, I. D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of publicly naming biased AI systems. Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, 59–68.
20.Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017). Right for the right reasons: Training differentiable agents by constraining their explanations. Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2662–2670.
21.Selvavarapu, R. K., Mohit, J., Singh, A., Berg, A. C., & Berg, T. L. (2020). Choosing the right stakes for fairness. arXiv preprint arXiv:2012.06738.
22.Shi, C., Li, S., Guo, S., Xie, S., Wu, W., Dou, J., ... & Chua, T. S. (2025). Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation. arXiv preprint arXiv:2511.17282.
23.Srinivasan, K., & Chander, A. (2021). Biases in generative art: A causal look from the lens of art history. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency.
24.Tan, H., & Bansal, M. (2019). LXMERT: Learning cross-modality encoder representations from transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
25.Wang, A., Narayanan, A., & Russakovsky, O. (2020). REVISE: A tool for measuring and mitigating bias in visual datasets. International Conference on European Conference on Computer Vision, 733–751.
26.Wang, T., Zhao, J., Yatskar, M., Chang, K. W., & Ordonez, V. (2019). Balanced datasets are not enough: Estimating and mitigating gender bias in deep image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, 5310–5319.
27.Whittaker, M., Crawford, K., Dobbe, R., Fried, G., Kaziunas, E., Varner, M., ... & West, S. M. (2018). AI Now Report 2018. AI Now Institute at New York University.
28.Yang, K., Qin, K., Duan, Y., & Russakovsky, O. (2020). Towards fairer datasets: Filtering and balancing the distribution of the people category in ImageNet. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 547–558.
29.Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). HellaSwag: Can a machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
30.Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K. W. (2017). Men also do laundry: Multi-attribute bias amplification its mitigation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2979–2989.
31.Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9), 2337–2348.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Computational Intelligence Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
