Multiple Instance Attention Learning for Multimodal Classification and Detection for Breast Cancer Diagnosis

This project addresses the challenging problem of breast cancer image analysis, which is of high clinical relevance. Breast cancer is the most prevalent cancer among women worldwide, and timely detection of potential lesions using medical imaging is crucial for improving survival rates. The objective of this project is to assist radiologists by providing a “second reader opinion” through a multimodal analysis system, focusing on two key tasks: (1) image classification and (2) lesion delineation. While the classification task involves assigning a score to the image, the delineation task is more demanding as it requires the radiologist to outline the lesions. To tackle these challenges, we propose a weakly supervised training approach that leverages only image-level labels. This approach enables simultaneous image classification and lesion delineation, offering a comprehensive solution for assisting radiologists in their diagnostic workflow.


This project focuses on improving the analysis of breast cancer images, which is important for detecting and treating breast cancer in women. We want to help radiologists by providing a computer system that can assist in two main tasks: classifying images and outlining the areas of concern. The first task involves giving a score to the image, while the second task requires marking the specific areas of potential cancer. To achieve this, we are developing a system that can learn from examples without needing detailed annotations. This means we can use more readily available image-level labels instead of requiring specific annotations for each lesion. By doing this, we hope to make the diagnosis process faster and more accurate, potentially saving lives by aiding in early detection of breast cancer. This research is important not only for medical professionals but also for the broader community as it can improve the effectiveness and efficiency of breast cancer screening and diagnosis. In the future, this approach may have broader applications in other areas of medical imaging and help researchers develop better diagnostic tools for various diseases.


01/04/2023 - 30/06/2026

Funding scheme

Fundação para a Ciência e Tecnologia

Project reference



Hospital Professor Doutor Fernando Fonseca, EPE; Institute for Systems and Robotics (ISR-Lisboa)

Project website


HCI Subfields

Health; Specific Applications Areas