Combining multiple drugs to treat patients is a common practice, especially in complex diseases and severe conditions. However, this approach can cause drug–drug interactions (DDIs) when two or more drugs interact in ways that can affect their effectiveness and behaviour within the body. These interactions can pose significant risks to patient health [1].

The rise of artificial intelligence (AI) has transformed how we approach various medical challenges. Some approaches used classification networks that take scientific sentences as input to extract the interactions [2]. However, the performance of text-based models in the biomedical literature is often lower than in general documents [3]. This is mainly because biomedical texts require specialised knowledge to grasp complex concepts. The success of fine-tuned large language models (LLMs), such as BioMedBERT [4], highlights the necessity of specialised models.

Drug often contains a wealth of information that can be leveraged for DDI extraction. A drug formula shows the elements in a compound and their relative proportions, while the drug description covers uses, dosage, and side effects. Additionally, a molecular graph represents the structure of a compound, illustrating connections between atoms. Integrating these diverse modalities can improve the prediction of drug behaviour and interactions.

Moreover, multimodal approaches [5] offer a promising opportunity for more comprehensive DDI extraction. However, a major challenge is the heterogeneity of the data sources, which makes it difficult to process the modalities. While researchers have proposed several fusion techniques to effectively capture multimodal features, their performance in DDI remains largely unexplored. Therefore, we aim to enhance DDI classification accuracy through these fusion techniques.

In this study, we propose a deep multimodal fusion model that uses the representation of biomedical text assisted by four modalities that describe the contexts of the drug. We first improve an LLM-based model to process textual information from scientific articles. Then, we integrate it with augmented information, including molecular structure graphs, molecular structure images, chemical formulas, and textual drug descriptions. We also conducted experiments with different phases of fusions (i.e., early, intermediate, and late fusion) to explore their performance. Our main contributions are summarised as follows:

We build a novel deep multimodal fusion model to combine diverse modalities for DDI extraction, including textual data from input sentences and drug descriptions, molecular structure images, molecular structure graphs, and chemical formulas.

We explore integrating various fusion techniques with different phases, including early, intermediate, and late fusion, to analyse their impact on model accuracy and robustness.

We compare several large language models (LLMs), including BioMedBERT, SciBERT, and BioBERT, to generate rich representations for textual data. Additionally, we utilise a large vision model, namely Vision Transformer (ViT), to process molecular structure images. This multimodal approach captures complex relationships across textual and visual modalities, improving drug–drug interaction (DDI) extraction.

Statement of Significance