Automated Data Pipelines for Real-Time Monitoring of Global Weapon Transfers

Project granted by the German Foundation for Peace Research/Deutsche Stiftung Friedensforschung

Principal investigators:  Prof. Dr. Paul W. Thurner und Dr. Oliver Pamp

Duration: 12 Months

Project Description: The international arms trade has a profound impact on global security, with the transfer of small arms and light weapons (SALW) to state and non-state actors playing a critical role in the escalation and de-escalation of conflicts. Despite its importance, data on SALW transfers remain limited due to the cessation of critical prior data initiatives and significant underreporting issues in existing sources. To address this gap, the pilot project aims to explore the feasibility of automated data pipelines for efficiently collecting and processing data on global weapon transfers. These pipelines will leverage machine learning and large language models to track global arms transfers to both state and non-state actors in real-time. The project, in partnership with SIPRI and leveraging support from the Munich Data Science hub, seeks to create a sustainable, updated data collection technology. The initiative will focus primarily on SALW but will also encompass major conventional weapons (MCW), aiming for greater accuracy and sustainability in arms monitoring practices.

The envisioned data pipelines are intended to leverage pre-existing datasets in conjunction with continually updated governmental records, trade statistics, text corpora, and news media sources. The system will integrate AI tools and custom pipelines to evaluate performance and conduct multilingual text analyses to identify actors and weapon types. The streamed data is standardized using Named Entity Recognition and Deep Learning algorithms to provide detailed information on providers, recipients, and weapon details. The data engineering process will be semi-supervised, requiring domain experts due to substantial domain knowledge demands. The aim is to demonstrate efficiency gains and forensic possibilities by incorporating new data sources and integrating data streams.

The implementation of automated data pipelines is expected to transform and improve the monitoring of global weapon transfers, enhancing policymaking and international collaboration on arms control. This will also contribute to new research data infrastructure in peace and conflict studies. The Pilot Project will be crucial in evaluating the feasibility of these pipelines by analyzing data sources, refining extraction methods, establishing data cleaning and integration protocols, and exploring automation of extraction processes with potential real-time monitoring systems.