Innova-tsn at the SAS Hackathon 2024
The Innova-tsn team has recently participated in the prestigious competition SAS Hackathon 2024. This year, we decided to take on the challenge of designing an Innovative solution for a critical social issue: poverty and vulnerability.
Our tool, designed with SAS Viya and enhanced by Large Language Models (LLMs) focuses on analysing real-time digital content from platforms such as Facebook, X, Google News and Google Trends. With this approach, the aim is to identify issues of poverty and vulnerability through what citizens perceive in social networks and media, offering a perspective on the geographical areas where poverty and vulnerability are greast. This method helps us understand social well-being, life expectations and their relation to income and access to goods and services.
As a starting point for this project, we seek to undestand Colombia’s socio-economic reality and the methods implemented to officially measure poverty in the country through the National Administrative Department for Stadistics (DANE). In Colombia there are measurement methodologies: monetary poverty and multidimensional poverty. The former assesses per capita income of households compared to the income required to cover the basic food basket, while the latter measures access to goods and services and deprivations in the quality of life of households, differentiating five dimensions: education, health, work, housing and conditions of children and adolescents. Traditional methods require costly surveys and long collection periods and do not allow for disaggregation a delay in the publication of results and the impossibility of geographically disaggregated analysis, limiting the efficient allocation of resources. Our solution integrates real-time data from various sources, closing a critical gap in socio-economic analysis and contributing to the targeting of social programmes.
The development of our tool started with the identification of key information sources such as Google Trends, Googles News, Facebook and X. From these platforms, the search was carried out based on the 5 dimensions establishment by DANE for the Colombian MPI, and considering the subject matter, we included two more indicators for the search and categorisation of information: inequality and hunger.
By using web scraping techniques by means of Python’s Selenium library, we gathered the texts concerning the defined subject matters through news, searches and comments on social media, which were later analysed with LLMs and SAS Viya. This analysis included the deletion of unrelated texts through cosine similarity, sentiment categorisation of the post (positive, neutral or negative) and the identification of geographical location through names of departments of municipalities.
Users’ texts and comments were structured in three variables: Department/Region, Dimension and Sentiment. This allowed us to create indicators to identify the ratio of negative texts that indicate alerts about social problems. As a result, we created an aggregated indicator by deparment and dimension, facilitating the identification of priority areas and providing a summary of the alerts for decision-makers.
Our tool not only identify highly vulnerable ares, but also listen to the citizens’ voice to facilitate the approach and effectiveness of the implementation of social policies and programmes, thus allowing a better distribution os resources. The integration of Generative AI techniques enhances our solution, enablig the analysis of large data volumes.
With a great potential for scalability, our tool can adapt for monitoring changes in public perception regarding economic issues, natural disasters and political situations. This innovative approach has the potential to revolutionise the way resources are distributed globally, setting a new standard for the analysis and comparison of social problems in different regions.
As future developments of this solution, in order to have a wider reach, an interface with a chatbot could be realised that would allow it to probe further into the issues encountered, providing policy makers with answers to questions that traditional methods cannot address.
We are excited about the opportunities our tool offers for the transformation of socio-economic analysis and targeting of social programmes.
We believe that, when using real-tim data and Artificial Intelligence, we can contribute to a more equitable and sustainable future for everybody. In addition, we are very proud of having participated, one more year, in such a prestigious international contest as the SAS Hackathon 2024 and we hope to continue doing so in the following editions.
This year, we managed to achieve 3rd place worldwide in the Data4Good category, as well as being recognised by the jury with the People’s Choice Winners badge.
Thank you for being with us in this journey towards innovation and social change!
