Use big data wisely and it can provide big insights

Big Data_Flickr_ infocux Technologies
Copyright: Flickr/infocux Technologies

Speed read

  • The Global Pulse initiative is using big data to assess attitudes and trends
  • Social media analysis can reveal food price indicators, perceptions and the results of advocacy efforts
  • But local knowledge is needed to correctly interpret data patterns

Send to a friend

The details you provide on this page will not be used to send unsolicited email, and will not be sold to a 3rd party. See privacy policy.

The variety of data generated every day — from satellite images to people’s geotagged photos of their breakfast — is astonishing. The sheer quantity now being produced is hard to fathom: experts estimate that 90 per cent of all the digital data in the world has been generated over the last two years.
The term ‘big data’ refers to the information generated by the everyday use of digital services, produced so frequently, in such diverse ways and at scales so large that it cannot be stored or analysed through traditional means. Analysis of this information can provide valuable insights about human behaviour and intentions.

This new era of big data represents significant challenges and opportunities for global development. With support and guidance, the development community can embrace the craft of ‘data science’ to enhance evidence-based decision-making. Yet it is important that humanitarian and development practitioners and policymakers understand that data on its own can’t inform the development agenda without a detailed understanding of its local context.

Global Pulse, a UN innovation initiative established by secretary-general Ban Ki-moon in 2009, is working to build development sector capacity to make use of new sources of digital data while understanding its local context.

The power of social data

The initiative is exploring how powerful the insights from big data can be. One example is our research project looking at how Twitter can provide information on how people are affected by food price spikes. We analysed millions of public tweets from between 2010 and 2011 for particular key words. In one data set containing thousands of tweets from Indonesia discussing the price of rice, we saw that the volume of tweets about staple foods correlated closely with actual spikes in their cost. [1] Learning how to identify and understand such data could help the development community anticipate trends.

“The development community can embrace the craft of ‘data science’ to enhance evidence-based decision-making.”

Robert Kirkpatrick

And there’s more. Tracking sentiments on social media and blogs provides an opportunity to get real-time awareness and feedback about particular communities’ attitudes. For example, a recent project by UNICEF (the UN Children’s Fund) showed how it is possible to track parents’ attitudes to immunisation by identifying trends in the sentiments of their public posts on social media. [2]
Finally, there is an opportunity to use big data to evaluate and measure the impact of specific humanitarian, aid or development interventions. We applied this concept to help the UN’s Every Woman Every Child initiative, which campaigns to improve the health of women and children around the world. Our preliminary analysis of millions of social media posts, selected based on their use of relevant keywords, revealed a significant shift towards statements showing an awareness of parental health, previously an overlooked dimension, over the past two years. This provides an indication that the programme’s message could be getting through.

Of course, harvesting insights from big data will not replace traditional data sources such as surveys, censuses and official statistics because not every demographic is represented. Rather, it will supplement them: providing new streams of timely information for development practitioners and policymakers tackling hunger, poverty and disease.

Local labs

These examples show how useful big data can be. Yet data need to be understood in context. To achieve this, Global Pulse is organised into a network of Pulse Labs in different regions of the world. There is a headquarters lab in New York, United States, a lab in Jakarta, Indonesia and lab is opening this month in Kampala, Uganda. These collaborative research centres include multidisciplinary teams of data scientists, engineers, analysts and digital communications experts who prototype, test and share big data techniques and tools across the UN system and beyond.  
The local expertise in Pulse Labs is crucial for gaining an awareness of cultural contexts and specific development challenges. Understanding the ethnographic dimensions of how different cultures and communities use digital services such as mobile phones or social media is key to being able to leverage big data for development purposes.
Indonesians, for example, are avid users of social media. Jakarta is the most tweeting city in the world, more vocal than New York or London. Yet in other regions, different activities generate high volumes of digital data. In Kenya, for example, social media use is much less widespread, but the use of mobile phones to transfer money is popular and could be a rich source of data. [3,4]
Big data, and the art of data science, offer new options in the policymaking toolkit. Data science is not a quick, technology-based solution, however. It takes collaboration, intuition and cultural insight. To be able to correctly interpret patterns in big data that are relevant to human development requires a nuanced understanding of local contexts, and therefore it is beneficial to do the analysis close to where the development needs are. We at Global Pulse are encouraged by the emerging practice of applying data science to development, but urge newcomers to the field to remember the importance of involving local partners and building the capacity to do data science from the ground up.
Robert Kirkpatrick is the director of UN Global Pulse. To find out more about how to collaborate on projects with Pulse Labs, contact [email protected]
See below for a video by Global Pulse about the Twitter project in Indonesia:


[1] UN Global Pulse Twitter and Perceptions of Crisis-Related Stress (UN Global Pulse, accessed 28 November 2013)

[2] UNICEF Tracking Anti-Vaccination Sentiment in Eastern European Social Media Networks (UNICEF, April 2013)

[3] UN Global Pulse Landscaping Study: Digital Signals & Access to Finance in Kenya (UN Global Pulse, accessed 28 November 2013) 

[4] Kaplan, E. Searching for Digital Signals in Kenya (UN Global Pulse, 3 September 2013)