Skip to main content
Exploring what’s next in tech – Insights, information, and ideas for today’s IT and business leaders

Feeding the world, one terabyte at a time

As the global population explodes, HPE's Memory-Driven Computing Sandbox is the power behind a new effort to improve the world's food security.

Among the COVID-19 pandemic's many economic and social impacts has been the unprecedented strain it has put on the global food supply. At once, it has disrupted farm production, food processing, transportation and logistics, and consumer demand. But while the effect has been worldwide, it has unfolded differently in every country, demanding multifold solutions rather than a single magic bullet.

The task of uncovering those solutions has been taken on by the Consortium for Global International Agriculture Research (CGIAR) in partnership with Hewlett Packard Enterprise. CGIAR is a global research partnership of 14 nonprofit agricultural research institutes working in more than 100 countries on research into virtually every aspect of food security. It uses data from satellite images, weather records, crop performance, economic activities, and surveys to generate a complete, timely picture of what is occurring in areas of high food production, known as food baskets. By applying HPE's Memory-Driven Computing Sandbox to these massive datasets, CGIAR is able to model food systems and uncover insights to help predict how food security challenges are unfolding from the pandemic and inform effective responses.

The growing food crisis

The global food supply chain was facing significant challenges even before the COVID-19 pandemic struck. Around the world, nearly 800 million people are chronically undernourished and 2 billion are affected by vitamin or mineral deficiency. Contributing to the hunger problem is a decline in the number of smaller farms—those two acres or less, which account for about 80 percent of the world's farming—because of difficulties with ongoing viability. Global hunger is projected to only get worse: The global population could reach between 8.5 billion and 9.8 billion by 2030, depending on the source, necessitating 70 percent more food than is currently consumed.

When the pandemic struck, it wasn't a question of whether its impacts would worsen the food crisis, but how they would take shape.

Please read: 6 ways in which big data is taking a bite out of the food industry

"As a global community, we found that we were kind of flying blind in terms of knowing what the impact of COVID was going to be on food availability," says Brian King, coordinator of CGIAR's Platform for Big Data in Agriculture. "We knew there would be shocks, but we didn't know the nature of the shocks. I don't know that we've gone through this acute level of the same crisis, unfolding in thousands of different ways at the same time around the world."

COVID fosters a collaboration

Shortly after the pandemic began, HPE began looking to partner with pharmaceutical, medical, and research organizations to offer help in their efforts around addressing COVID issues, says Janice Zdankus, vice president of innovation for social impact at HPE. The company connected with CGIAR, where King told them the organization was running several research projects to understand the effects of COVID shutdowns on emissions and that it could use help calculating analytics across 250 cities. HPE quickly agreed to join the efforts and provided access to the Sandbox.

The project involved compiling lockdown data, satellite records, emissions signatures, and other information, and then uploading terabytes of that data to the Sandbox to train a model for recognizing greenhouse gas, particularly nitrous oxide, from space. While CGIAR has high-performance computing clusters at several of its centers, the need for this type of timely, localized information and analysis was beyond its existing capabilities.

"We don't have the compute power to do that," King says. "To set something up like that would take several weeks to run on our infrastructure, and you'd have to deprioritize other things. In this case, the Sandbox provided the speed and the scale that was needed to generate useful insights about the world with a timeliness that allows us to do something about them. That's what made the difference."

The Memory-Driven Computing Sandbox, which takes its name from the controlled environment in which allows customers to experiment, was designed for the kind of big data processing that is at the center of CGIAR's work. In a typical system, small amounts of memory are tethered to each processor. One processor must request access to data held in the others to perform any tasks, which is prohibitively inefficient with massive datasets.

The Sandbox gives each processor, which contains up to 64 sockets, access to up to 48 terabytes of shared memory. This makes huge, diverse datasets available in memory simultaneously, removing the computational impediments that can slow down research and discovery and hamper the timeliness required for work like CGIAR's. Emissions analysis on a single point on the earth would have taken four to five hours to run using CGIAR's existing technology. Using the Sandbox, CGIAR can run multiple analyses across multiple points with the necessary frequency to inform preventative action.

Monitoring the future of food security

Today, CGIAR can build high-frequency views of food systems that link together crop modeling—including weather records, crop performance, and yield amounts, all by year and location—survey data, and overall economic activity. This allows researchers to monitor emissions from up to 1,000 points across India and East Africa. Changes in any of these emission levels, King explains, could signal changes in economic activity that would give CGIAR critical context for understanding how food security challenges are developing and allow it to compare that picture with crop and survey data to determine how, say, an individual crop will impact the overall food supply.

Ultimately, the insights gathered from this data will allow CGIAR to predict how pandemic-related food security issues are unfolding in individual food producing cities and settlements. It will be able to share these data insights with farmers, policymakers, and food relief organizations to help support a collective response to supply chain disruptions, food shortages, and other challenges.

Please read: Precision agriculture yields higher profits, lower risks

While COVID-related food supply challenges are the most pressing issues CGIAR is addressing, its larger goal is to evaluate the effects of climate change on the future of food security. With the help of the Sandbox's processing power, it is building models to predict droughts, floods, cyclones, and other climate-related natural disasters that may threaten the livelihoods of small farmers in developing economies. With the insights the organization gathers, it can help farmers and policymakers prepare for these shock events to reduce the impact on the food system.

"Initially, the big compute job was the main constraint," King says. "Now, we can implement this [data] in more of a monitoring system way, making it available in season to see if there are any potential disruptions to economic activity that could have follow-on effects for food availability—a sort of early warning system. If all of a sudden something gets really disrupted in the wheat-producing area of India, for example, we'd be able to detect that. And because of our on-the-ground research networks, we should be able to start to understand why."

This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.