Skip to main content

7 lessons to take away from a failed big data project

When a big data initiative fails to meet its goals, all is not lost. These guidelines teach you how to examine past mistakes and spot important clues that lead to future success.

According to Gartner analyst Nick Heudecker, nearly 85 percent of big data projects fail. That's an awful number, but don't lose hope. Even when a big data initiative crumbles into ashes, IT can take away valuable lessons that help everyone achieve success the next time around.

"Most big data initiatives never live up to their expected potential because they aren’t scoped properly," explains Leon Morales, managing director of DNA Behavior International, a behavioral sciences company that helps businesses make informed decisions. Among the common reasons why most big data initiatives never go beyond the experimentation stage and are eventually abandoned are people, technologies, overly optimistic expectations, and poor planning, he says: "They typically start out with a good idea, but then end up as an extra project for someone who is already busy."

Terminating or freezing a big data initiative that failed to meet its goals is never a happy experience. Yet, there's also a bright side. Here are seven lessons you can take away from a failed initiative to ensure that your next big data venture will be successful:

1. Understand what went wrong

Begin by performing an after-action review, Morales advises. "This is where every member of the team is allowed to debrief what went well, what didn’t go well, and what should have not been done at all," he notes. Documenting specific actions helps team members identify the root cause of failure.

"This is a discipline that I put into place in every work environment I’ve worked in," Morales says. "We learn how to respect our differences, but also how to get projects into the performance stage."

2. Do a gut-check on leadership skills

All enterprise data programs run into challenges. When it gets tough, leaders need to support and champion the initiative. "Every member of the team is critical to the overall success," Morales notes. He recalls a $50 million big data initiative focused on a call center. "It had sponsorship and funding, but what it didn't have was a steering committee that was committed to seeing the program through until the end," he says. 

A year earlier, Morales had worked on another $50 million program. That one met all of its deliverables, on time and on budget. "The difference was a strong steering committee and having crucial conversations with team members to remove blockages," he reports.

3. Ensure staff buy-in

Even if a big data project has complete management-level support, failing to secure lower-level employees' acceptance and cooperation can doom a project just as effectively.

"A lot of projects get a huge push back from staff who will be affected…because they fear their jobs might be taken away," explains Andrew Pearson, managing director of Intelligencia, a software consulting firm that serves the casino, hospitality, entertainment, e-sports, and sports betting industries. Education and training are critical to allaying employee concerns. "If employees are shown that their jobs are safe and might actually become more meaningful—because they can spend their days analyzing data for true business value rather than just wrangling the data—they will become allies rather than detractors," Pearson says.

Need help with your artificial intelligence deep learning strategy? We have a Dummies Guide for that.

4. Align the project with corporate objectives

Big data projects can be confusing and challenging to implement. For that reason, precise alignment with major enterprise objectives is essential to the success of any big data project. "That’s what's so often missing from the planning process," observes Briana Brownell, founder and CEO of Pure Strategy, a provider of AI-based decision-support technology.

"A project might not impact something meaningful to the business, like retaining customers or driving sales," Brownell notes. "When that happens, it’s doomed to either fail or be shelved, because it distracts from more important projects." Aligning a big data project with corporate objectives also means that employees don’t need permission to act on the results. "They already know it’s in line with the goals of their specific role," she says.

5. Use high-quality data

Data that's inaccurate, poorly formatted, or obtained from dodgy sources can kill even the most carefully planned big data project. Standardized data from verifiable sources is key to the success of any big data initiative.

Pearson offers some context: "We work with a casino in a country where anyone who wants to visit a casino has to provide a federally issued ID. This makes creating a single customer view much easier than for some of our [other] clients, who are using IDs from multiple countries." He notes that the clean data from a single trusted source allows him to help clients obtain "highly accurate figures on customer spend, which means we can get reliable figures on things like customer lifetime value, which is a necessary component for proper marketing."

6. Look for better software

The analytics software market is evolving and improving rapidly, so if your big data project failed because it wasn't providing meaningful or accurate insights, it may be a good idea to see what other software choices are available.

"On one particular big data project, I came to realize that part of the problem was that the software we had sold was old and wasn't able to do what we needed it to do," Pearson recalls. "I discovered the company also had other software products that would actually be a better fit, so we changed course. A short while later, we were able to deliver what we had been contracted to do."

7. Decide whether to continue or start over

Should a failed project be tossed out or redesigned and improved? It depends, Morales says. "If the technology isn’t proven, then it could be worth scratching," he says. On the other hand, if the people who are leading, planning, and building the initiative are skilled and capable of learning from their past mistakes, it might make sense to attempt resurrecting a stuck project.

Morales says big data projects can be narrowed down to three core areas: people, process, and technology. "This helps to isolate where the blockages are and how to get the blockages remediated or unblocked," he notes. Getting a stalled project moving again sometimes demands a different level of thinking and open communication. "Never underestimate the power of connecting with an individual," Morales says. "Usually, there is a golden nugget to get the program moving by respecting those behavioral differences with individuals."

A final takeaway

All of these ideas are easy to implement if you have a leader who truly respects each team member's behavioral differences, Morales observes. "Relationship versus results is how we view the world in our company," he explains. "We need an element of both to be successful, but you don’t necessarily want your project managers to be heavily wired to be relational when it comes to planning."

This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.