How Not to Drown in the Big Data Alphabet Soup

October 13, 2015 • Blog Post • By Todd Wasserman, NewsCred


  • Today's IT leaders are often tasked with developing their company's Big Data strategy
  • Joy King from Hewlett Packard Enterprise's Big Data Platform business shares best practices for creating a Big Data ecosystem that meets your company's needs

Choosing the right technologies based on business needs, not hype

Hadoop, R, Spark, Storm, Kafka, SQL, NoSQL, MapReduce, Hive, PIG. This list shows that Big Data doesn't lack unique names and terminology. To an outsider, it can almost seem like such monikers are designed to alienate, but to an insider there's a more pressing issue. New Big Data technologies come along so often and with so much hype that it's hard to determine which ones merit your attention.

The problem is compounded by the reality of the IT professional's evolving job these days. Thanks to the cloud, there has been a shift in the responsibilities and expectation of the role away from just maintaining infrastructure. Instead, the modern CIO or CTO is often tasked with developing a Big Data strategy, acting as head of procurement for technologies and working closely with business teams to determine their feasibility and the technical viability of pursuing new lines of business.

That means that more time is spent scrutinizing emerging technologies. Since the tech press tends to go bonkers for every shiny new object - and big marketing budgets only fuel the hype - how do you separate the sizzle from the steak?

Joy King, VP of Product Marketing & Field Engagement in Hewlett Packard Enterprise's Big Data Platform business, has offered the following best practices for IT decision makers who face the growing, and potentially befuddling, Big Data lexicon.

1. Define your business goals first to determine your tech needs. Every bit of data needs a different "pipe." For example, Kafka is an amazing tool for streaming data, so if your business strategy centers on streaming data, then it's important. If not, it's not worth your attention, no matter now trendy it is.

2. Be aware of the bias toward the new. Staying informed about the latest technologies is the new social currency. Since content is needed to fuel social media buzz, new technologies get a disproportionate amount of attention before securing credibility. It's more fun to talk about the latest thing - especially if it has an interesting name - than something that's been around for a decade. In the end, new is cool but enterprise security and stability are mandatory.

3. Take a look at the size of the project's developer community. Many open source projects boast about "thousands" of people contributing to it, but that's a pretty vague statistic. Is it 2,000? 20,000? 200,000? The actual manpower (and sometimes the one or two corporations whose developers are guiding it) behind the community makes a world of difference in its ability to scale the technology quickly and meet enterprise-grade needs.

4. Keep in mind that a lot of hype comes from marketing spend. A new Big Data offering might appear on billboards, banner ads and TV commercials, but how does the technology apply to you? The loudest voice is not necessarily the most valuable to your business.

5. Take cues from your partners. Gauge whether a new technology is worth considering by maintaining an open dialogue with your technology partners and your customers, both internal and external. Building strong relationships and having continuous discussions with these constituents will help IT leaders better assess which Big Data initiatives have the strength to meet your (and your customers') needs.

There are a lot of opportunities in the Big Data market, which means a lot of choices for IT leaders. Developing a strategy means putting together the right pieces - meaning the right open source projects and emerging technologies, along with enterprise services and software - to create a Big Data ecosystem that meets your business needs.