Start making sense: Building modern data platforms

The modern enterprise has a data problem. Tremendously successful at capturing and storing data, the challenge is making sense of it, especially taking advantage of AI/ML-powered analytics. The reality is that, despite major investments in capturing and storing data—estimated to be already in excess of a petabyte, or 1 million gigabytes—most of the data collected by enterprises is never used.

This is not hyperbole. Forrester Research estimates that nearly 75 percent of all data collected by enterprises goes unused for analytics. Meanwhile, another study indicates that enterprises don't understand how to use more than 55 percent of the data they collect—or even know with any certainty what they've captured in the first place.

A system that makes access to all of the organization's data easier and logical would bring this lost data out of the shadows and put it to work.

A broken system

So, why is it so difficult for enterprises to turn their data into something of value to their organizations?

A primary reason is that data is so decentralized that getting access to it in a timely fashion is very difficult. In the modern enterprise, data resides just about everywhere: internal data centers, third-party data centers, cloud repositories, and at the edge.

For each new business use case, enterprises tend to build the technologies they need to address that specific challenge. They create a new database, a new dashboard, a new and different data feed system. The result is a hodgepodge of data silos replicated with each use case having its own technology stack, complete with its own data store and management.

Please read: Data deluge: How to get data management strategy in sync

This is a broken system that is already becoming unmanageable and costly to maintain. Not only does it make it a challenge for enterprises to access their data, but it also fails the organization's end users—the employees who need data to do their jobs better.

But what if this data paradigm was flipped on its head? Instead of a decentralized data model, what if the entire data ecosystem—including storage, compute, analytic tools, visualization, and other middleware apps—was coordinated so that users had one unified system to access data, wherever it may reside? That model is the idea behind the modern data platform.

So, what would that look like? And what is the best way to get there?

Architect for the future

The concept of the modern data platform is broader in scope than just a single product. It's not just a database with a website in front of it. A modern data platform is an architecture—an end-to-end, secure, flexible, and unified architecture—and more of a solution that addresses all the data-related challenges enterprises face today.

Please read: A data fabric enables a comprehensive data strategy

Enterprises can architect their data platform for the future in two basic ways. The first is to rip and replace their prior investments in data systems, start over from scratch, and go "big bang" all at once. That path would provide all the components for a modern data platform from Day One.

It is far more likely, however, that an enterprise might prefer to take this process in phases, approaching it step by step, use case by use case. Not in the same way their data system was developed before, with no overarching plan or vision, but with an architectural blueprint to ensure that the new data platform is able to expand to incorporate additional types of data and AI use cases in the future.

If the first use case called for a relational database, for example, the architectural plan would ensure that the platform was capable of storing other types of nonrelational data as well.

Please read: Create a single trusted view of all data, from edge to cloud

Viewed from this holistic perspective, a modern data platform is an engine for any kind or number of data-driven use cases. Most important, it allows enterprises to derive value from their data in a coherent way rather than having to go and find the data they need every single time.

The platform architecture

So, what are the components of a modern data platform? There are basically two major segments: data storage and data processing (compute).

On the data store side, the components or layers might include some kind of data fabric, like Hewlett Packard Enterprise's Ezmeral, to simplify management and ensure consistent operations across all endpoints on-prem, across clouds, and on the edge. This software knitting also serves the purpose of tying everything together so that the platform and everything within it can scale easily.

The second major component on the data storage side consists of the storage mechanisms themselves. Centralizing the data from all of the various types of databases in a single, easily accessible place solves a number of the most difficult data issues. It makes it easy for enterprises to find, transfer, combine, process, or scrub data quickly and efficiently.

But that's only half the story. There is also the data processing/compute side. This layer contains all the tools and technologies for collating, accessing, retrieving, analyzing, reporting, creating new applications and services, and in general, working with the data stored on the platform.

Please read: Report: The state of container adoption

Key to the compute side of the modern data platform is the integration of new container-based technologies like Kubernetes, which allow enterprises to simplify, accelerate, and orchestrate new application development and deployment. A container is a self-contained software package that allows enterprises to test and develop, or "spin up," new network applications or services without interfering with other applications. Kubernetes, a container orchestration platform, revolutionizes data processing by allowing users to work with the data from any number of different sources efficiently and simultaneously.

All data everywhere, from anywhere

For users, this is the front end to the modern data platform. Using just one single interface and dashboard, users can pull the data they need and run their data processing and analytics applications, machine learning, data visualization, and other middleware apps.

A modern data platform offers the enterprise a host of benefits by supplying users with direct access to centralized data, even though that data may be anywhere and anyplace, even in different countries or on different continents.

Please watch: Create a single trusted view of all your data

Enterprises continue to store and collect enormous amounts of data. By accelerating the process of building a modern data platform, they can finally start making sense of their new and stored data in a timely fashion and turn it into significant value for their organizations.

Best practices for enterprise data

Best practices for organizations looking to build a modern data platform include:

• Start with a consultative approach. What are your business goals? What are your business problems? How can you start to use data to solve those problems?
• Think tactically. What's the best use case? What's the low-hanging fruit that can be plucked to give you the best value for the least amount of investment. Prove the value of this investment to your company's stakeholders, and use that to build on and deliver the next use case.
• Architect the future. Design a data platform that will map to your organization's present and future business goals and accommodate multiple types of data.
• Begin building the foundational elements. This will help satisfy the first use case and prepare for the next.
• Create a self-perpetuating cycle of value. Having reinforced the value by solving problems that improve the business, continue to get the funding necessary for the next use case.