Data pipelines

What are data pipelines?

Data pipelines are used to move data from a source to a destination, such as a data lake or data warehouse

What are the components of a data pipeline?

A data pipeline is comprised of three steps: a data source, a data processing or data transformation step and a data destination or data storage location. The data source is where the data comes from. Common data sources include databases, CRM systems, IoT sensors and more. The data processing or data transformation step includes all of the operations that change the data, including transportation, translation, sorting, consolidation, deduplication, validation and analysis. The final step of a data pipeline, data storage, is where the transformed data is stored so that users are able to access it. Typical data storage locations include data warehouses, data lakes and data marts.

ETL pipelines are considered to be a subcategory of data pipelines. The main difference between an ETL pipeline and a data pipeline is that ETL pipelines can transform data in more ways than a data pipeline can. For example, an ETL pipeline can combine specific metric data to make it easier to analyse. ETL pipelines can also transfer data on a set schedule, such as when network traffic is slower instead of in real time, allowing data to be transferred at regular intervals instead of continuously.

Related HPE Solutions, Products or Services

What are the types of data pipelines?

Real-time pipelines

Real-time pipelines are often used in financial insights industries or enterprises that process data directly through streaming services, such as analytics and weather reporting. This system processes data instantly through an architecture with the capacity to process millions of events at scale, providing incredibly reliable insights.

Open-source pipelines

Open-source pipelines are a budget-friendly system utilised by smaller businesses and the general public to move, process and store data. Tools used to facilitate this type of pipeline are more affordable than those that facilitate real-time or cloud-based data pipeline systems. These systems are open to the public, requiring intentional customisation in all use cases.

Cloud pipelines

Cloud pipelines are aptly named to utilise, transform and analyse cloud-based data. By removing the need for on-site storage infrastructure, organisations can both collect and analyse data within a cloud-based structure. Cloud-native pipelines often include extensive security offerings due to the nature of the service.

Batch processing pipelines

Batch processing pipelines are one of the most popular choices for data pipeline storage systems. Often used to move and store massive amounts of data on a consistent basis, organisations utilise batch processing pipeline systems to translate and move their data to be stored and analysed at a slower rate than real-time systems due to the sheer volume of data being moved.

Streaming pipelines

Streaming pipelines and batch processing pipelines are the two most common forms of data pipeline. Streaming pipelines allow users to ingest both structured and unstructured data from a variety of different data sources.

What is data pipeline architecture?

Data pipeline architecture refers to the systems that connect the data sources, data processing systems, analytical tools and applications.

Data pipeline architecture ensures that all relevant data is collected, allowing data scientists to draw insights from the data in order to target behaviours, promote efficiency in customer journeys and amplify user experiences. Data pipelines take raw data, route it to an appropriate storage site and transform it into actionable insights. The architecture is dynamically layered, beginning with intake and ending with continual oversight.

Foundationally, raw data involves an array of data points – far too many from which to gain insights. The architecture of data pipelines comprises the system created to capture, structure and move data to draw insights from and analyse for deeper understanding and utilisation. This is often accomplished through automation, software and data storage solutions.

Storage locations are determined depending on the format of the data collected. Sending data to the correct storage location is a critical step in data pipeline architecture, with options for storing mastered data within a structured storage system such as a data warehouse and more loosely structured data within a data lake. Data analysts can gather data insights from loosely structured data within data lakes or analyse mastered data within a central storage location. Without proper placement in a storage environment, there cannot be practical oversight within the architecture, further limiting future applications. 

HPE and data pipelines

HPE Ezmeral is a hybrid analytics and data science platform designed to drive data-first modernisation, enabling enterprises to unlock the value of their data wherever it lives. HPE Ezmeral powers HPE GreenLake analytics services to help customers unify, modernise and analyse all their data across edge to cloud.

HPE Ezmeral helps unlock data’s value and innovate faster with choice, efficiency and flexibility that are not available from niche and cloud-based solutions. It does this by:

Providing a unified software platform built on 100% open source and designed for both cloud-native and non-cloud-native (legacy) applications running on any infrastructure, either on-prem or hybrid and multicloud environments.

Unifying data and modernising apps with the industry’s first integrated data fabric optimised for high-performance analytics. It accelerates time to insights by combining files, objects, event streams and NoSQL databases into a single logical infrastructure and file system to provide global access to synchronised data.

Addressing the challenges associated with operationalising ML models at enterprise scale with a solution that delivers DevOps-like speed and agility, combined with a cloud-like experience that accelerates your workloads.

Delivering a consistent experience across teams with a single platform that leverages a wide range of analytical and ML tools. Built-in automation and a cloud-native experience simplify the process of connecting users and their tools to the right data, compute engines and storage, freeing teams to focus on unlocking data’s value.

Gaining freedom and flexibility with integrated open-source tools and frameworks in a unified hybrid data lakehouse. An integrated app store, or HPE Ezmeral Marketplace, enables the rapid creation of streamlined, customised engines and environments based on full-stack, validated solutions from trusted ISV partners.