HPE Ezmeral Container Platform operates on two networks, as shown here:
The two networks are laid out as follows:
- Network for the Controller, Worker, and Gateway hosts: This network must be both routable and part of the organization's network that is managed by that organization's IT department.
- Network for virtual nodes (Docker containers):HPE Ezmeral Container Platform creates and manages this network, which can be either public (routable) or private (non-routable). For Kubernetes, Canal is used as the Container Network (CN) Network Provider. The container network is typically private (non-routable) instead of public (routable). See Public (Routable) Virtual Node Network and Private (Non-Routable) Virtual Node Network, below.
Private (Non-Routable) Virtual Node Network
Private, non-routable virtual node networks keep the virtual node IP addresses private and hidden within the private network. As described in Gateway Hosts and Load Balancing, the Gateway hosts proxy service endpoint ports. IP masquerading replaces the IP addresses of outgoing and incoming packets when the containers are in a private, non-routable network, as shown here.
HPE Ezmeral Container Platform is deployed on a set of hosts. Each host has an IP address and FQDN such as Host IP1, Host IP2, etc. Hosts are typically deployed as one or more racks of servers that are connected to an external switch for access to other subnets in the organization (such as end user network etc.). See Multiple Subnets, below.
HPE Ezmeral Container Platform provisions clusters of embedded, fully-managed Docker containers. Each cluster spins up within a tenant and receives distinct assigned IP addresses and FQDNs from a user-provided IP range, which appear in the diagram above as IP1, IP2, etc. Kubernetes supports network isolation via network policies.
End-user access to services in the containers (such as SSH or web applications) is routed through a Gateway host that runs the HAProxy service. This access is purely for control traffic. All other traffic, including access to remote HDFS or other enterprise systems such as Active Directory (AD), MIT KDC (Kerberos provider), SSO (Identity providers), and Certificate Authority (CA), is performed via the host network interface masquerading, as opposed to the Gateway host port proxying.
Public (Routable) Virtual Node Network
Non-Kubernetes network traffic in a public HPE Ezmeral Container Platform virtual node network flows as shown here.
The above diagram depicts a deployment where a routable IP range is used for containers. Unlike private non-routable virtual node network configuration, all of the hosts must be in the same subnet. All the key functionality is identical to the recommended approach for a non-routable private IP range (see Private (Non-Routable) Virtual Node Network). This approach allows the Docker containers to directly access the external network via the network interface connector (NIC) on the hosts where they reside.
Multiple Subnets
HPE Ezmeral Container Platform can be deployed across multiple subnets as long as it is configured to use private non-routable virtual network (see Private (Non-Routable) Virtual Node Network). This diagram displays a sample deployment using multiple subnets.
When multiple subnets are used:
- The hosts may be located on-premises and/or in a public cloud. For example, hosts can reside on multiple racks and/or can be virtual machines residing on cloud-based services (such as AWS, Azure, or GCP).
- If the deployment includes cloud-based hosts, then the container network must be private and non-routable.
- If the Controller and Worker hosts are on different subnets, then the path MTU settings must be the same for both subnets.
- If the Controller and Worker hosts are on a single subnet and the Gateway hosts are on the different subnet, then the subnet with the Gateway hosts may use an MTU setting that is lower than or equal to the MTU setting on the other subnet with no further action needed. If the MTU of the Gateway host is larger, then it must be at least 1,000 bytes larger than the MTU setting of the other subnet.
- The subnets used by Gateway hosts may have different path MTU settings, subject to the above.
- If the Controller and Shadow Controller are on one subnet, the Worker hosts on a second subnet, and the Gateway hosts on a third subnet, then each of the hosts must have the same path MTU setting.