| Internet-Draft | Benchmarking Containerized Infra | September 2022 |
| Sun, et al. | Expires 27 March 2023 | [Page] |
Recently, the Benchmarking Methodology Working Group has extended the laboratory characterization from physical network functions (PNFs) to virtual network functions (VNFs). Considering the network function implementation trend moving from virtual machine-based to container-based, system configurations and deployment scenarios for benchmarking will be partially changed by how the resource allocation and network technologies are specified for containerized VNFs. This draft describes additional considerations for benchmarking network performance when network functions are containerized and performed in general-purpose hardware.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 27 March 2023.¶
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The Benchmarking Methodology Working Group(BMWG) has recently expanded its benchmarking scope from Physical Network Function(PNF) running on a dedicated hardware system to Network Function Virtualization(NFV) infrastructure and Virtualized Network Function(VNF). [RFC8172] described considerations for configuring NFV infrastructure and benchmarking metrics, and [RFC8204] gives guidelines for benchmarking virtual switch which connects VNFs in Open Platform for NFV(OPNFV).¶
Recently NFV infrastructure has evolved to include a lightweight virtualized platform called the containerized infrastructure, where network functions are virtualized by using the host operating system (OS) virtualization instead of hardware virtualization in virtual machine (VM)-based infrastructure based on the hypervisor. In comparison to VMs, containers do not have a separate hardware and kernel. Containerized virtual network functions (C-VNF) share the same kernel space on the same host, while their resources are logically isolated in different namespaces. Considering this architecture difference between container-based and virtual-machine based NFV systems, containerized NFV network performance benchmarking might have different System Under Test(SUT) and Device Under Test(DUT) configurations compared with both black-box benchmarking and VM-based NFV infrastructure as described in [RFC8172].¶
In terms of networking, to route traffic between containers which are isolated in different network namespaces, virtual ethernet (vETH) interface pairs are used to create a tunnel to Linux bridge or virtual switch (vSwitch) instead of TAP virtual networking device in VM case. Based on the location of the C-VNF (at baremetal or inside a VM), apart from container-to-container scenarios described in [ETSI-TST-009], additional SUT's deployment scenarios should be considered. Besides, containerized network performance is also affected by multiple different packet acceleration techniques which have been applied recently in containerized infrastructure to achieve high throughput and line-rate transmission speed. Each kind of acceleration technique has different deployment location and usage of vSwitch, which is an important aspect of the NFV infrastructure as stated in [RFC8204]. Therefore, different networking models considerations based on the usage characteristic of vSwitch in containerized infrastructure should be noticed while benchmarking containerized network performance.¶
This draft aims to provide additional considerations as specifications to guide containerized infrastructure benchmarking compared with the previous benchmarking methodology of common NFV infrastructure. These considerations include additional deployment scenarios, configuration parameters, investigation of multiple networking models based on the usage of vSwitch in different packet acceleration techniques, and investigation of several deployment configurations that might impact on containerized network performance such as resource isolation, hugepages, service function chaining. The benchmark experiences of these mentioned considerations are also presented in this draft as references. Note that, although the detailed configurations of both infrastructures differ, the new benchmarks and metrics defined in [RFC8172] and [RFC8204] can be equally applied in containerized infrastructure from a generic-NFV point of view, and therefore defining additional evaluation metrics or methodologies are out of scope.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document is to be interpreted as described in [RFC2119]. This document uses the terminology described in [RFC8172], [RFC8204], [ETSI-TST-009].¶
For benchmarking of the containerized infrastructure, as mentioned in [RFC8172], the basic approach is to reuse existing benchmarking methods developed within the BMWG. Various network function specifications defined in BMWG should still be applied to containerized VNF(C-VNF)s for the performance comparison with physical network functions and VM-based VNFs. A major distinction of the containerized infrastructure from the VM-based infrastructure is the absence of a hypervisor. Without hypervisor, all C- VNFs share the same host and kernel space. Storage, computing, and networking resources are logically isolated between containers via different namespaces. Networking between containers is provided by different kind of Container Network Plugins (CNI). There are multiple kinds of CNI, however, these plugins share the same responsibility of creating virtual networking tunnel between containers and host namespaces via vETH pairs. One end of the vETH pair is attached to containers, and the other end is attached to the existing vSwitch in the host node. These architectural differences bring additional considerations when benchmarking network performance in containerized infrastructure¶
In a common containerized infrastructure, thanks to the proliferation of Kubernetes, the pod is defined as a basic unit for orchestration and management that can host multiple containers. Based on that, [ETSI-TST-009] defined two test scenario for container infrastructure as follows.¶
o Container2Container: Communication between containers running in the same pod. it can be done by shared volumes or Inter-process communication (IPC).¶
o Pod2Pod: Communication between containers running in the different pods.¶
As mentioned in [RFC8204], vSwitch is also an important aspect of the containerized infrastructure. For Pod2Pod communication, every pod has only one virtual Ethernet (vETH) interface. This interface is connected to the vSwitch via vETH pair for each container. Not only Pod2Pod but also Pod2External scenario that communicates with an external node is also required. In this case, vSwitch SHOULD support gateway and Network Address Translation (NAT) functionalities.¶
Figure 1 shows briefly differences of network architectures based on container deployment models. Basically, on bare metal, C-VNFs can be deployed as a cluster called POD by Kubernetes. Otherwise, each C-VNF can be deployed separately using Docker. In the former case, there is only one external network interface, even a POD containing more than one C-VNF. An additional deployment model considers a scenario where C-VNFs or PODs are running on VM. In our draft, we define new terminologies; BMP, which is Pod on bare metal, and VMP, which is Pod on VM.¶
+---------------------------------------------------------------------+
| Baremetal Node |
| +--------------+ +--------------+ +-------------- + +-------------+ |
| | | | POD | | VM | | VM | |
| | | |+------------+| |+-------------+| | +-------+ | |
| | C-VNF(A) | || C-VNFs(B) || || C-VNFs(C) || | |PODs(D)| | |
| | | |+------------+| |+-----^-------+| | +---^---+ | |
| | | | | | | | | | | |
| | +------+ | | +------+ | | +--v---+ | | +---v--+ | |
| +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ |
| +--^---+ +---^--+ +--^---+ +---^--+ |
| | | | | |
| | | +--v---+ +---v--+ |
| +------|-----------------|------------|vhost |---------|vhost |---+ |
| | | | +--^---+ +---^--+ | |
| | | | | | | |
| | +--v---+ +---v--+ +--v---+ +---v--+ | |
| | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | |
| | | +--^---+ +---^--+ +--^---+ +---^--+ | | |
| | | | | vSwitch | | | | |
| | | +--|-----------------|---------------|-----------------|--+ | | |
| | +-| | | Bridge | | |-+ | |
| | +--|-----------------|---------------|-----------------|--+ | |
| | | +---------+ | +--|-----------------|---+ | |
| | | |Container| | | | Hypervisor | | | |
| | | | Engine | | | | | | | |
| | | +---------+ | +--|-----------------|---+ | |
| | | | Host Kernel | | | |
| +------|-----------------|---------------|-----------------|------+ |
| +--v-----------------v---------------v-----------------v--+ |
+-----| physical network |-----+
+---------------------------------------------------------+
In [ETSI-TST-009], they described data plane test scenarios in a single host. In that document, there are two scenarios for containerized infrastructure; Container2Container, which is internal communication between two containers in the same Pod, and the Pod2Pod model, which is communication between two containers running in different Pods. According to our new terminologies, we can call the Pod2Pod model the BMP2BMP scenario. When we consider container running on VM as an additional deployment option, there can be more single host test scenarios as follows;¶
o BMP2VMP scenario¶
+---------------------------------------------------------------------+
| HOST +-----------------------------+ |
| |VM +-------------------+ | |
| | | C-VNF | | |
| +--------------------+ | | +--------------+ | | |
| | C-VNF | | | | Logical Port | | | |
| | +--------------+ | | +-+--^-------^---+--+ | |
| | | Logical Port | | | +----|-------|---+ | |
| +-+--^-------^---+---+ | | Logical Port | | |
| | | +---+----^-------^---+--------+ |
| | | | | |
| +----v-------|----------------------------|-------v-------------+ |
| | l----------------------------l | |
| | Data Plane Networking | |
| | (Kernel or User space) | |
| +----^--------------------------------------------^-------------+ |
| | | |
| +----v------+ +----v------+ |
| | Phy Port | | Phy Port | |
| +-----------+ +-----------+
+-------^--------------------------------------------^----------------+
| |
+-------v--------------------------------------------v----------------+
| |
| Traffic Generator |
| |
+---------------------------------------------------------------------+
o VMP2VMP scenario¶
+---------------------------------------------------------------------+
| HOST |
| +-----------------------------+ +-----------------------------+ |
| |VM +-------------------+ | |VM +-------------------+ | |
| | | C-VNF | | | | C-VNF | | |
| | | +--------------+ | | | | +--------------+ | | |
| | | | Logical Port | | | | | | Logical Port | | | |
| | +-+--^-------^---+--+ | | +-+--^-------^---+--+ | |
| | +----|-------|---+ | | +----|-------|---+ | |
| | | Logical Port | | | | Logical Port | | |
| +---+----^-------^---+--------+ +---+----^-------^---+--------+ |
| | | | | |
| +--------v-------v------------------------|-------v-------------+ |
| | l------------------------l | |
| | Data Plane Networking | |
| | (Kernel or User space) | |
| +----^--------------------------------------------^-------------+ |
| | | |
| +----v------+ +----v------+ |
| | Phy Port | | Phy Port | |
| +-----------+ +-----------+ |
+-------^--------------------------------------------^----------------+
| |
+-------v--------------------------------------------v----------------+
| |
| Traffic Generator |
| |
+---------------------------------------------------------------------+
Configuration parameter settings play a crucial role in ensuring the consistency and repeatability of benchmarking results. Most of the configuration parameters for containerized network performance benchmarking are the same with those are listed in [RFC8172] and [RFC8204]. This section lists some additional parameters for containerized infrastructure:¶
o Selected Container Runtime o Selected Container Network Plugin o Selected packet acceleration networking model o Container Network Plugin o Number of C-VNF o Memory, NUMA allocation to C-VNF¶
Container networking services are provided as network plugins. Basically, by using them, network services are deployed as an isolation environment from container runtime through the host namespace, creating a virtual interface, allocating interface and IP address to C-VNF. Since the containerized infrastructure has different network architecture depending on its using plugins, it is necessary to specify the plugin used in the infrastructure. Especially for Kubernetes infrastructure, several Container Networking Interface (CNI) plugins are developed, which describes network configuration files in JSON format, and plugins are instantiated as new namespaces. When the CNI plugin is initiated, it pushes forwarding rules and networking policies to the existing vSwitch (i.e., Linux bridge, Open vSwitch) or creates its own switch functions to provide networking service.¶
The container network model can be classified according to the location of the vSwitch component.¶
+------------------------------------------------------------------+
| User Space |
| +-----------+ +-----------+ |
| | C-VNF | | C-VNF | |
| | +-------+ | | +-------+ | |
| +-| eth |-+ +-| eth |-+ |
| +---^---+ +---^---+ |
| | | |
| | +----------------------------------+ | |
| | | | | |
| | | Networking Controller / Agent | | |
| | | | | |
| | +-----------------^^---------------+ | |
----------|-----------------------||---------------------|----------
| +---v---+ || +---v---+ |
| +--| veth |-------------------vv-----------------| veth |--+ |
| | +-------+ vSwitch Component +-------+ | |
| | (OVS Kernel Datapath, Linux Bridge, ..) | |
| | | |
| +-------------------------------^----------------------------+ |
| | |
| Kernel Space +-----------v----------+ |
+----------------------| NIC |--------------------+
+----------------------+
Figure 4 shows kernel-space vSwitch model. In this model, because the vSwitch component is running on kernel space, data packets should be processed in-network stack of host kernel before transferring packets to the C-VNF running in user-space. Not only pod2External but also pod2pod traffic should be processed in the kernel space. For dynamic networking configuration, the Forwarding policy can be pushed by the controller/agent located in the user-space. In the case of Open vSwitch (OVS) [OVS], the first packet of flow can be sent to the user space agent (ovs-switchd) for forwarding decision. Kernel-space vSwitch models are listed below;¶
o Docker Network[Docker-network], Flannel Network[Flannel], OVS(OpenvSwitch)[OVS], OVN(Open Virtual Network)[OVN]¶
+------------------------------------------------------------------+
| User Space |
| +---------------+ +---------------+ |
| | C-VNF | | C-VNF | |
| | +-----------+ | +-----------------+ | +-----------+ | |
| | |virtio-user| | | Networking | | |virtio-user|-| |
| +-| / eth |-+ | Controller/Agent| +-| / eth |-+ |
| +-----^-----+ +-------^^--------+ +-----^-----+ |
| | || | |
| | || | |
| +-----v-----+ || +-----v-----+ |
| | vhost-user| || | vhost-user| |
| +--| / memif |--------------vv--------------| / memif |--+ |
| | +-----------+ +-----------+ | |
| | vSwitch | |
| | +--------------+ | |
| +----------------------| PMD Driver |----------------------+ |
| | | |
| +-------^------+ |
----------------------------------|---------------------------------
| | |
| | |
| | |
| Kernel Space +----------V-----------+ |
+----------------------| NIC |--------------------+
+----------------------+
Figure 5 shows user-space vSwitch model, in which data packets from physical network port are bypassed kernel processing and delivered directly to the vSwitch running on user-space. This model is commonly considered as Data Plane Acceleration (DPA) technology since it can achieve high-rate packet processing than a kernel-space network with limited packet throughput. For bypassing kernel and directly transferring the packet to vSwitch, Data Plane Development Kit (DPDK) is essentially required. With DPDK, an additional driver called Pull-Mode Driver (PMD) is created on vSwtich. PMD driver must be created for each NIC separately. Userspace CNI [userspace-cni] is required to create user-space acceleration container networking. User-space vSwitch models are listed below;¶
+------------------------------------------------------------------+
| User Space |
| +----------------+ +----------------+ |
| | C-VNF | | C-VNF | |
| | +------------+ | | +------------+ | |
| +-| veth |-+ +-| veth |-+ |
| +-----^------+ +------^-----+ |
| | | |
-------------|---------------------------------------|--------------
| +-----v-------+ +-----v-------+ |
| | +------+ | | +------+ | |
| | | eBPF | | | | eBPF | | |
| | +------+ | | +------+ | |
| | veth tc hook| | veth tc hook| |
| | / | | / | |
| | AFXDP | | AFXDP | |
| +-----^-------+ +------^------+ |
| | | |
| | +-------------------------------+ | |
| | | | | |
| | | Networking Stack | | |
| | | | | |
| | +-------------------------------+ | |
| | | |
| +-------|---------------------------------------|--------+ |
| | +-----|------+ +----|-------+| |
| | | +--v---+ | | +-v----+ || |
| | | | eBPF | | NIC Driver | | eBPF | || |
| | | +------+ | | +------+ || |
| | | XDP hook | | XDP hook || |
| | +------------+ +------------+| |
| +---------------------------^----------------------------+ |
| | |
| Kernel Space +--------v--------+ |
+-----------------------| NIC |------------------------+
+-----------------+
Figure 6 shows eBPF Acceleration model, which leverages extended Berkeley Packet Filter (eBPF) technology [eBPF] to achieve high-performance packet processing. It enables execution of sandboxed programs inside abstract virtual machines within the Linux kernel without changing the kernel source code or loading the kernel module. To accelerate data plane performance, eBPF programs are attached to different BPF hooks inside the linux kernel stack.¶
One type of BPF hook is the eXpress Data Path (XDP) at the networking driver. It is the first hook that triggers eBPF program upon packet reception from external network. The other type of BPF hook is Traffic Control Ingress/Egress eBPF hook (tc eBPF) at the vETH interface. The eBPF program running at the tc hook enforce policy on all traffic exit the pod, while the eBPF program running at the XDP hook enforce policy on all traffic coming from NIC.¶
On the egress datapath side, whenever a packet exits the pod, it first goes through the pod's vETH interface. Then, the destination that received the packet depends on the chosen CNI plugin that is used to create container networking. If the chosen CNI plugin is an eBPF-based CNI, the packet is received by the eBPF program running at vETH interface tc hook. If the chosen CNI plugin is an AFXDP-supported CNI, the packet is received by the AFXDP socket [AFXDP]. AFXDP socket is a new Linux socket type which allows a fast packet delivery tunnel between itself and the XDP hook at the networking driver. This tunnel bypasses the network stack in kernel space to provide high-performance raw packet networking. Packets are transmitted between user space and AFXDP socket via a shared memory buffer. Once the egress packet arrived at the AFXDP socket or tc hook, it is directly forwarded to the NIC.¶
On the ingress datapath side, eBPF programs at the XDP hook pick up packets from the network device and directly deliver it to the vETH interface pair or AFXDP socket, bypass all of the kernel network layer processing such as iptables. In case of Cilium CNI [Cilium], context-switching process to the pod network namespace can also be bypassed.¶
Notable eBPF Acceleration models are: Calico[Calico], Cilium[Cilium] (eBPF supported CNI), and Cloud Native Data Plane[CNDP] - (AFXDP supported CNI project). Using Userspace CNI with an AFXDP supporter user-space vSwitch such as OVS-DPDK or VPP is also a deployment option of AFXDP eBPF acceleration networking model. In the case of Cilium, eBPF/XDP program can be offloaded directly on the smart NIC card, which allows data plane acceleration without using the CPU. Container network performance of Cilium project is reported in [cilium-benchmark], while AFXDP performance and comparison against DPDK are reported in [intel-AFXDP] and [LPC18-DPDK-AFXDP], respectively.¶
+------------------------------------------------------------------+
| User Space |
| +-----------------+ +-----------------+ |
| | C-VNF | | C-VNF | |
| | +-------------+ | | +-------------+ | |
| +-| vf driver |-+ +-| vf driver |-+ |
| +-----^-------+ +------^------+ |
| | | |
-------------|---------------------------------------|--------------
| +---------+ +---------+ |
| +------|-------------------|------+ |
| | +----v-----+ +-----v----+ | |
| | | virtual | | virtual | | |
| | | function | | function | | |
| Kernel Space | +----^-----+ NIC +-----^----+ | |
+---------------| | | |----------------+
| +----v-------------------v----+ |
| | Classify and Queue | |
| +-----------------------------+ |
+---------------------------------+
Figure 7 shows Smart-NIC acceleration model, which does not use vSwitch component. This model can be separated into two technologies.¶
One is Single-Root I/O Virtualization (SR-IOV), which is an extension of PCIe specifications to enable multiple partitions running simultaneously within a system to share PCIe devices. In the NIC, there are virtual replicas of PCI functions known as virtual functions (VF), and each of them is directly connected to each container's network interfaces. Using SR-IOV, data packets from external bypass both kernel and user space and are directly forwarded to container's virtual network interface. SRIOV network device plugin for Kubernetes[SR-IOV] is recommended to create an SRIOV-based container networking.¶
The other technology is eBPF/XDP programs offloading to Smart-NIC card as mentioned in the previous section. It enables general acceleration of eBPF. eBPF programs are attached to XDP and run at the Smart-NIC card, which allows server CPUs to perform more application-level work. However, not all Smart-NIC cards provide eBPF/XDP offloading support.¶
+-------------------------------------------------------+
| User Space |
| +--------------------+ +--------------------+ |
| | C-VNF | | C-VNF | |
| | +------+ +------+ | | +------+ +------+ | |
| +-| veth |--| veth |-+ +-| veth |--| veth |-+ |
| +---^--+ +---^--+ +--^---+ +---^--+ |
| | | | | |
| | | | | |
| | +---v--------+ +-------v----+ | |
| | | vhost-user | | vhost-user | | |
| | +--| / memif |--| / memif |--+ | |
| | | +------------+ +------------+ | | |
| | | vSwitch | | |
| | +----------------------------------+ | |
| | | |
--------|----------------------------------------|-------
| +-----------+ +-------------+ |
| +----|--------------|---+ |
| |+---v--+ +---v--+| |
| || vf | | vf || |
| |+------+ +------+| |
| Kernel Space | | |
+--------------| NIC |----------------+
+-----------------------+
Figure 8 shows the networking model when combining user-space vSwitch model and Smart-NIC acceleration model. This model is frequently considered in service function chain scenarios when two different types of traffic flows are present. These two types are North/South traffic and East/West traffic.¶
North/South traffic is the type that packets are received from other servers and routed through VNF. For this traffic type, Smart-NIC model such as SR-IOV is preferred because packets always have to pass the NIC. User-space vSwitch involvement in north-south traffic will create more bottlenecks. On the other hand, East/West traffic is a form of sending and receiving data between containers deployed in the same server and can pass through multiple containers. For this type, user-space vSwitch models such as OVS-DPDK and VPP are preferred because packets are routed within the user space only and not through the NIC.¶
The throughput advantages of these different networking models with different traffic direction cases are reported in [Intel-SRIOV-NFV].¶
CPU pinning enables benefits such as maximizing cache utilization, eliminating operating system thread scheduling overhead as well as coordinating network I/O by guaranteeing resources. This technology is very effective in avoiding the "noisy neighbor" problem, and it is already proved in existing experience [Intel-EPA].¶
Using NUMA, performance will be increasing not CPU and memory but also network since that network interface connected PCIe slot of specific NUMA node have locality. Using NUMA requires a strong understanding of VNF's memory requirements. If VNF uses more memory than a single NUMA node contains, the overhead will occurr due to being spilled to another NUMA node. Network performance can be changed depending on the location of the NUMA node whether it is the same NUMA node where the physical network interface and CNF are attached to. There is benchmarking experience for cross-NUMA performance impacts [ViNePERF]. In that tests, they consist of cross-NUMA performance with 3 scenarios depending on the location of the traffic generator and traffic endpoint. As the results, it was verified as below:¶
o A single NUMA Node serving multiple interfaces is worse than Cross-NUMA Node performance degradation¶
o Worse performance with VNF sharing CPUs across NUMA¶
Hugepage configures a large page size of memory to reduce Translation Lookaside Buffer(TLB) miss rate and increase the application performance. This increases the performance of logical/virtual to physical address lookups performed by a CPU's memory management unit, and overall system performance. In the containerized infrastructure, the container is isolated at the application level, and administrators can set huge pages more granular level (e.g., Kubernetes allows to use of 512M bytes huge pages for the container as default values). Moreover, this page is dedicated to the application but another process, so the application uses the page more efficiently way. From a network benchmark point of view, however, the impact on general packet processing can be relatively negligible, and it may be necessary to consider the application level to measure the impact together. In the case of using the DPDK application, as reported in [Intel-EPA], it was verified to improve network performance because packet handling processes are running in the application together.¶
When we consider benchmarking for containerized and VM-based infrastructure and network functions, benchmarking scenarios may contain various operational use cases. Traditional black-box benchmarking focuses on measuring the in-out performance of packets from physical network ports since the hardware is tightly coupled with its function and only a single function is running on its dedicated hardware. However, in the NFV environment, the physical network port commonly will be connected to multiple VNFs(i.e., Multiple PVP test setup architectures were described in [ETSI-TST-009]) rather than dedicated to a single VNF. This scenario is called Service Function Chaining. Therefore, benchmarking scenarios should reflect operational considerations such as the number of VNFs or network services defined by a set of VNFs in a single host. [service-density] proposed a way for measuring the performance of multiple NFV service instances at a varied service density on a single host, which is one example of these operational benchmarking aspects. Another aspect in benchmarking service function chaining scenario should be considered is different network acceleration technologies. Network performance differences may occur because of different traffic patterns based on the provided acceleration method.¶
Apart from the single-host test scenario, the multi-hosts scenario should also be considered in container network benchmarking, where container services are deployed across different servers. To provide network connectivity for container-based VNFs between different server nodes, inter-node networking is required. According to [ETSI-NFV-IFA-038], there are several technologies to enable inter-node network: overlay technologies using a tunnel endpoint (e.g. VXLAN, IP in IP), routing using Border Gateway Protocol (BGP), layer 2 underlay, direct network using dedicated NIC for each pod, or load balancer using LoadBalancer service type in Kubernetes. Different protocols from these technologies may cause performance differences in container networking.¶
Benchmarking activities as described in this memo are limited to technology characterization of a Device Under Test/System Under Test (DUT/SUT) using controlled stimuli in a laboratory environment with dedicated address space and the constraints specified in the sections above.¶
The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network.¶
Further, benchmarking is performed on a "black-box" basis and relies solely on measurements observable external to the DUT/SUT.¶
Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks.¶
In this test, our purpose is to test the performance of user-space based model for container infrastructure and figure out the relationship between resource allocation and network performance. With respect to this, we set up Contiv-VPP, one of the user-space based network solutions in container infrastructure and tested like below.¶
o Three physical server for benchmarking¶
+-------------------+----------------------+--------------------------+ | Node Name | Specification | Description | +-------------------+----------------------+--------------------------+ | Conatiner Control |- Intel(R) Xeon(R) | Container Deployment | | for Master | CPU E5-2690 | and Network Allocation | | | (2Socket X 12Core) |- ubuntu 18.04 | | |- MEM 128G |- Kubernetes Master | | |- DISK 2T |- CNI Conterller | | |- Control plane : 1G |.. Contive VPP Controller | | | |.. Contive VPP Agent | +-------------------+----------------------+--------------------------+ | Conatiner Service |- Intel(R) Xeon(R) | Container Service | | for Worker | Gold 6148 |- ubuntu 18.04 | | | (2socket X 20Core) |- Kubernetes Worker | | |- MEM 128G |- CNI Agent | | |- DISK 2T |.. Contive VPP Agent | | |- Control plane : 1G | | | |- Data plane : MLX 10G| | | | (1NIC 2PORT) | | +-------------------+----------------------+--------------------------+ | Packet Generator |- Intel(R) Xeon(R) | Packet Generator | | | CPU E5-2690 |- CentOS 7 | | | (2Socket X 12Core) |- installed Trex 2.4 | | |- MEM 128G | | | |- DISK 2T | | | |- Control plane : 1G | | | |- Data plane : MLX 10G| | | | (1NIC 2PORT) | | +-------------------+----------------------+--------------------------+
o The architecture of benchmarking¶
+----+ +--------------------------------------------------------+
| | | Containerized Infrastructure Master Node |
| | | +-----------+ |
| <-------> 1G PORT 0 | |
| | | +-----------+ |
| | +--------------------------------------------------------+
| |
| | +--------------------------------------------------------+
| | | Containerized Infrastructure Worker Node |
| | | +---------------------------------+ |
| s | | +-----------+ | +------------+ +------------+ | |
| w <-------> 1G PORT 0 | | | 10G PORT 0 | | 10G PORT 1 | | |
| i | | +-----------+ | +------^-----+ +------^-----+ | |
| t | | +--------|----------------|-------+ |
| c | +-----------------------------|----------------|---------+
| h | | |
| | +-----------------------------|----------------|---------+
| | | Packet Generator Node | | |
| | | +--------|----------------|-------+ |
| | | +-----------+ | +------v-----+ +------v-----+ | |
| <-------> 1G PORT 0 | | | 10G PORT 0 | | 10G PORT 1 | | |
| | | +-----------+ | +------------+ +------------+ | |
| | | +---------------------------------+ |
| | | |
+----+ +--------------------------------------------------------+
o Network model of Containerized Infrastructure(User space Model)¶
+---------------------------------------------+---------------------+
| NUMA 0 | NUMA 0 |
+---------------------------------------------|---------------------+
| Containerized Infrastructure Worker Node | |
| +---------------------------+ | +----------------+ |
| | POD1 | | | POD2 | |
| | +-------------+ | | | +-------+ | |
| | | | | | | | | | |
| | +--v---+ +---v--+ | | | +-v--+ +-v--+ | |
| | | eth1 | | eth2 | | | | |eth1| |eth2| | |
| | +--^---+ +---^--+ | | | +-^--+ +-^--+ | |
| +------|-------------|------+ | +---|-------|----+ |
| +--- | | | | |
| | +-------|---------------|------+ | |
| | | | +------|--------------+ |
| +----------|--------|-------|--------|----+ | |
| | v v v v | | |
| | +-tap10--tap11-+ +-tap20--tap21-+ | | |
| | | ^ ^ | | ^ ^ | | | |
| | | | VRF1 | | | | VRF2 | | | | |
| | +--|--------|--+ +--|--------|--+ | | |
| | | +-----+ | +---+ | | |
| | +-tap01--|--|-------------|----|---+ | | |
| | | +------v--v-+ VRF0 +----v----v-+ | | | |
| | +-| 10G ETH0/0|------| 10G ETH0/1|-+ | | |
| | +---^-------+ +-------^---+ | | |
| | +---v-------+ +-------v---+ | | |
| +---| DPDK PMD0 |------| DPDK PMD1 |------+ | |
| +---^-------+ +-------^---+ | User Space |
+---------|----------------------|------------|---------------------+
| +-----|----------------------|-----+ | Kernal Space |
+---| +---V----+ +----v---+ |------|---------------------+
| | PORT 0 | 10G NIC | PORT 1 | | |
| +---^----+ +----^---+ |
+-----|----------------------|-----+
+-----|----------------------|-----+
+---| +---V----+ +----v---+ |----------------------------+
| | | PORT 0 | 10G NIC | PORT 1 | | Packet Generator (Trex) |
| | +--------+ +--------+ | |
| +----------------------------------+ |
+-------------------------------------------------------------------+
We set up a Contive-VPP network to benchmark the user space container network model in the containerized infrastructure worker node. We set up network interface at NUMA0, and we created different network subnets VRF1, VRF2 to classify input and output data traffic, respectively. And then, we assigned two interfaces which connected to VRF1, VRF2 and, we setup routing table to route Trex packet from eth1 interface to eth2 interface in POD.¶
In this environment, we confirmed that the routing table doesn't work when we send packets using Trex packet generator. The reason is that when kernel space based network configured, ip forwarding rule is processed to kernel stack level while 'ip packet forwarding rule' is processed only in vrf0, which is the default virtual routing and forwarding (VRF0) in VPP. The above testing architecture makes problem since vrf1 and vrf2 interface couldn't route packet. According to above result, we assigned vrf0 and vrf1 to POD and, data flow is like below.¶
+---------------------------------------------+---------------------+
| NUMA 0 | NUMA 0 |
+---------------------------------------------|---------------------+
| Containerized Infrastructure Worker Node | |
| +---------------------------+ | +----------------+ |
| | POD1 | | | POD2 | |
| | +-------------+ | | | +-------+ | |
| | +--v----+ +---v--+ | | | +-v--+ +-v--+ | |
| | | eth1 | | eth2 | | | | |eth1| |eth2| | |
| | +--^---+ +---^--+ | | | +-^--+ +-^--+ | |
| +------|-------------|------+ | +---|-------|----+ |
| +-------+ | | | | |
| | +-------------|---------------|------+ | |
| | | | +------|--------------+ |
| +-----|-------|-------------|--------|----+ | |
| | | | v v | | |
| | | | +-tap10--tap11-+ | | |
| | | | | ^ ^ | | | |
| | | | | | VRF1 | | | | |
| | | | +--|--------|--+ | | |
| | | | | +---+ | | |
| | +-*tap00--*tap01----------|----|---+ | | |
| | | +-V-------v-+ VRF0 +----v----v-+ | | | |
| | +-| 10G ETH0/0|------| 10G ETH0/1|-+ | | |
| | +-----^-----+ +------^----+ | | |
| | +-----v-----+ +------v----+ | | |
| +---|*DPDK PMD0 |------|*DPDK PMD1 |------+ | |
| +-----^-----+ +------^----+ | User Space |
+-----------|-------------------|-------------|---------------------+
v v
*- CPU pinning interface
We conducted benchmarking with three conditions. The test environments are as follows. - Basic VPP switch - General kubernetes (No CPU Pining) - Shared Mode / Exclusive mode. In the basic Kubernetes environment, all PODs share a host's CPU. Shared mode is that some POD share a pool of CPU assigned to specific PODs. Exclusive mode is that a specific POD dedicates a specific CPU to use. In shared mode, we assigned two CPUs for several PODs, in exclusive mode, we dedicated one CPU for one POD, independently. The result is like Figure 13. First, the test was conducted to figure out the line rate of the VPP switch, and the basic Kubernetes performance. After that, we applied NUMA to the network interface using Shared Mode and Exclusive Mode in the same node and different node. In Exclusive and Shared mode tests, we confirmed that Exclusive mode showed better performance than Shared mode when same NUMA CPU was assigned, respectively. However, we confirmed that performance is reduced at the section between the vpp switch and the POD, affecting the total result.¶
+--------------------+---------------------+-------------+
| Model | NUMA Mode (pinning)| Result(Gbps)|
+--------------------+---------------------+-------------+
| | N/A | 3.1 |
| Maximum Line Rate |---------------------+-------------+
| | same NUMA | 9.8 |
+--------------------+---------------------+-------------+
| Without CMK | N/A | 1.5 |
+--------------------+---------------------+-------------+
| | same NUMA | 4.7 |
| CMK-Exclusive Mode +---------------------+-------------+
| | Different NUMA | 3.1 |
+--------------------+---------------------+-------------+
| | same NUMA | 3.5 |
| CMK-shared Mode +---------------------+-------------+
| | Different NUMA | 2.3 |
+--------------------+---------------------+-------------+
In this test, our purpose is to test the performance of Smart-NIC acceleration model for container infrastructure and figure out relationship between resource allocation and network performance. With respect to this, we setup SRIOV combining with DPDK to bypass the Kernel space in container infrastructure and tested based on that.¶
o Three physical server for benchmarking¶
+-------------------+-------------------------+------------------------+ | Node Name | Specification | Description | +-------------------+-------------------------+------------------------+ | Conatiner Control |- Intel(R) Core(TM) | Container Deployment | | for Master | i5-6200U CPU | and Network Allocation | | | (1socket x 4Core) |- ubuntu 18.04 | | |- MEM 8G |- Kubernetes Master | | |- DISK 500GB |- CNI Conterller | | |- Control plane : 1G | MULTUS CNI | | | | SRIOV plugin with DPDK| +-------------------+-------------------------+------------------------+ | Conatiner Service |- Intel(R) Xeon(R) | Container Service | | for Worker | E5-2620 v3 @ 2.4Ghz |- Centos 7.7 | | | (1socket X 6Core) |- Kubernetes Worker | | |- MEM 128G |- CNI Agent | | |- DISK 2T | MULTUS CNI | | |- Control plane : 1G | SRIOV plugin with DPDK| | |- Data plane : XL710-qda2| | | | (1NIC 2PORT- 40Gb) | | +-------------------+-------------------------+------------------------+ | Packet Generator |- Intel(R) Xeon(R) | Packet Generator | | | Gold 6148 @ 2.4Ghz |- CentOS 7.7 | | | (2Socket X 20Core) |- installed Trex 2.4 | | |- MEM 128G | | | |- DISK 2T | | | |- Control plane : 1G | | | |- Data plane : XL710-qda2| | | | (1NIC 2PORT- 40Gb) | | +-------------------+-------------------------+------------------------+
o The architecture of benchmarking¶
+----+ +--------------------------------------------------------+
| | | Containerized Infrastructure Master Node |
| | | +-----------+ |
| <-------> 1G PORT 0 | |
| | | +-----------+ |
| | +--------------------------------------------------------+
| |
| | +--------------------------------------------------------+
| | | Containerized Infrastructure Worker Node |
| | | +---------------------------------+ |
| s | | +-----------+ | +------------+ +------------+ | |
| w <-------> 1G PORT 0 | | | 40G PORT 0 | | 40G PORT 1 | | |
| i | | +-----------+ | +------^-----+ +------^-----+ | |
| t | | +--------|----------------|-------+ |
| c | +-----------------------------|----------------|---------+
| h | | |
| | +-----------------------------|----------------|---------+
| | | Packet Generator Node | | |
| | | +--------|----------------|-------+ |
| | | +-----------+ | +------v-----+ +------v-----+ | |
| <-------> 1G PORT 0 | | | 40G PORT 0 | | 40G PORT 1 | | |
| | | +-----------+ | +------------+ +------------+ | |
| | | +---------------------------------+ |
| | | |
+----+ +--------------------------------------------------------+
o Network model of Containerized Infrastructure(User space Model)¶
+---------------------------------------------+---------------------+
| CMK shared core | CMK exclusive core |
+---------------------------------------------|---------------------+
| Containerized Infrastructure Worker Node | |
| +---------------------------+ | +----------------+ |
| | POD1 | | | POD2 | |
| | (testpmd) | | | (testpmd) | |
| | +-------------+ | | | +-------+ | |
| | | | | | | | | | |
| | +--v---+ +---v--+ | | | +-v--+ +-v--+ | |
| | | eth1 | | eth2 | | | | |eth1| |eth2| | |
| | +--^---+ +---^--+ | | | +-^--+ +-^--+ | |
| +------|-------------|------+ | +---|-------|----+ |
| | | | | | |
| +------ +-+ | | | |
| | +----|-----------------|------+ | |
| | | | +--------|--------------+ |
| | | | | | User Space|
+---------|------------|----|--------|--------|---------------------+
| | | | | | |
| +--+ +------| | | | |
| | | | | | Kernal Space|
+------|--------|-----------|--------|--------+---------------------+
| +----|--------|-----------|--------|-----+ | |
| | +--v--+ +--v--+ +--v--+ +--v--+ | | NIC|
| | | VF0 | | VF1 | | VF2 | | VF3 | | | |
| | +--|---+ +|----+ +----|+ +-|---+ | | |
| +----|------|---------------|-----|------+ | |
+---| +v------v+ +-v-----v+ |------|---------------------+
| | PORT 0 | 40G NIC | PORT 1 | |
| +---^----+ +----^---+ |
+-----|----------------------|-----+
+-----|----------------------|-----+
+---| +---V----+ +----v---+ |----------------------------+
| | | PORT 0 | 40G NIC | PORT 1 | | Packet Generator (Trex) |
| | +--------+ +--------+ | |
| +----------------------------------+ |
+-------------------------------------------------------------------+
We set up a Multus CNI, SRIOV CNI with DPDK to benchmark the user-space container network model in the containerized infrastructure worker node. The Multus CNI support creates multiple interfaces for a container. The traffic is bypassed the Kernel space by SRIOV with DPDK. We established two modes of CMK: shared core and exclusive core. We created VFs for each network interface of a container. Then, we set up TREX to route packet from eth1 to eth2 in a POD.¶
Figure 17 shows the test results when using 1518 bytes packet traffic from the T-Rex traffic generator. First, we get the maximum line rate of the system using SR-IOV as the packet acceleration technique. Then we measured throughput when applying the CMK feature. We observed similar results as VPP CPU Pinning test. The default Kubernetes system without CMK feature enabled had the worst performance as the CPU resources are shared without any isolation. When the CMK feature is enabled, Exclusive Mode performed better than Shared Mode because each pod had its own dedicated CPU.¶
+--------------------+-------------+
| Model | Result(Gbps)|
+--------------------+-------------+
| Maximum Line Rate | 39.3 |
+--------------------+-------------+
| Without CMK | 11.5 |
+--------------------+-------------+
| CMK-Exclusive Mode | 39.2 |
+--------------------+-------------+
| CMK-shared Mode | 29.6 |
+--------------------+-------------+
The main goal of this experience was to benchmark the multi-pod scenario, in which packets are traversed through two pods. To create additional interfaces for forwarding packets between two pods, Multus CNI was used. We compared two userspace-vSwitch model network technologies: OVS/DPDK and VPP-memif. Since that vpp-memif has a different packet forwarding mechanism by using shared memory interface, it is expected that vpp-memif may provide higher performance that OVS-DPDK. Also, we consider NUMA impact for both cases, and made 6 scenarios depending on CPU location of vSwitch and two pods. Figure 18 is packet forwarding scenario in this test, where two pods run on the same host and vSwitch delivers packets between two pods.¶
+----------------------------------------------------------------+
|Worker Node |
| +--------------------------------------------------------+ |
| |Kubernetes | |
| | +--------------+ +--------------+ | |
| | | pod1 | | pod2 | | |
| | | +--------+ | | +--------+ | | |
| | | | L2FWD | | | | L2FWD | | | |
| | | +---^--v-+ | | +--^--v--+ | | |
| | | | DPDK | | | | DPDK | | | |
| | | +---^--v-+ | | +--^--v--+ | | |
| | +------^--v----+ +-----^--v-----+ | |
| | ^ v ^ v | |
| | +------^--v>>>>>>>>>>>>>>>>>>>>>>>>>>>^--v-----+ | |
| | | ^ OVS-DPDK / VPP-memif vSwitch v | | |
| | +------^---------------------------------v-----+ | |
| | | ^ PMD Driver v | | |
| | +------^---------------------------------v-----+ | |
| | ^ v | |
| +----------^---------------------------------v-----------+ |
| ^ v |
| +----------^---------------------------------v---------+ |
| | ^ 40G NIC v | |
| | +------^-------+ +--------v-----+ | |
+---|---| Port 0 |----------------| Port 1 |---|-----+
| +------^-------+ +--------v-----+ |
+----------^---------------------------------v---------+
+------^-------+ +--------v-----+
+-------| Port 0 |----------------| Port 1 |---------+
| +------^-------+ +--------v-----+ |
| Traffic Generator (TRex) |
| |
+----------------------------------------------------------------+
+-------------------+-------------------------+------------------------+ | Node Name | Specification | Description | +-------------------+-------------------------+------------------------+ | Conatiner Control |- Intel(R) Core(TM) | Container Deployment | | for Master | E5-2620v3 @ 2.40GHz | and Network Allocation | | | (1socket x 12Cores) |- ubuntu 18.04 | | |- MEM 32GB |- Kubernetes Master | | |- DISK 1TB |- CNI Controller | | |- NIC: Control plane: 1G | - MULTUS CNI | | |- OS: CentOS Linux7.9 | - DPDK-OVS/VPP-memif | +-------------------+-------------------------+------------------------+ | Conatiner Service |- Intel(R) Xeon(R) |- Container dpdk-L2fwd | | for Worker | Gold 6148 @ 2.40GHz |- Kubernetes Worker | | | (2socket X 40Cores) |- CNI Agent | | |- MEM 256GB | - Multus CNI | | |- DISK 2TB | - DPDK-OVS/VPP-memif | | |- NIC | | | | - Control plane: 1G | | | | - Data plane: XL710-qda2| | | | (1NIC 2PORT- 40Gb) | | | |- OS: CentOS Linux 7.9 | | +-------------------+-------------------------+------------------------+ | Packet Generator |- Intel(R) Xeon(R) | Packet Generator | | | Gold 6148 @ 2.4Ghz |- Installed Trex v2.92 | | | (2Socket X 40Core) | | | |- MEM 256GB | | | |- DISK 2TB | | | |- NIC | | | | - Data plane: XL710-qda2| | | | (1NIC 2PORT - 40Gb) | | | |- OS: CentOS Lunix 7.9 | | +-------------------+-------------------------+------------------------+
For installations and configurations of CNIs, we used userspace-cni network plugin. Among this CNI, multus provides to create multiple interfaces for each pod. Both OVS-DPDK and VPP-memif bypass kernel with DPDK PMD driver. For CPU isolation and NUMA allocation, we used Intel CMK with exclusive mode. Since Trex generator is upgraded to the new version, we used the latest version of Trex.¶
To analyze benchmarking impacts of different NUMA allocation, we set 6 scenarios depending on CPU location allocating to two pods and vSwich. For this scenario, we did not consider cross-NUMA case, which allocates CPUs to pod or switch in a manner that two cores are located in different NUMA nodes. 6 scenarios we considered are listed in Table 1. Note that, NIC is attached to the NUMA1.¶
| Scenario # | vSwtich | pod1 | pod2 |
|---|---|---|---|
| S1 | NUMA1 | NUMA0 | NUMA0 |
| S2 | NUMA1 | NUMA1 | NUMA1 |
| S3 | NUMA0 | NUMA0 | NUMA0 |
| S4 | NUMA0 | NUMA1 | NUMA1 |
| S5 | NUMA1 | NUMA1 | NUMA0 |
| S6 | NUMA0 | NUMA0 | NUMA1 |
For multi-pod benchmarking, we discovered Non Drop Rate (NDR) with binary search algorithm. In Trex, it supports command to discover NDR for each testing. Also, we test for different ethernet frame sizes from 64bytes to 1518bytes. For running Trex, we used command as follows;¶
./ndr --stl --port 0 1 -v --profile stl/bench.py --prof-tun size=x --opt-bin-search¶
As the benchmarking results, Table 2 shows packet loss ratio using 1518 bytes packet in OVS-DPDK/vpp-memif. From that result, we can say that the vpp-memif has better performance that OVS-DPDK, which is came from the difference in the way to forward packets between vswitch and pod. Also, the impact of NUMA is bigger when vswitch and both pods are located in the same node than when allocating CPU to the node where NIC is attached.¶
| Networking Model | S1 | S2 | S3 | S4 | S5 | S6 |
|---|---|---|---|---|---|---|
| OVS-DPDK | 21.29 | 13.17 | 6.32 | 19.76 | 12.43 | 6.38 |
| vpp-memif | 59.96 | 34.17 | 45.13 | 57.1 | 33.47 | 44.92 |
Added new Section 4. Benchmarking Considerations. Previous Section 4. Networking Models in Containerized Infrastructure was moved into this new Section 4 as a subsection¶
Re-organized Additional Deployment Scenarios for containerized network benchmarking contents from Section 3. Containerized Infrastructure Overview to new Section 4. Benchmarking Considerations as the Addtional Deployment Scenarios subsection¶
Added new Addtional Configuration Parameters subsection to new Section 4. Benchmarking Considerations¶
Moved previous Section 5. Performance Impacts into new Section 4. Benchmarking Considerations as the Deployment settings impact on network performance section¶
Updated eBPF Acceleration Model with AFXDP deployment option¶
Enhanced Abstract and Introduction's description about the draft's motivation and contribution.¶
Added Change Log¶
Added eBPF Acceleration Model in Section 4. Networking Models in Containerized Infrastructure¶
Added Model Combination in Section 4. Networking Models in Containerized Infrastructure¶
Added Service Function Chaining in Section 5. Performance Impacts¶
Added Troubleshooting and Results for SRIOV-DPDK Benchmarking Experience¶
Added Benchmarking Experience of Multi-pod Test¶
Removed Section 3. Benchmarking Considerations, Removed Section 4. Benchmarking Scenarios for the Containerized Infrastructure¶
Added new Section 3. Containerized Infrastructure Overview, Added new Section 4. Networking Models in Containerized Infrastructure. Added new Section 5. Performance Impacts¶
Re-organized Subsection Comparison with the VM-based Infrastructure of previous Section 3. Benchmarking Considerations and previous Section 4.Benchmarking Scenarios for the Containerized Infrastructure to new Section 3. Containerized Infrastructure Overview¶
Re-organized Subsection Container Networking Classification of previous Section 3. Benchmarking Considerations to new Section 4. Networking Models in Containerized Infrastructure. Kernel-space vSwitch models and User-space vSwitch models were presented as seperate subsections in this new Section 4.¶
Re-organized Subsection Resource Considerations of previous Section 3. Benchmarking Considerations to new Section 5. Performance Impacts as 2 seperate subsections CPU Isolation / NUMA Affinity and Hugepages. Previous Section 5. Additional Considerations was moved into this new Section 5 as the Additional Considerations subsection.¶
Moved Benchmarking Experience contents to Appendix¶
Added Benchmarking Experience of SRIOV-DPDK.¶
Added Benchmarking Experience of Contiv-VPP.¶
Editorial changes only.¶
Editorial changes only.¶
Added Container Networking Classification in Section 3.Benchmarking Considerations (Kernel Space network model and User Space network model).¶
Added Resource Considerations in Section 3.Benchmarking Considerations(Hugepage, NUMA, RX/TX Multiple-Queue).¶
Renamed Section 4.Test Scenarios to Benchmarking Scenarios for the Containerized Infrastructure, added 2 additional scenarios BMP2VMP and VMP2VMP.¶
Added Additional Consideration as new Section 5.¶