I’m excited to finally have the opportunity to play with Juniper’s vMX! Since it was announced last year I’ve been eagerly waiting for release – a couple of client projects already have passed by where the vMX would have been a perfect fit. vMX already won an award earlier this year at Interop Tokyo 2015!
In this post I’ll be giving a bit of background on the vMX architecture and licensing, and then go on to walk through a lab based configuration of vMX.
The vMX is a virtual MX Series Router that is optimized to run as software on x86 servers. Like other MX routers, it runs Junos, and Trio has been compiled for x86! Yes, that means the sophisticated L2, L2.5 and L3 forwarding features we are used to on the MX are present on the vMX.
Architecture
vMX can be installed on server hardware of your choice, so long as it is x86 based and running Linux (although I’m sure a version to run on vmware won’t be too far away).
vMX itself actually consists of two separate VMs – a virtual forwarding plane (VFP) running the vTrio, and a virtual control plane (VCP) running Junos.
The Linux virtualisation solution KVM is what Juniper are using to spin up the virtual instances of the control and forwarding planes, and multiple instances of vMX can be run on the same hardware. To see Juniper using Linux and KVM is no surprise as this is what we are used to on Juniper’s other products such as the QFX.
The VMs are managed by a simple orchestration script which is used to create, stop and start the vMX instances. A simple configuration file defines parameters such as memory and vCPUs to allocate to the VCP and VFP.
A couple of Linux bridges are created by the orchestration script. Clearly VCP and VFP need to be able to communicate directly so an “internal” bridge is automatically created for each vMX instance to enable this communication. An “external” bridge is also created, this is used to enable the management interface on the Linux Physical host to be used for the virtual management interfaces on the VCP and VFP.
For data interfaces, there are a couple of techniques available for packet I/O depending on the required vMX throughput –
- Paravirtualisation using KVMs virtio drivers
- PCI passthrough using single root I/O virtualisation (SR-IOV), enabling packets to bypass the hypervisor and therefore increase I/O.
Juniper recommend virtio or SR-IOV up to 3Gbps, and SR-IOV over 3Gbps (using a minimum of 2 x 10GE interfaces).
Which you will choose will ultimately depend on your use case for the vMX.
Licensing
Now this is what I really like about vMX! Licensing is based on a combination of throughput and features, and the lowest available throughput license is 100Mbps! Yes – you don’t need to be shifting multi-Gigabits of traffic to start with vMX. You can start small and pay-as-you-grow with vMX.
Below 1Gbps there are only 3 options – 100Mbps, 250Mbps and 500Mbps. Full scale features are included! List price on the 100Mbps option is a very reasonable $750.
At 1Gbps and above, licences are a combination of features (Base, Advance, and Premium) and full duplex throughput (1G, 5G, 10G, 40G)
- Base – IP routing with 32,000 routes in the forwarding table. Basic Layer 2 functionality, Layer 2 bridging and switching.
- Advance – Features in the BASE application package IP routing with routes up to platform scale in the forwarding table. IP and MPLS switching for unicast and multicast applications. Layer 2 features include Layer 2 VPN, VPLS, EVPN, and Layer 2 Circuit
VXLAN.
- Premium – Features in the BASE and ADVANCE application packages. Layer 3 VPN for IP and multicast
Setting up vMX on Ubuntu
Now I’m going to walk through setting up vMX on Ubuntu 14.04 LTS server (Juniper’s recommended flavour of Linux for vMX). Just for fun this is actually running as a nested Vmware VM on my Macbook Pro – fine for a lab, but don’t try this in production! 🙂 I have allocated 8GB RAM, 4 vCPUs and two vNICs to the Ubuntu VM. Also the VM is enabled to support hypervisor applications within the VM.
At this point Ubuntu Server has been freshly installed, and the option to install virtualisation was selected during setup.
First things first, let’s update all packages, install the prerequsite packages and restart the system
mdinham@ubuntu:~$ sudo apt-get upgrade
<snip>
mdinham@ubuntu:~$ sudo apt-get install bridge-utils qemu-kvm libvirt-bin python python-netifaces vnc4server libyaml-dev python-yaml numactl libparted0-dev libpciaccess-dev libnuma-dev libyajl-dev libxml2-dev libglib2.0-dev libnl-dev python-pip python-dev libxml2-dev libxslt-dev
<snip>
mdinham@ubuntu:~$ sudo reboot
Configuring vMX
As this is a lab based build, I will be using virtio for the virtual NIC. There are two options on the VFP – a “Lite” version PFE for labs and performance version for normal operation.
Note: Ubuntu 14.04 provides libvirt 1.2.2 which works for VFP lite version. However for the VFP performance version you must upgrade to libvirt 1.2.8.
Let’s extract the vMX application bundle and get going!
mdinham@ubuntu:~$ tar xzf vmx-14.1R5.4-1.tgz
mdinham@ubuntu:~$ cd vmx-14.1R5.4-1/
mdinham@ubuntu:~/vmx-14.1R5.4-1$ ls
config drivers env images scripts vmx.sh
First of all we need to setup the vmx config file, this is done by editing config/vmx.conf
First of all I set an instance name for vmx, and set the correct vmx images. I’m using vPFE-lite.
---
#Configuration on the host side - management interface, VM images etc.
HOST:
identifier : vmx1 # Maximum 4 characters
host-management-interface : eth0
routing-engine-image : "/home/mdinham/vmx-14.1R5.4-1/images/jinstall64-vmx-14.1R5.4-domestic.img"
routing-engine-hdd : "/home/mdinham/vmx-14.1R5.4-1/images/vmxhdd.img"
forwarding-engine-image : "/home/mdinham/vmx-14.1R5.4-1/images/vPFE-lite-20150707.img"
Now the parameters the control plane and forwarding plane.
I’ve allocated 1 vCPU to vRE and 3 vCPU to vPFE. 1GB RAM to the RE and 6GB to the forwarding plane, as per the defaults for 14.1
UPDATE: Feb 2016
For vMX on 15.1, allocate 1 vCPU to vRE and 3 vCPU to vPFE. 2GB RAM to the RE and 8GB to the forwarding plane.
I have also tried vMX with 2GB allocated to the vPFE and the forwarding plane loaded, which could be fine for lab purposes. I’d expect 1GB to be the minimum on the vRE. 3 x vCPU seems to be the minimum for the vPFE.
Note that device-type is set to “virtio” for the interfaces.
---
#External bridge configuration
BRIDGES:
- type : external
name : br-ext # Max 10 characters
---
#vRE VM parameters
CONTROL_PLANE:
vcpus : 1
memory-mb : 2048
console_port: 8601
interfaces :
- type : static
ipaddr : 10.102.144.94
macaddr : "0A:00:DD:C0:DE:0E"
---
#vPFE VM parameters
FORWARDING_PLANE:
memory-mb : 6144
vcpus : 3
console_port: 8602
device-type : virtio
interfaces :
- type : static
ipaddr : 10.102.144.98
macaddr : "0A:00:DD:C0:DE:10"
---
#Interfaces
JUNOS_DEVICES:
- interface : ge-0/0/0
mac-address : "02:06:0A:0E:FF:F0"
description : "ge-0/0/0 interface"
I will only be using one interface in this lab, but up to 10 can be configured. For SR-IOV, things are done slightly differently – see this vMX doc for reference.
I now need to deploy the vMX instance using the orchestration script. “-lv” provides verbose logging. My vMX instance will be created by the script and automatically started.
mdinham@ubuntu:~/vmx-14.1R5.4-1$ sudo ./vmx.sh -lv --install
==================================================
Welcome to VMX
==================================================
Date..............................................07/18/15 13:19:03
VMX Identifier....................................vmx1
Config file......................................./home/mdinham/vmx-14.1R5.4-1/config/vmx.conf
Build Directory.................................../home/mdinham/vmx-14.1R5.4-1/build/vmx1
Environment file................................../home/mdinham/vmx-14.1R5.4-1/env/ubuntu_virtio.env
Junos Device Type.................................virtio
Initialize scripts................................[OK]
Copy images to build directory....................[OK]
==================================================
VMX Environment Setup Completed
==================================================
==================================================
VMX Install & Start
==================================================
Linux distribution................................ubuntu
Check GRUB........................................[Disabled]
Installation status of qemu-kvm...................[OK]
Installation status of libvirt-bin................[OK]
Installation status of bridge-utils...............[OK]
Installation status of python.....................[OK]
Installation status of libyaml-dev................[OK]
Installation status of python-yaml................[OK]
Installation status of numactl....................[OK]
Installation status of libnuma-dev................[OK]
Installation status of libparted0-dev.............[OK]
Installation status of libpciaccess-dev...........[OK]
Installation status of libyajl-dev................[OK]
Installation status of libxml2-dev................[OK]
Installation status of libglib2.0-dev.............[OK]
Installation status of libnl-dev..................[OK]
Check Kernel Version..............................[Disabled]
Check Qemu Version................................[Disabled]
Check libvirt Version.............................[Disabled]
Check virsh connectivity..........................[OK]
IXGBE Enabled.....................................[Disabled]
==================================================
Pre-Install Checks Completed
==================================================
Check for VM vcp-vmx1.............................[Not Running]
Check for VM vfp-vmx1.............................[Not Running]
Cleanup VM states.................................[OK]
Check if bridge br-ext exists.....................[No]
Cleanup VM bridge br-ext..........................[OK]
Cleanup VM bridge br-int-vmx1.....................[OK]
==================================================
VMX Stop Completed
==================================================
Check VCP image...................................[OK]
Check VFP image...................................[OK]
VMX Model.........................................Lite
Check VCP Config image............................[OK]
Check management interface........................[OK]
Setup huge pages to 8192..........................[OK]
Attempt to kill libvirt...........................[OK]
Attempt to start libvirt..........................[OK]
Sleep 2 secs......................................[OK]
Check libvirt support for hugepages...............[OK]
==================================================
System Setup Completed
==================================================
Get Management Address of eth0....................[OK]
Generate libvirt files............................[OK]
Sleep 2 secs......................................[OK]
Find configured management interface..............eth0
Find existing management gateway..................eth0
Check if eth0 is already enslaved to br-ext.......[No]
Gateway interface needs change....................[Yes]
Create br-ext.....................................[OK]
Get Management Gateway............................192.168.100.254
Flush eth0........................................[OK]
Start br-ext......................................[OK]
Bind eth0 to br-ext...............................[OK]
Get Management MAC................................00:0c:29:76:a8:15
Assign Management MAC 00:0c:29:76:a8:15...........[OK]
Add default gw 192.168.100.254....................[OK]
Create br-int-vmx1................................[OK]
Start br-int-vmx1.................................[OK]
Check and start default bridge....................[OK]
Define vcp-vmx1...................................[OK]
Define vfp-vmx1...................................[OK]
Wait 2 secs.......................................[OK]
Start vcp-vmx1....................................[OK]
Start vfp-vmx1....................................[OK]
Wait 2 secs.......................................[OK]
==================================================
VMX Bringup Completed
==================================================
Check if br-ext is created........................[Created]
Check if br-int-vmx1 is created...................[Created]
Check if VM vcp-vmx1 is running...................[Running]
Check if VM vfp-vmx1 is running...................[Running]
Check if tap interface vcp_ext-vmx1 exists........[OK]
Check if tap interface vcp_int-vmx1 exists........[OK]
Check if tap interface vfp_ext-vmx1 exists........[OK]
Check if tap interface vfp_int-vmx1 exists........[OK]
==================================================
VMX Status Verification Completed.
==================================================
Log file..........................................
/home/mdinham/vmx-14.1R5.4-1/build/vmx1/logs/vmx_1437221943.log
==================================================
Thankyou for using VMX
==================================================
Connecting to the console port on the VMs
We can now connect to the vMX control plane! This is done using the vmx.sh script again.
Specify vcp (control plane – Junos) or vcf (vPFE) and the instance name.
mdinham@ubuntu:~/vmx-14.1R5.4-1$ ./vmx.sh --console vcp vmx1
--
Login Console Port For vcp-vmx1 - 8601
Press Ctrl-] to exit anytime
--
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Amnesiac (ttyd0)
login:
After a while, the FPC and interfaces will come online
root> show interfaces terse | match ge-0/0/0
ge-0/0/0 up up
root> show chassis fpc
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online Absent 100 0 512 14 0
I’ll go ahead and add an IP address to ge-0/0/0. Note: if I was using the management interface I could configure interface FXP0 also now. Remember FXP0 will be bridged to the host eth0 adapter (or an adapter you specify).
root# set interfaces ge-0/0/0.0 family inet address 192.168.100.5/24
Can I ping anything?
root> ping 192.168.100.5
PING 192.168.100.5 (192.168.100.5): 56 data bytes
64 bytes from 192.168.100.5: icmp_seq=0 ttl=64 time=0.059 ms
^C
--- 192.168.100.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.059/0.059/0.059/0.000 ms
root> ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254): 56 data bytes
^C
--- 192.168.100.254 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
OK, so I can ping the interface but nothing else on the host. As I’m using virtio I need to create a device binding between the host physical NIC and the vMX interface.
Creating a virtio binding
This is done in the config file config/vmx-junosdev.conf.
virtio bindings are flexible and can be used to map multiple vMX instances to the same physical host interface, or to connect vMX instances together.
A new Linux bridge will be created between host interface eth1 and ge-0/0/0 on vmx1.
##############################################################
#
# vmx-junos-dev.conf
# - Config file for junos device bindings.
# - Uses YAML syntax.
# - Leave a space after ":" to specify the parameter value.
# - For physical NIC, set the 'type' as 'host_dev'
# - For junos devices, set the 'type' as 'junos_dev' and
# set the mandatory parameter 'vm-name' to the name of
# the vPFE where the device exists
# - For bridge devices, set the 'type' as 'bridge_dev'
#
##############################################################
interfaces :
- link_name : vmx_link
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/0
endpoint_2 :
- type : host_dev
dev_name : eth1
If eth1 is not already up on the Linux host, bring it up
sudo ifconfig eth1 up
Again the orchestration script vmx.sh is used to create the device bindings
mdinham@ubuntu:~/vmx-14.1R5.4-1$ sudo ./vmx.sh --bind-dev
Bind Link vmx_link(ge-0.0.0-vmx1, eth1)...........[OK]
And we can see a new bridge has been created called “vmx_link” as referenced in the bindings configuration file
mdinham@ubuntu:~/vmx-14.1R5.4-1$ brctl show
bridge name bridge id STP enabled interfaces
br-ext 8000.000c2976a815 yes br-ext-nic
eth0
vcp_ext-vmx1
vfp_ext-vmx1
br-int-vmx1 8000.52540050c859 yes br-int-vmx1-nic
vcp_int-vmx1
vfp_int-vmx1
virbr0 8000.fe060a0efff1 yes ge-0.0.1-vmx1
ge-0.0.2-vmx1
ge-0.0.3-vmx1
vmx_link 8000.000c2976a81f no eth1
ge-0.0.0-vmx1
Now to retry that ping!
mdinham@ubuntu:~/vmx-14.1R5.4-1$ ./vmx.sh --console vcp vmx1
--
Login Console Port For vcp-vmx1 - 8601
Press Ctrl-] to exit anytime
--
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
root> ping 192.168.100.254
PING 192.168.100.254 (192.168.100.254): 56 data bytes
64 bytes from 192.168.100.254: icmp_seq=0 ttl=64 time=4.951 ms
64 bytes from 192.168.100.254: icmp_seq=1 ttl=64 time=2.081 ms
^C
--- 192.168.100.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.081/3.516/4.951/1.435 ms
Success! At this point I’ve a working vMX with an interface mapped to a NIC on the Ubuntu host. What happens if I turn on OSPF and LDP?
root> show ospf neighbor
Address Interface State ID Pri Dead
192.168.100.254 ge-0/0/0.0 Full 10.0.0.2 1 37
192.168.100.1 ge-0/0/0.0 Full 10.0.0.1 128 39
root> show ldp neighbor
Address Interface Label space ID Hold time
192.168.100.254 ge-0/0/0.0 10.0.0.2:0 13
Excellent, now the fun can really begin, but I’ll save that for another time!
vPFE
One last thing – what does the VFP look like?
mdinham@ubuntu:~/vmx-14.1R5.4-1$ sudo ./vmx.sh --console vfp vmx1
--
Login Console Port For vfp-vmx1 - 8602
Press Ctrl-] to exit anytime
--
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Wind River Linux 6.0.0.12 localhost console
localhost login: root
Password:
Last login: Sat Jul 18 12:20:49 UTC 2015 on console
The riot process is where all the magic happens!
top - 12:52:55 up 33 min, 1 user, load average: 0.38, 0.74, 0.62
Tasks: 102 total, 1 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 2.1%sy, 0.0%ni, 96.8%id, 0.0%wa, 0.0%hi, 0.6%si, 0.0%st
Mem: 5824060k total, 4454308k used, 1369752k free, 12184k buffers
Swap: 0k total, 0k used, 0k free, 44552k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1019 root 20 0 36.6g 37m 10m S 12 0.7 4:25.04 riot
<snip>
Further reference
I hope you enjoyed this vMX post! For further reference on any of the above material please see the Juniper Release Notes for vMX