Part 1 of this series focussed on the interface MTU configuration, looking how different vendors implement the setting. Some include the layer 2 headers, some don’t. Part 3 will add MPLS MTU to the mix.

This post looks at Jumbo Frames, IP MTU and we’ll also introduce OSPF to the mix. OSPF exchanges IP MTU information in DBD packets when forming a neighbor adjacency and will detect any MTU mismatch, so if the MTU settings are wrong or mismatched, we won’t fully establish the adjacency. Of course, this feature can be turned off, but following the OSPF Version 2 specification (RFC 2328):

If the Interface MTU field in the Database Description packet
indicates an IP datagram size that is larger than the router can
accept on the receiving interface without fragmentation, the
Database Description packet is rejected.

Jumbo Frames

Why would anyone actually want to increase the MTU size beyond 1500? Well, back in the day, larger packets were desirable because they resulted in less overhead on the server – fewer CPU interrupts, fewer CPU cycles wasted etc. Today with Large Segment Offload (LSO) etc the performance increase might not be what you’d expect. Take a look here for more info. As always – implement something that is suitable for your environment, and test before you do.  Anyway, you are not here to debate the use of Jumbo frames or otherwise, so let’s crack on.

IP MTU

As with Ethernet frames, the protocol MTU can be changed for IP packets. The accepted “standard” payload for a Jumbo Frame is 9000 bytes (i.e. an IP MTU set to 9000).

In this post, the interface MTU will be increased to the maximum supported by the interface hardware, but for the purposes of this post and to demonstrate that the interface MTU and IP MTU can be different, we will set the IP MTU to a consistent value of 2000 bytes.

Topology

The same virtual topology will be used as Part 1.

Software revisions are as follows

• IOS (Cisco 7200 12.4(24)T)
• IOS-XE (CSR1000V 15.4(1)S)
• IOS-XR (IOS-XRv 5.1.1)
• Junos (12.3R5.7)
• Junos (Firefly 12.1X46)

IOS/IOS-XE

Our Interface config is below, the interface MTU has been changed to 9216, and the protocol MTU to 2000. From our tests earlier, we know what this means that the maximum IP payload is 2000 bytes, which would result in 2014 bytes being put on the wire including the L2 headers.

interface GigabitEthernet2
 mtu 9216
 ip address 192.168.1.4 255.255.255.0
 ip mtu 2000
 negotiation auto
 cdp enable
end

OK so let’s go ahead and enable OSPF, nothing much to see here

interface Loopback0
 ip address 4.4.4.4 255.255.255.255
 !
 router ospf 1
 network 4.4.4.4 0.0.0.0 area 0
 network 192.168.1.4 0.0.0.0 area 0
 !

I already enabled OSPF on another router, but didn’t change the MTU from 1500. We’re not going to get a full adjacency here but let’s troubleshoot

1000v#show ip ospf neighbor
 Neighbor ID Pri State Dead Time Address Interface
 192.168.1.3 1 INIT/DROTHER 00:00:35 192.168.1.3 GigabitEthernet2

1000v#debug ip ospf adj
 OSPF adjacency debugging is on
 1000v#
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Neighbor change event
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: DR/BDR election
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Elect BDR 192.168.1.3
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Elect DR 192.168.1.4
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: DR: 192.168.1.4 (Id) BDR: 192.168.1.3 (Id)
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Neighbor change event
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: DR/BDR election
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Elect BDR 192.168.1.3
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: Elect DR 192.168.1.4
 *Apr 3 18:31:08.455: OSPF-1 ADJ Gi2: DR: 192.168.1.4 (Id) BDR: 192.168.1.3 (Id)
 *Apr 3 18:31:08.484: OSPF-1 ADJ Gi2: Send DBD to 192.168.1.3 seq 0x341 opt 0x52 flag 0x7 len 32
 *Apr 3 18:31:08.484: OSPF-1 ADJ Gi2: Retransmitting DBD to 192.168.1.3 [2]
 *Apr 3 18:31:09.102: OSPF-1 ADJ Gi2: Rcv DBD from 192.168.1.3 seq 0x26B1 opt 0x52 flag 0x7 len 32 mtu 1500 state EXSTART
 *Apr 3 18:31:09.102: OSPF-1 ADJ Gi2: Nbr 192.168.1.3 has smaller interface MTU

Well that’s pretty clear why there is a problem. Let’s move on to IOS-XR.

IOS-XR

Let’s see what IOS-XR has to say about OSPF & MTU

RP/0/0/CPU0:ios#show ospf neighbor
* Indicates MADJ interface

Neighbors for OSPF 1

Neighbor ID     Pri   State           Dead Time   Address         Interface
4.4.4.4         1     EXSTART/BDR     00:00:38    192.168.1.4     GigabitEthernet0/0/0/0
    Neighbor is up for 00:00:06

RP/0/0/CPU0:ios#show ospf trace errors
11   Apr  3 18:33:41.776* ospf_rcv_dbd: WARN nbr 4.4.4.4 larger MTU dbd_if_mtu 2000 oi_ip_mtu 1500

Pretty clear that there is an MTU problem. Below I’ve set the Interface MTU to 9000 – remember IOS-XR includes the L2 headers in the Interface MTU, so the maximum encapsulated data on this wire would be 8986.

UPDATE 23/12/15:  I’m setting the interface MTU to a different value to the IOS router’s MTU setting,  to show that it’s the IP MTU that is the important setting and must match for two OSPF routers to establish an adjacency.

The IP MTU is set to 2000 to match the other router.

interface Loopback0
 ipv4 address 3.3.3.3 255.255.255.255
!
interface GigabitEthernet0/0/0/0
 mtu 9000
 ipv4 mtu 2000
 ipv4 address 192.168.1.3 255.255.255.0
!
router ospf 1
 area 0
 interface Loopback0
 !
 interface GigabitEthernet0/0/0/0
 !
 !
!
end

OSPF has established with the 1000v now the IPv4 MTU has been changed to 2000, and we can see the route the 1000v’s loopback interface 4.4.4.4

Notice that for OSPF to be happy, it only matters that the IP MTU is the same on both routers, the physical MTU can be different.

RP/0/0/CPU0:ios#show ospf neighbor
Thu Apr 3 21:07:09.053 UTC
* Indicates MADJ interface
Neighbors for OSPF 1
Neighbor ID Pri State Dead Time Address Interface
4.4.4.4 1 FULL/DR 00:00:31 192.168.1.4 GigabitEthernet0/0/0/0
 Neighbor is up for 01:13:55
Total neighbor count: 1

RP/0/0/CPU0:ios#show route 4.4.4.4
Thu Apr 3 21:07:29.291 UTC
Routing entry for 4.4.4.4/32
 Known via "ospf 1", distance 110, metric 2, type intra area
 Installed Apr 3 19:53:14.416 for 01:14:14
 Routing Descriptor Blocks
 192.168.1.4, from 4.4.4.4, via GigabitEthernet0/0/0/0
 Route metric is 2
 No advertising protos.

RP/0/0/CPU0:ios#ping 4.4.4.4 donotfrag size 2000 co 1
Thu Apr 3 21:07:36.781 UTC
Type escape sequence to abort.
Sending 1, 2000-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
RP/0/0/CPU0:ios#

Junos

Finally we’ll configure the Junos router and make sure that we are able to ping with 2000 bytes of protocol data to the 1000v and XRv.

The Junos configuration is as follows, I’ve not added the MTU settings yet.

interfaces {
    }
    ge-0/0/1 {
        unit 0 {
            family inet {
                address 192.168.1.5/24;
            }
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 5.5.5.5/32;
            }
        }
    }
}
protocols {
    ospf {
        area 0.0.0.0 {
            interface lo0.0 {
                passive;
            }
            interface ge-0/0/1.0;
        }
    }
}

The neigbor isn’t going to establish so let’s add some traceoptions to double check what the problem is.

root@firefly> show configuration protocols
ospf {
    traceoptions {
        file ospf-log;
        flag error;
    }
}
root@firefly> show log ospf-log
Apr 3 21:39:00 firefly clear-log[1261]: logfile cleared
Apr  3 21:39:01.701093 OSPF packet ignored: MTU mismatch from 192.168.1.3 on intf ge-0/0/1.0 area 0.0.0.0
Apr  3 21:39:04.794783 OSPF packet ignored: MTU mismatch from 192.168.1.4 on intf ge-0/0/1.0 area 0.0.0.0

OK, that’s pretty clear. Let’s fix the MTU and IP MTU.

root@firefly# show | compare
[edit interfaces ge-0/0/1]
+   mtu 9192;
[edit interfaces ge-0/0/1 unit 0 family inet]
+      mtu 2000;

Notice that the IP MTU is configured under family inet.

root@firefly> show interfaces ge-0/0/1 | match "ge-|MTU:"
Physical interface: ge-0/0/1, Enabled, Physical link is Up
  Link-level type: Ethernet, MTU: 9192, Link-mode: Full-duplex, Speed: 1000mbps,
  Logical interface ge-0/0/1.0 (Index 74) (SNMP ifIndex 519)
    Protocol inet, MTU: 2000
root@firefly> show ospf neighbor
Address          Interface              State     ID               Pri  Dead
192.168.1.4      ge-0/0/1.0             Full      4.4.4.4            1    38
192.168.1.3      ge-0/0/1.0             Full      3.3.3.3            1    37

root@firefly> show route protocol ospf

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[OSPF/10] 00:02:15, metric 2
                    > to 192.168.1.3 via ge-0/0/1.0
4.4.4.4/32         *[OSPF/10] 00:02:15, metric 2
                    > to 192.168.1.4 via ge-0/0/1.0

All good, we can see routes to 3.3.3.3 and 4.4.4.4

Now for the ping, remember that the Junos ping size excludes the ICMP (8 bytes) and IP (20 bytes) headers , so we’ll be expecting the maximum working ping size to be 1972 bytes, for 2000 bytes of protocol data and 2014 bytes on the wire.

root@firefly> ping rapid count 1 do-not-fragment size 1972 3.3.3.3
PING 3.3.3.3 (3.3.3.3): 1972 data bytes
!
--- 3.3.3.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 3.293/3.293/3.293/0.000 ms

root@firefly> ping rapid count 1 do-not-fragment size 1972 4.4.4.4
PING 4.4.4.4 (4.4.4.4): 1972 data bytes
!
--- 4.4.4.4 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.961/0.961/0.961/0.000 ms

root@firefly> ping rapid count 1 do-not-fragment size 1973 3.3.3.3
PING 3.3.3.3 (3.3.3.3): 1973 data bytes
ping: sendto: Message too long
.
--- 3.3.3.3 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

NX-OS

I don’t have a Nexus in my lab, but for completeness below is the config for updating the MTU on NX-OS.

On the 7k the system will be enabled for Jumbo by default, if you need to change this value it’s done like this

switch(config)#system jumbomtu 9216

Make sure you check the Interfaces and Vlan interfaces have the correct MTU. That’s done with the interface command “mtu X”. For Layer 2 interfaces, configure either the default MTU size (1500 bytes) or up to the system jumbo MTU size.

On the 5k it’s done a bit differently – in a QoS policy map! For NX-OS >4.1

switch(config)#policy-map type network-qos jumbo
switch(config-pmap-nq)#class type network-qos class-default
switch(config-pmap-c-nq)#mtu 9216
switch(config-pmap-c-nq)#exit
switch(config-pmap-nq)#exit
switch(config)#system qos
switch(config-sys-qos)#service-policy type network-qos jumbo

 Summary

In this post I’ve shown how the physical MTU can vary from the IP MTU, and how it’s important to have the same IP MTU when working with OSPF.

Also discussed were debugging steps to troubleshoot OSPF MTU issues on IOS and Junos.

Gotta love Junos for keeping things consistent!