This is part 4 of the following blog series:
- My first FlexPod! (Part 1 – Introduction)
- My first FlexPod! (Part 2 – Hardware Overview)
- My first FlexPod! (Part 3 – Hardware Configuration)
- My first FlexPod! (Part 4 – Quality of Services)
- My first FlexPod! (Part 5 – Hyper-V Cluster)
- My first FlexPod! (Part 6 – System Center)
Quality of Services:
I hope you enjoyed reading part 1, 2 and 3. As explained Cisco UCS is a converged infrastructure. A converged infrastructure uses less cabling and allows you to configure vNIC’s and vHBA’s as an abstraction layer on top of the physical infrastructure. This offers flexibility and has some benefits. It also adds an important requirement, which is QoS (Quality of Services). Without QoS, any type of traffic could saturate all available bandwidth resulting in slow network or storage performance. Which could potentially make your compute platform unstable. It is important that each type of traffic can consume as much bandwidth as possible when needed, but when the network gets congested each type of traffic gets a fare share of bandwidth, shaped by a pre-defined weight (%).
The nice thing is, Cisco UCS fully supports QoS. In fact, it is an absolute requirement which is easy to configure. QoS is not only used for traffic shaping, it is also used to handle certain type of traffic differently. Like storage traffic (especially Fiber Channel traffic) needs to be lossless (without packet drops). And you might want certain type of traffic (like iSCSI traffic) to use a different MTU packet size (e.g. Jumbo Frames with MTU 9000).
With this blog I am not going to explain QoS in technical detail. I will give you some links to technical resources that were useful to me. And I will explain what you can and need to configure in UCS Manager.
QoS System Class:
The first thing you need to configure in UCS Manager are the properties of the QoS System Class.
Cisco UCS handles the following QoS priorities:
- Platinum
- Gold
- Silver
- Bronze
- Best Effort
- Fiber Channel
You cannot rename them, but you can change their properties as shown in the screen-capture above. Once you have configured the QoS System Class you can configure so call QoS Policies and finally link those policies to vNIC’s and vHBA’s, which I will explain in a moment. Before I explain QoS Policies first allow me to explain some important properties of the QoS System Class.
CoS Value
You can configure each priority with a CoS (Class of Services) value. QoS accepts a CoS value of 0 to 7, whereas 0 and 7 are reserved by the system, and 3 is reserved by Fiber Channel. So apart from a CoS value of 3 you can configure 1, 2, 4, 5 and 6 for the remaining priorities. CoS operates only on 802.1Q VLAN ethernet, at the data link layer (layer 2). A CoS value is tagged outbound in the 802.11Q frame so that other network devices receive it inbound.
Although QoS is automatically configured under the hood by UCS it is important that the CoS values match your uplink devices. Because you always have uplink devices (like Cisco Nexus switches) connected to the Fabric Interconnects. And you want them to handle each type of traffic the same. For example, you want the MTU packet size to be maintained end-to-end. I will get back to this in a moment…
MTU Packet Size
You can configure each priority with a MTU packet size to match your requirement. Except for Fiber Channel which has a fixed MTU packet size of 2240. As you might know, it is common to enable Jumbo Frames (MTU 9000) for an iSCSI and Live Migration Network. Or maybe you want to enabled Jumbo Frames over the entire system. The CVD (Cisco Validated Designs) have their best-practices in terms MTU values. So it is best to follow those designs and make sure they match your uplink devices.
NOTE: The MTU Packet Size value is not the value that is applied to a vNIC or vHBA. Because a vNIC or vHBA has it’s own property to configure a MTU packet size. This value is used when traffic from other devices (like uplink switches) arrive at the FI’s which have tagged the same CoS value. Imagine you have a storage system (either iSCSI or Fiber Channel) that has QoS configured and tags a CoS value to the storage traffic. When that storage traffic arrives at an FI it will know how to handle the traffic according the QoS System Class. This also applies to a UCS Server on which you have configured QoS in the Operating System.
Weight
As the name suggest, the ‘weight’ value defines the amount of weight (%) a priority has when the network is congested. As the screen-capture above shows the priority named ‘Gold’ has a weight value of 9. With all weight together this equals to 40% of the entire bandwidth.
NOTE: Using the sceen-capture as an example; Traffic tagged as ‘Gold’ can use almost 100% of available bandwidth. But once the network gets congested it will be limited to 40% bandwidth.
So it is important that you think through what different type of traffic you have and how you want to classify them with QoS. One thing to keep in mind, always make sure your storage traffic (whether it is iSCSI or Fiber Channel) has enough bandwidth to operate properly. Also make sure it uses the right MTU packet size and has no packet loss.
QoS Policies
Once you have configured the QoS System Class you are ready to configure QoS Policies.
The screen-capture above is just one example. As you can see you have to select a ‘priority’ which you have configured earlier. The ‘Burst(Bytes)’ value essentially means the NIC speed. If you configure a value of 10240 and link the QoS Policy to a vNIC it will operate as a 10GbE NIC. If you configure it as 1024 it will operate as a 1GbE NIC. UCS even allows you to configure uncommon values like 2048 for 2GbE.
As mentioned the CoS value is tagged to the 802.11Q frame. The ‘Host Control’ setting defines whether the Operating System on a UCS Server is allowed to control QoS. If you have already configured QoS on your Operating System using CoS values, and do you do not want it to be overwritten by UCS you should set the ‘Host Control’ setting to ‘Full’.
As the following example of a vNIC illustrates you can select a QoS Policy:
Notice the QoS Policy ‘iSCSI’ is assigned. And also notice that the vNIC has it’s own property for the MTU packet size. The really nice thing about all this is, QoS is easy to configure. You only configure the QoS System Class once. Then you create some QoS Policies as you want and apply them to a vNIC or vHBA. If you would implement a Hyper-V Cluster you can configure QoS on each vNIC of a Hyper-V Server. This will make sure traffic is shaped accordingly.
If you want to know more about QoS on Cisco UCS I highly recommend you to have a look at the following video and blog from Brad Hedlund:
Cisco UCS Networking, Cisco VIC (Palo) QoS
https://www.youtube.com/watch?v=mMsYWkifWM0
VMware 10GE QoS Design Deep Dive with Cisco UCS, Nexus
http://bradhedlund.com/2010/09/15/vmware-10ge-qos-designs-cisco-ucs-nexus/#comment-127145
Challange (Uplink Switches and Jumbo Frames):
Now, there is something I need to share. During the design and implementation of our FlexPod environment I had a challange and found something that was not well documented in the CVD (Cisco Validated Designs). I was able to solve it but it took me some time to master. That is the reason why I wanted to add this part in the first place.
First of all, QoS is very easy to configure with Cisco UCS Manager. No CLI, just UCS Manager quick and easy. UCS Manager covers the entire UCS system. But in our infrastructure we use iSCSI over the entire system. I would have preferred Fiber Channel, but that was just not the case. We have connected two uplink Cisco Nexus 5548UP switches. And our NetApp Storage Array is connected to those switches. Please refer to the connectivity overview of part 2 from this blog. We wanted normal network traffic to use a default MTU packet size of 1500, and iSCSI traffic to use a MTU packet size of 9000 (Jumbo Frames). The thing is, with a Nexus 9000 series switch you can configure MTU values per VLAN. But with a Nexus 5500 series switch you cannot. Instead you can enable it over the entire switch for all traffic, or by manually configuring QoS. Back then I had no experience with QoS on Nexus, so I had to investigate it. The following excellent blogs and video from Matt Oswalt provided me all the information I needed.
Nexus 5000 QoS – Keeping It Classless Labs
http://keepingitclassless.net/2013/03/nexus-5000-qos-keeping-it-classless-labs
Part 1- Types of QoS Policies
http://keepingitclassless.net/2012/11/cisco-quality-of-service-part-1-types-of-qos-policies
Part 2 – Bringing it Together: Cisco Nexus, Cisco UCS, and VMware
http://keepingitclassless.net/2012/11/qos-part-2-qos-and-jumbo-frames-on-nexus-ucs-and-vmware
But I was not there yet. As a bonus I had to contact Cisco Support for an issue we had. We noticed that by default all iSCSI traffic coming from the NetApp Storage Array is always tagged with CoS value 4. Since we gave our iSCSI traffice priority ‘Gold’ I then had to make sure it had a CoS value of 4. As described in the following link from NetApp Support:
Does the storage system set the CoS field on a vlan tagged frame?
https://kb.netapp.com/support/index?page=content&id=3013708
At the end I was able to configured QoS on the Nexus 5548UP switches in such a way that it matches our UCS environment. All iSCSI traffic is now identified and classified to enable Jumbo Frames (MTU 9000) and a proper bandwidth weight. All other traffic is handled with MTU 1500. Working flawlessly! It took me some time to master. But once you get to know it you get a big smile on your face 😉
Now we are able to configure Hyper-V Servers the way we want. Just easily add as many vNICs we want, apply the right QoS Policies on them and have optimal performance.
Ok, that’s it for now. I hope this information was informative. Click on the link below to continue with the next part.
