The new vCloud Director 5.1 delivers many new features and enhancements, one in particular is the introduction and support of Virtual Extensible LAN (VXLAN). VXLAN is a technology that enables the expansion of isolated vCloud architectures across layer 2 domains beyond the limits imposed by the IEEE 802.1Q standard. By utilizing a new MAC-in-UDP encapsulation technique, a VXLAN ID adds a 24 bit identifier, which allows networks to push beyond the IEEE 802.1Q limit to a possible 16 million logical networks. Figure 1 below illustrates the changes added to the Ethernet frame by VXLAN.

Figure 1: Ethernet Frame with VXLAN Encapsulation

 While the conventional IEEE 802.1Q standard works perfectly, when trying to meet greater scalability demands VXLAN surpasses the IEEE 802.1Q limitation by offering scalable capabilities of up to 16 million possible networks. Because of the scalable and flexible capabilities offered by VXLAN, this technology is something to be consider for large scalable Cloud (vCloud) networks. For a quick crash course on VXLAN, take a look at Duncan Epping’s post “Understanding VXLAN and the value prop in just 4 Minutes…”

Configuring VXLAN in vCloud Director 5.1 required some initial steps that are outside of the vCloud Director 5.1 management interface which I want to illustrate here.

First a couple of facts:

A VXLAN network pool is automatically created in vCloud Director 5.1 whenever a Provider vDC is created. If the hosts of any given cluster are not prepared to use VXLAN first, the VXLAN network Pool in vCloud Director will display an error. I would recommend identifying all of the pre-requisites for the use of VXLAN from a network as well as the software dependency perspective before creating new Provider vDC in vCloud Director 5.1.

In order to prepare the resource clusters (hosts) to use VXLAN, log in the vCloud Networking and Security appliance (previously knows as vShield Manager). The preparation of the networks as well as the hosts requires the identification and assignment of the Segment ID Pool and the Multicast addresses. Below are the steps necessary to prepare and configure VXLAN for vCloud Director 5.1.

Step 1: Log into the vCloud Networking and Security appliance. Select the Datacenter. Then, select the Network Virtualization tab on the right side of the screen and click the Preparation hyperlink. This will reveal the Connectivity and Segment ID screen, as illustrated in figure 2.

Figure 2: Network Virtualization Settings



Step 2: Click the Edit button on the right end of the screen, and enter the required Segment ID Pool, and Multicast address that will be used by vCloud Networking and Security appliance. The Segment ID’s cannot be mapped directly to any one Multicast Address, as the possibility of one-to-one mapping doesn’t exist. This Segment ID and Multicast Address configuration is defined in ranges. Figure 3 illustrates the Segment ID and Multicast Address options.

Figure 3: Segment ID Pool and Multicast Address


Step 3: Click on the Connectivity button in the Network Virtualization tab to prepare the resource clusters (hosts) to be part of the VXLAN with vCloud Director. Choose the Distributed switch that is to be associated with the resource cluster, and enter the VLAN ID for the desired network segment that will be used to overlay the VXLAN traffic coming from the Distributed Switches. Figure 4 illustrates the configuration options.

Figure 4: Resource Cluster


Step 4: Specify the NIC teaming policy that applies to the respective Distributed Switch configuration, and the MTU settings. The MTU settings for VXLAN default to 1600 bytes due to the VXLAN ID encapsulation technique which increases the size of the packets. This behavior is similar for the configuration of vCDNI in vCloud Director.  vCDNI required the minimum MTU packet configuration of 1524. Overall, the important thing to understand here is the requirement to use jumbo frames across all network devices. Figure 5 illustrates the NIC teaming policies available as well as the default MTU settings and click Finish.

Figure 5: VXLAN Attributes


After choosing and completing the specification for the Distributed Switches, the VXLAN vmkernel modules are pushed and enabled on to all of the hosts that are part of the selected cluster. New dvPort Groups and vmknic interfaces are added and automatically created on the Distributed Switch associated to the VXLAN. The new dvPort group can be identified by the unique naming convention vxw-vmknicPg-dvs-xx-xx-xx-xx. Figure 6 offers an example of the adapter configuration.

Figure 6: VXLAN VMkernel Interfaces



A troublesome results of the automated network configuration process for the vmknics, is that all interfaces will be automatically assigned an IP address based on DHCP. This behavior can become a configuration management issue; unless there is a DHCP server on that network segment (normally the management network), all of the newly created interfaces will receive an IPv4 address within the 169.254/16 prefix that is valid for communication with other devices connected to the same physical link.

This configuration will not work as an IPv4 local addresses are not suitable for communication with devices not directly connected to the same physical or logical link, and are only used where stable, routable addresses are not available. As a result of this configuration the status of the hosts preparation will be displayed as “Not ready” in the vCloud Networking and Security appliance interface. Figure 7 illustrates the issue discussed above.

Figure 7: vmknics IP Address Status



The solution to this issue is simple: update the vmknics interface with automatically assigned IP with valid addresses. This can be achieved in a manual or automated format. Figure 8 illustrate the results of a successful configuration.

Figure 8: VXLAN Successful Preparation Results


Step 5: At this point, all the required network, and hosts preparation for the use of VXLAN with vCloud Director 5.1 have been completed. In order to start using the VXLAN feature in vCloud Director 5.1,  create a Provider vDC.  A VXLAN Pool is automatically created. Figure 9 illustrates the existence of VXLAN capable network pool in the management interface of vCloud Director.

Figure 9: VXLAN Network Pool in vCloud Director 5.1


There you have it, folks. You can now proceed with the creation and configuration of Organization, and vApp networks to harness the scalable features delivered by VXLAN in vCloud Director 5.1 infrastructures.




Tagged with →  
Share →

32 Responses to vCloud Director 5.1 VXLAN Configuration

  1. […] 5.1 – New Features and Enhancements in Networking (ESX Virtualization) vCloud Director 5.1 VXLAN Configuration (Punching Clouds) ESXi 5.1 Network Health Check with Jumbo Frames (Rickard Nobel) Distributed […]

  2. Alex says:

    How can we configure VLAN networking pool instead of VXLAN? Let’s say our switches dont support MTU greater 1500, how would we go about configuring provider VDC with VLAN backed network pools?

  3. Rawlinson says:

    Alex, to configure a VLAN backed pool all you have to do is create a vDS on the vSphere infrastructure that will be use for your VLANs and mapped the VLAN backed pool to that switch and enter the VLAN ID range that you want to allow in that vDS. VLAN is not like vCDNI (1524), or VXLAN (1600) where jumbo frames support and configurations are needed because of the packet size increase.
    Follow the provided instructions and you should be good to go.

  4. Abhilash says:

    Hi Rawlison,
    Thank you so much for the above documentation. I’m trying to implement the same thing in my lab but i’m unable to prepare the hosts for the VXLAN. The return an error telling the VIB failed to install. Can you eloborate a little more on the step 7 which will be really helpful.

  5. […] there is an excellent article about VXLAN setup in vCloud Director 5.1 it describes only the native vSphere VXLAN implementation. […]

  6. Yohan Wadia says:

    Hi Rawlinson,
    I had one query regarding VXLANs… Is it really necessary to set up VXLANs in a vCloud env?? Can’t I just have a vSphere Port group backed network pool??

    Awesome writeup though… really helpful 🙂

  7. Rawlinson says:

    vCloud Director will automatically create a VXLAN network pool. You don’t have to use it, but this is something that currently happens as part of the PVCD configuration workflow. You can continue and create your port group backed network pool as you would normally do before VXLAN.

  8. […] to know a bit more about how to configure VXLAN inside VCD? Rawlinson Rivera has a nice write-up that is worth […]

  9. […] to know a bit more about how to configure VXLAN inside VCD? Rawlinson Rivera has a nice write-up that is worth […]

  10. […] is a repost from Rawlinson’s personal blog, Punching […]

  11. […] is a repost from Rawlinson’s personal blog, Punching […]

  12. Andrew W. says:

    What do we get when we have a VXLAN configured?

  13. Rawlinson says:

    VXLAN creates a network abstraction layer over available physical networks. With a VXLAN fabric in a datacenter, it is possible to overlay multiple VXLAN- backed Layer 2 networks all over the datacenter, providing Layer 2 adjacency
    to VMs hosted in the datacenter . This makes it possible to create on-demand networks on top of this fabric, allowing unconstrained virtual machine placement within the datacenter and, at the same time, affording unencumbered virtual machine mobility .

  14. Brian says:

    Hi, Nice article, I have a question about the proper IPs to replace the autogenerated ones. How do you know which range they need to be in? Should the IPs be a valid IP on the external network? Also with regards to the segment ID range and the multicast range, how do you know what is valid for this range? sorry if this is a dumb question but I am not a networking expert, so any pointers greatly appreciated. thanks again. Brian

  15. Rawlinson says:

    Hey Brian let me see if I can answer your entire question.

    The IPs should be valid and routable as these are the IPs that are going to be to create the VXLAN fabric or also known as VXLAN Tunnel End Point which (VTEP) is esponsible for encapsulating the VM traffic in a VXLAN header as well as stripping it off and presenting the destination VM with the original L2 packet. In vCloud Director terms those interfaces are not necessarily external networks because if you notice how the IPs are setup and the fact they are setup as VMkernel interface with IPs configure outside of vCD. That is not the way you setup external networks in vCD.

    Regarding the SegmentID and multicast range, the segmentID are the VXLAN SegmentID which are normally called virtual wires or virtual network identifiers. This is what allows you to specified the number of networks to be create within the configuration. This is the thing that everyone is buzzing about in VXLAN, you can define up to 16 million logical networks, but who needs that many right? hopefully not you at the moment. Do the range of segmentID is starts at the 5000 range and you can go up from there. So for example if you wanted to create 20 logical networks your range will be 5000-5020. Now for each one of those segmentID you can define a dedicated multicast groups or If your network is limited to the number of multicast groups you can shared them as well. This sort of thing requires configuration on physical routers (enable multicast routing) and switches (enable IGMP, etc).

    One of the steps that I didnt add to the blog post was the preparation of the VXLAN network scope. You can do this by clicking on the Network Scope hyperlink next to the preparation link. There you can select the hosts and clusters that you want to be part of that network scope and start consuming the logical networks you defined in the pool (5000-5020). To create the logical networks or virtual wire just go to the networks link and add them there. You can see the VXLAN Network Scope created and you can identify the hosts and clusters and VMs to be configure on that logical network or virtual wire.

    I hope this answers your questions, I would have added screen shots to this message but replying from wordpress wont let me do that. Let me know.



  16. ashok says:

    1. how to decide the segment id pool range and multicast address ?
    2. i am getting one error while creating the vxlan port groups in vc

    Update opaque data for set of entities . A specified parameter was not correct selectionSet.dvsUuid.

    how to sort out this ?

  17. Rawlinson says:

    Asjok, In order to decide the segmentID pool range and multicast addresses I would recommend to check with the network engineers and see if multicast is currently being used on the network. You may want to dedicated multicast addresses for different purposes (traffic type, isolated systems communication, etc)
    in VCNS there are usable ranges for both SegmentID’s and Multicast (SegmentID range starts at 5000) (Multicast range starts at You can read one of my previous responses a gentleman named Brian on this post before. I provided a few examples on the creation of segments pools and configuration, that may be helpful.



  18. […] Exporting/importing/restoring Distributed Switch configs using vSphere Web client (KB Article) vCloud Director 5.1 VXLAN Configuration (Punching Clouds) ESXi 5.1 Network Health Check with Jumbo Frames (Rickard Nobel) ESXi 5.1 Network […]

  19. […] to configure VXLAN in vCloud Director 5.1 can be found on the punching clouds […]

  20. […] Note: Due to a cock-up on my behalf, I lost the screen grab for this part of the post. So I had to steal it from someone else. Namely from Rawlinson Riveria’s post which focuses on the same/similar setup – […]

  21. Selvakumaran says:

    i have only one vmk named vmk0. i am not able to change ip setting to automatically. how to do it.


  22. […] to vCNS and SSO. This is also a good time to start setting up VXLAN. VXLAN setup can be found here. At this point we are done. You can license it on the vCenter server after vCloud does it’s […]

  23. […] reference, the installation steps published by Rawlinson at are almost identical to the installation step I have followed, though I kept getting the following […]

  24. […] more detailed walk through for configuring VXLAN for a vCloud Director environment, check out this article by Rawlinson Rivera who takes you through the process step by […]

  25. […] is a repost from Rawlinson’s personal blog, Punching […]

  26. […] caso que lo fuere, en éste punto el Provider vDC ya estaría listo para hacer uso de redes VXLAN). El post de @punchingclouds brinda mayores luces al […]

  27. […] is also a good time to start setting up VXLAN. VXLAN setup can be found here. At this point we are done. You can license it on the vCenter server after vCloud does it’s […]

  28. […] vShield Manager, adding VMkernel ports, Network Scopes, Virtual Wires, and so on (Mike Laverick and Rawlinson Rivera both have easy to follow tutorials. The VMware VXLAN Deployment Guide is also a great read). Once […]

Leave a Reply