Now that I’ve cover the RVC primer in the previous post vSphere 5.5: Using RVC VSAN Observer Pt1, let’s dive into “vsan.observer” command. “vsan.observer” is more than just a simple RVC command, but instead it’s an entire performance monitoring utility for VSAN with a built-in webserver and web UI that provides insight into many different aspects of VSAN performance related characteristics.

By running the “vsan.observer” command and passing it to a VSAN enabled cluster, the command gathers performance statistics from vCenter Server and VSAN every 60 seconds by default.

VSAN Enabled Cluster

The collection time interval can be modified with the use of the interval parameter. Decreasing the time interval should be used with caution as the vsan.observer increases the collection frequency and can easily collect gigabytes of data per hour.

Running the command with the option listed below opens an unencrypted HTTP webserver on port 8010 by default. Using the port parameter and specifying the desire port the default port can be change when executing the command.

  • vsan.observer ~/computers// –run-webserver

VSAN Observer CLI

The vsan.observer command will run until it is manually stop with by using “<Ctrl>+<C>”.

Note: The vCenter Server retains the entire history of your observer session in memory until it is stopped with +. Because of this behavior the current version will shut itself off after two hours. This scenario presents a possible good reason to run on a dedicated vCenter Server virtual machine.

After all this preparation work we are now ready to look at in-depth VSAN performance data by opening a web browser and pointing to the vCenter Server running RVC:

  • http://<vc-rvc-ip>:<observer-port>

Example:

  • http://vcenter.local:8010 <-default port number

Note: The VSAN Observer default port number is 8010

It is recommended to use a recent version of the Google Chrome browser, but any modern browser should work. Internet Explorer 8 is not considered a modern browser, but may still work to some extend. Older versions of IE will definitely give you problems.

The VSAN Observer web UI is organized by subsystem. You should start familiarizing yourself with the UI starting with the VSAN client view.

VSAN Client

The VSAN Client view provides an overview over the level of service virtual machines are currently getting from VSAN. Every host in a VSAN cluster consumes storage that is distributed across all other hosts in the cluster, so seeing a performance issue in the VSAN Client view on Host-01 may in fact be due to overloaded disks on Host-02.

VSAN Client View

VSAN Disks

The VSAN Disks view provides the ability to look at VSAN from the physical disk layer perspective, and checking how storage contributing nodes are doing in terms of servicing IO from their local disks.

VSAN Disks View

VSAN Deep Dive

The VSAN Deep Dive view provides the ability to drill down further into a deep-dive of the VSAN disks layer on a per-host basis and seeing how VSAN splits IO among SSDs and HDDs.

VSAN Disks Deep Dive View

PCPU & Memory

VSAN shares compute resource with the rest of ESX, i.e. VSAN is consuming a slice of the same CPU and Memory resources that the VMs running on a given host are also consuming. You can inspect the VSAN CPU and Memory consumption in dedicated tabs in the observer, which may also come in handy detecting performance bottlenecks due to CPU or Memory limits.

VSAN PCPU View

 

VSAN Memory View

Distribution

The distribution view provides insight into VSAN cluster balance, distribution of objects, significant cluster events, etc. Every time you switch tabs, the graphs update automatically and reflect the latest information gathered by RVC in the background. Take a look around. Most tabs contain information about how to read the information presented in the graphs. But a lot of them no doubt require familiarity with storage performance.

VSAN Distribution View

DOM Owner

Don’t worry about this one; the information displayed is very similar to what is displayed under the VSAN Client view. This view is specifically is to be used by the folks at VMware support.

VSAN DOM Owner View

VMs

VSAN being a VM centric storage platform and its monitoring and management capabilities are entirely focused around virtual machines. This view provides visibility into the per-disk (VMDK) policy settings applied when using VM Storage Profiles. The VMs view provides the ability to look at performance on a per-VM or even per-virtual disk level in the VSAN observer.

VSAN VMs View

We hope you will enjoy the VSAN observer and find it useful in operating and troubleshooting your VSAN deployments. Performance troubleshooting is a very important but also difficult topic, and we hope the community will provide suggestions and feedback on your experience using the VSAN observer, so we can improve it as well as evolve our product direction overall to better address your needs.

Watch out for subsequent posts that will go into more detail about how to read VSAN observer graphs, how to understand storage performance, and how to draw conclusions and troubleshoot performance problems.

That will be it for now folks. Again thanks Christian Dickmann (@cdickmann)

– Enjoy

For future updates, be sure to follow me on Twitter at @PunchingClouds

X