Sep 01

ESXi mgmt network lost when adding ESXi to VDS

So the other day a had a very unpleasant experience when adding a few new ESXi hosts, version 5.5 U2 including the latest patch, to an existing vCenter Server (5.5 U2e) vNetwork Distributed switch (VDS). I used the wizard in the vSphere web client and followed the recommendation but i guess i could have made the change for just one ESXi host at the time and not all four at the same time.

However, when the task was completed i ended up with three new ESXi hosts attached to the VDS but one new ESXi host disconnected from vCenter Server:


So i connected to the Direct User Console Interface (DCUI) and enabled ESXi shell via troubleshooting options. I started by verifying the available network IPV4 interfaces via the command “esxcli network ip interface ipv4 get” and it all looked ok.


Next i checked the routing table using the command “esxcli network ip route ipv4 list” and it didn’t look that good since no default GW was configured.


So this means i had a vmk0 interface but the default GW specified via the interface was not enabled. Just to verify i ran the command “esxcli network ip interface list” and the output showed me what i was suspecting:

  • Enabled: false


   To enable a vmk interface you can simply run the command “esxcli network ip interface set -i vmk0 -e true” but in this case i received the following error:

  • Unable to kill DCUI (12096732)

One thing to understand is that the numbers next to DCUI, in my case 12096732, is not the DCUI pid. To find the pid you can run the following command “ps -j |grep -i dcui” and i my case it showed:

  • 35605 35605 dcui                 35605 /bin/dcuidcui06

So PID 35605 is the one you need to kill using the command “kill -9 35605” and when completed i could successfully enable the vmk0 interface by running the “esxcli network ip interface set -i vmk0 -e true” command once again.


The default GW was now listed via the “esxcli network ip route ipv4 list” command and the vmk0 interface had status “Enable: true”.
The ESXi host connected automatically to the vCenter Server and we were ready to continue with some more important stuff:)

Aug 27

vMotion – A general system error occurred


The other day i had an issue with vMotion migrations in my vSphere environment running the following versions:

  • vCenter Server 5.5.0 build 2183111
  • ESXi 5.5.0 build 2143827

The error i was seeing happened at the magic 14% threshold and the errors i saw were:

  • A general system error occurred: Migration to host X.Y.Y,224 failed with error Busy (0xbad0004)
  • Migration to host X.Y.Z.224 failed with error Busy (195887108)


There are a bunch of VMware KB regarding vMotion failure at 14% and here are a few:

After a lot of troubleshooting including reading a bunch of log files and a few configuration changes i came across VMware KB 1031636 which talks about “vMotion fails with the error: remote host IP_Address failed with status Busy (1031636)”.
Even though KB 1031636 is for ESX/ESXi 4.1 i decided to give the Migrate.VMotionResolveSwapType config change (use 0 instead of the default value 1) a try and after the change all vMotion processes completed successfully.

VMware will update KB 1031636 to include vSphere 5.5 as well.

Aug 24

Nutanix @ VMworld in San Francisco 2015

It’s about a week before this years VMworld event (30 August – 3 September) held in San Fransisco at Moscone Center. Nutanix will be at the solutions exchange as an event Gold sponsor. This gives you a great opportunity to step by our booth  #1729 to meet some excellent Nutanix representatives and discuss pretty much anything regarding our invisible and any application ready platform.

If you haven’t totally filled up your schedule yet i can recommend the following opportunities:

Unfortunately i’ll not participate in this years event but i hope to see you at VMworld in Barcelona later this fall instead….


Aug 21

VMware Tools for nested ESXi version 1.2 released

“VMTurbo"As i wrote almost two years ago here, VMware labs got a VMware Tools fling for nested ESXi. This fling is still relevant when running ESXi 5.5 but for ESXi 6.x it is installed by default.  A few days ago a new version was added including the following enhancements:

  • Guest Time Synchronization
  • Larger resource pool for programs started using the VIX API

The VIB can be installed from the ESXi host directly if you got internet connection using the following command:

  • esxcli software vib install -v

Another option is to download the VIB manually from here.

Read more about VMware Tools for nested ESXi here.

Aug 18

Upgrade Nutanix Operating System via CLI


The other day i had to upgrade one customers Nutanix Cluster via the command line. I will not go in to details about why but let’s say it had to do with internal policies. I have written about how to upgrade the Nutanix Operating System (NOS) before and as outlined in this blog post i normally use, and it is the recommended way, the Nutanix Prism (user interface) to do the NOS upgrades. There are situations where the CLI might be useful and if needed you can follow the procedure outline below to conduct an upgrade


  • Log on to one of the Nutanix Controller Virtual Machines and verify you got enough available disk space using the following command:
    • du -hs
  • Perform a Nutanix Cluster health check to verify everything looks good before upgrading your system. Use the following command:
    •  ncc health_checks run_all
  • Upload the upgrade file to one of the CVMs using e.g. the following command:
    • scp nutanix_installer_package-danube-4.1.4-6951.tar.gz nutanix@
  • Unpack the Nutanix Cluster installation file on the CVM using the following command
    • tar -xvf nutanix_installer_package-danube-4.1.4-6951.tar.gz

Perform the upgrade

  • Enable auto install via the CVM on the Nutanix Cluster using the following command:
    • cluster enable_auto_install
  • Perform the NOS upgrade via the CVM using the following command:
    • /home/nutanix/install/bin/cluster -i /home/nutanix/install upgrade

You can follow the upgrade process via e.g.:

  • tail the finish.out and genesis.out file using:
    • tail -f finish.out
    • tail -f genesis.out
  • run the command:
    • upgrade_status

Verify the upgrade

When the upgrade is finished you can disable auto_install if needed by running the following command:

  • cluster disable_auto_install

Verify that the new version is installed using the following command:

  • ncli cluster status
    In my case the output was:

    • Cluster Version : danube-4.1.4-stable
    • Changeset ID : xxxxxx
    • Changeset Date : 2015-07-xx xx:yy:zz -0700
    • ncli Version : danube-4.1.4-stable
    • Changeset ID : xxxxxx
    • Changeset Date : 2015-07-xx xx:yy:zz -0700
  • Per CVM you can also verify the current version via the following command:
    • cat /etc/nutanix/release_version

So this is what you can do if you are not able to upgrade NOS via Nutanix Prism which is much easier and absolutely the preferred way.

Aug 13

Nutanix Acropolis is Citrix XenApp and Citrix XenDesktop ready

Nutanix Acropolis hypervisor (AHV) was released a few months ago at the Nutanix .Next conference. AHV is included in every Nutanix software license (Standard, Pro and Ultimate) and can run on any Nutanix node. Yesterday, 2015-08-12, the announcement was made that AHV is now verified as Citrix ready for XenApp and XenDesktop.

Screen Shot 2015-08-13 at 23.51.29

This is something really good since it will give IT architects additional flexibility and options when designing a virtual desktop infrastructure (VDI) solution and i know a lot of Nutanix customers running Citrix have been asking for this .

The Citrix ready verification includes the following components:

  • XenApp 7.6
  • XenDesktop 7.6
  • NetScaler
  • ShareFile

XenApp and XenDesktop are planned to be supported from September 2015 by Citrix and Nutanix. Other, non AHV, Nutanix solutions are already verified as Citrix Ready and validated to run the XenDesktop 7.6, XenApp 7.6, NetScaler & ShareFile.

Additional information:

Aug 13

vShield Manager disk 100% full

A few days ago my vShield Manager, VSM, virtual machine (VM) disk system was filled up to 100%. Yes i know this is due to poor monitoring but it’s in my lab and i’m using it for a specific reason from time to time. At least i had some reporting of the issue so i could fix it.

All the times i have seen this it has been because of log files not being deleted.

The preferred way to solve this problem is to open a support request (SR) with VMware and let them take care of things but in many cases you might do some research yourself. This is what i did to solve the problem or simply free up some disk space.

  • Open the VSM console
  • Log on using the admin credentials
  • Type enable and the admin password once again
  • Start SSH, since i usually finds it easier to troubleshoot via an ssh session compared to the VM console, by typing the following command:
    • start ssh
  • Log on via ssh using the above described procedure
  • Verified the file system status/usage by typing the following command:
    • show filesystems
      It showed 100% utilisation for /dev/sda2 (root file system)
  • Clean up the log files by typing the following commands:
    • purge log manager
    • purge log system
  • Verified file system usage and it still showed 100% utilisation for the root file system
  • Reboot the VSM
  • Verified file system usage and it now showed 4 GB available disk space

This solution applies to vShield Manager version 5.5.3 build 2175697

Screen Shot 2015-07-22 at 14.01.27

Case closed:)

Apparently this works for NSX (more specifically at least for NSX 6.1.4) as well according to Mohammed Salem
hanks for letting me know.


Aug 11

Chance to 2 win free VMworld Barcelona 2015 tickets from VMTurbo

Today i again received some were good news from VMTurbo. Now they have decided to also give away 2 free VMworld 2015 full conference passes for the Barcelona, Spain, event. VMworld in Barcelona will be held at Fira Gran Vira 12 – 15 October later this year. As i wrote in this blog post they also gave you a chance to win VMworld San Francisco conference passes.

If you haven’t done so already sign up, between 7 August – 4 September, for the chance to win the VMTurbo sponsored VMworld tickets here and tune in for the 3 drawings that will be held:

  • 21 August
  • 28 August
  • 4 Septembe

You can read the VMturbo Sweepstakes term and agreements here


So again, a huge thank you VMturbo for this great way of sponsoring the virtualisation community.

This is not an advertising blog post payed for by my blog sponsor VMTurbo, this is just a way for you as a VDCX56 blog reader to get a chance to take advantage of this great offer from VMTurbo.

Aug 04

VMworld 2015 Barcelona blogger pass


For the third year have been selected by the VMware vExpert team to get a free VMworld blogger pass. I’m really grateful for this since it’s given to a selected few VMware vExperts and makes it possible to attend VMworld in San Francisco or VMworld in Barcelona for free. The blogger pass is a full conference pass meaning you can attend the sessions, visit the solutions exchange and attend the VMware party.

My blogger pass is for VMworld in Barcelona 12-15 October. The event will be held at FIRA GRAN VIA as usual. You can expect same blog post setup as i have used the past two years unless i decide to do something else.

Screen Shot 2015-07-30 at 23.18.51

You can find my old VMworld blogger blog posts here:

If you are not registered for either of the VMworld events, use the following registration link and sign up. Finally, again a big thank you to VMware people involved in the community and in the blogger pass selection.

See you in Barcelona and if you want to have a chat or a beer, just reach out to me via twitter or the author section of my blog.

Aug 01

Meet NPX 004 – Samir Roshan


  • Name: Samir Roshan
  • NPX Number: 004
  • Employer: Nutanix Inc
  • Blog URL:
  • Twitter: @kooltechies
  • Virtualisation background: 10+ years of virtualisation experience
  • Storage Background: Storage has been a continuous part of my journey in the virtualisation world sometimes mainstream some times not. I will say close to ~5 years overall.
  • Hyper Converged Infrastructure background: Since my joining Nutanix in 2014
  • Future of HCI and other emerging technologies: HCI will be the mainstream platform for running application and infrastructure going forward. HCI will replace 3 tier storage completly over a period of time. In other emerging technologies we will see a constant pull and push of new concepts and prototypes. There will be concrete steps taken in the field of Micro-services. Cloud computing and dev-ops will continue its rally , Openstack will have a hard time surviving.
  • Value of NPX: Enormous !! NPX is the first certification to blend Multi hypervisors enabling architects to advise on architect solutions which includes Private Cloud, Microservices and Public cloud. The certification is an extremely challenging and rewarding one from the perspective of experience and skill sets required to go through it. Architects who wants to pursue skills in multiple areas should really go for it.
  • What made you go for NPX: Primarily because of the value and learning by pursuing NPX. It has enabled me to look at the challenges in IT at a very holistic level. I am in a position to advise my customer on different hypervisors, HCI, Webscale, Microservices and on Cloud.
  • Advise to people looking into the NPX track: Preparation is the key, this is probably the top level certification in the world right now. There are no shortcuts in order to achieve this, you need to follow the exam blueprint covering each areas in detail. Solid architectural understanding, technical expertise on the selected hypervisors, expert knowledge on the Nutanix platform is key to success.
  • Other: For all NPX aspirants feel free to reach out for any clarification or if you need any mentorship for the program.

Older posts «