Mar 24

Storage vMotion fails with error Already disconnected

This blog post applies to vCenter Server version 5.5. U2d and ESXi server version 5.5 U2a.

A few days back i got involved in a virtual machine migration (VM) issue. The VMs were suppose to be migrated from one vSphere cluster to another without any chance of mounting one of the vSphere clusters datastore(s) to the other vSphere cluster. The VM(s) couldn’t be powered off during the migration so the option we had was using the Migrate option from the vSphere Web Client and select “Change both host and datastore”:

Screen Shot 2015-03-24 at 20.28.51

We had to move about 10 VMs and 6 of them completed without any errors but for the additional 4 VMs the process was interrupted randomly. The same VM could be running for either 10 minutes or 2 hours meaning it could fail for any of its VMDKs. VMs with only 1 VMDK failed as well. When changing both host and datastore one VMDK is migrated at the time and not several VMDKs at the same time as when you just change datastore and not host.

The error we got was:

  • Status
    • A general system error occurred:Migration to host <x.x.x.x> failed with error Already disconnected (0xbad002e)
  • Detail
    • Migrating the active state of virtual machine


The initial investigation included:

  • IP information for all vMotion interfaces including
    • IP Addresses
    • Subnet masks
    • Gateway
  • Network connectivity between all potential hosts
  • Limited the migration process to specific sender and receiver ESXi hosts
  • VM vmware.log file in the VM home directory
  • vmkernel log file on both sender and receiver ESXi host.
  • ESXi hosts for dropped packages using esxtop during the migration process
  • Network switches for dropped packages
  • Spanning Tree configuration, which was actually not in use but on our list of things to go through.
  • Verifying latest NIC driver was used by running the following command:
    •  esxcli software vib list | grep -i ixgbe
      Screen Shot 2015-03-24 at 20.53.15

What got us on the right track was checking for errors on the ESXi host network adapters using the following command:

  • ethtool -S vmnic0 | grep -i error
    • The command gave us quite a lot  “rx_missed_errors”

When i have seen this before it has been related to “Flow Control” not being turned on. When the networking team turned on Flow Control on the switches the rx_missed_errors counter did not increase and the Storage vMotion processes completed successfully on the first attempt.

Mar 17

Nutanix Platform Expert (NPX)

This is really awesome, today Nutanix has released the enterprise computing certification Nutanix Platform Expert (NPX).  Since the certification is released by Nutanix it will off course require that each candidate completely master the Nutanix web-scale hyper converged infrastructure (HCI) concept and products. What will make NPX absolutely outstanding, and is a huge differentiator compared to other high level certificates, is that it covers both a specific area/concept/product (Nutanix) and several other areas as well including but not limited to e.g:

  • Analyze business needs
  • Application implementation/layout on HCI
  • Incorporate new operational models into Nutanix solutions
  • HCI Infrastructure design/implementation/operation
  • Multi Hypervisor expert skills (2 of the below 3):
    • KVM
    • Microsoft Hyper-V
    • VMware ESXi
  • Transition of existing 3 tier architecture to HCI


There are no requirements for participating in the NPX program apart from being a Nutanix Platform Professional. However, the following figure gives you an idea of the recommended skills before joining the program. Let’s assume you want to create a design using vSphere and also be tested on Microsoft Hyper-V you should at least have the skills compared to VCDX (DCV, DT or Cloud) and e.g. MCSE Private Cloud.

Screen Shot 2015-03-17 at 16.20.42

The entire NPX process consists of several steps including:

  • Written exams
  • A web-scale based design
  • Hands-on labs
  • Design exercise
  • Live peer review

Nutanix will sponsor two NPX Design Review sessions in in 2015. Both sessions will be held at Nutanix Corporate Headquarters located at 1740 Technology Drive, Suite 150, San Jose, CA 95110

  • September 14-18
  • December 7-11

To participate in the program, register your interest via the Nutanix NPX Program Homepage, NPX Program Forum or contact the NPX program manager Mark Brunstad at

Just one more important thing. This certification is free of charge,


Mar 13

vRealize Automation 6.2.1 released

A new version of the vRealize Automation (vRA) formerly known as vCloud Automation Center (vCAC) software was released yesterday, actually the same day as when vSphere 6.0 was released.

Based on that and the fact that is just a minor release i can unedrstand it didn’t got that much attention.

However, there are one really good thing with this release and that is that the secure remote console capabilities for machines provisioned with vSphere are implemented (back in a secure way) in this release by means of WebMKS (HTML 5 console) through a console proxy.

This feature requires ESXi 5.1 or later to be used.

The vRA 6.2.1 release notes can be found here and you can download the vRA 6.2.1 software here.

Mar 13

Two years as a blogger

Today, March 13 2015, i have been a blogger for two years. During my first year as a blogger i posted 113 blog posts and the last year i followed up with 116 blog posts meaning a total of 229 blog posts. As i mention in my one year as a blogger summary, which you can find here, the numbers are more than i expected. I’m still targeting one blog post per week.

During this year a lot of fun stuff have happened:

  • I have start working for Nutanix.
  • My blog was voted #18 in the yearly vBlog top voting. See separate blog post here.
  • My blog was voted #1 in the year vBlog top voting in the category “New blog”. See separate blog post here.
  • I have been awarded the vExpert title for 2015.

The number of unique blog visitors per day are increasing, slowly but still.

Obviously a lot of my work has been related to Nutanix which have reflected my blogging and it seems like there is a lot of interest around our software and i can understand why:)
I have continued to work a lot with vRealize Automation (vRA) formerly known as vCloud Automation Center (vCAC) and I’m also picking up on vRealize Orchestration (vRO) formerly known as vCenter Orchestrator (vCO) and not to forget KVM. So expect a lot more blog posts in these three areas.

Finally, a huge thank you to all of you reading my blog, creating blog posts comments, link your blog posts to my blog posts, retweet my tweets, mention my blog posts on twitter, giving me linkedin likes.


Mar 10

vRealize Automation monitoring

Since i started to work as a consultant back in 2007 monitoring has been part of pretty much all design, implementations and heath checks i have done. Every now and then i receive questions about how to implement monitoring for certain software and this week i received a question about monitoring the vRealize Automation (vRA)  formerly known as vCloud Automation Center (vCAC).
This blog post will describe how you can monitor the vRA:

  • Virtual Appliance
  • Identity Appliance,
  • Infrastructure as a Service (IaaS) Manager
  • Infrastructure as a Service (IaaS) Web services.

The blog post will not include e.g. the Distributed Execution Manager (DEM) Orchestrator, DEM Worker, vSphere Proxy Agent and the databases Microsoft SQL (MSSQL) and PostgreSQL the most likely also will be included in the vRA setup.

The following applies to (at least) vRA versions 6.1 and 6.2.

vRA Virtual Appliance

You can call the following URL for health status monitoring of the vRA VA(s):

  • https://fdqn/vcac/services/api/status

Make sure the response includes the following:


vRA Identity Appliance

You can call the following URL for health status monitoring of the vRA Identity Appliance(s):

  • https://fdqn:7444/websso/HealthStatus

Make sure the response includes the following:



vRA IaaS Manager

You can call the following URL for health status monitoring of the vRA IaaS Manager(s)

  • https://fqdn/VMPS2

Make sure the response includes the following:

  • BasicHttpBinding_VMPSProxyAgent_policy

vRA IaaS Web

You can call the following URL for health status monitoring of the vRA IaaS Manager(s)

  • https://fqdn/WAPI/api/status

Make sure the response includes the following:

  •  iaas-service


Hope this will help you in setting up monitoring for your vRA components.

Mar 04

Top vBlog 2015 voting is now open

As usual, Eric Siebert, has made an awesome job putting together the annual top vBlog voting for the best virtualization blogs and i just wanted to let you know that the 2015 voting is now open unless you haven’t seen that already.


There are more than 400 blog to select from so you got plenty of alternatives. There are actually voting guidelines available where you are asked to consider the following when you vote:

  • Frequency (how often they post)
  • Quality (how well the posts are)
  • Length (how long the posts are)
  • Longevity (how long they have been blogging for)

However, i just think you should pick the blogs based on the once that helps you in your daily job and provides good news. Doesn’t have to make it harder than that:)

Just click the below vote button, select your top 10 blogs, rank the in your order of preference and move on from there. It will just take 2-5 minutes of your time.


The result is calculated based on the one you rank as number #1 will get 10 points and the one you rank as #10 will get 1 point.
The result will be presented March 19.





Mar 03

VMware vSphere Update Manager service fails

So a few days back i had some problem with the VMware Windows based vSphere Update Manager (VUM) Server that didn’t start after rebooting the virtual machine (VM) where the service runs. All other VMware related services running on the VM started without any problem.

vCenter Server (and VUM) version running was 5.5.U1b was running on Windows Server 2008 R2. Yes i know there are never versions available and i’ll not go into any discussion why this kind of old release was still running in the customers environment.

In Windows event viewer i found the following entry:

  • Level = Error
  • Date and Time = 01/22/2014 3:19:XX AM
  • Source = Service Control Manager
  • Event ID = 7031
  • Task = None
  • Category = The VMware vSphere Update Manager Service service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 300000 milliseconds: Restart the service.

I saw the above entry a couple of times since the auto restart feature tried to start the service.
The first entry was followed by this entry:

  • Level = Information
  • Date and Time = 01/22/2014 3:19:XX AM
  • Source = Service Control Manager
  • Event ID = 7036
  • Task = None
  • Category = The Application Experience service entered the stopped state

The VUM log file vum-server-log4cpp.log, locatedC:ProgramDataVMwareVMware Update ManagerLogs, contained e.g. the following information:

  • [DATE & TIME ‘ManagedJetty’ 5776 INFO] [managedJetty, 313] ManagedJetty Start=
  • [DATE & TIME ‘ManagedJetty’ 5776 ERROR] [managedJetty, 232] Failed to parse C:/Program Files (x86)/VMware/Infrastructure/Update Manager/jetty-vum-ssl.xml. Error: no element found
  • [DATE & TIME ‘VcIntegrity’ 5776 ERROR] [vcIntegrity, 480] Error starting Process monitor: no element found
  • [DATE & TIME ” 2900 ALERT] [logUtil, 265] Product = VMware Update Manager, Version = 5.5.0, Build = 1888318
  • [DATE & TIME ‘VcIntegrity’ 2900 DEBUG] [vcIntegrity, 1111] Getting managed process information from configuration

You’ll see this error when the jetty-vum-ssl.xml is corrupt. Luckily the customer had working backups which could be restored so after we restored the file the VUM service started and everything was back to normal.

Feb 26

Initialize the remote vCenter Orchestrator database


It doesn’t happen every day but once in a while you can immediately answer a question when being called by a customer and yesterday was my lucky day:)
The only requirement i had before answering the question was to get two print screens so i could put together this really short blog post. Here we go.

Customer was standing up a vCenter Orchestrator 5.5.2 (vCO), soon to be renamed to vRealize Orchestrator (vRO), and was trying to connect to an external Microsoft SQL database server version 2012 SP1 using the following settings.

Screen Shot 2015-01-20 at 22.03.04

He received the following error:

  • Database connection successful. No schema found. Please initialize database.

Screen Shot 2015-01-20 at 22.02.51

I must point out that this was the customer first vCO installation but if he had only scrolled up to the top of the page he would have found the solution since you actually got a link that says:

  • Create the database tables

Screen Shot 2015-01-20 at 22.03.10Click the link and the database is initialized and you are ready to start doing your vCO magic.



Feb 24

Horizon View and smart card login

This is a guest blog post created by my former colleague Toni Lindberg who now works for Cygate in Stockholm, Sweden.

Installing VMware Horizon View out of the box is not difficult. However, integrating the Horizon View solution with smart card to increase the security can sometimes cause problems.

This blog post consists of a short guide how to make it right.

Click here to read the entire blog post!

Feb 18

Nutanix Operating System Upgrade to 4.1.1

As you might have seen a new version (4.1.1) of the Nutanix Operating System a.k.a NOS was released a while ago.
See the new features in this blog post.

I have been through the upgrade process a few times now and last week i did one while running a customer workshop. The upgrade process just ran in the upper right corner of my screen so the customer could follow the process during the workshop.
This post will cover the process of upgrading your entire Nutanix Cluster without any downtime for your running virtual machines (VM). Follow the below process to perform the upgrade:

  • Log in to your Nutanix Controller Virtual Machine (CVM) via http://CVM-IP-ADDRESS:9440
    Screen Shot 2015-01-27 at 03.22.12
  • Click the wheel next to the admin link at the top right corner
    Screen Shot 2015-01-27 at 03.29.11
  • Select Upgrade Software
    Screen Shot 2015-01-27 at 03.32.03

Click here to read the entire blog post!

Older posts «