«

»

Jul 14 2015

vCenter Server won’t start

“VMTurbo"
This is actually the second blog post with the same name. I had the same problem, vCenter Server didn’t start, about a year ago and you can read that blog post here.

This blog post applies to vCenter Server 5.5.0 build 2442329 and the problem was that the vCenter Server service didn’t start after an unexpected reboot of the Windows 2012 R2 virtual machine (VM) running the the vCenter Server service.

While investigating the most recent vCenter Server log file, vpxd-xyz.log, located in the directory C:\ProgramData\VMware\VMware VirtualCenter\Logs i found the following:

“2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] CoreDump: unable to create the dump file: class Vmacore::SystemException(The file or directory is corrupted and unreadable. )”
See the entire log file here

Based on the log file entries i tested the following:

  • Create a file on the C: root directory and it failed.
  • Create a file on my desktop and it failed.
  • Create a file in a lot of different directories on the C: and all the tests failed.
  • Create a file on D: and this was successful

So it was obvious that we had problem with the C: drive but at least some, e.g. vCenter Server, log files could be created:)

I also found a lot of entries in the Windows Server Event Log e.g.:

  • Application log
    Event ID 1000
    Level Error
    Faulting application name: vpxd.exe, version: 5.5.0.43769, time stamp: 0x54b90698
    Faulting module name: KERNELBASE.dll, version: 6.3.9600.17415, time stamp: 0x54505737
    Exception code: 0xe06d7363
    Fault offset: 0x0000000000008b9c
    Faulting process id: 0x1e9c
    Faulting application start time: 0x01d0bc80dd30a512
    Faulting application path: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: 1aec81ff-2874-11e5-80d7-005056a041b3
    Faulting package full name:
    Faulting package-relative application ID:
  • Application Log
    Event ID 1001
    Level Information
    Source Windows Error Reporting
    Fault bucket , type 0
    Event Name: APPCRASH
    Response: Not available
    Cab Id: 0
    Problem signature:
    P1: vpxd.exe
    P2: 5.5.0.43769
    P3: 54b90698
    P4: KERNELBASE.dll
    P5: 6.3.9600.17415
    P6: 54505737
    P7: e06d7363
    P8: 0000000000008b9c

The first action taken to solve the problem was running check disk using the following command:

  • chkdsk C: /F
    This command requires a reboot based on the following command output:
    Chkdsk cannot run because the volume is in use by another
    process. Would you like to schedule this volume to be
    checked the next time the system restarts? (Y/N)

However, during the reboot the check disk ran for a very very short time and didn’t solve my problem so i ran check disk again using the scan option:

  •  chkdsk C: /scan

When finished i could copy files to my C: drive again and the vCenter Server started again so everything is back to normal. Now we just have to figure out why the VM rebooted but that’s another story.

vCenter Server log file

Section for VMware VirtualCenter, pid=1424, version=5.5.0, build=2442329, option=Release

—— In-memory logs start ——–
mem> 2015-07-xxT11:29:31.827+02:00 [04760 info ‘Hooks’] Hooks Initialized

—— In-memory logs end ——–
2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Logging uses fast path: true
2015-07-xxT11:29:31.859+02:00 [04760 info ‘Libs’] VThreadBase detected multiple threads.
2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Handling bora/lib logs with VmaCore facilities
2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Initialized channel manager
2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Current working directory: C:\Windows\system32
2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] ThreadPool windowsStackImmediateCommit = true
2015-07-xxT11:29:31.885+02:00 [04760 info ‘ThreadPool’] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
2015-07-xxT11:29:31.885+02:00 [04760 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.885+02:00 [00828 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Log path: C:\ProgramData\VMware\VMware VirtualCenter\Logs
2015-07-xxT11:29:31.885+02:00 [02408 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Initializing SSL
2015-07-xxT11:29:31.885+02:00 [01836 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.886+02:00 [01352 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.887+02:00 [04760 info ‘Default’] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
2015-07-xxT11:29:31.888+02:00 [04592 info ‘ThreadPool’] Thread enlisted
2015-07-xxT11:29:31.891+02:00 [04760 warning ‘Libs’] FileIOCreateRetry: Unrecognized error code: 1392
2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Win32 Exception (3765269347) detected at 00007ff8f01c8b9c
–>
2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] rip: 140707157019548 rsp: 86765840 rbp: 0
–>
2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] rax: 140707201861943 rbx: 86766208 rcx: 60162048
–> rdx: 0 rdi: 6447380920 rsi: 19301800
–> r8: 1761332359 r9: 34772056277007 r10: 23362607753396275
–> r11: 2207616008235 r12: 1 r13: 0
–> r14: 0 r15: 0
–>
2015-07-xxT11:29:31.893+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Backtrace
–> backtrace[00] rip 000000018018b7fa
–> backtrace[01] rip 0000000180104c78
–> backtrace[02] rip 0000000180105f6e
–> backtrace[03] rip 00007ff7936adada
–> backtrace[04] rip 00007ff8f02a1c72
–> backtrace[05] rip 00007ff8f2d01328
–> backtrace[06] rip 00007ff8f2ce2456
–> backtrace[07] rip 00007ff8f2cf3acd
–> backtrace[08] rip 00007ff8f2cb8407
–> backtrace[09] rip 00007ff8f2cb762d
–> backtrace[10] rip 00007ff8f01c8b9c
–> backtrace[11] rip 0000000068fb54ca
–> backtrace[12] rip 00000001800fac8f
–> backtrace[13] rip 0000000180112e0c
–> backtrace[14] rip 0000000180113830
–> backtrace[15] rip 00007ff7935d9c92
–> backtrace[16] rip 00007ff7933befa2
–> backtrace[17] rip 00007ff7935f7c9b
–> backtrace[18] rip 00007ff8f1ba7f2a
–> backtrace[19] rip 00007ff8f05013d2
–> backtrace[20] rip 00007ff8f2cce954
–>
2015-07-xxT11:29:31.893+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Generating minidump …
2015-07-xxT11:29:31.894+02:00 [04760 warning ‘Libs’] FileIOCreateRetry: Unrecognized error code: 1392
2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] CoreDump: unable to create the dump file: class Vmacore::SystemException(The file or directory is corrupted and unreadable. )
2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] (Log recursion level 2) VpxUnhandledException

—— In-memory logs start ——–
mem> 2015-07-xxT11:29:31.827+02:00 [04760 info ‘Hooks’] Hooks Initialized
mem> 2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Section for VMware VirtualCenter, pid=1424, version=5.5.0, build=2442329, option=Release
mem> –>
mem> 2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Logging uses fast path: true
mem> 2015-07-xxT11:29:31.859+02:00 [04760 info ‘Libs’] VThreadBase detected multiple threads.
mem> 2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Handling bora/lib logs with VmaCore facilities
mem> 2015-07-xxT11:29:31.859+02:00 [04760 info ‘Default’] Initialized channel manager
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Current working directory: C:\Windows\system32
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] ThreadPool windowsStackImmediateCommit = true
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘ThreadPool’] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.885+02:00 [00828 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Log path: C:\ProgramData\VMware\VMware VirtualCenter\Logs
mem> 2015-07-xxT11:29:31.885+02:00 [02408 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.885+02:00 [04760 info ‘Default’] Initializing SSL
mem> 2015-07-xxT11:29:31.885+02:00 [01836 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.886+02:00 [01352 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.887+02:00 [04760 info ‘Default’] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
mem> 2015-07-xxT11:29:31.888+02:00 [04592 info ‘ThreadPool’] Thread enlisted
mem> 2015-07-xxT11:29:31.890+02:00 [04760 verbose ‘vpxdvpxdSupportManager’] Uptime Log: C:\ProgramData\VMware\VMware VirtualCenter\Logs/1436693371889642.vpxduptime
mem> 2015-07-xxT11:29:31.891+02:00 [04760 verbose ‘vpxdvpxdSupportManager’] Moved 0 uptime log(s) to cache dir
mem> 2015-07-xxT11:29:31.891+02:00 [04760 warning ‘Libs’] FileIOCreateRetry: Unrecognized error code: 1392
mem> 2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Win32 Exception (3765269347) detected at 00007ff8f01c8b9c
mem> –>
mem> 2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] rip: 140707157019548 rsp: 86765840 rbp: 0
mem> –>
mem> 2015-07-xxT11:29:31.891+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] rax: 140707201861943 rbx: 86766208 rcx: 60162048
mem> –> rdx: 0 rdi: 6447380920 rsi: 19301800
mem> –> r8: 1761332359 r9: 34772056277007 r10: 23362607753396275
mem> –> r11: 2207616008235 r12: 1 r13: 0
mem> –> r14: 0 r15: 0
mem> –>
mem> 2015-07-xxT11:29:31.893+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Backtrace
mem> –> backtrace[00] rip 000000018018b7fa
mem> –> backtrace[01] rip 0000000180104c78
mem> –> backtrace[02] rip 0000000180105f6e
mem> –> backtrace[03] rip 00007ff7936adada
mem> –> backtrace[04] rip 00007ff8f02a1c72
mem> –> backtrace[05] rip 00007ff8f2d01328
mem> –> backtrace[06] rip 00007ff8f2ce2456
mem> –> backtrace[07] rip 00007ff8f2cf3acd
mem> –> backtrace[08] rip 00007ff8f2cb8407
mem> –> backtrace[09] rip 00007ff8f2cb762d
mem> –> backtrace[10] rip 00007ff8f01c8b9c
mem> –> backtrace[11] rip 0000000068fb54ca
mem> –> backtrace[12] rip 00000001800fac8f
mem> –> backtrace[13] rip 0000000180112e0c
mem> –> backtrace[14] rip 0000000180113830
mem> –> backtrace[15] rip 00007ff7935d9c92
mem> –> backtrace[16] rip 00007ff7933befa2
mem> –> backtrace[17] rip 00007ff7935f7c9b
mem> –> backtrace[18] rip 00007ff8f1ba7f2a
mem> –> backtrace[19] rip 00007ff8f05013d2
mem> –> backtrace[20] rip 00007ff8f2cce954
mem> –>
mem> 2015-07-xxT11:29:31.893+02:00 [04760 warning ‘win32vpxdOsLayer_win32’] [VpxUnhandledException] Generating minidump …
mem> 2015-07-xxT11:29:31.894+02:00 [04760 warning ‘Libs’] FileIOCreateRetry: Unrecognized error code: 1392
mem> 2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] CoreDump: unable to create the dump file: class Vmacore::SystemException(The file or directory is corrupted and unreadable. )
mem> 2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] (Log recursion level 2) VpxUnhandledException

—— In-memory logs end ——–
2015-07-xxT11:29:31.894+02:00 [04760 error ‘Default’] Section for VMware VirtualCenter, pid=1424, version=5.5.0, build=2442329, option=Release
–>

4 comments

Skip to comment form

  1. ggholl

    The first thing you do is have full support with vmware and then you contact them with the log files. I suspect from the sound of things you had a corrupt C Drive. Chkdsk /f will automatically fix any hardware errors on the actual drive. It sounds like the section of the SAN for the local storage that you had that via Milan had corrupt sectors. Next time contact vmware technical support upload the log files after you open a support ticket. These blog posts are not for finding out solutions you should go through vmware solutions support

  2. magander3

    Hi,
    well i usually blog about my challenges during my daily work and this was absolutely one of them. I’m not telling anyone to follow what i do, i just present the solutions to my problem and if it helps anyone i’m just happy.
    You’re right that you should absolutely file a service request with the software vendor when you got a problem.

    //Magnus

  3. sowhite

    Check your vcenter DB disk space.

  4. magander3

    Hi,
    disk space was not the problem. solution provided in the blog post

    thanks

    //Magnus

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">