VCSA 6.5 Broken Filesystem - "Welcome to Emergency Mode"

Today I managed to crash the storage used in my home lab. After fixing the FreeNAS box, my vCenter Server Appliance (Version 6.5 Update 1) refused to boot and after opening the console, it welcomed me with the following error message:

Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

Typically, this problem is caused by filesystem issues. This article explains how to fix the filesystem and get the appliance back up.

Please note that you should create a snapshot prior to fix VCSA filesystems and ask VMware GSS for support when you don't know what you are doing.

When opening the console, this message is displayed:

The system complains about services that failed to start, but the main reason has already scrolled out of the screen, so the first thing is to scroll up (Shift + Page Up):

There is a problem with the log filesystem as stated in the first line: "[FAILED] Failed to start Filesystem Check on /dev/log_vg/log.". Scroll back down and log in with the VCSA root password. (If you can't remember the root password, read this post for password recovery instructions.)

Use # df -h to verify that there are no full filesystems and compare the output to the filesystem table (# cat /etc/fstab). As you can see, the /storage/log filesystem is missing in the output from df -h:

So, as already stated in the first error message, there is a problem with the /dev/log_vg/log device. The next step is to run a filesystem check. Use the -y option to automatically fix all issues or press "y" for each identified error.

# e2fsck -y /dev/log_vg/log

When all errors are fixed, reboot vCenter Server Appliance:

# reboot -f

It should now be able to boot the vCenter Service Appliance.

 

  1. Good one

  2. Your post save my day.. Thank you so much..

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>