Log Disk Exhaustion on VCSA 7 Fix

Keeping a “happy” vCenter Server in your vSphere environment is crucial to having a vSphere environment that operates correctly and smoothly.  Space issues are definitely an issue that can cause problems.  I wanted to shed light on a problem I have seen a couple of times in my home lab environment with my VCSA 7.  After logging into the vSphere Client,  I have seen the message Log Disk Exhaustion on VCSA.  What is the error?  In this blog, I will walk you through triaging and resolving this issue with disk space on your vCenter Server VCSA appliance, so we can take a look at Log Disk Exhaustion on VCSA 7 Fix.   

Log Disk Exhaustion on VCSA 7 Update 2 Error

Have you seen this in your environment before with your VCSA appliance? You will probably see many posts, articles, and other resources in this case out there with others having the same issue. In the original blog post for the lab environment, I was running VCSA 7 Update 2.  Note the message in the screenshot below. You will see this error below. Note, this is not telling us we have an issue on a vSphere datastore (storage volume) or vSAN volume, but rather on a partition physically attached to vCenter Server (vCenter appliance).

By default, the error messages begin when disks are 80% full. When you see this, it means you need to give attention to space as it can certainly affect the performance of your VCSA appliance.

A bit of Googling led to the following VMware KB. You can find that support KB here: vCenter Server Appliance disk space is full (76563) (vmware.com). The official VMware Knowledge base article describes the steps to find the root cause of the issue with storage utilization in the vCenter Server in your VMware virtualization environment and begin troubleshooting. 

• Partitions are not sized correctly
• Large files in the partitions
• Tables in the database consuming space
• Services are failing to clean up files

For my VCSA appliance, I was using the standard, out-of-the-box partition sizes that come with a fresh deployment.  I don’t really think I am seeing an issue with the services cleaning up log space as I think the logs are simply accumulating and becoming large enough before being pruned that it is flagging the warning in the VCSA vSphere Client.  However, I wanted to clean up the space for the environment.

Steps to perform before troubleshooting 

I definitely recommend that you have good backups of your VCSA appliance before you begin performing any steps below for disk usage maintenance on your vCenter host. What are the best ways to backup your VCSA appliance?

1. One of the first things I do is take a snapshot of the VCSA appliance from the ESXi host, not from vCenter Server. So login to the host that has ownership of your VCSA virtual machine and take a quick snapshot.
2. You can also have a file-level backup of your vCenter Server. You can take this from the VAMI management interface
3. Have a full virtual machine backup using something like Veeam, NAKIVO, BDRSuite, etc. In a break glass situation, you can restore the full VM back to the cluster and be good to go.

Log Disk Exhaustion on VCSA 7 Fix and Solutions

To find where the disk space is being used, you can start by establishing an SSH connection to your VCSA appliance and use the df -h Linux command to verify which partition is having an issue.  On a Windows machine, you can use Putty. As suspected with the error message, the path /storage/log partition is the one having issues. However, each environment may have different partitions that are causing an issue. The methods and tips below though can apply to any of the vCenter partition hard disk configurations. 

Next, to start looking at where the storage space is being used specifically in the /storage/log partition, you can access the shell and issue the command from the shell and press enter:

du -a /storage/log | sort -n -r | head -n 20

By looking at the directory structure listed above by size, I drilled down into the /storage/log/vmware/eam/web partition and found quite a few .log files that were 8 MB a piece for the ESX Agent Manager logging directory.  After removing most if not all, I was able to reclaim a couple of gigs worth of space.

Next directory up for scrutiny is the /storage/log/vmware/lookupsvc directory listed in the output earlier.  In my environment, below is a screenshot of the log files that I had accumulated under the directory.  While not sizeable in and of themselves, combined with the sheer number, these were taking a good chunk of disk space also.    

Under the same directory another level deep is the tomcat directory.  Under the tomcat directory, I had another set of fairly numerous log files that had accumulated here.  Again, I axed the ones that were older and kept just the most recent.  This resulted in even more space gained back.  

After going through manually pruning the files listed in the directories above, I was able to reclaim quite a bit of space and now had over 4 gigs free compared to the under 1 gig free that I had when I started the exercise.  

VCSA Log Disk Full Video

Increase the size of a VCSA hard disk

There is a supported way to increase the size of the partitions on your VCSA. Note the following steps:

To identify the logical volume mapping to a partition, execute the following command:

lsblk -o NAME,HCTL,MOUNTPOINT,SIZE

For obtaining the detailed PCI device paths for each block device, use:

ls -al /sys/block/sd*

To pinpoint the partition requiring expansion, take, for example, /storage/lvm_snapshot which corresponds to the logical volume /sys/block/sdq. Ascertain the SCSI label for the related PCI device by executing:

cat /sys/devices/pci0000:00/0000:00:11.0/0000:02:00.0/label

Should the device path identified in the previous step lack a serial number format preceding the host directory (at the fourth directory level), it signifies mapping to SCSI0. For instance, /sys/devices/pci0000:00/0000:00:10.0/ is associated with SCSI0.

Understanding that the partition /sys/block/sdq aligns with SCSI device 1 and Unit number 1, derived from the third segment of HCTL from the initial step’s output, it is mapped as SCSI(1:1).

To change disk space through the BASH Shell, ensure no snapshot is attached to the vCenter Server VM, then:

SSH into the vCenter Server Appliance using root credentials.

Activate the Bash shell with:

shell.set --enabled true

Enter shell and press Enter.

Identify the disk with capacity issues by executing:

df -h; lsblk; lsscsi

If, for instance, /storage/log correlates with sde and its SCSI ID is SCSI(0:4), this information will be mirrored in the vsphere client’s virtual machine settings as the disk under SCSI(0:4).

To expand the disk space of the vCenter Server Appliance virtual machine, use the vSphere Client or vSphere Web Client.

Following the disk expansion, execute this script within the SSH session to automatically enlarge any logical volumes with extended physical volumes:

/usr/lib/applmgmt/support/scripts/autogrow.sh

Verify the successful disk enlargement with:

df -h

For disk space augmentation via the Appliance Management CLI:

SSH into the vCenter Server Appliance with root credentials.

Identify the disk facing capacity constraints. For vCenter Server Appliance versions:

6.5 and 6.7, execute:

com.vmware.appliance.version1.resources.storage.stats.list

Post disk expansion, to automatically resize logical volumes on the extended physical volumes for vCenter Server Appliance versions 6.5 and 6.7, run:

com.vmware.appliance.system.storage.resize

Lastly, re-run the storage stats list command for confirmation:

com.vmware.appliance.version1.resources.storage.stats.list

Monitoring

You definitely want to monitor your VMware vCenter Server with tools that can shed light on disk space issues when the capacity on your partitions is becoming exhausted or you reach a certain threshold on your disk resource. Be sure to setup an alert you can trigger in this scenario when disk usage has grown. You can use vRealize Operations (now VMware Aria Operations) or other tools like VMware Skyline. Plus, you get the benefit of other automation from these tools.

Wrapping up VCSA log disk exhaustion fix

If you get the message in the vSphere Client, Log Disk Exhaustion on VCSA, like me you are probably wondering what is going on with the log directory. Hopefully, the content here will help you to pinpoint the issue with vCenter and if you need to expand disk size in the VM settings. The Log Disk Exhaustion on VCSA 7 Fix for me was going through the directories listed in the output of the “du -a” command and finding directories there that had many log files contained in the directories. I didn’t really find a silver bullet 1 or two files that were generated and causing the disk space issues.

For me, it was just a combination of log files in the storage/log partition that was the culprit. Hopefully, in addition to the VMware KB article referenced in the post, this will help any who want to get past this error and issue on their VCSA appliance. Let me know if you have seen this issue in the comments below. Or, do you use Proxmox, Hyper-V, XCP-ng, Nutanix, or another solution? I would be curious to know that also. If you have any questions or suggestions let me know in a comment or the VHT forums.