Best Practices to backup VMware vSphere Virtual Machines
While containers and containerized workloads are certainly gaining momentum with organizations redesigning business-critical applications for the future, the truth is, virtual machines are here to stay for the foreseeable future. Containers and VMs work well together for most critical IT infrastructure. With this post, I want to get back to the basics of how we protect virtual machines in VMware vSphere. At the end of the day, backing up VMs and the data, services, and applications they contain is vitally important. This post will consider the best practices to backup VMware vSphere virtual machines and see what considerations need to be made to ensure your VMs are protected appropriately.
Best Practices to backup VMware vSphere Virtual Machines
Let’s take a look at several different best practice considerations that need to be made when backing up and recovering VMware vSphere virtual machines. We will discuss the following:
- Understand RPO and RTO and how these relate to recovery from backup
- Understand what constitutes a backup and what does not
- Use changed block tracking for backing up VMs
- Follow the 3-2-1 backup best practice methodology
- Don’t forget about the security of your backups
- Evaluate virtual machine housekeeping
- Use application-aware backups
- Keep up with the latest vSphere releases
- Leverage the cloud as an offsite storage location
- Protect against ransomware
1. Understand RPO and RTO and how these relate to recovery from backup
All too often organizations configure their backups without any consideration RPO and RTO. What are these? I can just about guarantee you, you will not read documentation regarding backup solutions without running into these two terms. RPO or Restore Point Objective, simply put, determines how much data loss the business is okay with. In other words, if you set your backups of a particular VM to daily backups, the worst case scenario would be that you could potentially lose 24 hours worth of data. Businesses must determine if this loss of data is acceptable. Following suite a backup scheduled for every six hours would be a potential for 6 hours of data loss, so on and so forth.
Setting VM backup schedules should not be an arbitrary, “stick our finger up to the wind,” and decide how often VMs should be backed up. This should be given careful consideration from the business perspective to determine what the acceptable loss should be. What about the RTO?
The RTO is the Restore Time Objective. This determines how much time it will take to recover your virtual machine. If you have your backups configured as hourly backups, you may only lose one hour’s worth of data. However, due to the amount of data, the time it takes to restore that VM might take three hours. Restore Time Objective defines what the acceptable amount of time your business can go without the data specified in the RPO.
Understanding both of these values in relation to your individual business is absolutely critical when thinking about the best practices to backup VMware vSphere virtual machines. There is no right or wrong answer for each business and these will most likely be different for every organization.
2. Understand what constitutes a backup and what does not
There are so many times that I have seen clients I have worked with and IT admins who assume they have what they consider to be a “backup” only to find it was not truly a backup. One of the most common scenarios that fits this description is viewing VMware vSphere virtual machine snapshots as backups. I will be the first to tell you that snapshots have their place and can be very valuable to the business in many scenarios. However, like everything else that you have read, snapshots are not backups. Why?
Let’s think about what a true backup really is. Backups should be a totally standalone copy of your virtual machine that allows restoring that virtual machine with no reliance on the production infrastructure. This is not the case with a VMware vSphere snapshot. Consider the fact that a VMware vSphere snapshot is made up of a chain of delta disks that rely on one another to make a complete copy of your data. If anything happens to one of the disks in the chain, the VM is toast as well as the snapshot. In that case, you can’t rely on the snapshot as a backup since it is not a full copy of the data. Also, it is not a standalone copy separate from your production infrastructure. If something happens to the physical infrastructure your production VMs are housed on, this means your VM, including the snapshot, is gone. Again, backups should not rely on production infrastructure.
3. Use changed block tracking for backing up VMs
Back in the old days of backups, each time a backup ran, it may have been configured to grab a full copy of the data. This is incredibly inefficient, both in terms of the backup time required and the backup storage space needed to store the multiple full copies of the data. A much more efficient way to backup data is to only copy the changes that have occurred since the last backup. By doing this, the backup is incredibly more efficient. The actual changes or additional data may be miniscule by comparison to the entire bulk of data.
One of the capabilities of the vSphere Storage APIs for data protection is Changed Block Tracking (CBT). What is CBT? Changed Block Tracking (CBT) is a VMkernel feature that keeps track of the storage blocks of virtual machines as they change over time. The VMkernel keeps track of block changes on virtual machines, which enhances the backup process for applications that have been developed to take advantage of VMware’s vStorage APIs. VMware vSphere keeps track of the changed blocks that have happened to the virtual machine. The backup solution can then take advantage of this information to copy only the changed blocks each time the backup of the virtual machine runs.
This leads to many benefits, including greatly reduced backup windows as well as much less backup storage needed for backing up virtual machines. One of the things that you want to make sure of when you are targeting a virtual machine for backup with a backup solution, CBT cannot be enabled on VMs that have snapshots present or that are powered off. This provides another very important reason to make sure you are monitoring and pruning snapshots in your environment. It is a best practice to do this anyway, and it leads to tremendous benefits for your VMware vSphere virtual infrastructure, including backups.
4. Follow the 3-2-1 backup best practice methodology
There is an industry best practice methodology that you have no doubt heard of or seen mentioned – the 3-2-1 backup strategy. What is it? It is a best practice backup methodology that helps to design your backups in such a way that you always have multiple copies of your data, stored in a protected manner. The 3-2-1 backup strategy helps to ensure that it would be extremely unlikely, if not impossible, for you to lose all copies of your production data.
The 3-2-1 backup strategy recommends having (3) copies of your data, stored on at least (2) different types of media, with at least (1) copy stored offsite. As you can see by this description, there is storage diversity mandated by these principles. First, you have multiple copies of your data. You have those multiple copies stored on different media types. This may include storing backups on both hard disk and tape media. Finally, you have at least one copy of the backup stored offsite. This ensures that if all other copies of data are lost on-premises, you will have another copy of your data that can be used to recover.
Many businesses today are leveraging the cloud for this aspect of the 3-2-1 backup strategy. Cloud storage is a cheap, efficient, and effective storage location that allows keeping a copy of your data offsite. This helps to prevent a scenario such as can happen with ransomware, where ransomware infects all storage locations on-premises. It may even encrypt all the copies of your backups. Having the offsite storage location in the cloud helps to ensure there is a copy of your data safe from these types of risks.
5. Don’t forget about the security of your backups
When creating and architecting your backups, do not forget about security. Securing your backups is vitally important. When you think about the data contained in backups, it is production data. If someone were to get access to backups containing production data, they essentially have compromised your production environment.
Encrypting your backups should be a standard practice in 2021. If you are not doing this, or you have a backup solution that can’t do this, you need to look elsewhere, period. Also, make sure encryption is happening both in-flight and at rest. Even if you are encrypting data at rest, without in-flight encryption, it is plainly viewable as it crosses the network.
When it comes to storing tape media, pay attention to the physical security of the storage location. You also don’t want a situation where it is easy for someone to walk in unattended and carry off a set of tapes.
6. Evaluate virtual machine housekeeping
As a VMware vSphere environment continues to grow, you can certainly have virtual machine sprawl in the environment. This sprawl can also affect your backups. Keeping a lean vSphere estate helps to ensure you are not backing up anything that does not need to be backed up, and you are not keeping around backup data that is no longer needed.
Also, when talking about virtual machine housekeeping, make sure to keep your VMware vSphere virtual machines free of ad-hoc snapshots as well. Keeping virtual disks tidy helps to reduce the likelihood of corruption and other unwanted side effects. Modern backup solutions leverage snapshots to redirect the I/O from the base disk so the data can be copied to the backup. If you already have snapshots on your VM that is the target of a backup, the backup solution is creating yet another snapshot on top of the existing snapshots. This can further degrade performance and increases the risks of snapshots not rolling off correctly under high load and other scenarios.
7. Use application-aware backups
For most businesses, the majority of VMs that are in production are generally running some type of application that is served out to end-users. Many of these may be database related. When thinking about best practices to backup VMware vSphere virtual machines, using application-aware backups should be the standard you use for backing up these types of VMs. Why?
Application-aware backups ensure that all data is flushed to disk from pending I/O operations and from memory to ensure the data copied from the disk is consistent. Without using application-aware backups, you are simply getting a crash consistent backup that could potentially have data or consistency corruption. Also, by using application-aware backups by your backup solution of choice, you generally have the ability to perform granular restore operations for databases such as restoring a single database or even a single table in a database. All of these features with application-aware backups help to reduce the Restore Time Objective when there is a data loss event.
Another benefit of application-aware backups is they can perform database housekeeping automatically. Instead of having to create a separate SQL job to truncate your logs, you can simply have the backup solution with application-aware backups enabled, truncate the DBs for you.
8. Keep up with the latest vSphere updates
You may wonder, what does keeping up with the latest vSphere updates have to do with backups? For one, keeping your vSphere environment up-to-date is a general best practice across the board. It just helps to ensure that things are running smoothly. However, it also helps to ensure you are benefiting from the latest improvements in terms of performance and other tweaks. VMware has done a lot of work to make things more efficient in the latest releases of vSphere, including with snapshot technology.
Having recent versions of vSphere ensures you are benefiting from these improvements with your data protection solution. One word of caution here. While staying up-to-date is recommended, always make sure your data protection solution is compatible with the latest version of vSphere. Generally, they are a little behind the curve when a new release of vSphere drops.
9. Leverage the cloud as an offsite storage location
In thinking about your 3-2-1 backup strategy, it is a good idea to think about your storage locations and where you are housing your backup data. There is no question that most organizations are using cloud storage for many different use cases. It makes a lot of sense for backup storage as it is relatively cheap, virtually unlimited, scalable, and elastic. Your business does not have to provision, maintain, and continually allocate physical infrastructure to satisfy backup storage needs. This helps to ensure that physical backup storage is not a blocker to effective and efficient backups.
Cloud storage from various cloud service providers also has great features built-in such as immutable backups. This helps to protect against ransomware. This relates to our next section.
10. Protect against ransomware
Ransomware has become a huge issue for businesses today. As was seen in the Colonial Pipeline hack and the JBS meat packing attack, it can shut down and affect critical services that can take days if not weeks to recover from. The fallout from devastating ransomware attacks is felt across entire nations. Protecting your VMware vSphere virtual machine backups against ransomware is a must.
Make sure your backup environment is air-gapped in a sense, either by credentials, or low-level file access from the main production network environment. If malicious processes can’t connect to or have the permissions to access, it protects those backups from being encrypted. Attackers know if they have encrypted your backups, you have no choice but to pay the ransom demanded. They even target various vendor-specific backup files to encrypt these.
In your 3-2-1 backup strategy, make sure you have the technology solutions and best practices in play that allow protecting backups at all costs.
Wrapping Up
As we have discussed, there are certainly best practices to backup VMware vSphere Virtual Machines that need to be considered. There may be others that are more specific to your organization. The backup solution you choose can also have a big part to play in capabilities that align with best practices. NAKIVO Backup & Replication is the solution I use in the lab to backup critical VMs and applications. It provides the key functionality to align with best practices such as changed block tracking support, replication & backup copies to align with 3-2-1, and cloud storage support. It also sports encrypted backups, backup testing, and immutable backups with Amazon S3 object lock. They are also one of the best to provide beta and free trial access to the NAKIVO appliance which literally takes a couple of minutes to deploy.
Download and learn more about NAKIVO here: Full-Featured Free Trial – VMware | Hyper-V | AWS EC2 (nakivo.com)
