VMware Snapshots over SSH

[pierreschmitz]

Hello everyone,

in TrueNAS the VMware snapshot feature ist connected to the VMware storage API which is only available on paid versions of ESXi. A home lab combination of TrueNAS Core and VMware ESXi Free as a central datastore doesn't make much sense. So I ended up storing my virtual machines on the ESXi host and tried to figure out, how I can backup my machines to TrueNAS.

I found the community project ghettoVCB:

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

Broadcom Community - VMTN, Mainframe, Symantec, Carbon Black

communities.vmware.com

This piece of software is installed as a packet on the host and communicates over ssh to take snapshpots of the VMs to have a consistent state for the backup. Then it copies the data to another location. The storage API is not needed at all and it works fine with the free version of ESXi.

Now I wondered, if it was possible, to do a similar thing in TrueNAS. The storage had to connect to ESXi over SSH, take a snapshot of the VMs, then take a snapshot of his own zvol or whatever and afterwards delete the snapshot on the ESXi. This could be automated over cron.

What do you think, does this make any sense or do you see any possible issues? Unfortunately I'm a total beginner on TrueNAS and can't do this by myself.

Cheers
Pierre

 

[pierreschmitz]

By the way: The Storage API is available in ESXi free, but only in read-only mode. So, a list of the registered datastores can be retrieved.

 

[jgreco]

ghettoVCB is designed to run on the hypervisor itself, and typically backs up to an NFS-mounted share from your storage. SSH doesn't play any role at all in that, except to install gVCB in the first place. It takes a snapshot of each VM, copies that to NFS, and then most of the time it removes the snapshot, except when it doesn't, because in practice this process sucks a bit.

There are lots of things that you could theoretically do on Free ESXi for backing stuff up, if only you had someone to author them for you. Nothing precludes you from writing software to do ghettoVCB-like snapshotting and copying the files off via SSH, except that you say you "can't do this by [yourself]", and this isn't a beginner-level project. Most people end up using ghettoVCB, or using one of the hacky ways to keep ESXi running in registered mode so that they can use a product that deals with the storage API, such as Veeam. This includes signing up for VMUG eval experience licensing (not for production use) or perpetually locking in evaluation mode (almost certainly a licensing violation), and then acquiring a Veeam NFR.

 

[pierreschmitz]

To simplify the task, maybe TrueNAS and ESXi do not have to communicate with each other at all. If ESXi could perform a VM snapshot at a given time and wait 5 minutes so that TrueNAS can do a snapshot of his zvol and after that ESXi deleted his snapshot, then everything would be fine. If both systems take their time from the same internal (LAN) NTP server a cron-based scheduling on each system should work. Maybe ghettoVCB could be used to take the snapshot and since the backup functionality is not needed, the backup volume could be configured to /dev/null. But then all the data would be transferred from TrueNAS over FC to the host for nothing. A ghettoVCB-like ESXi packet, which only takes snapshots, would be optimal. Regularly, scheduled VM snapshots are only availabe in vCenter.

 

[jgreco]

ESXi snapshots take a nonzero amount of time, and generally need to be done on a per-VM basis, so asynchronous operation trying to align with cron would be very dicey. Restoration is also somewhat complicated when snapshots are involved.

 

[NugentS]

Back in VMWare 3 - I used the VMware API's to write a script that:

1. Made a list of the available VM's on the host
2. Snapshot the guest
3. Copy the snapshot to another location
4. Remove the snapshot

Given how much I knew about scripting on linux (next to nothing) I was amazed it worked as well as it did. It was crude and basic but it did me for a while. All the details needed are in the API documents that VMWare publish

It had a couple of features like if the machine name (as per VMWare) has a space in it then that machine didn't backup and it really didn't handle script errors well. Then I found ghetto backup which does exactly the same thing (just without so many features). It works though restore is a matter of working out what to do and typing the right command in. This is done at the esxi level and has nothing to do with the storage system other than esxi uses it. You define the machines to backup and where to put backups.

The only issue really is that each backup is a complete backup - there is no incremental / differential - each backup is the whole VM

 

[pierreschmitz]

So, I think I figured it out. It is really simple (please feel free to criticize, sometimes the simplest solution isn't the best), but it is the only way I know.

First I enabled the SSH Service in ESXi with automatic service start at bootup. Then I created a SSH Keypair of the root account on TrueNAS Core. After that I stored the Public Key on ESXi and from then on I could connect from the TrueNAS root shell to ESXi root without a password. Pretty standard.

SSH access to ESXi with public keys:

https://kb.vmware.com/s/article/1002866

Now I just have to write a shell script, that logs into ESXi, takes the snapshots, logs out, takes the TrueNAS snapshot(s), logs in again, deletes the VM snapshots and done. This script can be scheduled over the Tasks function in the TrueNAS menu.

How to create VM snapshots in ESXi:

blog :: Create snapshots via commandline in ESXi 5.5

How to take ZFS Snapshots (sorry in German, but thats a general documentation file, users familiar to ZFS know this):

https://docs.oracle.com/cd/E19253-01/820-2313/6ndu3p9cf/index.html

More is not needed, or did I get it wrong? Are there possible security concerns? The only disadvantage I see, is that the script has to be edited by hand every time a new VM is set up or something changes. But I think for a home lab environment I can live with that.

 

[NugentS]

Why do you need to do this?
I don't even use the VMWare Snapshot feature in TrueNAS - mostly cos everytime I try to configure it it doesn't work - and everything works without it anyway

 

[pierreschmitz]

Crash consistent snapshots are always an option, but this is not the way it should be done.