Wednesday, August 24, 2022

Create sparsebundle image manually to speed up Time Machine Backup process

Just found Time Machine backup is getting slower and revised what can be done on sparse image format. Here we come with the manual method to create sparsebundle image faster enough for Time Machine backup on network shared drive.

Here's the syntax:

$ hdiutil create -size 1024GB -type SPARSEBUNDLE -encryption AES-128 -nospotlight -fs "Journaled HFS+" -imagekey sparse-band-size=262144 -verbose -volname "Backup_X" Backup_X.sparsebundle

Note:
-imagekey sparse-band-size=size can be used to specify the number of 512-byte sectors that will be added each time the image grows. Valid values for SPARSEBUNDLE range from 2048 to 16777216 sectors (1 MB to 8 GB). 
sparse-band-size represents the size of the ‘chunks’ that make up the sparseimage (they aren’t just one single file). Normally, the images are made in 8M chunks but those perform quite poorly over the network. The value provided above represents 128M chunks (2 * 128 * 1024 512-Byte-Blocks = 262144 = 128MB) which is a good size for a hard disk backup.

Ref:
https://github.com/0xdevalias/devalias.net/issues/89
https://mike.peay.us/blog/archives/248
https://eclecticlight.co/2020/06/18/selecting-sizes-for-sparse-bundles/
https://ss64.com/osx/hdiutil.html

Friday, August 19, 2022

Extend the storage of WD MyCloud with external USB hard drive for Mac Users

Home backup solution WD MyCloud provides simple approach to extend the size of storage by attaching additional external hard drive through its USB port. 

First, take a look at what partition format of USB hard drive is supported by WD MyCloud:

My Cloud: External USB Drive Supported File Systems


The table lists the USB drive file systems supported on the My Cloud products.
Note:
  • exFAT support requires firmware version 2.21.111 and higher
  • exFAT with EFI System Partitions are not supported and will result in "Unable to mount USB device" message
    My Cloud device
    NTFS
    HFS+
    FAT32
    exFAT
    My Cloud
    My Cloud Mirror
    My Cloud Mirror Gen2
    My Cloud EX2
    My Cloud EX4
    My Cloud EX2 Ultra✔ *
    My Cloud EX2100✔ *
    My Cloud EX4100✔ *
    My Cloud DL2100✔ *
    My Cloud DL4100✔ *
    My Cloud PR2100✔ *
    My Cloud PR4100✔ *

The best Partition Format of all?

Choosing a partition format for external USB hard drive can be confusing for home users. Consider yourself using Windows or MacOS, the type of filesystem format would have impact on daily backup and the maintenance in case of drive failure. Default format of WD products (external USB hard drive) mostly comes with NTFS partition. Windows users may not aware of such difference until they switch to use Mac computer for home use. 

MacOS doesn't come with stable support (for free, I mean) of READ/WRITE operations to NTFS partition. So, the best bet is to reformat the external drive to HFS+ for better support. If you mainly work on MacOS platform then HFS+ would be your best choice. This takes consideration of easier maintenance and recovery using Apple computer.

People may ask if FAT32 or exFAT would be more appropriate for cross platform support among Windows and Mac computers. FAT32 partition is limit to store files whereas each file has a limit in size less than 4GB. exFAT doesn't have such limit. However, exFAT is not journaled filesystem so the chances of data loss in case of power outage is bigger than journaled filesystem like NTFS and HFS+. We are talking about hard drive running on 24/7 basis.

For reliable choices, we have now limited to NTFS and HFS+. Luckily, nearly all WD MyCloud products support both of these journaled filesystems. For Mac Users, HFS+ is definitely a good choice for extending NAS storage. In case of MyCloud server failure, the chances to recover data from HFS+ partition is higher than that from NTFS partition.

HFS+ is so far the most reliable filesystem with Time Machine support across different MacOS versions. Newer filesystem APFS has just officially supported since Big Sur. Older MyCloud products don't even support APFS. For backward compatibility, HFS+ is still the best choice as storage media.

What about Multiple Time Machine backups on WD MyCloud?

WD MyCloud has builtin support of Time Machine shared folder to accomodate more than one backup at a time. So if you have more than one computer required for home backup, you can opt for the builtin Time Machine feature on WD MyCloud server. The only exception is that you have no control over the size limit of each backup. WD MyCloud uses one single folder for Time Machine Backup for all. When the space runs out, whoever which backup in next operation may start to delete its older snapshots.

If you want more control over Time Machine backup, then you may want to manually create backup image file/folder for use. This way you can set a size limit to the image file/folder. The type of backup image option should be *.sparsebundle which is commonly used by Time Machine operations.

Once external HFS+ formatted drive is connected to WD MyCloud, it will hook up as USB drive and the drive would appear on Finder as new drive. Furthermore, you may protect the drive with password secured access on WD MyCloud web interface. This HFS+ format is just for storage level. If you are considering using APFS or other filesystem, then you should keep reading. We are going to create the actual backup image in its preferred filesystem afterwards.

Manually create *.sparsebundle on network shared drive


For better control over backup size limit, you must create *.sparsebundle image through Disk Utility tool. Click [File] > [New Image] > [Blank Image...] and then select a location on network shared drive to store your backup image.

For backup format, you will have many more options, i.e., Mac OS Extended (Journaled), APFS or even exFAT. This is the filesystem for the actual Time Machine Backup. 

For Image Format, you can simply pick [sparse bundle disk image]. 

For Partitions option, please select [Single partition - GUID Partition Map] for compatibility with Time Machine requirements. 

Finally, you must set the Size to your desired value. The backup image size must be smaller than the total space available on network shared drive. 

For further security, you can select Encryption like 128/256 bit encryption and set your password.

Once you create *.sparsebundle image successfully. A new drive linked to this image will appear on Finder.

Tell Time Machine where to find the new backup location

You can set destination with Time Machine command line like this:


$ sudo tmutil setdestination /Volumes/YOUR_TIMEMACHINE_BACKUP_DRIVE

Make network shared volume automatically mounted

To make the mounted volume surviving at the next boot time:

  1. Go to System Preferences
  2. Click Users and Groups
  3. Select your user and then click Login Items
  4. Click the + button and then choose the blank image (.sparsebundle) created in step 1 above
  5. Repeat step 4 and choose the target volume so it will be mounted automatically.



Ref.:

The way to manage multiple copies of Time Machine Backups is different than builtin setup of TimeMachineBackup shared folder on WD My Cloud.

Creating sparsebundle image manually on NAS share drive gives flexibilities and control over image size for each backup. 


Configure time machine backup on Samba drive:

https://manjaro.site/how-to-configure-time-machine-to-backup-to-samba-shared-folder/


Repair sparsebundle on NAS backup:

https://www.garth.org/archives/2011,08,27,169,fix-time-machine-sparsebundle-nas-based-backup-errors.html

https://expobrain.net/2016/12/10/fix-corrupted-time-machine-spase-bundles/

https://macmanus.nl/2014/01/31/fixed-use-terminal-to-repair-corrupt-sparsebundle-file/


Limit the size of time machine sparse bundles:

https://community.wd.com/t/how-can-i-limit-the-size-of-the-time-machine-sparse-bundles-on-the-mbl/55021/6


Time machine backups on external USB drive attached to WD My Cloud:

https://support-en.wd.com/app/answers/detailweb/a_id/19225

https://community.wd.com/t/how-to-expand-disk-size-for-time-machine-backups/230284/2

https://community.wd.com/t/usb-port-for-time-machine-backup/266335