[Macos, APFS, VmWare] Shrinking VM images

[Rollo62 536]

Hello there,

 

I was making some experiments with shrinking larger VmWare Fusion VM images, which is not as simple as it should.
My configuration is:

Macos Host: Mojave, VmWare 11.5.0

Macos Guest: Catalina

I have found the following process to shrink the OS disk, which looks as if it works well so far.
The disk-sace is free and reduced and the VM is accessible.
Unfortunately the last step, to shrink the VM image itself from the host seems to be the issue now.

The "pre-occupy disk space" checkbos is NOT checked, so it should be a sparse file.
There is the "vmware-vdiskmanager" tool, which should be able to do this, and it comes back with "Shrink: 100% done.".

It seems that after this the VM size is still original, and the VM disk is not accessible any more.
Does anybody has a good tip what may go wrong here ?
 

Quote

Howto Shrink/Resize existing VmWare image
--> ! Backup your VmWare image, no warranty if anything went wrong.
--> Tested on Catalina APFS image, with VmWare Fusion 11.5.0

01.) Prepare VmWare image
--> Remove any snapshots
--> Ensure the disk tool is a sparse disk (no pre-occupied space clicked)

02.) Run VmWare Fusion Macos guest,
--> Open diskutil app
--> Check disk space distribution
--> Note the desired "tgt size" you want to achive after shrinking
--> CLose diskutil

03.) Open Terminal

04.) Get current Container ID and info
diskutil list
--> Get identifier of desired container
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *107.4 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         107.2 GB   disk0s2 //<======

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +107.2 GB   disk1  //<==== size
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD - Daten    23.7 GB    disk1s1
   2:                APFS Volume Preboot                 85.0 MB    disk1s2
   3:                APFS Volume Recovery                528.9 MB   disk1s3
   4:                APFS Volume VM                      1.1 MB     disk1s4
   5:                APFS Volume Macintosh HD            10.6 GB    disk1s5

05.) Shrink Container
//                                      "tgt size"      "extra size"
sudo diskutil apfs resizeContainer disk0s2 60g jhfs+ Extra 47.2g
--> Be sure the currently needed space is less than "tgt size"
--> The xx.2g decimals seems to be acceptable for the command
--> This command ends up in some processing line, like
    Shrinking APFS data structures
    [ | 0%..10%..20%..30%..40%..50%..60%..................... ] 68.4%
--> After shrinking complete:
    Shrinking APFS data structures
    Shrinking partition
    Modifying partition map
    Initialized /dev/rdisk0s3 as a 44 GB case-insensitive HFS Plus volume with a 8192k journal
    Mounting disk
    1 new disk created or changed due to APFS operation
    Disk from APFS operation: disk0s3
    Finished APFS operation

06.) Check status and get identifier
diskutil list
-->
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *107.4 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         60.0 GB    disk0s2  //<== resized to "tgt size"
   3:                  Apple_HFS Extra                   47.0 GB    disk0s3 //<== added separate free space

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +60.0 GB    disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD - Daten    23.8 GB    disk1s1
   2:                APFS Volume Preboot                 85.0 MB    disk1s2
   3:                APFS Volume Recovery                528.9 MB   disk1s3
   4:                APFS Volume VM                      1.1 MB     disk1s4
   5:                APFS Volume Macintosh HD            10.6 GB    disk1s5

07.) Erase the extra volume
sudo diskutil eraseVolume "Free Space" %noformat% /dev/disk0s3
-->
    Started erase on disk0s3 Extra
    Unmounting disk
    Finished erase on disk0

08.) Check status
diskutil list
-->
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *107.4 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         60.0 GB    disk0s2  //<== "tgt size" reached

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +60.0 GB    disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD - Daten    23.8 GB    disk1s1
   2:                APFS Volume Preboot                 85.0 MB    disk1s2
   3:                APFS Volume Recovery                528.9 MB   disk1s3
   4:                APFS Volume VM                      1.1 MB     disk1s4
   5:                APFS Volume Macintosh HD            10.6 GB    disk1s5

09.) Possible Alternative: Erase freespace
sudo diskutil secureErase freespace 0 MySpecialDiskIdentifier
--> untested

10.) Close the VmWare guest

11.) Resize the VmWare disk image
--> From Macos host Finder search, locate the position of the Virtual Disk Manager tool
    Locate the path insize the VmWare fusion package
    Something like:
    /Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager
    (Get path from Finder: Press Ctrl + Option + P to show path footer.
     Select and right click element at footer.
     Choose Copy pathname to get the real path to an object)
--> Get location of vmdk inside the .vmwarevm
    /user/myusername/documents/Virtuelle Maschinen/macOS 10.14.7_Rx1032B_2_BETA.vmwarevm/Contents/Virtuelle Festplatte.vmdk

--> From Macos host teminal call:
    Something like this: (!! Keep the \ before spaces in the path name)
    sudo /Applications/VMware\ Fusion.app/Contents/Library/vmware-vdiskmanager -k /Users/myusername/Documents/Virtuelle\ Maschinen/macOS\ 10.14.7_Rx1032B_2_BETA.vmwarevm/Virtuelle\ Festplatte.vmdk

--> During processing shows:
    Shrink: 48% done.
--> When finished shows:
    Shrink: 100% done.
    Shrink completed successfully.

 

 

Edited October 11, 2019 by Rollo62

[Rollo62 536]

Still no luck, but I added the step fragmentation before the shrinking.

After the shrinking, I've got missing user access, so I solved this by giving access to everyone again:

--> sudo chmod -v 666 /Users/myusername/Documents/Virtuelle\ Maschinen/macOS\ 10.14.7_Rx1032B_2_BETA.vmwarevm/Virtuelle\ Festplatte.vmdk

 

After that the VM is running again, but still the physical size is not shrinked, although the internal partition is shrinked as it should.

 

Maybe somebody has a clue how the VM image could be shrinked physically as well, maybe there are some tools out there (Paragon, etc.),
but I could find any tool yet solving such specific issues with VmWare Fusion images.

 

I nobody has some clue, maybe I have to  restart again with some older, clean images, but I would like to know howto cleanup existing images for the future.
What I cannot believe is that a modern file system like AFPS and VmWare are not able to allow a simple shrink of the file, only with a lot of terminal command steps, as below.
Isn't this the 21st century now

 

Edited October 11, 2019 by Rollo62

[HolgerX 7]

Hmm..

 

With VirtualPC (Microsoft..) it was necessary, after defragmenting, to overwrite the now 'free' space with e.g. '0', only then had its dynamic reduction works.

Maybe 'thinks' VMWare that even in the actually empty areas are still data?

 

[Rollo62 536]

Interesting, do you have a script which you used for that ?
When I check my partitions and volums, it looks pretty clear whare the "free space" is.

The problem is that VmWare seems not to offer a shrinking option for Macos, maybe only with some tweaks deep in the VmWare tools ?

Since the drive images are only files, I think this should be possible somehow, maybe in best case by just editing a HEX variable encoding the physical size, and trunkating somehow.

Because the partitions looks fine, only the physical size didn't change, which makes sense in the physical world.

Edited October 15, 2019 by Rollo62

[HolgerX 7]

Hmm..

 

Unfortunately, this was not possible with a simple script.

 

I had a simple tool written, which after defrag (with move to the beginning!) Just new files (each 40 MB size) with always # 0 as a character has generated.

Thus I filled the 'empty' area of the hard disk completely with # 0. Finally, just delete these files again.

 

Was just a workarround for VirtualPC ..

 

With the DiskManager of VirtualPC, the physical size of the hard disk could be reduced.

 

 

Do you already know this link?

https://virtualman.wordpress.com/2016/02/24/shrink-a-vmware-virtual-machine-disk-vmdk/

 

This was linked from
https://communities.vmware.com/thread/572240

 

(Translated from german with Goolge 😉 )

Edited October 15, 2019 by HolgerX

[Guest]

Also see Sysinternals sdelete -z:

-z	Zero free space (good for virtual disk optimization).

[Rollo62 536]

Thanks for the info.
I thought about script on MAcos, like

Quote

sudo diskutil eraseVolume "Free Space" %noformat% /dev/disk0s3

I have read some notes that this also supports "wipe" all unused bytes.

 

Since I tried shrinking only via VmWare Fusion yet, on my Macos host, maybe its a good point to check with VmWare Spere, what this can offer.
This seems not an easy task at all, fortunately the vmdsk images are exchangeable between Fusion, Workstation and Sphere (hope so).

 

However, if this adventure turns out to get that difficult, I could create a new, empty VM from scratch, as usual.
Only that the smallest disk size I can create via VmWare Fusion seems to be 40 GB (~30-35GB physical),
while a really cleaned up VM (removed all garbage) maybe only would need only 20-25GB (I would assume).

 

[Rollo62 536]

@Ondrej Kelle

Thanks, but I'm on Macos host and Macos guest in the VM.
On Windows this task seems to be easier, but I think the real issue lays in VmWare images, which need to be resized.
If VmWare itself doesn't support them, maybe some 3rd party may exists.