So I''m looking at files on my ZFS volume that are compressed, and I''m wondering to myself, "self, are the values shown here the size on disk, or are they the pre-compressed values". Google gives me no great results on the first few pages, so I headed here. This really relates to my VMware environment. I had some "things" happen on my platform that required me to Storage Vmotion everything off of a particular zpool. When I did that, I saw most VM''s inflate to nearly their thick provisioned size. What didn''t swell to that size went to about 2/3 provisioned (non-Nexenta storage). I have been seeing 1.3-1.5x compression ratios on pretty much everything I turn compression on for (these are general use VM''s - webservers,SQL,firewall,etc). My question is this - when I''m looking in the file structure, or in the datastore browser in VMware, am I seeing the uncompressed file size, or the compressed filesize? My gut tells me that since they inflated _so_ badly when I storage vmotioned them, that they are the compressed values, but I would love to know for sure. -Matt Breitbach
2011-11-23 7:39, Matt Breitbach wrote:> So I''m looking at files on my ZFS volume that are compressed, and I''m > wondering to myself, "self, are the values shown here the size on disk, or > are they the pre-compressed values". Google gives me no great results on > the first few pages, so I headed here.Alas, I can''t give a good hint about VMWare - which values it uses. But here are some numbers it might see (likely "du" or "ls" sizes are in play): Locally on a ZFS-enabled system you can use "ls" to normally list your files. This would show you the logical POSIX file size, including any referenced-but-not-allocated sparse blocks (logical size = big, physical size = zero), etc. Basically, this just gives a range of byte numbers that you can address in the file, and depending on the underlying FS all or not all of these bytes are backed by physical storage 1:1. If you use "du" on the ZFS filesystem, you''ll see the logical storage size, which takes into account compression and sparse bytes. So the "du" size should be not greater than "ls" size. Harder to see would be the physical allocation size, which refers to your data pool''s redundancy (raidzN level, copies=N and so on). But you can more or less calculate that from "du" size. Also your files on ZFS indirectly consume space by requiring some metadata blocks (usually one per data block, and usually they are comparatively small) which is pool metadata and does not show up easily as "file size" as well. If you''re too interested, you might search for howtos on "zdb" command usage to "debug" your ZFS pool and gather stats. If your new storage system does support some sort of compression, at least for contiguous ranges of zero-bytes, you might write some large zero-filled files into your VM''s filesystems. This should empty the blocks previously used by files now deleted in the VM and may give temporary space savings.> > This really relates to my VMware environment. I had some "things" happen on > my platform that required me to Storage Vmotion everything off of a > particular zpool. When I did that, I saw most VM''s inflate to nearly their > thick provisioned size. What didn''t swell to that size went to about 2/3 > provisioned (non-Nexenta storage). > > I have been seeing 1.3-1.5x compression ratios on pretty much everything I > turn compression on for (these are general use VM''s - > webservers,SQL,firewall,etc). > > My question is this - when I''m looking in the file structure, or in the > datastore browser in VMware, am I seeing the uncompressed file size, or the > compressed filesize? > > My gut tells me that since they inflated _so_ badly when I storage vmotioned > them, that they are the compressed values, but I would love to know for > sure. > > -Matt BreitbachHTH, //Jim Klimov
On 11/23/11 04:58 PM, Jim Klimov wrote:> 2011-11-23 7:39, Matt Breitbach wrote: >> So I''m looking at files on my ZFS volume that are compressed, and I''m >> wondering to myself, "self, are the values shown here the size on disk, or >> are they the pre-compressed values". Google gives me no great results on >> the first few pages, so I headed here. > Alas, I can''t give a good hint about VMWare - which values > it uses. But here are some numbers it might see (likely > "du" or "ls" sizes are in play): > > Locally on a ZFS-enabled system you can use "ls" to normally > list your files. This would show you the logical POSIX file > size, including any referenced-but-not-allocated sparse blocks > (logical size = big, physical size = zero), etc. > Basically, this just gives a range of byte numbers that you > can address in the file, and depending on the underlying FS > all or not all of these bytes are backed by physical storage 1:1. > > If you use "du" on the ZFS filesystem, you''ll see the logical > storage size, which takes into account compression and sparse > bytes. So the "du" size should be not greater than "ls" size.It can be significantly bigger: ls -sh x 2 x du -sh x 1K x -- Ian.
2011-11-23 8:21, Ian Collins wrote:>> If you use "du" on the ZFS filesystem, you''ll see the logical >> storage size, which takes into account compression and sparse >> bytes. So the "du" size should be not greater than "ls" size. > > It can be significantly bigger: > > ls -sh x > 2 x > > du -sh x > 1K xPun accepted ;) Ian is right, that "du" size also reflects block-size usage, and that''s how many bytes are actually used on the FS (over redundancy layer). Even if your files are smaller than a single block, that''s the minimum they will bite off your disk anyway. However, the original question was about VM datastores, so large files were assumed. //Jim
Hi Matt, On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote:> So I''m looking at files on my ZFS volume that are compressed, and I''m > wondering to myself, "self, are the values shown here the size on disk, or > are they the pre-compressed values". Google gives me no great results on > the first few pages, so I headed here. > > This really relates to my VMware environment. I had some "things" happen on > my platform that required me to Storage Vmotion everything off of a > particular zpool. When I did that, I saw most VM''s inflate to nearly their > thick provisioned size. What didn''t swell to that size went to about 2/3 > provisioned (non-Nexenta storage). > > I have been seeing 1.3-1.5x compression ratios on pretty much everything I > turn compression on for (these are general use VM''s - > webservers,SQL,firewall,etc). > > My question is this - when I''m looking in the file structure, or in the > datastore browser in VMware, am I seeing the uncompressed file size, or the > compressed filesize? > > My gut tells me that since they inflated _so_ badly when I storage vmotioned > them, that they are the compressed values, but I would love to know for > sure.How are you measuring the space? Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? -- richard -- ZFS and performance consulting http://www.RichardElling.com LISA ''11, Boston, MA, December 4-9
Currently using NFS to access the datastore. -Matt -----Original Message----- From: Richard Elling [mailto:richard.elling at gmail.com] Sent: Tuesday, November 22, 2011 11:10 PM To: Matt Breitbach Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] Compression Hi Matt, On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote:> So I''m looking at files on my ZFS volume that are compressed, and I''m > wondering to myself, "self, are the values shown here the size on disk, or > are they the pre-compressed values". Google gives me no great results on > the first few pages, so I headed here. > > This really relates to my VMware environment. I had some "things" happenon> my platform that required me to Storage Vmotion everything off of a > particular zpool. When I did that, I saw most VM''s inflate to nearlytheir> thick provisioned size. What didn''t swell to that size went to about 2/3 > provisioned (non-Nexenta storage). > > I have been seeing 1.3-1.5x compression ratios on pretty much everything I > turn compression on for (these are general use VM''s - > webservers,SQL,firewall,etc). > > My question is this - when I''m looking in the file structure, or in the > datastore browser in VMware, am I seeing the uncompressed file size, orthe> compressed filesize? > > My gut tells me that since they inflated _so_ badly when I storagevmotioned> them, that they are the compressed values, but I would love to know for > sure.How are you measuring the space? Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? -- richard -- ZFS and performance consulting http://www.RichardElling.com LISA ''11, Boston, MA, December 4-9
After additional digging and investigation, looks like it''s showing me the "compressed" size, which is good. I''ve regroomed the storage pools and started moving things back, and I''m seeing files deflate back to their expected size. What really tipped me off was when I decided to log in to one of the VM''s. VMware showed 120GB provisioned, 92GB allocated on the standard block storage. After storage vmotioning back to NFS, it showed 62GB allocated. The key was when I logged in to the machine, Windows said it was using 90GB. All said and done, it was a good adventure, and I got the info that I needed. Thanks to all that took the time to reply. -Matt Breitbach -----Original Message----- From: Donal Farrell [mailto:vmlinuz101 at gmail.com] Sent: Wednesday, November 23, 2011 10:42 AM To: Matt Breitbach Subject: Re: [zfs-discuss] Compression is this on esx 3.5.x? or 4.x or greater? The reason i ask is that svmotion to and from nfs datastores did not work correctly in esx 3.5.x Also can you past the output of cat /vmfs/volumes/datastore/vmname.vmdk here? On Wed, Nov 23, 2011 at 1:55 PM, Matt Breitbach <matthewb at flash.shanje.com> wrote:> Currently using NFS to access the datastore. > > -Matt > > -----Original Message----- > From: Richard Elling [mailto:richard.elling at gmail.com] > Sent: Tuesday, November 22, 2011 11:10 PM > To: Matt Breitbach > Cc: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] Compression > > Hi Matt, > > On Nov 22, 2011, at 7:39 PM, Matt Breitbach wrote: > >> So I''m looking at files on my ZFS volume that are compressed, and I''m >> wondering to myself, "self, are the values shown here the size on disk,or>> are they the pre-compressed values". ?Google gives me no great results on >> the first few pages, so I headed here. >> >> This really relates to my VMware environment. ?I had some "things" happen > on >> my platform that required me to Storage Vmotion everything off of a >> particular zpool. ?When I did that, I saw most VM''s inflate to nearly > their >> thick provisioned size. ?What didn''t swell to that size went to about 2/3 >> provisioned (non-Nexenta storage). >> >> I have been seeing 1.3-1.5x compression ratios on pretty much everythingI>> turn compression on for (these are general use VM''s - >> webservers,SQL,firewall,etc). >> >> My question is this - when I''m looking in the file structure, or in the >> datastore browser in VMware, am I seeing the uncompressed file size, or > the >> compressed filesize? >> >> My gut tells me that since they inflated _so_ badly when I storage > vmotioned >> them, that they are the compressed values, but I would love to know for >> sure. > > How are you measuring the space? > > Are you using block (iscsi/fc) or NFS to access the datastores from ESXi? > ?-- richard > > -- > > ZFS and performance consulting > http://www.RichardElling.com > LISA ''11, Boston, MA, December 4-9 > > > > > > > > > > > > > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >