Guys, Forgive my ignorance on this but I am wondering how ZFS uses space when a volume is created with the -V parameter and then shared as an iSCSI lun. For example, I created 5 volumes 2 * 1TB in size and 3 * 2TB in size, but the space usage appears to be in addition to this size specification:- $ zfs list -o space -r ib NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD ib 168G 8.76T 0 21K 0 8.76T ib/iscsi 168G 8.76T 0 19K 0 8.76T ib/iscsi/esx-server1-00 2.16T 2.04T 58K 39.3G 2.00T 0 ib/iscsi/esx-server1-01 2.16T 2.00T 32.6M 258M 2.00T 0 ib/iscsi/esx-server2-00 1.16T 1.39T 862K 400G 1024G 0 ib/iscsi/esx-server2-01 1.16T 1.00T 859K 168M 1024G 0 ib/iscsi/esx-server2-02 2.16T 2.33T 166M 347G 1.99T 0 Even though the filesystem ib/iscsi/esx-server2-00 was specified as 1TB in size, and only 400GB is used the space usage is now at 1.4TB, i.e. 1TB + 400GB, is this normal? How can I get it to only use the 1TB of space that I reserved for the volume? $ zfs get all ib/iscsi/esx-server2-00 NAME PROPERTY VALUE SOURCE ib/iscsi/esx-server2-00 type volume - ib/iscsi/esx-server2-00 creation Fri Nov 13 11:01 2009 - ib/iscsi/esx-server2-00 used 1.39T - ib/iscsi/esx-server2-00 available 1.16T - ib/iscsi/esx-server2-00 referenced 400G - ib/iscsi/esx-server2-00 compressratio 1.00x - ib/iscsi/esx-server2-00 reservation none default ib/iscsi/esx-server2-00 volsize 1T - ib/iscsi/esx-server2-00 volblocksize 8K - ib/iscsi/esx-server2-00 checksum on default ib/iscsi/esx-server2-00 compression off default ib/iscsi/esx-server2-00 readonly off default ib/iscsi/esx-server2-00 shareiscsi on local ib/iscsi/esx-server2-00 copies 1 default ib/iscsi/esx-server2-00 refreservation 1T local ib/iscsi/esx-server2-00 primarycache all default ib/iscsi/esx-server2-00 secondarycache all default ib/iscsi/esx-server2-00 usedbysnapshots 862K - ib/iscsi/esx-server2-00 usedbydataset 400G - ib/iscsi/esx-server2-00 usedbychildren 0 - ib/iscsi/esx-server2-00 usedbyrefreservation 1024G - TIA -- This message posted from opensolaris.org
On Nov 18, 2009, at 7:28 PM, Duncan Bradey wrote:> Guys, > > Forgive my ignorance on this but I am wondering how ZFS uses space > when a volume is created with the -V parameter and then shared as an > iSCSI lun. > > For example, I created 5 volumes 2 * 1TB in size and 3 * 2TB in > size, but the space usage appears to be in addition to this size > specification:- > > $ zfs list -o space -r ib > NAME AVAIL USED USEDSNAP USEDDS > USEDREFRESERV USEDCHILD > ib 168G 8.76T 0 > 21K 0 8.76T > ib/iscsi 168G 8.76T 0 > 19K 0 8.76T > ib/iscsi/esx-server1-00 2.16T 2.04T 58K 39.3G > 2.00T 0 > ib/iscsi/esx-server1-01 2.16T 2.00T 32.6M 258M > 2.00T 0 > ib/iscsi/esx-server2-00 1.16T 1.39T 862K 400G > 1024G 0 > ib/iscsi/esx-server2-01 1.16T 1.00T 859K 168M > 1024G 0 > ib/iscsi/esx-server2-02 2.16T 2.33T 166M 347G > 1.99T 0 > > Even though the filesystem ib/iscsi/esx-server2-00 was specified as > 1TB in size, and only 400GB is used the space usage is now at 1.4TB, > i.e. 1TB + 400GB, is this normal?Yes, this is normal. refreservation does not include snapshots (USEDSNAP above).> How can I get it to only use the 1TB of space that I reserved for > the volume?Avoid taking snapshots of the volume. Unlike file systems, volumes do not have a quota or refquota, but they do have a maximum size. -- richard> > $ zfs get all ib/iscsi/esx-server2-00 > NAME PROPERTY > VALUE SOURCE > ib/iscsi/esx-server2-00 type > volume - > ib/iscsi/esx-server2-00 creation Fri Nov 13 11:01 > 2009 - > ib/iscsi/esx-server2-00 used > 1.39T - > ib/iscsi/esx-server2-00 available > 1.16T - > ib/iscsi/esx-server2-00 referenced > 400G - > ib/iscsi/esx-server2-00 compressratio > 1.00x - > ib/iscsi/esx-server2-00 reservation > none default > ib/iscsi/esx-server2-00 volsize > 1T - > ib/iscsi/esx-server2-00 volblocksize > 8K - > ib/iscsi/esx-server2-00 checksum > on default > ib/iscsi/esx-server2-00 compression > off default > ib/iscsi/esx-server2-00 readonly > off default > ib/iscsi/esx-server2-00 shareiscsi > on local > ib/iscsi/esx-server2-00 copies > 1 default > ib/iscsi/esx-server2-00 refreservation > 1T local > ib/iscsi/esx-server2-00 primarycache > all default > ib/iscsi/esx-server2-00 secondarycache > all default > ib/iscsi/esx-server2-00 usedbysnapshots > 862K - > ib/iscsi/esx-server2-00 usedbydataset > 400G - > ib/iscsi/esx-server2-00 usedbychildren > 0 - > ib/iscsi/esx-server2-00 usedbyrefreservation > 1024G - > > TIA > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Richard, Thanks for this, this explains why I am seeing this. I am using snapshots as I am replicating the data to other servers (via zfs send/recieve) is there another way to prevent this behaviour and still use snapshots? Or do I need to create these volumes as thin provisioned to get around this? Duncan -- This message posted from opensolaris.org
On Nov 18, 2009, at 8:54 PM, Duncan Bradey wrote:> Richard, > > Thanks for this, this explains why I am seeing this. I am using > snapshots as I am replicating the data to other servers (via zfs > send/recieve) is there another way to prevent this behaviour and > still use snapshots? Or do I need to create these volumes as thin > provisioned to get around this?IMHO, many volume use cases can be sparse. For virtual machine disks, I use sparse, compressed, and (soon) deduplicated volumes. -- richard
Thanks Richard, I''ve set the refreservation down and this has "freed" up space, I''m now setting up a process to monitor and update the refreservation attribute on zfs volumes used so we can thin provision yet keep some storage (half the remaining volume size) available to ensure that these volumes dont run out of space. Duncan -- This message posted from opensolaris.org