Erick Perez - Quadrian Enterprises
2020-May-03 05:33 UTC
[CentOS] Understanding VDO vs ZFS
sorry corrections:
For this test I created a 40GB lvm volume group with /dev/sdb and /dev/sdc
then a 40GB LV
then a 60GB VDO vol (for testing purposes)
vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
output from just created vdoas
[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6
'saving
percent'
physical blocks : 10483712
logical blocks : 15728640
1K-blocks : 41934848
1K-blocks used : 4212024
1K-blocks available : 37722824
used percent : 10
saving percent : 99
[root at localhost ~]#
FIRST copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas from source
outside vdo volume
[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6
'saving
percent'
1K-blocks used : 4721348
1K-blocks available : 37213500
used percent : 11
saving percent : 9
SECOND copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas form source
outside vdo volume
#cp /root/CentOS-7-x86_64-Minimal-2003.iso
/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso
1K-blocks used : 5239012
1K-blocks available : 36695836
used percent : 12
saving percent : 52
THIRD copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to
vdoas form inside vdo volume to inside vdo volume
1K-blocks used : 5248060
1K-blocks available : 36686788
used percent : 12
saving percent : 67
Then I did this a total of 9 more times to have 10 ISOs copied. Total data
copied 10.6GB.
Do note this:
When using DF, it will show the VDO size, in my case 60G
when using vdostats it will show the size of the LV, in my case 40G
Remeber dedupe AND compression are enabled.
The df -hT output shows the logical space occupied by these iso files as
seen by the filesystem on the VDO volume.
Since VDO manages a logical to physical block map, df sees logical space
consumed according to the file system that resides on top of the VDO
volume.
vdostats --hu is viewing the physical block device as managed by VDO.
Physically a single .ISO image is residing on the disk, but logically the
file system thinks there are 10 copies, occupying 10.6GB.
So at the end I have 10 .ISOs of 1086 1MB blocks (total 10860 1MB blocks)
that yield these results:
1K-blocks used : 5248212
1K-blocks available : 36686636
used percent : 12
saving percent : 89
So at the end it is using 5248212 1K blocks minus 4212024 initial used 1K
blocks, gives (5248212 - 4212024) = 1036188 1K blocks / 1024 = about 1012MB
total.
Hope this helps understanding where the space goes.
BTW: Testing system is CentOS Linux release 7.8.2003 stock. with only "yum
install vdo kmod-kvdo"
History of commands:
[root at localhost vdomounts]# history
2 pvcreate /dev/sdb
3 pvcreate /dev/sdc
8 vgcreate -v -A y vgvol01 /dev/sdb /dev/sdc
9 vgdisplay
13 lvcreate -l 100%FREE -n lvvdo01 vgvol01
14 yum install vdo kmod-kvdo
18 vdo create --name=vdoas --device=/dev/vgvol01/lvvdo01
--vdoLogicalSize=60G --writePolicy=async
19 mkfs.xfs -K /dev/mapper/vdoas
20 ls /mnt
21 mkdir /mnt/vdomounts
22 mount /dev/mapper/vdoas /mnt//vdomounts/
26 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
28 cp /root/CentOS-7-x86_64-Minimal-2003.iso /mnt/vdomounts/ -vvv
29 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
30 cp /root/CentOS-7-x86_64-Minimal-2003.iso
/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso
31 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
33 cd /mnt/vdomounts/
35 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version3.iso
36 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
37 df
39 vdostats --hu
40 ls -l --block-size=1MB /root/CentOS-7-x86_64-Minimal-2003.iso
41 df -hT
42 vdo status | grep Dedupl
43 vdostats --hu
44 vdostats
48 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version4.iso
49 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version5.iso
50 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version6.iso
51 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version7.iso
52 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version8.iso
53 cp CentOS-7-x86_64-Minimal-2003-version2.iso
./CentOS-7-x86_64-Minimal-2003-version9.iso
54 df -hT
55 ls -l --block-size=1MB
56 vdostats --hu
57 df -hT
58 df
59 vdostats --hu
60 vdostats
61 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent'
62 cat /etc/centos-release
63 history
[root at localhost vdomounts]#
On Sat, May 2, 2020 at 10:07 PM Erick Perez - Quadrian Enterprises <
eperez at quadrianweb.com> wrote:
> My two cents:
> 1- Do you have an encrypted filesystem on top of VDO? If yes, you will see
> no benefit from dedupe.
> 2- can you post the stats of vdostats ?verbose /dev/mapper/xxxxx (replace
> with your device)
>
> you can do something like: "vdostats -verbose /dev/mapper/xxxxxxxx |
grep
> -B6 'save percentage'
>
>
>
>
> On Sat, May 2, 2020 at 9:54 PM david <david at daku.org> wrote:
>
>> Folks
>>
>> I'm looking for a solution for backups because ZFS has failed on me
>> too many times. In my environment, I have a large amount of data
>> (around 2tb) that I periodically back up. I keep the last 5
>> "snapshots". I use rsync so that when I overwrite the oldest
backup,
>> most of the data is already there and the backup completes quickly,
>> because only a small number of files have actually changed.
>>
>> Because of this low change rate, I have used ZFS with its
>> deduplication feature to store the data. I started using a Centos-6
>> installation, and upgraded years ago to Centos7. Centos 8 is on my
>> agenda. However, I've had several data-loss events with ZFS where
>> because of a combination of errors and/or mistakes, the entire store
>> was lost. I've also noticed that ZFS is maintained separately from
>> Centos. At this moment, the Centos 8 update causes ZFS to
>> fail. Looking for an alternate, I'm trying VDO.
>>
>> In the VDO installation, I created a logical volume containing two
>> hard-drives, and defined VDO on top of that logical volume. It
>> appears to be running, yet I find the deduplication numbers don't
>> pass the smell test. I would expect that if the logical volume
>> contains three copies of essentially identical data, I should see
>> deduplication numbers close to 3.00, but instead I'm seeing numbers
>> like 1.15. I compute the compression number as follows:
>> Use df and extract the value for "1k blocks used" from the
third column
>> use vdostats --verbose and extract the number titled "1K-blocks
used"
>>
>> Divide the first by the second.
>>
>> Can you provide any advice on my use of ZFS or VDO without telling me
>> that I should be doing backups differently?
>>
>> Thanks
>>
>> David
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS at centos.org
>> https://lists.centos.org/mailman/listinfo/centos
>>
>
>
> --
>
> ---------------------
> Erick Perez
>
>
--
---------------------
Erick Perez
Quadrian Enterprises S.A. - Panama, Republica de Panama
Skype chat: eaperezh
WhatsApp IM: +507-6675-5083
---------------------
On May 3, 2020 8:33:33 AM GMT+03:00, Erick Perez - Quadrian Enterprises <eperez at quadrianweb.com> wrote:>sorry corrections: >For this test I created a 40GB lvm volume group with /dev/sdb and >/dev/sdc >then a 40GB LV >then a 60GB VDO vol (for testing purposes) > >vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' >output from just created vdoas > >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 >'saving >percent' >physical blocks : 10483712 > logical blocks : 15728640 > 1K-blocks : 41934848 > 1K-blocks used : 4212024 > 1K-blocks available : 37722824 > used percent : 10 > saving percent : 99 >[root at localhost ~]# > >FIRST copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas from source >outside vdo volume >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 >'saving >percent' > 1K-blocks used : 4721348 > 1K-blocks available : 37213500 > used percent : 11 > saving percent : 9 > >SECOND copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas form >source >outside vdo volume >#cp /root/CentOS-7-x86_64-Minimal-2003.iso >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > 1K-blocks used : 5239012 > 1K-blocks available : 36695836 > used percent : 12 > saving percent : 52 > >THIRD copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to >vdoas form inside vdo volume to inside vdo volume > 1K-blocks used : 5248060 > 1K-blocks available : 36686788 > used percent : 12 > saving percent : 67 > >Then I did this a total of 9 more times to have 10 ISOs copied. Total >data >copied 10.6GB. > > >Do note this: >When using DF, it will show the VDO size, in my case 60G >when using vdostats it will show the size of the LV, in my case 40G >Remeber dedupe AND compression are enabled. > >The df -hT output shows the logical space occupied by these iso files >as >seen by the filesystem on the VDO volume. >Since VDO manages a logical to physical block map, df sees logical >space >consumed according to the file system that resides on top of the VDO >volume. >vdostats --hu is viewing the physical block device as managed by VDO. >Physically a single .ISO image is residing on the disk, but logically >the >file system thinks there are 10 copies, occupying 10.6GB. > >So at the end I have 10 .ISOs of 1086 1MB blocks (total 10860 1MB >blocks) >that yield these results: > 1K-blocks used : 5248212 > 1K-blocks available : 36686636 > used percent : 12 > saving percent : 89 > >So at the end it is using 5248212 1K blocks minus 4212024 initial >used 1K >blocks, gives (5248212 - 4212024) = 1036188 1K blocks / 1024 = about >1012MB >total. > >Hope this helps understanding where the space goes. > >BTW: Testing system is CentOS Linux release 7.8.2003 stock. with only >"yum >install vdo kmod-kvdo" > >History of commands: >[root at localhost vdomounts]# history > 2 pvcreate /dev/sdb > 3 pvcreate /dev/sdc > 8 vgcreate -v -A y vgvol01 /dev/sdb /dev/sdc > 9 vgdisplay > 13 lvcreate -l 100%FREE -n lvvdo01 vgvol01 > 14 yum install vdo kmod-kvdo > 18 vdo create --name=vdoas --device=/dev/vgvol01/lvvdo01 >--vdoLogicalSize=60G --writePolicy=async > 19 mkfs.xfs -K /dev/mapper/vdoas > 20 ls /mnt > 21 mkdir /mnt/vdomounts > 22 mount /dev/mapper/vdoas /mnt//vdomounts/ > 26 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > 28 cp /root/CentOS-7-x86_64-Minimal-2003.iso /mnt/vdomounts/ -vvv > 29 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > 30 cp /root/CentOS-7-x86_64-Minimal-2003.iso >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > 31 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > 33 cd /mnt/vdomounts/ > 35 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version3.iso > 36 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > 37 df > 39 vdostats --hu > 40 ls -l --block-size=1MB /root/CentOS-7-x86_64-Minimal-2003.iso > 41 df -hT > 42 vdo status | grep Dedupl > 43 vdostats --hu > 44 vdostats > 48 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version4.iso > 49 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version5.iso > 50 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version6.iso > 51 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version7.iso > 52 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version8.iso > 53 cp CentOS-7-x86_64-Minimal-2003-version2.iso >./CentOS-7-x86_64-Minimal-2003-version9.iso > 54 df -hT > 55 ls -l --block-size=1MB > 56 vdostats --hu > 57 df -hT > 58 df > 59 vdostats --hu > 60 vdostats > 61 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > 62 cat /etc/centos-release > 63 history >[root at localhost vdomounts]# > > > > > >On Sat, May 2, 2020 at 10:07 PM Erick Perez - Quadrian Enterprises < >eperez at quadrianweb.com> wrote: > >> My two cents: >> 1- Do you have an encrypted filesystem on top of VDO? If yes, you >will see >> no benefit from dedupe. >> 2- can you post the stats of vdostats ?verbose /dev/mapper/xxxxx >(replace >> with your device) >> >> you can do something like: "vdostats -verbose /dev/mapper/xxxxxxxx | >grep >> -B6 'save percentage' >> >> >> >> >> On Sat, May 2, 2020 at 9:54 PM david <david at daku.org> wrote: >> >>> Folks >>> >>> I'm looking for a solution for backups because ZFS has failed on me >>> too many times. In my environment, I have a large amount of data >>> (around 2tb) that I periodically back up. I keep the last 5 >>> "snapshots". I use rsync so that when I overwrite the oldest >backup, >>> most of the data is already there and the backup completes quickly, >>> because only a small number of files have actually changed. >>> >>> Because of this low change rate, I have used ZFS with its >>> deduplication feature to store the data. I started using a Centos-6 >>> installation, and upgraded years ago to Centos7. Centos 8 is on my >>> agenda. However, I've had several data-loss events with ZFS where >>> because of a combination of errors and/or mistakes, the entire store >>> was lost. I've also noticed that ZFS is maintained separately from >>> Centos. At this moment, the Centos 8 update causes ZFS to >>> fail. Looking for an alternate, I'm trying VDO. >>> >>> In the VDO installation, I created a logical volume containing two >>> hard-drives, and defined VDO on top of that logical volume. It >>> appears to be running, yet I find the deduplication numbers don't >>> pass the smell test. I would expect that if the logical volume >>> contains three copies of essentially identical data, I should see >>> deduplication numbers close to 3.00, but instead I'm seeing numbers >>> like 1.15. I compute the compression number as follows: >>> Use df and extract the value for "1k blocks used" from the third >column >>> use vdostats --verbose and extract the number titled "1K-blocks >used" >>> >>> Divide the first by the second. >>> >>> Can you provide any advice on my use of ZFS or VDO without telling >me >>> that I should be doing backups differently? >>> >>> Thanks >>> >>> David >>> >>> _______________________________________________ >>> CentOS mailing list >>> CentOS at centos.org >>> https://lists.centos.org/mailman/listinfo/centos >>> >> >> >> -- >> >> --------------------- >> Erick Perez >> >> > >-- > >--------------------- >Erick Perez >Quadrian Enterprises S.A. - Panama, Republica de Panama >Skype chat: eaperezh >WhatsApp IM: +507-6675-5083 >--------------------- >_______________________________________________ >CentOS mailing list >CentOS at centos.org >https://lists.centos.org/mailman/listinfo/centosHi Erick, In the end how much space is used as per vdostats ? Keep in mind that VDO keeps one slab for itself (default is 2GB) and also some metadata is written : https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-qs-requirements#vdo-ig-storage-reqts Best Regards, Strahil Nikolov
Erick Perez - Quadrian Enterprises
2020-May-04 06:28 UTC
[CentOS] Understanding VDO vs ZFS
Strahil, I am using about 1012MB for the first ISO. I believe it's because of compression. From there vdostats --hu reports 5.0G usage and 12% in percentage. With savings of 89% for original + 9 copies of the same ISO. On Sun, May 3, 2020 at 1:17 AM Strahil Nikolov <hunter86_bg at yahoo.com> wrote:> On May 3, 2020 8:33:33 AM GMT+03:00, Erick Perez - Quadrian Enterprises < > eperez at quadrianweb.com> wrote: > >sorry corrections: > >For this test I created a 40GB lvm volume group with /dev/sdb and > >/dev/sdc > >then a 40GB LV > >then a 60GB VDO vol (for testing purposes) > > > >vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > >output from just created vdoas > > > >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 > >'saving > >percent' > >physical blocks : 10483712 > > logical blocks : 15728640 > > 1K-blocks : 41934848 > > 1K-blocks used : 4212024 > > 1K-blocks available : 37722824 > > used percent : 10 > > saving percent : 99 > >[root at localhost ~]# > > > >FIRST copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas from source > >outside vdo volume > >[root at localhost ~]# vdostats --verbose /dev/mapper/vdoas | grep -B6 > >'saving > >percent' > > 1K-blocks used : 4721348 > > 1K-blocks available : 37213500 > > used percent : 11 > > saving percent : 9 > > > >SECOND copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to vdoas form > >source > >outside vdo volume > >#cp /root/CentOS-7-x86_64-Minimal-2003.iso > >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > > 1K-blocks used : 5239012 > > 1K-blocks available : 36695836 > > used percent : 12 > > saving percent : 52 > > > >THIRD copy CentOS-7-x86_64-Minimal-2003.iso (1.1G) to > >vdoas form inside vdo volume to inside vdo volume > > 1K-blocks used : 5248060 > > 1K-blocks available : 36686788 > > used percent : 12 > > saving percent : 67 > > > >Then I did this a total of 9 more times to have 10 ISOs copied. Total > >data > >copied 10.6GB. > > > > > >Do note this: > >When using DF, it will show the VDO size, in my case 60G > >when using vdostats it will show the size of the LV, in my case 40G > >Remeber dedupe AND compression are enabled. > > > >The df -hT output shows the logical space occupied by these iso files > >as > >seen by the filesystem on the VDO volume. > >Since VDO manages a logical to physical block map, df sees logical > >space > >consumed according to the file system that resides on top of the VDO > >volume. > >vdostats --hu is viewing the physical block device as managed by VDO. > >Physically a single .ISO image is residing on the disk, but logically > >the > >file system thinks there are 10 copies, occupying 10.6GB. > > > >So at the end I have 10 .ISOs of 1086 1MB blocks (total 10860 1MB > >blocks) > >that yield these results: > > 1K-blocks used : 5248212 > > 1K-blocks available : 36686636 > > used percent : 12 > > saving percent : 89 > > > >So at the end it is using 5248212 1K blocks minus 4212024 initial > >used 1K > >blocks, gives (5248212 - 4212024) = 1036188 1K blocks / 1024 = about > >1012MB > >total. > > > >Hope this helps understanding where the space goes. > > > >BTW: Testing system is CentOS Linux release 7.8.2003 stock. with only > >"yum > >install vdo kmod-kvdo" > > > >History of commands: > >[root at localhost vdomounts]# history > > 2 pvcreate /dev/sdb > > 3 pvcreate /dev/sdc > > 8 vgcreate -v -A y vgvol01 /dev/sdb /dev/sdc > > 9 vgdisplay > > 13 lvcreate -l 100%FREE -n lvvdo01 vgvol01 > > 14 yum install vdo kmod-kvdo > > 18 vdo create --name=vdoas --device=/dev/vgvol01/lvvdo01 > >--vdoLogicalSize=60G --writePolicy=async > > 19 mkfs.xfs -K /dev/mapper/vdoas > > 20 ls /mnt > > 21 mkdir /mnt/vdomounts > > 22 mount /dev/mapper/vdoas /mnt//vdomounts/ > > 26 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 28 cp /root/CentOS-7-x86_64-Minimal-2003.iso /mnt/vdomounts/ -vvv > > 29 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 30 cp /root/CentOS-7-x86_64-Minimal-2003.iso > >/mnt/vdomounts/CentOS-7-x86_64-Minimal-2003-version2.iso > > 31 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 33 cd /mnt/vdomounts/ > > 35 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version3.iso > > 36 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 37 df > > 39 vdostats --hu > > 40 ls -l --block-size=1MB /root/CentOS-7-x86_64-Minimal-2003.iso > > 41 df -hT > > 42 vdo status | grep Dedupl > > 43 vdostats --hu > > 44 vdostats > > 48 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version4.iso > > 49 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version5.iso > > 50 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version6.iso > > 51 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version7.iso > > 52 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version8.iso > > 53 cp CentOS-7-x86_64-Minimal-2003-version2.iso > >./CentOS-7-x86_64-Minimal-2003-version9.iso > > 54 df -hT > > 55 ls -l --block-size=1MB > > 56 vdostats --hu > > 57 df -hT > > 58 df > > 59 vdostats --hu > > 60 vdostats > > 61 vdostats --verbose /dev/mapper/vdoas | grep -B6 'saving percent' > > 62 cat /etc/centos-release > > 63 history > >[root at localhost vdomounts]# > > > > > > > > > > > >On Sat, May 2, 2020 at 10:07 PM Erick Perez - Quadrian Enterprises < > >eperez at quadrianweb.com> wrote: > > > >> My two cents: > >> 1- Do you have an encrypted filesystem on top of VDO? If yes, you > >will see > >> no benefit from dedupe. > >> 2- can you post the stats of vdostats ?verbose /dev/mapper/xxxxx > >(replace > >> with your device) > >> > >> you can do something like: "vdostats -verbose /dev/mapper/xxxxxxxx | > >grep > >> -B6 'save percentage' > >> > >> > >> > >> > >> On Sat, May 2, 2020 at 9:54 PM david <david at daku.org> wrote: > >> > >>> Folks > >>> > >>> I'm looking for a solution for backups because ZFS has failed on me > >>> too many times. In my environment, I have a large amount of data > >>> (around 2tb) that I periodically back up. I keep the last 5 > >>> "snapshots". I use rsync so that when I overwrite the oldest > >backup, > >>> most of the data is already there and the backup completes quickly, > >>> because only a small number of files have actually changed. > >>> > >>> Because of this low change rate, I have used ZFS with its > >>> deduplication feature to store the data. I started using a Centos-6 > >>> installation, and upgraded years ago to Centos7. Centos 8 is on my > >>> agenda. However, I've had several data-loss events with ZFS where > >>> because of a combination of errors and/or mistakes, the entire store > >>> was lost. I've also noticed that ZFS is maintained separately from > >>> Centos. At this moment, the Centos 8 update causes ZFS to > >>> fail. Looking for an alternate, I'm trying VDO. > >>> > >>> In the VDO installation, I created a logical volume containing two > >>> hard-drives, and defined VDO on top of that logical volume. It > >>> appears to be running, yet I find the deduplication numbers don't > >>> pass the smell test. I would expect that if the logical volume > >>> contains three copies of essentially identical data, I should see > >>> deduplication numbers close to 3.00, but instead I'm seeing numbers > >>> like 1.15. I compute the compression number as follows: > >>> Use df and extract the value for "1k blocks used" from the third > >column > >>> use vdostats --verbose and extract the number titled "1K-blocks > >used" > >>> > >>> Divide the first by the second. > >>> > >>> Can you provide any advice on my use of ZFS or VDO without telling > >me > >>> that I should be doing backups differently? > >>> > >>> Thanks > >>> > >>> David > >>> > >>> _______________________________________________ > >>> CentOS mailing list > >>> CentOS at centos.org > >>> https://lists.centos.org/mailman/listinfo/centos > >>> > >> > >> > >> -- > >> > >> --------------------- > >> Erick Perez > >> > >> > > > >-- > > > >--------------------- > >Erick Perez > >Quadrian Enterprises S.A. - Panama, Republica de Panama > >Skype chat: eaperezh > >WhatsApp IM: +507-6675-5083 > >--------------------- > >_______________________________________________ > >CentOS mailing list > >CentOS at centos.org > >https://lists.centos.org/mailman/listinfo/centos > > Hi Erick, > > In the end how much space is used as per vdostats ? > Keep in mind that VDO keeps one slab for itself (default is 2GB) and also > some metadata is written : > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-qs-requirements#vdo-ig-storage-reqts > > Best Regards, > Strahil Nikolov >-- --------------------- Erick Perez Quadrian Enterprises S.A. - Panama, Republica de Panama Skype chat: eaperezh WhatsApp IM: +507-6675-5083 ---------------------