zfs discuss - Apr 2010 - crypted zvol bandwith => lofidevice=`pfexec lofiadm -a /dev/zvol/rdsk/$volumepath -c aes-256-cbc`

Hello everyone!

First!
Great thanks for this great technology that is ZFS!

Then!
I need some advices about a weird thing I just find out.

Seems my I/O on a crypted zvol is 3 time more than the corresponding ones off
the pool on the lofi device.
I have attached a file containing all information I know about.

That should be very easy to reproduce.

Did someone could explain that behavior to me?

Thanks in advance,
Best Regards,
Mickael.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100416/373d83d3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs.fs.properties
Type: application/octet-stream
Size: 18179 bytes
Desc: zfs.fs.properties
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100416/373d83d3/attachment.obj>

On 16/04/2010 10:19, Mickael Lambert wrote:
> First!
> Great thanks for this great technology that is ZFS!
>
> Then!
> I need some advices about a weird thing I just find out.
>
> Seems my I/O on a crypted zvol is 3 time more than the corresponding
> ones off the pool on the lofi device.
> I have attached a file containing all information I know about.
>
> That should be very easy to reproduce.
>
> Did someone could explain that behavior to me?
Encryption costs and in this setup you have multiple layers here too, a 
ZFS pool ontop of lofi (doing the encryption) ontop of a ZVOL which is 
in your rpool.

So any write to the filesystems in "apool" have to go to ZFS, then to 
lofi (and be encrypted) then to the ZVOL that is in your rpool, that 
alone without the encryption adds to the IO requirements.

A fair comparison would be to do the same setup with the multiple pools 
and lofi but don''t have lofi do encryption.  That would tell you the 
overhead of the encryption that lofi does.

-- 
Darren J Moffat

Hello Darren,

Thanks for your answer!

I have try without encryption and seems the same.

My mean is about bandwidth. what I see is that if I write
xMb/s to zfs fs then zfs write nearly xMb/s to the pool and that''s
attended.
This pool write nearly xMb/s to lofi and that''s attended.
iostat lofi seems showing xMb/s input attended also.

I am not enough skilled to investigate on what''s happening then between
lofi and the zvol but the other pool write 3*xMb/s to the physical drive.
I am not sure if lofi and or the zvol made some kind of allocate/read/write/read
that make the zvol write 2 or more time each blocks.

If someone has a time windows to make a trial it could be great.
I really would like to understand if it is attended and why.

Thanks in advance,
Best regards,
Mickael.





-----Original Message-----
From: Darren J Moffat [mailto:darrenm at opensolaris.org]
Sent: Fri 4/16/2010 6:04 PM
To: Lambert, Mickael KGSM (GZIE:3655)
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] crypted zvol bandwith => lofidevice=`pfexec
lofiadm -a /dev/zvol/rdsk/$volumepath -c aes-256-cbc`
 
On 16/04/2010 10:19, Mickael Lambert wrote:
> First!
> Great thanks for this great technology that is ZFS!
>
> Then!
> I need some advices about a weird thing I just find out.
>
> Seems my I/O on a crypted zvol is 3 time more than the corresponding
> ones off the pool on the lofi device.
> I have attached a file containing all information I know about.
>
> That should be very easy to reproduce.
>
> Did someone could explain that behavior to me?
Encryption costs and in this setup you have multiple layers here too, a 
ZFS pool ontop of lofi (doing the encryption) ontop of a ZVOL which is 
in your rpool.

So any write to the filesystems in "apool" have to go to ZFS, then to 
lofi (and be encrypted) then to the ZVOL that is in your rpool, that 
alone without the encryption adds to the IO requirements.

A fair comparison would be to do the same setup with the multiple pools 
and lofi but don''t have lofi do encryption.  That would tell you the 
overhead of the encryption that lofi does.

-- 
Darren J Moffat



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100417/69c43296/attachment.html>

On 17/04/2010 10:53, Mickael Lambert wrote:
> My mean is about bandwidth. what I see is that if I write
> xMb/s to zfs fs then zfs write nearly xMb/s to the pool and that''s
attended.
> This pool write nearly xMb/s to lofi and that''s attended.
> iostat lofi seems showing xMb/s input attended also.
>
> I am not enough skilled to investigate on what''s happening then
between
> lofi and the zvol but the other pool write 3*xMb/s to the physical drive.
> I am not sure if lofi and or the zvol made some kind of
> allocate/read/write/read that make the zvol write 2 or more time each
> blocks.
I don''t think there is a bug it is just a side effect of what happens 
because of the "a pool on lofi on zvol in a pool setup" you have.

-- 
Darren J Moffat