zfs discuss - Jan 2010 - server hang with compression on, ping timeouts from remote machine

Hello All,

I am running NTFS over iSCSI on a ZFS ZVOL volume with compression=gzip-9 and
blocksize=8K. The server is 2 core P4 3.0 Ghz with 5 GB of RAM.

Whenever I start copying files from Windows onto the ZFS disk, after about
100-200 Mb been copied the server starts to experience freezes. I have iostat
running, which freezes as well. Even pings on both of the network adapters are
reporting either 4000 ms or timeouts for when the freeze is happening.

I have reproduce the same behavior with a 1 GB test ZVOL. Whenever I do
sequential writes of 64 Kb with compression=gzip-9 I experience the freezes.
With compression=off it''s all good.

I''ve also experienced similar behavior (short freezes) when running zfs
send|zfs receive with compression on LOCALLY on ZVOLs again.

Has anyone else experienced this ? Know any of bug? This is on snv117.
-- 
This message posted from opensolaris.org

Hello Christo,

On Jan 31, 2010, at 4:07 PM, Christo Kutrovsky wrote:
> Hello All,
> 
> I am running NTFS over iSCSI on a ZFS ZVOL volume with compression=gzip-9
and blocksize=8K. The server is 2 core P4 3.0 Ghz with 5 GB of RAM.
> 
> Whenever I start copying files from Windows onto the ZFS disk, after about
100-200 Mb been copied the server starts to experience freezes. I have iostat
running, which freezes as well. Even pings on both of the network adapters are
reporting either 4000 ms or timeouts for when the freeze is happening.
> 
> I have reproduce the same behavior with a 1 GB test ZVOL. Whenever I do
sequential writes of 64 Kb with compression=gzip-9 I experience the freezes.
With compression=off it''s all good.
> 
> I''ve also experienced similar behavior (short freezes) when
running zfs send|zfs receive with compression on LOCALLY on ZVOLs again.
I think gzip in ZFS have a reputation being somewhat heavy on system resources,
that said it would be nice if it did not have such a large impact on low level
functions. Have a look in the archive, search for example death-spriral or
Death-spriral revisited. Have you tried using the default compression algorithm
also (lzjb, compresison=on)?

Regards

Henrik
http://sparcv9.blogspot.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100131/e27a72a9/attachment.html>

On Jan 31, 2010, at 7:21 AM, Henrik Johansson wrote:
> Hello Christo,
> 
> On Jan 31, 2010, at 4:07 PM, Christo Kutrovsky wrote:
> 
>> Hello All,
>> 
>> I am running NTFS over iSCSI on a ZFS ZVOL volume with
compression=gzip-9 and blocksize=8K. The server is 2 core P4 3.0 Ghz with 5 GB
of RAM.
For NTFS, use recordsize=4 KB, but I wouldn''t worry too much about
compression at
that recordsize.
>> Whenever I start copying files from Windows onto the ZFS disk, after
about 100-200 Mb been copied the server starts to experience freezes. I have
iostat running, which freezes as well. Even pings on both of the network
adapters are reporting either 4000 ms or timeouts for when the freeze is
happening.
>> 
>> I have reproduce the same behavior with a 1 GB test ZVOL. Whenever I do
sequential writes of 64 Kb with compression=gzip-9 I experience the freezes.
With compression=off it''s all good.
gzip-9 is a pig.  b115 includes the fix for:
	CR6586537  async zio taskqs can block out userland commands
which greatly reduced this effect.  But you might consider the default gzip-6
instead.

Back to my note above, compression is done to records, but the size is still
512 byte sectors. In other words, there are 8 sectors in a 4 KB record, so
compression is bounded by 12.5% chunks.
>> I''ve also experienced similar behavior (short freezes) when
running zfs send|zfs receive with compression on LOCALLY on ZVOLs again.
> 
> I think gzip in ZFS have a reputation being somewhat heavy on system
resources, that said it would be nice if it did not have such a large impact on
low level functions. Have a look in the archive, search for example
death-spriral or Death-spriral revisited. Have you tried using the default
compression algorithm also (lzjb, compresison=on)?
Good idea, and for small records compression gains less.
 -- richard

Thanks for your replies.

I am aware of the 512 bytes concept, thus my selection of 8 KB (matched with 8KB
ntfs). Even 20% reduction is still good, that''s like having 20% extra
ram (for cache).

I haven''t experimented with the default lzjb compression. If I want to
compress something usually I want it compressed well. Originally I had tried 64
Kb, but then I discovered windows does partial reads and writes (not entire
clusters), thus I decided to pick 8K something that fits in 9k jumbo frame.

Either way, I think it''s very bad for an OS compression to cause your
server to not respond to pings (other side affects aside).

I am running 117, thus the fix should be in place. Nevertheless it does point
out that there could be other things wrong with gzip compression and zfs.
-- 
This message posted from opensolaris.org

On 01/31/10 07:07, Christo Kutrovsky wrote:
> I''ve also experienced similar behavior (short freezes) when
running
> zfs send|zfs receive with compression on LOCALLY on ZVOLs again.
>
> Has anyone else experienced this ? Know any of bug? This is on
> snv117.
you might also get better results after the fix to:

6881015 ZFS write activity prevents other threads from running in a 
timely manner

which was fixed in build 129.

As a workaround, try a lower gzip compression level -- higher gzip
levels usually burn lots more CPU without significantly increasing the
compression ratio.

					- Bill

Thanks Bill, that looks relevant. Note however this only happens with gzip
compression, but it''s definiteness something I''ve experienced.

I''ve decided to wait for the next full release before upgrading. I was
just wondering if the problem was resolved.

I''ll migrate to COMSTAR soon, I hope the kernel mode iscsi will make a
difference.
-- 
This message posted from opensolaris.org