zfs discuss - Mar 2011 - ZFS send/recv horribly slow on system with 1800+ filesystems

Hi all, I have a test system with a large amount of filesystems which
we take snapshots of and do send/recvs with.

On our test machine, we have 1800+ filesystems and about 5,000
snapshots.The system has 48GB of RAM, and 8 cores (x86). The
filesystem is comprised of 2 regular 1TB in a mirror with a 320GB
FusionIO flash card acting as a ZIL and read cache.

We''ve noticed that on systems with just a handful of filesystems, ZFS
send (recursive) is quite quick, but on our 1800+ fs box, it''s
horribly slow.

For example,

root at testbox:~# zfs send -R chunky/0 at async-2011-02-28-15:11:20| pv -i
1 > /dev/null

2.51GB 0:04:57 [47.4kB/s] [            <=>                                
]
^C

The other odd thing I''ve noticed is that during the ''zfs
send'' to
/dev/null, zpool iostat shows we''re actually *writing* to the zpool at
the rate of 4MB-8MB/s, but reading almost nothing. How can this be the
case?

So I''m left with 2 questions -

1.) Does ZFS get immensely slow once we have thousands of filesystems?

2.) Why do we see 4MB-8MB/s of *writes* to the filesystem when we do a
''zfs send'' to /dev/null ?


-Moazam

On Mon, Feb 28, 2011 at 10:38 PM, Moazam Raja <moazam at gmail.com>
wrote:
> We''ve noticed that on systems with just a handful of filesystems,
ZFS
> send (recursive) is quite quick, but on our 1800+ fs box, it''s
> horribly slow.
When doing an incremental send, the system has to identify what blocks
have changed, which can take some time. If not much data has changed,
the delay can take longer than the actual send.

I''ve noticed that there''s a small delay when starting a send
of a new
snapshot and when starting the receive of one. Putting something like
mbuffer in the path helps to smooth things out. It won''t help in the
example you''ve cited below, but it will help in real world use.
> The other odd thing I''ve noticed is that during the ''zfs
send'' to
> /dev/null, zpool iostat shows we''re actually *writing* to the
zpool at
> the rate of 4MB-8MB/s, but reading almost nothing. How can this be the
> case?
The writing seems odd, but the lack of reads doesn''t. You might have
most or all of the data in the ARC or L2ARC, so your zpool doesn''t
need to be read from.
> 1.) Does ZFS get immensely slow once we have thousands of filesystems?
No. Incremental sends might take longer, as I mentioned above.
> 2.) Why do we see 4MB-8MB/s of *writes* to the filesystem when we do a
> ''zfs send'' to /dev/null ?
Is anything else using the filesystems in the pool?

-B

-- 
Brandon High : bhigh at freaks.com