zfs discuss - Nov 2010 - ZFS send/receive and locking

Hello,

We are trying to setup a pair of ZFS file servers, each backing-up data from
another.
The simplified setup is as follows:

server1
tank/prod/web
tank/backup/mail

server2
tank/prod/mail
tank/backup/web

server1:tank/prod/web is a test setup with 10GB of data for 60 websites.
server2:tank/prod/mail is quite heavily loaded: it is 900GB with about 20k mail
accounts accessed from 3 mail servers(postfix+pop+imap) over NFS

We periodically sync production file systems between the two servers.
server2 pulls changes from server1:tank/prod/web as follows:
1. server1: zfs snapshot tank/prod/web at repli_N+1
2. server2: ssh server1 zfs send -R -I @repli_N tank/prod/web at repli_N+1 |
zfs recv -d -F tank/back
3. server1: zfs destroy tank/prod/web at repli_N
4. next send/receive in 30 minutes.

Our problem is that receiving the file system on server2 into tank/backup
causes a lock of the production file system tank/prod/mail.
The NFS clients (mail servers) cannot reach the NFS server2 at all.
All postfix processes at mail serves hang on the I/O and the load goes up to
100+.
After a while everything goes back to normal.

This behaviour is most noticeable when we try to add new data to
server1:tank/prod/web.
The mail servers scream , mail is not delivered, we get downtime notifications.
Yes, the snapshot diff is bigger then, because we are adding new data.
However the total of server1:tank/prod/web is 9.5GB. It is not that big.

We did not succeed to reproduce this behaviour with simple dd + zfs send + zfs
receive.
Real production data seems to be needed.

The questions are:
1. Is it possible that receiving a fs at one path blocks a file system at
another. Is the whole pool locked?
2. Is it possible to save the setup with 2 production servers backing up each
other
3. Otherwise would it help to have two storage pools - prod/ and backup/ instead
of tank/prod tank/backup
4. Or should we switch to a Master->Replicator setup?

Best regards,
Byte tech team


---------------------------------------
zpool status tank:

 pool: tank
state: ONLINE
scrub: none requested
config:

       NAME         STATE     READ WRITE CKSUM
       tank         ONLINE       0     0     0
         raidz2-0   ONLINE       0     0     0
           c2t0d0   ONLINE       0     0     0
           c2t1d0   ONLINE       0     0     0
           c2t2d0   ONLINE       0     0     0
           c2t3d0   ONLINE       0     0     0
           c2t4d0   ONLINE       0     0     0
           c2t5d0   ONLINE       0     0     0
         raidz2-1   ONLINE       0     0     0
           c2t6d0   ONLINE       0     0     0
           c2t7d0   ONLINE       0     0     0
           c2t8d0   ONLINE       0     0     0
           c2t9d0   ONLINE       0     0     0
           c2t10d0  ONLINE       0     0     0
           c2t11d0  ONLINE       0     0     0
       logs
         mirror-2   ONLINE       0     0     0
           c3t2d0   ONLINE       0     0     0
           c3t3d0   ONLINE       0     0     0
-- 
This message posted from opensolaris.org

Sorry i''m not able to provide more insight but I thought some of the
concepts in this article might help you, as well as Mike''s replication
script, also available on this page:
http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs

You also might want to look at InfraGeeks auto-replicate script:

http://www.infrageeks.com/groups/infrageeks/wiki/8fb35/zfs_autoreplicate_script.html

If the auto-replicate script works without locking things up, then you are
probably doing something wrong.  This script works great for me on b134
hosts.  One thing you might do to enable faster transfer speeds is enable
blowfish-cbc in your /etc/ssh/sshd_config and then modify the script to use
that cipher.

Cheers,

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20101103/c7d76c2d/attachment.html>

The problem is not with how the replication is done.  The locking happens during
the basic zfs operations.

We noticed:
on server2 (which is quite busy serving maildirs) we did

zfs create tank/newfs
rsync 4GB from someotherserver to /tank/newfs
zfs destroy tank/newfs

Destroying newfs took more than 30 minutes, and during this time the production
filesystem was inaccessible via NFS.

We got a hint on priv that we should try the following experiment

zfs create tank/tmp  
dd of=/tank/tmp/data [...]
zpool scrub tank 
zfs destroy tank/tmp

At this point the zfs destroy command gets suspended. 
Issuing 
zpool scrub -s  
causes the destroy to finish immediately.

The other part of the hint is that it is an issue with I/O scheduler and we
should upgrade.
We will provide the details a soon as we have sorted this out.
-- 
This message posted from opensolaris.org