zfs discuss - Feb 2011 - Concurrent zfs send/recv for 100 file systems

Hi All,

I have two text files:

1. *snap_prev.txt:* This one lists the name of the immediate previous
snapshots of the 100 zfs file systems. One per line.

2. *snap_latest.txt:* This one contains the name of the corresponding latest
snapshots of those 100 zfs file systems. One per line.

Is it posible to send/recv these 100 snapshots to a second systems
concurrently? Can someone point me to some sample bash script or to some
URL?

If I send them one after another, it takes more time based on the size of
the filesystems. My idea is to send them concurrently so that it takes much
more less time for send/recv operation to the second system.

Thanks,

-- Rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110218/95b289bb/attachment.html>

Sorry, I forgot to mention. This is for Incremental Snapshots

On Fri, Feb 18, 2011 at 4:00 PM, Rahul Deb <rahul.deb at gmail.com> wrote:
> Hi All,
>
> I have two text files:
>
> 1. *snap_prev.txt:* This one lists the name of the immediate previous
> snapshots of the 100 zfs file systems. One per line.
>
> 2. *snap_latest.txt:* This one contains the name of the corresponding
> latest snapshots of those 100 zfs file systems. One per line.
>
> Is it posible to send/recv these 100 snapshots to a second systems
> concurrently? Can someone point me to some sample bash script or to some
> URL?
>
> If I send them one after another, it takes more time based on the size of
> the filesystems. My idea is to send them concurrently so that it takes much
> more less time for send/recv operation to the second system.
>
> Thanks,
>
> -- Rahul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110218/a0bf1e1e/attachment.html>

On 02/19/11 01:00 PM, Rahul Deb wrote:
> Hi All,
>
> I have two text files:
>
> 1. *snap_prev.txt:* This one lists the name of the immediate previous 
> snapshots of the 100 zfs file systems. One per line.
>
> 2. *snap_latest.txt:* This one contains the name of the corresponding 
> latest snapshots of those 100 zfs file systems. One per line.
>
> Is it posible to send/recv these 100 snapshots to a second systems 
> concurrently? Can someone point me to some sample bash script or to 
> some URL?
>
You could, but when I investigated this a couple of years ago it didn''t
gain much in the way of performance (I though I could make better use of 
network bandwidth, but discovered the receive end was IOP rather than 
throughput limited) and was a great way to lock up the receiving 
system.  The lockups are thankfully an thing of the past, but I still 
stagger sends to our backup server form other hosts.
> If I send them one after another, it takes more time based on the size 
> of the filesystems. My idea is to send them concurrently so that it 
> takes much more less time for send/recv operation to the second system.
>
I''d run some tests with typical filesystems.  Incremental sends with 
lots of small updates are a lot slower than ones with fewer, bigger changes.

-- 
Ian.