zfs discuss - Oct 2012 - Building an On-Site and Off-Size ZFS server, replication question

Good morning.

I am in the process of planning a system which will have 2 ZFS servers, one
on site, one off site. The on site server will be used by workstations and
servers in house, and most of that will stay in house. There will, however,
be data i want backed up somewhere else, which is where the offsite server
comes in... This server will be sitting in a Data Center and will have some
storage available to it (the whole server currently has 2 3Tb drives,
though they are not dedicated to the ZFS box, they are on VMware ESXi).
There is then some storage (currently 100Gb, but more can be requested) of
SFTP enabled backup which i plan to use for some snapshots, but more on
that later.

Anyway, i want to confirm my plan and make sure i am not missing anything
here...

* build server in house with storage, pools, etc...
* have a server in data center with enough storage for its reason, plus the
extra for offsite backup
* have one pool set as my "offsite" pool... anything in here should be
backed up off site also...
* possibly have another set as "very offsite" which will also be
pushed to
the SFTP server, but not sure...
* give these pools out via SMB/NFS/iSCSI
* every 6 or so hours take a snapshot of the 2 offsite pools.
* do a ZFS send to the data center box
* nightly, on the very offsite pool, do a ZFS send to the SFTP server
* if anything goes wrong (my server dies, DC server dies, etc), Panic,
download, pray... the usual... :)

Anyway, I want to make sure i am doing this correctly... Is there anything
on that list that sounds stupid or am i doing anything wrong? am i missing
anything?

Also, as a follow up question, but slightly unrelated, when it comes to the
ZFS Send, i could use SSH to do the send, directly to the machine... Or i
could upload the compressed, and possibly encrypted dump to the server...
Which, for resume-ability and speed, would be suggested? And if i where to
go with an upload option, any suggestions on what i should use?

Thanks.
-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/4a8c8221/attachment.html>

2012-10-05 11:17, Tiernan OToole wrote:
> Also, as a follow up question, but slightly unrelated, when it comes to
> the ZFS Send, i could use SSH to do the send, directly to the machine...
> Or i could upload the compressed, and possibly encrypted dump to the
> server... Which, for resume-ability and speed, would be suggested? And
> if i where to go with an upload option, any suggestions on what i should
> use?
As for this, the answer depends on network bandwidth, reliability,
and snapshot file size - ultimately, on the probability and retry
cost of an error during transmission.

Many posters on the list strongly object to using files as storage
for snapshot streams, because in reliability this is (may be) worse
than a single-disk pool and bitrot on it - a single-bit error in
a snapshot file can render it and all newer snapshots invalid and
un-importable.

Still, given enough scratch space on the sending and receiving sides
and a bad (slow, glitchy) network in-between, I did go with compressed
files of zfs-send streams (perhaps making recursion myself and using
smaller files of one snapshot each - YMMV). For compression on multiCPU
senders I can strongly suggest "pigz --fast $filename" (I did have
problems in pigz-1.7.1 compressing several files with one command,
maybe that''s fixed now). If you''re tight on space/transfer
size more
than on CPU, you can try other parallel algos - pbzip2, p7zip, etc.
Likewise, you can also pass the file into an encryptor of your choice.

Then I can rsync these files to the receiving server, using "rsync -c"
and/or md5sum, sha256sum, sha1sum or whatever tool(s) of your liking
to validate that the files received match those sent - better safe
than sorry. I''m usually using "rsync -cavPHK" for any work,
which
gives you retryable transfers in case network goes down, status bar,
directory recursion and hardlink support among other things.

NFS is also retryable if so configured (even if the receiver gets
rebooted in the process), and if you, for example, already have
VPN between two sites, you might find it faster than rsync which
involves extra encryption - maybe redundant in VPN case.

When the scratch area on the receiver has got and validated the
compressed snapshot stream, I can gzcat it and pipe into zfs recv.
This ultimately validates that the zfs-send stream arrived intact
and is fully receivable, and only then I can delete the temporary
files involved - or retry the send from different steps (it is
possible that the initial file was corrupted in RAM, etc.)

Note that such approach via files essentially disables zfs-send
deduplication which may be available in protocol between two
active zfs commands, but AFAIK this does not preclude you from
receiving data into deduped datasets - local dedup happens upon
block writes anyway, like compression, encryption and stuff like
that.

HTH,
//Jim Klimov

Thanks for that Jim!

Sounds like a plan there... One question about the storing ZFS dumps in a
file... So, the idea of storing the data in a SFTP server which has an
unknown underlying file system... Is that defiantly off limits, or can it
be done? and should i be doing a full dump or just an incremental one?
maybe incremental daily, and then a full dump weekly?

Thanks!

--Tiernan

On Fri, Oct 5, 2012 at 9:36 AM, Jim Klimov <jimklimov at cos.ru> wrote:
> 2012-10-05 11:17, Tiernan OToole wrote:
>
>> Also, as a follow up question, but slightly unrelated, when it comes to
>> the ZFS Send, i could use SSH to do the send, directly to the
machine...
>> Or i could upload the compressed, and possibly encrypted dump to the
>> server... Which, for resume-ability and speed, would be suggested? And
>> if i where to go with an upload option, any suggestions on what i
should
>> use?
>>
>
> As for this, the answer depends on network bandwidth, reliability,
> and snapshot file size - ultimately, on the probability and retry
> cost of an error during transmission.
>
> Many posters on the list strongly object to using files as storage
> for snapshot streams, because in reliability this is (may be) worse
> than a single-disk pool and bitrot on it - a single-bit error in
> a snapshot file can render it and all newer snapshots invalid and
> un-importable.
>
> Still, given enough scratch space on the sending and receiving sides
> and a bad (slow, glitchy) network in-between, I did go with compressed
> files of zfs-send streams (perhaps making recursion myself and using
> smaller files of one snapshot each - YMMV). For compression on multiCPU
> senders I can strongly suggest "pigz --fast $filename" (I did
have
> problems in pigz-1.7.1 compressing several files with one command,
> maybe that''s fixed now). If you''re tight on
space/transfer size more
> than on CPU, you can try other parallel algos - pbzip2, p7zip, etc.
> Likewise, you can also pass the file into an encryptor of your choice.
>
> Then I can rsync these files to the receiving server, using "rsync
-c"
> and/or md5sum, sha256sum, sha1sum or whatever tool(s) of your liking
> to validate that the files received match those sent - better safe
> than sorry. I''m usually using "rsync -cavPHK" for any
work, which
> gives you retryable transfers in case network goes down, status bar,
> directory recursion and hardlink support among other things.
>
> NFS is also retryable if so configured (even if the receiver gets
> rebooted in the process), and if you, for example, already have
> VPN between two sites, you might find it faster than rsync which
> involves extra encryption - maybe redundant in VPN case.
>
> When the scratch area on the receiver has got and validated the
> compressed snapshot stream, I can gzcat it and pipe into zfs recv.
> This ultimately validates that the zfs-send stream arrived intact
> and is fully receivable, and only then I can delete the temporary
> files involved - or retry the send from different steps (it is
> possible that the initial file was corrupted in RAM, etc.)
>
> Note that such approach via files essentially disables zfs-send
> deduplication which may be available in protocol between two
> active zfs commands, but AFAIK this does not preclude you from
> receiving data into deduped datasets - local dedup happens upon
> block writes anyway, like compression, encryption and stuff like
> that.
>
> HTH,
> //Jim Klimov
>
> ______________________________**_________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
>
http://mail.opensolaris.org/**mailman/listinfo/zfs-discuss<http://mail.opensolaris.org/mailman/listinfo/zfs-discuss>
>


-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/e87fd6bb/attachment-0001.html>

2012-10-05 13:13, Tiernan OToole wrote:
> Thanks for that Jim!
>
> Sounds like a plan there... One question about the storing ZFS dumps in
> a file... So, the idea of storing the data in a SFTP server which has an
> unknown underlying file system... Is that defiantly off limits, or can
> it be done?
Mileages do vary. Maybe you should pack the stream files into
archives with error-correction codes (or at least verification
CRCs) like ZIP, RAR, likely p7zip, maybe others; and also keep
checksum files. At least this can help detect or even fix small
nasty surprises.

The general concern is that zfs send streams have no built-in
redundancy, I''m not sure about error-checking - likely it is
there. And it is widely assumed that this being a stream, a small
error can redirect the flow widely differently from expectations
and cause the whole dataset state to be invalid (likely this
snapshot receiving will be aborted, and then you can''t receive
any newer ones over it).

That said, some people do keep the streams on tape; the NDMP
tools and protocol from Sun IIRC do the same for backups.
So it''s not off-limits, but precautions may be due (keep 2+
copies, do CRC/ECC and so on).

 > and should i be doing a full dump or just an incremental
 > one? maybe incremental daily, and then a full dump weekly?

A full dump of a large filesystem can be unbearably large for
storage and transfers. Still, the idea of storing occasional
full snapshots and a full history of incrementals (so that you
can try to recover starting from any of the full snapshots you
have) sounds sane. This way you have a sort of second copy by
virtue of a full snapshot incorporating some state of the dataset
and if the most recent one is broken - you can try to recover
with the one(s) before it and applying more incremental snapshots.
Likewise, errors in very old snapshots become irrelevant when a
newer full snapshot is intact. But sometimes two or more things
can break - or be detected to break - at once ;)

In particular, regular drills should be done (and provisioned
for) to test that you can in fact recover from your backups,
and that they do contain all the data you need. Older configs
can become obsolete as live systems evolve...

HTH,
//Jim

Thanks again Jim. Very handy info. This is now my weekend project, so
hopefully things go well!

--Tiernan

On Fri, Oct 5, 2012 at 10:40 AM, Jim Klimov <jimklimov at cos.ru> wrote:
> 2012-10-05 13:13, Tiernan OToole wrote:
>
>> Thanks for that Jim!
>>
>> Sounds like a plan there... One question about the storing ZFS dumps in
>> a file... So, the idea of storing the data in a SFTP server which has
an
>> unknown underlying file system... Is that defiantly off limits, or can
>> it be done?
>>
>
> Mileages do vary. Maybe you should pack the stream files into
> archives with error-correction codes (or at least verification
> CRCs) like ZIP, RAR, likely p7zip, maybe others; and also keep
> checksum files. At least this can help detect or even fix small
> nasty surprises.
>
> The general concern is that zfs send streams have no built-in
> redundancy, I''m not sure about error-checking - likely it is
> there. And it is widely assumed that this being a stream, a small
> error can redirect the flow widely differently from expectations
> and cause the whole dataset state to be invalid (likely this
> snapshot receiving will be aborted, and then you can''t receive
> any newer ones over it).
>
> That said, some people do keep the streams on tape; the NDMP
> tools and protocol from Sun IIRC do the same for backups.
> So it''s not off-limits, but precautions may be due (keep 2+
> copies, do CRC/ECC and so on).
>
>
> > and should i be doing a full dump or just an incremental
> > one? maybe incremental daily, and then a full dump weekly?
>
> A full dump of a large filesystem can be unbearably large for
> storage and transfers. Still, the idea of storing occasional
> full snapshots and a full history of incrementals (so that you
> can try to recover starting from any of the full snapshots you
> have) sounds sane. This way you have a sort of second copy by
> virtue of a full snapshot incorporating some state of the dataset
> and if the most recent one is broken - you can try to recover
> with the one(s) before it and applying more incremental snapshots.
> Likewise, errors in very old snapshots become irrelevant when a
> newer full snapshot is intact. But sometimes two or more things
> can break - or be detected to break - at once ;)
>
> In particular, regular drills should be done (and provisioned
> for) to test that you can in fact recover from your backups,
> and that they do contain all the data you need. Older configs
> can become obsolete as live systems evolve...
>
> HTH,
> //Jim
>
>
> ______________________________**_________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
>
http://mail.opensolaris.org/**mailman/listinfo/zfs-discuss<http://mail.opensolaris.org/mailman/listinfo/zfs-discuss>
>


-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/61fea756/attachment.html>

On 10/05/12 21:36, Jim Klimov wrote:
> 2012-10-05 11:17, Tiernan OToole wrote:
>> Also, as a follow up question, but slightly unrelated, when it comes to
>> the ZFS Send, i could use SSH to do the send, directly to the
machine...
>> Or i could upload the compressed, and possibly encrypted dump to the
>> server... Which, for resume-ability and speed, would be suggested? And
>> if i where to go with an upload option, any suggestions on what i
should
>> use?
> As for this, the answer depends on network bandwidth, reliability,
> and snapshot file size - ultimately, on the probability and retry
> cost of an error during transmission.
>
> Many posters on the list strongly object to using files as storage
> for snapshot streams, because in reliability this is (may be) worse
> than a single-disk pool and bitrot on it - a single-bit error in
> a snapshot file can render it and all newer snapshots invalid and
> un-importable.
>
> Still, given enough scratch space on the sending and receiving sides
> and a bad (slow, glitchy) network in-between, I did go with compressed
> files of zfs-send streams (perhaps making recursion myself and using
> smaller files of one snapshot each - YMMV). For compression on multiCPU
> senders I can strongly suggest "pigz --fast $filename" (I did
have
> problems in pigz-1.7.1 compressing several files with one command,
> maybe that''s fixed now). If you''re tight on
space/transfer size more
> than on CPU, you can try other parallel algos - pbzip2, p7zip, etc.
> Likewise, you can also pass the file into an encryptor of your choice.
I do have to suffer a slow, glitchy WAN to a remote server and rather 
than send stream files, I broke the data on the remote server into a 
more fine grained set of filesystems than I would do normally.  In this 
case, I made the directories under what would have been the leaf 
filesystems filesystems themselves.

By spreading the data over more filesystems, the individual incremental 
sends are smaller, so there is less data to resend if the link burps 
during a transfer.

-- 
Ian.

Thanks Ian. That sounds like an option also. The plan was to break up the
file systems anyway, since some i will want to be replicated remotely, and
others not as much.

--Tiernan

On Fri, Oct 5, 2012 at 11:17 AM, Ian Collins <ian at ianshome.com> wrote:
> On 10/05/12 21:36, Jim Klimov wrote:
>
>> 2012-10-05 11:17, Tiernan OToole wrote:
>>
>>> Also, as a follow up question, but slightly unrelated, when it
comes to
>>> the ZFS Send, i could use SSH to do the send, directly to the
machine...
>>> Or i could upload the compressed, and possibly encrypted dump to
the
>>> server... Which, for resume-ability and speed, would be suggested?
And
>>> if i where to go with an upload option, any suggestions on what i
should
>>> use?
>>>
>> As for this, the answer depends on network bandwidth, reliability,
>> and snapshot file size - ultimately, on the probability and retry
>> cost of an error during transmission.
>>
>> Many posters on the list strongly object to using files as storage
>> for snapshot streams, because in reliability this is (may be) worse
>> than a single-disk pool and bitrot on it - a single-bit error in
>> a snapshot file can render it and all newer snapshots invalid and
>> un-importable.
>>
>> Still, given enough scratch space on the sending and receiving sides
>> and a bad (slow, glitchy) network in-between, I did go with compressed
>> files of zfs-send streams (perhaps making recursion myself and using
>> smaller files of one snapshot each - YMMV). For compression on multiCPU
>> senders I can strongly suggest "pigz --fast $filename" (I did
have
>> problems in pigz-1.7.1 compressing several files with one command,
>> maybe that''s fixed now). If you''re tight on
space/transfer size more
>> than on CPU, you can try other parallel algos - pbzip2, p7zip, etc.
>> Likewise, you can also pass the file into an encryptor of your choice.
>>
>
> I do have to suffer a slow, glitchy WAN to a remote server and rather than
> send stream files, I broke the data on the remote server into a more fine
> grained set of filesystems than I would do normally.  In this case, I made
> the directories under what would have been the leaf filesystems filesystems
> themselves.
>
> By spreading the data over more filesystems, the individual incremental
> sends are smaller, so there is less data to resend if the link burps during
> a transfer.
>
> --
> Ian.
>
> ______________________________**_________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
>
http://mail.opensolaris.org/**mailman/listinfo/zfs-discuss<http://mail.opensolaris.org/mailman/listinfo/zfs-discuss>
>


-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/c9bb7912/attachment.html>

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Tiernan OToole
> 
> I am in the process of planning a system which will have 2 ZFS servers, one
on
> site, one off site. The on site server will be used by workstations and
servers
> in house, and most of that will stay in house. There will, however, be data
i
> want backed up somewhere else, which is where the offsite server comes
> in... This server will be sitting in a Data Center and will have some
storage
> available to it (the whole server currently has 2 3Tb drives, though they
are
> not dedicated to the ZFS box, they are on VMware ESXi). There is then some
> storage (currently 100Gb, but more can be requested) of SFTP enabled
> backup which i plan to use for some snapshots, but more on that later.
> 
> Anyway, i want to confirm my plan and make sure i am not missing anything
> here...
> 
> * build server in house with storage, pools, etc...
> * have a server in data center with enough storage for its reason, plus the
> extra for offsite backup
> * have one pool set as my "offsite" pool... anything in here
should be backed
> up off site also...
> * possibly have another set as "very offsite" which will also be
pushed to the
> SFTP server, but not sure...
> * give these pools out via SMB/NFS/iSCSI
> * every 6 or so hours take a snapshot of the 2 offsite pools.
> * do a ZFS send to the data center box
> * nightly, on the very offsite pool, do a ZFS send to the SFTP server
> * if anything goes wrong (my server dies, DC server dies, etc), Panic,
> download, pray... the usual... :)
> 
> Anyway, I want to make sure i am doing this correctly... Is there anything
on
> that list that sounds stupid or am i doing anything wrong? am i missing
> anything?
> 
> Also, as a follow up question, but slightly unrelated, when it comes to the
ZFS
> Send, i could use SSH to do the send, directly to the machine... Or i could
> upload the compressed, and possibly encrypted dump to the server... Which,
> for resume-ability and speed, would be?suggested? And if i where to go with
> an upload option, any suggestions on what i should use?
It is recommended, whenever possible, you should pipe the "zfs send"
directly into a "zfs receive" on the receiving system.  For two solid
reasons:

If a single bit is corrupted, the whole stream checksum is wrong and therefore
the whole stream is rejected.  So if this occurs, you want to detect it (in the
form of one incremental failed) and then correct it (in the form of the next
incremental succeeding).  Whereas, if you store your streams on storage, it will
go undetected, and everything after that point will be broken.

If you need to do a restore, from a stream stored on storage, then your only
choice is to restore the whole stream.  You cannot look inside and just get one
file.  But if you had been doing send | receive, then you obviously can look
inside the receiving filesystem and extract some individual specifics.

If the recipient system doesn''t support "zfs receive," you
might consider exporting an iscsi device, and allowing the sender system deal
with it directly.  Or share a filesystem (such as NFS) with the sender system,
and let the sender create a recipient filesystem inside a file container, so the
sender can deal with it directly.

On Fri, Oct 5, 2012 at 3:17 AM, Ian Collins <ian at ianshome.com> wrote:
> I do have to suffer a slow, glitchy WAN to a remote server and rather than
> send stream files, I broke the data *on the remote server* into a more
> fine grained set of filesystems than I would do normally.  In this case, I
> made the directories under what would have been the leaf filesystems
> filesystems themselves.
>
Meaning you also broke the data on the LOCAL server into the same set of
more granular filesystems?  Or is it now possible to zfs send a
subdirectory of a filesystem?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/34e30e14/attachment.html>

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Frank Cusack
> 
> On Fri, Oct 5, 2012 at 3:17 AM, Ian Collins <ian at ianshome.com>
wrote:
> I do have to suffer a slow, glitchy WAN to a remote server and rather than
> send stream files, I broke the data on the remote server into a more fine
> grained set of filesystems than I would do normally. ?In this case, I made
the
> directories under what would have been the leaf filesystems filesystems
> themselves.
> 
> Meaning you also broke the data on the LOCAL server into the same set of
> more granular filesystems?? Or is it now possible to zfs send a
subdirectory of
> a filesystem?
"zfs create" instead of "mkdir"

As Ian said - he didn''t zfs send subdirs, he made filesystems where he
otherwise would have used subdirs.

On 10/06/12 07:57, Edward Ned Harvey 
(opensolarisisdeadlongliveopensolaris) wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Frank Cusack
>>
>> On Fri, Oct 5, 2012 at 3:17 AM, Ian Collins<ian at ianshome.com> 
wrote:
>> I do have to suffer a slow, glitchy WAN to a remote server and rather
than
>> send stream files, I broke the data on the remote server into a more
fine
>> grained set of filesystems than I would do normally.  In this case, I
made the
>> directories under what would have been the leaf filesystems filesystems
>> themselves.
>>
>> Meaning you also broke the data on the LOCAL server into the same set
of
>> more granular filesystems?  Or is it now possible to zfs send a
subdirectory of
>> a filesystem?
> "zfs create" instead of "mkdir"
>
> As Ian said - he didn''t zfs send subdirs, he made filesystems
where he otherwise would have used subdirs.
>
That right.

I do have a lot of what would appear to be unnecessary filesystems, but 
after loosing the WAN 3 days into a large transfer, a change of tactic 
was required!

-- 
Ian.

On Fri, Oct 5, 2012 at 1:28 PM, Ian Collins <ian at ianshome.com> wrote:
> I do have a lot of what would appear to be unnecessary filesystems, but
> after loosing the WAN 3 days into a large transfer, a change of tactic was
> required!
>
I''ve recently (last year or so) gone the other way, and have made an
effort
to combine filesystems.  I''m now thinking of remote replication so
maybe
I''ll break them up again.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121005/758dd05c/attachment.html>

On 05/10/2012 15:01, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- 
>> bounces at opensolaris.org] On Behalf Of Tiernan OToole
>> 
>> I am in the process of planning a system which will have 2 ZFS 
>> servers, one on site, one off site. The on site server will be
>> used by workstations and servers in house, and most of that will
>> stay in house. There will, however, be data i want backed up
>> somewhere else, which is where the offsite server comes in... This
>> server will be sitting in a Data Center and will have some storage 
>> available to it (the whole server currently has 2 3Tb drives, 
>> though they are not dedicated to the ZFS box, they are on VMware 
>> ESXi). There is then some storage (currently 100Gb, but more can
>> be requested) of SFTP enabled backup which i plan to use for some 
>> snapshots, but more on that later.
>> 
>> Anyway, i want to confirm my plan and make sure i am not missing 
>> anything here...
>> 
>> * build server in house with storage, pools, etc... * have a
>> server in data center with enough storage for its reason, plus the
>> extra for offsite backup * have one pool set as my "offsite"
>> pool... anything in here should be backed up off site also... *
>> possibly have another set as "very offsite" which will also
be
>> pushed to the SFTP server, but not sure... * give these pools out
>> via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 
>> offsite pools. * do a ZFS send to the data center box * nightly,
>> on the very offsite pool, do a ZFS send to the SFTP server * if 
>> anything goes wrong (my server dies, DC server dies, etc), Panic, 
>> download, pray... the usual... :)
>> 
>> Anyway, I want to make sure i am doing this correctly... Is there 
>> anything on that list that sounds stupid or am i doing anything 
>> wrong? am i missing anything?
>> 
>> Also, as a follow up question, but slightly unrelated, when it 
>> comes to the ZFS Send, i could use SSH to do the send, directly to 
>> the machine... Or i could upload the compressed, and possibly 
>> encrypted dump to the server... Which, for resume-ability and 
>> speed, would be suggested? And if i where to go with an upload 
>> option, any suggestions on what i should use?
> 
> It is recommended, whenever possible, you should pipe the "zfs
send"
> directly into a "zfs receive" on the receiving system.  For two
> solid reasons:
> 
> If a single bit is corrupted, the whole stream checksum is wrong and 
> therefore the whole stream is rejected.  So if this occurs, you want 
> to detect it (in the form of one incremental failed) and then
> correct it (in the form of the next incremental succeeding).
> Whereas, if you store your streams on storage, it will go undetected,
> and everything after that point will be broken.
> 
> If you need to do a restore, from a stream stored on storage, then 
> your only choice is to restore the whole stream.  You cannot look 
> inside and just get one file.  But if you had been doing send | 
> receive, then you obviously can look inside the receiving filesystem 
> and extract some individual specifics.
> 
> If the recipient system doesn''t support "zfs receive,"
[...]
On that note, is there a minimal user-mode zfs thing that would allow
receiving a stream into an image file? No need for file/directory access
etc.
I was thinking maybe the zfs-fuse-on-linux project may have suitable bits?

On Oct 7, 2012, at 3:50 PM, Johannes Totz <johannes at jo-t.de> wrote:
> On 05/10/2012 15:01, Edward Ned Harvey
> (opensolarisisdeadlongliveopensolaris) wrote:
>>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- 
>>> bounces at opensolaris.org] On Behalf Of Tiernan OToole
>>> 
>>> I am in the process of planning a system which will have 2 ZFS 
>>> servers, one on site, one off site. The on site server will be
>>> used by workstations and servers in house, and most of that will
>>> stay in house. There will, however, be data i want backed up
>>> somewhere else, which is where the offsite server comes in... This
>>> server will be sitting in a Data Center and will have some storage 
>>> available to it (the whole server currently has 2 3Tb drives, 
>>> though they are not dedicated to the ZFS box, they are on VMware 
>>> ESXi). There is then some storage (currently 100Gb, but more can
>>> be requested) of SFTP enabled backup which i plan to use for some 
>>> snapshots, but more on that later.
>>> 
>>> Anyway, i want to confirm my plan and make sure i am not missing 
>>> anything here...
>>> 
>>> * build server in house with storage, pools, etc... * have a
>>> server in data center with enough storage for its reason, plus the
>>> extra for offsite backup * have one pool set as my
"offsite"
>>> pool... anything in here should be backed up off site also... *
>>> possibly have another set as "very offsite" which will
also be
>>> pushed to the SFTP server, but not sure... * give these pools out
>>> via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the 2 
>>> offsite pools. * do a ZFS send to the data center box * nightly,
>>> on the very offsite pool, do a ZFS send to the SFTP server * if 
>>> anything goes wrong (my server dies, DC server dies, etc), Panic, 
>>> download, pray... the usual... :)
>>> 
>>> Anyway, I want to make sure i am doing this correctly... Is there 
>>> anything on that list that sounds stupid or am i doing anything 
>>> wrong? am i missing anything?
>>> 
>>> Also, as a follow up question, but slightly unrelated, when it 
>>> comes to the ZFS Send, i could use SSH to do the send, directly to 
>>> the machine... Or i could upload the compressed, and possibly 
>>> encrypted dump to the server... Which, for resume-ability and 
>>> speed, would be suggested? And if i where to go with an upload 
>>> option, any suggestions on what i should use?
>> 
>> It is recommended, whenever possible, you should pipe the "zfs
send"
>> directly into a "zfs receive" on the receiving system.  For
two
>> solid reasons:
>> 
>> If a single bit is corrupted, the whole stream checksum is wrong and 
>> therefore the whole stream is rejected.  So if this occurs, you want 
>> to detect it (in the form of one incremental failed) and then
>> correct it (in the form of the next incremental succeeding).
>> Whereas, if you store your streams on storage, it will go undetected,
>> and everything after that point will be broken.
>> 
>> If you need to do a restore, from a stream stored on storage, then 
>> your only choice is to restore the whole stream.  You cannot look 
>> inside and just get one file.  But if you had been doing send | 
>> receive, then you obviously can look inside the receiving filesystem 
>> and extract some individual specifics.
>> 
>> If the recipient system doesn''t support "zfs
receive," [...]
> 
> On that note, is there a minimal user-mode zfs thing that would allow
> receiving a stream into an image file? No need for file/directory access
> etc.
cat :-)
> I was thinking maybe the zfs-fuse-on-linux project may have suitable bits?
I''m sure most Linux distros have cat
 -- richard

Ok, so, after reading a bit more of this discussion and after playing
around at the weekend, i have a couple of questions to ask...

1: Do my pools need to be the same? for example, the pool in the datacenter
is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual drives in
RAIDZ1, giving 800Gb usable. If i am backing up stuff to the home server,
can i still do a ZFS Send, even though underlying system is different?
2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended as
normal, or is there any difference?

Thanks.

--Tiernan

On Mon, Oct 8, 2012 at 3:51 AM, Richard Elling <richard.elling at
gmail.com>wrote:
> On Oct 7, 2012, at 3:50 PM, Johannes Totz <johannes at jo-t.de>
wrote:
>
> > On 05/10/2012 15:01, Edward Ned Harvey
> > (opensolarisisdeadlongliveopensolaris) wrote:
> >>> From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-
> >>> bounces at opensolaris.org] On Behalf Of Tiernan OToole
> >>>
> >>> I am in the process of planning a system which will have 2 ZFS
> >>> servers, one on site, one off site. The on site server will be
> >>> used by workstations and servers in house, and most of that
will
> >>> stay in house. There will, however, be data i want backed up
> >>> somewhere else, which is where the offsite server comes in...
This
> >>> server will be sitting in a Data Center and will have some
storage
> >>> available to it (the whole server currently has 2 3Tb drives,
> >>> though they are not dedicated to the ZFS box, they are on
VMware
> >>> ESXi). There is then some storage (currently 100Gb, but more
can
> >>> be requested) of SFTP enabled backup which i plan to use for
some
> >>> snapshots, but more on that later.
> >>>
> >>> Anyway, i want to confirm my plan and make sure i am not
missing
> >>> anything here...
> >>>
> >>> * build server in house with storage, pools, etc... * have a
> >>> server in data center with enough storage for its reason, plus
the
> >>> extra for offsite backup * have one pool set as my
"offsite"
> >>> pool... anything in here should be backed up off site also...
*
> >>> possibly have another set as "very offsite" which
will also be
> >>> pushed to the SFTP server, but not sure... * give these pools
out
> >>> via SMB/NFS/iSCSI * every 6 or so hours take a snapshot of the
2
> >>> offsite pools. * do a ZFS send to the data center box *
nightly,
> >>> on the very offsite pool, do a ZFS send to the SFTP server *
if
> >>> anything goes wrong (my server dies, DC server dies, etc),
Panic,
> >>> download, pray... the usual... :)
> >>>
> >>> Anyway, I want to make sure i am doing this correctly... Is
there
> >>> anything on that list that sounds stupid or am i doing
anything
> >>> wrong? am i missing anything?
> >>>
> >>> Also, as a follow up question, but slightly unrelated, when it
> >>> comes to the ZFS Send, i could use SSH to do the send,
directly to
> >>> the machine... Or i could upload the compressed, and possibly
> >>> encrypted dump to the server... Which, for resume-ability and
> >>> speed, would be suggested? And if i where to go with an upload
> >>> option, any suggestions on what i should use?
> >>
> >> It is recommended, whenever possible, you should pipe the
"zfs send"
> >> directly into a "zfs receive" on the receiving system. 
For two
> >> solid reasons:
> >>
> >> If a single bit is corrupted, the whole stream checksum is wrong
and
> >> therefore the whole stream is rejected.  So if this occurs, you
want
> >> to detect it (in the form of one incremental failed) and then
> >> correct it (in the form of the next incremental succeeding).
> >> Whereas, if you store your streams on storage, it will go
undetected,
> >> and everything after that point will be broken.
> >>
> >> If you need to do a restore, from a stream stored on storage, then
> >> your only choice is to restore the whole stream.  You cannot look
> >> inside and just get one file.  But if you had been doing send |
> >> receive, then you obviously can look inside the receiving
filesystem
> >> and extract some individual specifics.
> >>
> >> If the recipient system doesn''t support "zfs
receive," [...]
> >
> > On that note, is there a minimal user-mode zfs thing that would allow
> > receiving a stream into an image file? No need for file/directory
access
> > etc.
>
> cat :-)
>
> > I was thinking maybe the zfs-fuse-on-linux project may have suitable
> bits?
>
> I''m sure most Linux distros have cat
>  -- richard
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>


-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121008/bd270954/attachment.html>

On 10/08/12 20:08, Tiernan OToole wrote:
> Ok, so, after reading a bit more of this discussion and after playing 
> around at the weekend, i have a couple of questions to ask...
>
> 1: Do my pools need to be the same? for example, the pool in the 
> datacenter is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual 
> drives in RAIDZ1, giving 800Gb usable. If i am backing up stuff to the 
> home server, can i still do a ZFS Send, even though underlying system 
> is different?
Yes you can, just make sure you have enough space!

> 2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended 
> as normal, or is there any difference?
>
It can be sent as normal.

-- 
Ian.

Cool beans lads. Thanks!

On Mon, Oct 8, 2012 at 8:17 AM, Ian Collins <ian at ianshome.com> wrote:
> On 10/08/12 20:08, Tiernan OToole wrote:
>
>> Ok, so, after reading a bit more of this discussion and after playing
>> around at the weekend, i have a couple of questions to ask...
>>
>> 1: Do my pools need to be the same? for example, the pool in the
>> datacenter is 2 1Tb drives in Mirror. in house i have 5 200Gb virtual
>> drives in RAIDZ1, giving 800Gb usable. If i am backing up stuff to the
home
>> server, can i still do a ZFS Send, even though underlying system is
>> different?
>>
>
> Yes you can, just make sure you have enough space!
>
>
>  2: If i give out a partition as an iSCSI LUN, can this be ZFS Sended as
>> normal, or is there any difference?
>>
>>
> It can be sent as normal.
>
> --
> Ian.
>
> ______________________________**_________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
>
http://mail.opensolaris.org/**mailman/listinfo/zfs-discuss<http://mail.opensolaris.org/mailman/listinfo/zfs-discuss>
>


-- 
Tiernan O''Toole
blog.lotas-smartman.net
www.geekphotographer.com
www.tiernanotoole.ie
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121008/f350ee14/attachment-0001.html>

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Richard Elling
> 
> >> If the recipient system doesn''t support "zfs
receive," [...]
> >
> > On that note, is there a minimal user-mode zfs thing that would allow
> > receiving a stream into an image file? No need for file/directory
access
> > etc.
> 
> cat :-)
He was asking if it''s possible to do "zfs receive" on a
system that doesn''t natively support zfs.  The answer is no, unless you
want to consider fuse or similar.  I can''t speak about zfs on fuse or
anything - except that I personally wouldn''t trust it.  There are
differences even between zfs on solaris versus freebsd, vs whatever, all of
which are fully supported, much better than zfs on fuse.  But different people
use and swear by all of these things - so maybe it would actually be a good
solution for you.

The direction I would personally go would be an openindiana virtual machine to
do the zfs receive.

> > I was thinking maybe the zfs-fuse-on-linux project may have suitable
bits?
> 
> I''m sure most Linux distros have cat
hehe.  Anyway.  Answered above.

On Oct 10, 2012, at 9:29 AM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) <opensolarisisdeadlongliveopensolaris
at nedharvey.com> wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Richard Elling
>> 
>>>> If the recipient system doesn''t support "zfs
receive," [...]
>>> 
>>> On that note, is there a minimal user-mode zfs thing that would
allow
>>> receiving a stream into an image file? No need for file/directory
access
>>> etc.
>> 
>> cat :-)
> 
> He was asking if it''s possible to do "zfs receive" on a
system that doesn''t natively support zfs.  The answer is no, unless you
want to consider fuse or similar.
Read it again he asked, "On that note, is there a minimal user-mode zfs
thing that would allow
receiving a stream into an image file?"  Something like:
	zfs send ... | ssh user at host "cat > file"
>  I can''t speak about zfs on fuse or anything - except that I
personally wouldn''t trust it.  There are differences even between zfs
on solaris versus freebsd, vs whatever, all of which are fully supported, much
better than zfs on fuse.  But different people use and swear by all of these
things - so maybe it would actually be a good solution for you.
> 
> The direction I would personally go would be an openindiana virtual machine
to do the zfs receive.
> 
> 
>>> I was thinking maybe the zfs-fuse-on-linux project may have
suitable bits?
>> 
>> I''m sure most Linux distros have cat
> 
> hehe.  Anyway.  Answered above.
> 

 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422









-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121010/416fc169/attachment.html>

> From: Richard Elling [mailto:richard.elling at gmail.com]
> 
> Read it again he asked, "On that note, is there a minimal user-mode
zfs thing
> that would allow
> receiving a stream into an image file?" ?Something like:
> 	zfs send ... | ssh user at host "cat > file"
He didn''t say he wanted to cat to a file.  But it doesn''t
matter.  It was only clear from context, responding to the advice of "zfs
receive"ing into a zpool-in-a-file, that it was clear he was asking about
doing a "zfs receive" into a file, not just cat.  If you
weren''t paying close attention to the thread, it would be easy to
misunderstand what he was asking for.

When he asked for "minimal user-mode" he meant, something less than a
full-blown OS installation just for the purpose of zfs receive.  He went on to
say, he was considering zfs-fuse-on-linux.

On Oct 11, 2012, at 6:03 AM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) <opensolarisisdeadlongliveopensolaris
at nedharvey.com> wrote:
>> From: Richard Elling [mailto:richard.elling at gmail.com]
>> 
>> Read it again he asked, "On that note, is there a minimal
user-mode zfs thing
>> that would allow
>> receiving a stream into an image file?"  Something like:
>> 	zfs send ... | ssh user at host "cat > file"
> 
> He didn''t say he wanted to cat to a file.  But it doesn''t
matter.  It was only clear from context, responding to the advice of "zfs
receive"ing into a zpool-in-a-file, that it was clear he was asking about
doing a "zfs receive" into a file, not just cat.  If you
weren''t paying close attention to the thread, it would be easy to
misunderstand what he was asking for.
Pedantically, a pool can be made in a file, so it works the same...
> 
> When he asked for "minimal user-mode" he meant, something less
than a full-blown OS installation just for the purpose of zfs receive.  He went
on to say, he was considering zfs-fuse-on-linux.
... though I''m not convinced zfs-fuse supports files, whereas
illumos/Solaris does.
Perhaps a linux fuse person can respond.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422









-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121011/3bc51ce1/attachment-0001.html>

> From: Richard Elling [mailto:richard.elling at gmail.com]
> 
> Pedantically, a pool can be made in a file, so it works the same...
Pool can only be made in a file, by a system that is able to create a pool. 
Point is, his receiving system runs linux and doesn''t have any zfs; his
receiving system is remote from his sending system, and it has been suggested
that he might consider making an iscsi target available, so the sending system
could "zpool create" and "zfs receive" directly into a file
or device on the receiving system, but it doesn''t seem as if
that''s going to be possible for him - he''s expecting to
transport the data over ssh.  So he''s looking for a way to do a
"zfs receive" on a linux system, transported over ssh.  Suggested
answers so far include building a VM on the receiving side, to run openindiana
(or whatever) or using zfs-fuse-linux.

He is currently writing his "zfs send" datastream into a series of
files on the receiving system, but this has a few disadvantages as compared to
doing "zfs receive" on the receiving side.  Namely, increased risk of
data loss and less granularity for restores.  For these reasons, it''s
been suggested to find a way of receiving via "zfs receive" and
he''s exploring the possibilities of how to improve upon this situation.
Namely, how to "zfs receive" on a remote linux system via ssh, instead
of cat''ing or redirecting into a series of files.

There, I think I''ve recapped the whole thread now.   ;-)

2012-10-12 16:50, Edward Ned Harvey 
(opensolarisisdeadlongliveopensolaris) ?????:
>  So he''s looking for a way to do a "zfs receive" on a
linux system,
transported over ssh.  Suggested answers so far include building a VM on 
the receiving side, to run openindiana (or whatever) or using 
zfs-fuse-linux.


For completeness, if iSCSI target on the receiving host or another
similar solution is implemented, the secure networking part of
"zfs send over ssh" ("local" sending into a pool on an iSCSI
target)
can be done by a VPN, i.e. OpenVPN which uses the same OpenSSL
encryption.

//Jim

Jim, I''m trying to contact you off-list, but it doesn''t seem
to be working.  Can you please contact me off-list?

On Oct 12, 2012, at 5:50 AM, Edward Ned Harvey
(opensolarisisdeadlongliveopensolaris) <opensolarisisdeadlongliveopensolaris
at nedharvey.com> wrote:
>> From: Richard Elling [mailto:richard.elling at gmail.com]
>> 
>> Pedantically, a pool can be made in a file, so it works the same...
> 
> Pool can only be made in a file, by a system that is able to create a pool.
You can''t send a pool, you can only send a dataset. Whether you receive
the dataset
into a pool or file is a minor nit, the send stream itself is consistent.
> Point is, his receiving system runs linux and doesn''t have any
zfs; his receiving system is remote from his sending system, and it has been
suggested that he might consider making an iscsi target available, so the
sending system could "zpool create" and "zfs receive"
directly into a file or device on the receiving system, but it doesn''t
seem as if that''s going to be possible for him - he''s
expecting to transport the data over ssh.  So he''s looking for a way to
do a "zfs receive" on a linux system, transported over ssh.  Suggested
answers so far include building a VM on the receiving side, to run openindiana
(or whatever) or using zfs-fuse-linux.
> 
> He is currently writing his "zfs send" datastream into a series
of files on the receiving system, but this has a few disadvantages as compared
to doing "zfs receive" on the receiving side.  Namely, increased risk
of data loss and less granularity for restores.  For these reasons,
it''s been suggested to find a way of receiving via "zfs
receive" and he''s exploring the possibilities of how to improve
upon this situation.  Namely, how to "zfs receive" on a remote linux
system via ssh, instead of cat''ing or redirecting into a series of
files.
> 
> There, I think I''ve recapped the whole thread now.   ;-)

Yep, and cat works fine.
 -- richard

--

Richard.Elling at RichardElling.com
+1-760-896-4422









-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121012/9753d265/attachment-0001.html>