zfs discuss - Oct 2012 - [zfs] portable zfs send streams (preview webrev)

We have finished a beta version of the feature. A webrev for it
can be found here:

http://cr.illumos.org/~webrev/sensille/fits-send/

It adds a command ''zfs fits-send''. The resulting streams can
currently only be received on btrfs, but more receivers will
follow.
It would be great if anyone interested could give it some testing
and/or review. If there are no objections, I''ll send a formal
webrev soon.

Thanks,
Arne


On 10.10.2012 21:38, Arne Jansen wrote:
> Hi,
> 
> We''re currently working on a feature to send zfs streams in a
portable
> format that can be received on any filesystem.
> It is a kernel patch that generates the stream directly from kernel,
> analogous to what zfs send does.
> The stream format is based on the btrfs send format. The basic idea
> is to just send commands like mkdir, unlink, create, write, etc.
> For incremental sends it''s the same idea.
> The receiving side is user mode only, so it''s very easy to port it
to
> any system. If the receiving side has the capability to create
> snapshots, those from the sending side will get recreated. If not,
> you still have the benefit of fast incremental transfers.
> 
> My question is if there''s any interest in this feature and if it
had
> chances of getting accepted. Also, would it be acceptable to start with
> a working version and add performance optimizations later on?
> 
> -Arne

Arne Jansen wrote:
> We have finished a beta version of the feature.
What does FITS stand for?

On 10/18/2012 10:19 PM, Andrew Gabriel wrote:
> Arne Jansen wrote:
>> We have finished a beta version of the feature.
> 
> What does FITS stand for?
Filesystem Incremental Transport Stream
(or Filesystem Independent Transport Stream)

Arne Jansen <sensille at gmx.net> wrote:
> On 10/18/2012 10:19 PM, Andrew Gabriel wrote:
> > Arne Jansen wrote:
> >> We have finished a beta version of the feature.
> > 
> > What does FITS stand for?
>
> Filesystem Incremental Transport Stream
> (or Filesystem Independent Transport Stream)
Is this an attempt to create a competition for TAR?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On 19.10.2012 10:47, Joerg Schilling wrote:
> Arne Jansen <sensille at gmx.net> wrote:
> 
>> On 10/18/2012 10:19 PM, Andrew Gabriel wrote:
>>> Arne Jansen wrote:
>>>> We have finished a beta version of the feature.
>>>
>>> What does FITS stand for?
>>
>> Filesystem Incremental Transport Stream
>> (or Filesystem Independent Transport Stream)
> 
> Is this an attempt to create a competition for TAR?
Not really. We''d have preferred tar if it would have been powerful
enough.
It''s more an alternative to rsync for incremental updates. I really
like the send/receive feature and want to make it available for cross-
platform syncs.

Arne
> 
> J?rg
>

On Wed, Oct 17, 2012 at 2:29 PM, Arne Jansen <sensille at gmx.net>
wrote:
> We have finished a beta version of the feature. A webrev for it
> can be found here:
>
> http://cr.illumos.org/~webrev/sensille/fits-send/
>
> It adds a command ''zfs fits-send''. The resulting streams
can
> currently only be received on btrfs, but more receivers will
> follow.
> It would be great if anyone interested could give it some testing
> and/or review. If there are no objections, I''ll send a formal
> webrev soon.
Why are you trying to reinvent the wheel? AFAIK some tar versions and
AT&T AST pax support deltas based on a standard (I''ll have to dig
out
the exact specification, but from looking at it you did double work).

Irek

On 19.10.2012 11:16, Irek Szczesniak wrote:
> On Wed, Oct 17, 2012 at 2:29 PM, Arne Jansen <sensille at gmx.net>
wrote:
>> We have finished a beta version of the feature. A webrev for it
>> can be found here:
>>
>> http://cr.illumos.org/~webrev/sensille/fits-send/
>>
>> It adds a command ''zfs fits-send''. The resulting
streams can
>> currently only be received on btrfs, but more receivers will
>> follow.
>> It would be great if anyone interested could give it some testing
>> and/or review. If there are no objections, I''ll send a formal
>> webrev soon.
> 
> Why are you trying to reinvent the wheel? AFAIK some tar versions and
> AT&T AST pax support deltas based on a standard (I''ll have to
dig out
> the exact specification, but from looking at it you did double work).
> 
I haven''t done the research myself, but the result was that pax would
have needed significant extension, but I don''t have the details. If
you dig out a format already in use that supports everything we need
(like sharing data between files, needed for btrfs reflinks), it should
be easy to change the format. Stuffing the data into a specific format
is not an essential part of the work and can be changed with a limited
amount of work.

-Arne
> Irek

Arne Jansen <sensille at gmx.net> wrote:
> > Is this an attempt to create a competition for TAR?
>
> Not really. We''d have preferred tar if it would have been powerful
enough.
> It''s more an alternative to rsync for incremental updates. I
really
> like the send/receive feature and want to make it available for cross-
> platform syncs.
TAR with the star extensions that are also implemented by many other recent TAR 
programs should do, what are you missing?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On 19.10.2012 12:17, Joerg Schilling wrote:
> Arne Jansen <sensille at gmx.net> wrote:
> 
>>> Is this an attempt to create a competition for TAR?
>>
>> Not really. We''d have preferred tar if it would have been
powerful enough.
>> It''s more an alternative to rsync for incremental updates. I
really
>> like the send/receive feature and want to make it available for cross-
>> platform syncs.
> 
> TAR with the star extensions that are also implemented by many other recent
TAR
> programs should do, what are you missing?
As said I''ve not done the research myself, but operations that come to
mind
include
 - partial updates of files
 - sparse files
 - punch hole
 - truncate
 - rename
 - referencing parts of other files as the data to write (reflinks)
 - create snapshot

Do star support these operation? Are they part of any standard?

Also, are chmod/chown/set atime/mtime possible on existing files?

I''d be happy to find a well maintained receiver/format to send the
data in, this way I don''t need to care about a cross-platform
receiver :)

-Arne
> 
> J?rg
>

On Fri, Oct 19, 2012 at 11:23 AM, Arne Jansen <sensille at gmx.net>
wrote:
> On 19.10.2012 11:16, Irek Szczesniak wrote:
>> On Wed, Oct 17, 2012 at 2:29 PM, Arne Jansen <sensille at
gmx.net> wrote:
>>> We have finished a beta version of the feature. A webrev for it
>>> can be found here:
>>>
>>> http://cr.illumos.org/~webrev/sensille/fits-send/
>>>
>>> It adds a command ''zfs fits-send''. The resulting
streams can
>>> currently only be received on btrfs, but more receivers will
>>> follow.
>>> It would be great if anyone interested could give it some testing
>>> and/or review. If there are no objections, I''ll send a
formal
>>> webrev soon.
>>
>> Why are you trying to reinvent the wheel? AFAIK some tar versions and
>> AT&T AST pax support deltas based on a standard (I''ll have
to dig out
>> the exact specification, but from looking at it you did double work).
>>
>
> I haven''t done the research myself, but the result was that pax
would
> have needed significant extension, but I don''t have the details.
If
> you dig out a format already in use that supports everything we need
> (like sharing data between files, needed for btrfs reflinks), it should
> be easy to change the format. Stuffing the data into a specific format
> is not an essential part of the work and can be changed with a limited
> amount of work.
>
> -Arne
>
>> Irek
>
tar/pax was the initial format that was chosen for btrfs send/receive
as it looked like the best and most compatible way. In the middle of
development however I realized that we need more then storing whole
and incremental files/dirs in the format. We needed to store
information about moved, renamed, deleted, reflinked and even partial
clones where only some bits of a file are shared with another. This
can for sure all be implemented in pax, but then the next problem is
that in some situations renamed/moved files need multiple entries to
get to the desired result. For example, file a may be renamed to b
while at the same time file b got renamed to a. In such cases we need
3 entries that use a temporary name so that we don''t loose one of the
files while receiving. There are much more complex examples where it
gets quite complicated.

Also, it needed support for metadata (mode, size, uid/gid, ...)
changes on already existing files/dirs. Reusing already existent
tar/pax entry types was not possible for this as standard tar would
overwrite the original files with empty files.

I had all that implemented with pax, using a lot of custom pax
entries. A lot...so many that it didn''t look like tar/pax anymore. It
actually mutated from a list of file/dir/link entries (which tar/pax
is meant to be) to a list of filesystem instructions (rename, link,
unlink, rmdir, write parts of a file, clone parts of a file, chmod,
...).

My thought was, that this was already a big misuse of tar/pax, so I
decided to implement a simple format for this purpose only. Using pax
gave no advantages anymore. In tar/pax every entry must have a file
name, even the pax header entries need a file name. The problem now
is, that plain tar will treat every unknown entry type as regular file
and blindly overwrite existing ones which may result in data loss. To
prevent this, I always added something to the file name so that
unpacking with tar would not hurt the user. The unavoidable side
effect however is that the result of a plain untar is unusable without
further interpretation, which will be hard because tar by default does
not dump pax headers but instead ignores unknown entries.

Also, using tar/pax as the format for send/receive may give a user the
wrong impression that he can later use his good old standard tar to
restore his backups...this could be fatal for him.

Alex.

Arne Jansen <sensille at gmx.net> wrote:
> On 19.10.2012 12:17, Joerg Schilling wrote:
> > Arne Jansen <sensille at gmx.net> wrote:
> > 
> >>> Is this an attempt to create a competition for TAR?
> >>
> >> Not really. We''d have preferred tar if it would have been
powerful enough.
> >> It''s more an alternative to rsync for incremental
updates. I really
> >> like the send/receive feature and want to make it available for
cross-
> >> platform syncs.
> > 
> > TAR with the star extensions that are also implemented by many other
recent TAR
> > programs should do, what are you missing?
>
> As said I''ve not done the research myself, but operations that
come to mind
> include
>  - partial updates of files
How do you intend to detect this _after_ the original file was updated?
>  - sparse files
supported in an efficient way by star
>  - punch hole
As this is a specific case of a sparse file, it could be added
>  - truncate
see above
>  - rename
part of the incremental restore architecture from star, but needs a restore 
symbol table on the receiving system
>  - referencing parts of other files as the data to write (reflinks)
There is no user space  interface to detect this, why do you need it?
>  - create snapshot
star supports incrementals. or do you mean that a snapshot should be set up on 
the reeiving site?
> Do star support these operation? Are they part of any standard?
>
> Also, are chmod/chown/set atime/mtime possible on existing files?
star allows to call:

	star -x -xmeta

to _only_ extract meta data from a normal tar archive and it allows to create a 
specific meta data only archve via star -c -meta

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On 19.10.2012 13:53, Joerg Schilling wrote:
> Arne Jansen <sensille at gmx.net> wrote:
> 
>> On 19.10.2012 12:17, Joerg Schilling wrote:
>>> Arne Jansen <sensille at gmx.net> wrote:
>>>
>>>>> Is this an attempt to create a competition for TAR?
>>>>
>>>> Not really. We''d have preferred tar if it would have
been powerful enough.
>>>> It''s more an alternative to rsync for incremental
updates. I really
>>>> like the send/receive feature and want to make it available for
cross-
>>>> platform syncs.
>>>
>>> TAR with the star extensions that are also implemented by many
other recent TAR
>>> programs should do, what are you missing?
>>
>> As said I''ve not done the research myself, but operations that
come to mind
>> include
>>  - partial updates of files
> 
> How do you intend to detect this _after_ the original file was updated?
the patch is a kernel-side patch. It detects it by efficiently comparing
zfs snapshots. If you change a block in a file, we''ll send only the
changed
block.
> 
>>  - sparse files
> 
> supported in an efficient way by star
> 
>>  - punch hole
> 
> As this is a specific case of a sparse file, it could be added
While ''sparse file'' meant the initial transport of a sparse
file, ''punch hole''
is meant in the incremental context. Linux supports punching holes in files.
> 
>>  - truncate
> 
> see above
> 
>>  - rename
> 
> part of the incremental restore architecture from star, but needs a restore
> symbol table on the receiving system
Here, the sending sides determines renames.
> 
>>  - referencing parts of other files as the data to write (reflinks)
> 
> There is no user space  interface to detect this, why do you need it?
As above, it is detected in kernel. In the case of btrfs, each datablock
has backreferences to all its users. In zfs, the problem does not exist
in this form, nevertheless a common format has to be able to transport
this.
> 
>>  - create snapshot
> 
> star supports incrementals. or do you mean that a snapshot should be set up
on
> the reeiving site?
snapshot on the receiving side. The aim is to exactly replicate a number of
snapshots, just as zfs send/receive does.

Alexander Block (the author of btrfs send/receive) commented in another
subthread and explained the motivation to move away from tar/pax.

-Arne
> 
>> Do star support these operation? Are they part of any standard?
>>
>> Also, are chmod/chown/set atime/mtime possible on existing files?
> 
> star allows to call:
> 
> 	star -x -xmeta
> 
> to _only_ extract meta data from a normal tar archive and it allows to
create a
> specific meta data only archve via star -c -meta
> 
> J?rg
>

On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at gmx.net> wrote:
> We have finished a beta version of the feature. A webrev for it
> can be found here:
>
> http://cr.illumos.org/~webrev/sensille/fits-send/
>
> It adds a command ''zfs fits-send''. The resulting streams
can
> currently only be received on btrfs, but more receivers will
> follow.
> It would be great if anyone interested could give it some testing
> and/or review. If there are no objections, I''ll send a formal
> webrev soon.

I haven''t looked at the FITS-specific code yet but have some high-level
comments:

I''m not sure that we should add another subcommand for this, vs. a new
flag
to zfs send.

The final review should include the code to receive this stream with a new
command in /usr/bin.

Please don''t bother changing libzfs (and proliferating the copypasta
there)
-- do it like lzc_send().

Likewise, zfs_ioc_fits_send should use the new-style API.  See the comment
at the beginning of zfs_ioctl.c.

I''m not a fan of the name "FITS" but I suppose somebody else
already named
the format.  If we are going to follow someone else''s format though, it
at
least needs to be well-documented.  Where can we find the documentation?

FYI, #1 google hit for "FITS":  http://en.wikipedia.org/wiki/FITS
#3 hit:  http://code.google.com/p/fits/

Both have to do with file formats.  The entire first page of google results
for "FITS format" and "FITS file format" are related to
these two formats.
 "FITS btrfs" didn''t return anything specific to the file
format, either.

--matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121019/19689d75/attachment.html>

On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
> On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at gmx.net
> <mailto:sensille at gmx.net>> wrote:
> 
>     We have finished a beta version of the feature. A webrev for it
>     can be found here:
> 
>     http://cr.illumos.org/~webrev/sensille/fits-send/
> 
>     It adds a command ''zfs fits-send''. The resulting
streams can
>     currently only be received on btrfs, but more receivers will
>     follow.
>     It would be great if anyone interested could give it some testing
>     and/or review. If there are no objections, I''ll send a formal
>     webrev soon.
> 
> 
> 
> Please don''t bother changing libzfs (and proliferating the
copypasta
> there) -- do it like lzc_send().
> 
ok. It would be easier though if zfs_send would also already use the
new style. Is it in the pipeline already?
> Likewise, zfs_ioc_fits_send should use the new-style API.  See the
> comment at the beginning of zfs_ioctl.c.
> 
> I''m not a fan of the name "FITS" but I suppose somebody
else already
> named the format.  If we are going to follow someone else''s format
> though, it at least needs to be well-documented.  Where can we find the
> documentation?
> 
> FYI, #1 google hit for "FITS":  http://en.wikipedia.org/wiki/FITS
> #3 hit:  http://code.google.com/p/fits/
> 
> Both have to do with file formats.  The entire first page of google
> results for "FITS format" and "FITS file format" are
related to these
> two formats.  "FITS btrfs" didn''t return anything
specific to the file
> format, either.
It''s not too late to change it, but I have a hard time coming up with
some better name. Also, the format is still very new and I''m sure
it''ll
need some adjustments.

-arne
> 
> --matt
> 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at gmx.net> wrote:
> On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
> > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at gmx.net
> > <mailto:sensille at gmx.net>> wrote:
> >
> >     We have finished a beta version of the feature. A webrev for it
> >     can be found here:
> >
> >     http://cr.illumos.org/~webrev/sensille/fits-send/
> >
> >     It adds a command ''zfs fits-send''. The resulting
streams can
> >     currently only be received on btrfs, but more receivers will
> >     follow.
> >     It would be great if anyone interested could give it some testing
> >     and/or review. If there are no objections, I''ll send a
formal
> >     webrev soon.
> >
> >
> >
> > Please don''t bother changing libzfs (and proliferating the
copypasta
> > there) -- do it like lzc_send().
> >
>
> ok. It would be easier though if zfs_send would also already use the
> new style. Is it in the pipeline already?
>
> > Likewise, zfs_ioc_fits_send should use the new-style API.  See the
> > comment at the beginning of zfs_ioctl.c.
> >
> > I''m not a fan of the name "FITS" but I suppose
somebody else already
> > named the format.  If we are going to follow someone else''s
format
> > though, it at least needs to be well-documented.  Where can we find
the
> > documentation?
> >
> > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
> > #3 hit:  http://code.google.com/p/fits/
> >
> > Both have to do with file formats.  The entire first page of google
> > results for "FITS format" and "FITS file format"
are related to these
> > two formats.  "FITS btrfs" didn''t return anything
specific to the file
> > format, either.
>
> It''s not too late to change it, but I have a hard time coming up
with
> some better name. Also, the format is still very new and I''m sure
it''ll
> need some adjustments.
>
> -arne
>
> >
> > --matt
>
>
I''m sure we can come up with something.  Are you planning on this being
solely for ZFS, or a larger architecture for replication both directions in
the future?

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121019/8c3b6843/attachment.html>

On Fri, Oct 19, 2012 at 1:46 PM, Arne Jansen <sensille at gmx.net> wrote:
> On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
> > Please don''t bother changing libzfs (and proliferating the
copypasta
> > there) -- do it like lzc_send().
> >
>
> ok. It would be easier though if zfs_send would also already use the
> new style. Is it in the pipeline already?
>
> > Likewise, zfs_ioc_fits_send should use the new-style API.  See the
> > comment at the beginning of zfs_ioctl.c.
>
I''m saying to use lzc_send() as an example, rather than zfs_send().
 lzc_send() already uses the new style.  I don''t see how your job would
be
made easier by converting zfs_send().

It would be nice to convert ZFS_IOC_SEND to the new IOCTL format someday,
but I don''t think that the complexities of zfs_send() would be
appropriate
for libzfs_core.  Programmatic consumers typically know exactly what
snapshots they want sent and would prefer the clean error handling of
lzc_send().

--matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121019/1d29f733/attachment.html>

On 10/20/2012 01:10 AM, Tim Cook wrote:
> 
> 
> On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at gmx.net
> <mailto:sensille at gmx.net>> wrote:
> 
>     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
>     > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at
gmx.net
>     <mailto:sensille at gmx.net>
>     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>> wrote:
>     >
>     >     We have finished a beta version of the feature. A webrev for
it
>     >     can be found here:
>     >
>     >     http://cr.illumos.org/~webrev/sensille/fits-send/
>     >
>     >     It adds a command ''zfs fits-send''. The
resulting streams can
>     >     currently only be received on btrfs, but more receivers will
>     >     follow.
>     >     It would be great if anyone interested could give it some
testing
>     >     and/or review. If there are no objections, I''ll send
a formal
>     >     webrev soon.
>     >
>     >
>     >
>     > Please don''t bother changing libzfs (and proliferating
the copypasta
>     > there) -- do it like lzc_send().
>     >
> 
>     ok. It would be easier though if zfs_send would also already use the
>     new style. Is it in the pipeline already?
> 
>     > Likewise, zfs_ioc_fits_send should use the new-style API.  See the
>     > comment at the beginning of zfs_ioctl.c.
>     >
>     > I''m not a fan of the name "FITS" but I suppose
somebody else already
>     > named the format.  If we are going to follow someone
else''s format
>     > though, it at least needs to be well-documented.  Where can we
>     find the
>     > documentation?
>     >
>     > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
>     > #3 hit:  http://code.google.com/p/fits/
>     >
>     > Both have to do with file formats.  The entire first page of
google
>     > results for "FITS format" and "FITS file
format" are related to these
>     > two formats.  "FITS btrfs" didn''t return
anything specific to the file
>     > format, either.
> 
>     It''s not too late to change it, but I have a hard time coming
up with
>     some better name. Also, the format is still very new and I''m
sure it''ll
>     need some adjustments.
> 
>     -arne
> 
>     >
>     > --matt
> 
> 
> 
> I''m sure we can come up with something.  Are you planning on this
being
> solely for ZFS, or a larger architecture for replication both directions
> in the future?
We have senders for zfs and btrfs. The planned receiver will be mostly
filesystem agnostic and can work on a much broader range. It basically
only needs to know how to create snapshots and where to store a few
meta informations.
It would be great if more filesystems would join on the sending side,
but I have no involvement there.

I see no basic problem in choosing a name that''s already in use.
Especially with file extensions most will be already taken. How about
something with ''portable'' and ''backup'', like
pib or pibs? ''i'' for
incremental.

-Arne

> 
> --Tim
>  
>

On 10/20/2012 01:21 AM, Matthew Ahrens wrote:
> On Fri, Oct 19, 2012 at 1:46 PM, Arne Jansen <sensille at gmx.net
> <mailto:sensille at gmx.net>> wrote:
> 
>     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
>     > Please don''t bother changing libzfs (and proliferating
the copypasta
>     > there) -- do it like lzc_send().
>     >
> 
>     ok. It would be easier though if zfs_send would also already use the
>     new style. Is it in the pipeline already?
> 
>     > Likewise, zfs_ioc_fits_send should use the new-style API.  See the
>     > comment at the beginning of zfs_ioctl.c.
> 
> 
> I''m saying to use lzc_send() as an example, rather than
zfs_send().
>  lzc_send() already uses the new style.  I don''t see how your job
would
> be made easier by converting zfs_send().
Yeah, but the zfs util still uses the old version.
> 
> It would be nice to convert ZFS_IOC_SEND to the new IOCTL format
> someday, but I don''t think that the complexities of zfs_send()
would be
> appropriate for libzfs_core.  Programmatic consumers typically know
> exactly what snapshots they want sent and would prefer the clean error
> handling of lzc_send().
What I meant was if you want the full-blown zfs send-functionality with
the ton of options, it would be much easier to reuse the existing logic
and only call *_send_fits instead of *_send when requested.
If you''re content with just the -i option I''ve currently
implemented,
it''s certainly easy to convert. I on my part have mostly programmatic
consumers.

-Arne
> 
> --matt

On Sat, Oct 20, 2012 at 2:54 AM, Arne Jansen <sensille at gmx.net> wrote:
> On 10/20/2012 01:10 AM, Tim Cook wrote:
> >
> >
> > On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at gmx.net
> > <mailto:sensille at gmx.net>> wrote:
> >
> >     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
> >     > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at
gmx.net
> >     <mailto:sensille at gmx.net>
> >     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>> wrote:
> >     >
> >     >     We have finished a beta version of the feature. A webrev
for it
> >     >     can be found here:
> >     >
> >     >     http://cr.illumos.org/~webrev/sensille/fits-send/
> >     >
> >     >     It adds a command ''zfs fits-send''. The
resulting streams can
> >     >     currently only be received on btrfs, but more receivers
will
> >     >     follow.
> >     >     It would be great if anyone interested could give it some
> testing
> >     >     and/or review. If there are no objections, I''ll
send a formal
> >     >     webrev soon.
> >     >
> >     >
> >     >
> >     > Please don''t bother changing libzfs (and
proliferating the
> copypasta
> >     > there) -- do it like lzc_send().
> >     >
> >
> >     ok. It would be easier though if zfs_send would also already use
the
> >     new style. Is it in the pipeline already?
> >
> >     > Likewise, zfs_ioc_fits_send should use the new-style API. 
See the
> >     > comment at the beginning of zfs_ioctl.c.
> >     >
> >     > I''m not a fan of the name "FITS" but I
suppose somebody else
> already
> >     > named the format.  If we are going to follow someone
else''s format
> >     > though, it at least needs to be well-documented.  Where can
we
> >     find the
> >     > documentation?
> >     >
> >     > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
> >     > #3 hit:  http://code.google.com/p/fits/
> >     >
> >     > Both have to do with file formats.  The entire first page of
google
> >     > results for "FITS format" and "FITS file
format" are related to
> these
> >     > two formats.  "FITS btrfs" didn''t return
anything specific to the
> file
> >     > format, either.
> >
> >     It''s not too late to change it, but I have a hard time
coming up with
> >     some better name. Also, the format is still very new and
I''m sure
> it''ll
> >     need some adjustments.
> >
> >     -arne
> >
> >     >
> >     > --matt
> >
> >
> >
> > I''m sure we can come up with something.  Are you planning on
this being
> > solely for ZFS, or a larger architecture for replication both
directions
> > in the future?
>
> We have senders for zfs and btrfs. The planned receiver will be mostly
> filesystem agnostic and can work on a much broader range. It basically
> only needs to know how to create snapshots and where to store a few
> meta informations.
> It would be great if more filesystems would join on the sending side,
> but I have no involvement there.
>
> I see no basic problem in choosing a name that''s already in use.
> Especially with file extensions most will be already taken. How about
> something with ''portable'' and ''backup'',
like pib or pibs? ''i'' for
> incremental.
>
> -Arne
>
>
Re-using names generally isn''t a big deal, but in this case the
existing
name is a technology that''s extremely similar to what you''re
doing - which
WILL cause a ton of confusion in the userbase, and make troubleshooting far
more difficult when searching google/etc looking for links to documents
that are applicable.

Maybe something like far - filesystem agnostic replication?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121020/619a6887/attachment-0001.html>

On Sat, Oct 20, 2012 at 1:23 AM, Arne Jansen <sensille at gmx.net> wrote:
> On 10/20/2012 01:21 AM, Matthew Ahrens wrote:
> > On Fri, Oct 19, 2012 at 1:46 PM, Arne Jansen <sensille at gmx.net
> > <mailto:sensille at gmx.net>> wrote:
> >
> >     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
> >     > Please don''t bother changing libzfs (and
proliferating the
> copypasta
> >     > there) -- do it like lzc_send().
> >     >
> >
> >     ok. It would be easier though if zfs_send would also already use
the
> >     new style. Is it in the pipeline already?
> >
> >     > Likewise, zfs_ioc_fits_send should use the new-style API. 
See the
> >     > comment at the beginning of zfs_ioctl.c.
> >
> >
> > I''m saying to use lzc_send() as an example, rather than
zfs_send().
> >  lzc_send() already uses the new style.  I don''t see how your
job would
> > be made easier by converting zfs_send().
>
> Yeah, but the zfs util still uses the old version.
>
True, but it needs to do lots of crazy stuff (e.g. zfs rename; properties),
which isn''t supported by your routines.

> >
> > It would be nice to convert ZFS_IOC_SEND to the new IOCTL format
> > someday, but I don''t think that the complexities of
zfs_send() would be
> > appropriate for libzfs_core.  Programmatic consumers typically know
> > exactly what snapshots they want sent and would prefer the clean error
> > handling of lzc_send().
>
> What I meant was if you want the full-blown zfs send-functionality with
> the ton of options, it would be much easier to reuse the existing logic
> and only call *_send_fits instead of *_send when requested.
> If you''re content with just the -i option I''ve currently
implemented,
> it''s certainly easy to convert. I on my part have mostly
programmatic
> consumers.

I think it would be hard to make the full-blown zfs send work in
a receiving-platform-agnostic way -- it''s just too ZFS specific
(e.g. hierarchical filesystems, each of our properties (checksum, copies,
aclmode, ...).  As long as we are starting anew with nothing to be
compatibile with, might as well keep it simple.

--matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121021/876965f2/attachment.html>

On Sat, Oct 20, 2012 at 1:24 PM, Tim Cook <tim at cook.ms> wrote:
>
>
> On Sat, Oct 20, 2012 at 2:54 AM, Arne Jansen <sensille at gmx.net>
wrote:
>
>> On 10/20/2012 01:10 AM, Tim Cook wrote:
>> >
>> >
>> > On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at
gmx.net
>> > <mailto:sensille at gmx.net>> wrote:
>> >
>> >     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
>> >     > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille
at gmx.net
>> >     <mailto:sensille at gmx.net>
>> >     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>> wrote:
>> >     >
>> >     >     We have finished a beta version of the feature. A
webrev for
>> it
>> >     >     can be found here:
>> >     >
>> >     >     http://cr.illumos.org/~webrev/sensille/fits-send/
>> >     >
>> >     >     It adds a command ''zfs fits-send''.
The resulting streams can
>> >     >     currently only be received on btrfs, but more
receivers will
>> >     >     follow.
>> >     >     It would be great if anyone interested could give it
some
>> testing
>> >     >     and/or review. If there are no objections,
I''ll send a formal
>> >     >     webrev soon.
>> >     >
>> >     >
>> >     >
>> >     > Please don''t bother changing libzfs (and
proliferating the
>> copypasta
>> >     > there) -- do it like lzc_send().
>> >     >
>> >
>> >     ok. It would be easier though if zfs_send would also already
use the
>> >     new style. Is it in the pipeline already?
>> >
>> >     > Likewise, zfs_ioc_fits_send should use the new-style API.
See the
>> >     > comment at the beginning of zfs_ioctl.c.
>> >     >
>> >     > I''m not a fan of the name "FITS" but I
suppose somebody else
>> already
>> >     > named the format.  If we are going to follow someone
else''s format
>> >     > though, it at least needs to be well-documented.  Where
can we
>> >     find the
>> >     > documentation?
>> >     >
>> >     > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
>> >     > #3 hit:  http://code.google.com/p/fits/
>> >     >
>> >     > Both have to do with file formats.  The entire first page
of
>> google
>> >     > results for "FITS format" and "FITS file
format" are related to
>> these
>> >     > two formats.  "FITS btrfs" didn''t
return anything specific to the
>> file
>> >     > format, either.
>> >
>> >     It''s not too late to change it, but I have a hard
time coming up
>> with
>> >     some better name. Also, the format is still very new and
I''m sure
>> it''ll
>> >     need some adjustments.
>> >
>> >     -arne
>> >
>> >     >
>> >     > --matt
>> >
>> >
>> >
>> > I''m sure we can come up with something.  Are you planning
on this being
>> > solely for ZFS, or a larger architecture for replication both
directions
>> > in the future?
>>
>> We have senders for zfs and btrfs. The planned receiver will be mostly
>> filesystem agnostic and can work on a much broader range. It basically
>> only needs to know how to create snapshots and where to store a few
>> meta informations.
>> It would be great if more filesystems would join on the sending side,
>> but I have no involvement there.
>>
>> I see no basic problem in choosing a name that''s already in
use.
>> Especially with file extensions most will be already taken. How about
>> something with ''portable'' and
''backup'', like pib or pibs? ''i'' for
>> incremental.
>>
>> -Arne
>>
>>
> Re-using names generally isn''t a big deal, but in this case the
existing
> name is a technology that''s extremely similar to what
you''re doing - which
> WILL cause a ton of confusion in the userbase, and make troubleshooting far
> more difficult when searching google/etc looking for links to documents
> that are applicable.
>
> Maybe something like far - filesystem agnostic replication?
>
All else being equal, I like this name (FAR).  It ends in "AR" like
several
other archive formats (TAR, WAR, JAR).  Plus not a lot of false positives
when googling around for it.

However, if compatibility with the existing format is an explicit goal, we
should use the same name, and the btrfs authors may be averse to changing
the name.

--matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121021/c17f21a3/attachment-0001.html>

On 20.10.2012 22:24, Tim Cook wrote:
> 
> 
> On Sat, Oct 20, 2012 at 2:54 AM, Arne Jansen <sensille at gmx.net
> <mailto:sensille at gmx.net>> wrote:
> 
>     On 10/20/2012 01:10 AM, Tim Cook wrote:
>     >
>     >
>     > On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at
gmx.net
>     <mailto:sensille at gmx.net>
>     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>> wrote:
>     >
>     >     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
>     >     > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille
at gmx.net
>     <mailto:sensille at gmx.net>
>     >     <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>
>     >     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>
>     <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>>> wrote:
>     >     >
>     >     >     We have finished a beta version of the feature. A
webrev for it
>     >     >     can be found here:
>     >     >
>     >     >     http://cr.illumos.org/~webrev/sensille/fits-send/
>     >     >
>     >     >     It adds a command ''zfs fits-send''.
The resulting streams can
>     >     >     currently only be received on btrfs, but more
receivers will
>     >     >     follow.
>     >     >     It would be great if anyone interested could give it
some testing
>     >     >     and/or review. If there are no objections,
I''ll send a formal
>     >     >     webrev soon.
>     >     >
>     >     >
>     >     >
>     >     > Please don''t bother changing libzfs (and
proliferating the copypasta
>     >     > there) -- do it like lzc_send().
>     >     >
>     >
>     >     ok. It would be easier though if zfs_send would also already
use the
>     >     new style. Is it in the pipeline already?
>     >
>     >     > Likewise, zfs_ioc_fits_send should use the new-style API.
See the
>     >     > comment at the beginning of zfs_ioctl.c.
>     >     >
>     >     > I''m not a fan of the name "FITS" but I
suppose somebody else already
>     >     > named the format.  If we are going to follow someone
else''s format
>     >     > though, it at least needs to be well-documented.  Where
can we
>     >     find the
>     >     > documentation?
>     >     >
>     >     > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
>     >     > #3 hit:  http://code.google.com/p/fits/
>     >     >
>     >     > Both have to do with file formats.  The entire first page
of google
>     >     > results for "FITS format" and "FITS file
format" are related to these
>     >     > two formats.  "FITS btrfs" didn''t
return anything specific to the file
>     >     > format, either.
>     >
>     >     It''s not too late to change it, but I have a hard
time coming up with
>     >     some better name. Also, the format is still very new and
I''m sure it''ll
>     >     need some adjustments.
>     >
>     >     -arne
>     >
>     >     >
>     >     > --matt
>     >
>     >
>     >
>     > I''m sure we can come up with something.  Are you planning
on this being
>     > solely for ZFS, or a larger architecture for replication both
directions
>     > in the future?
> 
>     We have senders for zfs and btrfs. The planned receiver will be mostly
>     filesystem agnostic and can work on a much broader range. It basically
>     only needs to know how to create snapshots and where to store a few
>     meta informations.
>     It would be great if more filesystems would join on the sending side,
>     but I have no involvement there.
> 
>     I see no basic problem in choosing a name that''s already in
use.
>     Especially with file extensions most will be already taken. How about
>     something with ''portable'' and
''backup'', like pib or pibs? ''i'' for
>     incremental.
> 
>     -Arne
> 
> 
> Re-using names generally isn''t a big deal, but in this case the
existing name is
> a technology that''s extremely similar to what you''re
doing - which WILL cause a
> ton of confusion in the userbase, and make troubleshooting far more
difficult
> when searching google/etc looking for links to documents that are
applicable.
> 
> Maybe something like far - filesystem agnostic replication?   
I like that one. It has a nice connotation to ''remote''. So
''far'' it be.
Thanks!

-Arne

On 22.10.2012 06:32, Matthew Ahrens wrote:
> On Sat, Oct 20, 2012 at 1:24 PM, Tim Cook <tim at cook.ms <mailto:tim
at cook.ms>> wrote:
> 
> 
> 
>     On Sat, Oct 20, 2012 at 2:54 AM, Arne Jansen <sensille at gmx.net
>     <mailto:sensille at gmx.net>> wrote:
> 
>         On 10/20/2012 01:10 AM, Tim Cook wrote:
>         >
>         >
>         > On Fri, Oct 19, 2012 at 3:46 PM, Arne Jansen <sensille at
gmx.net
>         <mailto:sensille at gmx.net>
>         > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>> wrote:
>         >
>         >     On 10/19/2012 09:58 PM, Matthew Ahrens wrote:
>         >     > On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen
<sensille at gmx.net
>         <mailto:sensille at gmx.net>
>         >     <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>
>         >     > <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>
>         <mailto:sensille at gmx.net <mailto:sensille at
gmx.net>>>> wrote:
>         >     >
>         >     >     We have finished a beta version of the feature. A
webrev for it
>         >     >     can be found here:
>         >     >
>         >     >     http://cr.illumos.org/~webrev/sensille/fits-send/
>         >     >
>         >     >     It adds a command ''zfs
fits-send''. The resulting streams can
>         >     >     currently only be received on btrfs, but more
receivers will
>         >     >     follow.
>         >     >     It would be great if anyone interested could give
it some
>         testing
>         >     >     and/or review. If there are no objections,
I''ll send a formal
>         >     >     webrev soon.
>         >     >
>         >     >
>         >     >
>         >     > Please don''t bother changing libzfs (and
proliferating the copypasta
>         >     > there) -- do it like lzc_send().
>         >     >
>         >
>         >     ok. It would be easier though if zfs_send would also
already use the
>         >     new style. Is it in the pipeline already?
>         >
>         >     > Likewise, zfs_ioc_fits_send should use the new-style
API.  See the
>         >     > comment at the beginning of zfs_ioctl.c.
>         >     >
>         >     > I''m not a fan of the name "FITS"
but I suppose somebody else already
>         >     > named the format.  If we are going to follow someone
else''s format
>         >     > though, it at least needs to be well-documented. 
Where can we
>         >     find the
>         >     > documentation?
>         >     >
>         >     > FYI, #1 google hit for "FITS": 
http://en.wikipedia.org/wiki/FITS
>         >     > #3 hit:  http://code.google.com/p/fits/
>         >     >
>         >     > Both have to do with file formats.  The entire first
page of google
>         >     > results for "FITS format" and "FITS
file format" are related to
>         these
>         >     > two formats.  "FITS btrfs" didn''t
return anything specific to
>         the file
>         >     > format, either.
>         >
>         >     It''s not too late to change it, but I have a hard
time coming up with
>         >     some better name. Also, the format is still very new and
I''m sure
>         it''ll
>         >     need some adjustments.
>         >
>         >     -arne
>         >
>         >     >
>         >     > --matt
>         >
>         >
>         >
>         > I''m sure we can come up with something.  Are you
planning on this being
>         > solely for ZFS, or a larger architecture for replication both
directions
>         > in the future?
> 
>         We have senders for zfs and btrfs. The planned receiver will be
mostly
>         filesystem agnostic and can work on a much broader range. It
basically
>         only needs to know how to create snapshots and where to store a few
>         meta informations.
>         It would be great if more filesystems would join on the sending
side,
>         but I have no involvement there.
> 
>         I see no basic problem in choosing a name that''s already
in use.
>         Especially with file extensions most will be already taken. How
about
>         something with ''portable'' and
''backup'', like pib or pibs? ''i'' for
>         incremental.
> 
>         -Arne
> 
> 
>     Re-using names generally isn''t a big deal, but in this case
the existing
>     name is a technology that''s extremely similar to what
you''re doing - which
>     WILL cause a ton of confusion in the userbase, and make troubleshooting
far
>     more difficult when searching google/etc looking for links to documents
that
>     are applicable.  
> 
>     Maybe something like far - filesystem agnostic replication?   
> 
> 
> All else being equal, I like this name (FAR).  It ends in "AR"
like several
> other archive formats (TAR, WAR, JAR).  Plus not a lot of false positives
when
> googling around for it.   
> 
> However, if compatibility with the existing format is an explicit goal, we
> should use the same name, and the btrfs authors may be averse to changing
the name.
There''s really nothing to keep. In the btrfs world, like in the zfs
world, the
stream has no special name, it''s just a ''btrfs send
stream'', like the ''zfs send
stream''. The necessity for a name only arises from the wish to build a
bridge
between the worlds.
The author of btrfs is also fine with ''far''.

-Arne
> 
> --matt

Alexander Block <ablock84 at googlemail.com> wrote:
> tar/pax was the initial format that was chosen for btrfs send/receive
> as it looked like the best and most compatible way. In the middle of
> development however I realized that we need more then storing whole
> and incremental files/dirs in the format. We needed to store
> information about moved, renamed, deleted, reflinked and even partial
> clones where only some bits of a file are shared with another. This
> can for sure all be implemented in pax, but then the next problem is
> that in some situations renamed/moved files need multiple entries to
> get to the desired result. For example, file a may be renamed to b
> while at the same time file b got renamed to a. In such cases we need
> 3 entries that use a temporary name so that we don''t loose one of
the
> files while receiving. There are much more complex examples where it
> gets quite complicated.
The problems of complex renames has been solved in star with the incremental 
backup/restore concept 8 years ago already. Renames are done based on inode 
numbers.
> Also, it needed support for metadata (mode, size, uid/gid, ...)
> changes on already existing files/dirs. Reusing already existent
> tar/pax entry types was not possible for this as standard tar would
> overwrite the original files with empty files.
This is not true. Star implements this since more than 8 years.

> I had all that implemented with pax, using a lot of custom pax
> entries. A lot...so many that it didn''t look like tar/pax anymore.
It
> actually mutated from a list of file/dir/link entries (which tar/pax
> is meant to be) to a list of filesystem instructions (rename, link,
> unlink, rmdir, write parts of a file, clone parts of a file, chmod,
> ...).
If you end up in something that does not look like an enhanced tar archivem you 
did probably not follow the rules.
> My thought was, that this was already a big misuse of tar/pax, so I
> decided to implement a simple format for this purpose only. Using pax
> gave no advantages anymore. In tar/pax every entry must have a file
> name, even the pax header entries need a file name. The problem now
> is, that plain tar will treat every unknown entry type as regular file
> and blindly overwrite existing ones which may result in data loss. To
> prevent this, I always added something to the file name so that
> unpacking with tar would not hurt the user. The unavoidable side
> effect however is that the result of a plain untar is unusable without
> further interpretation, which will be hard because tar by default does
> not dump pax headers but instead ignores unknown entries.
The problem with possible overrides with too dump archivers has been solved 
in star long ago. Star let''s old software believe that there is an EOF
when a
meta data only entry is found.
> Also, using tar/pax as the format for send/receive may give a user the
> wrong impression that he can later use his good old standard tar to
> restore his backups...this could be fatal for him.
This works fine for the incremental backup/restore system used by star.

Enhancing existing data structures without breaking the philosophy is not 
trivial and may take some time, but it is usually better than reinventing the 
wheel.

Star now exists since more than 30 years (it exists since August 1982). Star is 
where other tar implementaions take their ideas from ;-)

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Arne, I took a look at far.c in
http://cr.illumos.org/~webrev/sensille/far-send/.  Here are some
high-level comments:

Why did you choose to do this all in the kernel?  As opposed to the
way "zfs diff" works, where the kernel generates the list of changed
items and then userland sorts out what exactly changed in each block
and how that relates to the "from" snapshot.  Do you need information
that isn''t available through the regular POSIX ioctls?

Why do you need far_find_from_bp(), rather than reading the from
snapshot through the "front door", e.g. dmu_buf_hold()?

The big comment at the beginning is very helpful.  However there are
several typos / missing words.  Also could you explain what you mean
by "put back"?

A comment says that far_sa_setup was copied from zfs_znode.c.  If this
is the case, you need to also copy the copyright notices in
zfs_znode.c (or just the ones that apply, if you want to bother
digging through the history).

far_find_from_bp():  why do you need to clear out the lower level
blklevel_t''s every time through the loop?  Would it suffice to clear
them all out at the beginning?

In Illumos, we declare structs as "typedef struct x { ... } x_t;".
(note there is no underscore before the type name).

The ASSERT3U() family of macros are pretty useful.  Consider using
them instead of plain ASSERT() where possible.

The blklevel_t structure is pretty complicated; is there a comment
somewhere saying what it represents?  E.g. that bl_bp is an array of
length bl_nslots?  Additionally, you could give the variables
descriptive names ("bl_bps" and "bl_nbps" would be maximally
terse but
still give some clue as to what they are).  Also it seems like bp_bp
== bl_buf->b_data?

--matt


On Wed, Oct 17, 2012 at 5:29 AM, Arne Jansen <sensille at gmx.net>
wrote:
> We have finished a beta version of the feature. A webrev for it
> can be found here:
>
> http://cr.illumos.org/~webrev/sensille/fits-send/
>
> It adds a command ''zfs fits-send''. The resulting streams
can
> currently only be received on btrfs, but more receivers will
> follow.
> It would be great if anyone interested could give it some testing
> and/or review. If there are no objections, I''ll send a formal
> webrev soon.
>
> Thanks,
> Arne
>
>
> On 10.10.2012 21:38, Arne Jansen wrote:
>> Hi,
>>
>> We''re currently working on a feature to send zfs streams in a
portable
>> format that can be received on any filesystem.
>> It is a kernel patch that generates the stream directly from kernel,
>> analogous to what zfs send does.
>> The stream format is based on the btrfs send format. The basic idea
>> is to just send commands like mkdir, unlink, create, write, etc.
>> For incremental sends it''s the same idea.
>> The receiving side is user mode only, so it''s very easy to
port it to
>> any system. If the receiving side has the capability to create
>> snapshots, those from the sending side will get recreated. If not,
>> you still have the benefit of fast incremental transfers.
>>
>> My question is if there''s any interest in this feature and if
it had
>> chances of getting accepted. Also, would it be acceptable to start with
>> a working version and add performance optimizations later on?
>>
>> -Arne
>
>
> -------------------------------------------
> illumos-zfs
> Archives: https://www.listbox.com/member/archive/182191/=now
> RSS Feed:
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
> Modify Your Subscription:
https://www.listbox.com/member/?member_id=21635000&id_secret=21635000-73dc201a
> Powered by Listbox: http://www.listbox.com