zfs discuss - Sep 2011 - commercial zfs-based storage replication software?

Hi,

Does anyone know a good commercial zfs-based storage replication
software that runs on Solaris (i.e. not an appliance, not another OS
based on solaris)?
Kinda like Amanda, but for replication (not backup).

Thanks,

Fajar

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
> 
> Does anyone know a good commercial zfs-based storage replication
> software that runs on Solaris (i.e. not an appliance, not another OS
> based on solaris)?
> Kinda like Amanda, but for replication (not backup).
Please define replication, not backup?  To me, your question is unclear what
you want to accomplish.  What don''t you like about zfs send | zfs
receive?

On Fri, Sep 30, 2011 at 7:22 PM, Edward Ned Harvey
<opensolarisisdeadlongliveopensolaris at nedharvey.com>
wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
>>
>> Does anyone know a good commercial zfs-based storage replication
>> software that runs on Solaris (i.e. not an appliance, not another OS
>> based on solaris)?
>> Kinda like Amanda, but for replication (not backup).
>
> Please define replication, not backup? ?To me, your question is unclear
what
> you want to accomplish. ?What don''t you like about zfs send | zfs
receive?
Basically I need something that does zfs send | zfs receive, plus
GUI/web interface to configure stuff (e.g. which fs to backup,
schedule, etc.), support, and a price tag.

Believe it or not the last two requirement are actually important
(don''t ask :P ), and are the main reasons why I can''t use
automated
send - receive scripts already available from the internet.

CMIIW, Amanda can use "zfs send" but it only store the resulting
stream somewhere, while the requirement for this one is that the send
stream must be received on a different server (e.g. DR site) and be
accessible there.

-- 
Fajar

Maybe I''m missing something here, but Amanda has a whole bunch of bells
and whistles, and scans the filesystem to determine what should be backed up. 
Way overkill for this task I think.

Seems to me like zfs send blah | ssh replicatehost zfs receive ?  more than
meets the requirement when combined with just plain old crontab.

If it''s a graphical interface you''re looking for, I''m
sure someone has hacked together somethings in TCL/Tk pr Perl/TK as an interface
to cron which you could probably hack to have construct your particular crontab
entry.

Just a thought,

Mike

---
Michael Sullivan                   
mps at axsh.us
http://www.axsh.us/
Phone: +1-662-259-8888
Mobile: +1-662-202-7716

On 30 Sep 11, at 07:33 , Fajar A. Nugraha wrote:
> On Fri, Sep 30, 2011 at 7:22 PM, Edward Ned Harvey
> <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
>>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>>> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
>>> 
>>> Does anyone know a good commercial zfs-based storage replication
>>> software that runs on Solaris (i.e. not an appliance, not another
OS
>>> based on solaris)?
>>> Kinda like Amanda, but for replication (not backup).
>> 
>> Please define replication, not backup?  To me, your question is unclear
what
>> you want to accomplish.  What don''t you like about zfs send |
zfs receive?
> 
> Basically I need something that does zfs send | zfs receive, plus
> GUI/web interface to configure stuff (e.g. which fs to backup,
> schedule, etc.), support, and a price tag.
> 
> Believe it or not the last two requirement are actually important
> (don''t ask :P ), and are the main reasons why I can''t use
automated
> send - receive scripts already available from the internet.
> 
> CMIIW, Amanda can use "zfs send" but it only store the resulting
> stream somewhere, while the requirement for this one is that the send
> stream must be received on a different server (e.g. DR site) and be
> accessible there.
> 
> -- 
> Fajar
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Mike

---
Michael Sullivan                   
mps at axsh.us
http://www.axsh.us/
Phone: +1-662-259-8888
Mobile: +1-662-202-7716

On Fri, Sep 30, 2011 at 8:20 PM, Michael Sullivan
<michael.p.sullivan at mac.com> wrote:
> Maybe I''m missing something here, but Amanda has a whole bunch of
bells and whistles, and scans the filesystem to determine what should be backed
up. ?Way overkill for this task I think.
I was using Amanda as an example because it has GUI, schedule, and
supports zfs send. Probably not the best example :)
>
> Seems to me like zfs send blah | ssh replicatehost zfs receive ? ?more than
meets the requirement when combined with just plain old crontab.
Functionality wise, it meets some requirement. However GUI is a
requirement here.
>
> If it''s a graphical interface you''re looking for,
I''m sure someone has hacked together somethings in TCL/Tk pr Perl/TK
preferably web-based
> as an interface to cron
not just cron. It needs some form of configuration (e.g. input which
datasets to sync, to which remote host, and how often), reporting
(e.g. there are x jobs yesterday, all completed in 60 minutes, 1GB
data transferred).
> which you could probably hack to have construct your particular crontab
entry.
... and if they have a company that can provide support and put a
reasonable price tag on it, that''d be perfect :)
I haven''t found any so far though, which is why I asked here.

-- 
Fajar

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
> 
> Basically I need something that does zfs send | zfs receive, plus
> GUI/web interface to configure stuff (e.g. which fs to backup,
> schedule, etc.), support, and a price tag.
> 
> Believe it or not the last two requirement are actually important
> (don''t ask :P ), and are the main reasons why I can''t use
automated
> send - receive scripts already available from the internet.
> 
> CMIIW, Amanda can use "zfs send" but it only store the resulting
> stream somewhere, while the requirement for this one is that the send
> stream must be received on a different server (e.g. DR site) and be
> accessible there.
Have you looked at Sun Unified Storage, AKA the 7000 series?
http://www.oracle.com/us/products/servers-storage/storage/unified-storage/in
dex.html

I''ve never used them myself, personally, because I prefer to have the
control of building my system from the ground up and knowing all the stuff
under the hood.  But based on marketing materials that I''ve seen, I
think
it''s exactly what you''re describing.

On Sat, Oct 1, 2011 at 5:06 AM, Edward Ned Harvey
<opensolarisisdeadlongliveopensolaris at nedharvey.com>
wrote:
> Have you looked at Sun Unified Storage, AKA the 7000 series?
>
http://www.oracle.com/us/products/servers-storage/storage/unified-storage/in
> dex.html
>
> I''ve never used them myself, personally, because I prefer to have
the
> control of building my system from the ground up and knowing all the stuff
> under the hood. ?But based on marketing materials that I''ve seen,
I think
> it''s exactly what you''re describing.
Thanks, that would be my fallback plan (along with nexentastor and netapp).

-- 
Fajar

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
> 
> On Sat, Oct 1, 2011 at 5:06 AM, Edward Ned Harvey
> <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
> > Have you looked at Sun Unified Storage, AKA the 7000 series?
> 
> Thanks, that would be my fallback plan (along with nexentastor and
netapp).

So you''re basically looking for installable 3rd party software that
replicates that functionality?  I don''t know of any, but
that''s not saying
much, because when it comes to ZFS, I''m not very platform explorative.

On Sat, Oct 1, 2011 at 8:01 PM, Edward Ned Harvey
<opensolarisisdeadlongliveopensolaris at nedharvey.com>
wrote:
>> On Sat, Oct 1, 2011 at 5:06 AM, Edward Ned Harvey
>> <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
>> > Have you looked at Sun Unified Storage, AKA the 7000 series?
>>
>> Thanks, that would be my fallback plan (along with nexentastor and
> netapp).
>
> So you''re basically looking for installable 3rd party software
that
> replicates that functionality?
Thank you all for the responses so far.

@Edward: storage appliance is not really what I''m looking for since
they provide SAN/NAS-based replication instead of host-based, but it''s
close enough in the sense that I can use it if I can''t find exact
solution match. Also I don''t need all of their features (e.g. I
don''t
need the ability to export dataset as FC/iscsi/nfs).

@Brad: From their product page it looks like MenloWare has nice GUI,
but it''s not capable of managing replication, only provisioning and
snapshot.

-- 
Fajar

On 1 Oct 11, at 08:01 , Edward Ned Harvey wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Fajar A. Nugraha
>> 
>> On Sat, Oct 1, 2011 at 5:06 AM, Edward Ned Harvey
>> <opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
>>> Have you looked at Sun Unified Storage, AKA the 7000 series?
>> 
>> Thanks, that would be my fallback plan (along with nexentastor and
> netapp).
> 
> So you''re basically looking for installable 3rd party software
that
> replicates that functionality?  I don''t know of any, but
that''s not saying
> much, because when it comes to ZFS, I''m not very platform
explorative.
A I said before, hack an open source job scheduler or find one which allows
creating jobs with parameters or panels for custom fields to put together the
crontab command and wrap it in something which preserves the output of cron
rather than email, but stores it in a database or something as well as keeps
track of success or failure and notifies someone in the event of failure and/or
restarts.  Which also probably means it needs to be do distributed process
management to kickoff everything it needs to.  It should probably be ZFS aware
so it can present filesystems and select based on filesystem rather than job.

Oracle Enterprise Manager does this.  It''s commercial, and I''m
sure they would negotiate on price for you and give you a good deal if you are
good at bargaining with your Oracle Sales Rep.

I think his requirements are being driven by a PHB who wants to see a
"GUI".

crontab, ssh - functionality already there, simple and not many "moving
parts" but obviously too obfuscated for the PHB to understand.

Good luck.

Mike

---
Michael Sullivan                   
mps at axsh.us
http://www.axsh.us/
Phone: +1-662-259-8888
Mobile: +1-662-202-7716

Have you looked at the time-slider functionality that is already in 
Solaris ?

There is a GUI for configuration of the snapshots and time-slider can be 
configured to do a ''zfs send'' or ''rsync''. 
The GUI doesn''t have the
ability to set the ''zfs recv'' command but that is set one-time
in the
SMF service properties.

--
Darren J Moffat

On Tue, Oct 11, 2011 at 5:26 PM, Darren J Moffat
<darrenm at opensolaris.org> wrote:
> Have you looked at the time-slider functionality that is already in Solaris
> ?
Hi Darren. Is it available for Solaris 10? I just installed Solaris 10
u10 and couldn''t find it.
>
> There is a GUI for configuration of the snapshots
the screenshots that I can find all refer to opensolaris
> and time-slider can be
> configured to do a ''zfs send'' or
''rsync''. ?The GUI doesn''t have the ability
> to set the ''zfs recv'' command but that is set one-time in
the SMF service
> properties.
Is there a reference on how to get/install this functionality on Solaris 10?

Thanks,

Fajar

On 10/13/11 09:27, Fajar A. Nugraha wrote:
> On Tue, Oct 11, 2011 at 5:26 PM, Darren J Moffat
> <darrenm at opensolaris.org>  wrote:
>> Have you looked at the time-slider functionality that is already in
Solaris
>> ?
>
> Hi Darren. Is it available for Solaris 10? I just installed Solaris 10
> u10 and couldn''t find it.
No it is not.
>> There is a GUI for configuration of the snapshots
>
> the screenshots that I can find all refer to opensolaris
>
>> and time-slider can be
>> configured to do a ''zfs send'' or
''rsync''.  The GUI doesn''t have the ability
>> to set the ''zfs recv'' command but that is set
one-time in the SMF service
>> properties.
>
> Is there a reference on how to get/install this functionality on Solaris
10?
No because it doesn''t exist on Solaris 10.

-- 
Darren J Moffat

2011-10-13 13:27, Darren J Moffat ?????:
> On 10/13/11 09:27, Fajar A. Nugraha wrote:
>> On Tue, Oct 11, 2011 at 5:26 PM, Darren J Moffat
>> <darrenm at opensolaris.org> wrote:
>>> Have you looked at the time-slider functionality that is already in
>>> Solaris
>>> ?
>>
>> Hi Darren. Is it available for Solaris 10? I just installed Solaris 10
>> u10 and couldn''t find it.
>
> No it is not.
>> Is there a reference on how to get/install this functionality on 
>> Solaris 10?
>
> No because it doesn''t exist on Solaris 10.
>
Well, just for the sake of completeness: most of our systems are using
zfs-auto-snap service, including Solaris 10 systems datiing from Sol10u6.
Installation of relevant packages from SXCE (ranging snv_117-snv_130)
was trivial, but some script-patching was in order. I think, replacement
of the ksh interpreter to ksh93.

I haven''t used the GUI part and I guess my experience relates to the
script-based zfs-auto-snap (before it was remade into current binary
form, or so I read). We kind of got stuck with SXCE systems which
still "just work" finely ;)

The point is, even if unsupported (may be a problem in OP''s case)
it is likely that one or another version of zfs-auto-snap or TimeSlider
can be made to work in Sol10 with little effort.

HTH,
//Jim

On Wed, Oct 19, 2011 at 7:52 PM, Jim Klimov <jimklimov at cos.ru>
wrote:
> 2011-10-13 13:27, Darren J Moffat ?????:
>>
>> On 10/13/11 09:27, Fajar A. Nugraha wrote:
>>>
>>> On Tue, Oct 11, 2011 at 5:26 PM, Darren J Moffat
>>> <darrenm at opensolaris.org> wrote:
>>>>
>>>> Have you looked at the time-slider functionality that is
already in
>>>> Solaris
>>>> ?
>>>
>>> Hi Darren. Is it available for Solaris 10? I just installed Solaris
10
>>> u10 and couldn''t find it.
>>
>> No it is not.
>>>
>>> Is there a reference on how to get/install this functionality on
Solaris
>>> 10?
>>
>> No because it doesn''t exist on Solaris 10.
>>
>
> Well, just for the sake of completeness: most of our systems are using
> zfs-auto-snap service, including Solaris 10 systems datiing from Sol10u6.
> Installation of relevant packages from SXCE (ranging snv_117-snv_130)
> was trivial, but some script-patching was in order. I think, replacement
> of the ksh interpreter to ksh93.
Yes, I remembered reading about that.
>
> I haven''t used the GUI part and I guess my experience relates to
the
> script-based zfs-auto-snap (before it was remade into current binary
> form, or so I read). We kind of got stuck with SXCE systems which
> still "just work" finely ;)
>
> The point is, even if unsupported (may be a problem in OP''s case)
> it is likely that one or another version of zfs-auto-snap or TimeSlider
> can be made to work in Sol10 with little effort.
To be honest, if it''s just to get it work, I''d just make my
own. Or
running SE with a solaris 10 zone inside it, with SE managing
time-slider/replication and S10 zone running the application.

But for this particular case support is essential. That''s why I
mentioned earlier if I can''t get a supported solution for this setup
(with a reasonable price), storage-based replication would have to do.

-- 
Fajar

2011-10-19 17:54, Fajar A. Nugraha ?????:
> On Wed, Oct 19, 2011 at 7:52 PM, Jim Klimov<jimklimov at cos.ru> 
wrote:
>> Well, just for the sake of completeness: most of our systems are 
>> using zfs-auto-snap service, including Solaris 10 systems datiing 
>> from Sol10u6. Installation of relevant packages from SXCE (ranging 
>> snv_117-snv_130) was trivial, but some script-patching was in order. 
>> I think, replacement of the ksh interpreter to ksh93. 
> Yes, I remembered reading about that.
>
Actually, I revised the systems: the scripts are kept in original form, 
but those sol10 servers where ksh93 was absent, got a symlink:

/usr/bin/ksh93 -> ../dt/bin/dtksh

2011/10/20 Jim Klimov <jimklimov at cos.ru>:
> 2011-10-19 17:54, Fajar A. Nugraha ?????:
>>
>> On Wed, Oct 19, 2011 at 7:52 PM, Jim Klimov<jimklimov at cos.ru>
?wrote:
>>>
>>> Well, just for the sake of completeness: most of our systems are
using
>>> zfs-auto-snap service, including Solaris 10 systems datiing from
Sol10u6.
>>> Installation of relevant packages from SXCE (ranging
snv_117-snv_130) was
>>> trivial, but some script-patching was in order. I think,
replacement of the
>>> ksh interpreter to ksh93.
>>
>> Yes, I remembered reading about that.
>>
>
> Actually, I revised the systems: the scripts are kept in original form, but
> those sol10 servers where ksh93 was absent, got a symlink:
>
> /usr/bin/ksh93 -> ../dt/bin/dtksh
For the sake of completeness, I tried using zfs-auto-snapshot 0.12
(the last version before obsoleted by  time-slider). I used opencsw''s
ksh 93 (http://www.opencsw.org/package/ksh/) and symlink
/opt/csw/bin/ksh to /usr/bin/ksh93.

It kinda works when only used for automated snapshot. But when using
zfs/backup-save-cmd property ( to make remote system do a "zfs receive
-d -F", problem started to happen. It complains the snapshot already
exist. Probably the result of interaction between
com.sun:auto-snapshot user property, child dataset, and "zfs send -R".
If only "zfs receive" has the option to "force delete the
snapshot if
it already exist, rollback to the previous one, and retry the
incremental receive" :)

Since zfs-auto-snapshot has been obsoleted (and obviously
unsupported), it wouldn''t make sense for me to try to fix this. I
might revisit this option again later using time-slider when solaris
11 is released though.

Thanks for the info so far.

-- 
Fajar