Hello all, I am using 2008.11 with the Timeslider automatic snapshots, and using it to automatically send snapshots to a remote host every 15 minutes. Both sides are X4540''s, with the remote filesystem mounted read-only as I read earlier that would cause problems. The snapshots send fine for several days, I accumulate many snapshots at regular intervals, and they are sent without any problems. Then I will get the dreaded: " cannot receive incremental stream: most recent snapshot of pdxfilu02 does not match incremental source " Manually sending does not work, or destroying snapshots on the remote side and resending the batch again from the earliest point in time. The only way I have found that works, is to destroy the entire zfs filesystem on the remote side, and begin anew. Is there a way to force a ZFS receive, or to get more information about what changed on the remote system to cause it not to accept anymore snapshots? Thank you in advance -- Brent Jones brent at servuhome.net
Hi Brent, Brent Jones wrote:> I am using 2008.11 with the Timeslider automatic snapshots, and using > it to automatically send snapshots to a remote host every 15 minutes. > Both sides are X4540''s, with the remote filesystem mounted read-only > as I read earlier that would cause problems. > The snapshots send fine for several days, I accumulate many snapshots > at regular intervals, and they are sent without any problems. > Then I will get the dreaded: > " > cannot receive incremental stream: most recent snapshot of pdxfilu02 > does not match incremental source > " >Which command line are you using? Maybe you need to do a rollback first (zfs receive -F)? Cheers Carsten
On Sun, Jan 4, 2009 at 11:33 PM, Carsten Aulbert <carsten.aulbert at aei.mpg.de> wrote:> Hi Brent, > > Brent Jones wrote: >> I am using 2008.11 with the Timeslider automatic snapshots, and using >> it to automatically send snapshots to a remote host every 15 minutes. >> Both sides are X4540''s, with the remote filesystem mounted read-only >> as I read earlier that would cause problems. >> The snapshots send fine for several days, I accumulate many snapshots >> at regular intervals, and they are sent without any problems. >> Then I will get the dreaded: >> " >> cannot receive incremental stream: most recent snapshot of pdxfilu02 >> does not match incremental source >> " >> > > Which command line are you using? > > Maybe you need to do a rollback first (zfs receive -F)? > > Cheers > > Carsten > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >I am using a command similar to this: zfs send -i pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:30 pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:45 | ssh -c blowfish user at host.com /sbin/zfs recv -d pdxfilu02 It normally works, then after some time it will stop. It is still doing a full snapshot replication at this time (very slowly it seems, I''m bit by the bug of slow zfs send/resv) Once I get back on my regular snapshotting, if it comes out of sync again, I''ll try doing a -F rollback and see if that helps. -- Brent Jones brent at servuhome.net
Correlation question below... Brent Jones wrote:> On Sun, Jan 4, 2009 at 11:33 PM, Carsten Aulbert > <carsten.aulbert at aei.mpg.de> wrote: > >> Hi Brent, >> >> Brent Jones wrote: >> >>> I am using 2008.11 with the Timeslider automatic snapshots, and using >>> it to automatically send snapshots to a remote host every 15 minutes. >>> Both sides are X4540''s, with the remote filesystem mounted read-only >>> as I read earlier that would cause problems. >>> The snapshots send fine for several days, I accumulate many snapshots >>> at regular intervals, and they are sent without any problems. >>> Then I will get the dreaded: >>> " >>> cannot receive incremental stream: most recent snapshot of pdxfilu02 >>> does not match incremental source >>> " >>> >>> >> Which command line are you using? >> >> Maybe you need to do a rollback first (zfs receive -F)? >> >> Cheers >> >> Carsten >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> > > I am using a command similar to this: > > zfs send -i pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:30 > pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:45 | ssh -c > blowfish user at host.com /sbin/zfs recv -d pdxfilu02 > > It normally works, then after some time it will stop. It is still > doing a full snapshot replication at this time (very slowly it seems, > I''m bit by the bug of slow zfs send/resv) > > Once I get back on my regular snapshotting, if it comes out of sync > again, I''ll try doing a -F rollback and see if that helps. >When this gets slow, are the other snapshot-related commands also slow? For example, normally I see "zfs list -t snapshot" completing in a few seconds, but sometimes it takes minutes? -- richard
On Mon, Jan 5, 2009 at 2:50 PM, Richard Elling <Richard.Elling at sun.com> wrote:> Correlation question below... > > Brent Jones wrote: >> >> On Sun, Jan 4, 2009 at 11:33 PM, Carsten Aulbert >> <carsten.aulbert at aei.mpg.de> wrote: >> >>> >>> Hi Brent, >>> >>> Brent Jones wrote: >>> >>>> >>>> I am using 2008.11 with the Timeslider automatic snapshots, and using >>>> it to automatically send snapshots to a remote host every 15 minutes. >>>> Both sides are X4540''s, with the remote filesystem mounted read-only >>>> as I read earlier that would cause problems. >>>> The snapshots send fine for several days, I accumulate many snapshots >>>> at regular intervals, and they are sent without any problems. >>>> Then I will get the dreaded: >>>> " >>>> cannot receive incremental stream: most recent snapshot of pdxfilu02 >>>> does not match incremental source >>>> " >>>> >>>> >>> >>> Which command line are you using? >>> >>> Maybe you need to do a rollback first (zfs receive -F)? >>> >>> Cheers >>> >>> Carsten >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >>> >> >> I am using a command similar to this: >> >> zfs send -i pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:30 >> pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:45 | ssh -c >> blowfish user at host.com /sbin/zfs recv -d pdxfilu02 >> >> It normally works, then after some time it will stop. It is still >> doing a full snapshot replication at this time (very slowly it seems, >> I''m bit by the bug of slow zfs send/resv) >> >> Once I get back on my regular snapshotting, if it comes out of sync >> again, I''ll try doing a -F rollback and see if that helps. >> > > When this gets slow, are the other snapshot-related commands also > slow? For example, normally I see "zfs list -t snapshot" completing > in a few seconds, but sometimes it takes minutes? > -- richard > >I''m not seeing zfs related commands any slower. On the remote side, it builds up thousands of snapshots, and aside from SSH scrolling as fast as it can over the network, no other slowness. But the actual send and receive is getting very very slow, almost to the point of needing the scrap the project and find some other way to ship data around! -- Brent Jones brent at servuhome.net
On Mon, Jan 5, 2009 at 4:29 PM, Brent Jones <brent at servuhome.net> wrote:> On Mon, Jan 5, 2009 at 2:50 PM, Richard Elling <Richard.Elling at sun.com> wrote: >> Correlation question below... >> >> Brent Jones wrote: >>> >>> On Sun, Jan 4, 2009 at 11:33 PM, Carsten Aulbert >>> <carsten.aulbert at aei.mpg.de> wrote: >>> >>>> >>>> Hi Brent, >>>> >>>> Brent Jones wrote: >>>> >>>>> >>>>> I am using 2008.11 with the Timeslider automatic snapshots, and using >>>>> it to automatically send snapshots to a remote host every 15 minutes. >>>>> Both sides are X4540''s, with the remote filesystem mounted read-only >>>>> as I read earlier that would cause problems. >>>>> The snapshots send fine for several days, I accumulate many snapshots >>>>> at regular intervals, and they are sent without any problems. >>>>> Then I will get the dreaded: >>>>> " >>>>> cannot receive incremental stream: most recent snapshot of pdxfilu02 >>>>> does not match incremental source >>>>> " >>>>> >>>>> >>>> >>>> Which command line are you using? >>>> >>>> Maybe you need to do a rollback first (zfs receive -F)? >>>> >>>> Cheers >>>> >>>> Carsten >>>> _______________________________________________ >>>> zfs-discuss mailing list >>>> zfs-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>> >>>> >>> >>> I am using a command similar to this: >>> >>> zfs send -i pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:30 >>> pdxfilu01/Archive at zfs-auto-snap:frequent-2009-01-04-03:45 | ssh -c >>> blowfish user at host.com /sbin/zfs recv -d pdxfilu02 >>> >>> It normally works, then after some time it will stop. It is still >>> doing a full snapshot replication at this time (very slowly it seems, >>> I''m bit by the bug of slow zfs send/resv) >>> >>> Once I get back on my regular snapshotting, if it comes out of sync >>> again, I''ll try doing a -F rollback and see if that helps. >>> >> >> When this gets slow, are the other snapshot-related commands also >> slow? For example, normally I see "zfs list -t snapshot" completing >> in a few seconds, but sometimes it takes minutes? >> -- richard >> >> > > I''m not seeing zfs related commands any slower. On the remote side, it > builds up thousands of snapshots, and aside from SSH scrolling as fast > as it can over the network, no other slowness. > But the actual send and receive is getting very very slow, almost to > the point of needing the scrap the project and find some other way to > ship data around! > > -- > Brent Jones > brent at servuhome.net >Got a small update on the ZFS send, I am in fact seeing ZFS list taking several minutes to complete. I must have timed it correctly during the send, and both sides are not completing the ZFS list, and its been about 5 minutes already. There is a small amount of network traffic between the two hosts, so maybe it''s comparing what needs to be sent, not sure. I''ll update when/if it completes. -- Brent Jones brent at servuhome.net