zfs discuss - Dec 2009 - ZFS incremental receives fail

We''ve been using ZFS for about two years now and make a lot of use of
zfs
send/receive to send our data from one X4500 to another. This has been
working well for the past 18 months that we''ve been doing the sends.

I recently upgraded the receiving thumper to Solaris 10 u8 and since then,
I''ve been having no end of problems. We were asked to upgrade to
attempt to
resolve a driver issue we''ve been having with the X4500s and I also
upgraded the zfs version running on the system.

The last snapshot received was named thumperpool/mnt at 200911301000 and since
then we''ve been completely unable to receive any snapshots -- even if
I''ve
literally just snapshotted, removed back to the previous snapshot, and then
tried to receive that same snapshot.

Here''s a demo of the behaviour we''re seeing:

117 root at thumper2:~> cat 1000_1100 | zfs receive -v thumperpool
receiving incremental stream of vlepool/mnt at 200911301100 into thumperpool at
200911301100
cannot receive incremental stream: destination thumperpool has been modified
since most recent snapshot

If I modify any data on the system, snapshot it, rollback the system, and
attempt to receive the snapshot back, I get the same:

122 root at thumper2:/thumperpool/mnt> zfs list | tail
thumperpool/mnt at 200911291700             131M      -  1.77T  -
thumperpool/mnt at 200911292200             114M      -  1.77T  -
thumperpool/mnt at 200911300000            85.2M      -  1.77T  -
thumperpool/mnt at 200911300100            87.2M      -  1.77T  -
thumperpool/mnt at 200911300300             102M      -  1.77T  -
thumperpool/mnt at 200911300400            81.0M      -  1.77T  -
thumperpool/mnt at 200911300500            83.3M      -  1.77T  -
thumperpool/mnt at 200911300700            86.7M      -  1.77T  -
thumperpool/mnt at 200911300900             102M      -  1.77T  -
thumperpool/mnt at 200911301000                0      -  1.77T  -
123 root at thumper2:/thumperpool/mnt> dd if=/dev/zero of=mymodification
bs=512 count=100
100+0 records in
100+0 records out
124 root at thumper2:/thumperpool/mnt> zfs list | tail
thumperpool/mnt at 200911291700             131M      -  1.77T  -
thumperpool/mnt at 200911292200             114M      -  1.77T  -
thumperpool/mnt at 200911300000            85.2M      -  1.77T  -
thumperpool/mnt at 200911300100            87.2M      -  1.77T  -
thumperpool/mnt at 200911300300             102M      -  1.77T  -
thumperpool/mnt at 200911300400            81.0M      -  1.77T  -
thumperpool/mnt at 200911300500            83.3M      -  1.77T  -
thumperpool/mnt at 200911300700            86.7M      -  1.77T  -
thumperpool/mnt at 200911300900             102M      -  1.77T  -
thumperpool/mnt at 200911301000            73.1K      -  1.77T  -
125 root at thumper2:/thumperpool/mnt> zfs snapshot thumperpool/mnt at
fail_demo
126 root at thumper2:/thumperpool/mnt> zfs list | tail
thumperpool/mnt at 200911292200             114M      -  1.77T  -
thumperpool/mnt at 200911300000            85.2M      -  1.77T  -
thumperpool/mnt at 200911300100            87.2M      -  1.77T  -
thumperpool/mnt at 200911300300             102M      -  1.77T  -
thumperpool/mnt at 200911300400            81.0M      -  1.77T  -
thumperpool/mnt at 200911300500            83.3M      -  1.77T  -
thumperpool/mnt at 200911300700            86.7M      -  1.77T  -
thumperpool/mnt at 200911300900             102M      -  1.77T  -
thumperpool/mnt at 200911301000            73.1K      -  1.77T  -
thumperpool/mnt at fail_demo                   0      -  1.77T  -
127 root at thumper2:/thumperpool/mnt> zfs send -i thumperpool/mnt at
200911301000 thumperpool/mnt at fail_demo > /tmp/fail_demo
128 root at thumper2:/thumperpool/mnt> zfs rollback thumperpool/mnt at
200911301000
cannot rollback to ''thumperpool/mnt at 200911301000'': more
recent snapshots exist
use ''-r'' to force deletion of the following snapshots:
thumperpool/mnt at fail_demo
129 root at thumper2:/thumperpool/mnt> zfs rollback -r thumperpool/mnt at
200911301000
130 root at thumper2:/thumperpool/mnt> zfs list | tail
thumperpool/mnt at 200911291700             131M      -  1.77T  -
thumperpool/mnt at 200911292200             114M      -  1.77T  -
thumperpool/mnt at 200911300000            85.2M      -  1.77T  -
thumperpool/mnt at 200911300100            87.2M      -  1.77T  -
thumperpool/mnt at 200911300300             102M      -  1.77T  -
thumperpool/mnt at 200911300400            81.0M      -  1.77T  -
thumperpool/mnt at 200911300500            83.3M      -  1.77T  -
thumperpool/mnt at 200911300700            86.7M      -  1.77T  -
thumperpool/mnt at 200911300900             102M      -  1.77T  -
thumperpool/mnt at 200911301000                0      -  1.77T  -
131 root at thumper2:/thumperpool/mnt> cat /tmp/fail_demo | zfs receive
thumperpool -v
too many arguments
usage:
        receive [-vnF] <filesystem|volume|snapshot>
        receive [-vnF] -d <filesystem>

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
132 root at thumper2:/thumperpool/mnt> cat /tmp/fail_demo | zfs receive -v
thumperpool
receiving incremental stream of thumperpool/mnt at fail_demo into thumperpool at
fail_demo
cannot receive incremental stream: destination thumperpool has been modified
since most recent snapshot
133 root at thumper2:/thumperpool/mnt> cat /tmp/fail_demo | zfs receive -vF
thumperpool
receiving incremental stream of thumperpool/mnt at fail_demo into thumperpool at
fail_demo
cannot receive incremental stream: most recent snapshot of thumperpool does not
match incremental source




-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 04311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091210/0e95b661/attachment.bin>

On Thu, Dec 10, 2009 at 09:50:43AM +0000, Andrew Robert Nicols
wrote:
> We''ve been using ZFS for about two years now and make a lot of use
of zfs
> send/receive to send our data from one X4500 to another. This has been
> working well for the past 18 months that we''ve been doing the
sends.
> 
> I recently upgraded the receiving thumper to Solaris 10 u8 and since then,
> I''ve been having no end of problems. We were asked to upgrade to
attempt to
> resolve a driver issue we''ve been having with the X4500s and I
also
> upgraded the zfs version running on the system.
> 
> The last snapshot received was named thumperpool/mnt at 200911301000 and
since
> then we''ve been completely unable to receive any snapshots -- even
if I''ve
> literally just snapshotted, removed back to the previous snapshot, and then
> tried to receive that same snapshot.
A bit more information about the set up:

zpool is running at version 15
zfs is running version 2 (version 4 is available)
Kernel version: SunOS 5.10 Generic_141445-09
Kernel architecture: i86pc
Application architecture: i386

Server is a SunFire X4500 with 48 x 500Gb disks running firmware version
1.7.

I''ve tried a re-install of the server and this hasn''t helped
matters.

-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 04311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091210/cd6759d3/attachment.bin>

On Thu, Dec 10, 2009 at 1:50 AM, Andrew Robert Nicols
<andrew.nicols at luns.net.uk> wrote:
> The last snapshot received was named thumperpool/mnt at 200911301000 and
since
> then we''ve been completely unable to receive any snapshots -- even
if I''ve
> literally just snapshotted, removed back to the previous snapshot, and then
> tried to receive that same snapshot.
Have you tried using ''zfs receive -F thumperpool'' ?

-B


-- 
Brandon High : bhigh at freaks.com
If it wasn''t for pacifists, we could achieve peace.

> We''ve been using ZFS for about two years now and make a lot of use
of
> zfs
> send/receive to send our data from one X4500 to another. This has been
> working well for the past 18 months that we''ve been doing the
sends.
> 
> I recently upgraded the receiving thumper to Solaris 10 u8 and since
> then,
> I''ve been having no end of problems. 
I had a similar problem - the main filer was 10u6, and we built the snapshot
filer later, when the hardware platform had been updated, and 10u6 was no
longer supported on that platform, so I was force to go to 10u8 for driver
compatibility reasons.  The backup scripts that formerly worked, suddenly,
didn''t work anymore.  Even though there''s no conceivable
possibility of
writing to the secondary filer, the complaint was "filesystem has
changed"
and unable to receive incremental snap.

I solved it with this:
zfs set readonly=on targetfs

Hope that solves it for you too.