Hi all, I''ve managed to ruin my btrfs filesystem, but I don''t know how. I basically have it split up into two subvolumes, one for the root and one for home both of which are creating hourly snapshots. I''ve ''btrfs send'' some of those to an external machine every now and then. (At least once a day) At some point ''btrfs send'' stopped working but gave no better error message than invalid argument. I did a balance on the drive which resulted in a kernel panic after a while of working. It then repeated the balance automatically on the next bootup until it finally said ''balance failed'' when I checked with btrfs fi balance status. I''m running a funky setup here where two partitions on the same drive are both added into the same btrfs volume resulting in this setup: Label: none uuid: 4e74fa05-4bf2-40da-946b-8cd53ce92242 Total devices 2 FS bytes used 54.60GB devid 1 size 60.52GB used 52.04GB path /dev/sda7 devid 2 size 16.32GB used 8.03GB path /dev/sda5 Data: total=54.00GB, used=53.51GB System, RAID1: total=32.00MB, used=12.00KB System: total=4.00MB, used=0.00 Metadata, RAID1: total=3.00GB, used=1.11GB The state that I''m in is that everything is working, I haven''t found any data corruption, scrubs work and confirm that there are no errors but I can no longer use ''btrfs send'' to create differential images and I can also no longer complete a balance. I tried running balance start -dconvert=single to get rid of my second partition but that also failed (without a panic though) I have backed up the newest state of the filesystem via rsync and am about to wipe and restore from backup with a new fs. Is anyone interested in a btrfs-image to debug this state? If so please tell me what command to run since I have never done this before... Thanks Harald -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
The dmesg from the kernel panic can be seen here: http://pastebin.com/3QZqrqAX I''ve tried to create a btrfs-image but I''m getting a crash of the utility after only 50 MB... Check tree block failed, want=320304799744, have=0 Check tree block failed, want=320305049600, have=0 Check tree block failed, want=320298184704, have=0 Check tree block failed, want=320298184704, have=0 Check tree block failed, want=320305221632, have=0 Check tree block failed, want=320308678656, have=0 Check tree block failed, want=320308678656, have=0 Check tree block failed, want=320297295872, have=0 Check tree block failed, want=320297295872, have=0 Check tree block failed, want=320297295872, have=721320548710005777 Check tree block failed, want=320297295872, have=721320548710005777 Check tree block failed, want=320297295872, have=721320548710005777 read block failed check_tree_block btrfs-image: btrfs-image.c:518: create_metadump: Assertion `!(ret < 0)'' failed. [1] 32513 abort (core dumped) On Mon, Apr 8, 2013 at 9:48 AM, Harald Glatt <mail@hachre.de> wrote:> Hi all, > > I''ve managed to ruin my btrfs filesystem, but I don''t know how. I > basically have it split up into two subvolumes, one for the root and > one for home both of which are creating hourly snapshots. > > I''ve ''btrfs send'' some of those to an external machine every now and > then. (At least once a day) > > At some point ''btrfs send'' stopped working but gave no better error > message than invalid argument. > > I did a balance on the drive which resulted in a kernel panic after a > while of working. It then repeated the balance automatically on the > next bootup until it finally said ''balance failed'' when I checked with > btrfs fi balance status. > > I''m running a funky setup here where two partitions on the same drive > are both added into the same btrfs volume resulting in this setup: > > Label: none uuid: 4e74fa05-4bf2-40da-946b-8cd53ce92242 > Total devices 2 FS bytes used 54.60GB > devid 1 size 60.52GB used 52.04GB path /dev/sda7 > devid 2 size 16.32GB used 8.03GB path /dev/sda5 > > Data: total=54.00GB, used=53.51GB > System, RAID1: total=32.00MB, used=12.00KB > System: total=4.00MB, used=0.00 > Metadata, RAID1: total=3.00GB, used=1.11GB > > The state that I''m in is that everything is working, I haven''t found > any data corruption, scrubs work and confirm that there are no errors > but I can no longer use ''btrfs send'' to create differential images and > I can also no longer complete a balance. > > I tried running balance start -dconvert=single to get rid of my second > partition but that also failed (without a panic though) > > I have backed up the newest state of the filesystem via rsync and am > about to wipe and restore from backup with a new fs. > > Is anyone interested in a btrfs-image to debug this state? If so > please tell me what command to run since I have never done this > before... > > Thanks > Harald-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Apr 8, 2013 at 10:45 AM, Harald Glatt <mail@hachre.de> wrote:> The dmesg from the kernel panic can be seen here: > http://pastebin.com/3QZqrqAX > > I''ve tried to create a btrfs-image but I''m getting a crash of the > utility after only 50 MB... > > Check tree block failed, want=320304799744, have=0 > Check tree block failed, want=320305049600, have=0 > Check tree block failed, want=320298184704, have=0 > Check tree block failed, want=320298184704, have=0 > Check tree block failed, want=320305221632, have=0 > Check tree block failed, want=320308678656, have=0 > Check tree block failed, want=320308678656, have=0 > Check tree block failed, want=320297295872, have=0 > Check tree block failed, want=320297295872, have=0 > Check tree block failed, want=320297295872, have=721320548710005777 > Check tree block failed, want=320297295872, have=721320548710005777 > Check tree block failed, want=320297295872, have=721320548710005777 > read block failed check_tree_block > btrfs-image: btrfs-image.c:518: create_metadump: Assertion `!(ret < 0)'' failed. > [1] 32513 abort (core dumped) > > > On Mon, Apr 8, 2013 at 9:48 AM, Harald Glatt <mail@hachre.de> wrote: >> Hi all, >> >> I''ve managed to ruin my btrfs filesystem, but I don''t know how. I >> basically have it split up into two subvolumes, one for the root and >> one for home both of which are creating hourly snapshots. >> >> I''ve ''btrfs send'' some of those to an external machine every now and >> then. (At least once a day) >> >> At some point ''btrfs send'' stopped working but gave no better error >> message than invalid argument. >> >> I did a balance on the drive which resulted in a kernel panic after a >> while of working. It then repeated the balance automatically on the >> next bootup until it finally said ''balance failed'' when I checked with >> btrfs fi balance status. >> >> I''m running a funky setup here where two partitions on the same drive >> are both added into the same btrfs volume resulting in this setup: >> >> Label: none uuid: 4e74fa05-4bf2-40da-946b-8cd53ce92242 >> Total devices 2 FS bytes used 54.60GB >> devid 1 size 60.52GB used 52.04GB path /dev/sda7 >> devid 2 size 16.32GB used 8.03GB path /dev/sda5 >> >> Data: total=54.00GB, used=53.51GB >> System, RAID1: total=32.00MB, used=12.00KB >> System: total=4.00MB, used=0.00 >> Metadata, RAID1: total=3.00GB, used=1.11GB >> >> The state that I''m in is that everything is working, I haven''t found >> any data corruption, scrubs work and confirm that there are no errors >> but I can no longer use ''btrfs send'' to create differential images and >> I can also no longer complete a balance. >> >> I tried running balance start -dconvert=single to get rid of my second >> partition but that also failed (without a panic though) >> >> I have backed up the newest state of the filesystem via rsync and am >> about to wipe and restore from backup with a new fs. >> >> Is anyone interested in a btrfs-image to debug this state? If so >> please tell me what command to run since I have never done this >> before... >> >> Thanks >> HaraldI didn''t realize it had to be unmounted for the image. I''ve uploaded a successful image with -c9. I''ll give out the URL on demand. However there are news: I managed after creating that image to successfully balance down to single and remove the second drive. I readded it again and now everything is back to normal and working... It is possible that after that kernel panic it indeed finished the balance correctly and I misread the error message saying something along the lines of ''error while getting status: balance not in progress'' for something that was telling me the balance failed... Ultimately that reduces this issue down to, something caused ''btrfs send'' to fail, a balance then kernel paniced and afterwards everything was back to normal. I don''t think the image that I have will be of any help because that imagine is of course from after that kernel panic and I suppose completeling that balance then fixed the issues - even if it paniced along the way. Thanks Harald -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 8 Apr 2013 09:48:55 +0200 Harald Glatt <mail@hachre.de> wrote:> At some point ''btrfs send'' stopped working but gave no better error > message than invalid argument.Usually in cases like this there are more details in ''dmesg''.> I did a balance on the drive which resulted in a kernel panic after a > while of working.Maybe you faced the same problem that I had recently with "Object already exists" . Could''ve been useful if you saved the actual kernel panic message. -- With respect, Roman
You''re right, I''m seeing this ''object already exists'' message in my dmesg log. Nice catch!! I''m posting it again in case others missed it too, this was the kernel panic dmesg that happened during the balance that I did in order to fix the ''btrfs send'' not working anymore problem: http://pastebin.com/3QZqrqAX On Mon, Apr 8, 2013 at 1:14 PM, Roman Mamedov <rm@romanrm.ru> wrote:> On Mon, 8 Apr 2013 09:48:55 +0200 > Harald Glatt <mail@hachre.de> wrote: > >> At some point ''btrfs send'' stopped working but gave no better error >> message than invalid argument. > > Usually in cases like this there are more details in ''dmesg''. > >> I did a balance on the drive which resulted in a kernel panic after a >> while of working. > > Maybe you faced the same problem that I had recently with "Object already > exists" . Could''ve been useful if you saved the actual kernel panic message. > > -- > With respect, > Roman-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 8 Apr 2013 13:19:13 +0200 Harald Glatt <mail@hachre.de> wrote:> You''re right, I''m seeing this ''object already exists'' message in my > dmesg log. Nice catch!!See this thread: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23514.html This patch fixes the problem: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23534.html But it is / will be only in the very latest kernels (including ''stable'' updates), because it was posted only like a week ago. -- With respect, Roman
That''s not gonna be a problem, I''m always running on the most recent final kernel - in the case of my panic it was 3.8.6. Thanks! On Mon, Apr 8, 2013 at 1:22 PM, Roman Mamedov <rm@romanrm.ru> wrote:> On Mon, 8 Apr 2013 13:19:13 +0200 > Harald Glatt <mail@hachre.de> wrote: > >> You''re right, I''m seeing this ''object already exists'' message in my >> dmesg log. Nice catch!! > > See this thread: > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23514.html > > This patch fixes the problem: > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23534.html > > But it is / will be only in the very latest kernels (including ''stable'' > updates), because it was posted only like a week ago. > > -- > With respect, > Roman-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html