Bartosz Kulicki
2013-Nov-06 21:53 UTC
Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
Hi, As per subject. Seems UUID tree creation failed after upgrade. I could not mount filesystem under 3.12. Going back to 3.8.10 allowed me to mount fs but I could no longer perform any deletes, writes etc. I''ve opened a bug report here. https://bugzilla.kernel.org/show_bug.cgi?id=64461 I''ve included link to image captured with btrfs-image. Please CC me if further information is needed as I''m not subscribed to the mailing list. regards, Bartosz -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Stefan Behrens
2013-Nov-07 08:49 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
On Wed, 6 Nov 2013 21:53:25 +0000, Bartosz Kulicki wrote:> As per subject. Seems UUID tree creation failed after upgrade. I could > not mount filesystem under 3.12. Going back to 3.8.10 allowed me to > mount fs but I could no longer perform any deletes, writes etc. > > I''ve opened a bug report here. > https://bugzilla.kernel.org/show_bug.cgi?id=64461 > > I''ve included link to image captured with btrfs-image. > > Please CC me if further information is needed as I''m not subscribed to > the mailing list. >ENOSPC means you''re out of disk space. A copy-on-write filesystem needs disk space for delete operations, that''s not a bug. What does ''btrfs fi df /mountpoint'' say? How much disk space do you have, how much is allocated? And try the procedure that is described in the wiki: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Bartosz Kulicki
2013-Nov-07 11:33 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
Hi Stefan, Yes, I realise that. As I pointed out in bug report I have tried workarounds suggested in wiki and more. To be precise: partial balancing didn''t work (completed without error but no change), balacing fs failed with ENOSPC, clobbering a file didn''t work (ENOSPC), btrfs-zero-log completed but no change. The filesystem was quite full 98% but I could still write to it before rebooting with new kernel. I had to nuke the fs since then and restore from backup but I captured image with btrfs-image before running btrfs-zero-log. regards, Bart On Thu, Nov 7, 2013 at 8:49 AM, Stefan Behrens <sbehrens@giantdisaster.de> wrote:> On Wed, 6 Nov 2013 21:53:25 +0000, Bartosz Kulicki wrote: >> As per subject. Seems UUID tree creation failed after upgrade. I could >> not mount filesystem under 3.12. Going back to 3.8.10 allowed me to >> mount fs but I could no longer perform any deletes, writes etc. >> >> I''ve opened a bug report here. >> https://bugzilla.kernel.org/show_bug.cgi?id=64461 >> >> I''ve included link to image captured with btrfs-image. >> >> Please CC me if further information is needed as I''m not subscribed to >> the mailing list. >> > > ENOSPC means you''re out of disk space. A copy-on-write filesystem needs > disk space for delete operations, that''s not a bug. > > What does ''btrfs fi df /mountpoint'' say? > How much disk space do you have, how much is allocated? > > And try the procedure that is described in the wiki: > > https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space > >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Bartosz Kulicki
2013-Nov-07 11:43 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
FWIW - just before nuking the fs I have added a 3GB loopback device to btrfs. This restored ability to delete the files but I could not remove the loopback after deleting some large files (if I remember correctly error I got was "block device required") cheers, Bart -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Duncan
2013-Nov-07 14:04 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
Bartosz Kulicki posted on Thu, 07 Nov 2013 11:43:15 +0000 as excerpted:> FWIW - just before nuking the fs I have added a 3GB loopback device to > btrfs. > > This restored ability to delete the files but I could not remove the > loopback after deleting some large files (if I remember correctly error > I got was "block device required")Having read the wiki and the list but not having (yet) gotten into that sort of situation myself, I''m following this with some interest, and was just about to ask if you''d tried that. OK, so you added the loop device and were then able to do deletes but afterward could not delete the device... The device thus obviously added some room for the filesystem to work, so you could delete files. After that file-delete, did you then try a balance again? Because as documented, and as demonstrated by the initial now allowed file-delete, that should have given you just enough extra metadata space to get you out of the jam, allowing a balance to finish, which in theory at least would have freed up enough space due to the compaction of the balance, to then finally do a proper btrfs device delete on the temporary loop device. But without that last balance attempt after the loop add and successful file delete, we''ll never know if the temporary loop device could have then been successfully btrfs device deleted or not. Which is a bit frustrating here, since that''s exactly the bit omitted from your posting what you tried, but what I''m counting on to get me out of a similar jam, should I ever get in it... -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Russell Coker
2013-Nov-10 13:10 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
On Thu, 7 Nov 2013, Bartosz Kulicki <bartosz.kulicki@gmail.com> wrote:> FWIW - just before nuking the fs I have added a 3GB loopback device to > btrfs. > > This restored ability to delete the files but I could not remove the > loopback after deleting some large files (if I remember correctly > error I got was "block device required")I once had a problem where I added a second block device and started a balance. But for some reason the balance decided to make metadata a RAID-1 and even when there was enough space I couldn''t remove it (you must have 2 devices for RAID-1). So I added a third device, that allowed me to delete the second device which made the meta-data no longer RAID-1 and I could then delete the third device and have the single-device BTRFS filesystem I wanted. That was a while ago, maybe running kernel 3.10 or 3.8. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Duncan
2013-Nov-10 15:42 UTC
Re: Fwd: unable to delete files after kernel upgrade from 3.8.10 to 3.12
Russell Coker posted on Mon, 11 Nov 2013 00:10:36 +1100 as excerpted:> On Thu, 7 Nov 2013, Bartosz Kulicki <bartosz.kulicki@gmail.com> wrote: >> FWIW - just before nuking the fs I have added a 3GB loopback device to >> btrfs. >> >> This restored ability to delete the files but I could not remove the >> loopback after deleting some large files (if I remember correctly error >> I got was "block device required") > > I once had a problem where I added a second block device and started a > balance. But for some reason the balance decided to make metadata a > RAID-1 and even when there was enough space I couldn''t remove it (you > must have 2 devices for RAID-1). > > So I added a third device, that allowed me to delete the second device > which made the meta-data no longer RAID-1 and I could then delete the > third device and have the single-device BTRFS filesystem I wanted. > > That was a while ago, maybe running kernel 3.10 or 3.8.Hmm... Very good point that I guess the classic "add a device to get out of the jam" recommendation doesn''t cover, without a more complex explanation, at least! Thanks for bringing it up! For safety reasons btrfs (almost[1]) always defaults to two copies of metadata. On a single device, that''s DUP mode, two copies obviously on the same device. But with two or more devices it''ll default metadata to raid1 mode, trying to keep one copy of metadata on each of two different devices thus allowing the chance to recover at least the data that''s on surviving drives in the event of a failure. So if there''s only a single existing device and the "add-a-device-to-get- out-of-the-jam" method is used, either adding a /third/ device may be needed (your solution), or alternatively, doing the balance using options to force single mode may be necessary: btrfs balance -mconvert=single -f <path> Or possibly -mconvert=dup, to force metadata to stay dup mode, but I''m not sure without trying it whether dup will work on more than a single device. --- [1] The exception is SSD, I believe only with a single device, where SINGLE mode is the default because some SSDs automatically dedup in any case, so even DUP mode would only actually be physically stored only once, second-guessing btrfs'' efforts to keep two separate copies. I''m not sure why the dedup feature changes the default for /all/ ssds as it seems to me SSDs without that feature should arguably still get DUP by default which means it''s a bad exception, do DUP regardless and if the hardware dedups let the hardware dedup seems more reasonable, but that''s what''s documented. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html