miyamoto moesasji
2009-Nov-05 20:38 UTC
Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5
I''ve just finished installing onto an OCZ Agilent v2 SSD with btrfs as filesystem. However to my surprise I''ve hit an ENOSPC condition one one of the partitions within less than a day of uptime, while the filesystem on that partition only reported 50% to be in use, which is far from the 75% limit people mention on the ML. Note that this occurs using a vanilla 2.6.32-rc5 kernel on a 64-bit gentoo system. Error-message from logs: 2009-11-05T07:55:57.586574+00:00 PulsarX4 kernel: [ 136.095961] no space left, need 4096, 4440064 delalloc bytes, 10704142336 bytes_used, 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use 10708582400 total 2009-11-05T07:55:57.645314+00:00 PulsarX4 kernel: [ 136.154217] no space left, need 4096, 4448256 delalloc bytes, 10704134144 bytes_used, 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use 10708582400 total Further details: 0) The partition that reports ENOSPC is mounted as: /dev/sda3 /usr btrfs defaults,rw,nodev,noatime 1) df -h reports : /dev/sda3 21G 11G 9.5G 53% /usr 2) btrfs-show : Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 Total devices 1 FS bytes used 10.60GB devid 1 size 20.00GB used 20.00GB path /dev/sda3 3) The other partitions using btrfs show a similar relatively large difference between the space reported by df -h and the size being taken up according to btrfs-show. Although this potentially is a problem between screen and chair I don''t see what I am doing wrong. Note that the ENOSPC issue occurs on a partition that still allows me to boot, so it is possible to run tests if needed when this indeed turns out to be a bug. Please let me know in case I need to provide further details from the logs. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
miyamoto moesasji
2009-Nov-05 20:55 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5
> Hmm looks like quite a bit of your fs got taken up for metadata. Perhaps try > running btrfsctl -b /usr and see if that frees up some space for you.Thanks, 1) The btrfsctl does not have the -b option on my system, is that an option that only is only enabled with the debug-utilities? 2) I would find it surprising if it is meta-data just looking at the numbers. Below is the output of btrfs-show for all partitions with btrfs. Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 Total devices 1 FS bytes used 89.53MB devid 1 size 8.00GB used 5.63GB path /dev/sda5 Label: none uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff Total devices 1 FS bytes used 150.14MB devid 1 size 4.00GB used 2.04GB path /dev/sda2 Label: none uuid: 558766bb-5e0d-48dd-9a13-7117f3047710 Total devices 1 FS bytes used 180.94MB devid 1 size 20.00GB used 5.04GB path /dev/sda6 Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 Total devices 1 FS bytes used 10.60GB devid 1 size 20.00GB used 20.00GB path /dev/sda3 Label: none uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b Total devices 1 FS bytes used 28.00KB devid 1 size 32.00GB used 2.04GB path /dev/sdb2 As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show demonstrates 5.63GB being used of the 8GB. It almost looks as if files that are overwritten don''t get removed from the space being in use. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik
2009-Nov-05 21:02 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5
On Thu, Nov 05, 2009 at 08:38:18PM +0000, miyamoto moesasji wrote:> I''ve just finished installing onto an OCZ Agilent v2 SSD with btrfs as > filesystem. However to my surprise I''ve hit an ENOSPC condition one > one of the partitions within less than a day of uptime, while the > filesystem on that partition only reported 50% to be in use, which is > far from the 75% limit people mention on the ML. > > Note that this occurs using a vanilla 2.6.32-rc5 kernel on a 64-bit > gentoo system. > > Error-message from logs: > > 2009-11-05T07:55:57.586574+00:00 PulsarX4 kernel: [ 136.095961] no > space left, need 4096, 4440064 delalloc bytes, 10704142336 bytes_used, > 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use > 10708582400 total > 2009-11-05T07:55:57.645314+00:00 PulsarX4 kernel: [ 136.154217] no > space left, need 4096, 4448256 delalloc bytes, 10704134144 bytes_used, > 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use > 10708582400 total > > Further details: > > 0) The partition that reports ENOSPC is mounted as: > /dev/sda3 /usr btrfs defaults,rw,nodev,noatime > > 1) df -h reports : /dev/sda3 21G 11G 9.5G 53% /usr > > 2) btrfs-show : > Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 > Total devices 1 FS bytes used 10.60GB > devid 1 size 20.00GB used 20.00GB path /dev/sda3 > > 3) The other partitions using btrfs show a similar relatively large > difference between the space reported by df -h and the size being > taken up according to btrfs-show. > > Although this potentially is a problem between screen and chair I > don''t see what I am doing wrong. Note that the ENOSPC issue occurs on > a partition that still allows me to boot, so it is possible to run > tests if needed when this indeed turns out to be a bug. > > Please let me know in case I need to provide further details from the logs.Hmm looks like quite a bit of your fs got taken up for metadata. Perhaps try running btrfsctl -b /usr and see if that frees up some space for you. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik
2009-Nov-05 21:55 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5
On Thu, Nov 05, 2009 at 08:55:12PM +0000, miyamoto moesasji wrote:> > > Hmm looks like quite a bit of your fs got taken up for metadata. Perhaps try > > running btrfsctl -b /usr and see if that frees up some space for you. > Thanks, > > 1) The btrfsctl does not have the -b option on my system, is that an option > that only is only enabled with the debug-utilities? >Sorry I always get that wrong, its btrfs-vol -b.> 2) I would find it surprising if it is meta-data just looking at the numbers. > Below is the output of btrfs-show for all partitions with btrfs. > > Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 > Total devices 1 FS bytes used 89.53MB > devid 1 size 8.00GB used 5.63GB path /dev/sda5 > > Label: none uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff > Total devices 1 FS bytes used 150.14MB > devid 1 size 4.00GB used 2.04GB path /dev/sda2 > > Label: none uuid: 558766bb-5e0d-48dd-9a13-7117f3047710 > Total devices 1 FS bytes used 180.94MB > devid 1 size 20.00GB used 5.04GB path /dev/sda6 > > Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 > Total devices 1 FS bytes used 10.60GB > devid 1 size 20.00GB used 20.00GB path /dev/sda3 > > Label: none uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b > Total devices 1 FS bytes used 28.00KB > devid 1 size 32.00GB used 2.04GB path /dev/sdb2 > > As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show > demonstrates 5.63GB being used of the 8GB. It almost looks as if files that are > overwritten don''t get removed from the space being in use. >The used part is how much of the volume is allocated into chunks. So when it says 5.63 gb is being used, that means that 5.63 gbs of the volume has been carved up into data/metadata chunks. I hope to at some point make one of our utilities tell you how much of that is for data and how much of that is metadata. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
miyamoto moesasji
2009-Nov-05 22:37 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5
1) Running btrfs-vol -b indeed does free up some space on the completely full partition, but not much, just 1GB. However I can use it again, so that is very helpful. Many thanks Josef! For completeness: After running it on sda5 and sda3: Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 Total devices 1 FS bytes used 90.31MB devid 1 size 8.00GB used 6.42GB path /dev/sda5 Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 Total devices 1 FS bytes used 10.60GB devid 1 size 20.00GB used 18.99GB path /dev/sda3 2) However I would still like to point out that I find it very surprising to see the amount of space taken up by data+meta-data, which looks dangerous to me seeing how quickly I got into a disk full situation while normal df indicated no problem whatsoever (if on root I would basically have had a kernel panic). Is this really expected behavior or is this a known problem already so no need to trouble-shoot? On 11/5/09, Josef Bacik <josef@redhat.com> wrote:> On Thu, Nov 05, 2009 at 08:55:12PM +0000, miyamoto moesasji wrote: >> >> > Hmm looks like quite a bit of your fs got taken up for metadata. >> > Perhaps try >> > running btrfsctl -b /usr and see if that frees up some space for you. >> Thanks, >> >> 1) The btrfsctl does not have the -b option on my system, is that an >> option >> that only is only enabled with the debug-utilities? >> > > Sorry I always get that wrong, its btrfs-vol -b. > >> 2) I would find it surprising if it is meta-data just looking at the >> numbers. >> Below is the output of btrfs-show for all partitions with btrfs. >> >> Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 >> Total devices 1 FS bytes used 89.53MB >> devid 1 size 8.00GB used 5.63GB path /dev/sda5 >> >> Label: none uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff >> Total devices 1 FS bytes used 150.14MB >> devid 1 size 4.00GB used 2.04GB path /dev/sda2 >> >> Label: none uuid: 558766bb-5e0d-48dd-9a13-7117f3047710 >> Total devices 1 FS bytes used 180.94MB >> devid 1 size 20.00GB used 5.04GB path /dev/sda6 >> >> Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 >> Total devices 1 FS bytes used 10.60GB >> devid 1 size 20.00GB used 20.00GB path /dev/sda3 >> >> Label: none uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b >> Total devices 1 FS bytes used 28.00KB >> devid 1 size 32.00GB used 2.04GB path /dev/sdb2 >> >> As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show >> demonstrates 5.63GB being used of the 8GB. It almost looks as if files >> that are >> overwritten don''t get removed from the space being in use. >> > > The used part is how much of the volume is allocated into chunks. So when > it > says 5.63 gb is being used, that means that 5.63 gbs of the volume has been > carved up into data/metadata chunks. I hope to at some point make one of > our > utilities tell you how much of that is for data and how much of that is > metadata. Thanks, > > Josef >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik
2009-Nov-05 22:43 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5
On Thu, Nov 05, 2009 at 10:37:10PM +0000, miyamoto moesasji wrote:> 1) Running btrfs-vol -b indeed does free up some space on the > completely full partition, but not much, just 1GB. However I can use > it again, so that is very helpful. Many thanks Josef! > > For completeness: After running it on sda5 and sda3: > Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 > Total devices 1 FS bytes used 90.31MB > devid 1 size 8.00GB used 6.42GB path /dev/sda5 > > Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 > Total devices 1 FS bytes used 10.60GB > devid 1 size 20.00GB used 18.99GB path /dev/sda3 > > 2) However I would still like to point out that I find it very > surprising to see the amount of space taken up by data+meta-data, > which looks dangerous to me seeing how quickly I got into a disk full > situation while normal df indicated no problem whatsoever (if on root > I would basically have had a kernel panic). Is this really expected > behavior or is this a known problem already so no need to > trouble-shoot? >Have you been using btrfs since 2.6.32-rc3 or have you used it for a while now and just recently gone to a 2.6.32-rc kernel? The reason I ask is because we did all sorts of things to try and make sure users didn''t run out of space, which included being overly agressive about making sure there was plenty of metadata space. The enospc patches that went into 2.6.32-rc3 (i think, somewhere in there) has made this much better and you shouldn''t be seeing this bad of an imbalance towards metadata. So, if this fs was created pre 2.6.32-rc3 then this is expected and unfortunate. If this fs was post that time then this is a problem and we need to figure out whats wrong. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
miyamoto moesasji
2009-Nov-05 23:07 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5
I''m not 100% sure what is the correct answer. This is a clean install, just installed this weekend so the system itself has only been running with the 2.6.32-rc5 kernel, nothing older than that and the drive itself was completely new/clean. However for the install itself I''ve used SystemRescueCD which was using a 2.6.31.1 kernel. Hence the partitions have been formatted using the kernel 2.6.31.1 and that is also the kernel used during the install of the system. On 11/5/09, Josef Bacik <josef@redhat.com> wrote:> On Thu, Nov 05, 2009 at 10:37:10PM +0000, miyamoto moesasji wrote: >> 1) Running btrfs-vol -b indeed does free up some space on the >> completely full partition, but not much, just 1GB. However I can use >> it again, so that is very helpful. Many thanks Josef! >> >> For completeness: After running it on sda5 and sda3: >> Label: none uuid: a12ac0e9-cbea-4acf-bb26-181146940714 >> Total devices 1 FS bytes used 90.31MB >> devid 1 size 8.00GB used 6.42GB path /dev/sda5 >> >> Label: none uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5 >> Total devices 1 FS bytes used 10.60GB >> devid 1 size 20.00GB used 18.99GB path /dev/sda3 >> >> 2) However I would still like to point out that I find it very >> surprising to see the amount of space taken up by data+meta-data, >> which looks dangerous to me seeing how quickly I got into a disk full >> situation while normal df indicated no problem whatsoever (if on root >> I would basically have had a kernel panic). Is this really expected >> behavior or is this a known problem already so no need to >> trouble-shoot? >> > > Have you been using btrfs since 2.6.32-rc3 or have you used it for a while > now > and just recently gone to a 2.6.32-rc kernel? The reason I ask is because > we > did all sorts of things to try and make sure users didn''t run out of space, > which included being overly agressive about making sure there was plenty of > metadata space. The enospc patches that went into 2.6.32-rc3 (i think, > somewhere in there) has made this much better and you shouldn''t be seeing > this > bad of an imbalance towards metadata. So, if this fs was created pre > 2.6.32-rc3 then this is expected and unfortunate. If this fs was post that > time > then this is a problem and we need to figure out whats wrong. Thanks, > > Josef >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik
2009-Nov-06 01:25 UTC
Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5
On Thu, Nov 05, 2009 at 11:07:27PM +0000, miyamoto moesasji wrote:> I''m not 100% sure what is the correct answer. > > This is a clean install, just installed this weekend so the system > itself has only been running with the 2.6.32-rc5 kernel, nothing older > than that and the drive itself was completely new/clean. However for > the install itself I''ve used SystemRescueCD which was using a 2.6.31.1 > kernel. Hence the partitions have been formatted using the kernel > 2.6.31.1 and that is also the kernel used during the install of the > system. >Yeah if most of the data was put on the disk under the old kernel then its likely everything was skewed towards metadata and thats why you are having problems. Theres not much more I can say other than sorry :(. If you can, try copying the data from that volume to a new volume, format the old volume, and copy it back under the new kernel and it should work out better. Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html