I have a Btrfs partition that is reliably reproducing premature ENOSPC when restoring the disk from a tar file, but it is only happening with zlib compression (lzo or no compression proceeds normally). I''ve had the same issue at least back through the 3.1 kernel series, and I''ve been having intermittent issues even further back. I am currently using a 3.2.1 kernel merged with Chris'' latest integration branch. I''ve performed about 12 trials trying to explore various combinations of compress, compress-force, compress[-force]=[zlib,lzo] and autodefrag. If I use no compression, or if I explicitly declare lzo compression, I don''t receive the premature ENOSPC when untarring my restoration archive to the empty partition. If I don''t specify compression (zlib is the default) or specify zlib, I get consistent premature ENOSPC errors regardless of other combinations. I apologize if this is already general knowledge, but I couldn''t see where this has been posted to the list before. As time allows, I will try to capture exactly where this ENOSPC is being issued in btrfs by inserting WARN_ON''s in my local version where-ever ENOSPC is set. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jan 18, 2012 at 5:13 PM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:> I have a Btrfs partition that is reliably reproducing premature ENOSPC > when restoring the disk from a tar file, but it is only happening with > zlib compression (lzo or no compression proceeds normally).thank you very much for this info. i did the same thing: compress=lzo now i can do a full system installation to an fresh btrfs filesystem without "--bwlimit 4096" to rsync again! my kernel: vanilla 3.2.1 Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jan 18, 2012 at 10:13 AM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:> I have a Btrfs partition that is reliably reproducing premature ENOSPC > when restoring the disk from a tar file, but it is only happening with > zlib compression (lzo or no compression proceeds normally). > > I''ve had the same issue at least back through the 3.1 kernel series, > and I''ve been having intermittent issues even further back. > > I am currently using a 3.2.1 kernel merged with Chris'' latest > integration branch. > > I''ve performed about 12 trials trying to explore various combinations > of compress, compress-force, compress[-force]=[zlib,lzo] and > autodefrag. > > If I use no compression, or if I explicitly declare lzo compression, I > don''t receive the premature ENOSPC when untarring my restoration > archive to the empty partition. > > If I don''t specify compression (zlib is the default) or specify zlib, > I get consistent premature ENOSPC errors regardless of other > combinations. > > I apologize if this is already general knowledge, but I couldn''t see > where this has been posted to the list before. > > As time allows, I will try to capture exactly where this ENOSPC is > being issued in btrfs by inserting WARN_ON''s in my local version > where-ever ENOSPC is set.Some follow-up... I''ve injected some debugging code to isolate when the ENOSPC is being generated when using zlib compression. When using zlib, I''m getting intermittent ENOSPC in the may_commit_transaction() function in extent-tree.c at this point: if (delayed_rsv->size < bytes) { spin_unlock(&delayed_rsv->lock); return -ENOSPC; } The typical values for (delayed_rsv->size < bytes) have been: delayed_rsv->size ( = 0x60000) < bytes ( = 0x78000) This typically occurs when unzipping a section of my backup that contains lots of small files that are probably being mostly in-lined. I don''t see errors in this section when using lzo or no compression. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 02/09/2012 05:01 AM, Mitch Harder wrote:> On Wed, Jan 18, 2012 at 10:13 AM, Mitch Harder > <mitch.harder@sabayonlinux.org> wrote: >> I have a Btrfs partition that is reliably reproducing premature ENOSPC >> when restoring the disk from a tar file, but it is only happening with >> zlib compression (lzo or no compression proceeds normally). >> >> I''ve had the same issue at least back through the 3.1 kernel series, >> and I''ve been having intermittent issues even further back. >> >> I am currently using a 3.2.1 kernel merged with Chris'' latest >> integration branch. >> >> I''ve performed about 12 trials trying to explore various combinations >> of compress, compress-force, compress[-force]=[zlib,lzo] and >> autodefrag. >> >> If I use no compression, or if I explicitly declare lzo compression, I >> don''t receive the premature ENOSPC when untarring my restoration >> archive to the empty partition. >> >> If I don''t specify compression (zlib is the default) or specify zlib, >> I get consistent premature ENOSPC errors regardless of other >> combinations. >> >> I apologize if this is already general knowledge, but I couldn''t see >> where this has been posted to the list before. >> >> As time allows, I will try to capture exactly where this ENOSPC is >> being issued in btrfs by inserting WARN_ON''s in my local version >> where-ever ENOSPC is set. > > Some follow-up... > > I''ve injected some debugging code to isolate when the ENOSPC is being > generated when using zlib compression. > > When using zlib, I''m getting intermittent ENOSPC in the > may_commit_transaction() function in extent-tree.c at this point: > > if (delayed_rsv->size < bytes) { > spin_unlock(&delayed_rsv->lock); > return -ENOSPC; > } > > The typical values for (delayed_rsv->size < bytes) have been: > delayed_rsv->size ( = 0x60000) < bytes ( = 0x78000) > > This typically occurs when unzipping a section of my backup that > contains lots of small files that are probably being mostly in-lined. > > I don''t see errors in this section when using lzo or no compression.Hi Mitch, Would you like to try this patch? thanks, liubo diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 8603ee4..d83b15e 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -3483,28 +3483,34 @@ static int may_commit_transaction(struct btrfs_root *root, if (force) goto commit; - /* See if there is enough pinned space to make this reservation */ - spin_lock(&space_info->lock); - if (space_info->bytes_pinned >= bytes) { + if (space_info != delayed_rsv->space_info) { + /* + * For DATA: + * See if there is enough pinned space to make this reservation + */ + spin_lock(&space_info->lock); + if (space_info->bytes_pinned < bytes) { + spin_unlock(&space_info->lock); + return -ENOSPC; + } spin_unlock(&space_info->lock); - goto commit; - } - spin_unlock(&space_info->lock); - - /* - * See if there is some space in the delayed insertion reservation for - * this reservation. - */ - if (space_info != delayed_rsv->space_info) - return -ENOSPC; - - spin_lock(&delayed_rsv->lock); - if (delayed_rsv->size < bytes) { + } else { + /* + * For METADATA: + * See if there is enough space(pinned and delayed insertion) + * to make this reservation + */ + spin_lock(&space_info->lock); + spin_lock(&delayed_rsv->lock); + if (space_info->bytes_pinned + delayed_rsv->size < bytes) { + spin_unlock(&delayed_rsv->lock); + spin_unlock(&space_info->lock); + return -ENOSPC; + } spin_unlock(&delayed_rsv->lock); - return -ENOSPC; - } - spin_unlock(&delayed_rsv->lock); + spin_unlock(&space_info->lock); + } commit: trans = btrfs_join_transaction(root); if (IS_ERR(trans)) -- 1.6.5.2> -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Feb 8, 2012 at 8:14 PM, Liu Bo <liubo2009@cn.fujitsu.com> wrote:> On 02/09/2012 05:01 AM, Mitch Harder wrote: >> On Wed, Jan 18, 2012 at 10:13 AM, Mitch Harder >> <mitch.harder@sabayonlinux.org> wrote: >>> I have a Btrfs partition that is reliably reproducing premature ENOSPC >>> when restoring the disk from a tar file, but it is only happening with >>> zlib compression (lzo or no compression proceeds normally). >>> >>> I''ve had the same issue at least back through the 3.1 kernel series, >>> and I''ve been having intermittent issues even further back. >>> >>> I am currently using a 3.2.1 kernel merged with Chris'' latest >>> integration branch. >>> >>> I''ve performed about 12 trials trying to explore various combinations >>> of compress, compress-force, compress[-force]=[zlib,lzo] and >>> autodefrag. >>> >>> If I use no compression, or if I explicitly declare lzo compression, I >>> don''t receive the premature ENOSPC when untarring my restoration >>> archive to the empty partition. >>> >>> If I don''t specify compression (zlib is the default) or specify zlib, >>> I get consistent premature ENOSPC errors regardless of other >>> combinations. >>> >>> I apologize if this is already general knowledge, but I couldn''t see >>> where this has been posted to the list before. >>> >>> As time allows, I will try to capture exactly where this ENOSPC is >>> being issued in btrfs by inserting WARN_ON''s in my local version >>> where-ever ENOSPC is set. >> >> Some follow-up... >> >> I''ve injected some debugging code to isolate when the ENOSPC is being >> generated when using zlib compression. >> >> When using zlib, I''m getting intermittent ENOSPC in the >> may_commit_transaction() function in extent-tree.c at this point: >> >> if (delayed_rsv->size < bytes) { >> spin_unlock(&delayed_rsv->lock); >> return -ENOSPC; >> } >> >> The typical values for (delayed_rsv->size < bytes) have been: >> delayed_rsv->size ( = 0x60000) < bytes ( = 0x78000) >> >> This typically occurs when unzipping a section of my backup that >> contains lots of small files that are probably being mostly in-lined. >> >> I don''t see errors in this section when using lzo or no compression. > > Hi Mitch, > > Would you like to try this patch? > > thanks, > liubo > > diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c > index 8603ee4..d83b15e 100644 > --- a/fs/btrfs/extent-tree.c > +++ b/fs/btrfs/extent-tree.c > @@ -3483,28 +3483,34 @@ static int may_commit_transaction(struct btrfs_root *root, > if (force) > goto commit; > > - /* See if there is enough pinned space to make this reservation */ > - spin_lock(&space_info->lock); > - if (space_info->bytes_pinned >= bytes) { > + if (space_info != delayed_rsv->space_info) { > + /* > + * For DATA: > + * See if there is enough pinned space to make this reservation > + */ > + spin_lock(&space_info->lock); > + if (space_info->bytes_pinned < bytes) { > + spin_unlock(&space_info->lock); > + return -ENOSPC; > + } > spin_unlock(&space_info->lock); > - goto commit; > - } > - spin_unlock(&space_info->lock); > - > - /* > - * See if there is some space in the delayed insertion reservation for > - * this reservation. > - */ > - if (space_info != delayed_rsv->space_info) > - return -ENOSPC; > - > - spin_lock(&delayed_rsv->lock); > - if (delayed_rsv->size < bytes) { > + } else { > + /* > + * For METADATA: > + * See if there is enough space(pinned and delayed insertion) > + * to make this reservation > + */ > + spin_lock(&space_info->lock); > + spin_lock(&delayed_rsv->lock); > + if (space_info->bytes_pinned + delayed_rsv->size < bytes) { > + spin_unlock(&delayed_rsv->lock); > + spin_unlock(&space_info->lock); > + return -ENOSPC; > + } > spin_unlock(&delayed_rsv->lock); > - return -ENOSPC; > - } > - spin_unlock(&delayed_rsv->lock); > + spin_unlock(&space_info->lock); > > + } > commit: > trans = btrfs_join_transaction(root); > if (IS_ERR(trans)) > -- > 1.6.5.2 > >Thanks for looking at this, but I''m still getting ENOSPC errors at the same point after applying this patch. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jan 18, 2012 at 5:13 PM, Mitch Harder <mitch.harder@sabayonlinux.org> wrote:> I have a Btrfs partition that is reliably reproducing premature ENOSPC > when restoring the disk from a tar file, but it is only happening with > zlib compression (lzo or no compression proceeds normally).Ive just unsquashfs''ed an 5GiB image on a fresh btrfs and a much faster System using: compress=lzo,noatime and got ENOSPC again :( so it doesnt matter if lzo or zlib: ENOSPC with compression enabled! my kernel: vanilla 3.2.5 for testing, i suggest making a squashfs of your whole system "/" and unsquashfs that image into a new subvolume. this really seems to trigger this ENOSPC problem, because it creates the files fast enough. Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thursday 16 February 2012 19:06:52 Ahmet Inan wrote:> and got ENOSPC again :( > > so it doesnt matter if lzo or zlib: ENOSPC with compression > enabled! > > my kernel: vanilla 3.2.5I know that there has been an ENOSPC fix go in to the 3.3 RC kernels since the 3.2 release, any chance you''d be able to see if it still happens with the current 3.3 RC ? cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
On Thu, Feb 16, 2012 at 11:31 AM, Chris Samuel <chris@csamuel.org> wrote:> On Thursday 16 February 2012 19:06:52 Ahmet Inan wrote: > >> and got ENOSPC again :( >> >> so it doesnt matter if lzo or zlib: ENOSPC with compression >> enabled! >> >> my kernel: vanilla 3.2.5 > > I know that there has been an ENOSPC fix go in to the 3.3 RC kernels > since the 3.2 release, any chance you''d be able to see if it still > happens with the current 3.3 RC ?just tested 3.3.0-rc3, and it works like a charm! but there are lots of issues with external packages not compiling with 3.3, why i wont be able to deploy it to our systems here yet. could this one bugfix be backported to 3.2? here are some results from my unsquashfs''ing just to show how awesome btrfs with lzo is. these results are _not_ from an ssd btw! (spinning disks, rebooted between tests) # time unsquashfs /dev/nbd0 Parallel unsquashfs: Using 8 processors 792630 inodes (870469 blocks) to write [==================/] 870469/870469 100% created 759645 files created 67271 directories created 27250 symlinks created 5141 devices created 0 fifos Timing Results: zlib: real 6m38.586s user 2m16.021s sys 0m31.138s lzo: real 1m53.429s user 1m55.622s sys 0m46.664s no compression: real 1m16.975s user 1m47.173s sys 0m53.520s complete system installation in under 5minutes, that simply rocks! Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Feb 16, 2012 at 04:05:40PM +0100, Ahmet Inan wrote:> On Thu, Feb 16, 2012 at 11:31 AM, Chris Samuel <chris@csamuel.org> wrote: > > On Thursday 16 February 2012 19:06:52 Ahmet Inan wrote: > > > >> and got ENOSPC again :( > >> > >> so it doesnt matter if lzo or zlib: ENOSPC with compression > >> enabled! > >> > >> my kernel: vanilla 3.2.5 > > > > I know that there has been an ENOSPC fix go in to the 3.3 RC kernels > > since the 3.2 release, any chance you''d be able to see if it still > > happens with the current 3.3 RC ? > > just tested 3.3.0-rc3, and it works like a charm! > > but there are lots of issues with external packages not compiling with 3.3, > why i wont be able to deploy it to our systems here yet. > > could this one bugfix be backported to 3.2?All of the btrfs patches in 3.3-rc are against 3.2 in my git tree: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus> > here are some results from my unsquashfs''ing just to show how > awesome btrfs with lzo is. these results are _not_ from an ssd btw! > (spinning disks, rebooted between tests)Very nice ;) -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Feb 16, 2012 at 6:55 PM, Chris Mason <chris.mason@oracle.com> wrote:> On Thu, Feb 16, 2012 at 04:05:40PM +0100, Ahmet Inan wrote: >> On Thu, Feb 16, 2012 at 11:31 AM, Chris Samuel <chris@csamuel.org> wrote: >> > On Thursday 16 February 2012 19:06:52 Ahmet Inan wrote: >> > >> >> and got ENOSPC again :( >> >> >> >> so it doesnt matter if lzo or zlib: ENOSPC with compression >> >> enabled! >> >> >> >> my kernel: vanilla 3.2.5 >> > >> > I know that there has been an ENOSPC fix go in to the 3.3 RC kernels >> > since the 3.2 release, any chance you''d be able to see if it still >> > happens with the current 3.3 RC ? >> >> just tested 3.3.0-rc3, and it works like a charm! >> >> but there are lots of issues with external packages not compiling with 3.3, >> why i wont be able to deploy it to our systems here yet. >> >> could this one bugfix be backported to 3.2? > > All of the btrfs patches in 3.3-rc are against 3.2 in my git tree: > > git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linusthank you very much, it works as it should. had to couse some traffic on kernel.org but finally i got a local linux repo! is for-linus considered stable and can i pull regularly? here what i did for those interested: cd /usr/src/ rm linux git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux cd linux git checkout linux-3.2.y git pull git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus cp /boot/config .config make menuconfig # look around, save and exit make -j8 make modules_install make install and thats it, 3.2.6 stable kernel from git with latest for-linus btrfs from git Ahmet -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html