I''ve been trying to run PostgreSQL with a ~20-30 GB database on a btrfs filesystem (rest of the filesystem filled with KVM images, compressed archives etc.). Snapshots are cool, and let you rollback to a different database version easily ;) Unfortunately, running PostgreSQL on btrfs wasn''t very reliable as I was getting rather frequent "No space left" for some disk intensive operations (CREATE INDEX, REINDEX, VACUUM ANALYZE). As the error happens, I still have around 80 GB free space on the filesystem: Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 255G 80G 77% /mnt/btrfs What''s even more interesting, after the error happened, I''m able to write a ~85 GB file with dd before it exits with "out of space" (with and without the oflag=direct): # dd if=/dev/urandom of=bigfile bs=1M dd: writing `bigfile'': No space left on device 81056+0 records in 81055+0 records out 84992327680 bytes (85 GB) copied, 9429.28 s, 9.0 MB/s The kernel used was 2.6.39-rc5. If it matters, I had the filesystem mounted with compress-force flag: /dev/sdb4 on /mnt/btrfs type btrfs (rw,noatime,compress-force) -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 16.05.2011 20:08, Tomasz Chmielewski wrote:> The kernel used was 2.6.39-rc5. > > If it matters, I had the filesystem mounted with compress-force flag:Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left? # df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 257G 78G 77% /mnt/btrfs coredb=# DELETE FROM core_trevent where "date" < E''2011-05-07 00:00''; ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device CONTEXT: writing block 1014999 of relation base/16384/16521 -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:> > Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left? > > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sdb4 336G 257G 78G 77% /mnt/btrfs > > > coredb=# DELETE FROM core_trevent where "date" < E''2011-05-07 00:00''; > ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device > CONTEXT: writing block 1014999 of relation base/16384/16521 >Hi Tomasz, What does btrfs fi df /mountpoint show? Please see this link: https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21 As well as this one: https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_to_check_the_amount_of_free_space.3F -- MG -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 21.05.2011 01:05, Miguel Garrido wrote:> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org> wrote: >> >> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left? >> >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb4 336G 257G 78G 77% /mnt/btrfs >> >> >> coredb=# DELETE FROM core_trevent where "date"< E''2011-05-07 00:00''; >> ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device >> CONTEXT: writing block 1014999 of relation base/16384/16521 >> > > Hi Tomasz, > > What does btrfs fi df /mountpoint show?Used space is almost identical as with normal df.> Please see this link: > https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21Doesn''t apply in my case: kernel is 2.6.39, device is relatively large. It is *probably* related to compression used (compress-force - disabled the option, and didn''t see out of space since). -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 21.05.2011 01:05, Miguel Garrido wrote:> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org> wrote: >> >> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left? >> >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb4 336G 257G 78G 77% /mnt/btrfs >> >> >> coredb=# DELETE FROM core_trevent where "date"< E''2011-05-07 00:00''; >> ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device >> CONTEXT: writing block 1014999 of relation base/16384/16521Hmm, this "No space left on device" only shows up if I write to the btrfs filesystem relatively fast. For example, this way, I can only write around 1 GB (~60 MB/s write): # cp -v acer.img.xz /mnt/btrfs/ `ac.img.xz'' -> `/mnt/btrfs/ac.img.xz'' cp: writing `/mnt/btrfs/ac.img.xz'': No space left on device This way, when the writes are CPU-bound, I''m able to fully fill the btrfs filesystem (~8 MB/s): # dd if=/dev/urandom of=/mnt/btrfs/bigfile Delayed allocation? -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 21.05.2011 01:05, Miguel Garrido wrote:> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org> wrote: >> >> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left? >> >> >> # df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb4 336G 257G 78G 77% /mnt/btrfs >> >> >> coredb=# DELETE FROM core_trevent where "date"< E''2011-05-07 00:00''; >> ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device >> CONTEXT: writing block 1014999 of relation base/16384/16521Just a follow up - I''m able to reproduce the issue with 3.0.1 kernel, with plain rsync. I''m trying to write a ~ 4 GB file to btrfs. "Fast" writes will make btrfs output "No space left on device". On the other hand, if I write slowly, btrfs will be happy and write the whole file. cp ("full speed copy") fails after a few seconds, so let''s try to write with ~5 MB/s - still fails: # rsync -a -v --progress --bwlimit=5000 . /mnt/btrfs/temp sending incremental file list ./ randomfile 645136384 13% 4.87MB/s 0:13:32 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "/mnt/btrfs/temp/randomfile": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7] rsync: connection unexpectedly closed (31 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7] Let''s write with ~1 MB/s then - succeeds: # rsync -a -v --progress --bwlimit=1000 . /mnt/btrfs/temp sending incremental file list ./ randomfile 4693411469 100% 988.30kB/s 1:17:17 (xfer#1, to-check=6/8) (...) sent 4758749321 bytes received 112 bytes 1012176.84 bytes/sec total size is 4758167972 speedup is 1.00 I don''t think it''s expected? I see these in dmesg while copying the files: [ 1703.978600] btrfs: free space inode generation (0) did not match free space cache generation (22496) [ 1703.978605] btrfs: failed to load free space cache for block group 92371156992 [ 1704.026476] btrfs: free space inode generation (0) did not match free space cache generation (22502) [ 1704.026481] btrfs: failed to load free space cache for block group 114919735296 [ 1704.035239] btrfs: free space inode generation (0) did not match free space cache generation (22503) [ 1704.035243] btrfs: failed to load free space cache for block group 118140960768 [ 1704.035252] btrfs: free space inode generation (0) did not match free space cache generation (22481) [ 1704.035255] btrfs: failed to load free space cache for block group 119214702592 [ 1704.294757] btrfs: free space inode generation (0) did not match free space cache generation (22503) [ 1704.294761] btrfs: failed to load free space cache for block group 243231883264 -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html