Hello, I have 3 partitions with btrfs(/, /home and /data). All of them have following mount options noatime,space_cache,inode_cache,compress=lzo,defaults Whenever there is a unclean shutdown(which happens a lot in my case), the next reboot, system comes up relatively at the same speed but as systemd is starting up daemons, the disk is continuously ( and unusally long) grinding. This causes random delays with various daemons such as postgresql failing to start, kdm timing out of xorg servers etc and I have to reboot after the dust settles to bring back the system to the normal. At one time, even keyboad/mouse were not responding as some debus service timed out.. I think it is rebuilding the space cache because I saw similar long disk activity when I activated it first. How can I confirm that it is the space cache rebuild thats taking time? if space cache rebuild is the reason, is there any way to improve it? I am running archlinux/systemd/kde setup with two 7200 RPM seagate sata disks(no RAID, one 80 GB for / and /home, other 500GB for data). The kernel is 3.9.8 x86_64. Thanks. -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 6-30-13 19:26:16 Shridhar Daithankar wrote:> Whenever there is a unclean shutdown(which happens a lot in my > case), the next reboot, system comes up relatively at the same speed > but as systemd is starting up daemons, the disk is continuously (and > unusally long) grinding.[snip]> How can I confirm that it is the space cache rebuild thats taking > time? > > if space cache rebuild is the reason, is there any way to improve > it? > > I am running archlinux/systemd/kdeI suspect this is, at least in part, related to severe fragmentation in /home. There are large files in these directories that are updated frequently by various components of KDE and the Chrome browser. (Firefox has its own databases that are frequently updated, too.) ~/.local/share/akonadi ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend ~/.cache/chromium/Default/Cache ~/.cache/chromium/Default/Media\ Cache I improved performance dramatically (orders of magnitude) by copying the database files into an empty file that was modified with: chattr -C and renaming to make the files no COW. (Note that this is the only way to change an existing file to no COW.) I also set the same attribute on the owning directories so that all new files inherit the no COW attribute. I suspect there are other files that fragment badly since I see periods of high disk activity coming back slowly over a few weeks of use after making the modifications above. I intend to track them down and do the same. Also, see these: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work https://btrfs.wiki.kernel.org/index.php/UseCases#How_do_I_defragment_many_files.3F $ uname -r 3.9.6-200.fc18.x86_64 $ -- Garry T. Williams -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/30/2013 06:53 PM, Garry T. Williams wrote:> On 6-30-13 19:26:16 Shridhar Daithankar wrote: >> Whenever there is a unclean shutdown(which happens a lot in my >> case), the next reboot, system comes up relatively at the same speed >> but as systemd is starting up daemons, the disk is continuously (and >> unusally long) grinding. >>> I am running archlinux/systemd/kde > > I suspect this is, at least in part, related to severe fragmentation > in /home. >I''m wondering if this is affecting myself. I have a big issue with my data drive slowing down and there being near long periods of high disk IO that prevent me doing anything else. I''ve noticed from iotop various btrfs processes hogging the IO for long periods, e.g. btrfs-transacti... & btrfs-submit I''ve been running kde which has got unusable (not from reboot, but in general). xfce is less hampered but IO still seems like an issue at times. Of course, xfce hits different files. I''ve been using this file system a couple of months and not defragged before. I started defragging the various subvolumes a week or two ago - but I did not realise this was not recursive until this weekend. I''ve got a python script running defrag on various files and folders - I can better track what it is defragging. But it is _slow_ many many minutes for a rarely accessed folder with little content. Is this normal? I too had an issue with unclean shutdowns. I, relatively infrequently, get lockups. However, I had a spate last week which I have yet to resolve. I wonder if that is related. I wonder, if I defrag everything on say a weekly basis then will these performance issues go away? Running a 3.9.3 kernel. Pete> There are large files in these directories that are updated frequently > by various components of KDE and the Chrome browser. (Firefox has its > own databases that are frequently updated, too.) > > ~/.local/share/akonadi > ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend > ~/.cache/chromium/Default/Cache > ~/.cache/chromium/Default/Media\ Cache > > I improved performance dramatically (orders of magnitude) by copying > the database files into an empty file that was modified with: > > chattr -C > > and renaming to make the files no COW. (Note that this is the only > way to change an existing file to no COW.) I also set the same > attribute on the owning directories so that all new files inherit the > no COW attribute. > > I suspect there are other files that fragment badly since I see > periods of high disk activity coming back slowly over a few weeks of > use after making the modifications above. I intend to track them down > and do the same. > > Also, see these: > > https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_directory_doesn.27t_work > https://btrfs.wiki.kernel.org/index.php/UseCases#How_do_I_defragment_many_files.3F > > $ uname -r > 3.9.6-200.fc18.x86_64 > $ >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,> I suspect this is, at least in part, related to severe fragmentation > in /home.In his cause those issues are only present after an unclean shutdown - whereas fragmentation would show its effect after every reboot. Regards, Clemens -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Garry T. Williams posted on Sun, 30 Jun 2013 13:53:48 -0400 as excerpted:> I suspect this is, at least in part, related to severe fragmentation in > /home. > > There are large files in these directories that are updated frequently > by various components of KDE and the Chrome browser. (Firefox has its > own databases that are frequently updated, too.) > > ~/.local/share/akonadi > ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend > ~/.cache/chromium/Default/Cache ~/.cache/chromium/Default/Media\ > Cache > > I improved performance dramatically (orders of magnitude) by copying the > database files into an empty file that was modified with: > > chattr -C > > and renaming to make the files no COW. (Note that this is the only way > to change an existing file to no COW.) I also set the same attribute on > the owning directories so that all new files inherit the no COW > attribute. > > I suspect there are other files that fragment badly since I see periods > of high disk activity coming back slowly over a few weeks of use after > making the modifications above. I intend to track them down and do the > same.This definitely won''t be practical for everyone, but... 1) I run kde here, but switched away from kmail, akregator, basically anything kdepim related, when that akonadified. I had been using kmail for nearly a decade, and it had converted MSOE mail in it from before the turn of the century (!!), but one day when akonadi simply lost an email for the Nth time in so many days, the question occurred to me, why do I put up with this when there''s so many sane alternatives? Yes, I could have probably recovered that mail as I had others by redoing the akonadi resources or whatever, but the question was, why should I *HAVE* to, again, when there''s all sorts of saner alternatives that, like kmail before the akonadi insanity, didn''t lose mail in the first place? So I switched, choosing claws-mail to replace both kmail and akregator here, FWIW, but there''s other alternatives for those who don''t like claws- mail. And when I switched that off, I began wondering about semantic-desktop at all, even tho it was run-time switched off. So being a gentooer who had the option, I set USE=-semantic-desktop and a few other flags and rebuilt the affected bits of kde. Now no more semantic-desktop AT ALL! =:^) (Unfortunately, the gentoo/kde folks decided to kill the option and hard- enable semantic-desktop for the coming 4.11, which I''m running the betas of ATM, but using the diffs between the 4.10 ebuilds with the option and 4.11 builds without, I was able to patch out the the support, so now run with semantic-desktop build-time hard-disabled (instead of the normal gentoo 4.11 hard-enabled) here. So no gigabytes of nepomuk and akonadi files doing nothing but create problems for me, here! I do run firefox, but haven''t seen a problem with it, either, possibly due to #2... 2) Throw hardware at the problem. About a month ago I finally bit the financial bullet and upgraded to (mostly) SSD. My media partition and backups are still on spinning rust (on reiserfs since btrfs is still experimental), but the main system and /home are now on dual (fairly fast, Corsair Neutron) SSD, on btrfs in raid1 mode (both data/metadata). That''s actually why I''m running btrfs here, as my old standby, reiserfs, while highly reliable on spinning rust (yes, I know the rumors, but after a decade on reiserfs on spinning rust, it has been /extremely/ reliable for me, at least it has been since the data=ordered switch in kernel 2.6.16 IIRC, even thru various hardware issues!), isn''t particularly appropriate for SSD. So I run btrfs, which detects my SSDs and activates SSD mode automatically, here. I use dual SSDs in btrfs raid1 mode, in ordered to take advantage of btrfs data integrity with the checksumming. And with the SSDs, there''s no mechanical seek latency and the IOPS are high enough that, at least with the btrfs autodefrag mount option activated from the beginning, I''ve seen no noticeable fragmentation slowdowns either. (It may also help that I''m close to 100% overprovisioned on the SSDs as my effectively write-once-read-many media and backups remain on reiserfs on the spinning rust, so even without the trim option, the SSDs have plenty of room to do their write-management.) What I''ve noticed most is that the stuff that small, often written files that caused fragmentation issues on reiserfs on spinning rust, generally aren''t an issue at all on btrfs on SSD. This includes my main gentoo package tree and overlays, my git kernel checkout, and pan''s nntp message cache (by default 10 MB with two-week header expiry, but I''m running nearly a gig of text messages with no expiry, with messages on some gmane mailing-list groups I follow going back to 2002). All these are an order of magnitude at least faster on btrfs on the SSDs than they were on reiserfs on spinning rust, without the slowdowns I saw due to fragmentation on spinning rust, either. So that has been my answer to the fragmentation issue. The space-cache issue of the OP may be different. I had some problems with that and unclean shutdown when I first experimented with btrfs a bit over a year ago (still on spinning rust at the time), but I decided it wasn''t ready for me then and waited a year, and I haven''t had problems this go-round. I''ve only had a couple unclean shutdowns, however, but the system did seem to come right back up afterward. The SSDs may be playing some role in that too, tho. But I''ve really not had enough unclean shutdowns to be sure, yet. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30/06/13 10:53, Garry T. Williams wrote:> ~/.cache/chromium/Default/Cache ~/.cache/chromium/Default/Media\ CacheI''ve taken to making ~/.cache be tmpfs and all the apps have been fine with that. It also meant I didn''t have to worry about my btrfs snapshots being full of transient web junk. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlHQu0wACgkQmOOfHg372QRTMACg1YQx1B6liiLnVpOZLxnoHC+W 5ewAn1Z40V/52dongHBpg6OUdprUVqwo =601F -----END PGP SIGNATURE----- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sunday, June 30, 2013 01:53:48 PM Garry T. Williams wrote:> I suspect this is, at least in part, related to severe fragmentation > in /home.I don''t think so. The problem I have described occur only before anybody logs in to the system and /home being a separate partition, it is not the problem in this case. w> > There are large files in these directories that are updated frequently > by various components of KDE and the Chrome browser. (Firefox has its > own databases that are frequently updated, too.) > > ~/.local/share/akonadiThats 3.9MB in my case since I point the akonadi db to a systemwide postgresql instance. of course, it will just shift defragmentation there.> ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackenddamn! # filefrag soprano-virtuoso.db soprano-virtuoso.db: 10518 extents found # btrfs fi defrag soprano-virtuoso.db # filefrag soprano-virtuoso.db soprano-virtuoso.db: 957 extents found How much is an extend anyways? is it a page or 256M?> ~/.cache/chromium/Default/Cache > ~/.cache/chromium/Default/Media\ CacheI don''t use chromium. but I get the idea. But in general, how to find out most fragmented files and folders? mouting with autodefrag is a serious degradation..> I improved performance dramatically (orders of magnitude) by copying > the database files into an empty file that was modified with: > > chattr -C > > and renaming to make the files no COW. (Note that this is the only > way to change an existing file to no COW.) I also set the same > attribute on the owning directories so that all new files inherit the > no COW attribute. > > I suspect there are other files that fragment badly since I see > periods of high disk activity coming back slowly over a few weeks of > use after making the modifications above. I intend to track them down > and do the same.hmm.. a trick to find most badly fragmented files/directories and defragment them should do too I think. -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Shridhar Daithankar posted on Mon, 01 Jul 2013 08:20:19 +0530 as excerpted:> On Sunday, June 30, 2013 01:53:48 PM Garry T. Williams wrote:[discussing fragmentation]> >> ~/.kde/share/apps/nepomuk/repository/main/data/virtuosobackend > > damn! > > # filefrag soprano-virtuoso.db soprano-virtuoso.db: 10518 extents found > > # btrfs fi defrag soprano-virtuoso.db > > # filefrag soprano-virtuoso.db soprano-virtuoso.db: 957 extents foundWhile you evidently had quite some fragmentation as the number of extents dropped considerably, if you''re running btrfs compression, it''s worth noting that (based on earlier posts here) filefrag always counts compressed files as fragmented, even if they''re not. So a sufficiently sized file will almost certainly show fragmentation via filefrag if it''s compressed, and all you can do is use filefrag as a hint in that case; defrag may well not do anything if it''s not actually fragmented.> How much is an extend anyways? is it a page or 256M?I don''t know...> But in general, how to find out most fragmented files and folders? > mouting with autodefrag is a serious degradation..It is? AFAIK, all the autodefrag mount option does is scan files for fragmentation as they are written and queue any fragmentation-detected files for background defrag by the defrag thread. I had expected, particularly on spinning rust, that the benefits of autodefrag to far exceed the costs, so your performance drag claim is interesting to me indeed. If my expectation is wrong, which it could be, I''d love to know why, and see some numbers. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Monday, July 01, 2013 09:10:41 AM Duncan wrote:> > But in general, how to find out most fragmented files and folders? > > mouting with autodefrag is a serious degradation.. > > It is? AFAIK, all the autodefrag mount option does is scan files for > fragmentation as they are written and queue any fragmentation-detected > files for background defrag by the defrag thread. > > I had expected, particularly on spinning rust, that the benefits of > autodefrag to far exceed the costs, so your performance drag claim is > interesting to me indeed. If my expectation is wrong, which it could be, > I''d love to know why, and see some numbers.while I don''t have numbers, I enabled autodefrag on all the partitions and rebooted(twice, just to confirm) and its slow.. everything has a 10 second tail of disk activity and has quite some visible latency. Moving mouse, switching windows, starting new programs, everything has visible latency thats unusable. It seems autodefrag is being too aggressive for its own good.. I am sticking with defragging folders individually. /var, /home and a 1GB squid cache is what I have narrowed down and things are reasonably fast. -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Shridhar Daithankar posted on Mon, 01 Jul 2013 21:49:16 +0530 as excerpted:> On Monday, July 01, 2013 09:10:41 AM Duncan wrote: >> >>> mouting with autodefrag is a serious degradation.. >> >> It is? AFAIK, all the autodefrag mount option does is scan files for >> fragmentation as they are written and queue any fragmentation-detected >> files for background defrag by the defrag thread. >> >> I had expected, particularly on spinning rust, that the benefits of >> autodefrag to far exceed the costs, so your performance drag claim is >> interesting to me indeed. If my expectation is wrong, which it could >> be, I''d love to know why, and see some numbers. > > while I don''t have numbers, I enabled autodefrag on all the partitions > and rebooted(twice, just to confirm) and its slow.. > > everything has a 10 second tail of disk activity and has quite some > visible latency. Moving mouse, switching windows, starting new programs, > everything has visible latency thats unusable. > > It seems autodefrag is being too aggressive for its own good..Just to be clear, your system, your call. I''d never /dream/ of interfering with that due to the implications for my own system (which is certainly highly customized even matched against a peer-group of other gentoo installs =:^). That said... I''m guessing what you experiences with the autodefrag mount option was because you were not in stable-state yet. The original btrfs filesystem setup and fill was very likely without the flag on[2], so there''s quite a lot of existing fragmentation what would have to be worked thru before the filesystem gets defragged and you reach stable-state, at which I''d expect the autodefrag mount option to have little overhead. Tho if what you''re saying is correct[1] then it may be that the background defrag thread isn''t (io-)niced as I would have expected it to be. But I''d still expect there to be some better performance steady state after a few mounts gets the basic filesystem defragged. Tho if the fileystem is heavily fragmented[2], in practice it may well be easier to backup the filesystem content, do a clean mkfs and mount with autodefrag and restore from backup, thus ensuring autodefrag is on while filling the filesystem in the first place, than to wait for autodefrag to reach a stable system state in normal operation over many mounts. All that stated, you''ve definitely demonstrated that I hadn''t put enough thought into my initial general-case assumptions, which now come with far more qualifiers than they did before this subthread. Thanks. =:^) [1] I simply don''t know from personal experience as I (1) ensured I enabled autodefrag on the empty filesystem before I started filling it, and (2) I''m on fast ssd, an entirely different world from spinning rust. [2] Various comments I''ve read seem to hint that, surprisingly, certain distro installers leave a brand new install in a heavily fragmented state, as they apparently install without autodefrag on during the install and also apparently do heavy rewriting into existing files thereby triggering heavy fragmentation without the flag during that install. No, I''ve not bothered to track which distros, I simply ensured autodefrag was on, here, before filling my filesystems in the first place. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tuesday, July 02, 2013 01:00:29 PM Duncan wrote:> Just to be clear, your system, your call. I''d never /dream/ of > interfering with that due to the implications for my own system (which is > certainly highly customized even matched against a peer-group of other > gentoo installs =:^). That said... > > I''m guessing what you experiences with the autodefrag mount option was > because you were not in stable-state yet. The original btrfs filesystem > setup and fill was very likely without the flag on[2], so there''s quite a > lot of existing fragmentation what would have to be worked thru before > the filesystem gets defragged and you reach stable-state, at which I''d > expect the autodefrag mount option to have little overhead.Yes, I suspect as much. One of the data parition I have is over an year old and never defragged, /home about 3 months old. and I can see the defragmentation working when run by hand. The initial run on /var(only directories, to defrag metadata) was 6 min. Now that I am running it daily by hand, entire /var(including files, that includes 1.8G pacman cache, few tiny postgresql databases and /var/tmp where kde generates tons of IO) takes about 6-7 min. Despite of my experience with autodefrag, I want to use it because thats the best solution(kinda like autovacuum in postgresql) that keeps things clean, all the time.> > Tho if what you''re saying is correct[1] then it may be that the > background defrag thread isn''t (io-)niced as I would have expected it to > be. > > But I''d still expect there to be some better performance steady state > after a few mounts gets the basic filesystem defragged. Tho if the > fileystem is heavily fragmented[2], in practice it may well be easier to > backup the filesystem content, do a clean mkfs and mount with autodefrag > and restore from backup, thus ensuring autodefrag is on while filling the > filesystem in the first place, than to wait for autodefrag to reach a > stable system state in normal operation over many mounts.well, I think I will bite the bullet and defrag entire / overnight and repeat the autodefrag mount option. That should work too. Thanks -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tuesday, July 02, 2013 09:19:07 PM Shridhar Daithankar wrote:> On Tuesday, July 02, 2013 01:00:29 PM Duncan wrote: > > But I''d still expect there to be some better performance steady state > > after a few mounts gets the basic filesystem defragged. Tho if the > > fileystem is heavily fragmented[2], in practice it may well be easier to > > backup the filesystem content, do a clean mkfs and mount with autodefrag > > and restore from backup, thus ensuring autodefrag is on while filling the > > filesystem in the first place, than to wait for autodefrag to reach a > > stable system state in normal operation over many mounts. > > well, I think I will bite the bullet and defrag entire / overnight and > repeat the autodefrag mount option. That should work too.and that worked. defragged all the mount points including files and dirs, enabled autodefrag and reboot. Took about 2 hour to defrag the existing files. but the filesystem is now extremely smooth. faster than ext4 I might say. sure there are occasional stalls but they are more of noticable than annoyance and thats pretty much compensated by the significant improvement in latency in everything. fun fact, pg_test_fsync now reports 44 fsyncs per seconds instead of earlier 20. I don''t know if that down to defragmentation or compression. Also another bigger disk of 500GB, the score is around 24 fsyncs per second. So I suspect it has to do with tree size. anyways, good things overall thanks for the help and suggestions. -- Regards Shridhar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html