I''ve been using btrfs for two months now. Every day between 02:00 and 08:00 I rsync some 300GB data (milions of files) to btrfs device and then make snapshot. Next day i rsync again 300GG little changed (rsync "in place"). First days it worked perfectly. Then loadavg (sys load) started to rise. Now, after 60 days of rsync+shapshots sys load is so high that whole server becames quite unresponsive betweend 02:00 and 08:00. I never deleted any shapshot (yet) to its not rebalancing thing. What all those btrfs benchmark does not tell you that its performance decreases (sys load increases) with growing size of btree. Creating btrfs filesystem is instantaneous because initial tree is just nothing... I tried using btrfs and nilfs2 for a half year now and I would never trust a byte of production data to it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 23 Nov 2011 15:43:14 +0100 "krzf83@gmail.com " <krzf83@gmail.com> wrote:> I''ve been using btrfs for two months now. Every day between 02:00 and > 08:00 I rsync some 300GB data (milions of files) to btrfs device and > then make snapshot. Next day i rsync again 300GG little changed (rsync > "in place"). First days it worked perfectly. Then loadavg (sys load) > started to rise. Now, after 60 days of rsync+shapshots sys load is so > high that whole server becames quite unresponsive betweend 02:00 and > 08:00. I never deleted any shapshot (yet) to its not rebalancing > thing.Hello, I employ the same scenario, except that my rsync-(actually mirrordir-) destination btrfs device is only mounted briefly for the copy-then-snapshot operation, and unmounted during the rest of the time. Haven''t noticed any performance problems yet, but even though I have about 2 TB of data there, it''s probably not millions of files. Have you tried unmounting the FS and then mounting it again, to see if it solves the performance problem (at least for a while)? Or just keeping it mostly unmounted, I suppose what you have is a partition with backups, so keeping it mostly offline is better reliability-wise. -- With respect, Roman ~~~~~~~~~~~~~~~~~~~~~~~~~~~ "Stallman had a printer, with code he could not see. So he began to tinker, and set the software free."
Yes, yesterday I''ve umouted partition and re-mounted it. Nothing has changed this night. You can look at my load graphs here: http://img42.imageshack.us/img42/4661/33737291.png http://img210.imageshack.us/img210/2742/46527625.png On the second one blue is SYS load. I bet you can reasily spot time when backup to btrfs partition was running :) I doubt its a bug in btrfs. It is just broken concept :( # uname -a Linux sv15.vipserv.org 3.0.4-xxxx-std-ipv6-64 #1 SMP Mon Oct 10 01:07:40 CEST 2011 x86_64 x86_64 x86_64 GNU/Linux -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I see that this high sys load and loadavg did not appear gradualy. It appeared first time about 7 days ago and were present every backup, every night since then. During backup top shows higher cpu load on 4216 root 20 0 0 0 0 R 20.3 0.0 284:25.83 btrfs-delayed-m 4222 root 20 0 0 0 0 D 19.6 0.0 203:23.89 btrfs-transacti however one cannot really debug sys load so it does not tell much. Scenario here is quite simple (rsync to working dir and then snapshot, next night rsync (in place) to working dir and again snapshot). Rsync is killed at 08:00. I will try rebooting server this night but I doubt it will change anything on those high loads during backup to btrfs partition. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 11/23/2011 9:43 AM, krzf83@gmail.com wrote:> What all those btrfs benchmark does not tell you that its performance > decreases (sys load increases) with growing size of btree. Creating > btrfs filesystem is instantaneous because initial tree is just > nothing...While something is clearly wrong, this isn''t it. Each snapshot is its own btree, and you said there is little churn each day, so they aren''t getting significantly larger over time. Each snapshot is tracked in the tree of tree roots, so technically it is growing each time you take a snapshot, but 60 items is nothing. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I''ve rebooted server and run backup to btrfs partition again. It seems that problem is gone, high sys load does not occur now. So it is some bug in btrfs... Before reboot server had 30 days uptime so its really not much. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 24/11/11 02:08, krzf83@gmail.com wrote:> Linux sv15.vipserv.org 3.0.4-xxxx-std-ipv6-64 #1 SMP Mon Oct 10 > 01:07:40 CEST 2011 x86_64 x86_64 x86_64 GNU/LinuxYou''re running quite an old kernel btrfs wise - it would be very interesting to see if you have the same issue with Linus''s current git version (which has the latest btrfs code) or wait for 3.2-rc3. Another possibility I *think* is that you could try 3.1 with Chris Mason''s for-linus git branch pulled into it. Hopefully someone who knows the procedure better than I can correct me on this! :-) There''s a lot of work gone on both in fixing up corruption issues on power loss and also reducing a lot of unnecessary I/O that was happening due to bugs since that release. cheers! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Nov 24, 2011 at 8:00 AM, Chris Samuel <chris@csamuel.org> wrote:> Another possibility I *think* is that you could try 3.1 with > Chris Mason''s for-linus git branch pulled into it. Hopefully > someone who knows the procedure better than I can correct me > on this! :-)My method is: - use 3.1.1 (latest stable at the time I was compiling it), compiie btrfs as module - use only fs/btrfs directory from Chris'' for-linus tree, compile the module externally (make -C /lib/modules/`pwd`/build M=$(pwd) modules) - put the resulting btrfs.ko on /lib/modules/`uname -r`/updates/manual/ - depmod -a - verify the new module is selected by default (modinfo btrfs) - rebuild initrd - reboot -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 24/11/11 12:00, Chris Samuel wrote:> You''re running quite an old kernel btrfs wise - it would be > very interesting to see if you have the same issue with Linus''s > current git version (which has the latest btrfs code) or > wait for 3.2-rc3.3.2-rc3 is now out, Linus is after testers around the world because of some holiday in the US. cheers! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I''ve upgraded to newest kernel 3.1.2 but problem is not fixed. After reboot btrfs works fine to maybe few hours but after that system stats to choke on sys load, system becames unresponsive, and backups to btrfs becames extremly slow. Anything you want me to check so you can diagnose the problem - before I will return to ext4? blue on graph is sys load. It should not be visible at all. Something is seriously wrong with btrfs http://img265.imageshack.us/img265/1416/37248826.png I use snapshots for every important directory (after each backup) so there quite lot of them but this is a reason why I use btrfs. # btrfs sub list /backup|wc -l 7651 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 02/12/11 15:14, krzf83@gmail.com wrote:> I''ve upgraded to newest kernel 3.1.2 but problem is not fixed.As btrfs is an experimental filesystem I don''t believe that fixes are backported to those stable releases, they only go into Linus''s tree for his next release (3.2 in this case). So I''m afraid if you want to pick up fixes as they appear you''ll need to track Linus''s tree really (or one of Chris''s). I guess once btrfs is stable then that''ll start to happen, but for the moment I get the feeling that fixes and features are so tightly entangled that it would be an awful lot of work to backport them to the older code in the last stable release. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html