Hi all! My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow. There are many files and many links on the FS. # btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TB System, DUP: total=8.00MB, used=496.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=154.88GB, used=70.56GB Metadata: total=8.00MB, used=0.00 Maybe it''s because there is so much Metadata and it needs so many seeks on the discs when deleting? I''d like to delete some of the old Files but its so horrible slow that i think its maybe faster to copy all needed Data to a different Disc, killing the FS and move the Files back... The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus". Any hints how i could speed it up? Tobias -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/16/2011 02:49 AM, Tobias wrote:> Hi all! > > My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and > deleting is horrible slow. > > There are many files and many links on the FS. > > # btrfs filesystem df /srv/storage > Data: total=3.09TB, used=3.07TB > System, DUP: total=8.00MB, used=496.00KB > System: total=4.00MB, used=0.00 > Metadata, DUP: total=154.88GB, used=70.56GB > Metadata: total=8.00MB, used=0.00 > > Maybe it''s because there is so much Metadata and it needs so many seeks > on the discs when deleting? > > I''d like to delete some of the old Files but its so horrible slow that i > think its maybe faster to copy all needed Data to a different Disc, > killing the FS and move the Files back... > > The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus". > > Any hints how i could speed it up? >I recommend to run defragment on your files, it should help you, although defrag maybe slow, too. thanks, liubo> Tobias > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Dec 16, 2011 at 1:49 AM, Tobias <tracer@robotech.de> wrote:> Hi all! > > My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and > deleting is horrible slow. > > There are many files and many links on the FS. > > # btrfs filesystem df /srv/storage > Data: total=3.09TB, used=3.07TBthis is ... what, over 99% full? The slow down is normal, somewhat. Same thing happens on zfs, which became slower when usage is above 80-90%.> Maybe it''s because there is so much Metadata and it needs so many seeks on > the discs when deleting?I doubt it.> I''d like to delete some of the old Files but its so horrible slow that i > think its maybe faster to copy all needed Data to a different Disc, killing > the FS and move the Files back... > > The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus". > > Any hints how i could speed it up?Try: - mounting it with nodatacow: https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F - clobbering a big file: https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29 ... until you have at least 20% free space available. -- Fajar -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha <list@fajar.net> wrote:> On Fri, Dec 16, 2011 at 1:49 AM, Tobias <tracer@robotech.de> wrote: >> Hi all! >> >> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and >> deleting is horrible slow. >> >> There are many files and many links on the FS. >> >> # btrfs filesystem df /srv/storage >> Data: total=3.09TB, used=3.07TB > > this is ... what, over 99% full? > The slow down is normal, somewhat. Same thing happens on zfs, which > became slower when usage is above 80-90%.I don''t think "total" actually means "total space available" it increases as you use up more space.> >> Maybe it''s because there is so much Metadata and it needs so many seeks on >> the discs when deleting? > > I doubt it. > >> I''d like to delete some of the old Files but its so horrible slow that i >> think its maybe faster to copy all needed Data to a different Disc, killing >> the FS and move the Files back... >> >> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus". >> >> Any hints how i could speed it up? > > Try: > - mounting it with nodatacow: > https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F > - clobbering a big file: > https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29 > > ... until you have at least 20% free space available. > > -- > Fajar > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/16/2011 09:14 AM, Chester wrote:> On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha<list@fajar.net> wrote: >> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de> wrote: >>> Hi all! >>> >>> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and >>> deleting is horrible slow. >>> >>> There are many files and many links on the FS. >>> >>> # btrfs filesystem df /srv/storage >>> Data: total=3.09TB, used=3.07TB >> this is ... what, over 99% full? >> The slow down is normal, somewhat. Same thing happens on zfs, which >> became slower when usage is above 80-90%. > I don''t think "total" actually means "total space available" it > increases as you use up more space. >>> Maybe it''s because there is so much Metadata and it needs so many seeks on >>> the discs when deleting? >> I doubt it. >> >>> I''d like to delete some of the old Files but its so horrible slow that i >>> think its maybe faster to copy all needed Data to a different Disc, killing >>> the FS and move the Files back... >>> >>> The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus". >>> >>> Any hints how i could speed it up? >> Try: >> - mounting it with nodatacow: >> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F >> - clobbering a big file: >> https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29 >> >> ... until you have at least 20% free space available. >> >> -- >> Fajar >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.htmlWouldn''t it be more valuable to first collect enough info/data to debug the problem, before suggesting ways to get out of this situation? AFAIK, the slowness could be due to several reasons: 1. Slow disk writes. 2. Too many things to commit to the disk. 3. Too many things to do to actually accomplish the writes. 4. Important threads suspending due to lack of resources. 5. Something else which I''ve missed above. Is there some sort of profiling which we can enable, which can give us info about the speeds and quantities of read and write traffic on the fs? Aren''t there btrfs statistics that we can print out, which can give us these info? If not, we probably should think about adding some. P.S: btrfs n00b talking! Regards, Shyam -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 16.12.2011 03:09, schrieb Liu Bo:> On 12/16/2011 02:49 AM, Tobias wrote: >> Any hints how i could speed it up? >> > > I recommend to run defragment on your files, > it should help you, although defrag maybe slow, too. >When deleting files does it matter if the data is fragmented? When i do normal file-access the speed of reading is ok so until not i didn''t care about fragmentation. Especially because the data isn''t changes after the first write. Can i do a defrag on the metadata only? Tobias -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 16.12.2011 05:15, schrieb Shyam Prasad N:> On 12/16/2011 09:14 AM, Chester wrote: >> On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha<list@fajar.net> >> wrote: >>> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de> wrote: >>>> Hi all! >>>> >>>> My BTRFS-FS ist getting really slow. Reading is ok, writing is slow >>>> and >>>> deleting is horrible slow. >>>> >>>> There are many files and many links on the FS. >>>> >>>> # btrfs filesystem df /srv/storage >>>> Data: total=3.09TB, used=3.07TB >>> this is ... what, over 99% full? >>> The slow down is normal, somewhat. Same thing happens on zfs, which >>> became slower when usage is above 80-90%. >> I don''t think "total" actually means "total space available" it >> increases as you use up more space.Chester is right, the disk is a 8TB Array - the fs is less than half filled.> Wouldn''t it be more valuable to first collect enough info/data to > debug the problem, before suggesting ways to get out of this situation? > > AFAIK, the slowness could be due to several reasons: > 1. Slow disk writes.That depends on what is slow for you. Its a SATA-Disc-Array so good linear Read/Write and low IOPs (compared to SSDs or SAS-Discs)> 2. Too many things to commit to the disk.I think there should be very little data to write/commit when deleting files. Remember: writing is slow, but acceptable - deleting is the problem> 3. Too many things to do to actually accomplish the writes.I just do rm -rf xxx. What has BTRFS to do to actually delete a file or directory?> Is there some sort of profiling which we can enable, which can give us > info about the speeds and quantities of read and write traffic on the fs? > Aren''t there btrfs statistics that we can print out, which can give us > these info? If not, we probably should think about adding some.Good question - maybe a dev can say? Tobias -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Dec 16, 2011 at 09:51:56AM +0100, Tobias wrote:> Am 16.12.2011 03:09, schrieb Liu Bo: > >On 12/16/2011 02:49 AM, Tobias wrote: > >>Any hints how i could speed it up? > >> > > > >I recommend to run defragment on your files, > >it should help you, although defrag maybe slow, too. > > > > When deleting files does it matter if the data is fragmented?Yes. Deleting a file involves visiting the metadata for every extent (i.e. fragment) and updating it to reduce the reference count and possibly free up the space -- which itself involves more writes, as the free space cache is updated.> When i do normal file-access the speed of reading is ok so until not > i didn''t care about fragmentation. Especially because the data isn''t > changes after the first write. > > Can i do a defrag on the metadata only?Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- UNIX: Italian pen maker. ---
Tobias wrote (ao):> >>>On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de> wrote: > >>>>My BTRFS-FS ist getting really slow. Reading is ok, writing is > >>>>slow and deleting is horrible slow. > >>>> > >>>>There are many files and many links on the FS.Do you happen to have (many) snapshots? Are btrfs kernel threads using a lot of cpu? Sander -- Humilis IT Services and Solutions http://www.humilis.net -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 16.12.2011 10:19, schrieb Sander:> Tobias wrote (ao): >>>>> On Fri, Dec 16, 2011 at 1:49 AM, Tobias<tracer@robotech.de> wrote: >>>>>> My BTRFS-FS ist getting really slow. Reading is ok, writing is >>>>>> slow and deleting is horrible slow. >>>>>> >>>>>> There are many files and many links on the FS. > Do you happen to have (many) snapshots? Are btrfs kernel threads using a > lot of cpu? >No Snapshots at all. Only compress-force=lzo, nothing else special. The CPU is almost idle, IOwait is at 100%; looks likes the discs are accessing many small blocks. (seeking like hell...) Output from iostat (average each over 120 seconds) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.42 195.31 190.44 781.23 1657.90 12.65 64.91 168.90 3.42 338.60 2.51 96.64 sdb 0.00 0.50 75.00 336.57 300.00 2028.60 11.32 105.41 255.83 3.16 312.14 2.24 92.38 sdb 0.00 0.62 72.03 441.72 288.11 2164.88 9.55 116.39 226.27 4.20 262.48 1.89 97.29 sdb 0.00 0.20 1.07 468.74 4.26 2604.99 11.11 145.07 308.91 581.30 308.29 2.13 100.00 sdb 0.00 0.00 20.00 326.97 80.00 2114.03 12.65 38.56 111.41 58.02 114.67 2.83 98.28 sdb 0.00 0.00 5.02 563.35 20.08 3102.45 10.99 137.71 242.26 6.44 244.36 1.70 96.62 Tobias -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html