Hello, Reporting this on behalf of ric. He was running the following fs_mark command ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we''ve just locked up behind a page lock somewhere. I have to run off to class so I can''t look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks, Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote:> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: > > Hello, > > > > Reporting this on behalf of ric. He was running the following fs_mark command > > > > ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F > > > > Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, > > which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom > > of the document. He could ctrl+c and unmount the fs so its not a hard hang. > > Looks like we''ve just locked up behind a page lock somewhere. I have to run off > > to class so I can''t look into it too deeply so throwing this out there hoping > > somebody else figures it out :). Thanks, > > Which kernel was this? > >Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you''ve pushed into git recently, it has yan''s backref work plus a few other fixes. Thanks, Josef>-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote:> Hello, > > Reporting this on behalf of ric. He was running the following fs_mark command > > ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F > > Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, > which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom > of the document. He could ctrl+c and unmount the fs so its not a hard hang. > Looks like we''ve just locked up behind a page lock somewhere. I have to run off > to class so I can''t look into it too deeply so throwing this out there hoping > somebody else figures it out :). Thanks,Which kernel was this? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote:> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote: > > On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: > > > Hello, > > > > > > Reporting this on behalf of ric. He was running the following fs_mark command > > > > > > ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F > > > > > > Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, > > > which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom > > > of the document. He could ctrl+c and unmount the fs so its not a hard hang. > > > Looks like we''ve just locked up behind a page lock somewhere. I have to run off > > > to class so I can''t look into it too deeply so throwing this out there hoping > > > somebody else figures it out :). Thanks, > > > > Which kernel was this?> Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he > said, its just shy whatever patches you''ve pushed into git recently, it has > yan''s backref work plus a few other fixes. Thanks, >Well, the trace is strange because it looks like everyone is waiting on IO. Any chance once of the procs was spinning in system time? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Josef Bacik wrote:> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote: > >> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: >> >>> Hello, >>> >>> Reporting this on behalf of ric. He was running the following fs_mark command >>> >>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F >>> >>> Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, >>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom >>> of the document. He could ctrl+c and unmount the fs so its not a hard hang. >>> Looks like we''ve just locked up behind a page lock somewhere. I have to run off >>> to class so I can''t look into it too deeply so throwing this out there hoping >>> somebody else figures it out :). Thanks, >>> >> Which kernel was this? >> >> >> > > Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he > said, its just shy whatever patches you''ve pushed into git recently, it has > yan''s backref work plus a few other fixes. Thanks, > > Josef >I can fire off a new test with the latest from unstable if that is of interest, ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason wrote:> On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote: > >> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote: >> >>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: >>> >>>> Hello, >>>> >>>> Reporting this on behalf of ric. He was running the following fs_mark command >>>> >>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F >>>> >>>> Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, >>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom >>>> of the document. He could ctrl+c and unmount the fs so its not a hard hang. >>>> Looks like we''ve just locked up behind a page lock somewhere. I have to run off >>>> to class so I can''t look into it too deeply so throwing this out there hoping >>>> somebody else figures it out :). Thanks, >>>> >>> Which kernel was this? >>> > > >> Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he >> said, its just shy whatever patches you''ve pushed into git recently, it has >> yan''s backref work plus a few other fixes. Thanks, >> >> > > Well, the trace is strange because it looks like everyone is waiting on > IO. Any chance once of the procs was spinning in system time? > > -chris > >It actually cleaned up very nicely after I killed the fs_mark processes & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense). ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 16:37 -0400, Ric Wheeler wrote:> Chris Mason wrote: > > On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote: > > > >> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote: > >> > >>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: > >>> > >>>> Hello, > >>>> > >>>> Reporting this on behalf of ric. He was running the following fs_mark command > >>>> > >>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F > >>>> > >>>> Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, > >>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom > >>>> of the document. He could ctrl+c and unmount the fs so its not a hard hang. > >>>> Looks like we''ve just locked up behind a page lock somewhere. I have to run off > >>>> to class so I can''t look into it too deeply so throwing this out there hoping > >>>> somebody else figures it out :). Thanks, > >>>> > >>> Which kernel was this?> It actually cleaned up very nicely after I killed the fs_mark processes > & unmounted the btrfs file system. Before doing that, the box was > sluggish and had the feeling of a system with something that might have > been spinning (but that is just an observation, not measured in any > strict sense).Ok, I have that fs_mark test running here. How far did yours get before it stopped? -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason wrote:> On Thu, 2008-09-25 at 16:37 -0400, Ric Wheeler wrote: > >> Chris Mason wrote: >> >>> On Thu, 2008-09-25 at 14:34 -0400, Josef Bacik wrote: >>> >>> >>>> On Thu, Sep 25, 2008 at 02:37:02PM -0400, Chris Mason wrote: >>>> >>>> >>>>> On Thu, 2008-09-25 at 13:56 -0400, Josef Bacik wrote: >>>>> >>>>> >>>>>> Hello, >>>>>> >>>>>> Reporting this on behalf of ric. He was running the following fs_mark command >>>>>> >>>>>> ./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F >>>>>> >>>>>> Seems it hung and wasn''t making any progress. He managed to get some sysrq-t, >>>>>> which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom >>>>>> of the document. He could ctrl+c and unmount the fs so its not a hard hang. >>>>>> Looks like we''ve just locked up behind a page lock somewhere. I have to run off >>>>>> to class so I can''t look into it too deeply so throwing this out there hoping >>>>>> somebody else figures it out :). Thanks, >>>>>> >>>>>> >>>>> Which kernel was this? >>>>> > > >> It actually cleaned up very nicely after I killed the fs_mark processes >> & unmounted the btrfs file system. Before doing that, the box was >> sluggish and had the feeling of a system with something that might have >> been spinning (but that is just an observation, not measured in any >> strict sense). >> > > Ok, I have that fs_mark test running here. How far did yours get before > it stopped? > > -chris > >I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk: 17 8064000 20480 5.6 15301404 This is the new (no system sync() call) Chris special fs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million. Maybe it really was not hung, just extremely slow... ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 17:10 -0400, Ric Wheeler wrote:> > Ok, I have that fs_mark test running here. How far did yours get before > > it stopped? > > > > -chris > > > > > I had gone (in heavy fsync mode) up to about 8 million files on a 1TB > s-ata disk: > > 17 8064000 20480 5.6 15301404 > > This is the new (no system sync() call) Chris special fs_mark. The rate > had been quite reasonable, starting out at around 160 20k files/sec, > went under 100 files/sec at around 3 million files and then fell under > 50 files/sec at around 7.5 million before hitting this really low speed > at just under 8 million. > > Maybe it really was not hung, just extremely slow... >I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we''re spinning around due to bad accounting on the reserved extents, and that Yan''s latest patch set will fix it. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason wrote:> On Thu, 2008-09-25 at 17:10 -0400, Ric Wheeler wrote: > > >>> Ok, I have that fs_mark test running here. How far did yours get before >>> it stopped? >>> >>> -chris >>> >>> >>> >> I had gone (in heavy fsync mode) up to about 8 million files on a 1TB >> s-ata disk: >> >> 17 8064000 20480 5.6 15301404 >> >> This is the new (no system sync() call) Chris special fs_mark. The rate >> had been quite reasonable, starting out at around 160 20k files/sec, >> went under 100 files/sec at around 3 million files and then fell under >> 50 files/sec at around 7.5 million before hitting this really low speed >> at just under 8 million. >> >> Maybe it really was not hung, just extremely slow... >> >> > > I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I > get 155 files/sec ;) My hope is that we''re spinning around due to bad > accounting on the reserved extents, and that Yan''s latest patch set will > fix it. > > -chrisI can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-) ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote:> > > > I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I > > get 155 files/sec ;) My hope is that we''re spinning around due to bad > > accounting on the reserved extents, and that Yan''s latest patch set will > > fix it. > > > > -chris > I can update & restart my test as well. It is an odd box (8 CPUs, only > 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in > testing out edge conditions ;-)I''ll push out Yan''s patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Chris Mason wrote:> On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote: > >>> I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I >>> get 155 files/sec ;) My hope is that we''re spinning around due to bad >>> accounting on the reserved extents, and that Yan''s latest patch set will >>> fix it. >>> >>> -chris >>> >> I can update & restart my test as well. It is an odd box (8 CPUs, only >> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in >> testing out edge conditions ;-) >> > > I''ll push out Yan''s patches tomorrow. My box here is at 17.5 million > files and still going at 148 files/sec > > -chris > > >Sounds like a plan, thanks! Ric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 21:01 -0400, Ric Wheeler wrote:> Chris Mason wrote: > > On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote: > > > >>> I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I > >>> get 155 files/sec ;) My hope is that we''re spinning around due to bad > >>> accounting on the reserved extents, and that Yan''s latest patch set will > >>> fix it. > >>> > >>> -chris > >>> > >> I can update & restart my test as well. It is an odd box (8 CPUs, only > >> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in > >> testing out edge conditions ;-) > >> > > > > I''ll push out Yan''s patches tomorrow. My box here is at 17.5 million > > files and still going at 148 files/sec > > > > -chris > > > > > > > Sounds like a plan, thanks!I declined a bit down to 60 files/sec, but overnight made it up to 58 or so million files without stalling. It is possible that my metadata threshold changes caused problems for you, which might explain why my 4GB of ram lasted longer than your 1GB. I''ll try to rework the thresholds. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2008-09-25 at 21:01 -0400, Ric Wheeler wrote:> Chris Mason wrote: > > On Thu, 2008-09-25 at 18:58 -0400, Ric Wheeler wrote: > > > >>> I''m at 6.9 million files so far on a 500GB disk, and not surprisingly, I > >>> get 155 files/sec ;) My hope is that we''re spinning around due to bad > >>> accounting on the reserved extents, and that Yan''s latest patch set will > >>> fix it. > >>> > >>> -chris > >>> > >> I can update & restart my test as well. It is an odd box (8 CPUs, only > >> 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in > >> testing out edge conditions ;-) > >> > > > > I''ll push out Yan''s patches tomorrow. My box here is at 17.5 million > > files and still going at 148 files/sec > > > Sounds like a plan, thanks!I turned up a long standing balancing bug while testing Yan''s latest code. It took forever to track down, but I''ve finally pushed out his patches. My fs_mark run never did stall, but I''m retesting now with all of the latest changes. You''ll need to pull the latest unstable btrfs-progs and kernel from git. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html