thr3ads.net - Btrfs devel - No space left on device (28) [Mar 2013]

If this information is useful, please help other people find it:
Share via:

Stefan Priebe - Profihost AG

2013-Mar-21 08:03 UTC

No space left on device (28)

Hi,

while copying a 15GB file to my btrfs volume i''m getting "No space
left
on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00

# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt

# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3

# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Greets.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Chris Mason

2013-Mar-21 18:00 UTC

head link

Re: No space left on device (28)

Quoting Stefan Priebe - Profihost AG (2013-03-21
04:03:04)> Hi,
> 
> while copying a 15GB file to my btrfs volume i''m getting "No
space left
> on device".
> 
> Some information:
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.97GB
> Metadata: total=8.00MB, used=0.00
> 
> # df -h
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> 
> # btrfs fi show
> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>         Total devices 1 FS bytes used 15.07TB
>         devid    1 size 18.19TB used 18.19TB path /dev/dm-3
> 
> # btrfs fi balance start -dusage=5 /mnt/
> Done, had to relocate 0 out of 18604 chunks
> 
> What wrong here? What is about the last 3TB?
Which kernel are you running?

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-21 18:35 UTC

head link

Re: No space left on device (28)

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>> Hi,
>> 
>> while copying a 15GB file to my btrfs volume i''m getting
"No space left
>> on device".
>> 
>> Some information:
>> 
>> # btrfs filesystem df /mnt/
>> Data: total=18.14TB, used=15.05TB
>> System, DUP: total=8.00MB, used=1.94MB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=23.98GB, used=17.97GB
>> Metadata: total=8.00MB, used=0.00
>> 
>> # df -h
>> Filesystem             Size  Used Avail Use% Mounted on
>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>> 
>> # btrfs fi show
>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>        Total devices 1 FS bytes used 15.07TB
>>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>> 
>> # btrfs fi balance start -dusage=5 /mnt/
>> Done, had to relocate 0 out of 18604 chunks
>> 
>> What wrong here? What is about the last 3TB?
> 
> Which kernel are you running?
> 
> -chris
vanilla 3.8.3.

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Chris Mason

2013-Mar-21 19:02 UTC

head link

Re: No space left on device (28)

Quoting Stefan Priebe - Profihost AG (2013-03-21
14:35:59)> Hi Chris,
> 
> Am 21.03.2013 um 19:00 schrieb Chris Mason
<chris.mason@fusionio.com>:
> 
> > Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
> >> Hi,
> >> 
> >> while copying a 15GB file to my btrfs volume i''m getting
"No space left
> >> on device".
> >> 
> >> Some information:
> >> 
> >> # btrfs filesystem df /mnt/
> >> Data: total=18.14TB, used=15.05TB
> >> System, DUP: total=8.00MB, used=1.94MB
> >> System: total=4.00MB, used=0.00
> >> Metadata, DUP: total=23.98GB, used=17.97GB
> >> Metadata: total=8.00MB, used=0.00
> >> 
> >> # df -h
> >> Filesystem             Size  Used Avail Use% Mounted on
> >> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> >> 
> >> # btrfs fi show
> >> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
> >>        Total devices 1 FS bytes used 15.07TB
> >>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
> >> 
> >> # btrfs fi balance start -dusage=5 /mnt/
> >> Done, had to relocate 0 out of 18604 chunks
> >> 
> >> What wrong here? What is about the last 3TB?
> > 
> > Which kernel are you running?
> > 
> > -chris
> 
> vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations.
Are you able to try a slightly more experimental kernel?

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-21 19:42 UTC

head link

Re: No space left on device (28)

Yes, i''ve now upgraded to 3.9-rc3 same result.

rsync: rename 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP"
-> 
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>> Hi Chris,
>>
>> Am 21.03.2013 um 19:00 schrieb Chris Mason
<chris.mason@fusionio.com>:
>>
>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>> Hi,
>>>>
>>>> while copying a 15GB file to my btrfs volume i''m
getting "No space left
>>>> on device".
>>>>
>>>> Some information:
>>>>
>>>> # btrfs filesystem df /mnt/
>>>> Data: total=18.14TB, used=15.05TB
>>>> System, DUP: total=8.00MB, used=1.94MB
>>>> System: total=4.00MB, used=0.00
>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>> Metadata: total=8.00MB, used=0.00
>>>>
>>>> # df -h
>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>
>>>> # btrfs fi show
>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>         Total devices 1 FS bytes used 15.07TB
>>>>         devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>>>>
>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>> Done, had to relocate 0 out of 18604 chunks
>>>>
>>>> What wrong here? What is about the last 3TB?
>>>
>>> Which kernel are you running?
>>>
>>> -chris
>>
>> vanilla 3.8.3.
>
> Ok, with the 3.9 merge window Josef changed how we do the reservations.
> Are you able to try a slightly more experimental kernel?
>
> -chris
>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 06:08 UTC

head link

Re: No space left on device (28)

Should i try to downgrade?

Greets,
Stefan

Am 21.03.2013 um 20:42 schrieb Stefan Priebe <s.priebe@profihost.ag>:
> Yes, i''ve now upgraded to 3.9-rc3 same result.
> 
> rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP"
->
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
No space left on device (28)
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.98GB
> Metadata: total=8.00MB, used=0.00
> 
> Stefan
> 
> Am 21.03.2013 20:02, schrieb Chris Mason:
>> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>>> Hi Chris,
>>> 
>>> Am 21.03.2013 um 19:00 schrieb Chris Mason
<chris.mason@fusionio.com>:
>>> 
>>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>>> Hi,
>>>>> 
>>>>> while copying a 15GB file to my btrfs volume i''m
getting "No space left
>>>>> on device".
>>>>> 
>>>>> Some information:
>>>>> 
>>>>> # btrfs filesystem df /mnt/
>>>>> Data: total=18.14TB, used=15.05TB
>>>>> System, DUP: total=8.00MB, used=1.94MB
>>>>> System: total=4.00MB, used=0.00
>>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>>> Metadata: total=8.00MB, used=0.00
>>>>> 
>>>>> # df -h
>>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>> 
>>>>> # btrfs fi show
>>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>>        Total devices 1 FS bytes used 15.07TB
>>>>>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>>>>> 
>>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>>> Done, had to relocate 0 out of 18604 chunks
>>>>> 
>>>>> What wrong here? What is about the last 3TB?
>>>> 
>>>> Which kernel are you running?
>>>> 
>>>> -chris
>>> 
>>> vanilla 3.8.3.
>> 
>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>> Are you able to try a slightly more experimental kernel?
>> 
>> -chris
>> --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Roman Mamedov

2013-Mar-22 06:13 UTC

head link

Re: No space left on device (28)

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <s.priebe@profihost.ag> wrote:

I might be wrong here, but doesn''t this
> rsync: rename 
>
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP"
> -> 
>
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
...try to move a file from 

  "/mnt/.software/"

to

  ".software/"

(relative to current dir)??

Maybe you end up trying to rsync your array to a small root partition or
something?
> No space left on device (28)
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.98GB
> Metadata: total=8.00MB, used=0.00
> 
> Stefan
> 
> Am 21.03.2013 20:02, schrieb Chris Mason:
> > Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
> >> Hi Chris,
> >>
> >> Am 21.03.2013 um 19:00 schrieb Chris Mason
<chris.mason@fusionio.com>:
> >>
> >>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
> >>>> Hi,
> >>>>
> >>>> while copying a 15GB file to my btrfs volume i''m
getting "No space left
> >>>> on device".
> >>>>
> >>>> Some information:
> >>>>
> >>>> # btrfs filesystem df /mnt/
> >>>> Data: total=18.14TB, used=15.05TB
> >>>> System, DUP: total=8.00MB, used=1.94MB
> >>>> System: total=4.00MB, used=0.00
> >>>> Metadata, DUP: total=23.98GB, used=17.97GB
> >>>> Metadata: total=8.00MB, used=0.00
> >>>>
> >>>> # df -h
> >>>> Filesystem             Size  Used Avail Use% Mounted on
> >>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> >>>>
> >>>> # btrfs fi show
> >>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
> >>>>         Total devices 1 FS bytes used 15.07TB
> >>>>         devid    1 size 18.19TB used 18.19TB path
/dev/dm-3
> >>>>
> >>>> # btrfs fi balance start -dusage=5 /mnt/
> >>>> Done, had to relocate 0 out of 18604 chunks
> >>>>
> >>>> What wrong here? What is about the last 3TB?
> >>>
> >>> Which kernel are you running?
> >>>
> >>> -chris
> >>
> >> vanilla 3.8.3.
> >
> > Ok, with the 3.9 merge window Josef changed how we do the
reservations.
> > Are you able to try a slightly more experimental kernel?
> >
> > -chris
> >
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
With respect,
Roman

cwillu

2013-Mar-22 06:24 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov <rm@romanrm.ru>
wrote:> On Thu, 21 Mar 2013 20:42:28 +0100
> Stefan Priebe <s.priebe@profihost.ag> wrote:
>
> I might be wrong here, but doesn''t this
>
>> rsync: rename
>>
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>> ->
>>
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>
> ...try to move a file from
>
>   "/mnt/.software/"
>
> to
>
>   ".software/"
>
> (relative to current dir)??
No; that''s rsync giving the full path, and then the target path
relative to the command it was given.  The filename itself
(".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to
temporarily, so it can mv it over the original in an atomic fashion...

Stefan: ...which means that the actual copy succeeded, which suggests
that this is more of a metadata enospc thing.

You might try btrfs balance start -musage=5 (instead of -dusage), and
if that doesn''t report any chunks balanced, try a high number until it
does.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 06:38 UTC

head link

Re: No space left on device (28)

No this is fine it''s rsync style same happens with cp.

Am 22.03.2013 um 07:13 schrieb Roman Mamedov <rm@romanrm.ru>:
> On Thu, 21 Mar 2013 20:42:28 +0100
> Stefan Priebe <s.priebe@profihost.ag> wrote:
> 
> I might be wrong here, but doesn''t this
> 
>> rsync: rename 
>>
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP"
>> -> 
>>
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
> 
> ...try to move a file from 
> 
>  "/mnt/.software/"
> 
> to
> 
>  ".software/"
> 
> (relative to current dir)??
> 
> Maybe you end up trying to rsync your array to a small root partition or
> something?
> 
>> No space left on device (28)
>> 
>> # btrfs filesystem df /mnt/
>> Data: total=18.14TB, used=15.05TB
>> System, DUP: total=8.00MB, used=1.94MB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=23.98GB, used=17.98GB
>> Metadata: total=8.00MB, used=0.00
>> 
>> Stefan
>> 
>> Am 21.03.2013 20:02, schrieb Chris Mason:
>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>>>> Hi Chris,
>>>> 
>>>> Am 21.03.2013 um 19:00 schrieb Chris Mason
<chris.mason@fusionio.com>:
>>>> 
>>>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>>>> Hi,
>>>>>> 
>>>>>> while copying a 15GB file to my btrfs volume
i''m getting "No space left
>>>>>> on device".
>>>>>> 
>>>>>> Some information:
>>>>>> 
>>>>>> # btrfs filesystem df /mnt/
>>>>>> Data: total=18.14TB, used=15.05TB
>>>>>> System, DUP: total=8.00MB, used=1.94MB
>>>>>> System: total=4.00MB, used=0.00
>>>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>>>> Metadata: total=8.00MB, used=0.00
>>>>>> 
>>>>>> # df -h
>>>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>>> 
>>>>>> # btrfs fi show
>>>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>>>        Total devices 1 FS bytes used 15.07TB
>>>>>>        devid    1 size 18.19TB used 18.19TB path
/dev/dm-3
>>>>>> 
>>>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>>>> Done, had to relocate 0 out of 18604 chunks
>>>>>> 
>>>>>> What wrong here? What is about the last 3TB?
>>>>> 
>>>>> Which kernel are you running?
>>>>> 
>>>>> -chris
>>>> 
>>>> vanilla 3.8.3.
>>> 
>>> Ok, with the 3.9 merge window Josef changed how we do the
reservations.
>>> Are you able to try a slightly more experimental kernel?
>>> 
>>> -chris
>> --
>> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> -- 
> With respect,
> Roman--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 06:39 UTC

head link

Re: No space left on device (28)

Already tried with value 5 did not help ;-( and it also happens with plain cp
copying a 15gb file and aborts at about 80%

Am 22.03.2013 um 07:24 schrieb cwillu <cwillu@cwillu.com>:
> On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov <rm@romanrm.ru>
wrote:
>> On Thu, 21 Mar 2013 20:42:28 +0100
>> Stefan Priebe <s.priebe@profihost.ag> wrote:
>> 
>> I might be wrong here, but doesn''t this
>> 
>>> rsync: rename
>>>
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>>> ->
>>>
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>> 
>> ...try to move a file from
>> 
>>  "/mnt/.software/"
>> 
>> to
>> 
>>  ".software/"
>> 
>> (relative to current dir)??
> 
> No; that''s rsync giving the full path, and then the target path
> relative to the command it was given.  The filename itself
> (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write
to
> temporarily, so it can mv it over the original in an atomic fashion...
> 
> Stefan: ...which means that the actual copy succeeded, which suggests
> that this is more of a metadata enospc thing.
> 
> You might try btrfs balance start -musage=5 (instead of -dusage), and
> if that doesn''t report any chunks balanced, try a high number
until it
> does.--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

cwillu

2013-Mar-22 06:41 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:> Already tried with value 5 did not help ;-( and it also happens with plain
cp copying a 15gb file and aborts at about 80%
You tried -musage=5?  Your original email said -dusage=5.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 07:06 UTC

head link

Re: No space left on device (28)

Hi,

Am 22.03.2013 07:41, schrieb cwillu:> On Fri, Mar 22, 2013 at 12:39 AM,
Stefan Priebe - Profihost AG> <s.priebe@profihost.ag> wrote:
>> Already tried with value 5 did not help ;-( and it also happens with
>> plain cp copying a 15gb file and aborts at about 80%
>
> You tried -musage=5?  Your original email said -dusage=5.
sorry i missed the difference - but it does not work:

~# btrfs fi balance start -musage=5 /mnt/
ERROR: error during balancing ''/mnt/'' - No space left on
device
There may be more info in syslog - try dmesg | tail

~# dmesg|tail
[ 1183.019367] device fsid fc23c4a8-a351-4c91-96a2-ee6abeffe59a devid 1
transid 6224 /dev/mapper/raid54tb1
[ 1183.019860] btrfs: disk space caching is enabled
[42719.915781] btrfs: relocating block group 4194304 flags 4
[42727.916141] btrfs: 5 enospc errors during balance

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 12:11 UTC

head link

Re: No space left on device (28)

Hi Chris,
>>>>> Which kernel are you running?
>>>>>
>>>>> -chris
>>>>
>>>> vanilla 3.8.3.
>>>
>>> Ok, with the 3.9 merge window Josef changed how we do the
reservations.
>>> Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.

Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-22 13:53 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG
wrote:> Hi Chris,
> 
> >>>>> Which kernel are you running?
> >>>>>
> >>>>> -chris
> >>>>
> >>>> vanilla 3.8.3.
> >>>
> >>> Ok, with the 3.9 merge window Josef changed how we do the
reservations.
> >>> Are you able to try a slightly more experimental kernel?
> 
> any ideas what i can check? 3.9-rc3 gives me same results.
> 
Sorry Stefan I''m almost done with what I''m working on and then
I''ll work up a
patch for you to run so I can narrow down what''s going on.  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-22 13:56 UTC

head link

Re: No space left on device (28)

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG
wrote:
>> Hi Chris,
>>
>>>>>>> Which kernel are you running?
>>>>>>>
>>>>>>> -chris
>>>>>>
>>>>>> vanilla 3.8.3.
>>>>>
>>>>> Ok, with the 3.9 merge window Josef changed how we do the
reservations.
>>>>> Are you able to try a slightly more experimental kernel?
>>
>> any ideas what i can check? 3.9-rc3 gives me same results.
>>
> 
> Sorry Stefan I''m almost done with what I''m working on and
then I''ll work up a
> patch for you to run so I can narrow down what''s going on. 
Thanks,
Great!

Thanks - just wanted to know that it''s not my fault. I''m happy
to test
the patch and provide feedback.

Stefan

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-22 15:54 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG
wrote:> Hi Josef,
> Am 22.03.2013 14:53, schrieb Josef Bacik:
> > On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG
wrote:
> >> Hi Chris,
> >>
> >>>>>>> Which kernel are you running?
> >>>>>>>
> >>>>>>> -chris
> >>>>>>
> >>>>>> vanilla 3.8.3.
> >>>>>
> >>>>> Ok, with the 3.9 merge window Josef changed how we do
the reservations.
> >>>>> Are you able to try a slightly more experimental
kernel?
> >>
> >> any ideas what i can check? 3.9-rc3 gives me same results.
> >>
> > 
> > Sorry Stefan I''m almost done with what I''m working
on and then I''ll work up a
> > patch for you to run so I can narrow down what''s going on. 
Thanks,
> 
> Great!
> 
> Thanks - just wanted to know that it''s not my fault. I''m
happy to test
> the patch and provide feedback.
> 
Ok I think we are way over-reserving for rename, can you give this patch a whirl
and see what happens?  If it still fails can you capture dmesg and reply with
that so I can see what''s going on.  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 9ac2eca..aabaea6 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4161,6 +4161,11 @@ out:
 			ret = 0;
 	}
 	if (flushing) {
+		if (ret == -ENOSPC) {
+			printk(KERN_ERR "returning enospc, dumping space info\n");
+			dump_space_info(space_info, 0, 0);
+		}
+
 		spin_lock(&space_info->lock);
 		space_info->flush = 0;
 		wake_up_all(&space_info->wait);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ca1b767..3980ae7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3679,11 +3679,9 @@ static struct btrfs_trans_handle
*__unlink_start_trans(struct inode *dir,
 	 * 1 for the dir item
 	 * 1 for the dir index
 	 * 1 for the inode ref
-	 * 1 for the inode ref in the tree log
-	 * 2 for the dir entries in the log
 	 * 1 for the inode
 	 */
-	trans = btrfs_start_transaction(root, 8);
+	trans = btrfs_start_transaction(root, 5);
 	if (!IS_ERR(trans) || PTR_ERR(trans) != -ENOSPC)
 		return trans;
 
@@ -8127,7 +8125,7 @@ static int btrfs_rename(struct inode *old_dir, struct
dentry *old_dentry,
 	 * inodes.  So 5 * 2 is 10, plus 1 for the new link, so 11 total items
 	 * should cover the worst case number of items we''ll modify.
 	 */
-	trans = btrfs_start_transaction(root, 20);
+	trans = btrfs_start_transaction(root, 11);
 	if (IS_ERR(trans)) {
                 ret = PTR_ERR(trans);
                 goto out_notrans;
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-22 19:10 UTC

head link

Re: No space left on device (28)

Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:> On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG
wrote:
>> Hi Josef,
>> Am 22.03.2013 14:53, schrieb Josef Bacik:
>>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost
AG wrote:
>>>> Hi Chris,
>>>>
>>>>>>>>> Which kernel are you running?
>>>>>>>>>
>>>>>>>>> -chris
>>>>>>>>
>>>>>>>> vanilla 3.8.3.
>>>>>>>
>>>>>>> Ok, with the 3.9 merge window Josef changed how we
do the reservations.
>>>>>>> Are you able to try a slightly more experimental
kernel?
>>>>
>>>> any ideas what i can check? 3.9-rc3 gives me same results.
>>>>
>>>
>>> Sorry Stefan I''m almost done with what I''m
working on and then I''ll work up a
>>> patch for you to run so I can narrow down what''s going on.
Thanks,
>>
>> Great!
>>
>> Thanks - just wanted to know that it''s not my fault.
I''m happy to test
>> the patch and provide feedback.
>>
> Ok I think we are way over-reserving for rename, can you give this patch a
whirl
> and see what happens?  If it still fails can you capture dmesg and reply
with
> that so I can see what''s going on.  Thanks,
Thanks for the patch. I was able to copy some files and i''m still able 
put it copy some then fails on some then copies some then fails on some.

Rsync output:
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-wm8350.c
        12156 100%   11.17kB/s    0:00:01 (xfer#21395, to-check=1008/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-x1205.c
        16460 100%   15.05kB/s    0:00:01 (xfer#21396, to-check=1007/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/systohc.c
         1223 100%    1.12kB/s    0:00:01 (xfer#21397, to-check=1006/52625)
.software/kernel/linux-3.9-rc3/drivers/s390/
.software/kernel/linux-3.9-rc3/drivers/s390/Makefile
          144 100%    0.13kB/s    0:00:01 (xfer#21398, to-check=1052/52672)
.software/kernel/linux-3.9-rc3/drivers/s390/block/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/block" failed: No 
space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/char/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/char" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/cio/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/cio" failed: No
space
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.88pm8607.c.YBi0Rz"
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/crypto/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/crypto" failed: No 
space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/kvm/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/kvm" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/net/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/net" failed: No
space
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Kconfig.0lsfuE"
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Makefile.C9wI7I"
failed: No space left on device (28)
.software/kernel/linux-3.9-rc3/drivers/s390/scsi/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/scsi" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/sbus/
.software/kernel/linux-3.9-rc3/drivers/sbus/Makefile
           70 100%    0.06kB/s    0:00:01 (xfer#21399, to-check=1003/52824)
.software/kernel/linux-3.9-rc3/drivers/sbus/char/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/sbus/char" failed: No
space
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.aat2870-regulator.c.BGDwPN"
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/scsi/
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.ko.cmd
          208 100%    0.16kB/s    0:00:01 (xfer#21400, to-check=1407/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.mod.o.cmd
        27987 100%   21.54kB/s    0:00:01 (xfer#21401, to-check=1406/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.o.cmd
        43340 100%   32.63kB/s    0:00:01 (xfer#21402, to-check=1405/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.ko.cmd
          204 100%   15.32kB/s    0:00:00 (xfer#21403, to-check=1404/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.mod.o.cmd
        27975 100%    1.91MB/s    0:00:00 (xfer#21404, to-check=1403/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.o.cmd
        43327 100%  983.99kB/s    0:00:00 (xfer#21405, to-check=1402/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.ko.cmd
          208 100%    4.72kB/s    0:00:00 (xfer#21406, to-check=1401/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.mod.o.cmd
        27987 100%  621.16kB/s    0:00:00 (xfer#21407, to-check=1400/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.o.cmd
        43340 100%  742.53kB/s    0:00:00 (xfer#21408, to-check=1399/53242)

dmesg Output was pretty long so i posted to pastebin:
http://pastebin.com/raw.php?i=5kUS5MGA

Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-22 20:49 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe
wrote:> Hi Josef,
> Am 22.03.2013 16:54, schrieb Josef Bacik:
> > On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG
wrote:
> >> Hi Josef,
> >> Am 22.03.2013 14:53, schrieb Josef Bacik:
> >>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe -
Profihost AG wrote:
> >>>> Hi Chris,
> >>>>
> >>>>>>>>> Which kernel are you running?
> >>>>>>>>>
> >>>>>>>>> -chris
> >>>>>>>>
> >>>>>>>> vanilla 3.8.3.
> >>>>>>>
> >>>>>>> Ok, with the 3.9 merge window Josef changed
how we do the reservations.
> >>>>>>> Are you able to try a slightly more
experimental kernel?
> >>>>
> >>>> any ideas what i can check? 3.9-rc3 gives me same results.
> >>>>
> >>>
> >>> Sorry Stefan I''m almost done with what I''m
working on and then I''ll work up a
> >>> patch for you to run so I can narrow down what''s
going on.  Thanks,
> >>
> >> Great!
> >>
> >> Thanks - just wanted to know that it''s not my fault.
I''m happy to test
> >> the patch and provide feedback.
> >>
> > Ok I think we are way over-reserving for rename, can you give this
patch a whirl
> > and see what happens?  If it still fails can you capture dmesg and
reply with
> > that so I can see what''s going on.  Thanks,
> 
> Thanks for the patch. I was able to copy some files and i''m still
able
> put it copy some then fails on some then copies some then fails on some.
> 
Ok so I''ve been working on another ENOSPC related problem that I think
is
affecting you as well.  So I''m going to nail that down and see if that
helps you
too, if not we''ll poke around your problem some more and try and work
out what''s
happening.  I''ll get back to this fresh Monday morning.  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-22 20:55 UTC

head link

Re: No space left on device (28)

Hi Jsoef,

thanks!

Am 22.03.2013 21:49, schrieb Josef Bacik:> On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe wrote:
>> Hi Josef,
>> Am 22.03.2013 16:54, schrieb Josef Bacik:
>>> On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost
AG wrote:
>>>> Hi Josef,
>>>> Am 22.03.2013 14:53, schrieb Josef Bacik:
>>>>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe -
Profihost AG wrote:
>>>>>> Hi Chris,
>>>>>>
>>>>>>>>>>> Which kernel are you running?
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>
>>>>>>>>>> vanilla 3.8.3.
>>>>>>>>>
>>>>>>>>> Ok, with the 3.9 merge window Josef changed
how we do the reservations.
>>>>>>>>> Are you able to try a slightly more
experimental kernel?
>>>>>>
>>>>>> any ideas what i can check? 3.9-rc3 gives me same
results.
>>>>>>
>>>>>
>>>>> Sorry Stefan I''m almost done with what
I''m working on and then I''ll work up a
>>>>> patch for you to run so I can narrow down what''s
going on.  Thanks,
>>>>
>>>> Great!
>>>>
>>>> Thanks - just wanted to know that it''s not my fault.
I''m happy to test
>>>> the patch and provide feedback.
>>>>
>>> Ok I think we are way over-reserving for rename, can you give this
patch a whirl
>>> and see what happens?  If it still fails can you capture dmesg and
reply with
>>> that so I can see what''s going on.  Thanks,
>>
>> Thanks for the patch. I was able to copy some files and i''m
still able
>> put it copy some then fails on some then copies some then fails on
some.
>>
>
> Ok so I''ve been working on another ENOSPC related problem that I
think is
> affecting you as well.  So I''m going to nail that down and see if
that helps you
> too, if not we''ll poke around your problem some more and try and
work out what''s
> happening.  I''ll get back to this fresh Monday morning.  Thanks,
>
> Josef
>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-25 20:14 UTC

head link

Re: No space left on device (28)

On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe
wrote:> Hi Jsoef,
> 
> thanks!
Ok I don''t think the thing I just fixed will make any difference for
you so
here''s another debug patch, just apply it on top of what I''ve
already sent you
and re-run and give me the dmesg again.  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index aabaea6..bf6433f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4162,7 +4162,11 @@ out:
 	}
 	if (flushing) {
 		if (ret == -ENOSPC) {
-			printk(KERN_ERR "returning enospc, dumping space info\n");
+			spin_lock(&block_rsv->lock);
+			printk(KERN_ERR "returning enospc, space_info %u, size %Lu reserved
%Lu, "
+			       "flush %d, flush_state %d  dumping space info\n",
block_rsv->type,
+			       block_rsv->size, block_rsv->reserved, flush, flush_state);
+			spin_unlock(&block_rsv->lock);
 			dump_space_info(space_info, 0, 0);
 		}
 
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-26 07:45 UTC

head link

Re: No space left on device (28)

Hi,

output here:
[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
reserved=73728, may_use=6438297600, readonly=65536

Am 25.03.2013 21:14, schrieb Josef Bacik:> On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe wrote:
>> Hi Jsoef,
>>
>> thanks!
>
> Ok I don''t think the thing I just fixed will make any difference
for you so
> here''s another debug patch, just apply it on top of what
I''ve already sent you
> and re-run and give me the dmesg again.  Thanks,
>
> Josef
>
>
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index aabaea6..bf6433f 100644
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -4162,7 +4162,11 @@ out:
>   	}
>   	if (flushing) {
>   		if (ret == -ENOSPC) {
> -			printk(KERN_ERR "returning enospc, dumping space info\n");
> +			spin_lock(&block_rsv->lock);
> +			printk(KERN_ERR "returning enospc, space_info %u, size %Lu
reserved %Lu, "
> +			       "flush %d, flush_state %d  dumping space info\n",
block_rsv->type,
> +			       block_rsv->size, block_rsv->reserved, flush, flush_state);
> +			spin_unlock(&block_rsv->lock);
>   			dump_space_info(space_info, 0, 0);
>   		}
>
>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 12:53 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe
wrote:> Hi,
> 
> output here:
> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.548280] space_info 4 has 6439292928 free, is full
> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=32768, may_use=6438354944, readonly=65536
> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.552264] space_info 4 has 6439284736 free, is full
> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=40960, may_use=6438354944, readonly=65536
> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.556258] space_info 4 has 6439284736 free, is full
> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=40960, may_use=6438354944, readonly=65536
> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  591.147373] space_info 4 has 6439235584 free, is full
> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=90112, may_use=6438354944, readonly=65536
> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  595.562390] space_info 4 has 6439120896 free, is full
> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
> reserved=73728, may_use=6438297600, readonly=65536
> 
Weird, we have all the flushing stuff set and yet there is still a whole lot of
outstanding reservations.  Do you have compression enabled?  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-26 12:55 UTC

head link

Re: No space left on device (28)

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
>> Hi,
>>
>> output here:
>> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  590.548280] space_info 4 has 6439292928 free, is full
>> [  590.548283] space_info total=25748307968, used=19308916736,
pinned=0,
>> reserved=32768, may_use=6438354944, readonly=65536
>> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  590.552264] space_info 4 has 6439284736 free, is full
>> [  590.552267] space_info total=25748307968, used=19308916736,
pinned=0,
>> reserved=40960, may_use=6438354944, readonly=65536
>> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  590.556258] space_info 4 has 6439284736 free, is full
>> [  590.556261] space_info total=25748307968, used=19308916736,
pinned=0,
>> reserved=40960, may_use=6438354944, readonly=65536
>> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  591.147373] space_info 4 has 6439235584 free, is full
>> [  591.147375] space_info total=25748307968, used=19308916736,
pinned=0,
>> reserved=90112, may_use=6438354944, readonly=65536
>> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  595.562390] space_info 4 has 6439120896 free, is full
>> [  595.562393] space_info total=25748307968, used=19309047808,
pinned=0,
>> reserved=73728, may_use=6438297600, readonly=65536
>>
> 
> Weird, we have all the flushing stuff set and yet there is still a whole
lot of
> outstanding reservations.  Do you have compression enabled?  Thanks,
no - it''s just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 13:30 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG
wrote:> Hi Josef,
> 
> Am 26.03.2013 13:53, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >> Hi,
> >>
> >> output here:
> >> [  590.546162] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  590.548280] space_info 4 has 6439292928 free, is full
> >> [  590.548283] space_info total=25748307968, used=19308916736,
pinned=0,
> >> reserved=32768, may_use=6438354944, readonly=65536
> >> [  590.550147] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  590.552264] space_info 4 has 6439284736 free, is full
> >> [  590.552267] space_info total=25748307968, used=19308916736,
pinned=0,
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [  590.554141] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  590.556258] space_info 4 has 6439284736 free, is full
> >> [  590.556261] space_info total=25748307968, used=19308916736,
pinned=0,
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [  591.145255] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  591.147373] space_info 4 has 6439235584 free, is full
> >> [  591.147375] space_info total=25748307968, used=19308916736,
pinned=0,
> >> reserved=90112, may_use=6438354944, readonly=65536
> >> [  595.560257] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  595.562390] space_info 4 has 6439120896 free, is full
> >> [  595.562393] space_info total=25748307968, used=19309047808,
pinned=0,
> >> reserved=73728, may_use=6438297600, readonly=65536
> >>
> > 
> > Weird, we have all the flushing stuff set and yet there is still a
whole lot of
> > outstanding reservations.  Do you have compression enabled?  Thanks,
> 
> no - it''s just mounted with mount -o noatime
> 
> :~# cat /proc/mounts | grep btrfs
> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>
Ok I think I see what''s going on.  Can you try this patch and see if it
fixes
it?  Thanks,

Josef

 
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index bf6433f..84e8909 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3803,6 +3803,19 @@ static int can_overcommit(struct btrfs_root *root,
 	return 0;
 }
 
+static int btrfs_try_writeback(struct super_block *sb, unsigned long nr_pages,
+			       enum wb_reason reason)
+{
+	if (!writeback_in_progress(sb->s_bdi) &&
+	    down_read_trylock(&sb->s_umount)) {
+		writeback_inodes_sb_nr(sb, nr_pages, reason);
+		up_read(&sb->s_umount);
+		return 1;
+	}
+
+	return 0;
+}
+
 void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 				  unsigned long nr_pages)
 {
@@ -3810,8 +3823,7 @@ void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 	int started;
 
 	/* If we can not start writeback, just sync all the delalloc file. */
-	started = try_to_writeback_inodes_sb_nr(sb, nr_pages,
-						      WB_REASON_FS_FREE_SPACE);
+	started = btrfs_try_writeback(sb, nr_pages, WB_REASON_FS_FREE_SPACE);
 	if (!started) {
 		/*
 		 * We needn''t worry the filesystem going from r/w to r/o though
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-26 13:49 UTC

head link

Re: No space left on device (28)

Hi Josef,


Am 26.03.2013 14:30, schrieb Josef Bacik:> On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG
wrote:
>> Hi Josef,
>>
>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
>>>> Hi,
>>>>
>>>> output here:
>>>> [  590.546162] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  590.548280] space_info 4 has 6439292928 free, is full
>>>> [  590.548283] space_info total=25748307968, used=19308916736,
pinned=0,
>>>> reserved=32768, may_use=6438354944, readonly=65536
>>>> [  590.550147] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  590.552264] space_info 4 has 6439284736 free, is full
>>>> [  590.552267] space_info total=25748307968, used=19308916736,
pinned=0,
>>>> reserved=40960, may_use=6438354944, readonly=65536
>>>> [  590.554141] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  590.556258] space_info 4 has 6439284736 free, is full
>>>> [  590.556261] space_info total=25748307968, used=19308916736,
pinned=0,
>>>> reserved=40960, may_use=6438354944, readonly=65536
>>>> [  591.145255] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  591.147373] space_info 4 has 6439235584 free, is full
>>>> [  591.147375] space_info total=25748307968, used=19308916736,
pinned=0,
>>>> reserved=90112, may_use=6438354944, readonly=65536
>>>> [  595.560257] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  595.562390] space_info 4 has 6439120896 free, is full
>>>> [  595.562393] space_info total=25748307968, used=19309047808,
pinned=0,
>>>> reserved=73728, may_use=6438297600, readonly=65536
>>>>
>>>
>>> Weird, we have all the flushing stuff set and yet there is still a
whole lot of
>>> outstanding reservations.  Do you have compression enabled? 
Thanks,
>>
>> no - it''s just mounted with mount -o noatime
>>
>> :~# cat /proc/mounts | grep btrfs
>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>
> 
> Ok I think I see what''s going on.  Can you try this patch and see
if it fixes
> it?  Thanks,
It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then
continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt
         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
".etc_openvpn/ipp.txt": No space left on device (28)
.log/
.log/UcliEvt.log
      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01

the dmesg output looks like this:
[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
reserved=49152, may_use=6438453248, readonly=65536

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 14:44 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 07:49:16AM -0600, Stefan Priebe - Profihost AG
wrote:> Hi Josef,
> 
> 
> Am 26.03.2013 14:30, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG
wrote:
> >> Hi Josef,
> >>
> >> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >>>> Hi,
> >>>>
> >>>> output here:
> >>>> [  590.546162] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.548280] space_info 4 has 6439292928 free, is full
> >>>> [  590.548283] space_info total=25748307968,
used=19308916736, pinned=0,
> >>>> reserved=32768, may_use=6438354944, readonly=65536
> >>>> [  590.550147] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.552264] space_info 4 has 6439284736 free, is full
> >>>> [  590.552267] space_info total=25748307968,
used=19308916736, pinned=0,
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  590.554141] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.556258] space_info 4 has 6439284736 free, is full
> >>>> [  590.556261] space_info total=25748307968,
used=19308916736, pinned=0,
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  591.145255] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  591.147373] space_info 4 has 6439235584 free, is full
> >>>> [  591.147375] space_info total=25748307968,
used=19308916736, pinned=0,
> >>>> reserved=90112, may_use=6438354944, readonly=65536
> >>>> [  595.560257] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  595.562390] space_info 4 has 6439120896 free, is full
> >>>> [  595.562393] space_info total=25748307968,
used=19309047808, pinned=0,
> >>>> reserved=73728, may_use=6438297600, readonly=65536
> >>>>
> >>>
> >>> Weird, we have all the flushing stuff set and yet there is
still a whole lot of
> >>> outstanding reservations.  Do you have compression enabled? 
Thanks,
> >>
> >> no - it''s just mounted with mount -o noatime
> >>
> >> :~# cat /proc/mounts | grep btrfs
> >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>
> > 
> > Ok I think I see what''s going on.  Can you try this patch and
see if it fixes
> > it?  Thanks,
> 
> It still does not fix the problem.
> 
> The rsync output looks like this so it does not work for file a but then
> continues on c d e, ...
> 
> sync -av --progress /backup/ /mnt/
> sending incremental file list
> .etc_openvpn/ipp.txt
>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> .etc_openvpn/openvpn-status.log
>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> ".etc_openvpn/ipp.txt": No space left on device (28)
> .log/
> .log/UcliEvt.log
>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> .log/auth.log
>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> .log/auth.log.1
>     19431424  61%    7.35MB/s    0:00:01
> 
> the dmesg output looks like this:
> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7  dumping space info
> [  551.323694] space_info 4 has 6439526400 free, is full
> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> reserved=49152, may_use=6438453248, readonly=65536
>
Ok so then this is probably it, let me know if it helps.  Thanks,

Josef
 
diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index dc08d77..005c45d 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int
delay_iput)
 	INIT_LIST_HEAD(&splice);
 	INIT_LIST_HEAD(&works);
 
+	mutex_lock(&root->fs_info->ordered_operations_mutex);
 	spin_lock(&root->fs_info->ordered_extent_lock);
 	list_splice_init(&root->fs_info->ordered_extents, &splice);
 	while (!list_empty(&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int
delay_iput)
 
 		cond_resched();
 	}
+	mutex_unlock(&root->fs_info->ordered_operations_mutex);
 }
 
 /*
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

2013-Mar-26 15:03 UTC

head link

Re: No space left on device (28)

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>> no - it''s just mounted with mount -o noatime
>>>>
>>>> :~# cat /proc/mounts | grep btrfs
>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>>>
>>>
>>> Ok I think I see what''s going on.  Can you try this patch
and see if it fixes
>>> it?  Thanks,
>>
>> It still does not fix the problem.
>>
>> The rsync output looks like this so it does not work for file a but
then
>> continues on c d e, ...
>>
>> sync -av --progress /backup/ /mnt/
>> sending incremental file list
>> .etc_openvpn/ipp.txt
>>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
>> .etc_openvpn/openvpn-status.log
>>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>> ".etc_openvpn/ipp.txt": No space left on device (28)
>> .log/
>> .log/UcliEvt.log
>>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
>> .log/auth.log
>>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
>> .log/auth.log.1
>>     19431424  61%    7.35MB/s    0:00:01
>>
>> the dmesg output looks like this:
>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  551.323694] space_info 4 has 6439526400 free, is full
>> [  551.323696] space_info total=25748307968, used=19308666880,
pinned=0,
>> reserved=49152, may_use=6438453248, readonly=65536
>>
> 
> Ok so then this is probably it, let me know if it helps.  Thanks,
OK it now has copied a lot of files (170) without an error all were very
small.

Then this again:
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.vhost_net.mod.GrhWet"
->
".software/kernel/linux-3.9-rc3/.tmp_versions/vhost_net.mod": No space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rhine.mod.X2Ofhz"
->
".software/kernel/linux-3.9-rc3/.tmp_versions/via-rhine.mod": No space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rng.mod.RAyxkF"
-> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rng.mod": No
space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.virtio.mod.w7INoL"
-> ".software/kernel/linux-3.9-rc3/.tmp_versions/virtio.mod": No
space
left on device (28)^C

[ 5012.468988] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.472981] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.472982] space_info 4 has 6439399424 free, is full
[ 5012.472982] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.476974] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.476975] space_info 4 has 6439399424 free, is full
[ 5012.476976] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.480968] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.480969] space_info 4 has 6439399424 free, is full

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 15:25 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG
wrote:> Hi,
> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>> no - it''s just mounted with mount -o noatime
> >>>>
> >>>> :~# cat /proc/mounts | grep btrfs
> >>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0
0
> >>>>
> >>>
> >>> Ok I think I see what''s going on.  Can you try this
patch and see if it fixes
> >>> it?  Thanks,
> >>
> >> It still does not fix the problem.
> >>
> >> The rsync output looks like this so it does not work for file a
but then
> >> continues on c d e, ...
> >>
> >> sync -av --progress /backup/ /mnt/
> >> sending incremental file list
> >> .etc_openvpn/ipp.txt
> >>          229 100%    3.99kB/s    0:00:00 (xfer#2,
to-check=1009/1196)
> >> .etc_openvpn/openvpn-status.log
> >>          360 100%    6.28kB/s    0:00:00 (xfer#3,
to-check=1007/1196)
> >> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >> ".etc_openvpn/ipp.txt": No space left on device (28)
> >> .log/
> >> .log/UcliEvt.log
> >>       104188 100%  147.67kB/s    0:00:00 (xfer#4,
to-check=1131/2700)
> >> .log/auth.log
> >>     15211522 100%    2.97MB/s    0:00:04 (xfer#5,
to-check=1105/2700)
> >> .log/auth.log.1
> >>     19431424  61%    7.35MB/s    0:00:01
> >>
> >> the dmesg output looks like this:
> >> [  551.321576] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [  551.323694] space_info 4 has 6439526400 free, is full
> >> [  551.323696] space_info total=25748307968, used=19308666880,
pinned=0,
> >> reserved=49152, may_use=6438453248, readonly=65536
> >>
> > 
> > Ok so then this is probably it, let me know if it helps.  Thanks,
> 
> OK it now has copied a lot of files (170) without an error all were very
> small.
>
Welp progress is good.  Throw this into the mix and go again, it''s just
adding
some more debugging so I can make sure I''m going down the right rabbit
hole.
Thanks,

Josef 

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 84e8909..1cf810a 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4026,6 +4026,15 @@ static int flush_space(struct btrfs_root *root,
 
 	return ret;
 }
+
+static void dump_block_rsv(struct btrfs_block_rsv *block_rsv)
+{
+	spin_lock(&block_rsv->lock);
+	printk(KERN_ERR "dumping block rsv %u, size %Lu reserved %Lu\n",
+	       block_rsv->type, block_rsv->size, block_rsv->reserved);
+	spin_unlock(&block_rsv->lock);
+}
+
 /**
  * reserve_metadata_bytes - try to reserve bytes from the block_rsv''s
space
  * @root - the root we''re allocating for
@@ -4179,6 +4188,9 @@ out:
 			       "flush %d, flush_state %d  dumping space info\n",
block_rsv->type,
 			       block_rsv->size, block_rsv->reserved, flush, flush_state);
 			spin_unlock(&block_rsv->lock);
+			dump_block_rsv(&root->fs_info->delalloc_block_rsv);
+			dump_block_rsv(&root->fs_info->delayed_block_rsv);
+			dump_block_rsv(&root->fs_info->global_block_rsv);
 			dump_space_info(space_info, 0, 0);
 		}
 
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-26 16:19 UTC

head link

Re: No space left on device (28)

HI,


Am 26.03.2013 16:25, schrieb Josef Bacik:> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG
wrote:
>> Hi,
>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>>>> no - it''s just mounted with mount -o noatime
>>>>>>
>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache
0 0
>>>>>>
>>>>>
>>>>> Ok I think I see what''s going on.  Can you try
this patch and see if it fixes
>>>>> it?  Thanks,
>>>>
>>>> It still does not fix the problem.
>>>>
>>>> The rsync output looks like this so it does not work for file a
but then
>>>> continues on c d e, ...
>>>>
>>>> sync -av --progress /backup/ /mnt/
>>>> sending incremental file list
>>>> .etc_openvpn/ipp.txt
>>>>           229 100%    3.99kB/s    0:00:00 (xfer#2,
to-check=1009/1196)
>>>> .etc_openvpn/openvpn-status.log
>>>>           360 100%    6.28kB/s    0:00:00 (xfer#3,
to-check=1007/1196)
>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX"
->
>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
>>>> .log/
>>>> .log/UcliEvt.log
>>>>        104188 100%  147.67kB/s    0:00:00 (xfer#4,
to-check=1131/2700)
>>>> .log/auth.log
>>>>      15211522 100%    2.97MB/s    0:00:04 (xfer#5,
to-check=1105/2700)
>>>> .log/auth.log.1
>>>>      19431424  61%    7.35MB/s    0:00:01
>>>>
>>>> the dmesg output looks like this:
>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  551.323694] space_info 4 has 6439526400 free, is full
>>>> [  551.323696] space_info total=25748307968, used=19308666880,
pinned=0,
>>>> reserved=49152, may_use=6438453248, readonly=65536
>>>>
>>>
>>> Ok so then this is probably it, let me know if it helps.  Thanks,
>>
>> OK it now has copied a lot of files (170) without an error all were
very
>> small.
>>
>
> Welp progress is good.  Throw this into the mix and go again, it''s
just adding
> some more debugging so I can make sure I''m going down the right
rabbit hole.
> Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, 
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, 
reserved=45056, may_use=6438354944, readonly=65536

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 17:45 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 10:19:19AM -0600, Stefan Priebe
wrote:> HI,
> 
> 
> Am 26.03.2013 16:25, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG
wrote:
> >> Hi,
> >> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>>>> no - it''s just mounted with mount -o
noatime
> >>>>>>
> >>>>>> :~# cat /proc/mounts | grep btrfs
> >>>>>> /dev/mapper/raid54tb1 /mnt btrfs
rw,noatime,space_cache 0 0
> >>>>>>
> >>>>>
> >>>>> Ok I think I see what''s going on.  Can you
try this patch and see if it fixes
> >>>>> it?  Thanks,
> >>>>
> >>>> It still does not fix the problem.
> >>>>
> >>>> The rsync output looks like this so it does not work for
file a but then
> >>>> continues on c d e, ...
> >>>>
> >>>> sync -av --progress /backup/ /mnt/
> >>>> sending incremental file list
> >>>> .etc_openvpn/ipp.txt
> >>>>           229 100%    3.99kB/s    0:00:00 (xfer#2,
to-check=1009/1196)
> >>>> .etc_openvpn/openvpn-status.log
> >>>>           360 100%    6.28kB/s    0:00:00 (xfer#3,
to-check=1007/1196)
> >>>> rsync: rename
"/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >>>> ".etc_openvpn/ipp.txt": No space left on device
(28)
> >>>> .log/
> >>>> .log/UcliEvt.log
> >>>>        104188 100%  147.67kB/s    0:00:00 (xfer#4,
to-check=1131/2700)
> >>>> .log/auth.log
> >>>>      15211522 100%    2.97MB/s    0:00:04 (xfer#5,
to-check=1105/2700)
> >>>> .log/auth.log.1
> >>>>      19431424  61%    7.35MB/s    0:00:01
> >>>>
> >>>> the dmesg output looks like this:
> >>>> [  551.321576] returning enospc, space_info 3, size 0
reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  551.323694] space_info 4 has 6439526400 free, is full
> >>>> [  551.323696] space_info total=25748307968,
used=19308666880, pinned=0,
> >>>> reserved=49152, may_use=6438453248, readonly=65536
> >>>>
> >>>
> >>> Ok so then this is probably it, let me know if it helps. 
Thanks,
> >>
> >> OK it now has copied a lot of files (170) without an error all
were very
> >> small.
> >>
> >
> > Welp progress is good.  Throw this into the mix and go again,
it''s just adding
> > some more debugging so I can make sure I''m going down the
right rabbit hole.
> > Thanks,
> 
> Output is now:
> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
> [ 9587.646958] space_info 4 has 6439428096 free, is full
> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, 
> reserved=45056, may_use=6438453248, readonly=65536
> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
> [ 9587.839935] space_info 4 has 6439428096 free, is full
> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, 
> reserved=45056, may_use=6438354944, readonly=65536
> 
Well then that looks like I was going down the wrong rabbit hole.  This should
fix you up, for real this time ;).  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 1cf810a..ac415cf7 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4471,7 +4471,12 @@ static void update_global_block_rsv(struct btrfs_fs_info
*fs_info)
 	spin_lock(&sinfo->lock);
 	spin_lock(&block_rsv->lock);
 
-	block_rsv->size = num_bytes;
+	/*
+	 * Limit the global block rsv to 512mb, we have infrastructure in place
+	 * to throttle reservations if we start getting low on global block rsv
+	 * space.
+	 */
+	block_rsv->size = min_t(u64, num_bytes, 512 * 1024 * 1024);
 
 	num_bytes = sinfo->bytes_used + sinfo->bytes_pinned +
 		    sinfo->bytes_reserved + sinfo->bytes_readonly +
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-26 19:05 UTC

head link

Re: No space left on device (28)

Hi Josef,

Am 26.03.2013 18:45, schrieb Josef Bacik:>> Am 26.03.2013 16:25, schrieb Josef Bacik:
>>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost
AG wrote:
>>>> Hi,
>>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>>>>>> no - it''s just mounted with mount -o
noatime
>>>>>>>>
>>>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs
rw,noatime,space_cache 0 0
>>>>>>>>
>>>>>>>
>>>>>>> Ok I think I see what''s going on.  Can you
try this patch and see if it fixes
>>>>>>> it?  Thanks,
>>>>>>
>>>>>> It still does not fix the problem.
>>>>>>
>>>>>> The rsync output looks like this so it does not work
for file a but then
>>>>>> continues on c d e, ...
>>>>>>
>>>>>> sync -av --progress /backup/ /mnt/
>>>>>> sending incremental file list
>>>>>> .etc_openvpn/ipp.txt
>>>>>>            229 100%    3.99kB/s    0:00:00 (xfer#2,
to-check=1009/1196)
>>>>>> .etc_openvpn/openvpn-status.log
>>>>>>            360 100%    6.28kB/s    0:00:00 (xfer#3,
to-check=1007/1196)
>>>>>> rsync: rename
"/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>>>>>> ".etc_openvpn/ipp.txt": No space left on
device (28)
>>>>>> .log/
>>>>>> .log/UcliEvt.log
>>>>>>         104188 100%  147.67kB/s    0:00:00 (xfer#4,
to-check=1131/2700)
>>>>>> .log/auth.log
>>>>>>       15211522 100%    2.97MB/s    0:00:04 (xfer#5,
to-check=1105/2700)
>>>>>> .log/auth.log.1
>>>>>>       19431424  61%    7.35MB/s    0:00:01
>>>>>>
>>>>>> the dmesg output looks like this:
>>>>>> [  551.321576] returning enospc, space_info 3, size 0
reserved 0, flush
>>>>>> 2, flush_state 7  dumping space info
>>>>>> [  551.323694] space_info 4 has 6439526400 free, is
full
>>>>>> [  551.323696] space_info total=25748307968,
used=19308666880, pinned=0,
>>>>>> reserved=49152, may_use=6438453248, readonly=65536
>>>>>>
>>>>>
>>>>> Ok so then this is probably it, let me know if it helps. 
Thanks,
>>>>
>>>> OK it now has copied a lot of files (170) without an error all
were very
>>>> small.
>>>>
>>>
>>> Welp progress is good.  Throw this into the mix and go again,
it''s just adding
>>> some more debugging so I can make sure I''m going down the
right rabbit hole.
>>> Thanks,
>>
>> Output is now:
>> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
>> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
>> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
>> [ 9587.646958] space_info 4 has 6439428096 free, is full
>> [ 9587.646963] space_info total=25748307968, used=19308769280,
pinned=0,
>> reserved=45056, may_use=6438453248, readonly=65536
>> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
>> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
>> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
>> [ 9587.839935] space_info 4 has 6439428096 free, is full
>> [ 9587.839936] space_info total=25748307968, used=19308769280,
pinned=0,
>> reserved=45056, may_use=6438354944, readonly=65536
>>
>
> Well then that looks like I was going down the wrong rabbit hole.  This
should
> fix you up, for real this time ;).  Thanks,
Yes - this works now. Which of the patches can i drop? Do i just need 
the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0?
Will the fix be part of 3.9-rc5?

Thanks and greets,
Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 19:16 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe
wrote:> Hi Josef,
> 
> Am 26.03.2013 18:45, schrieb Josef Bacik:
> >> Am 26.03.2013 16:25, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe -
Profihost AG wrote:
> >>>> Hi,
> >>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>>>>>> no - it''s just mounted with mount
-o noatime
> >>>>>>>>
> >>>>>>>> :~# cat /proc/mounts | grep btrfs
> >>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs
rw,noatime,space_cache 0 0
> >>>>>>>>
> >>>>>>>
> >>>>>>> Ok I think I see what''s going on. 
Can you try this patch and see if it fixes
> >>>>>>> it?  Thanks,
> >>>>>>
> >>>>>> It still does not fix the problem.
> >>>>>>
> >>>>>> The rsync output looks like this so it does not
work for file a but then
> >>>>>> continues on c d e, ...
> >>>>>>
> >>>>>> sync -av --progress /backup/ /mnt/
> >>>>>> sending incremental file list
> >>>>>> .etc_openvpn/ipp.txt
> >>>>>>            229 100%    3.99kB/s    0:00:00
(xfer#2, to-check=1009/1196)
> >>>>>> .etc_openvpn/openvpn-status.log
> >>>>>>            360 100%    6.28kB/s    0:00:00
(xfer#3, to-check=1007/1196)
> >>>>>> rsync: rename
"/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >>>>>> ".etc_openvpn/ipp.txt": No space left on
device (28)
> >>>>>> .log/
> >>>>>> .log/UcliEvt.log
> >>>>>>         104188 100%  147.67kB/s    0:00:00
(xfer#4, to-check=1131/2700)
> >>>>>> .log/auth.log
> >>>>>>       15211522 100%    2.97MB/s    0:00:04
(xfer#5, to-check=1105/2700)
> >>>>>> .log/auth.log.1
> >>>>>>       19431424  61%    7.35MB/s    0:00:01
> >>>>>>
> >>>>>> the dmesg output looks like this:
> >>>>>> [  551.321576] returning enospc, space_info 3,
size 0 reserved 0, flush
> >>>>>> 2, flush_state 7  dumping space info
> >>>>>> [  551.323694] space_info 4 has 6439526400 free,
is full
> >>>>>> [  551.323696] space_info total=25748307968,
used=19308666880, pinned=0,
> >>>>>> reserved=49152, may_use=6438453248, readonly=65536
> >>>>>>
> >>>>>
> >>>>> Ok so then this is probably it, let me know if it
helps.  Thanks,
> >>>>
> >>>> OK it now has copied a lot of files (170) without an error
all were very
> >>>> small.
> >>>>
> >>>
> >>> Welp progress is good.  Throw this into the mix and go again,
it''s just adding
> >>> some more debugging so I can make sure I''m going down
the right rabbit hole.
> >>> Thanks,
> >>
> >> Output is now:
> >> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
> >> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved
6438256640
> >> [ 9587.646958] space_info 4 has 6439428096 free, is full
> >> [ 9587.646963] space_info total=25748307968, used=19308769280,
pinned=0,
> >> reserved=45056, may_use=6438453248, readonly=65536
> >> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0,
flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
> >> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved
6438256640
> >> [ 9587.839935] space_info 4 has 6439428096 free, is full
> >> [ 9587.839936] space_info total=25748307968, used=19308769280,
pinned=0,
> >> reserved=45056, may_use=6438354944, readonly=65536
> >>
> >
> > Well then that looks like I was going down the wrong rabbit hole. 
This should
> > fix you up, for real this time ;).  Thanks,
> 
> Yes - this works now. Which of the patches can i drop? Do i just need 
> the last one?
> Is it safe to add another 18TB raid via converting it to btrfs raid0?
> Will the fix be part of 3.9-rc5?
> 
So I''ll put together all of the patches that actually need to go up for
this and
post them, but basically its the mutex patch, the last patch I sent you and the
one that adjusts the reservations for rename and delete.  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-26 19:22 UTC

head link

Re: No space left on device (28)

Hi,

but when i transfer big files i see now this one:
[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
[20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[20368.895140] rsync           D ffffffff8160f580     0 14911      1 
0x00000000
[20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 
ffff88022394f800
[20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 
0000000000012c40
[20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 
ffff8801cefce130
[20368.895169] Call Trace:
[20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
[20368.895207]  [<ffffffffa0158c75>] 
wait_current_trans.isra.32+0x95/0x100 [btrfs]
[20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
[20368.895236]  [<ffffffffa015a45d>] 
start_transaction.part.33+0x13d/0x4d0 [btrfs]
[20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
[20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
[20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 
[btrfs]
[20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460
[btrfs]
[20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
[20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
[20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
[20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
[20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
[20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
[20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
[20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
[20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b

Speed is just 100kb/s instead of 100MB/s.

Stefan

Am 26.03.2013 20:16, schrieb Josef Bacik:> On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote:
>> Hi Josef,
>>
>> Am 26.03.2013 18:45, schrieb Josef Bacik:
>>>> Am 26.03.2013 16:25, schrieb Josef Bacik:
>>>>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe -
Profihost AG wrote:
>>>>>> Hi,
>>>>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>>>>>> Am 26.03.2013 13:53, schrieb Josef
Bacik:
>>>>>>>>>> no - it''s just mounted with
mount -o noatime
>>>>>>>>>>
>>>>>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs
rw,noatime,space_cache 0 0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok I think I see what''s going on. 
Can you try this patch and see if it fixes
>>>>>>>>> it?  Thanks,
>>>>>>>>
>>>>>>>> It still does not fix the problem.
>>>>>>>>
>>>>>>>> The rsync output looks like this so it does not
work for file a but then
>>>>>>>> continues on c d e, ...
>>>>>>>>
>>>>>>>> sync -av --progress /backup/ /mnt/
>>>>>>>> sending incremental file list
>>>>>>>> .etc_openvpn/ipp.txt
>>>>>>>>             229 100%    3.99kB/s    0:00:00
(xfer#2, to-check=1009/1196)
>>>>>>>> .etc_openvpn/openvpn-status.log
>>>>>>>>             360 100%    6.28kB/s    0:00:00
(xfer#3, to-check=1007/1196)
>>>>>>>> rsync: rename
"/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>>>>>>>> ".etc_openvpn/ipp.txt": No space left
on device (28)
>>>>>>>> .log/
>>>>>>>> .log/UcliEvt.log
>>>>>>>>          104188 100%  147.67kB/s    0:00:00
(xfer#4, to-check=1131/2700)
>>>>>>>> .log/auth.log
>>>>>>>>        15211522 100%    2.97MB/s    0:00:04
(xfer#5, to-check=1105/2700)
>>>>>>>> .log/auth.log.1
>>>>>>>>        19431424  61%    7.35MB/s    0:00:01
>>>>>>>>
>>>>>>>> the dmesg output looks like this:
>>>>>>>> [  551.321576] returning enospc, space_info 3,
size 0 reserved 0, flush
>>>>>>>> 2, flush_state 7  dumping space info
>>>>>>>> [  551.323694] space_info 4 has 6439526400
free, is full
>>>>>>>> [  551.323696] space_info total=25748307968,
used=19308666880, pinned=0,
>>>>>>>> reserved=49152, may_use=6438453248,
readonly=65536
>>>>>>>>
>>>>>>>
>>>>>>> Ok so then this is probably it, let me know if it
helps.  Thanks,
>>>>>>
>>>>>> OK it now has copied a lot of files (170) without an
error all were very
>>>>>> small.
>>>>>>
>>>>>
>>>>> Welp progress is good.  Throw this into the mix and go
again, it''s just adding
>>>>> some more debugging so I can make sure I''m going
down the right rabbit hole.
>>>>> Thanks,
>>>>
>>>> Output is now:
>>>> [ 9587.445642] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
>>>> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
>>>> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved
6438256640
>>>> [ 9587.646958] space_info 4 has 6439428096 free, is full
>>>> [ 9587.646963] space_info total=25748307968, used=19308769280,
pinned=0,
>>>> reserved=45056, may_use=6438453248, readonly=65536
>>>> [ 9587.649410] returning enospc, space_info 3, size 0 reserved
0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
>>>> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
>>>> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved
6438256640
>>>> [ 9587.839935] space_info 4 has 6439428096 free, is full
>>>> [ 9587.839936] space_info total=25748307968, used=19308769280,
pinned=0,
>>>> reserved=45056, may_use=6438354944, readonly=65536
>>>>
>>>
>>> Well then that looks like I was going down the wrong rabbit hole. 
This should
>>> fix you up, for real this time ;).  Thanks,
>>
>> Yes - this works now. Which of the patches can i drop? Do i just need
>> the last one?
>> Is it safe to add another 18TB raid via converting it to btrfs raid0?
>> Will the fix be part of 3.9-rc5?
>>
>
> So I''ll put together all of the patches that actually need to go
up for this and
> post them, but basically its the mutex patch, the last patch I sent you and
the
> one that adjusts the reservations for rename and delete.  Thanks,
>
> Josef
>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Josef Bacik

2013-Mar-26 19:38 UTC

head link

Re: No space left on device (28)

On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe
wrote:> Hi,
> 
> but when i transfer big files i see now this one:
> [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
> [20368.821978] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [20368.895140] rsync           D ffffffff8160f580     0 14911      1 
> 0x00000000
> [20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 
> ffff88022394f800
> [20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 
> 0000000000012c40
> [20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 
> ffff8801cefce130
> [20368.895169] Call Trace:
> [20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
> [20368.895207]  [<ffffffffa0158c75>] 
> wait_current_trans.isra.32+0x95/0x100 [btrfs]
> [20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
> [20368.895236]  [<ffffffffa015a45d>] 
> start_transaction.part.33+0x13d/0x4d0 [btrfs]
> [20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
> [20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30
[btrfs]
> [20368.895287]  [<ffffffffa015aae3>]
btrfs_start_transaction+0x13/0x20
> [btrfs]
> [20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460
[btrfs]
> [20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
> [20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
> [20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
> [20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
> [20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
> [20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
> [20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
> [20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
> [20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
> 
> Speed is just 100kb/s instead of 100MB/s.
> 
Hrm I wonder if 512 is too small for your case, can you add this patch to the
pile and see what dmesg says when you are having these problems?  Thanks,

Josef


diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 50767bb..d19c9f6 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -31,6 +31,7 @@
 #include "inode-map.h"
 #include "volumes.h"
 #include "dev-replace.h"
+#include "math.h"
 
 #define BTRFS_ROOT_TRANS_TAG 0
 
@@ -576,10 +577,19 @@ void btrfs_throttle(struct btrfs_root *root)
 static int should_end_transaction(struct btrfs_trans_handle *trans,
 				  struct btrfs_root *root)
 {
-	int ret;
+	struct btrfs_block_rsv *block_rsv =
&root->fs_info->global_block_rsv;
+	u64 num_bytes = 0;
+	int ret = 1;
 
-	ret = btrfs_block_rsv_check(root, &root->fs_info->global_block_rsv,
5);
-	return ret ? 1 : 0;
+	spin_lock(&block_rsv->lock);
+	num_bytes = div_factor(block_rsv->size, 5);
+	if (block_rsv->reserved >= num_bytes)
+		ret = 0;
+	else
+		printk(KERN_ERR "we''re pretty low, setting blocked, reserved
%Lu, size %Lu, num %Lu\n",
+		       block_rsv->reserved, block_rsv->size, num_bytes);
+	spin_unlock(&block_rsv->lock);
+	return ret;
 }
 
 int btrfs_should_end_transaction(struct btrfs_trans_handle *trans,
-- 
1.7.7.6

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

2013-Mar-26 19:47 UTC

head link

Re: No space left on device (28)

HI,
Am 26.03.2013 20:38, schrieb Josef Bacik:> On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe wrote:
>> Hi,
>>
>> but when i transfer big files i see now this one:
>> [20368.784736] INFO: task rsync:14911 blocked for more than 120
seconds.
>> [20368.821978] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [20368.895140] rsync           D ffffffff8160f580     0 14911      1
>> 0x00000000
>> [20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198
>> ffff88022394f800
>> [20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8
>> 0000000000012c40
>> [20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88
>> ffff8801cefce130
>> [20368.895169] Call Trace:
>> [20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
>> [20368.895207]  [<ffffffffa0158c75>]
>> wait_current_trans.isra.32+0x95/0x100 [btrfs]
>> [20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
>> [20368.895236]  [<ffffffffa015a45d>]
>> start_transaction.part.33+0x13d/0x4d0 [btrfs]
>> [20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
>> [20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30
[btrfs]
>> [20368.895287]  [<ffffffffa015aae3>]
btrfs_start_transaction+0x13/0x20
>> [btrfs]
>> [20368.895302]  [<ffffffffa015b2f0>]
__unlink_start_trans+0x70/0x460 [btrfs]
>> [20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
>> [20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
>> [20368.895317]  [<ffffffff811413bd>] ?
generic_permission+0xbd/0x110
>> [20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0
[btrfs]
>> [20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
>> [20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
>> [20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
>> [20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
>> [20368.895359]  [<ffffffff8151c2e9>]
system_call_fastpath+0x16/0x1b
>>
>> Speed is just 100kb/s instead of 100MB/s.
>>
>
> Hrm I wonder if 512 is too small for your case, can you add this patch to
the
> pile and see what dmesg says when you are having these problems?  Thanks,
No idea why but i can''t reproduce... ;-(

Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Btrfs devel - Mar 2013 - No space left on device (28)

No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)

Re: No space left on device (28)