Btrfs devel - Mar 2013 - Btrfs in multiple different disc -d sigle -m raid1 one drive failure...

Hi all,
I''m preparing a strorage pool for large data with quite low importance
- there will be at least 3 hdd in "-d single" and "-m raid1"
configuration.

mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc

What happen if one hdd fails? Do I lost everything from all three
discs or only data from one disc? (if from only one disc, then is it
acceptable otherwise not...)

I read all documentation and a lot of discussions on the web and
answer is not clear at all...

Kind Regards

Jan.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Sigh... failed to reply all...

On Mon, Mar 18, 2013 at 11:52 AM, Michael Johnson - MJ <mj@revmj.com>
wrote:
> Hi Jan,
>
> In that configuration, you should assume that a single disk failure
> will cause complete data loss.  While some data may be recoverable, I
> don''t believe btrfs gives you any guarantees in that regard.
>
> But, I have a suggestion for you that may be suitable for what you want.
>
> * Create 3 individual btrfs file systems, one on each disk.
>
> * Use ''aufs'' to present these filesystems as if they were
a single file system.
>
> I use a setup like this for data I don''t consider important, but
would
> rather not lose it all.  This ensures that individual files are always
> completely contained on a single disk.  It also has the additional
> advantage of preventing filesystem corruption from causing total data
> loss.
>
>
> On Mon, Mar 18, 2013 at 11:32 AM, Jan Beranek <jan233321@gmail.com>
wrote:
>> Hi all,
>> I''m preparing a strorage pool for large data with quite low
importance
>> - there will be at least 3 hdd in "-d single" and "-m
raid1"
>> configuration.
>>
>> mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
>>
>> What happen if one hdd fails? Do I lost everything from all three
>> discs or only data from one disc? (if from only one disc, then is it
>> acceptable otherwise not...)
>>
>> I read all documentation and a lot of discussions on the web and
>> answer is not clear at all...
>>
>> Kind Regards
>>
>> Jan.
>> --
>> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Michael Johnson - MJ


-- 
Michael Johnson - MJ
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Michael,
thank you for your advice - looks nice, I never used "aufs" before - I
will give a try.

Kind Regards

Jan.

On Mon, Mar 18, 2013 at 7:55 PM, Michael Johnson - MJ <mj@revmj.com>
wrote:
> Sigh... failed to reply all...
>
> On Mon, Mar 18, 2013 at 11:52 AM, Michael Johnson - MJ <mj@revmj.com>
wrote:
>> Hi Jan,
>>
>> In that configuration, you should assume that a single disk failure
>> will cause complete data loss.  While some data may be recoverable, I
>> don''t believe btrfs gives you any guarantees in that regard.
>>
>> But, I have a suggestion for you that may be suitable for what you
want.
>>
>> * Create 3 individual btrfs file systems, one on each disk.
>>
>> * Use ''aufs'' to present these filesystems as if they
were a single file system.
>>
>> I use a setup like this for data I don''t consider important,
but would
>> rather not lose it all.  This ensures that individual files are always
>> completely contained on a single disk.  It also has the additional
>> advantage of preventing filesystem corruption from causing total data
>> loss.
>>
>>
>> On Mon, Mar 18, 2013 at 11:32 AM, Jan Beranek
<jan233321@gmail.com> wrote:
>>> Hi all,
>>> I''m preparing a strorage pool for large data with quite
low importance
>>> - there will be at least 3 hdd in "-d single" and
"-m raid1"
>>> configuration.
>>>
>>> mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
>>>
>>> What happen if one hdd fails? Do I lost everything from all three
>>> discs or only data from one disc? (if from only one disc, then is
it
>>> acceptable otherwise not...)
>>>
>>> I read all documentation and a lot of discussions on the web and
>>> answer is not clear at all...
>>>
>>> Kind Regards
>>>
>>> Jan.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Michael Johnson - MJ
>
>
>
> --
> Michael Johnson - MJ


--
S pozdravem

Jan Beránek
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Mar 18, 2013 at 12:32 PM, Jan Beranek <jan233321@gmail.com>
wrote:
> Hi all,
> I''m preparing a strorage pool for large data with quite low
importance
> - there will be at least 3 hdd in "-d single" and "-m
raid1"
> configuration.
>
> mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
>
> What happen if one hdd fails? Do I lost everything from all three
> discs or only data from one disc? (if from only one disc, then is it
> acceptable otherwise not...)
I just finished doing some testing to check:  It will work, kinda sorta.

You''ll be forced to mount read-only, and any reads of file extents
that existed on the missing disk will return an io error.  As I
understand it, single doesn''t force files to be on a single disk,
instead it _doesn''t_ force them to be _several_ disks; the implication
being that a large file (say, a 4gb movie) may still end up with
pieces on each disk.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thank you for the test.

I will do some tests also (later as now I do not have a space for it)
- when you disconnected one disc (or deleted a file) did you try btrfs
fi balance? I wonder if missing files simply "disappears" or not,
because in the fact they are present as metadata information only. if
not, then would be necessary to  write a small script which will check
all the files and report missing/broken ones (or use aufs on top).

When I use "-d single" option, then preffered way is fill the disc
completelly and then continue with another disc. And you are right,
there is no guarantee that some files wil not be spread over all three
(or more) disc. (as I understand it from docs).

On Mon, Mar 18, 2013 at 9:03 PM, Jan Beranek <jan233321@gmail.com>
wrote:
> Thank you for your test.
>
> I will do some tests also (later as now I do not have a space for it) -
when
> you disconnected one disc (or deleted a file) did you try btrfs fi balance?
> I wonder if missing files simply "disappears" or not, because in
the fact
> they are present as metadata information only. if not, then would be
> necessary to  write a small script which will check all the files and
report
> missing/broken ones (or use aufs on top).
>
> When I use "-d single" option, then preffered way is fill the
disc
> completelly and then continue with another disc. And you are right, there
is
> no guarantee that some files wil not be spread over all three (or more)
> disc. (as I understand it from docs).
>
>
> On Mon, Mar 18, 2013 at 8:24 PM, cwillu <cwillu@cwillu.com> wrote:
>>
>> On Mon, Mar 18, 2013 at 12:32 PM, Jan Beranek
<jan233321@gmail.com> wrote:
>> > Hi all,
>> > I''m preparing a strorage pool for large data with quite
low importance
>> > - there will be at least 3 hdd in "-d single" and
"-m raid1"
>> > configuration.
>> >
>> > mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
>> >
>> > What happen if one hdd fails? Do I lost everything from all three
>> > discs or only data from one disc? (if from only one disc, then is
it
>> > acceptable otherwise not...)
>>
>> I just finished doing some testing to check:  It will work, kinda
sorta.
>>
>> You''ll be forced to mount read-only, and any reads of file
extents
>> that existed on the missing disk will return an io error.  As I
>> understand it, single doesn''t force files to be on a single
disk,
>> instead it _doesn''t_ force them to be _several_ disks; the
implication
>> being that a large file (say, a 4gb movie) may still end up with
>> pieces on each disk.
>
>
>
>
> --
> S pozdravem
>
> Jan Beránek


--
S pozdravem

Jan Beránek
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Mar 18, 2013 at 09:04:16PM +0100, Jan Beranek
wrote:
> Thank you for the test.
> 
> I will do some tests also (later as now I do not have a space for it)
> - when you disconnected one disc (or deleted a file) did you try btrfs
> fi balance? I wonder if missing files simply "disappears" or not,
> because in the fact they are present as metadata information only. if
> not, then would be necessary to  write a small script which will check
> all the files and report missing/broken ones (or use aufs on top).
   You''ll still see the file, because the metadata has two copies, so
it''ll still be there. You''ll only get problems when you try to
read it.
> When I use "-d single" option, then preffered way is fill the
disc
> completelly and then continue with another disc. And you are right,
> there is no guarantee that some files wil not be spread over all three
> (or more) disc. (as I understand it from docs).
   Correct. The solution to this, as discussed on this list several
times over the last year, is to use alternative chunk and extent
allocators which try to fill disks sequentially and keep extents of
the same files on the same device.

   We don''t have those yet, and I''m not aware of anyone
planning on
implementing them.

   Hugo.
> On Mon, Mar 18, 2013 at 9:03 PM, Jan Beranek <jan233321@gmail.com>
wrote:
> > Thank you for your test.
> >
> > I will do some tests also (later as now I do not have a space for it)
- when
> > you disconnected one disc (or deleted a file) did you try btrfs fi
balance?
> > I wonder if missing files simply "disappears" or not,
because in the fact
> > they are present as metadata information only. if not, then would be
> > necessary to  write a small script which will check all the files and
report
> > missing/broken ones (or use aufs on top).
> >
> > When I use "-d single" option, then preffered way is fill
the disc
> > completelly and then continue with another disc. And you are right,
there is
> > no guarantee that some files wil not be spread over all three (or
more)
> > disc. (as I understand it from docs).
> >
> >
> > On Mon, Mar 18, 2013 at 8:24 PM, cwillu <cwillu@cwillu.com>
wrote:
> >>
> >> On Mon, Mar 18, 2013 at 12:32 PM, Jan Beranek
<jan233321@gmail.com> wrote:
> >> > Hi all,
> >> > I''m preparing a strorage pool for large data with
quite low importance
> >> > - there will be at least 3 hdd in "-d single" and
"-m raid1"
> >> > configuration.
> >> >
> >> > mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
> >> >
> >> > What happen if one hdd fails? Do I lost everything from all
three
> >> > discs or only data from one disc? (if from only one disc,
then is it
> >> > acceptable otherwise not...)
> >>
> >> I just finished doing some testing to check:  It will work, kinda
sorta.
> >>
> >> You''ll be forced to mount read-only, and any reads of
file extents
> >> that existed on the missing disk will return an io error.  As I
> >> understand it, single doesn''t force files to be on a
single disk,
> >> instead it _doesn''t_ force them to be _several_ disks;
the implication
> >> being that a large file (say, a 4gb movie) may still end up with
> >> pieces on each disk.
> >
> >
> >
> >
-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ==  PGP
key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
              --- w.w.w.  : England''s batting scorecard ---

While in specific use cases it may be preferable to fill up one disk
and then move on to the next, that does not get you the best
performance, so I suspect you will be hard pressed to find any "real"
filesystem that behaves in such a way.  You may be able to do that
with aufs, but if you you want to limit data loss in the event of a
failure, you probably want to always write to the disk with the most
free space which aufs can do.

On Mon, Mar 18, 2013 at 1:04 PM, Jan Beranek <jan233321@gmail.com>
wrote:
> Thank you for the test.
>
> I will do some tests also (later as now I do not have a space for it)
> - when you disconnected one disc (or deleted a file) did you try btrfs
> fi balance? I wonder if missing files simply "disappears" or not,
> because in the fact they are present as metadata information only. if
> not, then would be necessary to  write a small script which will check
> all the files and report missing/broken ones (or use aufs on top).
>
> When I use "-d single" option, then preffered way is fill the
disc
> completelly and then continue with another disc. And you are right,
> there is no guarantee that some files wil not be spread over all three
> (or more) disc. (as I understand it from docs).
>
> On Mon, Mar 18, 2013 at 9:03 PM, Jan Beranek <jan233321@gmail.com>
wrote:
>> Thank you for your test.
>>
>> I will do some tests also (later as now I do not have a space for it) -
when
>> you disconnected one disc (or deleted a file) did you try btrfs fi
balance?
>> I wonder if missing files simply "disappears" or not, because
in the fact
>> they are present as metadata information only. if not, then would be
>> necessary to  write a small script which will check all the files and
report
>> missing/broken ones (or use aufs on top).
>>
>> When I use "-d single" option, then preffered way is fill the
disc
>> completelly and then continue with another disc. And you are right,
there is
>> no guarantee that some files wil not be spread over all three (or more)
>> disc. (as I understand it from docs).
>>
>>
>> On Mon, Mar 18, 2013 at 8:24 PM, cwillu <cwillu@cwillu.com>
wrote:
>>>
>>> On Mon, Mar 18, 2013 at 12:32 PM, Jan Beranek
<jan233321@gmail.com> wrote:
>>> > Hi all,
>>> > I''m preparing a strorage pool for large data with
quite low importance
>>> > - there will be at least 3 hdd in "-d single" and
"-m raid1"
>>> > configuration.
>>> >
>>> > mkfs.btrfs -d single -m raid1 /dev/sda /dev/sdb /dec/sdc
>>> >
>>> > What happen if one hdd fails? Do I lost everything from all
three
>>> > discs or only data from one disc? (if from only one disc, then
is it
>>> > acceptable otherwise not...)
>>>
>>> I just finished doing some testing to check:  It will work, kinda
sorta.
>>>
>>> You''ll be forced to mount read-only, and any reads of file
extents
>>> that existed on the missing disk will return an io error.  As I
>>> understand it, single doesn''t force files to be on a
single disk,
>>> instead it _doesn''t_ force them to be _several_ disks; the
implication
>>> being that a large file (say, a 4gb movie) may still end up with
>>> pieces on each disk.
>>
>>
>>
>>
>> --
>> S pozdravem
>>
>> Jan Beránek
>
>
>
> --
> S pozdravem
>
> Jan Beránek
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Michael Johnson - MJ
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html