Btrfs devel - Oct 2013 - Some questions after devices addition to existing raid 1 btrfs filesystem

Hi,

I have added 2x2Tb to my existing 2x2Tb raid 1 btrfs filesystem and
then ran a balance:

# btrfs filesystem show
        Total devices 4 FS bytes used 1.74TB
        devid    3 size 1.82TB used 0.00 path /dev/sdd
        devid    4 size 1.82TB used 0.00 path /dev/sde
        devid    2 size 1.82TB used 1.75TB path /dev/sdc
        devid    1 size 1.82TB used 1.75TB path /dev/sdb
# btrfs filesystem balance btrfs_root/
# btrfs filesystem show
        Total devices 4 FS bytes used 1.74TB
        devid    3 size 1.82TB used 892.00GB path /dev/sdd
        devid    4 size 1.82TB used 892.00GB path /dev/sde
        devid    2 size 1.82TB used 891.03GB path /dev/sdc
        devid    1 size 1.82TB used 891.04GB path /dev/sdb

It took 59 hours to complete the balance.

I checked on a couple of files and all seems fine but I have some questions:
- is there some kind of ''overall filesystem health/integrity
check''
that I should do on the filesystem now that the balance is done?
- also, I ran the command while some of the btrfs subvolumes were
mounted (as well as the btrfs_root/ of course), does this impact on
the balance job?
- the mounted btrfs devices were mounted using -o
space_cache,inode_cache but the btrfs_root/ was not, also, does this
impact on the balance job?
- about those options, a few months ago, I oftent had
btrfs-cache-1/btrfs-endio-met processes taking some cpu/hd time.  I
was advised to mount -o space_cache,inode_cache, which seems to have
quiet the processes down.  Are those options still necessary now?
- as the job took 60+ hours but the CPU rarely went above 10%, the
computer seemed still usable.  I left it do its job of course but
could I have accessed or write anything on the subvolumes while the
balance was running and if yes, would this have any impact on the
filesystem?

Thank you for your help,
All the best,
Laurent
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, Oct 07, 2013 at 01:45:29PM +0200, Laurent Humblet
wrote:
> I have added 2x2Tb to my existing 2x2Tb raid 1 btrfs filesystem and
> then ran a balance:
> 
> # btrfs filesystem show
>         Total devices 4 FS bytes used 1.74TB
>         devid    3 size 1.82TB used 0.00 path /dev/sdd
>         devid    4 size 1.82TB used 0.00 path /dev/sde
>         devid    2 size 1.82TB used 1.75TB path /dev/sdc
>         devid    1 size 1.82TB used 1.75TB path /dev/sdb
> # btrfs filesystem balance btrfs_root/
> # btrfs filesystem show
>         Total devices 4 FS bytes used 1.74TB
>         devid    3 size 1.82TB used 892.00GB path /dev/sdd
>         devid    4 size 1.82TB used 892.00GB path /dev/sde
>         devid    2 size 1.82TB used 891.03GB path /dev/sdc
>         devid    1 size 1.82TB used 891.04GB path /dev/sdb
> 
> It took 59 hours to complete the balance.
> 
> I checked on a couple of files and all seems fine but I have some
questions:
> - is there some kind of ''overall filesystem health/integrity
check''
> that I should do on the filesystem now that the balance is done?
   See btrfs scrub start
> - also, I ran the command while some of the btrfs subvolumes were
> mounted (as well as the btrfs_root/ of course), does this impact on
> the balance job?
   No.
> - the mounted btrfs devices were mounted using -o
> space_cache,inode_cache but the btrfs_root/ was not, also, does this
> impact on the balance job?
   No.
> - about those options, a few months ago, I oftent had
> btrfs-cache-1/btrfs-endio-met processes taking some cpu/hd time.  I
> was advised to mount -o space_cache,inode_cache, which seems to have
> quiet the processes down.  Are those options still necessary now?
   No, once you''ve mounted with them once (and had the caches rebuilt)
they''re not necessary to use any more.
> - as the job took 60+ hours but the CPU rarely went above 10%, the
> computer seemed still usable.  I left it do its job of course but
> could I have accessed or write anything on the subvolumes while the
> balance was running and if yes, would this have any impact on the
> filesystem?
   Absolutely, yes, you could have done. It would probably be slower
than normal to access the files while the balance is happening,
because the balance is using up I/O bandwidth, but other than that
there should be no impact.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ==  PGP
key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                        --- argc, argv, argh! ---

Hugo Mills posted on Mon, 07 Oct 2013 13:06:23 +0100 as excerpted:
> On Mon, Oct 07, 2013 at 01:45:29PM +0200, Laurent Humblet wrote:
>> I have added 2x2Tb to my existing 2x2Tb raid 1 btrfs filesystem and
>> then ran a balance:
>> 
>> # btrfs filesystem show
>>         Total devices 4 FS bytes used 1.74TB
>>         devid    3 size 1.82TB used 0.00 path /dev/sdd
>>         devid    4 size 1.82TB used 0.00 path /dev/sde
>>         devid    2 size 1.82TB used 1.75TB path /dev/sdc
>>         devid    1 size 1.82TB used 1.75TB path /dev/sdb
>> # btrfs filesystem balance btrfs_root/
>> # btrfs filesystem show
>>         Total devices 4 FS bytes used 1.74TB
>>         devid    3 size 1.82TB used 892.00GB path /dev/sdd
>>         devid    4 size 1.82TB used 892.00GB path /dev/sde
>>         devid    2 size 1.82TB used 891.03GB path /dev/sdc
>>         devid    1 size 1.82TB used 891.04GB path /dev/sdb
>> 
>> It took 59 hours to complete the balance.
Hugo''s answers are great and he and the wiki are where I learned most
of
what I know about btrfs.  I''ve just a few details to add.

First, the wiki site in case you weren''t aware of it:

https://btrfs.wiki.kernel.org

Second, in case you didn''t know but as you can surmise from the numbers
above, note that btrfs raid1 mode with greater than two devices is *NOT* 
the same as common or mdadm raid1 in a similar case.  Unlike mdadm which 
mirrors the data/metadata N-ways, four devices means four copies, on 
btrfs only two copies of the data/metadata are kept regardless of the 
number of devices.  That''s why adding two additional devices to the
first
two halved the usage on each device, after the balance was done.
>> I checked on a couple of files and all seems fine but I have some
>> questions:
>> - is there some kind of ''overall filesystem health/integrity
check''
>> that I should do on the filesystem now that the balance is done?
> 
>    See btrfs scrub start
btrfs scrub is great.  It verifies checksums and copies data from any 
good copy available if any checksums fail.  Of course raid1 gives you 
that extra copy to copy from in case one /is/ bad, and I''ve had scrub
fix
things up after a bad shutdown or two. =:^)  (Well, it was really suspend/
resume cycles that failed to resume properly that scrub fixed for me, but 
anyway...)
>> - also, I ran the command while some of the btrfs subvolumes were
>> mounted (as well as the btrfs_root/ of course), does this impact on the
>> balance job?
> 
>    No.
> 
>> - the mounted btrfs devices were mounted using -o
>> space_cache,inode_cache but the btrfs_root/ was not, also, does this
>> impact on the balance job?
> 
>    No.
AFAIK, btrfs doesn''t have separate filesystem-specific mount option 
handling yet.  IOW, if one subvolume''s mounted with cache, they all
will
be.  However, non-filesystem specific options such as ro/rw, should still 
be subvolume specific.  I''m not actually sure about noatime vs.
relatime,
but I''d guess that''s per subvolume too, since it''s a
general filesystem
option not specific to btrfs.

Talking about noatime... you didn''t mention it, but it''s
definitely
recommended with btrfs, especially if you''re doing regular
snapshotting,
as atime updates may be a big part of the snapshot unique metadata 
otherwise.
>> - about those options, a few months ago, I oftent had
>> btrfs-cache-1/btrfs-endio-met processes taking some cpu/hd time.  I was
>> advised to mount -o space_cache,inode_cache, which seems to have quiet
>> the processes down.  Are those options still necessary now?
> 
>    No, once you''ve mounted with them once (and had the caches
rebuilt)
> they''re not necessary to use any more.
Hugo, this might be worth noting.  AFAIK, space_cache is now actually the 
default, from sometime before 3.10, AFAIK, as that''s when I setup my 
btrfs filesystems here, never actually invoking space_cache on them at 
all, but a check of /proc/mounts said they''re using it anyway.  And the
problems with a bad space_cache preventing mount seem to have been fixed 
now -- btrfs appears to automatically fix a bad space_cache on mount, 
now, instead of failing the mount as it did a year and a half ago when I 
first tried btrfs, and decided it wasn''t yet ready for me, in part due
to
problems such as that.

Inode_cache, however, was not the default yet, at least not with 3.10, 
and in at least one thread since 3.10 I''ve seen the devs recommend 
against using it unless it''s actually needed.  I''ve not seen
the details
explained, but I''ve interpreted that to mean that inode_cache is much 
more a special-case option not recommended for everyone, while 
space_cache, as I said, seems to be stable enough it''s being enabled by
default even without being specifically invoked the first time, as used 
to be necessary.
>> - as the job took 60+ hours but the CPU rarely went above 10%, the
>> computer seemed still usable.  I left it do its job of course but could
>> I have accessed or write anything on the subvolumes while the balance
>> was running and if yes, would this have any impact on the filesystem?
> 
>    Absolutely, yes, you could have done. It would probably be slower
> than normal to access the files while the balance is happening, because
> the balance is using up I/O bandwidth, but other than that there should
> be no impact.
Agreed.

The only misgiving I''d have about using a filesystem while
it''s being
balanced is that (as Hugo mentioned) that will of course split the 
available I/O bandwidth, which means both the balance and whatever else 
you''re doing will take longer than they would otherwise.  (For those 
familiar with it, this is very much like mdraid during a sync/resync 
operation.)

And of course I''d put off doing anything that might have a chance of 
crashing the system, since while btrfs should recover just fine after a 
crash even in the middle of a balance, it /is/ still an experimental 
filesystem with bugs regularly being found and fixed, and I''d not wish
to
test fate.  I''d not be overly worried if a crash does happen in the 
middle of a balance, but I''d not want to do anything that''d
make it more
likely...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html