CentOS - Sep 2011 - LVM lvresize/lvextend requires some space in /etc to grow a logical volume?

Side note:

In CentOS-6, I noticed a new option in lvresize / lvextend:

       -r, --resizefs
              Resize underlying filesystem together with the logical
volume using
              fsadm(8).

Nice. Two steps (lvresize and resize2fs) can now be combined into one!
Works great.

But that has nothing to do with my question, just thought I'd share
the discovery.

=
Anyway, here's the real issue with LVM, at least in CentOS-6:

[root at jttest ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_main-lv_root
                       1548144   1548144         0 100% /

[root at jttest ~]# lvextend -L 2G /dev/vg_main/lv_root
  /etc/lvm/archive/.lvm_jttest.pp.local_5523_51321310: write error
failed: No space left on device
  Volume group "vg_main" metadata archive failed.
  /etc/lvm/cache/.cache.tmp: write error failed: No space left on device

So I can't extend a logical volume if there's no space in /etc?

Granted, I only need to delete like 2k worth of files in the same
logical volume as /etc to make the above work, but I don't recall
lvresize or lvextend requiring space in /etc to do a resize. Has this
always been the case? I don't have any 5.x servers handy to test this
with. Can anyone verify?

If this is new, this seems less-than-ideal. /etc itself rarely fills
up, so, in theory, I could make it its own LV, but that's even less
ideal. For now, I'll stick to deleting just enough to do the
extend/resize.

johnny

Am Montag, den 19.09.2011, 19:15 -0400 schrieb Johnny Tan:
> Anyway, here's the real issue with LVM, at least in CentOS-6:
> 
You would deal with the same issues in older lvm versions.
> [root at jttest ~]# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/vg_main-lv_root
>                        1548144   1548144         0 100% /
> 
Do i read that correctly and 1548144 1K Blocks are about 1,5G of Root
File System? Please tell me you have a seperated /var for log files.

If not, for production use, install a tool which gives you a warning
when reaching the 5% Level.
> [root at jttest ~]# lvextend -L 2G /dev/vg_main/lv_root
>   /etc/lvm/archive/.lvm_jttest.pp.local_5523_51321310: write error
> failed: No space left on device
>   Volume group "vg_main" metadata archive failed.
>   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
> 
> So I can't extend a logical volume if there's no space in /etc?
lvm tries to be helpfull if you ran into a power failure and can then
recover due his tmp files in /etc. 
> Granted, I only need to delete like 2k worth of files in the same
> logical volume as /etc to make the above work, but I don't recall
> lvresize or lvextend requiring space in /etc to do a resize. Has this
> always been the case? I don't have any 5.x servers handy to test this
> with. Can anyone verify?
> 
You will have the same problem, i think this procedure is the same in
all lvm2 versions, which seems a long time now.
> If this is new, this seems less-than-ideal. /etc itself rarely fills
> up, so, in theory, I could make it its own LV, but that's even less
> ideal. For now, I'll stick to deleting just enough to do the
> extend/resize.
> 
This is why the older unix guys always seperate stuff from / which could
fill up the root file system. ;) 

-- 

Stefan Held                      VI has only 2 Modes:
obi unixkiste org                The first one is for beeping all the time,
FreeNode: foo_bar                the second destroys the text.
---------------------------------------------------------------------------
perl -e'map{print pack
c,($|++?1:13)+ord,select$,,$,,$,,$|}split//,ESEL.$/'
---------------------------------------------------------------------------

On Mon, Sep 19, 2011 at 07:15:09PM -0400, Johnny Tan
wrote:
> Anyway, here's the real issue with LVM, at least in CentOS-6:
> 
> [root at jttest ~]# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/vg_main-lv_root
>                        1548144   1548144         0 100% /
> 
> [root at jttest ~]# lvextend -L 2G /dev/vg_main/lv_root
>   /etc/lvm/archive/.lvm_jttest.pp.local_5523_51321310: write error
> failed: No space left on device
>   Volume group "vg_main" metadata archive failed.
>   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
Many LVM modifications make a backup of the lvm metadata, which
typically resides in /etc/lvm.  You might try the -A n switch to
disable the automatic backups (after strongly heeding the advice
in man lvm; but as you said, on a test system, it's fine to play
with).  On a production system, if your / fills to 0 available blocks
you've probably got bigger problems.  :)  So I imagine it's probably not
a problem the LVM developers are too worried about (i.e., they may feel
that the -A switch is sufficient to handle it).

--keith

-- 
kkeller at wombat.san-francisco.ca.us

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL:
<http://lists.centos.org/pipermail/centos/attachments/20110919/1fb094c8/attachment-0002.sig>

On Mon, Sep 19, 2011 at 10:19 PM, Keith Keller
<kkeller at wombat.san-francisco.ca.us> wrote:
> On Mon, Sep 19, 2011 at 07:15:09PM -0400, Johnny Tan wrote:
>> Anyway, here's the real issue with LVM, at least in CentOS-6:
>>
>> [root at jttest ~]# df
>> Filesystem ? ? ? ? ? 1K-blocks ? ? ?Used Available Use% Mounted on
>> /dev/mapper/vg_main-lv_root
>> ? ? ? ? ? ? ? ? ? ? ? ?1548144 ? 1548144 ? ? ? ? 0 100% /
>>
>> [root at jttest ~]# lvextend -L 2G /dev/vg_main/lv_root
>> ? /etc/lvm/archive/.lvm_jttest.pp.local_5523_51321310: write error
>> failed: No space left on device
>> ? Volume group "vg_main" metadata archive failed.
>> ? /etc/lvm/cache/.cache.tmp: write error failed: No space left on
device
>
> Many LVM modifications make a backup of the lvm metadata, which
> typically resides in /etc/lvm. ?You might try the -A n switch to
> disable the automatic backups (after strongly heeding the advice
> in man lvm; but as you said, on a test system, it's fine to play
> with).
That did the trick:

[root at jttest~]# lvresize -L 2G /dev/vg_main/lv_root
  /etc/lvm/archive/.lvm_jttest.pp.local_1416_1029753972: write error
failed: No space left on device
  Volume group "vg_main" metadata archive failed.
  /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
[root at jttest~]# lvresize -A n -L 2G /dev/vg_main/lv_root
  Extending logical volume lv_root to 2.00 GiB
  WARNING: This metadata update is NOT backed up
  Logical volume lv_root successfully resized
  /etc/lvm/cache/.cache.tmp: write error failed: No space left on device

(And the subsequent resize worked too.)

Thank you for pointing out the -A flag (missed the fine print of
common options being in man lvm).

As you noted, I would probably never run into this situation in a
non-testing environment and, if I did, would have much bigger
problems. This probably explains why I never saw it previously. Good
to know.

johnny

Johnny Tan wrote:
<snip>
> Anyway, here's the real issue with LVM, at least in CentOS-6:
>
> [root at jttest ~]# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/vg_main-lv_root
>                        1548144   1548144         0 100% /
>
> [root at jttest ~]# lvextend -L 2G /dev/vg_main/lv_root
>   /etc/lvm/archive/.lvm_jttest.pp.local_5523_51321310: write error
> failed: No space left on device
>   Volume group "vg_main" metadata archive failed.
>   /etc/lvm/cache/.cache.tmp: write error failed: No space left on device
>
> So I can't extend a logical volume if there's no space in /etc?
<snip>
Um, just how do you have your system partitioned? I've never seen /etc on
its own partition; it's on the same partition as /... and if so, and
*that's* full, you've got a much bigger problem.

        mark