Sean Son
2019-Feb-13 01:05 UTC
[CentOS] /boot partition running out of space randomly. Please help!
Hello all First off, I am running Oracle Linux 7.6 on a Hyper-V 2016 VM for a customer. I know this is not an Oracle Linux mailling list, but because Oracle Linux and CentOS are so similar, to an extent, I figured why not ask on here because someone MIGHT know the answer.. Here is the issue. I have a 600MB /boot partition allocated on a UEFI system. The /boot/efi partition is on a separate EFI partition. Recently, I noticed that this system has been crashing every few minutes and when I checked the disk space, I noticed that the /boot partition has zero free space available. I removed all of the old kernels and left the running kernel in place, in hopes that will free up some space. It freed up about 50MB or so, but then the system would crash again. After I would reboot the VM to bring the system back up, I ran a df -h /boot, and the results were reporting ZERO disk space again for the /boot partition.. It makes absolutely no sense how a partition which is generally static UNLESS you move something into it, is running out of space after space has been manually freed up in the partition! What boggles me even more is that when I do an ls -lh /boot, the file systems do not add up to 600M (well 594M) at all. See below: df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.8G 0 2.8G 0% /dev tmpfs 2.8G 0 2.8G 0% /dev/shm tmpfs 2.8G 8.5M 2.8G 1% /run tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup /dev/mapper/VolGroup00-LogVolRoot 30G 19G 12G 63% / /dev/sda2 594M 594M 0 100% /boot /dev/sda1 238M 9.7M 229M 5% /boot/efi /dev/mapper/VolGroup00-LogVolHome 3.3G 415M 2.9G 13% /home tmpfs 565M 0 565M 0% /run/user/54321 tmpfs 565M 0 565M 0% /run/user/1000 ]$ ls -lh /boot total 92M -rw-r--r-- 1 root root 179K Dec 12 22:52 config-4.14.35-1844.0.7.el7uek.x86_64 drwx------ 3 root root 16K Dec 31 1969 efi drwx------. 2 root root 21 Feb 8 15:55 grub2 -rw-------. 1 root root 54M Aug 28 12:31 initramfs-0-rescue-0287c4db206d4a9abe14f750b9091a01.img -rw------- 1 root root 22M Dec 21 17:24 initramfs-4.14.35-1844.0.7.el7uek.x86_64.img -rw-r--r-- 1 root root 329K Dec 12 22:52 symvers-4.14.35-1844.0.7.el7uek.x86_64.gz -rw-r--r-- 1 root root 3.6M Dec 12 22:52 System.map-4.14.35-1844.0.7.el7uek.x86_64 -rwxr-xr-x. 1 root root 6.1M Aug 28 12:31 vmlinuz-0-rescue-0287c4db206d4a9abe14f750b9091a01 -rwxr-xr-x 1 root root 7.2M Dec 12 22:52 vmlinuz-4.14.35-1844.0.7.el7uek.x86_64 I have no idea what is going on here and why the space keeps filling up and the VM crashing! ANY and all help will be greatly appreciated! Thanks! I am running the following kernel: 4.14.35-1844.0.7.el7uek.x86_64 Thanks! Sean S.
Rob Kampen
2019-Feb-13 02:39 UTC
[CentOS] /boot partition running out of space randomly. Please help!
On 13/02/19 2:05 PM, Sean Son wrote:> Hello all > > First off, I am running Oracle Linux 7.6 on a Hyper-V 2016 VM for a > customer. I know this is not an Oracle Linux mailling list, but because > Oracle Linux and CentOS are so similar, to an extent, I figured why not ask > on here because someone MIGHT know the answer.. Here is the issue. I have > a 600MB /boot partition allocated on a UEFI system. The /boot/efi partition > is on a separate EFI partition. Recently, I noticed that this system has > been crashing every few minutes and when I checked the disk space, I > noticed that the /boot partition has zero free space available. I removed > all of the old kernels and left the running kernel in place, in hopes that > will free up some space. It freed up about 50MB or so, but then the system > would crash again. After I would reboot the VM to bring the system back up, > I ran a df -h /boot, and the results were reporting ZERO disk space again > for the /boot partition.. It makes absolutely no sense how a partition > which is generally static UNLESS you move something into it, is running out > of space after space has been manually freed up in the partition! What > boggles me even more is that when I do an ls -lh /boot, the file systems do > not add up to 600M (well 594M) at all. See below: > > df -h > Filesystem Size Used Avail Use% Mounted on > devtmpfs 2.8G 0 2.8G 0% /dev > tmpfs 2.8G 0 2.8G 0% /dev/shm > tmpfs 2.8G 8.5M 2.8G 1% /run > tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup > /dev/mapper/VolGroup00-LogVolRoot 30G 19G 12G 63% / > /dev/sda2 594M 594M 0 100% /boot > /dev/sda1 238M 9.7M 229M 5% /boot/efi > /dev/mapper/VolGroup00-LogVolHome 3.3G 415M 2.9G 13% /home > tmpfs 565M 0 565M 0% /run/user/54321 > tmpfs 565M 0 565M 0% /run/user/1000 > > ]$ ls -lh /boot > total 92M > -rw-r--r-- 1 root root 179K Dec 12 22:52 > config-4.14.35-1844.0.7.el7uek.x86_64 > drwx------ 3 root root 16K Dec 31 1969 efi > drwx------. 2 root root 21 Feb 8 15:55 grub2 > -rw-------. 1 root root 54M Aug 28 12:31 > initramfs-0-rescue-0287c4db206d4a9abe14f750b9091a01.img > -rw------- 1 root root 22M Dec 21 17:24 > initramfs-4.14.35-1844.0.7.el7uek.x86_64.img > -rw-r--r-- 1 root root 329K Dec 12 22:52 > symvers-4.14.35-1844.0.7.el7uek.x86_64.gz > -rw-r--r-- 1 root root 3.6M Dec 12 22:52 > System.map-4.14.35-1844.0.7.el7uek.x86_64 > -rwxr-xr-x. 1 root root 6.1M Aug 28 12:31 > vmlinuz-0-rescue-0287c4db206d4a9abe14f750b9091a01 > -rwxr-xr-x 1 root root 7.2M Dec 12 22:52 > vmlinuz-4.14.35-1844.0.7.el7uek.x86_64 > > I have no idea what is going on here and why the space keeps filling up and > the VM crashing! ANY and all help will be greatly appreciated! Thanks! > > I am running the following kernel: > 4.14.35-1844.0.7.el7uek.x86_64My stab in the dark is that the system is trying to write a crash / rescue image and there is not enough space. du --max-depth 1 is useful too.> > Thanks! > > Sean S. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos
Sean Son
2019-Feb-13 03:09 UTC
[CentOS] /boot partition running out of space randomly. Please help!
On Tue, Feb 12, 2019 at 9:39 PM Rob Kampen <rkampen at kampensonline.com> wrote:> On 13/02/19 2:05 PM, Sean Son wrote: > > Hello all > > > > First off, I am running Oracle Linux 7.6 on a Hyper-V 2016 VM for a > > customer. I know this is not an Oracle Linux mailling list, but because > > Oracle Linux and CentOS are so similar, to an extent, I figured why not > ask > > on here because someone MIGHT know the answer.. Here is the issue. I > have > > a 600MB /boot partition allocated on a UEFI system. The /boot/efi > partition > > is on a separate EFI partition. Recently, I noticed that this system has > > been crashing every few minutes and when I checked the disk space, I > > noticed that the /boot partition has zero free space available. I > removed > > all of the old kernels and left the running kernel in place, in hopes > that > > will free up some space. It freed up about 50MB or so, but then the > system > > would crash again. After I would reboot the VM to bring the system back > up, > > I ran a df -h /boot, and the results were reporting ZERO disk space again > > for the /boot partition.. It makes absolutely no sense how a partition > > which is generally static UNLESS you move something into it, is running > out > > of space after space has been manually freed up in the partition! What > > boggles me even more is that when I do an ls -lh /boot, the file systems > do > > not add up to 600M (well 594M) at all. See below: > > > > df -h > > Filesystem Size Used Avail Use% Mounted on > > devtmpfs 2.8G 0 2.8G 0% /dev > > tmpfs 2.8G 0 2.8G 0% /dev/shm > > tmpfs 2.8G 8.5M 2.8G 1% /run > > tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup > > /dev/mapper/VolGroup00-LogVolRoot 30G 19G 12G 63% / > > /dev/sda2 594M 594M 0 100% /boot > > /dev/sda1 238M 9.7M 229M 5% /boot/efi > > /dev/mapper/VolGroup00-LogVolHome 3.3G 415M 2.9G 13% /home > > tmpfs 565M 0 565M 0% /run/user/54321 > > tmpfs 565M 0 565M 0% /run/user/1000 > > > > ]$ ls -lh /boot > > total 92M > > -rw-r--r-- 1 root root 179K Dec 12 22:52 > > config-4.14.35-1844.0.7.el7uek.x86_64 > > drwx------ 3 root root 16K Dec 31 1969 efi > > drwx------. 2 root root 21 Feb 8 15:55 grub2 > > -rw-------. 1 root root 54M Aug 28 12:31 > > initramfs-0-rescue-0287c4db206d4a9abe14f750b9091a01.img > > -rw------- 1 root root 22M Dec 21 17:24 > > initramfs-4.14.35-1844.0.7.el7uek.x86_64.img > > -rw-r--r-- 1 root root 329K Dec 12 22:52 > > symvers-4.14.35-1844.0.7.el7uek.x86_64.gz > > -rw-r--r-- 1 root root 3.6M Dec 12 22:52 > > System.map-4.14.35-1844.0.7.el7uek.x86_64 > > -rwxr-xr-x. 1 root root 6.1M Aug 28 12:31 > > vmlinuz-0-rescue-0287c4db206d4a9abe14f750b9091a01 > > -rwxr-xr-x 1 root root 7.2M Dec 12 22:52 > > vmlinuz-4.14.35-1844.0.7.el7uek.x86_64 > > > > I have no idea what is going on here and why the space keeps filling up > and > > the VM crashing! ANY and all help will be greatly appreciated! Thanks! > > > > I am running the following kernel: > > 4.14.35-1844.0.7.el7uek.x86_64 > My stab in the dark is that the system is trying to write a crash / > rescue image and there is not enough space. du --max-depth 1 is useful too. > > > > Thanks! > > > > Sean S. > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > https://lists.centos.org/mailman/listinfo/centos > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centosHello Rob Thank you for the reply. What do you recommend I should do to prevent the crashing? Should I increase the /boot partition's disk space? I am worried that it will fill up again randomly like the current one is.. Should I create a new /boot partition? Thanks
Tony Mountifield
2019-Feb-13 09:07 UTC
[CentOS] /boot partition running out of space randomly. Please help!
In article <CAOKpjz9q41dr57t5mgG3wgqpbtgO82UgJYWQnmeQVR11xLsNeg at mail.gmail.com>, Sean Son <linuxmailinglistsemail at gmail.com> wrote:> Hello all > > First off, I am running Oracle Linux 7.6 on a Hyper-V 2016 VM for a > customer. I know this is not an Oracle Linux mailling list, but because > Oracle Linux and CentOS are so similar, to an extent, I figured why not ask > on here because someone MIGHT know the answer.. Here is the issue. I have > a 600MB /boot partition allocated on a UEFI system. The /boot/efi partition > is on a separate EFI partition. Recently, I noticed that this system has > been crashing every few minutes and when I checked the disk space, I > noticed that the /boot partition has zero free space available. I removed > all of the old kernels and left the running kernel in place, in hopes that > will free up some space. It freed up about 50MB or so, but then the system > would crash again. After I would reboot the VM to bring the system back up, > I ran a df -h /boot, and the results were reporting ZERO disk space again > for the /boot partition.. It makes absolutely no sense how a partition > which is generally static UNLESS you move something into it, is running out > of space after space has been manually freed up in the partition! What > boggles me even more is that when I do an ls -lh /boot, the file systems do > not add up to 600M (well 594M) at all. See below: > > df -h > Filesystem Size Used Avail Use% Mounted on > devtmpfs 2.8G 0 2.8G 0% /dev > tmpfs 2.8G 0 2.8G 0% /dev/shm > tmpfs 2.8G 8.5M 2.8G 1% /run > tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup > /dev/mapper/VolGroup00-LogVolRoot 30G 19G 12G 63% / > /dev/sda2 594M 594M 0 100% /boot > /dev/sda1 238M 9.7M 229M 5% /boot/efi > /dev/mapper/VolGroup00-LogVolHome 3.3G 415M 2.9G 13% /home > tmpfs 565M 0 565M 0% /run/user/54321 > tmpfs 565M 0 565M 0% /run/user/1000 > > ]$ ls -lh /boot > total 92M > -rw-r--r-- 1 root root 179K Dec 12 22:52 > config-4.14.35-1844.0.7.el7uek.x86_64 > drwx------ 3 root root 16K Dec 31 1969 efi > drwx------. 2 root root 21 Feb 8 15:55 grub2 > -rw-------. 1 root root 54M Aug 28 12:31 > initramfs-0-rescue-0287c4db206d4a9abe14f750b9091a01.img > -rw------- 1 root root 22M Dec 21 17:24 > initramfs-4.14.35-1844.0.7.el7uek.x86_64.img > -rw-r--r-- 1 root root 329K Dec 12 22:52 > symvers-4.14.35-1844.0.7.el7uek.x86_64.gz > -rw-r--r-- 1 root root 3.6M Dec 12 22:52 > System.map-4.14.35-1844.0.7.el7uek.x86_64 > -rwxr-xr-x. 1 root root 6.1M Aug 28 12:31 > vmlinuz-0-rescue-0287c4db206d4a9abe14f750b9091a01 > -rwxr-xr-x 1 root root 7.2M Dec 12 22:52 > vmlinuz-4.14.35-1844.0.7.el7uek.x86_64 > > I have no idea what is going on here and why the space keeps filling up and > the VM crashing! ANY and all help will be greatly appreciated! Thanks!Firstly, to see the space taken by everything on /boot without including the sub-mount /boot/efi, do this: # du -axk /boot Then if that doesn't account for all/most of the space, see if there are any processes holding a deleted file open: # fuser -m /boot Like you, I don't know what might be trying to fill up /boot when you are not installing a new kernel. Cheers Tony -- Tony Mountifield Work: tony at softins.co.uk - http://www.softins.co.uk Play: tony at mountifield.org - http://tony.mountifield.org
Gordon Messmer
2019-Feb-15 23:02 UTC
[CentOS] /boot partition running out of space randomly. Please help!
On 2/12/19 5:05 PM, Sean Son wrote:> I have no idea what is going on here and why the space keeps filling up and > the VM crashing! ANY and all help will be greatly appreciated! Thanks!What does "crashing" mean in this case?? Can you explain that in more detail?? A system crash shouldn't write anything into /boot, nor should a full /boot cause a system crash. As far as finding out what is using the space: umount /boot/efi find /boot/ -mount -printf "%s %p\n" | sort -n (It's safe to unmount /boot/efi when you're not modifying your grub.cfg)