Today I suddenly have two VMs that have read only file systems. The host is
CentOS 6, as are the two VMs with this problem.
The first symptom was on a new VM I installed ISPConfig onto. I got through the
entire process with only a dependency issue between php-pecl_apc and
php-accelerator.
After completing the installation I noticed some funny things, but I assumed it
might be the addition of quotas and remounting with quotas on. so I didn't
think much of it and rebooted the VM.
It failed to reboot because the file system should not be switched to
read-write. Since it was a new VM and installing ISPConfig was an experiment, I
just wiped it with the intention of starting over.
While I was creating another clone of a CentOS 6 image on the host, I looked
into one of the other VMs running on that host, which has been up and running
for 47 days. Same problem, without rebooting. For example running yum give
this:
[root at dev log]# yum update
Loaded plugins: fastestmirror, presto
Cannot open logfile /var/log/yum.log
Could not create lock at /var/run/yum.pid: [Errno 30] Read-only file system:
'/var/run/yum.pid'
Another app is currently holding the yum lock; waiting for it to exit...
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 254, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 103, in main
show_lock_owner(e.pid, logger)
File "/usr/share/yum-cli/utils.py", line 106, in show_lock_owner
ps = get_process_info(pid)
File "/usr/share/yum-cli/utils.py", line 61, in get_process_info
if (not os.path.exists("/proc/%d/status" % pid) or
TypeError: %d format: a number is required, not str
------------------------------------
And running mount gives this:
[root at dev log]# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/web on /NFS/web type none (rw,bind)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/etc/named on /var/named/chroot/etc/named type none (rw,bind)
/etc/named.rfc1912.zones on /var/named/chroot/etc/named.rfc1912.zones type none
(rw,bind)
/usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind)
/etc/named.iscdlv.key on /var/named/chroot/etc/named.iscdlv.key type none
(rw,bind)
/etc/named on /var/named/chroot/etc/named type none (rw,bind)
/usr/lib64/bind on /var/named/chroot/usr/lib64/bind type none (rw,bind)
mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
It's possible that information reported by mount(8) is not
up to date. For actual information about system mount points
check the /proc/mounts file.
-------------------------------
The VM is running, serving web pages and responding to DNS queries, but it is
clear, given my earlier experience with the ISPConfig machine, that I won't
be able to reboot it until I figure out the problem.
Now that I am looking at the output from the mount command I wonder where all
those named related mounts came from. Could it be webmin. Both VMs have webmin
installed. Mostly to allow be to configure bind, since system-config-bind is no
more.
Anybody have any idea what happened, or better yet, any ideas on how to fix
this?
Emmett