Just found my issue. After I removed the cephfs mounts it worked! I will debug ceph. I assumed because I could touch files on mounted cephfs it was working. Now virsh list works! thanks jerry Lars Kellogg-Stedman> On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote: >> Recently both virsh stopped talking to the libvirtd. Both stopped within >> a >> few days of each other. > > I've run into exactly the same problem. > > I'm running libvirt (libvirt-9.0.0-3.fc38.x86_64) on Fedora 38. On > Fedora, libvirtd is configured by default to use socket activation and > is run with the `--timeout 120` option. > > After some recent upgrades, I'm seeing the exact same symptoms that > Jerry described -- virsh commands simply get stuck at same call to > `poll()`. > > It looks like libvirtd is either crashing or failing to start, because > when virsh is in this state the `libvirtd` process isn't running. This > makes it *sound* like a systemd problem, but I'm not seeing errors > anywhere -- either from libvirtd or from systemd. > > I've worked around the problem locally by re-configuring libvirtd to > run persistently rather than using socket activation: > > systemctl disable --now libvirtd{,-ro,-admin}.socket > > cat > /etc/systemd/system/libvirtd.service.d/override.conf <<EOF > [Service] > EnvironmentFile> EOF > > systemctl restart libvirtd > > Package versions in case this helps correlate something: > > - libvirt-9.0.0-3.fc38.x86_64 > - systemd-253.5-1.fc38.x86_64 > - kernel-6.3.6-200.fc38.x86_64 > > Libvirt uri: qemu:///system > > -- > Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {irc,twitter,github} > http://blog.oddbit.com/ | N1LKS > >
Just a brief update: As soon as I umount cephfs virsh is able to talk to libvirtd. I tested the cephfs with: df (no problem) dd if=/dev/zero of=/cephstorage/a.img bs=1G count=1 oflag=dsync (no problem created random1G file, no I/O issues.) After mounting cephfs and restarting libvirtd "virsh" hangs again. Obviously virsh and libvirtd don't like the cephfs mount. I am just starting to debug the potential problem with cephfs and libvirtd/virsh. I originally noted when this problem occurred on two hypervisors the problem occurred a couple days a part which matched some updates that took place. I have not tried rolling back patches yet. I am curious if anyone uses cephfs filesystem and had similar problems recently. I will update the form if I find a solution. Thanks jerry Jerry Buburuz> > Just found my issue. > > After I removed the cephfs mounts it worked! > > I will debug ceph. > > I assumed because I could touch files on mounted cephfs it was working. > > Now virsh list works! > > thanks > jerry > > Lars Kellogg-Stedman >> On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote: >>> Recently both virsh stopped talking to the libvirtd. Both stopped >>> within >>> a >>> few days of each other. >> >> I've run into exactly the same problem. >> >> I'm running libvirt (libvirt-9.0.0-3.fc38.x86_64) on Fedora 38. On >> Fedora, libvirtd is configured by default to use socket activation and >> is run with the `--timeout 120` option. >> >> After some recent upgrades, I'm seeing the exact same symptoms that >> Jerry described -- virsh commands simply get stuck at same call to >> `poll()`. >> >> It looks like libvirtd is either crashing or failing to start, because >> when virsh is in this state the `libvirtd` process isn't running. This >> makes it *sound* like a systemd problem, but I'm not seeing errors >> anywhere -- either from libvirtd or from systemd. >> >> I've worked around the problem locally by re-configuring libvirtd to >> run persistently rather than using socket activation: >> >> systemctl disable --now libvirtd{,-ro,-admin}.socket >> >> cat > /etc/systemd/system/libvirtd.service.d/override.conf <<EOF >> [Service] >> EnvironmentFile>> EOF >> >> systemctl restart libvirtd >> >> Package versions in case this helps correlate something: >> >> - libvirt-9.0.0-3.fc38.x86_64 >> - systemd-253.5-1.fc38.x86_64 >> - kernel-6.3.6-200.fc38.x86_64 >> >> Libvirt uri: qemu:///system >> >> -- >> Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {irc,twitter,github} >> http://blog.oddbit.com/ | N1LKS >> >> > >
On 6/12/23 20:17, Jerry Buburuz wrote:> > Just found my issue. > > After I removed the cephfs mounts it worked! > > I will debug ceph. > > I assumed because I could touch files on mounted cephfs it was working. > > Now virsh list works!Out of curiosity. Do you perhaps have a storage pool defined over cephfs? I can see two possible sources for the problem: 1) autostarted storage pool that makes libvirt mount cephfs, or 2) a storage pool defined over a path where cephfs is mounted. The problem with 1) is obvious (in fact it's not specific to ceph, if it was NFS/iSCSI and the server wasn't responding then libvirtd would just hang). The problem with 2) is that for some types of storage pools ('dir' typically) libvirt assumes they are always 'running'. And proceeds to enumerate volumes in that pool (i.e. files under the dir). And if there's a stale mount point, this might stuck libvirtd. But again, this is not limited to ceph, any network FS might do this. Michal