Xen users - Jul 2006 - Error: Device 2065 (vbd) could not be connected ... is mounted in a guest domain

This was a bit interesting... I was moving a domain off an old xen 2.0 box
to a newer (and more stable) 3.0.0 system. I accidentally copied the
/etc/fstab off the old box over the version I had updated for the new xen3
box... which caused a couple reboots from single user mode. Somewhere
along the way xen (xenstored I think) got confused and wouldn''t let me
have one of my partitions. The error message being:

   Device /dev/vg1/domain-dns-data is mounted in a guest domain,
   and so cannot be mounted now.

but given that I hadn''t launched any new domains other than the one I
trying to get to start, this seemed clearly bogus.

To cut a long story short, I learned about the xenstore* commands and
though I''d mention them on the list.

   WARNING.  WARNING.  WARNING.  WARNING.
   --------------------------------------

   I''m sure you can use these to blow your feet off. I''m no
expert

   --------------------------------------

xenstore-list seems to be a crude equivalent of /bin/ls
e.g.
   [root@xen5 scripts]# xenstore-list /
   local

   [root@xen5 scripts]# xenstore-list /local
   domain

   [root@xen5 scripts]# xenstore-list /local/domain
   0
   1
   2

You can see we''re walking down a psuedo filesystem. At this point it
forks to show the 3 domains running on this box. This list is
actually important, because there is no domain # 6, but we will
see later that there is a device "connected" to domain 6, which is
what was causing my problem.

If you go down the dom0 tree, you get things like this:

   [root@xen5 scripts]# xenstore-list /local/domain/0/backend/vbd/1
   2049
   2065
   2081

which is a list of the block devices for dom1

my problem is to some degree shown here:

   [root@xen5 log]# xenstore-list /local/domain/0/backend/vbd
   1
   2
   6

Here is where we see that dom0 has a backend device for domains
1, 2, and the phantom domain 6

As a quick aside, the proper/more traditional view into this realm
comes from xm block-list DOMID

[root@xen5 log]# xm block-list  palladium
   (2049 ((virtual-device 2049) (backend-id 0) (state 4) (backend
   /local/domain/0/backend/vbd/1/2049) (ring-ref 8) (event-channel 9)))
   (2065 ((virtual-device 2065) (backend-id 0) (state 4) (backend
   /local/domain/0/backend/vbd/1/2065) (ring-ref 9) (event-channel 10)))
   (2081 ((virtual-device 2081) (backend-id 0) (state 4) (backend
   /local/domain/0/backend/vbd/1/2081) (ring-ref 10) (event-channel 11)))

which matches the view I showed above of the block devices for
dom1

when you get far enough down into this pseudo filesystem you hit
the equivalent of files, and you need to use xenstore-read to see
what is in those nodes. Here is an example that dumps all the
parameters for /local/domain/0/backend/vbd/1/2081

   root@xen5 xenstore]# D=/local/domain/0/backend/vbd/1/2081;  for d
   in `xenstore-list $D`; do echo -n "$d: "; xenstore-read $D/$d; done

   domain: palladium
   frontend: /local/domain/1/device/vbd/2081
   dev: sdc1
   state: 4
   params: /dev/vg1/pd-raid2
   mode: w
   frontend-id: 1
   type: phy
   physical-device: fd:4
   hotplug-status: connected
   sectors: 125829120
   info: 0
   sector-size: 512

Anyhow, returning to my issue... With the phantom domain 6...

   [root@xen5 xenstore]# xenstore-list  /local/domain/0/backend/vbd/6
   2065
   [root@xen5 xenstore]# xenstore-list /local/domain/0/backend/vbd/6/2065
   physical-device
   hotplug-status

   [root@xen5 xenstore]# xenstore-read
/local/domain/0/backend/vbd/6/2065/physical-device
   fd:6

   [root@xen5 xenstore]# xenstore-read
/local/domain/0/backend/vbd/6/2065/hotplug-status
   connected


fd:6 happens to match my target device (e.g 253,6)

   [root@xen5 xen]# ll /dev/mapper/vg1-domain--dns--data
   brw-rw----  1 root disk 253, 6 Jul  3 22:21 /dev/mapper/vg1-domain--dns--data

So that seems to pretty clearly be my problem. WARNING, so far we have
just been poking at thinks, and looking around...

let''s delete the node for that bogus backend device...

   [root@xen5 xenstore]# xenstore-rm /local/domain/0/backend/vbd/6/2065

and what does xenstore-list say?

   [root@xen5 xenstore]# xenstore-list /local/domain/0/backend/vbd/6/2065
   xenstore-list: could not list path /local/domain/0/backend/vbd/6/2065

good, it seems to be gone, and voila, my "xm create" command is
no longer bitching about "Device /dev/vg1/domain-dns-data is mounted
in a guest domain..."

Pity that it took me more than an hour to work through it all, and
I had the service down for a more than to couple of minutes it
should have taken to sync the filesystems :(

Anyhow, I hope someone else finds this usefull. This was with xen
3.0.0, hopefully 3.0.2 is more robust about tracking device
assignments. AFAIK, it should actually have been simple enough to find
this issue, as the device was "attached" to a non-existant
domain, and inconsistent assignments like that should be easy
enough to find, but I haven''t taken the time to build scripts to
try to handle that or make the process more friendly. (hhmm, 3.0.2
has a xenstore-ls which seems to be a tool which walks through the
xenstore-list output in a much more friendly manner... :)

My apologies if this is all documented and I''m just clued out
because I don''t do texinfo type docs

-Tom



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users