xen discuss - Oct 2007 - panic on domu install with b75a

While trying to install a b75a domu under Linux Xen 3.1 (running Debian Xen 3.1)
the installer panics while trying to create an ufs filesystem (after
partitioning).

Festplatte (c0d0) wird konfiguriert...
        - Fdisk-Partitionstabelle wird erstellt...
        - Solaris-Festplattenbezeichnung (VTOC) wird erstellt...

UFS-Dateisysteme werden erstellt und gepr?ft...
        - / wird erstellt(c0d0s0)...

panic[cpu0]/thread=d6008400: pfn_to_mfn(PFN_INVALID) not allowed

d54b4c8c unix:pfn_to_mfn+39 (ffffffff)
d54b4ca0 unix:pa_to_ma+1a (fffff000, fff)
d54b4cf0 xdf:xdf_prepare_rreq+175 (d3c52540, d6017650,)
d54b4d28 xdf:xdf_iostart+8f (d3c52540, d3c52540,)
d54b4d5c xdf:xdf_lb_rdwr+e7 (d4acb2a8, 1, 0, 9c,)
d54b4e00 xdf:xdf_ioctl+a9 (3040040, 422, d54b4)
d54b4e20 genunix:cdev_ioctl+2e (3040040, 422, d54b4)
d54b4f30 specfs:spec_fsync+e2 (d666bd80, 10, d49f7)
d54b4f64 genunix:fop_fsync+3d (d666bd80, 10, d49f7)
d54b4f84 genunix:fdsync+2a (6, 10, 445, d54b4fa)

panic: entering debugger (continue to save dump)
Loaded modules: [ scsi_vhci crypto neti ptm xpv_psm ufs mpt zfs uhci hook lofs 
ip logindmux usba specfs md random sctp arp ]
kmdb: target stopped at:
kmdb_enter+0xa: pushl  %eax

Is there anything I could try? (I changed the type of virtual hard disk from LVM
to image file, but without any change.)

Thanks,
Peter
 
 
This message posted from opensolaris.org

On Wed, Oct 31, 2007 at 02:13:47AM -0700, Peter Siering wrote:
> While trying to install a b75a domu under Linux Xen 3.1 (running Debian Xen
> 3.1) the installer panics while trying to create an ufs filesystem (after
> partitioning). 
> 
> Festplatte (c0d0) wird konfiguriert...
>         - Fdisk-Partitionstabelle wird erstellt...
>         - Solaris-Festplattenbezeichnung (VTOC) wird erstellt...
> 
> UFS-Dateisysteme werden erstellt und gepr?ft...
>         - / wird erstellt(c0d0s0)...
> 
> panic[cpu0]/thread=d6008400: pfn_to_mfn(PFN_INVALID) not allowed
> 
> d54b4c8c unix:pfn_to_mfn+39 (ffffffff)
> d54b4ca0 unix:pa_to_ma+1a (fffff000, fff)
> d54b4cf0 xdf:xdf_prepare_rreq+175 (d3c52540, d6017650,)
Can you file a bug?

thanks
john

Similar problem with b76 on Debian...

Preparing system for Solaris install

Configuring disk (c0d0)
        - Creating Fdisk partition table
        - Creating Solaris disk label (VTOC)

Creating and checking UFS file systems
        - Creating / (c0d0s0)

panic[cpu0]/thread=d33a8a00: pfn_to_mfn(PFN_INVALID) not allowed

d3440c8c unix:pfn_to_mfn+39 (ffffffff)
d3440ca0 unix:pa_to_ma+1a (fffff000, fff)
d3440cf0 xdf:xdf_prepare_rreq+175 (d2aa8700, d3f96ca0,)
d3440d28 xdf:xdf_iostart+8f (d2aa8700, d2aa8700,)
d3440d5c xdf:xdf_lb_rdwr+e7 (d2be62a8, 1, 0, 9c,)
d3440e00 xdf:xdf_ioctl+a9 (3040040, 422, d3440)
d3440e20 genunix:cdev_ioctl+2e (3040040, 422, d3440)
d3440f30 specfs:spec_fsync+e2 (d46059c0, 10, d2b35)
d3440f64 genunix:fop_fsync+3d (d46059c0, 10, d2b35)
d3440f84 genunix:fdsync+2a (6, 10, 9461, d3440f)

syncing file systems... done
dumping to /dev/dsk/c0d0s1, offset 427819008, content: kernel

panic[cpu0]/thread=d33a8a00: pfn_to_mfn(PFN_INVALID) not allowed
dump aborted: please record the above information!
rebooting...
WARNING: xdf_polled_io: timeout
 
 
This message posted from opensolaris.org

hexdump of the image file returns:

00000150  00 cd 10 b4 00 cd 16 61  c3 00 00 00 00 7c 00 00  |.......a.....|..|
00000160  80 00 43 61 6e 27 74 20  72 65 61 64 20 67 65 6f  |..Can''t
read geo|
00000170  6d 65 74 72 79 4e 6f 20  61 63 74 69 76 65 20 70  |metryNo active p|
00000180  61 72 74 69 74 69 6f 6e  43 61 6e 27 74 20 72 65 
|artitionCan''t re|
00000190  61 64 20 50 42 52 42 61  64 20 50 42 52 20 73 69  |ad PBRBad PBR si|
000001a0  67 21 21 21 4c 42 41 43  48 53 00 00 00 00 00 00  |g!!!LBACHS......|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 80 41  |...............A|

Do I need to prepare the image file with something special before passing it to
Solaris?
 
 
This message posted from opensolaris.org

> Similar problem with b76 on Debian...
> 
> Creating and checking UFS file systems
>         - Creating / (c0d0s0)
>
> panic[cpu0]/thread=d33a8a00: pfn_to_mfn(PFN_INVALID) not allowed
> 
> d3440c8c unix:pfn_to_mfn+39 (ffffffff)
> d3440ca0 unix:pa_to_ma+1a (fffff000, fff)
How much memory is allocated for the Solaris domU?

I guess the linux dom0 is running on a 32 bit hypervisor / kernel?
 
 
This message posted from opensolaris.org

> hexdump of the image file returns:
> 
> 00000150  00 cd 10 b4 00 cd 16 61  c3 00 00 00 00 7c 00 00 
|.......a.....|..|
> 00000160  80 00 43 61 6e 27 74 20  72 65 61 64 20 67 65 6f 
|..Can''t read geo|
> 00000170  6d 65 74 72 79 4e 6f 20  61 63 74 69 76 65 20 70  |metryNo active
p|
> 00000180  61 72 74 69 74 69 6f 6e  43 61 6e 27 74 20 72 65 
|artitionCan''t re|
> 00000190  61 64 20 50 42 52 42 61  64 20 50 42 52 20 73 69  |ad PBRBad PBR
si|
> 000001a0  67 21 21 21 4c 42 41 43  48 53 00 00 00 00 00 00 
|g!!!LBACHS......|
> 000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 80 41 
|...............A|
Looks like the standard Solaris master boot record code...
> Do I need to prepare the image file with something
> special before passing it to Solaris?
No.
 
 
This message posted from opensolaris.org

Yes, it "was" running a 32 bit hypervisor with 1.5G of memory.

I have upgraded Xen from 3.0.3 to 3.1 and it seems to have solved the issue.
I''m watching the Solaris Initial Install as I typed this message.
 
 
This message posted from opensolaris.org

Looks like it''s a Xen problem, the disk is fine.
 
 
This message posted from opensolaris.org

Peter Siering wrote:
> panic [cpu0]/thread=d6008400: pfn_to_mfn(PFN_INVALID) not allowed
> 
> d54b4c8c unix:pfn_to_mfn+39 (ffffffff)
> d54b4ca0 unix:pa_to_ma+1a (fffff000, fff)
> d54b4cf0 xdf:xdf_prepare_rreq+175 (d3c52540, d6017650,)
> d54b4d28 xdf:xdf_iostart+8f (d3c52540, d3c52540,)
> d54b4d5c xdf:xdf_lb_rdwr+e7 (d4acb2a8, 1, 0, 9c,)
> d54b4e00 xdf:xdf_ioctl+a9 (3040040, 422, d54b4)
Hmm, apparently it crashes when the xdf driver
gets an DKIOCFLUSHWRITECACHE ioctl (0x422).

The "xdf:xdf_lb_rdwr+e7 (d4acb2a8, 1, 0, 9c,)" call
looks like this one:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/xen/io/xdf.c#1105

			rc = xdf_lb_rdwr(vdp->xdf_dip, TG_WRITE,
			    vdp->xdf_cache_flush_block, xdf_flush_block,
			    DEV_BSIZE, (void *)dev);


Parameter TG_WRITE (== 1) is OK, parameter 
xdf_flush_block == DEFAULT_FLUSH_BLOCK == 0x9c is OK, too.

But parameter "vdp->xdf_cache_flush_block" appears to be a NULL
pointer.

Peter, could you retry the domU install under kmdb, and when it
panics, print the first arg to either xdf_iostart() or xdf_prepare_rreq()
as a "xdf_t" structure, e.g. for the original panic

    d3c52540::print xdf_t


Are the structure members xdf_flush_mem / xdf_cache_flush_block 0?
 
 
This message posted from opensolaris.org

> While trying to install a b75a domu under Linux Xen
> 3.1 (running Debian Xen 3.1) the installer panics
> while trying to create an ufs filesystem (after
> partitioning). 
> 
> Festplatte (c0d0) wird konfiguriert...
>         - Fdisk-Partitionstabelle wird erstellt...
>         - Solaris-Festplattenbezeichnung (VTOC) wird erstellt...
> 
> UFS-Dateisysteme werden erstellt und gepr?ft...
>         - / wird erstellt(c0d0s0)...
> panic [cpu0]/thread=d6008400: pfn_to_mfn(PFN_INVALID) not allowed
> 
> d54b4c8c unix:pfn_to_mfn+39 (ffffffff)
> d54b4ca0 unix:pa_to_ma+1a (fffff000, fff)
> d54b4cf0 xdf:xdf_prepare_rreq+175 (d3c52540, d6017650,)
> d54b4d28 xdf:xdf_iostart+8f (d3c52540, d3c52540,)
> d54b4d5c xdf:xdf_lb_rdwr+e7 (d4acb2a8, 1, 0, 9c,)
> d54b4e00 xdf:xdf_ioctl+a9 (3040040, 422, d54b4)
> d54b4e20 genunix:cdev_ioctl+2e (3040040, 422, d54b4)
> d54b4f30 specfs:spec_fsync+e2 (d666bd80, 10, d49f7)
> d54b4f64 genunix:fop_fsync+3d (d666bd80, 10, d49f7)
> d54b4f84 genunix:fdsync+2a (6, 10, 445, d54b4fa)
> 
> panic: entering debugger (continue to save dump)
> Loaded modules: [ scsi_vhci crypto neti ptm xpv_psm ufs mpt zfs uhci hook
lofs ip logindmux usba specfs md random sctp arp ]
> kmdb: target stopped at:
> kmdb_enter+0xa: pushl  %eax
> 
> Is there anything I could try? (I changed the type of
> virtual hard disk from LVM to image file, but without
> any change.)

I guess the problem is that the linux dom0 block backend
driver doesn''t report / support the "feature-barrier"
property,
and the Solaris block front driver has a problem when
"feature-barrier" isn''t there...


After starting the solaris domU install, try to list the 
backend vbd devices for the domU from the linux
dom0, something like (assuming the solaris domU is
using domid 2):

# xenstore-ls /local/domain/0/backend/vbd/2
0 = ""
 domain = "solaris"
 frontend = "/local/domain/2/device/vbd/0"
 uuid = "dfb8f5e2-b313-56ee-2900-9c7c91f9044b"
 dev = "0"
 state = "4"
 params = "/dev/zvol/dsk/hdd/sol-domu"
 mode = "w"
 online = "1"
 frontend-id = "2"
 type = "phy"
 physical-device = "0:0"
 hotplug-status = "connected"
 feature-barrier = "1"  
<<<<<<<<<<<<<<<<<<<<<<<<<<
 info = "0"
 sector-size = "512"
 sectors = "4194304"
 instance = "0"


Is there a "feature-barrier" property?


If not, try the following workaround:
- boot the Solaris domU kernel with options "-kd"
  e.g.
  extra = ''/platform/i86xpv/kernel/amd64/unix -kd - nowin -B
install_media=cdrom''

- in kmdb

  ::bp xdf`_init
  ::cont
  xdf_barrier_flush_disable/W 1
  ::cont


When I force a failure when looking up "feature-barrier"
in Solaris'' xdf_post_connect() function (32-bit kernel),
I get the following Solaris domU kernel error message:

    NOTICE: xdf@0: failed to read feature-barrier
    xdf@0: 4194304 blocks


And the next time Solaris'' domU xdf driver is told to
perform an ioctl(DKIOCFLUSHWRITECACHE), domU panics:


# newfs /dev/rdsk/c0d0s1
newfs: construct a new file system /dev/rdsk/c0d0s1: (y/n)? y
/dev/rdsk/c0d0s1:       786432 sectors in 192 cylinders of 128 tracks, 32 sector
s
        384.0MB in 12 cyl groups (16 c/g, 32.00MB/g, 15360 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 65600, 131168, 196736, 262304, 327872, 393440, 459008, 524576, 590144,
 655712, 721280

panic[cpu0]/thread=d6b2dc00: pfn_to_mfn(PFN_INVALID) not allowed

d6b69c8c unix:pfn_to_mfn+39 (ffffffff)
d6b69ca0 unix:pa_to_ma+1a (fffff000, fff)
d6b69cf0 xdf:xdf_prepare_rreq+175 (d484fc40, d6f61668,)
d6b69d28 xdf:xdf_iostart+8f (d484fc40, d484fc40,)
d6b69d5c xdf:xdf_lb_rdwr+e7 (d58bad10, 1, 0, 9c,)
d6b69e00 xdf:xdf_ioctl+a9 (3040001, 422, d6b69)
d6b69e20 genunix:cdev_ioctl+2e (3040001, 422, d6b69)
d6b69f30 specfs:spec_fsync+e2 (d6a54b40, 10, d6a4e)
d6b69f64 genunix:fop_fsync+3d (d6a54b40, 10, d6a4e)
d6b69f84 genunix:fdsync+2a (7, 10, ebf9, d6b69f)

panic: entering debugger (no dump device, continue to reboot)
Loaded modules: [ crypto neti ptm zfs uhci hook lofs ip logindmux usba md random
 sctp arp ]
kmdb: target stopped at:
kmdb_enter+0xa: pushl  %eax



xdf_ioctl(DKIOCFLUSHWRITECACHE) must check if
vdp->xdf_feature_barrier != 0, before trying to call:

                        rc = xdf_lb_rdwr(vdp->xdf_dip, TG_WRITE,
                            vdp->xdf_cache_flush_block, xdf_flush_block,
                            DEV_BSIZE, (void *)dev);


diff -r 58581342bfd9 usr/src/uts/common/xen/io/xdf.c
--- a/usr/src/uts/common/xen/io/xdf.c   Wed Nov 21 10:43:35 2007 -0800
+++ b/usr/src/uts/common/xen/io/xdf.c   Fri Nov 23 00:03:42 2007 +0100
@@ -1099,12 +1099,13 @@ xdf_ioctl(dev_t dev, int cmd, intptr_t a
                if (vdp->xdf_flush_supported) {
                        rc = xdf_lb_rdwr(vdp->xdf_dip, TG_WRITE,
                            NULL, 0, 0, (void *)dev);
-               } else {
-                       if (xdf_barrier_flush_disable)
-                               return (ENOTTY);
+               } else if (vdp->xdf_feature_barrier &&
+                   !xdf_barrier_flush_disable) {
                        rc = xdf_lb_rdwr(vdp->xdf_dip, TG_WRITE,
                            vdp->xdf_cache_flush_block, xdf_flush_block,
                            DEV_BSIZE, (void *)dev);
+               } else {
+                       rc = ENOTTY;
                }
                if ((mode & FKIOCTL) && (dkc != NULL) &&
                    (dkc->dkc_callback != NULL)) {
 
 
This message posted from opensolaris.org