thr3ads.net - xen discuss - DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Card [Oct 2007]

If this information is useful, please help other people find it:
Share via:

Kugutsumen

2007-Oct-10 08:50 UTC

DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Card

Hi,

The original thread was posted in the device drivers and zfs discussion.

http://www.opensolaris.org/jive/thread.jspa?messageID=162462&#162462

However the problem seems to only occur under the xen dom0: I boot the kernel
without the hypervisor, the problem completely disappear.

Upgrading the raid card firmware didn''t help, turning off write-back
didn''t help (actually made it worse.)

This is a 4 core amd 64 machine with 8 gig of ram. I remember there was an old
discussion on xen users related to dma on machine with big memory configuration.

Here is the description of the problem.

Just as I create a ZFS pool and copy the root partition to it.... the
performance seems to be really good then suddenly the system hangs all my
sesssions and displays on the console:

Oct 10 00:23:28 sunrise arcmsr: WARNING: arcmsr0: dma map got ''no
resources''
Oct 10 00:23:28 sunrise arcmsr: WARNING: arcmsr0: dma allocate fail 
Oct 10 00:23:28 sunrise arcmsr: WARNING: arcmsr0: dma allocate fail free scsi
hba

AR1100 was running in JBOD mode and write-back caching under Solaris Express Xen
Drop b66 07/07. Never had problem this setup under linux under heavy load.

ARECA SATA-SAS RAID Host Adapter Driver(i386) 1.20.00.13,REV=2006.08.14

Updated to latest firmware 1.43-70417 ... 

same problem..

WARNING: arcmsr0: dma map got ''no resources'' 

WARNING: arcmsr0: dma allocate fail 

WARNING: arcmsr0: dma allocate fail free scsi hba pkt 

WARNING: arcmsr0: dma map got ''no resources'' 

WARNING: arcmsr0: dma allocate fail 

WARNING: a

SunOS sunrise 5.11 xen-nv66-2007-06-24 i86pc i386 i86pc

prtconf -pv 
System Configuration:  Sun Microsystems  i86pc
Memory size: 8192 Megabytes
System Peripherals (PROM Nodes):

Node 0x000001
    bios-boot-device:  ''81''
    stdout:  00000000
    name:  ''i86pc''

    Node 0x000002
        existing:  00d5d000.00000000.02368001.00000000
        name:  ''ramdisk''

    Node 0x000003
        bus-type:  ''isa''
        device_type:  ''isa''
        name:  ''isa''

    Node 0x000004
        device_type:  ''pci''
        reg:  00000000.00000000.00000000
        #size-cells:  00000002
        #address-cells:  00000003
        name:  ''pci''

        Node 0x000005
            reg:  00000800.00000000.00000000.00000000.00000000
            compatible: ''pci1166,36.0'' +
''pci1166,36'' + ''pciclass,060400'' +
''pciclass,0604''
            model:  ''PCI-PCI bridge''
            ranges: 
82000000.00000000.ff400000.82000000.00000000.ff400000.00000000.00100000.c2000000.00000000.f6300000.c2000000.00000000.f6300000.00000000.00800000
            bus-range:  00000001.00000003
            #size-cells:  00000002
            #address-cells:  00000003
            device_type:  ''pci''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            unit-address:  ''1''
            class-code:  00060400
            revision-id:  00000000
            vendor-id:  00001166
            device-id:  00000036
            pcix-capid-pointer:  00000090
            name:  ''pci1166,36''

            Node 0x000018
                slot-names:  00000008.746f6c53.00000031
                reg:  00016800.00000000.00000000.00000000.00000000
                compatible: ''pci1166,104.c0'' +
''pci1166,104'' + ''pciclass,060400'' +
''pciclass,0604''
                model:  ''PCI-PCI bridge''
                ranges: 
82000000.00000000.ff400000.82000000.00000000.ff400000.00000000.00100000.c2000000.00000000.f6300000.c2000000.00000000.f6300000.00000000.00800000
                bus-range:  00000002.00000003
                #size-cells:  00000002
                #address-cells:  00000003
                device_type:  ''pci''
                power-consumption:  00000001.00000001
                66mhz-capable:  
                devsel-speed:  00000001
                unit-address:  ''d''
                class-code:  00060400
                revision-id:  000000c0
                vendor-id:  00001166
                device-id:  00000104
                pcix-capid-pointer:  00000090
                name:  ''pci1166,104''

                Node 0x000019
                    reg:  00021800.00000000.00000000.00000000.00000000
                    compatible: ''pci8086,335.a'' +
''pci8086,335'' + ''pciclass,060400'' +
''pciclass,0604''
                    model:  ''PCI-PCI bridge''
                    ranges: 
82000000.00000000.ff400000.82000000.00000000.ff400000.00000000.00100000.c2000000.00000000.f6300000.c2000000.00000000.f6300000.00000000.00800000
                    bus-range:  00000003.00000003
                    #size-cells:  00000002
                    #address-cells:  00000003
                    device_type:  ''pci''
                    power-consumption:  00000001.00000001
                    66mhz-capable:  
                    fast-back-to-back:  
                    devsel-speed:  00000001
                    unit-address:  ''3''
                    class-code:  00060400
                    revision-id:  0000000a
                    vendor-id:  00008086
                    device-id:  00000335
                    pcix-capid-pointer:  000000f0
                    name:  ''pci8086,335''

                    Node 0x00001a
                        assigned-addresses: 
82037010.00000000.ff4ff000.00000000.00001000.c2037018.00000000.f6400000.00000000.00400000
                        reg: 
00037000.00000000.00000000.00000000.00000000.02037010.00000000.00000000.00000000.00001000.42037018.00000000.00000000.00000000.00400000
                        compatible: ''pci17d3,1110.17d3.1110.0''
+ ''pci17d3,1110.17d3.1110'' + ''pci17d3,1110''
+ ''pci17d3,1110.0'' + ''pci17d3,1110'' +
''pciclass,010400'' + ''pciclass,0104''
                        model:  ''RAID controller''
                        power-consumption:  00000001.00000001
                        66mhz-capable:  
                        devsel-speed:  00000001
                        interrupts:  00000001
                        max-latency:  00000000
                        min-grant:  00000080
                        subsystem-vendor-id:  000017d3
                        subsystem-id:  00001110
                        unit-address:  ''e''
                        class-code:  00010400
                        revision-id:  00000000
                        vendor-id:  000017d3
                        device-id:  00001110
                        pcix-capid-pointer:  000000e0
                        name:  ''pci17d3,1110''

        Node 0x000006
            reg:  00001000.00000000.00000000.00000000.00000000
            compatible: ''pci1166,205.1166.201.0'' +
''pci1166,205.1166.201'' + ''pci1166,201'' +
''pci1166,205.0'' + ''pci1166,205'' +
''pciclass,060000'' + ''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000201
            unit-address:  ''2''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001166
            device-id:  00000205
            name:  ''pci1166,201''

        Node 0x000007
            #size-cells:  00000000
            #address-cells:  00000001
            device_type:  ''pci-ide''
            assigned-addresses: 
81001110.00000000.000001f0.00000000.00000008.81001114.00000000.000003f6.00000000.00000001.81001118.00000000.00000170.00000000.00000008.8100111c.00000000.00000376.00000000.00000001.81001120.00000000.0000ffa0.00000000.00000010
            reg: 
00001100.00000000.00000000.00000000.00000000.81001110.00000000.000001f0.00000000.00000008.81001114.00000000.000003f6.00000000.00000001.81001118.00000000.00000170.00000000.00000008.8100111c.00000000.00000376.00000000.00000001.01001120.00000000.00000000.00000000.00000010
            compatible: ''pci1166,214.1166.214.0'' +
''pci1166,214.1166.214'' + ''pci1166,214'' +
''pci1166,214.0'' + ''pci1166,214'' +
''pciclass,01018a'' + ''pciclass,0101''
            model:  ''IDE controller''
            power-consumption:  00000001.00000001
            devsel-speed:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000214
            unit-address:  ''2,1''
            class-code:  0001018a
            revision-id:  00000000
            vendor-id:  00001166
            device-id:  00000214
            name:  ''pci-ide''

            Node 0x000008
                reg:  00000000
                name:  ''ide''

            Node 0x000009
                reg:  00000001
                name:  ''ide''

        Node 0x00000a
            reg:  00001200.00000000.00000000.00000000.00000000
            compatible: ''pci1166,234.1166.230.0'' +
''pci1166,234.1166.230'' + ''pci1166,230'' +
''pci1166,234.0'' + ''pci1166,234'' +
''pciclass,060100'' + ''pciclass,0601''
            model:  ''ISA bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000230
            unit-address:  ''2,2''
            class-code:  00060100
            revision-id:  00000000
            vendor-id:  00001166
            device-id:  00000234
            name:  ''pci1166,230''

        Node 0x00000b
            assigned-addresses: 
82001810.00000000.ff6b4000.00000000.00001000.81001814.00000000.0000e000.00000000.00000100
            reg: 
00001800.00000000.00000000.00000000.00000000.02001810.00000000.00000000.00000000.00001000.01001814.00000000.00000000.00000000.00000100
            compatible: ''pci1166,223.1166.223.1'' +
''pci1166,223.1166.223'' + ''pci1166,223'' +
''pci1166,223.1'' + ''pci1166,223'' +
''pciclass,0c0310'' + ''pciclass,0c03''
            model:  ''Universal Serial Bus OHCI compliant''
            power-consumption:  00000001.00000001
            66mhz-capable:  
            fast-back-to-back:  
            devsel-speed:  00000001
            interrupts:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000223
            unit-address:  ''3''
            class-code:  000c0310
            revision-id:  00000001
            vendor-id:  00001166
            device-id:  00000223
            name:  ''pci1166,223''

        Node 0x00000c
            assigned-addresses: 
82001910.00000000.ff6b5000.00000000.00001000.81001914.00000000.0000e400.00000000.00000100
            reg: 
00001900.00000000.00000000.00000000.00000000.02001910.00000000.00000000.00000000.00001000.01001914.00000000.00000000.00000000.00000100
            compatible: ''pci1166,223.1166.223.1'' +
''pci1166,223.1166.223'' + ''pci1166,223'' +
''pci1166,223.1'' + ''pci1166,223'' +
''pciclass,0c0310'' + ''pciclass,0c03''
            model:  ''Universal Serial Bus OHCI compliant''
            power-consumption:  00000001.00000001
            66mhz-capable:  
            fast-back-to-back:  
            devsel-speed:  00000001
            interrupts:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000223
            unit-address:  ''3,1''
            class-code:  000c0310
            revision-id:  00000001
            vendor-id:  00001166
            device-id:  00000223
            name:  ''pci1166,223''

        Node 0x00000d
            assigned-addresses: 
82001a10.00000000.ff6b6000.00000000.00001000.81001a14.00000000.0000e800.00000000.00000100
            reg: 
00001a00.00000000.00000000.00000000.00000000.02001a10.00000000.00000000.00000000.00001000.01001a14.00000000.00000000.00000000.00000100
            compatible: ''pci1166,223.1166.223.1'' +
''pci1166,223.1166.223'' + ''pci1166,223'' +
''pci1166,223.1'' + ''pci1166,223'' +
''pciclass,0c0320'' + ''pciclass,0c03''
            model:  ''Universal Serial Bus EHCI compliant''
            power-consumption:  00000001.00000001
            66mhz-capable:  
            fast-back-to-back:  
            devsel-speed:  00000001
            interrupts:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  00001166
            subsystem-id:  00000223
            unit-address:  ''3,2''
            class-code:  000c0320
            revision-id:  00000001
            vendor-id:  00001166
            device-id:  00000223
            name:  ''pci1166,223''

        Node 0x00000e
            assigned-addresses: 
82002010.00000000.ff680000.00000000.00020000.82002014.00000000.ff660000.00000000.00020000.81002018.00000000.0000dc00.00000000.00000040
            reg: 
00002000.00000000.00000000.00000000.00000000.02002010.00000000.00000000.00000000.00020000.02002014.00000000.00000000.00000000.00020000.01002018.00000000.00000000.00000000.00000040
            compatible: ''pci8086,1076.10f1.3870.5'' +
''pci8086,1076.10f1.3870'' + ''pci10f1,3870'' +
''pci8086,1076.5'' + ''pci8086,1076'' +
''pciclass,020000'' + ''pciclass,0200''
            model:  ''Ethernet controller''
            power-consumption:  00000001.00000001
            66mhz-capable:  
            devsel-speed:  00000001
            interrupts:  00000001
            max-latency:  00000000
            min-grant:  000000ff
            subsystem-vendor-id:  000010f1
            subsystem-id:  00003870
            unit-address:  ''4''
            class-code:  00020000
            revision-id:  00000005
            vendor-id:  00008086
            device-id:  00001076
            pcix-capid-pointer:  000000e4
            name:  ''pci10f1,3870''

        Node 0x00000f
            video-adapter-type:  ''svga''
            display-type:  ''color''
            assigned-addresses: 
c2003010.00000000.f8000000.00000000.04000000.82003014.00000000.ff6c0000.00000000.00040000.81003018.00000000.0000ec00.00000000.00000080.a1003000.00000000.000003b0.00000000.0000000c.a1003000.00000000.000003c0.00000000.00000020.82003000.00000000.000a0000.00000000.00020000
            reg: 
00003000.00000000.00000000.00000000.00000000.42003010.00000000.00000000.00000000.04000000.02003014.00000000.00000000.00000000.00040000.01003018.00000000.00000000.00000000.00000080.a1003000.00000000.000003b0.00000000.0000000c.a1003000.00000000.000003c0.00000000.00000020.82003000.00000000.000a0000.00000000.00020000
            compatible: ''pci18ca,20.18ca.20.0'' +
''pci18ca,20.18ca.20'' + ''pci18ca,20'' +
''pci18ca,20.0'' + ''pci18ca,20'' +
''pciclass,030000'' + ''pciclass,0300''
            model:  ''VGA compatible controller''
            power-consumption:  00000001.00000001
            66mhz-capable:  
            devsel-speed:  00000001
            max-latency:  00000000
            min-grant:  00000000
            subsystem-vendor-id:  000018ca
            subsystem-id:  00000020
            device_type:  ''display''
            unit-address:  ''6''
            class-code:  00030000
            revision-id:  00000000
            vendor-id:  000018ca
            device-id:  00000020
            name:  ''display''

        Node 0x000010
            reg:  0000c000.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1100.0'' +
''pci1022,1100'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''18''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001100
            name:  ''pci1022,1100''

        Node 0x000011
            reg:  0000c100.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1101.0'' +
''pci1022,1101'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''18,1''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001101
            name:  ''pci1022,1101''

        Node 0x000012
            reg:  0000c200.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1102.0'' +
''pci1022,1102'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''18,2''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001102
            name:  ''pci1022,1102''

        Node 0x000013
            reg:  0000c300.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1103.0'' +
''pci1022,1103'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''18,3''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001103
            name:  ''pci1022,1103''

        Node 0x000014
            reg:  0000c800.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1100.0'' +
''pci1022,1100'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''19''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001100
            name:  ''pci1022,1100''

        Node 0x000015
            reg:  0000c900.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1101.0'' +
''pci1022,1101'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''19,1''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001101
            name:  ''pci1022,1101''

        Node 0x000016
            reg:  0000ca00.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1102.0'' +
''pci1022,1102'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''19,2''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001102
            name:  ''pci1022,1102''

        Node 0x000017
            reg:  0000cb00.00000000.00000000.00000000.00000000
            compatible: ''pci1022,1103.0'' +
''pci1022,1103'' + ''pciclass,060000'' +
''pciclass,0600''
            model:  ''Host bridge''
            power-consumption:  00000001.00000001
            devsel-speed:  00000000
            max-latency:  00000000
            min-grant:  00000000
            unit-address:  ''19,3''
            class-code:  00060000
            revision-id:  00000000
            vendor-id:  00001022
            device-id:  00001103
            name:  ''pci1022,1103''
 
 
This message posted from opensolaris.org

Kugutsumen

2007-Oct-10 11:37 UTC

head link

Re: DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Ca

Could this be related to this old problem:

http://www.nabble.com/high-memory-dma-update:-up-against-a-wall-t140777.html

I am going to try to limit the memory to 4 gig on the xen dom0 and see what
happened.
 
 
This message posted from opensolaris.org

Mark Johnson

2007-Oct-10 11:57 UTC

head link

Re: DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Ca

Kugutsumen wrote:> Could this be related to this old problem:
> 
>
http://www.nabble.com/high-memory-dma-update:-up-against-a-wall-t140777.html
> 
> I am going to try to limit the memory to 4 gig on the xen dom0 and see what
happened.
You''ll actually want to limit it to < 4G since some of
that the 4G needs to be used for device registers, etc.

e.g. You can limit it to 3G with

kernel$ /boot/$ISADIR/xen.gz mem=3G com1=9600,8n1 console=com1


Although, I think we should be able to do a better
job allocating low memory... So if this solves your
problem, we''ll need to look at it some more...



MRJ





-- 
Mark Johnson <mark.johnson@sun.com>
Sun Microsystems, Inc.
(781) 442-0869

Kugutsumen

2007-Oct-10 12:36 UTC

head link

Re: dom0 dma issue on large memory system with Areca 1100 SATA [solved]

Solved!! :D

I limit the dom0 memory to 1024M, reserving 7 gig for my domU''s.

kernel$ /boot/$ISADIR/xen.gz dom0_mem=1024M com1=57600,8n1 console=com1

The areca driver doesn''t seem to support dma above the first 32bit
region.

I will send a new support ticket to  areca, I hope they can update their
drivers; they should also fix the install script so you can install them without
patching it on nevada.

There has been a lot of discussion in the past around this 4G limit issue: 

http://lists.xensource.com/archives/html/xen-devel/2005-06/msg00121.html
 
 
This message posted from opensolaris.org

xen discuss - Oct 2007 - DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Card

DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Card

Re: DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Ca

Re: DMA critical problems under Xen dom0 (b66) with Areca 1110 SATA Raid Ca

Re: dom0 dma issue on large memory system with Areca 1100 SATA [solved]