All,
I''m having some issues with Xen disk IO being almost unbearable.
I''ve read
some other threads suggesting possibly the hardware did not get recognized
properly but I''m not seeing where that might be in dmesg.
First here are my hdparm results:
# hdparm -tT /dev/hda
/dev/hda:
Timing cached reads: 1476 MB in 2.00 seconds = 737.13 MB/sec
Timing buffered disk reads: 10 MB in 3.44 seconds = 2.90 MB/sec
# hdparm -tT /dev/hdb
/dev/hdb:
Timing cached reads: 1480 MB in 2.01 seconds = 738.15 MB/sec
Timing buffered disk reads: 6 MB in 3.15 seconds = 1.91 MB/sec
Here are basically the drive type on another machine:
# hdparm -tT /dev/hda
/dev/hda:
Timing cached reads: 1132 MB in 2.00 seconds = 565.80 MB/sec
Timing buffered disk reads: 170 MB in 3.01 seconds = 56.57 MB/sec
Anyway ... here are some relivent lines from dmesg. Can anyone suggest what
might be causing the problem?
ide_disk: no version for "struct_module" found: kernel tainted.
Probing IDE interface ide0...
hda: WDC WD400JB-00ENA0, ATA DISK drive
hdb: WDC WD2000JB-32FUA0, ATA DISK drive
Probing IDE interface ide1...
hdc: JLMS XJ-HD166S, ATAPI CD/DVD-ROM drive
hdd: HL-DT-ST GCE-8523B, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 78165360 sectors (40020 MB) w/8192KiB Cache, CHS=65535/16/63
hda: cache flushes not supported
hda: hda1 hda2
hdb: max request size: 512KiB
hdb: 390721968 sectors (200049 MB) w/8192KiB Cache, CHS=24321/255/63
hdb: cache flushes supported
hdb: hdb1
SCSI subsystem initialized
ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
ACPI: PCI Interrupt 0000:01:08.0[A] -> Link [APC3] -> GSI 18 (level, high)
->
IRQ 16
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
<Adaptec 2940 Ultra SCSI adapter>
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-devel@redhat.com
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
hdc: ATAPI 48X DVD-ROM drive, 512kB Cache
Uniform CD-ROM driver Revision: 3.20
hdd: ATAPI 52X CD-ROM CD-R/RW drive, 2048kB Cache
NFORCE2: IDE controller at PCI slot 0000:00:09.0
NFORCE2: chipset revision 162
NFORCE2: not 100% native mode: will probe irqs later
NFORCE2: BIOS didn''t set cable bits correctly. Enabling workaround.
NFORCE2: 0000:00:09.0 (rev a2) UDMA133 controller
NFORCE2: port 0x01f0 already claimed by ide0
NFORCE2: port 0x0170 already claimed by ide1
NFORCE2: neither IDE port enabled (BIOS)
forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.49.
ACPI: PCI Interrupt Link [APCH] enabled at IRQ 22
ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [APCH] -> GSI 22 (level, high)
->
IRQ 17
PCI: Setting latency timer of device 0000:00:04.0 to 64
eth0: forcedeth.c: subsystem: 01043:80a7 bound to 0000:00:04.0
ACPI: PCI Interrupt Link [APCJ] enabled at IRQ 21
ACPI: PCI Interrupt 0000:00:06.0[A] -> Link [APCJ] -> GSI 21 (level, high)
->
IRQ 18
PCI: Setting latency timer of device 0000:00:06.0 to 64
I even tried to set DMA on the drives:
# hdparm -d1 /dev/hda
/dev/hda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Operation not permitted
using_dma = 0 (off)
Any help is greatly appreciated ... I''d love this machine to be useable
:)
--
Lonny
--
"The glass is neither half empty nor half full ... it is twice as big as it
needs to be"
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
> First here are my hdparm results: > > # hdparm -tT /dev/hda > > /dev/hda: > Timing cached reads: 1476 MB in 2.00 seconds = 737.13 MB/sec > Timing buffered disk reads: 10 MB in 3.44 seconds = 2.90 MB/sec > > > # hdparm -tT /dev/hdb > > /dev/hdb: > Timing cached reads: 1480 MB in 2.01 seconds = 738.15 MB/sec > Timing buffered disk reads: 6 MB in 3.15 seconds = 1.91 MB/sec >Follow up ... here are the drives using the stock non-xen kernel ... I have some dmesg to weed through and figure out what driver I need. # hdparm -tT /dev/hda /dev/hda: Timing cached reads: 1396 MB in 2.00 seconds = 697.06 MB/sec Timing buffered disk reads: 142 MB in 3.02 seconds = 47.03 MB/sec [root@localhost ~]# hdparm -tT /dev/hdb /dev/hdb: Timing cached reads: 1448 MB in 2.00 seconds = 722.66 MB/sec Timing buffered disk reads: 136 MB in 3.01 seconds = 45.13 MB/sec Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx NFORCE2: IDE controller at PCI slot 0000:00:09.0 NFORCE2: chipset revision 162 NFORCE2: not 100% native mode: will probe irqs later NFORCE2: BIOS didn''t set cable bits correctly. Enabling workaround. NFORCE2: 0000:00:09.0 (rev a2) UDMA133 controller ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA Probing IDE interface ide0... hda: WDC WD400JB-00ENA0, ATA DISK drive hdb: WDC WD2000JB-32FUA0, ATA DISK drive Using cfq io scheduler ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 Probing IDE interface ide1... hdc: JLMS XJ-HD166S, ATAPI CD/DVD-ROM drive hdd: HL-DT-ST GCE-8523B, ATAPI CD/DVD-ROM drive ide1 at 0x170-0x177,0x376 on irq 15 Probing IDE interface ide2... Probing IDE interface ide3... Probing IDE interface ide4... Probing IDE interface ide5... hda: max request size: 128KiB hda: 78165360 sectors (40020 MB) w/8192KiB Cache, CHS=65535/16/63, UDMA(100) hda: cache flushes not supported hda: hda1 hda2 hdb: max request size: 1024KiB hdb: 390721968 sectors (200049 MB) w/8192KiB Cache, CHS=24321/255/63, UDMA(100) hdb: cache flushes supported hdb: hdb1 hdc: ATAPI 48X DVD-ROM drive, 512kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.20 hdd: ATAPI 52X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33) Possibly the top line? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ok so based on previous suggestions I did a mkinitrd: mkinitrd -f --preload=ide-disk --preload=ide-generic /boot/initrd-2.6.16.33-xen_3.0.4.1.img 2.6.16.33-xen_3.0.4.1 Upon looking further in the /lib/modules of the xen kernel I''m not seeing any other potential candidates to try preloading. When I made the initrd without the preload parameters, my drives weren''t found. Does anyone have any suggestions on what I should put into the initrd or specify explicitly in order to have my drives recognized properly? I installed xen from src rpm ... should I maybe just compile trial/error a kernel that might work? Thanks! Lonny _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 2/20/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote:> Ok so based on previous suggestions I did a mkinitrd: > > mkinitrd -f --preload=ide-disk --preload=ide-generic > /boot/initrd-2.6.16.33-xen_3.0.4.1.img 2.6.16.33-xen_3.0.4.1 > > Upon looking further in the /lib/modules of the xen kernel I''m not seeing any > other potential candidates to try preloading. When I made the initrd without > the preload parameters, my drives weren''t found. Does anyone have any > suggestions on what I should put into the initrd or specify explicitly in > order to have my drives recognized properly?This is the driver: NFORCE2: IDE controller at PCI slot 0000:00:09.0 Run "kmodule"? "cat /etc/modprobe.conf" might hint at something also. jerry -- "Pay no attention to that man behind the curtain!" _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> On 2/20/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote: >> Ok so based on previous suggestions I did a mkinitrd: >> >> mkinitrd -f --preload=ide-disk --preload=ide-generic >> /boot/initrd-2.6.16.33-xen_3.0.4.1.img 2.6.16.33-xen_3.0.4.1 >> >> Upon looking further in the /lib/modules of the xen kernel I''m not seeing >> any >> other potential candidates to try preloading. When I made the initrd without >> the preload parameters, my drives weren''t found. Does anyone have any >> suggestions on what I should put into the initrd or specify explicitly in >> order to have my drives recognized properly? > > This is the driver: > NFORCE2: IDE controller at PCI slot 0000:00:09.0 > > Run "kmodule"? > "cat /etc/modprobe.conf" might hint at something also.Thanks again Jerry, I had tried both of those previously and found nothing related to the nforce chipset: # kmodule NETWORK forcedeth SCSI aic7xxx AUDIO snd-intel8x0 USB ehci-hcd USB ohci-hcd USB ohci-hcd And modprobe.conf makes no reference either. The only deduction is the driver is compiled into the original CentOS kernel and not included as a module nor is it included in the ramdisk image as one. I''m assuming I have to compile a Xen kernel from source. I can''t seem to find any kind of archive that points to where I can get the sources or a method for building them. Does anyone have a link? or possibly another suggestion? Thanks again!! -- Lonny _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>>> On 2/20/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote: >>>> Ok so based on previous suggestions I did a mkinitrd: >>>> >>>> mkinitrd -f --preload=ide-disk --preload=ide-generic >>>> /boot/initrd-2.6.16.33-xen_3.0.4.1.img 2.6.16.33-xen_3.0.4.1 >>>> >>>> Upon looking further in the /lib/modules of the xen kernel I''m not seeing >>>> any >>>> other potential candidates to try preloading. When I made the initrd >>>> without >>>> the preload parameters, my drives weren''t found. Does anyone have any >>>> suggestions on what I should put into the initrd or specify explicitly in >>>> order to have my drives recognized properly? >>> >>> This is the driver: >>> NFORCE2: IDE controller at PCI slot 0000:00:09.0 >>> >>> Run "kmodule"? >>> "cat /etc/modprobe.conf" might hint at something also. >> >> Thanks again Jerry, I had tried both of those previously and found nothing >> related to the nforce chipset: >> # kmodule >> NETWORK forcedeth >> SCSI aic7xxx >> AUDIO snd-intel8x0 >> USB ehci-hcd >> USB ohci-hcd >> USB ohci-hcd >> >> And modprobe.conf makes no reference either. The only deduction is the >> driver >> is compiled into the original CentOS kernel and not included as a module nor >> is it included in the ramdisk image as one. I''m assuming I have to compile a >> Xen kernel from source. I can''t seem to find any kind of archive that points >> to where I can get the sources or a method for building them. >> >> Does anyone have a link? or possibly another suggestion? >> Thanks again!! > >I''m currently doing a make world on the xen sources and it''s building the kernel ... something tells me this is going to make no difference, it looks like exactly the same output from when I did a src.rpm install and doesn''t prompt you for any options in the kernel, suggesting all I''m doing is taking a longer road to the same desitnation. I need to figure out how to build a custom xen kernel and see if I can figure out which IDE driver I need (by name in the kernel build) -- "The glass is neither half empty nor half full ... it is twice as big as it needs to be" _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 2/21/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote:> >> Thanks again Jerry, I had tried both of those previously and found nothing > >> related to the nforce chipset:How about grep NFORCE2 /boot/config* jerry -- "Pay no attention to that man behind the curtain!" _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> On 2/21/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote: >> >> Thanks again Jerry, I had tried both of those previously and found >> nothing >> >> related to the nforce chipset: > > How about > grep NFORCE2 /boot/config*I found the only references to NFORCE in: /boot/config-2.6.9-42.0.8.EL:CONFIG_I2C_NFORCE2=m /boot/config-2.6.9-42.EL:CONFIG_I2C_NFORCE2=m Mind you I toasted my Xen kernel and am building one from scratch, removing the clean kernel build from the make. I think the driver is actually amd74xx so I tried building it into the kernel but then when I try to create the ramdisk image it errors asying that that module doesn''t exist .... Soooo I''m going to try to build the module and preload it when I create the initrd ... and then try to boot it remotely so if it dies I wont be able to fix it until later lol I''ll keep posting any progress. -- Lonny _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>>> On 2/21/07, Lonny Selinger <lonny@bangtherockstogether.net> wrote: >>>> >> Thanks again Jerry, I had tried both of those previously and found >>>> nothing >>>> >> related to the nforce chipset: >>> >>> How about >>> grep NFORCE2 /boot/config* >> >> I found the only references to NFORCE in: >> /boot/config-2.6.9-42.0.8.EL:CONFIG_I2C_NFORCE2=m >> /boot/config-2.6.9-42.EL:CONFIG_I2C_NFORCE2=m >> >> Mind you I toasted my Xen kernel and am building one from scratch, >> removing >> the clean kernel build from the make. I think the driver is actually >> amd74xx >> so I tried building it into the kernel but then when I try to create the >> ramdisk image it errors asying that that module doesn''t exist .... Soooo >> I''m >> going to try to build the module and preload it when I create the initrd >> ... >> and then try to boot it remotely so if it dies I wont be able to fix it >> until >> later lol >> >> I''ll keep posting any progress. >> >> -- >> Lonny >> > > > If you build the module into the kernel then it does not have to be > present in the initrd RAM disk.True ... which is why I couldn''t figure out why the ramdisk wouldn''t build ... it kept erroring out looking for a module that wasn''t there. Here''s the scoop ... everything is working now :D I rebuild a custom xen kernel ... basically I think for nothing (other than it was a newer version then what I had with the src.rpms) The key was the preload for the ide module required for my drives. After a VERY long brain fart I actually checked the driver I was using for my IDE devices in proc: Non Xen Kernel: # ls /proc/ide amd74xx drivers hda hdb hdc hdd ide0 ide1 and now WITH Xen Kernel # ls /proc/ide amd74xx drivers hda hdb hdc hdd ide0 ide1 after doing an hdparm: # for i in hda hdb; do> hdparm -tT /dev/$i; > done/dev/hda: Timing cached reads: 1380 MB in 2.00 seconds = 689.47 MB/sec Timing buffered disk reads: 142 MB in 3.02 seconds = 47.02 MB/sec /dev/hdb: Timing cached reads: 1452 MB in 2.00 seconds = 725.20 MB/sec Timing buffered disk reads: 138 MB in 3.02 seconds = 45.68 MB/sec Everything is MUCH happier. THanks everyone for your help!! -- Lonny -- "The glass is neither half empty nor half full ... it is twice as big as it needs to be" _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello. I saw your post:> /dev/hda: > Timing cached reads: 1380 MB in 2.00 seconds = 689.47 MB/sec > Timing buffered disk reads: 142 MB in 3.02 seconds = 47.02 MB/sec > > /dev/hdb: > Timing cached reads: 1452 MB in 2.00 seconds = 725.20 MB/sec > Timing buffered disk reads: 138 MB in 3.02 seconds = 45.68 MB/secI wondered about hdparm -tT, which I didn''t know. So I tries it myself:>hdparm -tT /dev/hd?/dev/hda: Timing cached reads: 266 MB in 2.00 seconds = 132.84 MB/sec Timing buffered disk reads: 144 MB in 3.01 seconds = 47.82 MB/sec /dev/hdb: Timing cached reads: 490 MB in 2.00 seconds = 244.92 MB/sec Timing buffered disk reads: 164 MB in 3.01 seconds = 54.48 MB/sec /dev/hdc: Timing cached reads: 506 MB in 2.00 seconds = 252.41 MB/sec Timing buffered disk reads: 50 MB in 3.02 seconds = 16.54 MB/sec /dev/hdd: Timing cached reads: 494 MB in 2.01 seconds = 246.32 MB/sec Timing buffered disk reads: 74 MB in 3.00 seconds = 24.64 MB/sec Looks to me, my hda and hdb are way slower für cached reads only, while hdc and hdd are way slower on both tests. Can someone please explain me, what these tests mean and what I could possibly do. Regards, Steffen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2007-02-21 at 21:33 +0100, Steffen Heil wrote:> Hello. > > I saw your post: > > /dev/hda: > > Timing cached reads: 1380 MB in 2.00 seconds = 689.47 MB/sec > > Timing buffered disk reads: 142 MB in 3.02 seconds = 47.02 MB/sec > > > > /dev/hdb: > > Timing cached reads: 1452 MB in 2.00 seconds = 725.20 MB/sec > > Timing buffered disk reads: 138 MB in 3.02 seconds = 45.68 MB/sec > > I wondered about hdparm -tT, which I didn''t know. > So I tries it myself:I tried that as well as a quick and dirty benchmark to see how closely it matched on dom-0 to a dom-u. I noticed that cached reads were slightly slower on the guest (by about 8 - 12%), but buffered reads were actually faster than dom-0 (by a small, but noticeable 3 - 5%). Never could quite explain it either, was just content that it was close and similar to readings on standard stock Linux systems. I''m kind of curious about this too. I mostly understand it, but not well enough to explain it to someone who does not, which means I don''t fully understand it but would like to :) Best, --Tim _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 21 Feb 2007 at 21:33, Steffen Heil wrote: [...]> >hdparm -tT /dev/hd? > > /dev/hda: > Timing cached reads: 266 MB in 2.00 seconds = 132.84 MB/sec > Timing buffered disk reads: 144 MB in 3.01 seconds = 47.82 MB/sec[...] Just for curiosity: As I''d expect XEN to buffer "virtual physical reads", did anybody repeat the commend twice in succession? That is hdparm -t /dev/hda hdparm -t /dev/hda I have made two tests here: First SLES9 without XEN (hardware RAID1, SCSI): /dev/sda: Timing buffered disk reads: 182 MB in 3.03 seconds = 60.13 MB/sec /dev/sda: Timing buffered disk reads: 162 MB in 3.01 seconds = 53.76 MB/sec Then SLES10 x86_64 with XEN 3.0.2 (hardware RAID1, SAS): /dev/sda1: Timing buffered disk reads: 202 MB in 3.01 seconds = 67.17 MB/sec /dev/sda1: Timing buffered disk reads: 170 MB in 3.03 seconds = 56.17 MB/sec While the performance doesn''t look verby bad, I think I can get about 150MB/s from the SAS RAID without XEN. The CPUs involved were a Pentium-4 with 2.4GHz, and an Opteron at 2.6GHz. Regards, Ulrich _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users