CentOS virt - Dec 2011 - New Guess OS Creation Problem

Hi All, 

I am running on CentOS Released 6.1 final. Been using and running Linux KVM
quite well for quite some time, something goes wrong after I perform yum
upgrade.

I created new VM yesterday without any problem, same exact installation
procedure, installed FreeBSD 8.2. I tried to create a new VM today after yum
upgrade, it's able to detect the hard disk, when I start commit FreeBSD 8.2
installation, it complains cannot write to disk as stated the error message
below.

block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)

Below is the log I capture from the VM log file

-- Log start -- 
2011-12-12 00:23:39.485: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/6
Using CPU model "cpu64-rhel6"

block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
-- Log End --

Below is the software version currently running;
gpxe-roms-qemu-0.9.7-6.7.el6.noarch
qemu-img-0.12.1.2-2.160.el6_1.8.x86_64
qemu-kvm-0.12.1.2-2.160.el6_1.8.x86_64
libvirt-client-0.8.7-18.el6_1.4.x86_64
libvirt-python-0.8.7-18.el6_1.4.x86_64
libvirt-0.8.7-18.el6_1.4.x86_64

Any of you having the problem as well?
I am planning to install CentOS as guest and see whether is has the same problem
as well. Thanks.

--
Paul Ooi

On Dec 12, 2011, at 6:56 PM, Stefan Hajnoczi wrote:
> On Sun, Dec 11, 2011 at 4:40 PM, takizo <paulooi at takizo.com>
wrote:
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>> char device redirected to /dev/pts/6
>> Using CPU model "cpu64-rhel6"
>> 
>> block I/O error in device 'drive-ide0-0-0': Invalid argument
(22)
> 
Hi Stefan, Thanks for responding. 
> Is /opt/cibai/database on a special filesystem or exotic storage setup?
It's an direct attached disk. 

/dev/sdb1             134G   30G   98G  23% /opt
> 
> Please check dmesg on the host for any kernel messages regarding I/O
errors.
I have checked dmesg, /var/log/messages and the one I pasted was from KVM
storage. Not much error I can see
> 
> Please try reading the file on the host to check whether the I/O error
> is happening on the host and not related to KVM:
> 
> $ dd if=/opt/cibai/database of=/dev/null iflag=direct
I have read the files. The file seems ok. 

dd if=/opt/vm/database.img of=/dev/null iflag=direct
52428800+0 records in52428800+0 records out
26843545600 bytes (27 GB) copied, 70.3967 s, 381 MB/s

I tried the installation again, same problem giving me IO error. Is there any
other to show more verbose logging?

2011-12-12 22:40:33.625: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
8490fea3-cb30-4ac5-2779-3681a77e7b03 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/vm/database.img,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:c8:1c:b9,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:3,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/7
Using CPU model "cpu64-rhel6"
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)
block I/O error in device 'drive-ide0-0-0': Invalid argument (22)

> 
> Stefan

On Dec 12, 2011, at 6:56 PM, Stefan Hajnoczi wrote:
> On Sun, Dec 11, 2011 at 4:40 PM, takizo <paulooi at takizo.com>
wrote:
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>> char device redirected to /dev/pts/6
>> Using CPU model "cpu64-rhel6"
>> 
>> block I/O error in device 'drive-ide0-0-0': Invalid argument
(22)
> 
> Is /opt/cibai/database on a special filesystem or exotic storage setup?
> 
> Please check dmesg on the host for any kernel messages regarding I/O
errors.
> 
> Please try reading the file on the host to check whether the I/O error
> is happening on the host and not related to KVM:
> 
> $ dd if=/opt/cibai/database of=/dev/null iflag=direct
> 
> Stefan
Hi Stefan, 

When I quit the installation, further see the error messages 

ide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING still
pending2011-12-12 22:47:51.427: shutting down

On Dec 12, 2011, at 6:56 PM, Stefan Hajnoczi wrote:
> On Sun, Dec 11, 2011 at 4:40 PM, takizo <paulooi at takizo.com>
wrote:
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>> char device redirected to /dev/pts/6
>> Using CPU model "cpu64-rhel6"
>> 
>> block I/O error in device 'drive-ide0-0-0': Invalid argument
(22)
> 
> Is /opt/cibai/database on a special filesystem or exotic storage setup?
> 
> Please check dmesg on the host for any kernel messages regarding I/O
errors.
> 
> Please try reading the file on the host to check whether the I/O error
> is happening on the host and not related to KVM:
> 
> $ dd if=/opt/cibai/database of=/dev/null iflag=direct
> 
> Stefan

Hi Stefan,

When I booted the VM, dmesg actually produced this message, I am not sure is it
a warning or error.

kvm: 6303: cpu0 unhandled wrmsr: 0x198 data 0
kvm: 6303: cpu0 unhandled wrmsr: 0x198 data 0

On Dec 12, 2011, at 11:08 PM, Stefan Hajnoczi wrote:
> On Mon, Dec 12, 2011 at 2:50 PM, takizo <paulooi at takizo.com>
wrote:
>> 
>> On Dec 12, 2011, at 6:56 PM, Stefan Hajnoczi wrote:
>> 
>>> On Sun, Dec 11, 2011 at 4:40 PM, takizo <paulooi at
takizo.com> wrote:
>>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096 -smp
1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>>>> char device redirected to /dev/pts/6
>>>> Using CPU model "cpu64-rhel6"
>>>> 
>>>> block I/O error in device 'drive-ide0-0-0': Invalid
argument (22)
>>> 
>>> Is /opt/cibai/database on a special filesystem or exotic storage
setup?
>>> 
>>> Please check dmesg on the host for any kernel messages regarding
I/O errors.
>>> 
>>> Please try reading the file on the host to check whether the I/O
error
>>> is happening on the host and not related to KVM:
>>> 
>>> $ dd if=/opt/cibai/database of=/dev/null iflag=direct
>>> 
>>> Stefan
>> 
>> Hi Stefan,
>> 
>> When I quit the installation, further see the error messages
>> 
>> ide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pendingide_dma_cancel: aiocb still pendingide_dma_cancel: BM_STATUS_DMAING
still pending2011-12-12 22:47:51.427: shutting down
>> 
Hi Stefan 
> Okay, so if I understand correctly you have CentOS 6.1 host running
> FreeBSD and CentOS guest installs to emulated IDE drives.  You're
> getting failed I/O requests with EINVAL (22).  You have verified that
> dd if=/opt/vm/database.img of=/dev/null iflag=direct reads the image
> file without errors.
That is right, both also emulated on IDE drive
> 
> Can you please try the CentOS guest install with a virtio-blk drive
> instead of IDE?  This should show whether this problem is related to
> IDE or a general issue in how QEMU accesses the host file.
Took your advice installed Centos on VirtIO drive and it's ok. 

Does it means it's IDE problem on the OS itself? 

2011-12-12 23:15:06.759: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 1024 -smp
1,sockets=1,cores=1,threads=1 -name centos -uuid
0752c74e-1d51-d529-f66f-8df9a28670ed -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/vm/centos.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/opt/ISO-Download/CentOS-6.0-x86_64-minimal.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:bc:5d,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-usb -device usb-tablet,id=input0 -vnc 127.0.0.1:3,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/9
Using CPU model "cpu64-rhel6"

> 
> Stefan

On Dec 12, 2011, at 11:33 PM, Stefan Hajnoczi wrote:
> On Mon, Dec 12, 2011 at 3:19 PM, takizo <paulooi at takizo.com>
wrote:
>> 
>> On Dec 12, 2011, at 11:08 PM, Stefan Hajnoczi wrote:
>> 
>>> On Mon, Dec 12, 2011 at 2:50 PM, takizo <paulooi at
takizo.com> wrote:
>>>> 
>>>> On Dec 12, 2011, at 6:56 PM, Stefan Hajnoczi wrote:
>>>> 
>>>>> On Sun, Dec 11, 2011 at 4:40 PM, takizo <paulooi at
takizo.com> wrote:
>>>>>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 4096
-smp 1,sockets=1,cores=1,threads=1 -name database -uuid
f3e9f320-7826-7e50-94bb-1833f7fd9dfb -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/database.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/cibai/database,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/opt/ISO-Download/FreeBSD-8.2-RELEASE-amd64-disc1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:77:a5:a6,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc
127.0.0.1:2,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>>>>>> char device redirected to /dev/pts/6
>>>>>> Using CPU model "cpu64-rhel6"
>>>>>> 
>>>>>> block I/O error in device 'drive-ide0-0-0':
Invalid argument (22)
>>>>> 
>>>>> Is /opt/cibai/database on a special filesystem or exotic
storage setup?
>>>>> 
>>>>> Please check dmesg on the host for any kernel messages
regarding I/O errors.
>>>>> 
>>>>> Please try reading the file on the host to check whether
the I/O error
>>>>> is happening on the host and not related to KVM:
>>>>> 
>>>>> $ dd if=/opt/cibai/database of=/dev/null iflag=direct
>>>>> 
>>>>> Stefan
>>>> 
>>>> Hi Stefan,
>>>> 
>>>> When I quit the installation, further see the error messages
>>>> 
>>>> ide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pendingide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pendingide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pendingide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pendingide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pendingide_dma_cancel: aiocb still pendingide_dma_cancel:
BM_STATUS_DMAING still pending2011-12-12 22:47:51.427: shutting down
>>>> 
>> 
>> Hi Stefan
>> 
>>> Okay, so if I understand correctly you have CentOS 6.1 host running
>>> FreeBSD and CentOS guest installs to emulated IDE drives. 
You're
>>> getting failed I/O requests with EINVAL (22).  You have verified
that
>>> dd if=/opt/vm/database.img of=/dev/null iflag=direct reads the
image
>>> file without errors.
>> 
>> That is right, both also emulated on IDE drive
>> 
>>> 
>>> Can you please try the CentOS guest install with a virtio-blk drive
>>> instead of IDE?  This should show whether this problem is related
to
>>> IDE or a general issue in how QEMU accesses the host file.
>> 
>> Took your advice installed Centos on VirtIO drive and it's ok.
>> 
>> Does it means it's IDE problem on the OS itself?
> 
> The physical drive in your machine is probably fine.  QEMU emulates an
> IDE controller and disks for the guest.  It looks like there is a bug
> in the IDE emulation code inside QEMU which leads to these errors.
I suspect the problem might came from QEMU, because day before it was working
before before I upgrade to latest QEMU. 
> 
> In general virtio-blk is the recommended storage interface because it
> delivers better performance.  I'm not up-to-speed on how stable the
> FreeBSD virtio drivers are but I think they are available if you
> search.
Unfortunately I am running FreeBSD 8.2, it's not natively supported yet. I
guess now interim solutions will be duplicate from some current running VM.
> 
>> 2011-12-12 23:15:06.759: starting up
>> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/libexec/qemu-kvm -S -M rhel6.1.0 -enable-kvm -m 1024 -smp
1,sockets=1,cores=1,threads=1 -name centos -uuid
0752c74e-1d51-d529-f66f-8df9a28670ed -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -drive
file=/opt/vm/centos.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=threads
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/opt/ISO-Download/CentOS-6.0-x86_64-minimal.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=threads
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:bc:5d,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-usb -device usb-tablet,id=input0 -vnc 127.0.0.1:3,password -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> char device redirected to /dev/pts/9
>> Using CPU model "cpu64-rhel6"
> 
> At this stage of debugging I would try enabling SystemTap probes in
> QEMU related to block I/O.  This produces a detailed log of what is
> happening inside QEMU.
> 
> strace -f -p $PID_OF_QEMU might also reveal how the I/O is failing.
> 
> Finally, gdb could be of use.
> 
> I'm not sure how familiar or easy it is for you to try these so
let's
> see if Kevin or someone else has other ideas before getting into the
> nitty-gritty of finding out why you get EINVAL (22).
I will see what I can grab and share in the list.

Thanks Stefan
> 
> Stefan