Xen users - Nov 2006 - xen, iscsi and resilience to short network outages

Hi. Here is the short version:

If dom0 experiences a short (< 120 second) network outage the guests
whose disks are on iSCSI LUNs get (seemingly) unrecoverable IO errors.
Is it possible to make Xen more resiliant to such problems?

And now the full version:

We''re testing Xen on iSCSI LUNs. The hardware/software configuration
is:

  * Dom0 and guest OS:  SLES10 x86_64
  * iSCSI LUN on NetApp filer

We connect to the LUN through dom0 and then "map" the device to a
guest
like so:

disk = [
''phy:/dev/disk/by-id/scsi-360a9800043346863483437714a643833,hda,w''
]

At around noon today (though it''s happend a few times in the last few
weeks) one
of our switches was powered off. At that time, here is what I see in
syslog of dom0:

Nov  9 12:06:04 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:13 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:21 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:29 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:38 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:46 egovxen1 iscsid: connect failed (113)
Nov  9 12:06:55 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:03 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:04 egovxen1 kernel: tg3: peth0: Link is up at 1000 Mbps,
full duplex.
Nov  9 12:07:04 egovxen1 kernel: tg3: peth0: Flow control is off for
TX and off for RX.
Nov  9 12:07:04 egovxen1 kernel: xenbr0: port 2(peth0) entering learning state
Nov  9 12:07:04 egovxen1 kernel: xenbr0: topology change detected, propagating
Nov  9 12:07:04 egovxen1 kernel: xenbr0: port 2(peth0) entering forwarding state
Nov  9 12:07:10 egovxen1 kernel:  session0: iscsi: session recovery
timed out after 120 secs
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: scsi: Device offlined -
not ready after error recovery
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: scsi: Device offlined -
not ready after error recovery
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: SCSI error: return code = 0x20000
Nov  9 12:07:10 egovxen1 kernel: end_request: I/O error, dev sdd,
sector 23349467
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: SCSI error: return code = 0x20000
Nov  9 12:07:10 egovxen1 kernel: end_request: I/O error, dev sdc, sector 6573193
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:3: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: rejecting I/O to offline device
Nov  9 12:07:10 egovxen1 kernel: sd 0:0:0:2: rejecting I/O to offline device
Nov  9 12:07:11 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:20 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:28 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:36 egovxen1 iscsid: connect failed (113)
Nov  9 12:07:44 egovxen1 iscsid: connection0:0 is operational after
recovery (19 attempts)

So it looks like the iSCSI connection was dropped at 12:06:04 and reestablished
at 12:07:44. But during this time the guests who''s disks were on the
iSCSI LUNs
get IO errors and do not recover. Here is what I got when I connected
to the console:

sfeehan@egovxen1:~> sudo xm console xenlb2
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: Id "1" respawning too fast: disabled for 5 minutes

Is it possible to adjust a timeout or otherwise make Xen a bit more tolerant of
short network outages? Thanks.

-- 
Steve Feehan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> INIT: cannot execute "/sbin/mingetty"
> INIT: cannot execute "/sbin/mingetty"
> INIT: cannot execute "/sbin/mingetty"
> INIT: Id "1" respawning too fast: disabled for 5 minutes
What happens after the five minutes is up?

Steven.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 11/10/06, Steven Smith <sos22-xen@srcf.ucam.org>
wrote:
> > INIT: cannot execute "/sbin/mingetty"
> > INIT: cannot execute "/sbin/mingetty"
> > INIT: cannot execute "/sbin/mingetty"
> > INIT: Id "1" respawning too fast: disabled for 5 minutes
> What happens after the five minutes is up?
>
> Steven.
Well, I''ve never waited 5 minutes to find out. But it was an hour
after the original IO timeout before I tried to login at the console
and saw these messages. And during this time I was not able to ssh
into the VM (but it was still pingable).

So I will force a timeout on Monday and see what happens after 5
minutes. But my guess is that the system is not going to recover.

-- 
Steve Feehan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 11/10/06, Steve Feehan <sfeehan@gmail.com>
wrote:
> On 11/10/06, Steven Smith <sos22-xen@srcf.ucam.org> wrote:
> > > INIT: cannot execute "/sbin/mingetty"
> > > INIT: cannot execute "/sbin/mingetty"
> > > INIT: cannot execute "/sbin/mingetty"
> > > INIT: Id "1" respawning too fast: disabled for 5
minutes
> > What happens after the five minutes is up?
> >
> > Steven.
>
> Well, I''ve never waited 5 minutes to find out. But it was an hour
> after the original IO timeout before I tried to login at the console
> and saw these messages. And during this time I was not able to ssh
> into the VM (but it was still pingable).
>
> So I will force a timeout on Monday and see what happens after 5
> minutes. But my guess is that the system is not going to recover.
As I suspected, the system never recovers from the IO errors. Someone
pulled the switch again and from the guest (connected via ssh):

sfeehan@extlb1:~> ls
-bash: /bin/ls: Input/output error

So I login to dom0 and start a console:

sfeehan@egovxen1:~> sudo xm console extlb1


extlb1 login: root
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: cannot execute "/sbin/mingetty"
INIT: Id "1" respawning too fast: disabled for 5 minutes
INIT: no more processes left in this runlevel

So I wait >5 minutes, reconnect and get no response.

Would I stand a better chance connecting to the iSCSI LUN from domU
rather than from dom0? My thought is that since the dom0 is able to
reconnect to the LUN when the network returns, perhaps this would be
the case for domU as well?

Root on NFS is also an option, but ataoe is unfortunately a
non-starter since we''ve invested in a NetApp and have to use it. ;)

I was hoping to avoid complicated initrd configuration (which I think
will be required for root on NFS or direct iSCSI connection).

I''m still curious if there is a configurable timeout in the domU
kernel that will be a little more tolerant of network outages.

Thanks,

-- 
Steve Feehan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Would I stand a better chance connecting to the iSCSI LUN from domU
> rather than from dom0? My thought is that since the dom0 is able to
> reconnect to the LUN when the network returns, perhaps this would be
> the case for domU as well?
Why would you *not* leave all of the iSCSI work to the domU?  It seems
like that would provide better predictability performance-wise (i.e.,
leaving your I/O to the built-in scheduling rather than everyone getting
a whatever''s-available slice of dom0''s cpu time).  

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 11/13/06, John Madden <jmadden@ivytech.edu>
wrote:
> > Would I stand a better chance connecting to the iSCSI LUN from domU
> > rather than from dom0? My thought is that since the dom0 is able to
> > reconnect to the LUN when the network returns, perhaps this would be
> > the case for domU as well?
>
> Why would you *not* leave all of the iSCSI work to the domU?  It seems
> like that would provide better predictability performance-wise (i.e.,
> leaving your I/O to the built-in scheduling rather than everyone getting
> a whatever''s-available slice of dom0''s cpu time).
Because it is not tolerant to short network outages? :)

Perhaps that is an excellent point, though I''m not familiar with IO
scheduling. But if pushing the IO up to domU makes it more reliable
then I might be able to tolerate a small loss in IO performance. This
presumes that it will be more reliable, which I won''t have time to
test right away.

-- 
Steve Feehan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Because it is not tolerant to short network outages? :)
Short network outages?  Are you seeing this in domU?
> Perhaps that is an excellent point, though I''m not familiar with
IO
> scheduling. But if pushing the IO up to domU makes it more reliable
> then I might be able to tolerate a small loss in IO performance. This
> presumes that it will be more reliable, which I won''t have time to
> test right away.
I can''t claim more "reliable," it just seems more logical and
that it
would likely be more controllable.  Then again, who knows, maybe you
dedicate a CPU to dom0 and do all of your IO there, leaving other CPU''s
to other workloads -- I guess it could be taken both ways.  But I would
think you''d want the hypervisor to be responsible for as little
userspace-type stuff as possible.

John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 11/13/06, Tom Brown <tbrown@baremetal.com> wrote:
> pushing the iscsi to dom0 means you''re abstracting the disk and
the domU
> can not know about iscsi timeouts, retries etc.
>
> pushing the iscsi to domU means you can do live migration, and your dom0
> is much simpler and should be more stable.
Live migration works in the iscsi in domU case also. You just have to be
very careful to limit access to the LUN and use a consisten device
name (ie. the ones provided by udev in /dev/disk/*).
> I''ve never used iscsi, so I can''t really comment about
retries, recoveries
> etc... but if the dom0 can recover, I would think that the domU could _if_
> it knew it was handling iSCSI, but that''s just a guess.
Well, I''m convinced it''s worth a try. When I can spare some
time to
experiment I''ll try both the root on NFS and iscsi in domU
configurations and report back to the list with my results.

Thanks,

-- 
Steve Feehan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users