Xen users - Aug 2008 - Xen and iSCSI - options and questions

Hello,

I have a small Xen farm of 8 dom0 servers with 64 virtual machines running
para-virtualized and this has been working great. Unfortunately, I''ve
hit a limit: my iSCSI hardware supports only 512 concurrent connections and so
I''m pretty much at the limit. (Wish I would have seen that problem
sooner!)

Of course, 87% of those connections are idle-- but necessary because I need to
have every volume mounted everywhere for migrations, etc. (And I have some
utility scripts I wrote to handle migrations and load balancing using Xen-API,
so it''s not an easy matter to simply connect to the iSCSI volumes as I
need them.)

I''m using stock Xen 3.2.1, btw. RPM that I compiled on a x86.
>From where I sit, I have several options, but I wanted to run this by the
list to tell me what others have done in this situation:
1. "Just-in-time" iSCSI connections from the iSCSI layer. So,
I''d have all of my device nodes in /dev/devices/by-path/... and iSCSI
would magically connect to them properly when the device node is opened.
Unfortunately, none of the Linux iSCSI clients that I can find support this
feature.

2. "Just-in-time" iSCSI connections from Xen. I found that
SuSE''s Xen seems to do this with a "block-iscsi" script in
/etc/xen/scripts, but it''s written for 3.0 and doesn''t seem to
work in 3.2. The  trick is that I''m doing all of my Xen management
through the XMLRPC API and I don''t see any way to do iSCSI mounts
there, so I suspect that their Xen 3.0 workaround doesn''t actually mesh
with Xen 3.2''s new way of doing things? (Otherwise, there would be a
way to do it through the API.)

3. Root-on-iSCSI boots for all the virtual hosts. This is messier, but I could
in theory change all 64 VMs to do root-on-iSCSI and (I presume) the iSCSI
connection that their local disks were on would be properly moved with a
"xm migrate". The downside is that RedHat Enterprise 5.1
doesn''t make this easy and I''m trying not to make this too
hacky. (And would I need to have little volumes for the iSCSI ramdisks? I
haven''t worked out how that scales yet.)

I think the best method is #2 and it seems like it SHOULD be possible. What am I
missing? How have others solved this dilemma?

Thanks for your help,

Joe Pranevich


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

http://kinkrsoftware.nl/contrib/xen/block-iscsi

Saw these?


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2008-08-04 22:02, jpranevich@kniggit.net wrote:
> I have a small Xen farm of 8 dom0 servers with 64 virtual machines running
para-virtualized and this has been working great. Unfortunately, I''ve
hit a limit: my iSCSI hardware supports only 512 concurrent connections and so
I''m pretty much at the limit. (Wish I would have seen that problem
sooner!)
[snip]
> 2. "Just-in-time" iSCSI connections from Xen. I found that
SuSE''s Xen seems to do this with a "block-iscsi" script in
/etc/xen/scripts, but it''s written for 3.0 and doesn''t seem to
work in 3.2. The  trick is that I''m doing all of my Xen management
through the XMLRPC API and I don''t see any way to do iSCSI mounts
there, so I suspect that their Xen 3.0 workaround doesn''t actually mesh
with Xen 3.2''s new way of doing things? (Otherwise, there would be a
way to do it through the API.)
As with the drbd problem I was having, this is a possible application 
for a qemu-dm wrapper.

You could try modifying the qemu-dm wrapper I wrote for drbd to do ISCSI 
connect/disconnect instead of the drbd-related stuff. This is 
encapsulated in the &setup and &cleanup subs. See the recent threads 
named "Centos 5.2 using LVM -> DRBD 0.82 -> Xen" and "drbd
8
primary/primary and xen migration on RHEL 5". This wrapper was written 
against Xen 3.0, not 3.2, so I don''t know if some qemu-dm semantics
have
changed in a way that would break the basic design. Also, I wrote that 
script for HVMs and it remains untested with PVMs.

-- 
Jefferson Ogata : Internetworker, Antibozo

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>http://kinkrsoftware.nl/contrib/xen/block-iscsi
> As with the drbd problem I was having, this is a possible application for a
> qemu-dm wrapper.

Are there any plans to get the block-iscsi or qemu-dm wrapper scripts
included in mainline Xen?

Cheers,
Todd



-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 5 Aug 2008, Todd Deshane wrote:
> >http://kinkrsoftware.nl/contrib/xen/block-iscsi
>
> > As with the drbd problem I was having, this is a possible application
for a
> > qemu-dm wrapper.
>
> Are there any plans to get the block-iscsi or qemu-dm wrapper scripts
> included in mainline Xen?
It was already so many times submitted on the mainlinglist, included in
Suse and if someone ask for it i''ll publish a URL.

If you want to bother to get it disbributed, get it distributed...


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

Stefan, the version that I could fine doesn''t seem to work with Xen
3.2. Is there some modification to /etc/xen/scripts/block that needs to be made
so it will be called in the right cases? I wasn''t able to figure it out
in my digging through it yesterday. (I had concluded that it must not work in
Xen 3.2 because there didn''t appear to be a way to do it using the
xen-api layer. But that may also be due to my ignorance of VBD/SR management and
how that all links together...)

I just installed a copy of SuSE on a machine here (seems to be Xen 3.0?) and
I''m looking, but I''m not sure I understand this all yet.

Thanks for your assistance,

Joe

----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: "Todd Deshane" <deshantm@gmail.com>
Cc: xen-users@lists.xensource.com, "Antibozo"
<xen-users@antibozo.net>
Sent: Tuesday, August 5, 2008 3:55:05 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-users] Xen and iSCSI - options and questions

On Tue, 5 Aug 2008, Todd Deshane wrote:
> >http://kinkrsoftware.nl/contrib/xen/block-iscsi
>
> > As with the drbd problem I was having, this is a possible application
for a
> > qemu-dm wrapper.
>
> Are there any plans to get the block-iscsi or qemu-dm wrapper scripts
> included in mainline Xen?
It was already so many times submitted on the mainlinglist, included in
Suse and if someone ask for it i''ll publish a URL.

If you want to bother to get it disbributed, get it distributed...


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2008-08-05 19:55, Stefan de Konink wrote:
> On Tue, 5 Aug 2008, Todd Deshane wrote:
>>> http://kinkrsoftware.nl/contrib/xen/block-iscsi
>>> As with the drbd problem I was having, this is a possible
application for a
>>> qemu-dm wrapper.
>> Are there any plans to get the block-iscsi or qemu-dm wrapper scripts
>> included in mainline Xen?
> 
> It was already so many times submitted on the mainlinglist, included in
> Suse and if someone ask for it i''ll publish a URL.
If you''re referring to block-drbd, this is distributed in current 
versions of drbd.

If you''re referring to my qemu-dm wrapper script, I only wrote it a 
couple of days ago, so I have no idea what you''re talking about. I
still
consider it beta code; I''m hoping people will test it out on
appropriate
applications.

-- 
Jefferson Ogata : Internetworker, Antibozo

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> Stefan, the version that I could fine doesn''t seem to work with
Xen
> 3.2. Is there some modification to /etc/xen/scripts/block that needs to
> be made so it will be called in the right cases? I wasn''t able to
figure
> it out in my digging through it yesterday. (I had concluded that it must
> not work in Xen 3.2 because there didn''t appear to be a way to do
it
> using the xen-api layer. But that may also be due to my ignorance of
> VBD/SR management and how that all links together...)
Nonsence, it is running from 3.1 to 3.2 for the last 9 months. Maybe you
don''t use the right disk syntax, but that isn''t really
something I
responsible for.
> I just installed a copy of SuSE on a machine here (seems to be Xen
> 3.0?) and I''m looking, but I''m not sure I understand this
all yet.
The updated script was even included in the new SuSe distro...


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

Ah, that is perhaps the problem then. I''m using "iscsi:"
instead of "phy:" and replacing the colon in the IQN with a
''@'' per instructions that I found in the SuSE
"xmexample.disks" file.

For example:

disk = [
"iscsi:iqn.2001-05.com.equallogic@0-8a0906-4bbfbc101-51f41f3a27647d08-jpranevich,xvda,w"
]

That returns an error: 

[root@vs6 SOURCES]# xm create jpranevich
Using config file "/etc/xen/jpranevich".
Error: Disk isn''t accessible

That error caused me to assume that the script didn''t work with 3.2. Is
there any other tweak that I need to make? Any specific way that iscsiadm should
be run first? I notice that just the line doesn''t include a portal
address-- just a IQN. My guess was that it just looked in /var/lib/iscsi/nodes/
for the right addresses and configuration, but perhaps not.

Any further advice that you can provide is appreciated.

Joe

----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: jpranevich@kniggit.net
Cc: xen-users@lists.xensource.com, "Antibozo"
<xen-users@antibozo.net>, "Todd Deshane"
<deshantm@gmail.com>
Sent: Tuesday, August 5, 2008 4:35:12 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-users] Xen and iSCSI - options and questions

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> Stefan, the version that I could fine doesn''t seem to work with
Xen
> 3.2. Is there some modification to /etc/xen/scripts/block that needs to
> be made so it will be called in the right cases? I wasn''t able to
figure
> it out in my digging through it yesterday. (I had concluded that it must
> not work in Xen 3.2 because there didn''t appear to be a way to do
it
> using the xen-api layer. But that may also be due to my ignorance of
> VBD/SR management and how that all links together...)
Nonsence, it is running from 3.1 to 3.2 for the last 9 months. Maybe you
don''t use the right disk syntax, but that isn''t really
something I
responsible for.
> I just installed a copy of SuSE on a machine here (seems to be Xen
> 3.0?) and I''m looking, but I''m not sure I understand this
all yet.
The updated script was even included in the new SuSe distro...


Stefan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> Hello,
>
> Ah, that is perhaps the problem then. I''m using "iscsi:"
instead of "phy:" and replacing the colon in the IQN with a
''@'' per instructions that I found in the SuSE
"xmexample.disks" file.
>
> For example:
>
> disk = [
"iscsi:iqn.2001-05.com.equallogic@0-8a0906-4bbfbc101-51f41f3a27647d08-jpranevich,xvda,w"
]
>
> That returns an error:
>
> [root@vs6 SOURCES]# xm create jpranevich
> Using config file "/etc/xen/jpranevich".
> Error: Disk isn''t accessible
>
> That error caused me to assume that the script didn''t work with
3.2. Is there any other tweak that I need to make? Any specific way that
iscsiadm should be run first? I notice that just the line doesn''t
include a portal address-- just a IQN. My guess was that it just looked in
/var/lib/iscsi/nodes/ for the right addresses and configuration, but perhaps
not.
>
> Any further advice that you can provide is appreciated.
iscsi://iqn.1986-03.com.sun:02:0d1b01cb-6e24-64bb-e0f7-a5e798e79984

Is what it was developed for. So I think you miss the //


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

----- "Stefan de Konink" <skinkie@xs4all.nl> wrote:
> > Any further advice that you can provide is appreciated.
> 
> iscsi://iqn.1986-03.com.sun:02:0d1b01cb-6e24-64bb-e0f7-a5e798e79984
> 
> Is what it was developed for. So I think you miss the //
> 
The good news is that, using your suggestion (which doesn''t match the
documentation, but whatever), I''m able to apparently get my
iSCSI-as-root VBD working on a SuSE test machine. (It doesn''t boot, but
that''s because I need to reconfigure for the SuSE bootloader rather
than Redhat''s pygrub. I at least get kernel messages and a
"can''t mount root" message.)

I did have to make a change in the script which worries me: (in find_sdev())

      #dev=`readlink $session/device/target*/*:0:*/block*`
      dev=`basename $session/device/target*/*:0:*/block*/*`

It''s not a link, so readlink wasn''t returning anything. It
*appears* to be trying to get the local /dev/ name for the iSCSI mount which it
created earlier in the script... and making this change does get the thing
moving, but there may be something deeper amiss.

Unfortunately, I do not have the same luck on my stock Xen 3.2.1 install.
I''ve added some simple debugging and realized that block-iscsi
doesn''t seem to be called at all and there are no lines written to
/var/log/xen-hotplug.log when I try and start my domain, though I do get lines
there on my SuSE instance.

Is there some hotplug configuration that I need to adjust? Anywhere obvious I
should look to see why it wouldn''t be being executed?

I''ll keep digging at this. Thanks for helping me get this far.

Joe

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> The good news is that, using your suggestion (which doesn''t match
the
> documentation, but whatever), I''m able to apparently get my
> iSCSI-as-root VBD working on a SuSE test machine. (It doesn''t
boot, but
> that''s because I need to reconfigure for the SuSE bootloader
rather than
> Redhat''s pygrub. I at least get kernel messages and a
"can''t mount root"
> message.)
So now you are complaining to me, the developer, that you didn''t read
the
mailinglist?
> I did have to make a change in the script which worries me: (in
find_sdev())
>
>       #dev=`readlink $session/device/target*/*:0:*/block*`
>       dev=`basename $session/device/target*/*:0:*/block*/*`
That line is *pretty* kernel specific... libvirt had many problems with it
too. Are you running .18 or .21?
> Unfortunately, I do not have the  same luck on my stock Xen 3.2.1
> install. I''ve added some simple debu gging and realized that
block-iscsi
> doesn''t seem to be called at all and  there are no lines written
to
> /var/log/xen-hotplug.log when I try an d start my domain, though I do
> get lines there on my SuSE instance.
No extra config is required.

Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

By way of an update, I now have a working block-iscsi test domain on SuSE. (It
took me a day or two because I''ve been busy with other work and it
needed a little massaging on my part. My images were written for pyGrub and I
just didn''t know the magic incantation to use the SuSE domUloader.)

Unfortunately, I''ve still no luck with the exact same configs on my
stock Xen 3.2.1 on Centos 5.2. From what I can tell,
''block-iscsi'' is never getting run (I added debugging that
writes to temp files) and nothing ever appears in the xen-hotplug.log.

Is there a doc someplace or a message board posting here that I can''t
find yet which describes the hotplug process in 3.2? (Especially where it
decides what scripts to execute.) I don''t really know where to start,
whether this is handled in the Python code or in the C, etc. Since Stefan and
others have this working perfectly well on 3.2.x, I''m sure that
it''s something easy... but no dice yet. I will just start diffing SuSE
code with the stock code soon and see if I find any hints that way.

(And, as an aside, when I poke and prod at the running block-iscsi instances
using the xen-api, it''s not how I expect. VDI.get_all()
doesn''t return anything and the VM object for my instance has a proper
VBD, but that VBD doesn''t have a VDI. This could be a xen-api artifact
of starting the instance using "xm" rather than the api, but
doesn''t get me any closer to seeing how to start a iscsi instance from
the api alone or to understanding how it funds its block device after a live
migration. More to dig, more to dig.)

And Stefan, I apologize if you felt I was dissing your documentation. My remark
was aimed at the SuSE maintainer who went and added an incorrect note in the
SuSE doc, rather than at you. I have the utmost respect and admiration for your
work.

Thanks again for all your help,

Joe


----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: jpranevich@kniggit.net
Cc: xen-users@lists.xensource.com, "Antibozo"
<xen-users@antibozo.net>, "Todd Deshane"
<deshantm@gmail.com>
Sent: Wednesday, August 6, 2008 4:21:41 AM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-users] Xen and iSCSI - options and questions

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> The good news is that, using your suggestion (which doesn''t match
the
> documentation, but whatever), I''m able to apparently get my
> iSCSI-as-root VBD working on a SuSE test machine. (It doesn''t
boot, but
> that''s because I need to reconfigure for the SuSE bootloader
rather than
> Redhat''s pygrub. I at least get kernel messages and a
"can''t mount root"
> message.)
So now you are complaining to me, the developer, that you didn''t read
the
mailinglist?
> I did have to make a change in the script which worries me: (in
find_sdev())
>
>       #dev=`readlink $session/device/target*/*:0:*/block*`
>       dev=`basename $session/device/target*/*:0:*/block*/*`
That line is *pretty* kernel specific... libvirt had many problems with it
too. Are you running .18 or .21?
> Unfortunately, I do not have the  same luck on my stock Xen 3.2.1
> install. I''ve added some simple debu gging and realized that
block-iscsi
> doesn''t seem to be called at all and  there are no lines written
to
> /var/log/xen-hotplug.log when I try an d start my domain, though I do
> get lines there on my SuSE instance.
No extra config is required.

Stefan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello,

So that it can be documented for the next generation of Xen users that have this
problem and come to the list via Google, I wanted to say that I''ve
tracked down my hotplug issue with the block-iscsi scripts on Centos 5.x and I
have the system nearly working. (It doesn''t work yet, but at least
it''s triggering the hotplug scripts properly now.)

The "bug" is in XendBootloader.py where it appears to check
os.access() too early when the "disk" parameter for iSCSI is the IQN.
Since the IQN doesn''t exist as a local file, it was bombing out.

The relevant fix is in the "xen-domUloader.diff" patch in the SuSE
SRPM. When applied to the stock install, my hotplug scripts are triggering
happily. I have to debug some issues with them under Centos and the iscsiadm
paths are different from SuSE, but I''m on the right path now.
(There''s a ton of other SuSE fixes in their SRPM as well, but I was
trying to keep a minimal change set so as to be as "stock" as
possible.)

I''ll put up a SRPM for Centos 5.x once I have this working, if anyone
is interested.

Thanks for everyone''s assistance with this.

Joe

----- Original Message -----
From: jpranevich@kniggit.net
To: xen-users@lists.xensource.com
Sent: Thursday, August 7, 2008 7:48:57 PM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-users] Xen and iSCSI - options and questions

Hello,

By way of an update, I now have a working block-iscsi test domain on SuSE. (It
took me a day or two because I''ve been busy with other work and it
needed a little massaging on my part. My images were written for pyGrub and I
just didn''t know the magic incantation to use the SuSE domUloader.)

Unfortunately, I''ve still no luck with the exact same configs on my
stock Xen 3.2.1 on Centos 5.2. From what I can tell,
''block-iscsi'' is never getting run (I added debugging that
writes to temp files) and nothing ever appears in the xen-hotplug.log.

Is there a doc someplace or a message board posting here that I can''t
find yet which describes the hotplug process in 3.2? (Especially where it
decides what scripts to execute.) I don''t really know where to start,
whether this is handled in the Python code or in the C, etc. Since Stefan and
others have this working perfectly well on 3.2.x, I''m sure that
it''s something easy... but no dice yet. I will just start diffing SuSE
code with the stock code soon and see if I find any hints that way.

(And, as an aside, when I poke and prod at the running block-iscsi instances
using the xen-api, it''s not how I expect. VDI.get_all()
doesn''t return anything and the VM object for my instance has a proper
VBD, but that VBD doesn''t have a VDI. This could be a xen-api artifact
of starting the instance using "xm" rather than the api, but
doesn''t get me any closer to seeing how to start a iscsi instance from
the api alone or to understanding how it funds its block device after a live
migration. More to dig, more to dig.)

And Stefan, I apologize if you felt I was dissing your documentation. My remark
was aimed at the SuSE maintainer who went and added an incorrect note in the
SuSE doc, rather than at you. I have the utmost respect and admiration for your
work.

Thanks again for all your help,

Joe


----- Original Message -----
From: "Stefan de Konink" <skinkie@xs4all.nl>
To: jpranevich@kniggit.net
Cc: xen-users@lists.xensource.com, "Antibozo"
<xen-users@antibozo.net>, "Todd Deshane"
<deshantm@gmail.com>
Sent: Wednesday, August 6, 2008 4:21:41 AM GMT -05:00 US/Canada Eastern
Subject: Re: [Xen-users] Xen and iSCSI - options and questions

On Tue, 5 Aug 2008 jpranevich@kniggit.net wrote:
> The good news is that, using your suggestion (which doesn''t match
the
> documentation, but whatever), I''m able to apparently get my
> iSCSI-as-root VBD working on a SuSE test machine. (It doesn''t
boot, but
> that''s because I need to reconfigure for the SuSE bootloader
rather than
> Redhat''s pygrub. I at least get kernel messages and a
"can''t mount root"
> message.)
So now you are complaining to me, the developer, that you didn''t read
the
mailinglist?
> I did have to make a change in the script which worries me: (in
find_sdev())
>
>       #dev=`readlink $session/device/target*/*:0:*/block*`
>       dev=`basename $session/device/target*/*:0:*/block*/*`
That line is *pretty* kernel specific... libvirt had many problems with it
too. Are you running .18 or .21?
> Unfortunately, I do not have the  same luck on my stock Xen 3.2.1
> install. I''ve added some simple debu gging and realized that
block-iscsi
> doesn''t seem to be called at all and  there are no lines written
to
> /var/log/xen-hotplug.log when I try an d start my domain, though I do
> get lines there on my SuSE instance.
No extra config is required.

Stefan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

Could you attach this patch please? I''d like to check this against my
ubuntu/debian patchqueue. Last SuSE I''ve seen was 6.x and I suspect
searching half a day for this...

Thanks in advance!

cheers,

Stephan

jpranevich@kniggit.net schrieb:
> Hello,
> 
> So that it can be documented for the next generation of Xen users that have
this problem and come to the list via Google, I wanted to say that I''ve
tracked down my hotplug issue with the block-iscsi scripts on Centos 5.x and I
have the system nearly working. (It doesn''t work yet, but at least
it''s triggering the hotplug scripts properly now.)
> 
> The "bug" is in XendBootloader.py where it appears to check
os.access() too early when the "disk" parameter for iSCSI is the IQN.
Since the IQN doesn''t exist as a local file, it was bombing out.
> 
> The relevant fix is in the "xen-domUloader.diff" patch in the
SuSE SRPM. When applied to the stock install, my hotplug scripts are triggering
happily. I have to debug some issues with them under Centos and the iscsiadm
paths are different from SuSE, but I''m on the right path now.
(There''s a ton of other SuSE fixes in their SRPM as well, but I was
trying to keep a minimal change set so as to be as "stock" as
possible.)
> 
> I''ll put up a SRPM for Centos 5.x once I have this working, if
anyone is interested.
> 
> Thanks for everyone''s assistance with this.




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Do you here mean that the bootloader falsely assumes that the uri is a 
file?!

Stefan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> The relevant fix is in the "xen-domUloader.diff" patch in the
SuSE
>  SRPM. When applied to the stock install, my hotplug scripts are
>  triggering happily. I have to debug some issues with them under Centos
>  and the iscsiadm paths are different from SuSE, but I''m on the
right
>  path now. (There''s a ton of other SuSE fixes in their SRPM as
well,
>  but I was trying to keep a minimal change set so as to be as
"stock"
>  as possible.)
Why not just side-step all of this and do your iSCSI initiating in dom0,
then LVM the LUN and boot the domU off of a phy: allocation?  I''ve seen
several posts lately regarding the complexities of iSCSI, booting
domU''s, etc., and I''m wondering why no one is just treated
like normal
SAN or SCSI LUNs.

John





-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

John Madden schreef:
>> The relevant fix is in the "xen-domUloader.diff" patch in the
SuSE
>>  SRPM. When applied to the stock install, my hotplug scripts are
>>  triggering happily. I have to debug some issues with them under Centos
>>  and the iscsiadm paths are different from SuSE, but I''m on
the right
>>  path now. (There''s a ton of other SuSE fixes in their SRPM as
well,
>>  but I was trying to keep a minimal change set so as to be as
"stock"
>>  as possible.)
> 
> Why not just side-step all of this and do your iSCSI initiating in dom0,
> then LVM the LUN and boot the domU off of a phy: allocation?  I''ve
seen
> several posts lately regarding the complexities of iSCSI, booting
> domU''s, etc., and I''m wondering why no one is just
treated like normal
> SAN or SCSI LUNs.
Migration?


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Why not just side-step all of this and do your iSCSI initiating in
dom0,
> > then LVM the LUN and boot the domU off of a phy: allocation? 
I''ve seen
> > several posts lately regarding the complexities of iSCSI, booting
> > domU''s, etc., and I''m wondering why no one is just
treated like normal
> > SAN or SCSI LUNs.
> 
> Migration?
Using multipath (and sync''ing /etc/multipath.conf), I migrate SAN LUNs
around between hosts in this manner so I think the only difference here
would be allowing multiple logins on the target.  

John



-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

John Madden schreef:
>>> Why not just side-step all of this and do your iSCSI initiating in
dom0,
>>> then LVM the LUN and boot the domU off of a phy: allocation? 
I''ve seen
>>> several posts lately regarding the complexities of iSCSI, booting
>>> domU''s, etc., and I''m wondering why no one is
just treated like normal
>>> SAN or SCSI LUNs.
>> Migration?
> 
> Using multipath (and sync''ing /etc/multipath.conf), I migrate SAN
LUNs
> around between hosts in this manner so I think the only difference here
> would be allowing multiple logins on the target.  
You cannot safely assume nobody is writing to that iSCSI LUN, therefore 
for example block-iscsi logins in and out for you between host upon 
create/shutdown/migration.


Stefan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Using multipath (and sync''ing /etc/multipath.conf), I migrate
SAN LUNs
> > around between hosts in this manner so I think the only difference
here
> > would be allowing multiple logins on the target.  
> 
> You cannot safely assume nobody is writing to that iSCSI LUN, therefore 
> for example block-iscsi logins in and out for you between host upon 
> create/shutdown/migration.
Well no, you can''t assume it, you have to ensure it with configuration.
In any shared disk scenario, you have to make the same assurances
(cluster filesystems, for example).  But yes, I suppose it would be
safer at least to do migrations with the block-iscsi scripts.  

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

----- "John Madden" <jmadden@ivytech.edu> wrote:
> Why not just side-step all of this and do your iSCSI initiating in
> dom0,
> then LVM the LUN and boot the domU off of a phy: allocation?  I''ve
> seen
> several posts lately regarding the complexities of iSCSI, booting
> domU''s, etc., and I''m wondering why no one is just
treated like
> normal
> SAN or SCSI LUNs.
Because the iSCSI hardware (Equallogics) I''m using has a maximum of 512
concurrent iSCSI connections per volume group. I have a Xen cluster of 40
servers and since each would need persistent connections to the iSCSI for each
possible host that could live on it (since I want migrations to be free;
I''ve written some software for this using the Xen-API that does simple
load balancing), that means I could only have 12 total virtual machines. And
that doesn''t work well at all. :)

If the hardware didn''t have a low limit, I would be doing exactly as
you indicate since it''s a much cleaner solution overall. But I have a
lot of $$$ invested in the gear now.

(Other options include using LVM and having each domU server using a different
slice of the same LVM, but I''ve read elsewhere that it isn''t
cluster-safe yet for cases where you need to add or remove volumes from the
group, etc. I don''t know the specifics since I read that weeks ago.)

Joe

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

----- "Stefan de Konink" <skinkie@xs4all.nl> wrote:
> Do you here mean that the bootloader falsely assumes that the uri is a
> file?!
> 
> Stefan
I traced my problem down to: (Xen 3.2.1)

./tools/python/xen/xend/XendBootloader.py:

def bootloader(blexec, disk, dom, quiet = False, blargs = '''',
kernel = '''',
               ramdisk = '''', kernel_args =
''''):

...

    if not os.access(blexec, os.X_OK):
        msg = "Bootloader isn''t executable"
        log.error(msg)
        raise VmError(msg)
    if not os.access(disk, os.R_OK):
        msg = "Disk isn''t accessible"
        log.error(msg)
        raise VmError(msg)


In the second os.access() check, I added additional debugging to see what it was
trying to access. Using the iscsi://IQN url which worked on my SuSE install,
"disk" was set to:
''//iqn.2001-05.com.equallogic@0-8a0906-4bbfbc101-51f41f3a27647d08-jpranevich
'' (Clearly, just the config line with iscsi: ripped off. I
didn''t trace the code to see where that happened.)

Checking the same file on the SuSE install, I noticed that it was very different
and I found the patch that made it different. It is possible (likely?) that some
other change in the patch may have been the "fix" since now that I
read more thoroughly, it seems to have similar logic, but does the check later.
(So, perhaps the switch to a local device name happens earlier in the process. I
don''t understand the flow enough and there are other changes to blkif
and such which could be the real fix. It could have been simply fortunate that I
picked the right patch.)

I hope this helps. I''m still having some issues, but it makes the iSCSI
connection properly, so I''m much of the way there.

Joe





_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

Am Mittwoch, 20. August 2008 15:25 schrieb John Madden:
> > The relevant fix is in the "xen-domUloader.diff" patch in
the SuSE
> >  SRPM. When applied to the stock install, my hotplug scripts are
> >  triggering happily. I have to debug some issues with them under
Centos
> >  and the iscsiadm paths are different from SuSE, but I''m on
the right
> >  path now. (There''s a ton of other SuSE fixes in their SRPM
as well,
> >  but I was trying to keep a minimal change set so as to be as
"stock"
> >  as possible.)
> Why not just side-step all of this and do your iSCSI initiating in dom0,
> then LVM the LUN and boot the domU off of a phy: allocation?  I''ve
seen
> several posts lately regarding the complexities of iSCSI, booting
> domU''s, etc., and I''m wondering why no one is just
treated like normal
> SAN or SCSI LUNs.
in a two Hardware-Setup it''s also better not to make iscsi over lo but
use the
logical volume direct.

So my solution ist to make stable device names (e.g. /dev/$DOMUNAME-sda 
and  /dev/$DOMUNAME-sdb) on every xen host. It doesn''t matter if these
are
real devices or links to real devices. OK, it''s a little overhead to
make
this stable naming scheme but with this my xen-host can also be storage-host 
while doing live migration.


-- 
greetings

eMHa


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users