Dube, Lutz
2010-Jul-23 06:31 UTC
[Xen-devel] Poor performance of xend on servers with more than 10000 FC device paths
On server startup xend start or a later xend restart needs approx. 30 min to
start/restart. Without attached FC devices xend start/restart needs only some
seconds.
server type: Fujitsu Primergy RX600-S5
The time gets lost in xen/xend/XendNode.py line 329 while calling
vscsi_util.get_all_scsi_device().
329 for pscsi_record in vscsi_util.get_all_scsi_devices():
330 scsi_id = pscsi_record[''scsi_id'']
331 if scsi_id:
332 saved_HBA_uuid = None
I think, in most cases we don''t need all the PSCSI devices registered
in
xend, but only a few of it.
So a good solution for this perforamce issue is to scan only the SCSI device
paths we need, controlled by a new option in xend-config.sxp.
I have made a patch to allow specification of scsi devices we need in xend
in the config file xend-config.sxp.
The new options pscsi-device-mask expects a list of device ids oder partial
device ids like the option of lsscsi, e.g.
(pscsi-device-mask (''<partial-dev-id1''
''partial-dev-id2'' ...))
Without this option set in xend-config.sxp or if lsscsi is not support, all
device paths are process like today.
Signed-off-by: Lutz Dube Lutz.Dube@ts.fujitsu.com
diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/examples/xend-config.sxp
--- a/tools/examples/xend-config.sxp Wed Jul 21 13:15:33 2010 +0200
+++ b/tools/examples/xend-config.sxp Fri Jul 23 08:23:19 2010 +0200
@@ -294,3 +294,11 @@
# we have to realize this may incur security issue and we can''t make
sure the
# device assignment could really work properly even after we do this.
#(pci-passthrough-strict-check yes)
+
+# If we have a very big scsi device configuration, start of xend is slow,
+# because xend scans all the device paths to build its internal PSCSI device
+# list. If we need only a few devices for assigning to a guest, we can reduce
+# the scan to this device. Set list list of device paths in same syntax like in
+# command lsscsi, e.g. (''16:0:0:0'' ''15:0'')
+# (pscsi-device-mask (''*''))
+
diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/util/vscsi_util.py
--- a/tools/python/xen/util/vscsi_util.py Wed Jul 21 13:15:33 2010
+0200
+++ b/tools/python/xen/util/vscsi_util.py Fri Jul 23 08:23:19 2010 +0200
@@ -148,11 +148,12 @@ def _vscsi_get_scsidevices_by_sysfs():
return devices
-def vscsi_get_scsidevices():
+def vscsi_get_scsidevices(mask=""):
""" get all scsi devices information """
- devices = _vscsi_get_scsidevices_by_lsscsi("")
- if devices:
+ devices = _vscsi_get_scsidevices_by_lsscsi("[%s]" % mask)
+ if devices or (len(mask) and mask[0] != "*"):
+ # devices found or partial device scan
return devices
return _vscsi_get_scsidevices_by_sysfs()
@@ -274,9 +275,9 @@ def get_scsi_device(pHCTL):
return _make_scsi_record(scsi_info)
return None
-def get_all_scsi_devices():
+def get_all_scsi_devices(mask=""):
scsi_records = []
- for scsi_info in vscsi_get_scsidevices():
+ for scsi_info in vscsi_get_scsidevices(mask):
scsi_record = _make_scsi_record(scsi_info)
scsi_records.append(scsi_record)
return scsi_records
diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/xend/XendNode.py
--- a/tools/python/xen/xend/XendNode.py Wed Jul 21 13:15:33 2010 +0200
+++ b/tools/python/xen/xend/XendNode.py Fri Jul 23 08:23:19 2010 +0200
@@ -326,7 +326,12 @@ class XendNode:
pscsi_table = {}
pscsi_HBA_table = {}
- for pscsi_record in vscsi_util.get_all_scsi_devices():
+ pscsi_records = []
+ for pscsi_mask in xendoptions().get_pscsi_device_mask():
+ pscsi_records += vscsi_util.get_all_scsi_devices(pscsi_mask)
+ log.debug("pscsi record count: %s" % len(pscsi_records))
+
+ for pscsi_record in pscsi_records:
scsi_id = pscsi_record[''scsi_id'']
if scsi_id:
saved_HBA_uuid = None
diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/xend/XendOptions.py
--- a/tools/python/xen/xend/XendOptions.py Wed Jul 21 13:15:33
2010 +0200
+++ b/tools/python/xen/xend/XendOptions.py Fri Jul 23 08:23:19 2010
+0200
@@ -164,6 +164,9 @@ class XendOptions:
"""
print >>sys.stderr, "xend [ERROR]", fmt % args
+ """Default mask for pscsi device scan."""
+ xend_pscsi_device_mask = [''*'']
+
def configure(self):
self.set_config()
@@ -429,6 +432,10 @@ class XendOptions:
def get_pci_dev_assign_strict_check(self):
return self.get_config_bool("pci-passthrough-strict-check",
self.pci_dev_assign_strict_check_default)
+
+ def get_pscsi_device_mask(self):
+ return self.get_config_value("pscsi-device-mask",
+ self.xend_pscsi_device_mask)
class XendOptionsFile(XendOptions):
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Masaki Kanno
2010-Jul-23 06:52 UTC
Re: [Xen-devel] Poor performance of xend on servers with more than 10000 FC device paths
Hi Lutz, Well done!! Best regards, Kan Fri, 23 Jul 2010 08:31:14 +0200, "Dube, Lutz" wrote:>On server startup xend start or a later xend restart needs approx. 30 min to >start/restart. Without attached FC devices xend start/restart needs only some >seconds. > >server type: Fujitsu Primergy RX600-S5 > >The time gets lost in xen/xend/XendNode.py line 329 while calling >vscsi_util.get_all_scsi_device(). > >329 for pscsi_record in vscsi_util.get_all_scsi_devices(): >330 scsi_id = pscsi_record[''scsi_id''] >331 if scsi_id: >332 saved_HBA_uuid = None > >I think, in most cases we don''t need all the PSCSI devices registered in >xend, but only a few of it. >So a good solution for this perforamce issue is to scan only the SCSI device >paths we need, controlled by a new option in xend-config.sxp. > >I have made a patch to allow specification of scsi devices we need in xend >in the config file xend-config.sxp. >The new options pscsi-device-mask expects a list of device ids oder partial >device ids like the option of lsscsi, e.g. >(pscsi-device-mask (''<partial-dev-id1'' ''partial-dev-id2'' ...)) > >Without this option set in xend-config.sxp or if lsscsi is not support, all >device paths are process like today. > >Signed-off-by: Lutz Dube Lutz.Dube@ts.fujitsu.com > >diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/examples/xend-config.sxp >--- a/tools/examples/xend-config.sxp Wed Jul 21 13:15:33 2010 +0200 >+++ b/tools/examples/xend-config.sxp Fri Jul 23 08:23:19 2010 >+0200 >@@ -294,3 +294,11 @@ > # we have to realize this may incur security issue and we can''t make sure >the > # device assignment could really work properly even after we do this. > #(pci-passthrough-strict-check yes) >+ >+# If we have a very big scsi device configuration, start of xend is slow, >+# because xend scans all the device paths to build its internal PSCSI device >+# list. If we need only a few devices for assigning to a guest, we can >reduce >+# the scan to this device. Set list list of device paths in same syntax >like in >+# command lsscsi, e.g. (''16:0:0:0'' ''15:0'') >+# (pscsi-device-mask (''*'')) >+ >diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/util/vscsi_util.py >--- a/tools/python/xen/util/vscsi_util.py Wed Jul 21 13:15:33 >2010 +0200 >+++ b/tools/python/xen/util/vscsi_util.py Fri Jul 23 08:23:19 2010 >+0200 >@@ -148,11 +148,12 @@ def _vscsi_get_scsidevices_by_sysfs(): > return devices > > >-def vscsi_get_scsidevices(): >+def vscsi_get_scsidevices(mask=""): > """ get all scsi devices information """ > >- devices = _vscsi_get_scsidevices_by_lsscsi("") >- if devices: >+ devices = _vscsi_get_scsidevices_by_lsscsi("[%s]" % mask) >+ if devices or (len(mask) and mask[0] != "*"): >+ # devices found or partial device scan > return devices > return _vscsi_get_scsidevices_by_sysfs() > >@@ -274,9 +275,9 @@ def get_scsi_device(pHCTL): > return _make_scsi_record(scsi_info) > return None > >-def get_all_scsi_devices(): >+def get_all_scsi_devices(mask=""): > scsi_records = [] >- for scsi_info in vscsi_get_scsidevices(): >+ for scsi_info in vscsi_get_scsidevices(mask): > scsi_record = _make_scsi_record(scsi_info) > scsi_records.append(scsi_record) > return scsi_records >diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/xend/XendNode.py >--- a/tools/python/xen/xend/XendNode.py Wed Jul 21 13:15:33 2010 +0200 >+++ b/tools/python/xen/xend/XendNode.py Fri Jul 23 08:23:19 2010 +0200 >@@ -326,7 +326,12 @@ class XendNode: > pscsi_table = {} > pscsi_HBA_table = {} > >- for pscsi_record in vscsi_util.get_all_scsi_devices(): >+ pscsi_records = [] >+ for pscsi_mask in xendoptions().get_pscsi_device_mask(): >+ pscsi_records += vscsi_util.get_all_scsi_devices(pscsi_mask) >+ log.debug("pscsi record count: %s" % len(pscsi_records)) >+ >+ for pscsi_record in pscsi_records: > scsi_id = pscsi_record[''scsi_id''] > if scsi_id: > saved_HBA_uuid = None >diff -r fac9d58d3c8a -r 02d55e7d0e94 tools/python/xen/xend/XendOptions.py >--- a/tools/python/xen/xend/XendOptions.py Wed Jul 21 13:15: >33 2010 +0200 >+++ b/tools/python/xen/xend/XendOptions.py Fri Jul 23 08:23:19 >2010 +0200 >@@ -164,6 +164,9 @@ class XendOptions: > """ > print >>sys.stderr, "xend [ERROR]", fmt % args > >+ """Default mask for pscsi device scan.""" >+ xend_pscsi_device_mask = [''*''] >+ > > def configure(self): > self.set_config() >@@ -429,6 +432,10 @@ class XendOptions: > def get_pci_dev_assign_strict_check(self): > return self.get_config_bool("pci-passthrough-strict-check", > self.pci_dev_assign_strict_check_defau >lt) >+ >+ def get_pscsi_device_mask(self): >+ return self.get_config_value("pscsi-device-mask", >+ self.xend_pscsi_device_mask) > > class XendOptionsFile(XendOptions): > > > >-------------------------------text/plain------------------------------- >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Jul-23 16:32 UTC
Re: [Xen-devel] Poor performance of xend on servers with more than 10000 FC device paths
Dube, Lutz writes ("[Xen-devel] Poor performance of xend on servers with
more than 10000 FC device paths"):> On server startup xend start or a later xend restart needs approx. 30 min
to
> start/restart. Without attached FC devices xend start/restart needs only
some
> seconds.
I confess I don''t follow this patch in its entirety but it seems safe
enough in the sense that it''s not going to affect users who
aren''t
using vscsi.
We are winding down xend maintenance in favour of the new libxl
library and xl command line tool. xl is supposed to be commandline
compatible with xm. So you should probably check that libxl does the
right thing with your setups.
I''ll apply your patch.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel