Hello, We need to use iSCSI in some of our domUs. By the moment, iSCSI is not for system filesystem, but for data filesystem. I am wondering what is the best way to use it. Is it better to configure it in dom0 and then attach the device to the domU? Or is it better to configure it directly in the domU? I am thinking that if we configure it in the dom0, then we can''t share that iscsi resource between differents domUs, can we? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, That depends on your scenario. If you've got more than one dom0(Xen-Servers), exspecially for redundancy, then it's not a good idea to make dom0 the iscsi-host. Of course, you can connect to any iscsi-target directly from the domUs, as you can connect the dom0 vias iscsi and share the ressource via NFS p.e. then. Me personally, i would connect it directly to the domUs, as this is straightforward. Why are you thinking about implementing another "hop" via dom0? What is the advantage you asume in this case? Cheers Florian> -----Ursprüngliche Nachricht----- > Von: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] Im Auftrag von > Angel L. Mateo > I am wondering what is the best way to use it. Is it > better to configure it in dom0 and then attach the device to > the domU? Or is it better to configure it directly in the domU? > > I am thinking that if we configure it in the dom0, > then we can't share that iscsi resource between differents > domUs, can we?_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rustedt, Florian wrote:>That depends on your scenario. If you''ve got more than one >dom0(Xen-Servers), exspecially for redundancy, then it''s not a good >idea to make dom0 the iscsi-host. > >Of course, you can connect to any iscsi-target directly from the >domUs, as you can connect the dom0 vias iscsi and share the >ressource via NFS p.e. then. > >Me personally, i would connect it directly to the domUs, as this is >straightforward.Much what I thought - attach directly from the DomU and then the connection is portable wherever you host that DomU. It didn''t work well for me (see below).>Why are you thinking about implementing another "hop" via dom0? What >is the advantage you asume in this case?Performance ! I''m not alone in finding iSCSI performance was "rather poor" connecting from the DomU. I reluctantly configured iSCSI in the Dom0 and passed the resulting SCSI device as a block device to the DomU and performance was "quite acceptable". From bits I''ve picked up here and there, I believe it''s something to do with the network stack in XEN/Dom0 being single threaded and killing performance - though I don''t know why it should affect iSCSI and not other networking. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
We had a similar experience, however with AoE rather than iSCSI. The xennet drivers were fast when benchmarked on a single DomU. We hadn''t anticipated the slowdown when many DomU''s were hitting xennet for block storage, cluster multicast and plain HTTP traffic all at once. Exporting block devices from DomU instead was much more successful. The performance bottleneck seemed to vanish, we could tune the network as we see fit (e.g. multipath / jumbo frames, good for storage), and the phy: backend allows for asynchronous writes (although the latter prevented us from doing this on shared GFS filesystems). This was on CentOS 5.2, in case anyone is wondering. -Jeff -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Simon Hobson Sent: Thursday, June 18, 2009 9:28 AM To: Xen-users@lists.xensource.com Subject: Re: AW: [Xen-users] Best way to use iSCSI in domU Rustedt, Florian wrote:>Why are you thinking about implementing another "hop" via dom0? What >is the advantage you asume in this case?Performance ! I''m not alone in finding iSCSI performance was "rather poor" connecting from the DomU. I reluctantly configured iSCSI in the Dom0 and passed the resulting SCSI device as a block device to the DomU and performance was "quite acceptable". From bits I''ve picked up here and there, I believe it''s something to do with the network stack in XEN/Dom0 being single threaded and killing performance - though I don''t know why it should affect iSCSI and not other networking. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El jue, 18-06-2009 a las 14:41 +0200, Rustedt, Florian escribió:> Hi, > > That depends on your scenario. If you''ve got more than one dom0(Xen-Servers), exspecially for redundancy, then it''s not a good idea to make dom0 the iscsi-host. > > Of course, you can connect to any iscsi-target directly from the domUs, as you can connect the dom0 vias iscsi and share the ressource via NFS p.e. then. > > Me personally, i would connect it directly to the domUs, as this is straightforward. > > Why are you thinking about implementing another "hop" via dom0? What is the advantage you asume in this case? >I don''t know. I''ve just raised the question to know about your experiences. I think that configuring it in the domU is easier and cleaner, but, for example, if you have more than one domU (in the same dom0) needing iSCSI resources (in our case, all iSCS disks are served by the same server) configuring it in the dom0 centralized the configuration. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El jue, 18-06-2009 a las 13:43 -0400, Jeff Sturm escribió:> We had a similar experience, however with AoE rather than iSCSI. The > xennet drivers were fast when benchmarked on a single DomU. We hadn''t > anticipated the slowdown when many DomU''s were hitting xennet for block > storage, cluster multicast and plain HTTP traffic all at once. > > Exporting block devices from DomU instead was much more successful. The > performance bottleneck seemed to vanish, we could tune the network as we > see fit (e.g. multipath / jumbo frames, good for storage), and the phy: > backend allows for asynchronous writes (although the latter prevented us > from doing this on shared GFS filesystems). >OK. But supppose I need two domUs (in the same dom0) needing the same iSCSI resource. Could they share it if I configure it in the dom0? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Angel L. Mateo > Sent: Friday, June 19, 2009 2:29 AM > To: xen-users@lists.xensource.com > Subject: RE: AW: [Xen-users] Best way to use iSCSI in domU > > El jue, 18-06-2009 a las 13:43 -0400, Jeff Sturm escribió: > > Exporting block devices from DomU instead was much more successful. The > > performance bottleneck seemed to vanish, we could tune the network as we > > see fit (e.g. multipath / jumbo frames, good for storage), and the phy: > > backend allows for asynchronous writes (although the latter prevented us > > from doing this on shared GFS filesystems). > > > OK. But supppose I need two domUs (in the same dom0) needing the same > iSCSI resource. Could they share it if I configure it in the dom0?Yes. If you export it read/write however, you need to override Xen's sanity check that would normally prevent you from doing so. Here's an example: node01: 'tap:sync:/dev/vg/wwwdocs,xvdg,w!' node02: 'tap:sync:/dev/vg/wwwdocs,xvdg,w!' In this example, /dev/vg/wwwdocs contains a clustered filesystem mounted read/write on several domU instances, some of which are on the same dom0, some not. We also use tap:sync: to ensure blocks are flushed to the SAN as they are written. If you have just one dom0 and could tolerate possible data loss, you're probably fine using phy: or tap:aio: instead. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
That wouldn't be any different than if they were on separate Dom0s. You're talking about having two systems accessing the same block device. Honestly, for data that I need shared between multiple Dom0s or DomUs, I use NFS, and it still works great. Best Regards Nathan Eisenberg Sr. Systems Administrator Atlas Networks, LLC support@atlasnetworks.us http://support.atlasnetworks.us/portal -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Angel L. Mateo Sent: Thursday, June 18, 2009 11:29 PM To: xen-users@lists.xensource.com Subject: RE: AW: [Xen-users] Best way to use iSCSI in domU El jue, 18-06-2009 a las 13:43 -0400, Jeff Sturm escribió:> We had a similar experience, however with AoE rather than iSCSI. The > xennet drivers were fast when benchmarked on a single DomU. We hadn't > anticipated the slowdown when many DomU's were hitting xennet for block > storage, cluster multicast and plain HTTP traffic all at once. > > Exporting block devices from DomU instead was much more successful. The > performance bottleneck seemed to vanish, we could tune the network as we > see fit (e.g. multipath / jumbo frames, good for storage), and the phy: > backend allows for asynchronous writes (although the latter prevented us > from doing this on shared GFS filesystems). >OK. But supppose I need two domUs (in the same dom0) needing the same iSCSI resource. Could they share it if I configure it in the dom0? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El vie, 19-06-2009 a las 10:22 -0700, Nathan Eisenberg escribió:> Honestly, for data that I need shared between multiple Dom0s or DomUs, I use NFS, and it still works great. >NFS is not an option. We are doing this to replace a NFS FS because of poor performance. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Interesting. I'd stand back and look at your existing setup before trying to rebuild with some other storage tech. In my experience, poor NFS performance is almost always due to misconfigurations (not your fault - NFS is suprisingly low level and can be very confusing at times). I have several very large NFS deployments that have absolutely fantastic performance. NFS is probably already a proven technology for whatever you're doing. The number one thing to do is ensure that you have NFS mounted without the synchronization option turned on. There's also buffer sizes to be calculated and set. If these are web servers, there is also an Apache directive regarding kernel caching of files; essentially, you can end up double caching files without the directive in place. Thank You, Nathan Eisenberg Sr. Systems Administrator Atlas Networks, LLC -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Angel L. Mateo Sent: Sunday, June 21, 2009 11:09 PM To: xen-users@lists.xensource.com Subject: RE: AW: [Xen-users] Best way to use iSCSI in domU El vie, 19-06-2009 a las 10:22 -0700, Nathan Eisenberg escribió:> Honestly, for data that I need shared between multiple Dom0s or DomUs, I use NFS, and it still works great. >NFS is not an option. We are doing this to replace a NFS FS because of poor performance. -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
El lun, 22-06-2009 a las 00:32 -0700, Nathan Eisenberg escribió:> Interesting. I''d stand back and look at your existing setup before trying to rebuild with some other storage tech. > > In my experience, poor NFS performance is almost always due to misconfigurations (not your fault - NFS is suprisingly low level and can be very confusing at times). I have several very large NFS deployments that have absolutely fantastic performance. NFS is probably already a proven technology for whatever you''re doing. > > The number one thing to do is ensure that you have NFS mounted without the synchronization option turned on. There''s also buffer sizes to be calculated and set. If these are web servers, there is also an Apache directive regarding kernel caching of files; essentially, you can end up double caching files without the directive in place. >We''ve done a lot of tests and checks. NFS performance is poor for what we need in this particular case (we are using it in other scenarios without any problem). -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información _o) y las Comunicaciones Aplicadas (ATICA) / \\ http://www.um.es/atica _(___V Tfo: 868887590 Fax: 868888337 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeff Sturm <jeff.sturm@eprize.com> writes:> We also use tap:sync: to ensure blocks are flushed to the SAN as > they are written. If you have just one dom0 and could tolerate > possible data loss, you''re probably fine using phy: or tap:aio: > instead.Doesn''t phy: use direct IO (avoiding buffer cache) as well? -- Feri. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: Ferenc Wagner [mailto:wferi@niif.hu] > Sent: Monday, June 22, 2009 8:55 AM > To: Jeff Sturm > Cc: xen-users@lists.xensource.com > Subject: Re: AW: Best way to use iSCSI in domU > > Jeff Sturm <jeff.sturm@eprize.com> writes: > > > We also use tap:sync: to ensure blocks are flushed to the SAN as > > they are written. If you have just one dom0 and could tolerate > > possible data loss, you''re probably fine using phy: or tap:aio: > > instead. > > Doesn''t phy: use direct IO (avoiding buffer cache) as well?Empirically, my guess is "no", or at least that writes are not performed synchronously. Our GFS cluster seems stable using tap:sync: as the backend, whereas it became corrupted on average once a week using phy:. We have since had 3 weeks of continuous uptime on the cluster. Frankly, I don''t understand the kernel interfaces well enough to say for certain. I asked a question about this a few weeks ago on this list, and didn''t receive a reply. If anyone knows for certain, it would be good to have confirmation, because I can''t completely rule out some other cause of our fs corruption. -Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users