I''m just curious what some of you guys out there are using for remote storage with XEN. We currently are a service provider using xen for our customers virtual servers. Right now each server is deployed with a raid controller and 4 disks in a raid 10 configuration. The raid controller + BBU are not cheap and add an extra expense to the server. Not only that but disk IO is what causes us to deploy a new host. For the most part these servers end up with a lot of unused Ram, CPU, and Disk Space. What we are considering doing is setting up a SAN, something like a 48 disk raid 10 array that the hosts can be attached to some how. I''m curious what some of you guys out there are doing and or using. Our virtual servers right now are PVM''s with logical volumes attached. I''ve been looking at ISCSI but the problem I''m seeing with ISCSI is that the disks that are exported to the initiator just pop up as /dev/sd devices and there seams to be no simple way to map that device to the guest easily using our automation system. I''ve also been looking a little into AOE but not sure if that would work. If we did disk based images the solution for the most part would be easy, but from what I''ve read LV''s attached to the guest perform a lot better than a raw disk image. Hopefully some of you can pass on your experience! Thanks! ~Shaun _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jun 23, 2011 at 10:32 PM, Shaun Reitan <mailinglists@unix-scripts.com> wrote:> I''m just curious what some of you guys out there are using for remote > storage with XEN. We currently are a service provider using xen for our > customers virtual servers. Right now each server is deployed with a raid > controller and 4 disks in a raid 10 configuration. The raid controller + > BBU are not cheap and add an extra expense to the server. Not only that but > disk IO is what causes us to deploy a new host. For the most part these > servers end up with a lot of unused Ram, CPU, and Disk Space. What we are > considering doing is setting up a SAN, something like a 48 disk raid 10 > array that the hosts can be attached to some how. > > I''m curious what some of you guys out there are doing and or using. Our > virtual servers right now are PVM''s with logical volumes attached. I''ve > been looking at ISCSI but the problem I''m seeing with ISCSI is that the > disks that are exported to the initiator just pop up as /dev/sd devices and > there seams to be no simple way to map that device to the guest easily using > our automation system. I''ve also been looking a little into AOE but not > sure if that would work. If we did disk based images the solution for the > most part would be easy, but from what I''ve read LV''s attached to the guest > perform a lot better than a raw disk image. > > Hopefully some of you can pass on your experience! Thanks! > > ~Shaun > > > _______________________________________________iSCSI and AOE export the storage as raw block devices to the hypervisors - which basically act the same as local storate (i.e. SATA / SAS HDD''s in the server) so you then either need to partition the iSCSI LUN the same way as you do with local drives, or the easier way is to use LVM to "slice" it up and then use LVM volumes for the domU virtual machines -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> iSCSI and AOE export the storage as raw block devices to the > hypervisors - which basically act the same as local storate (i.e. SATA > / SAS HDD''s in the server) so you then either need to partition the > iSCSI LUN the same way as you do with local drives, or the easier way > is to use LVM to "slice" it up and then use LVM volumes for the domU > virtual machines > >I don''t know about AOE but with ISCSI the problem I''m seeing is that if i create a volume group on the SAN there is no way to export that volume group. I could export the device that the volume group was created on but then the host doesn''t see that VG. I''m assuming i would need to use clustered LVM for that which i have been told to stay far away from. Another option was to carve out the LV''s on the SAN and to export each LV to the initiators but still the problem is that the host assigns these luns to /dev/sd devices and there''s no simple nice way to map them to who owns what disk. Least not that I''ve found yet. I''m fairly new to ISCSI though so maybe there''s something I''m missing. ~Shaun _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I don''t know about AOE but with ISCSI the problem I''m seeing is that if > i create a volume group on the SAN there is no way to export that volume > group. I could export the device that the volume group was created on > but then the host doesn''t see that VG. I''m assuming i would need to use > clustered LVM for that which i have been told to stay far away from. > Another option was to carve out the LV''s on the SAN and to export each > LV to the initiators but still the problem is that the host assigns > these luns to /dev/sd devices and there''s no simple nice way to map them > to who owns what disk. Least not that I''ve found yet. I''m fairly new > to ISCSI though so maybe there''s something I''m missing.A != B. Sure, scsi devices get assigned to each host in /dev but this has nothing to do with ownership. In a cluster, you''d want them all to have "ownership" at a SCSI level... cLVM will handle LVM metadata change locking but still isn''t strictly necessary. (I''ve begun to move away from it for my clusters -- it''s a /bitch/ and far more hassle than it''s worth.) For metadata changes, I pick a node and then run things like `vgscan` and `lvchange -ay` on the other nodes. ClusterSSH makes this a breeze. But that''s beside the point: share your disks over iSCSI, SAN, AOE, or JBOD, it makes no difference. John -- John Madden Sr UNIX Systems Engineer / Office of Technology Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jun 23, 2011 at 4:04 PM, Shaun Reitan <mailinglists@unix-scripts.com> wrote:> I don''t know about AOE but with ISCSI the problem I''m seeing is that if i > create a volume group on the SAN there is no way to export that volume > group. I could export the device that the volume group was created on but > then the host doesn''t see that VG. I''m assuming i would need to use > clustered LVM for that which i have been told to stay far away from. Another > option was to carve out the LV''s on the SAN and to export each LV to the > initiators but still the problem is that the host assigns these luns to > /dev/sd devices and there''s no simple nice way to map them to who owns what > disk. Least not that I''ve found yet. I''m fairly new to ISCSI though so > maybe there''s something I''m missing.your problem has two parts: A: the sharing/splitting. iSCSI shares block devices, and (as you''ve found) a VG is not a block device. you have to either export PVs or LVs there are three (main) ways to do it: 1: don''t worry about partitions/LVM on the target. Export disks (or rather, arrays). On the initator(s) set those as PVs to create a VG and then LVs. typically you do it on only one initiator and then vgscan on all the others (like John Madden). pros: simple, manageable. cons: no lock protection, resizing LVs can be dangerous without suspending initiators. 2: do as 1, but use cLVM. any LVM management done on any initiator is automatically and atomically seen by all of them. pros: safer. cons: the lock manager itself is a big pain to manage. 3: do all the LVM management on the target(s), export each LV as an iSCSI LUN. Most commercial iSCSI appliances include management on par with LVM, at the very least. pros: much safer, easier to handle, no need of shared locks. cons: you can''t have a LV spanning two targets. B: the Linux iSCSI initiator exposes each LUN as a full drive: /dev/sdd, /dev/sde, etc., not as a ''partition'' /dev/sdX1, /dev/sdX2, etc. That''s because they''re exactly the same as SCSI disks. but, just like SCSI disks, those LUNs are partitionable with any tool. just do `parted /dev/sdX` and create a full-size /dev/sdX1 partition. (BTW, wasn''t there an iSCSI-specific ''phy:'' driver? something like ''phy:iscsi:<lun name>''.... or am i mixing faint memories?) -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Jun 23, 2011 at 4:32 PM, Shaun Reitan <mailinglists@unix-scripts.com> wrote:> I''m curious what some of you guys out there are doing and or using. Our > virtual servers right now are PVM''s with logical volumes attached. I''ve > been looking at ISCSI but the problem I''m seeing with ISCSI is that the > disks that are exported to the initiator just pop up as /dev/sd devices and > there seams to be no simple way to map that device to the guest easily using > our automation system.Have a look at the /dev/disk/by-path/. It''s been a few years since I played with iSCSI on Linux but I think your device will be the iSCSI target name. This device path will not change between hosts and can be used to map an iSCSI target to a DomU if you''re doing one LUN per domain. I believe this will work for live migration if that''s a concern. If you''re just exposing large LUNs and want to use LVM to divide up that space then you''ll have to use cLVM if you want to mount that same VG on multiple DomU''s. -- David _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have personnaly setup two different types of SAN for use with Xen: 1st was FC fabrics using netapp device. - performs well even with several thounsands IOps and netapp embend a lot of tools to performs disk admin (if you don''t like them you can still use LVM). - Xen host sees LUN as local disks and if like i did you setup several fabrics to the same target(s) you can setup multipathd to ensure path redundancy. This is quite a mess to setup but is truely usefull! - you export LUN created with netapp tools so there is nothing like a volume group. Be aware that it is preferable to use clustered version of LVM2 (clvm) when using lvm on top off your exported LUNs. Unfortunately some basics features of LVM (like snapshots) do not work in clustered mode (at least they didn''t when I last tried). 2nd was AoE: - performs well too if you meet the ethernet network requirement - a lot cheaper! Many can''t afford netapp SAN device with brocade switch and QLOGIC HBA (for example) - have natice redundancy if your ethernet switches are. Redundancy path is a lot easier to setup and maintain than using iscsi - you can export raw devices, partitions or LVs (but of course no volume group) (even file but...) IMHO AoE is a really good alternative from an admin point of view. Unfortunately my AoE setup never reached the number a VM I got on the FC setup. So I am not able to say how many IOps it would support. However if you plan to deploy some 48 disks SAN using raid10 or even raid50 array you can expect good IO performances. regards. Le 23/06/2011 22:32, Shaun Reitan a écrit :> I''m just curious what some of you guys out there are using for remote > storage with XEN. We currently are a service provider using xen for > our customers virtual servers. Right now each server is deployed with > a raid controller and 4 disks in a raid 10 configuration. The raid > controller + BBU are not cheap and add an extra expense to the server. > Not only that but disk IO is what causes us to deploy a new host. For > the most part these servers end up with a lot of unused Ram, CPU, and > Disk Space. What we are considering doing is setting up a SAN, > something like a 48 disk raid 10 array that the hosts can be attached > to some how. > > I''m curious what some of you guys out there are doing and or using. > Our virtual servers right now are PVM''s with logical volumes > attached. I''ve been looking at ISCSI but the problem I''m seeing with > ISCSI is that the disks that are exported to the initiator just pop up > as /dev/sd devices and there seams to be no simple way to map that > device to the guest easily using our automation system. I''ve also > been looking a little into AOE but not sure if that would work. If we > did disk based images the solution for the most part would be easy, > but from what I''ve read LV''s attached to the guest perform a lot > better than a raw disk image. > > Hopefully some of you can pass on your experience! Thanks! > > ~Shaun > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- <http://www.horoa.net> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Viele Grüße. Christian Am 23.06.2011 um 23:41 schrieb Javier Guerra Giraldez <javier@guerrag.com>:> On Thu, Jun 23, 2011 at 4:04 PM, Shaun Reitan > <mailinglists@unix-scripts.com> wrote: >> I don''t know about AOE but with ISCSI the problem I''m seeing is that if i >> create a volume group on the SAN there is no way to export that volume >> group. I could export the device that the volume group was created on but >> then the host doesn''t see that VG. I''m assuming i would need to use >> clustered LVM for that which i have been told to stay far away from. Another >> option was to carve out the LV''s on the SAN and to export each LV to the >> initiators but still the problem is that the host assigns these luns to >> /dev/sd devices and there''s no simple nice way to map them to who owns what >> disk. Least not that I''ve found yet. I''m fairly new to ISCSI though so >> maybe there''s something I''m missing. > > > your problem has two parts: > > A: the sharing/splitting. iSCSI shares block devices, and (as you''ve > found) a VG is not a block device. you have to either export PVs or > LVsFibre channel is also easy to use.> there are three (main) ways to do it: > > 1: don''t worry about partitions/LVM on the target. Export disks (or > rather, arrays). On the initator(s) set those as PVs to create a VG > and then LVs. typically you do it on only one initiator and then > vgscan on all the others (like John Madden). > > pros: simple, manageable. cons: no lock protection, resizing LVs can > be dangerous without suspending initiators. > > > 2: do as 1, but use cLVM. any LVM management done on any initiator is > automatically and atomically seen by all of them. > > pros: safer. cons: the lock manager itself is a big pain to manage. >CLVM is not a pain. You only need cLVM if you follow some rules. IO fencing, pacemaker and all this complicated stuff is not really needed. Only cLVM compiled with openais. You can even make snapshots in a cluster environment with the latest version of LVM2. I use this since 3 years now and never lost a machine.> > 3: do all the LVM management on the target(s), export each LV as an > iSCSI LUN. Most commercial iSCSI appliances include management on par > with LVM, at the very least.This is a pain, when you have more then 100 machines, as I have.> > pros: much safer, easier to handle, no need of shared locks. cons: > you can''t have a LV spanning two targets. > > > B: the Linux iSCSI initiator exposes each LUN as a full drive: > /dev/sdd, /dev/sde, etc., not as a ''partition'' /dev/sdX1, /dev/sdX2, > etc. That''s because they''re exactly the same as SCSI disks. but, > just like SCSI disks, those LUNs are partitionable with any tool. > just do `parted /dev/sdX` and create a full-size /dev/sdX1 partition. > > (BTW, wasn''t there an iSCSI-specific ''phy:'' driver? something like > ''phy:iscsi:<lun name>''.... or am i mixing faint memories?) > > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Viele Grüße. Christian Am 23.06.2011 um 23:16 schrieb John Madden <jmadden@ivytech.edu>:>> I don''t know about AOE but with ISCSI the problem I''m seeing is that if >> i create a volume group on the SAN there is no way to export that volume >> group. I could export the device that the volume group was created on >> but then the host doesn''t see that VG. I''m assuming i would need to use >> clustered LVM for that which i have been told to stay far away from. >> Another option was to carve out the LV''s on the SAN and to export each >> LV to the initiators but still the problem is that the host assigns >> these luns to /dev/sd devices and there''s no simple nice way to map them >> to who owns what disk. Least not that I''ve found yet. I''m fairly new >> to ISCSI though so maybe there''s something I''m missing. > > A != B. Sure, scsi devices get assigned to each host in /dev but this has nothing to do with ownership. In a cluster, you''d want them all to have "ownership" at a SCSI level... cLVM will handle LVM metadata change locking but still isn''t strictly necessary. (I''ve begun to move away from it for my clusters -- it''s a /bitch/ and far more hassle than it''s worth.) For metadata changes, I pick a node and then run things like `vgscan` and `lvchange -ay` on the otherYes, this really works. But I use cLVM now for extra security. If you forget vgscan on one host, you can run into problems, when you create a new LV. Thats why I added cLVM to my setup. And when you activate a LV exclusively one one host, you can make even snapshots.> nodes. ClusterSSH makes this a breeze. But that''s beside the point: share your disks over iSCSI, SAN, AOE, or JBOD, it makes no difference. > > John > > > > -- > John Madden > Sr UNIX Systems Engineer / Office of Technology > Ivy Tech Community College of Indiana > jmadden@ivytech.edu > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 12:25 AM, Christian Motschke <christian@motschke.de> wrote:> CLVM is not a pain. You only need cLVM if you follow some rules. IO fencing, > pacemaker and all this complicated stuff is not really needed. Only cLVM > compiled with openais. You can even make snapshots in a cluster environment > with the latest version of LVM2. I use this since 3 years now and never lost > a machine.totally agree on openais. unfortunately, last time i checked, it wasn''t always easy to install on all distros. redhat ones are (were?) specially set on the old GFS ways. in the end, it''s all mostly a design problem; but you _do_ have to have all concepts very well defined. didn''t know about snapshots working at last! great news certainly. how recent is that? is it available out of the box on any distro? or do you have to compile all yourself?> 3: do all the LVM management on the target(s), export each LV as an > iSCSI LUN. Most commercial iSCSI appliances include management on par > with LVM, at the very least. > > This is a pain, when you have more then 100 machines, as I have.care to expand? i''ve never really used any commercial iSCSI box, just checked that they do comply with all checkpoints, and fooled a little on some control screens. AFAICT, there''s usually some (maybe optional, maybe expensive) global manager console that let''s you control many boxes on the same screen. or do you mean that it''s a pain when you have many Linux-based storage boxes? if so, i have to agree. it''s simple only when you have a handful of targets. after a point, you definitely need something else. can be cLVM, or glusterFS, or maybe just some clever scripts+frontend. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 3:32 AM, Shaun Reitan <mailinglists@unix-scripts.com> wrote:> I''m just curious what some of you guys out there are using for remote > storage with XEN. We currently are a service provider using xen for our > customers virtual servers. Right now each server is deployed with a raid > controller and 4 disks in a raid 10 configuration. The raid controller + > BBU are not cheap and add an extra expense to the server. Not only that but > disk IO is what causes us to deploy a new host. For the most part these > servers end up with a lot of unused Ram, CPU, and Disk Space. What we are > considering doing is setting up a SAN, something like a 48 disk raid 10 > array that the hosts can be attached to some how.Other than I/O requirement, do you need SAN? If you only need it for I/O, then there are other options which are both cheaper and easier to manage, which includes (but not limited to): - servers with lots of internal disk (something with 16 x 2.5" disk is quite common) - DAS/external JBOD array - go for SSD (some providers have taken this route) If you need SAN for other purposes (e.g. live migration), then others have already provided some good example. However, personally I''d check again with the requirement. It''s possible that the "best" setup for you might be: - a zfs-based storage appliance (e.g. napp-it, nexentastor) with lots of disks (mirror), lots of memory, and SSD for L2ARC/SLOG to provide high I/O throughput - nfs and image-based disks to provide easy management - tap:aio or tap:vhd on Xen side -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha wrote:>If you need SAN for other purposes (e.g. live migration), then others >have already provided some good example. However, personally I''d check >again with the requirement. It''s possible that the "best" setup for >you might be: >- a zfs-based storage appliance (e.g. napp-it, nexentastor) with lots >of disks (mirror), lots of memory, and SSD for L2ARC/SLOG to provide >high I/O throughput >- nfs and image-based disks to provide easy management >- tap:aio or tap:vhd on Xen sideDo you have any feeling for how such a setup might compare, performance wise, with a local disk setup with the same disks and LVM and a partition per client block device ? At the moment I''ve got 4 boxes, and have similar issues to the OP - one has spare disk but not enough ram, others have no spare ram but spare disk, and so on. Apart from one box which is probably temporary, they are "hand me down" Dell servers with SCSI disks which are (relatively) hard to obtain/expensive in large sizes. So I have been toying with setting up a box with a few SAS or SATA disks and migrating some of the guest storage to it. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 2:03 PM, Simon Hobson <linux@thehobsons.co.uk> wrote:> Fajar A. Nugraha wrote: > >> If you need SAN for other purposes (e.g. live migration), then others >> have already provided some good example. However, personally I''d check >> again with the requirement. It''s possible that the "best" setup for >> you might be: >> - a zfs-based storage appliance (e.g. napp-it, nexentastor) with lots >> of disks (mirror), lots of memory, and SSD for L2ARC/SLOG to provide >> high I/O throughput >> - nfs and image-based disks to provide easy management >> - tap:aio or tap:vhd on Xen side > > Do you have any feeling for how such a setup might compare, performance > wise, with a local disk setup with the same disks and LVM and a partition > per client block device ?Local disk with block devices should definitely be faster. But here''s the catch: notice I specifically suggest zfs, lots of memory, and SSD? zfs/zvol by itself has lower performance compared to plain block device. However it''s also the only filesystem that I know of that can use SSD to speed up both read and write. It''s something that LVM/ext4 can''t do. So it just might be able to offset performance penalty of zfs + nfs + file-based-image. ... or if you have the money, simply buy EMC/Netapp-base/Oracle storage with cache SSDs :D -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Am 24.06.2011 um 07:52 schrieb Javier Guerra Giraldez:> On Fri, Jun 24, 2011 at 12:25 AM, Christian Motschke > <christian@motschke.de> wrote: >> CLVM is not a pain. You only need cLVM if you follow some rules. IO fencing, >> pacemaker and all this complicated stuff is not really needed. Only cLVM >> compiled with openais. You can even make snapshots in a cluster environment >> with the latest version of LVM2. I use this since 3 years now and never lost >> a machine. > > totally agree on openais. unfortunately, last time i checked, it > wasn''t always easy to install on all distros. redhat ones are (were?) > specially set on the old GFS ways. in the end, it''s all mostly a > design problem; but you _do_ have to have all concepts very well > defined. > > didn''t know about snapshots working at last! great news certainly. > how recent is that? is it available out of the box on any distro? or > do you have to compile all yourself? >I made my own packages from the latest sources for debian. Clustered snapshots should work since january (2.0.2.84). I had some problems with it, but got it running yesterday with the latest sources of LVM2 ;-).> >> 3: do all the LVM management on the target(s), export each LV as an >> iSCSI LUN. Most commercial iSCSI appliances include management on par >> with LVM, at the very least. >> >> This is a pain, when you have more then 100 machines, as I have. > > care to expand? i''ve never really used any commercial iSCSI box, just > checked that they do comply with all checkpoints, and fooled a little > on some control screens. AFAICT, there''s usually some (maybe > optional, maybe expensive) global manager console that let''s you > control many boxes on the same screen.I mean, when you add or remove a new LV (iSCSI LUN) you have to do it with a graphical management console, set the access rights for the hosts etc. for every LUN. It is easier and faster to use lvcreate on one host.> > or do you mean that it''s a pain when you have many Linux-based storage > boxes? if so, i have to agree. it''s simple only when you have a > handful of targets. after a point, you definitely need something > else. can be cLVM, or glusterFS, or maybe just some clever > scripts+frontend. > > > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 9:17 AM, Fajar A. Nugraha <list@fajar.net> wrote:> On Fri, Jun 24, 2011 at 2:03 PM, Simon Hobson <linux@thehobsons.co.uk> wrote: >> Fajar A. Nugraha wrote: >> >>> If you need SAN for other purposes (e.g. live migration), then others >>> have already provided some good example. However, personally I''d check >>> again with the requirement. It''s possible that the "best" setup for >>> you might be: >>> - a zfs-based storage appliance (e.g. napp-it, nexentastor) with lots >>> of disks (mirror), lots of memory, and SSD for L2ARC/SLOG to provide >>> high I/O throughput >>> - nfs and image-based disks to provide easy management >>> - tap:aio or tap:vhd on Xen side >> >> Do you have any feeling for how such a setup might compare, performance >> wise, with a local disk setup with the same disks and LVM and a partition >> per client block device ? > > Local disk with block devices should definitely be faster. > > But here''s the catch: notice I specifically suggest zfs, lots of > memory, and SSD? > zfs/zvol by itself has lower performance compared to plain block > device. However it''s also the only filesystem that I know of that can > use SSD to speed up both read and write. It''s something that LVM/ext4 > can''t do. So it just might be able to offset performance penalty of > zfs + nfs + file-based-image. > > ... or if you have the money, simply buy EMC/Netapp-base/Oracle > storage with cache SSDs :D > > -- > Fajar > > _______________________________________________ >Fajar, Isnt''s there a way to use a local SSD drive as cache, when using a SAN? As matter of interest, you can store the EXT3/4 journal on a separate SSD drive to improve performance a bit, but it''s not nearly as effective as using SSD for ZFS''s ZIL + L2ARC cache drives. -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 2:17 AM, Fajar A. Nugraha <list@fajar.net> wrote:> However it''s also the only filesystem that I know of that can > use SSD to speed up both read and write. It''s something that LVM/ext4 > can''t do. So it just might be able to offset performance penalty of > zfs + nfs + file-based-image.there are a few SSD cache projects for Linux(https://github.com/facebook/flashcache, http://bcache.evilpiepirate.org/, http://users.cis.fiu.edu/~zhaom/dmcache/index.html), these operate at block level, accelerating any block device. while the obvious place to put them is in the target box, i''ve been thinking about using on the initiator boxes, to accelerate the iSCSI LUNs : obvious: harddisks ==> RAID/LVM ==> SSD ==> iSCSI target-----(network)---> iSCSI initiator ==> Xen VM food for thought: harddists ==> RAID/LVM ==> iSCSI target ----(network)---> iSCSI initiator ==> SSD cache ==>Xen VM this is feasible for VM images, since each LUN is used by only one machine at a time. clustered filesystems would immediately choke on such a setup. pros: could be _much_ faster, since the cache is local to the user. cons: you have to flush/disable the cache while live-migrating it''s still just an idea, i haven''t found the time (nor machines) to test it. but sounds doable, no? -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Jun 24, 2011 at 2:44 PM, Javier Guerra Giraldez <javier@guerrag.com> wrote:> On Fri, Jun 24, 2011 at 2:17 AM, Fajar A. Nugraha <list@fajar.net> wrote: >> However it''s also the only filesystem that I know of that can >> use SSD to speed up both read and write. It''s something that LVM/ext4 >> can''t do. So it just might be able to offset performance penalty of >> zfs + nfs + file-based-image. > > there are a few SSD cache projects for > Linux(https://github.com/facebook/flashcache, > http://bcache.evilpiepirate.org/, > http://users.cis.fiu.edu/~zhaom/dmcache/index.html), these operate at > block level, accelerating any block device.Cool, thanks for the link :D The comment seems a little discouraging though: "Bcache looks like it''s about ready for non critical use". That''s about the same state as btrfs or zfsonlinux is currently in (both of which I use mostly for dev purposes btw). If you have more experience with bcache (whether good or bad) let us know. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 23/06/11 22:04, Shaun Reitan wrote:> >> iSCSI and AOE export the storage as raw block devices to the >> hypervisors - which basically act the same as local storate (i.e. SATA >> / SAS HDD''s in the server) so you then either need to partition the >> iSCSI LUN the same way as you do with local drives, or the easier way >> is to use LVM to "slice" it up and then use LVM volumes for the domU >> virtual machines >> >> > > > I don''t know about AOE but with ISCSI the problem I''m seeing is that if > i create a volume group on the SAN there is no way to export that volume > group. I could export the device that the volume group was created on > but then the host doesn''t see that VG. I''m assuming i would need to use > clustered LVM for that which i have been told to stay far away from. > Another option was to carve out the LV''s on the SAN and to export each > LV to the initiators but still the problem is that the host assigns > these luns to /dev/sd devices and there''s no simple nice way to map them > to who owns what disk. Least not that I''ve found yet. I''m fairly new to > ISCSI though so maybe there''s something I''m missing. > > ~Shaun >Hi Shaun, I''ve not kept up with what Linux distro. you''re using, but the various udev rules tend to give persistent naming of devices under the /dev/disk/by-<id|path|uuid> directories. For example, in /dev/disk/by-path on one of my Xen dom0''s I can see ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.hinode-ops-lun-0 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.hinode-ops-lun-0-part1 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.hinode-ops-lun-0-part2 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.stereo-ext-lun-0 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.stereo-ext-lun-1 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.ukssdc-helpdesk-lun-0 ip-10.0.0.1:3260-iscsi-iqn.2005-09.uk.ac:storage.ukssdc-helpdesk-lun-1 I then use these paths in the vm config file. This is then independent of the order I login to the iscsi devices. Matthew -- Matthew Wild Tel.: +44 (0)1235 445173 Matthew.Wild@stfc.ac.uk URL http://www.ukssdc.ac.uk/ UK Solar System Data Centre and World Data Centre - Solar-Terrestrial Physics, Chilton Rutherford Appleton Laboratory, Chilton, Didcot, Oxon, OX11 0QX -- Scanned by iCritical. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users