Hi people... Here we use Xen 4 with Debian Lenny... We''re using kernel 2.6.31.13 pvops... As a storage system, we use AoE devices... So, we installed VM''s on AoE partition... The "NAS" server is a Intel based baremetal with SATA hard disc... However, sometime I feeling that VM''s is so slow... Also, all VM has GPLPV drivers installed... So, I am thing about change AoE to iSCSCI... However, I have many doubt with this procedure... Is it iSCSI better than AoE??? Please, if someboy can clarify this matter, I''ll be very thanks... Regards Gilberto Nunes TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 14:08:48 Gilberto Nunes wrote:> Hi people... > > Here we use Xen 4 with Debian Lenny... We''re using kernel 2.6.31.13 > pvops... > As a storage system, we use AoE devices... > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > based baremetal with SATA hard disc... > > However, sometime I feeling that VM''s is so slow... > > Also, all VM has GPLPV drivers installed... > > So, I am thing about change AoE to iSCSCI... > > However, I have many doubt with this procedure... Is it iSCSI better > than AoE??? > > Please, if someboy can clarify this matter, I''ll be very thanks... > > Regards > > > > Gilberto Nunes > TI > Selbetti Gestão de Documentos > Telefone: +55 (47) 3441-6004 > Celular: +55 (47) 8861-6672 > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >iSCSI will never outperform AoE in my opinion. But in stead of randomly switching technologies, I think you better try to find the culprit before. Like your storage itself for instance. SATA is not the fastest. What kind of RAID are you using and such ... (iotop?). Also what does your storage network look like. These can all be bottlenecks. B. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Hi people... > > Here we use Xen 4 with Debian Lenny... We're using kernel 2.6.31.13 > pvops... > As a storage system, we use AoE devices... > So, we installed VM's on AoE partition... The "NAS" server is a Intel > based baremetal with SATA hard disc... > > However, sometime I feeling that VM's is so slow... > > Also, all VM has GPLPV drivers installed... > > So, I am thing about change AoE to iSCSCI... > > However, I have many doubt with this procedure... Is it iSCSI better > than AoE??? >Over a single 'server' class network interface, iSCSI is probably better than AoE due to the TCP offload functions found in such adapters. If you are using the reference 'vblade' implementation of the AoE server then it is definitely known to suck performance-wise (see list archives for a discussion). I think the only way to be sure in your case is to set up iSCSI and AoE and run some performance tests yourself. We can argue the theoretical benefits of each on the list as much as we like, but the only way to be sure for yourself is to do some testing for yourself :) James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi!> As a storage system, we use AoE devices... > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > based baremetal with SATA hard disc...What target are you using? (vblade, gqaoed, ...) How many VMs? How many disks? How many Gigabit connectors between Dom0''s and NAS server?> However, sometime I feeling that VM''s is so slow...Why do you think AoE is the bottleneck? Did you already do some checks? -- Adi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I see... Well... Let me give more details here... I delivery aoe with a dedicated gigalan crossover between NAS Server and Xen Server... So, all traffic between this two server ran over this cross cable... The Xen server is a Dell PowerEdge 1950 and NAS Server is an Intel based PC as I said before... Sometimes, AoE seems lost connection with vblade, then one or two VM get down, and I can''t return this VM... So I restart both servers! (Amazing!!) I looking for better solutions here... Maybe I''ll install iSCSI and take some tests... Thanks for reply... Em Seg, 2010-07-05 às 14:14 +0200, Bart Coninckx escreveu:> On Monday 05 July 2010 14:08:48 Gilberto Nunes wrote: > > Hi people... > > > > Here we use Xen 4 with Debian Lenny... We''re using kernel 2.6.31.13 > > pvops... > > As a storage system, we use AoE devices... > > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > > based baremetal with SATA hard disc... > > > > However, sometime I feeling that VM''s is so slow... > > > > Also, all VM has GPLPV drivers installed... > > > > So, I am thing about change AoE to iSCSCI... > > > > However, I have many doubt with this procedure... Is it iSCSI better > > than AoE??? > > > > Please, if someboy can clarify this matter, I''ll be very thanks... > > > > Regards > > > > > > > > Gilberto Nunes > > TI > > Selbetti Gestão de Documentos > > Telefone: +55 (47) 3441-6004 > > Celular: +55 (47) 8861-6672 > > > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > iSCSI will never outperform AoE in my opinion. But in stead of randomly > switching technologies, I think you better try to find the culprit before. > Like your storage itself for instance. SATA is not the fastest. What kind of > RAID are you using and such ... (iotop?). Also what does your storage network > look like. These can all be bottlenecks. > > B. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersGilberto Nunes TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Adi... Answer your questions: I''m use vblade. Why??? Is not the better thing??? Actually I never use another piece of software.. I''m with about 13 VM: 12 with Windows 2003 Server and 1 with Debian Lenny... Between Nas and Xen server I have a dedicated Gigabyte crossover connection, with cable cat 5e... I hope this plus information can help... Thanks Em Seg, 2010-07-05 às 14:19 +0200, Adi Kriegisch escreveu:> Hi! > > > As a storage system, we use AoE devices... > > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > > based baremetal with SATA hard disc... > What target are you using? (vblade, gqaoed, ...) > How many VMs? How many disks? How many Gigabit connectors between Dom0''s > and NAS server? > > > However, sometime I feeling that VM''s is so slow... > Why do you think AoE is the bottleneck? Did you already do some checks? > > -- Adi > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersGilberto Nunes TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 14:27:20 Gilberto Nunes wrote:> I see... > > Well... Let me give more details here... > > I delivery aoe with a dedicated gigalan crossover between NAS Server and > Xen Server... > So, all traffic between this two server ran over this cross cable... > > The Xen server is a Dell PowerEdge 1950 and NAS Server is an Intel based > PC as I said before... > > Sometimes, AoE seems lost connection with vblade, then one or two VM get > down, and I can''t return this VM... So I restart both servers! > (Amazing!!) > > I looking for better solutions here... Maybe I''ll install iSCSI and take > some tests... > > Thanks for reply... > > Em Seg, 2010-07-05 às 14:14 +0200, Bart Coninckx escreveu: > > On Monday 05 July 2010 14:08:48 Gilberto Nunes wrote: > > > Hi people... > > > > > > Here we use Xen 4 with Debian Lenny... We''re using kernel 2.6.31.13 > > > pvops... > > > As a storage system, we use AoE devices... > > > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > > > based baremetal with SATA hard disc... > > > > > > However, sometime I feeling that VM''s is so slow... > > > > > > Also, all VM has GPLPV drivers installed... > > > > > > So, I am thing about change AoE to iSCSCI... > > > > > > However, I have many doubt with this procedure... Is it iSCSI better > > > than AoE??? > > > > > > Please, if someboy can clarify this matter, I''ll be very thanks... > > > > > > Regards > > > > > > > > > > > > Gilberto Nunes > > > TI > > > Selbetti Gestão de Documentos > > > Telefone: +55 (47) 3441-6004 > > > Celular: +55 (47) 8861-6672 > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > iSCSI will never outperform AoE in my opinion. But in stead of randomly > > switching technologies, I think you better try to find the culprit > > before. Like your storage itself for instance. SATA is not the fastest. > > What kind of RAID are you using and such ... (iotop?). Also what does > > your storage network look like. These can all be bottlenecks. > > > > B. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > Gilberto Nunes > TI > Selbetti Gestão de Documentos > Telefone: +55 (47) 3441-6004 > Celular: +55 (47) 8861-6672 >That''s working in reverse order. You need to test the speed of your storage locally first. That means without iSCSI, AoE ot whatever. If this already is slower than your network speed, you can do whatever you want with AoE or iSCSI, it won''t matter. Optimally you would test the IOPS, but you will get quite a good indication by running bonnie++ locally. If you hit less than 120 MB/sec in reading and/or writing, you have your bottleneck. Changing RAID5 to RAID50 or RAID1 to RAID10 can then already do a world of good. If that is not the bottleneck, you might consider upgrading your network path by using NIC bonding or multipathing. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 14:36:40 Gilberto Nunes wrote:> Hi Adi... > > Answer your questions: > > I''m use vblade. Why??? Is not the better thing??? Actually I never use > another piece of software.. > > I''m with about 13 VM: 12 with Windows 2003 Server and 1 with Debian > Lenny... > > Between Nas and Xen server I have a dedicated Gigabyte crossover > connection, with cable cat 5e... > > I hope this plus information can help... > > Thanks > > Em Seg, 2010-07-05 às 14:19 +0200, Adi Kriegisch escreveu: > > Hi! > > > > > As a storage system, we use AoE devices... > > > So, we installed VM''s on AoE partition... The "NAS" server is a Intel > > > based baremetal with SATA hard disc... > > > > What target are you using? (vblade, gqaoed, ...) > > How many VMs? How many disks? How many Gigabit connectors between Dom0''s > > and NAS server? > > > > > However, sometime I feeling that VM''s is so slow... > > > > Why do you think AoE is the bottleneck? Did you already do some checks? > > > > -- Adi > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > Gilberto Nunes > TI > Selbetti Gestão de Documentos > Telefone: +55 (47) 3441-6004 > Celular: +55 (47) 8861-6672 > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >You run 13 Windows machines of one gigabit connection and SATA disks?!? Man, that is pushing it. But I''m glad to hear your only complaint is "sometimes feeling slow". _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi!> I delivery aoe with a dedicated gigalan crossover between NAS Server and Xen > Server... > So, all traffic between this two server ran over this cross cable...Ok, so you''re limited to 1GBit in the storage backend.> The Xen server is a Dell PowerEdge 1950 and NAS Server is an Intel based PC as > I said before...Ok, you probably should check if your Dom0 has irqbalance installed[1]; this could help a little in boosting performance.> Sometimes, AoE seems lost connection with vblade, then one or two VM get down, > and I can''t return this VM... So I restart both servers! (Amazing!!)''vblade'' -- as mentioned before -- should be considered a reference implementation. It is single threaded and has no real queuing options. Just use ggaoed[2] or qaoed[3] and I am pretty sure, performance will improve. Another issue might be the cross-over cabling solution: You might want to check syslog on both servers to see if the connection is stable. Further more there are some NICs that are unable to provide desired performance with jumbo frames. There are even more cheapo NICs that do not provide performance at all. I''d suggest to find the culprit. There seem to be severe issues "somewhere", go find them. Or you won''t be happy with iSCSI either... -- Adi [1] http://lists.xensource.com/archives/html/xen-users/2010-04/msg00577.html [2] http://code.google.com/p/ggaoed/ [3] http://code.google.com/p/qaoed/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi!> > Answer your questions: > > > > I''m use vblade. Why??? Is not the better thing??? Actually I never use > > another piece of software.. > > > > I''m with about 13 VM: 12 with Windows 2003 Server and 1 with Debian > > Lenny... > > > > Between Nas and Xen server I have a dedicated Gigabyte crossover > > connection, with cable cat 5e... > You run 13 Windows machines of one gigabit connection and SATA disks?!? > Man, that is pushing it. But I''m glad to hear your only complaint is > "sometimes feeling slow".I agree... Just adding a second Gigabit interface may improve the situation. ;-) But (see my other mail as well): Use another AoE target implementation! (ggaoed is your friend. It will handle the load balancing of multiple interfaces automatically, as will the aoe driver -- aoe6-75 is the current one) Another thing that comes to my mind is the amount of disks you have. I''d say a not too busy Windows machine will fill up 1/3rd of the I/O pipe of a single SATA disk (I am not talking about peak performance). So you''ll need 13*(1/3)= ~5 disks to satisfy ''normal'' and idle load. Then add one disk for RAID5. So 6 disks in a RAID5 will feel a little slow when one VM starts to work... You will probably need more disks and/or faster disks. -- Adi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 14:58:42 Adi Kriegisch wrote:> Hi! > > > > Answer your questions: > > > > > > I''m use vblade. Why??? Is not the better thing??? Actually I never use > > > another piece of software.. > > > > > > I''m with about 13 VM: 12 with Windows 2003 Server and 1 with Debian > > > Lenny... > > > > > > Between Nas and Xen server I have a dedicated Gigabyte crossover > > > connection, with cable cat 5e... > > > > You run 13 Windows machines of one gigabit connection and SATA disks?!? > > Man, that is pushing it. But I''m glad to hear your only complaint is > > "sometimes feeling slow". > > I agree... Just adding a second Gigabit interface may improve the > situation. ;-) > > But (see my other mail as well): Use another AoE target implementation! > (ggaoed is your friend. It will handle the load balancing of multiple > interfaces automatically, as will the aoe driver -- aoe6-75 is the current > one) > > Another thing that comes to my mind is the amount of disks you have. I''d > say a not too busy Windows machine will fill up 1/3rd of the I/O pipe of a > single SATA disk (I am not talking about peak performance). So you''ll need > 13*(1/3)= ~5 disks to satisfy ''normal'' and idle load. Then add one disk for > RAID5. So 6 disks in a RAID5 will feel a little slow when one VM starts to > work... You will probably need more disks and/or faster disks. > > -- Adi >I agree on that, start off with your local storage. If it hits the 120 MB/sec ceiling, start highering your network bandwidth. 13 is a sh*tload of Windows guests, they need to be pampered with quite some I/O. Using RAID50 might make up for not having SAS if you use enough drives. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I agree on that, start off with your local storage. If it hits the 120 MB/sec ceiling, start highering your network bandwidth. 13 is a sh*tload of Windows guests, they need to be pampered with quite some I/O. Using RAID50 might make up for not having SAS if you use enough drives. --------------------------------------------------------------------------------------------------------------------------------- Sorry to hijack the thread, but is it only WIndows VMs that causes these problems? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 15:16:38 Jonathan Tripathy wrote:> I agree on that, start off with your local storage. If it hits the 120 > MB/sec ceiling, start highering your network bandwidth. > 13 is a sh*tload of Windows guests, they need to be pampered with quite > some I/O. Using RAID50 might make up for not having SAS if you use enough > drives. > > > --------------------------------------------------------------------------- > ------------------------------------------------------ > > Sorry to hijack the thread, but is it only WIndows VMs that causes these > problems? >No, but they will cause them sooner. Well, actually, relatively. It is a known fact that Windows in general used more resources than Linux, but if you have a Linux do way more than a Windows, it could also lead to over-suing your available resources. It''s simple math really: if demand > resources then problematic performance fi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Just to complemente...
I run bonnie++ like this:
bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root | bon_csv2txt >
test.txt
This is the result:
Version 1.03c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
bacula-selbet 1000M 53004 98 189796 37 97783 17 62844 99 1505552 99
+++++ +++
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
bacula-selbetti 1 483 3 +++++ +++ 307 1 455 5 +++++ +++
259 0
It''s tell something?
Thanks
Em Seg, 2010-07-05 às 14:42 +0200, Bart Coninckx escreveu:
> On Monday 05 July 2010 14:27:20 Gilberto Nunes wrote:
> > I see...
> >
> > Well... Let me give more details here...
> >
> > I delivery aoe with a dedicated gigalan crossover between NAS Server
and
> > Xen Server...
> > So, all traffic between this two server ran over this cross cable...
> >
> > The Xen server is a Dell PowerEdge 1950 and NAS Server is an Intel
based
> > PC as I said before...
> >
> > Sometimes, AoE seems lost connection with vblade, then one or two VM
get
> > down, and I can''t return this VM... So I restart both
servers!
> > (Amazing!!)
> >
> > I looking for better solutions here... Maybe I''ll install
iSCSI and take
> > some tests...
> >
> > Thanks for reply...
> >
> > Em Seg, 2010-07-05 às 14:14 +0200, Bart Coninckx escreveu:
> > > On Monday 05 July 2010 14:08:48 Gilberto Nunes wrote:
> > > > Hi people...
> > > >
> > > > Here we use Xen 4 with Debian Lenny... We''re using
kernel 2.6.31.13
> > > > pvops...
> > > > As a storage system, we use AoE devices...
> > > > So, we installed VM''s on AoE partition... The
"NAS" server is a Intel
> > > > based baremetal with SATA hard disc...
> > > >
> > > > However, sometime I feeling that VM''s is so slow...
> > > >
> > > > Also, all VM has GPLPV drivers installed...
> > > >
> > > > So, I am thing about change AoE to iSCSCI...
> > > >
> > > > However, I have many doubt with this procedure... Is it
iSCSI better
> > > > than AoE???
> > > >
> > > > Please, if someboy can clarify this matter, I''ll be
very thanks...
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > > Gilberto Nunes
> > > > TI
> > > > Selbetti Gestão de Documentos
> > > > Telefone: +55 (47) 3441-6004
> > > > Celular: +55 (47) 8861-6672
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Xen-users mailing list
> > > > Xen-users@lists.xensource.com
> > > > http://lists.xensource.com/xen-users
> > >
> > > iSCSI will never outperform AoE in my opinion. But in stead of
randomly
> > > switching technologies, I think you better try to find the
culprit
> > > before. Like your storage itself for instance. SATA is not the
fastest.
> > > What kind of RAID are you using and such ... (iotop?). Also what
does
> > > your storage network look like. These can all be bottlenecks.
> > >
> > > B.
> > >
> > > _______________________________________________
> > > Xen-users mailing list
> > > Xen-users@lists.xensource.com
> > > http://lists.xensource.com/xen-users
> >
> > Gilberto Nunes
> > TI
> > Selbetti Gestão de Documentos
> > Telefone: +55 (47) 3441-6004
> > Celular: +55 (47) 8861-6672
> >
>
> That''s working in reverse order. You need to test the speed of
your storage
> locally first. That means without iSCSI, AoE ot whatever. If this already
is
> slower than your network speed, you can do whatever you want with AoE or
> iSCSI, it won''t matter. Optimally you would test the IOPS, but you
will get
> quite a good indication by running bonnie++ locally. If you hit less than
120
> MB/sec in reading and/or writing, you have your bottleneck. Changing RAID5
to
> RAID50 or RAID1 to RAID10 can then already do a world of good.
> If that is not the bottleneck, you might consider upgrading your network
path
> by using NIC bonding or multipathing.
>
>
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
Gilberto Nunes
TI
Selbetti Gestão de Documentos
Telefone: +55 (47) 3441-6004
Celular: +55 (47) 8861-6672
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Hi!> I run bonnie++ like this: > bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root | bon_csv2txt > test.txtjust checking: your storage server has 500MB RAM? (-r)> This is the result: > > Version 1.03c ------Sequential Output------ --Sequential Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP > bacula-selbet 1000M 53004 98 189796 37 97783 17 62844 99 1505552 99 +++++[SNIP]> It''s tell something?Ja, your storage system can handle ~190MB/s sequential write. This means you will not get full peak performance to your clients as one gigabit interface is limited with 120MB/s. Your write speed (1,15GB/s) shows that you misspecified RAM size on your bonnie commandline because this is _WAY_ beyond what your disks are able to handle. (Good SATA disks will give you above 100MB/s read speed. Reading at that speed hints at 15 or more disks; the limit there is definitely bus speed and administrative overhead.) What you are really interested in (or should be) are IOPS (Input Output Operations per Second): A typical server or workstation no matter if virtual or ''real'' does a mixture between sequential and random I/O. Every server you run has its own partition on your storage backend. Just to get a better idea of what I am talking about consider the following: Every virtual machine does a sequential file read. What does that mean on the storage backend? -- There are 13 files being read at 13 different positions at the same time. This is a (close to) random I/O workload. Disk heads are flying around to satisfy all requests. No way you will be close to any high MB/s value: your disks are doing random I/O. Measuring sequential peak performances on network storage is pointless for this very reason. (People on this list were suggesting to do that just to verify your disk subsystem works fine.) To get an idea of what performance you might expect, you can do the following: 1. calculate IOPS that you might expect. You may use one of the online calculators that are available[1]. This begins with calculating IOPS per disk where you might need to consult your vendor''s datasheet or lookup the disks here[2]. You''ll immediately notice that SAS disks offer twice or more IOPS than SATA drives. When calculating IOPS you need to specify a workload as well. This means specify the read/write ratio. Average fileservers have around 80% read and 20% write. Read and write operations differ in the latency they have: The more latency a request has the fewer requests can be handled per second. (This is also the reason why local storage will always bring more IOPS than network storage: network transport simply adds to latency.) 2. measure the IOPS you get. I personally prefer using FIO[3] which is readily available in Debian. FIO is fully configurable; there are however some reasonable examples which you might use: /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical file server workload with 80% read. The IOPS calculator above[1] is only able to calculate IOPS for a fixed block size where this workload mixes blocksizes from 512byte to 64k. The result in IOPS cannot be directly compared. If you want to do so, you need to specify 4k blocks only in the config. WARNING: Do not use IOMeter itself on linux: it provides incorrect results as it cannot use aio on linux and therefor is unable to queue requests. Using the stock ''iometer-file-access-server'' profile you should get the something like: 3 disks/RAID5: 200-250 IOPS 4 disks/RAID5: 270-320 IOPS 5 disks/RAID5: 340-390 IOPS and so on (for SATA disks with AoE). 3. find the bottleneck in case you''re not getting what you can expect. Measure IOPS on the storage server with ''iostat 1'' ("tps" roughly corresponds to IOPS). ...ok, writing up how to debug a storage backend will take another hour... ask me if necessary. -- Adi [1] http://www.wmarow.com/strcalc/ [2] http://www.wmarow.com/strdir/hdd/ [3] http://freshmeat.net/projects/fio PS: Maybe there should be a wiki page about how to plan and implement a storage backend for a xen server? -- then others can add their knowledge more easily. ...and the question pops up every once in a while. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 18:43:20 Adi Kriegisch wrote:> Hi! > > > I run bonnie++ like this: > > bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root | bon_csv2txt > > > test.txt > > just checking: your storage server has 500MB RAM? (-r) > > > This is the result: > > > > Version 1.03c ------Sequential Output------ --Sequential Input- > > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > > /sec %CP bacula-selbet 1000M 53004 98 189796 37 97783 17 62844 99 > > 1505552 99 +++++ > > [SNIP] > > > It''s tell something? > > Ja, your storage system can handle ~190MB/s sequential write. This means > you will not get full peak performance to your clients as one gigabit > interface is limited with 120MB/s. > Your write speed (1,15GB/s) shows that you misspecified RAM size on your > bonnie commandline because this is _WAY_ beyond what your disks are able to > handle. (Good SATA disks will give you above 100MB/s read speed. Reading at > that speed hints at 15 or more disks; the limit there is definitely bus > speed and administrative overhead.) > > What you are really interested in (or should be) are IOPS (Input Output > Operations per Second): A typical server or workstation no matter if > virtual or ''real'' does a mixture between sequential and random I/O. > Every server you run has its own partition on your storage backend. Just to > get a better idea of what I am talking about consider the following: > Every virtual machine does a sequential file read. What does that mean on > the storage backend? -- There are 13 files being read at 13 different > positions at the same time. This is a (close to) random I/O workload. Disk > heads are flying around to satisfy all requests. No way you will be close > to any high MB/s value: your disks are doing random I/O. > Measuring sequential peak performances on network storage is pointless for > this very reason. (People on this list were suggesting to do that just to > verify your disk subsystem works fine.) > To get an idea of what performance you might expect, you can do the > following: > 1. calculate IOPS that you might expect. You may use one of the online > calculators that are available[1]. > This begins with calculating IOPS per disk where you might need to > consult your vendor''s datasheet or lookup the disks here[2]. You''ll > immediately notice that SAS disks offer twice or more IOPS than SATA > drives. > When calculating IOPS you need to specify a workload as well. This means > specify the read/write ratio. Average fileservers have around 80% read > and 20% write. Read and write operations differ in the latency they > have: The more latency a request has the fewer requests can be handled > per second. (This is also the reason why local storage will always bring > more IOPS than network storage: network transport simply adds to > latency.) 2. measure the IOPS you get. I personally prefer using FIO[3] > which is readily available in Debian. FIO is fully configurable; there are > however some reasonable examples which you might use: > /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical > file server workload with 80% read. The IOPS calculator above[1] is only > able to calculate IOPS for a fixed block size where this workload mixes > blocksizes from 512byte to 64k. The result in IOPS cannot be directly > compared. If you want to do so, you need to specify 4k blocks only in > the config. > WARNING: Do not use IOMeter itself on linux: it provides incorrect > results as it cannot use aio on linux and therefor is unable to queue > requests. > Using the stock ''iometer-file-access-server'' profile you should get the > something like: > 3 disks/RAID5: 200-250 IOPS > 4 disks/RAID5: 270-320 IOPS > 5 disks/RAID5: 340-390 IOPS > and so on (for SATA disks with AoE). > 3. find the bottleneck in case you''re not getting what you can expect. > Measure IOPS on the storage server with ''iostat 1'' ("tps" roughly > corresponds to IOPS). > ...ok, writing up how to debug a storage backend will take another > hour... ask me if necessary. > > -- Adi > > [1] http://www.wmarow.com/strcalc/ > [2] http://www.wmarow.com/strdir/hdd/ > [3] http://freshmeat.net/projects/fio > > PS: Maybe there should be a wiki page about how to plan and implement a > storage backend for a xen server? -- then others can add their knowledge > more easily. > ...and the question pops up every once in a while. >Adi, most educational, thank you. B. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hi again... I''m try to replace vblade to ggaoed, but I don''t know how to use it... I''m already installed it on server side, but I don''t know how use it on client side... Perhaps I can use aoetools too??? Please, somebody can point some how to??? Thanks a lot Em Seg, 2010-07-05 às 19:45 +0200, Bart Coninckx escreveu:> On Monday 05 July 2010 18:43:20 Adi Kriegisch wrote: > > Hi! > > > > > I run bonnie++ like this: > > > bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root | bon_csv2txt > > > > test.txt > > > > just checking: your storage server has 500MB RAM? (-r) > > > > > This is the result: > > > > > > Version 1.03c ------Sequential Output------ --Sequential Input- > > > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > > > /sec %CP bacula-selbet 1000M 53004 98 189796 37 97783 17 62844 99 > > > 1505552 99 +++++ > > > > [SNIP] > > > > > It''s tell something? > > > > Ja, your storage system can handle ~190MB/s sequential write. This means > > you will not get full peak performance to your clients as one gigabit > > interface is limited with 120MB/s. > > Your write speed (1,15GB/s) shows that you misspecified RAM size on your > > bonnie commandline because this is _WAY_ beyond what your disks are able to > > handle. (Good SATA disks will give you above 100MB/s read speed. Reading at > > that speed hints at 15 or more disks; the limit there is definitely bus > > speed and administrative overhead.) > > > > What you are really interested in (or should be) are IOPS (Input Output > > Operations per Second): A typical server or workstation no matter if > > virtual or ''real'' does a mixture between sequential and random I/O. > > Every server you run has its own partition on your storage backend. Just to > > get a better idea of what I am talking about consider the following: > > Every virtual machine does a sequential file read. What does that mean on > > the storage backend? -- There are 13 files being read at 13 different > > positions at the same time. This is a (close to) random I/O workload. Disk > > heads are flying around to satisfy all requests. No way you will be close > > to any high MB/s value: your disks are doing random I/O. > > Measuring sequential peak performances on network storage is pointless for > > this very reason. (People on this list were suggesting to do that just to > > verify your disk subsystem works fine.) > > To get an idea of what performance you might expect, you can do the > > following: > > 1. calculate IOPS that you might expect. You may use one of the online > > calculators that are available[1]. > > This begins with calculating IOPS per disk where you might need to > > consult your vendor''s datasheet or lookup the disks here[2]. You''ll > > immediately notice that SAS disks offer twice or more IOPS than SATA > > drives. > > When calculating IOPS you need to specify a workload as well. This means > > specify the read/write ratio. Average fileservers have around 80% read > > and 20% write. Read and write operations differ in the latency they > > have: The more latency a request has the fewer requests can be handled > > per second. (This is also the reason why local storage will always bring > > more IOPS than network storage: network transport simply adds to > > latency.) 2. measure the IOPS you get. I personally prefer using FIO[3] > > which is readily available in Debian. FIO is fully configurable; there are > > however some reasonable examples which you might use: > > /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical > > file server workload with 80% read. The IOPS calculator above[1] is only > > able to calculate IOPS for a fixed block size where this workload mixes > > blocksizes from 512byte to 64k. The result in IOPS cannot be directly > > compared. If you want to do so, you need to specify 4k blocks only in > > the config. > > WARNING: Do not use IOMeter itself on linux: it provides incorrect > > results as it cannot use aio on linux and therefor is unable to queue > > requests. > > Using the stock ''iometer-file-access-server'' profile you should get the > > something like: > > 3 disks/RAID5: 200-250 IOPS > > 4 disks/RAID5: 270-320 IOPS > > 5 disks/RAID5: 340-390 IOPS > > and so on (for SATA disks with AoE). > > 3. find the bottleneck in case you''re not getting what you can expect. > > Measure IOPS on the storage server with ''iostat 1'' ("tps" roughly > > corresponds to IOPS). > > ...ok, writing up how to debug a storage backend will take another > > hour... ask me if necessary. > > > > -- Adi > > > > [1] http://www.wmarow.com/strcalc/ > > [2] http://www.wmarow.com/strdir/hdd/ > > [3] http://freshmeat.net/projects/fio > > > > PS: Maybe there should be a wiki page about how to plan and implement a > > storage backend for a xen server? -- then others can add their knowledge > > more easily. > > ...and the question pops up every once in a while. > > > > Adi, most educational, thank you. > > B. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersGilberto Nunes TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi!> I''m try to replace vblade to ggaoed, but I don''t know how to use it... > I''m already installed it on server side, but I don''t know how use it on > client side...exactly like vblade -- they''re both doing the same thing: provide local storage via network using the ATA-over-Ethernet protocol.> Perhaps I can use aoetools too???Yes, you should install them. They contain useful tools and init scripts like: aoe-discover -- to start scanning for AoE targets in your network or aoe-stat -- to get a list of all available targets. Do not forget to have a quick look at /etc/default/aoetools! On the client side you only need to load the "aoe" kernel module. The version in the kernel is v47 -- upstream[1] already provides version v75. I recommend to download and install the newer module version upstream provides. (I already created some debian kmod packages for this but they''re not ready for prime time yet.) -- Adi [1] http://support.coraid.com/support/linux/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks you again!! I''ll try it! Em Ter, 2010-07-06 às 15:30 +0200, Adi Kriegisch escreveu:> Hi! > > > I''m try to replace vblade to ggaoed, but I don''t know how to use it... > > I''m already installed it on server side, but I don''t know how use it on > > client side... > exactly like vblade -- they''re both doing the same thing: provide local > storage via network using the ATA-over-Ethernet protocol. > > > Perhaps I can use aoetools too??? > Yes, you should install them. They contain useful tools and init scripts > like: > aoe-discover -- to start scanning for AoE targets in your network or > aoe-stat -- to get a list of all available targets. > Do not forget to have a quick look at /etc/default/aoetools! > > On the client side you only need to load the "aoe" kernel module. The > version in the kernel is v47 -- upstream[1] already provides version v75. > I recommend to download and install the newer module version upstream > provides. (I already created some debian kmod packages for this but they''re > not ready for prime time yet.) > > -- Adi > > [1] http://support.coraid.com/support/linux/Gilberto Nunes TI Selbetti Gestão de Documentos Telefone: +55 (47) 3441-6004 Celular: +55 (47) 8861-6672 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Monday 05 July 2010 18:43:20 Adi Kriegisch wrote:> Hi! > > > I run bonnie++ like this: > > bonnie++ -d /tmp/ -s 1000 -r 500 -n 1 -x 1 -u root | bon_csv2txt > > > test.txt > > just checking: your storage server has 500MB RAM? (-r) > > > This is the result: > > > > Version 1.03c ------Sequential Output------ --Sequential Input- > > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > > /sec %CP bacula-selbet 1000M 53004 98 189796 37 97783 17 62844 99 > > 1505552 99 +++++ > > [SNIP] > > > It''s tell something? > > Ja, your storage system can handle ~190MB/s sequential write. This means > you will not get full peak performance to your clients as one gigabit > interface is limited with 120MB/s. > Your write speed (1,15GB/s) shows that you misspecified RAM size on your > bonnie commandline because this is _WAY_ beyond what your disks are able to > handle. (Good SATA disks will give you above 100MB/s read speed. Reading at > that speed hints at 15 or more disks; the limit there is definitely bus > speed and administrative overhead.) > > What you are really interested in (or should be) are IOPS (Input Output > Operations per Second): A typical server or workstation no matter if > virtual or ''real'' does a mixture between sequential and random I/O. > Every server you run has its own partition on your storage backend. Just to > get a better idea of what I am talking about consider the following: > Every virtual machine does a sequential file read. What does that mean on > the storage backend? -- There are 13 files being read at 13 different > positions at the same time. This is a (close to) random I/O workload. Disk > heads are flying around to satisfy all requests. No way you will be close > to any high MB/s value: your disks are doing random I/O. > Measuring sequential peak performances on network storage is pointless for > this very reason. (People on this list were suggesting to do that just to > verify your disk subsystem works fine.) > To get an idea of what performance you might expect, you can do the > following: > 1. calculate IOPS that you might expect. You may use one of the online > calculators that are available[1]. > This begins with calculating IOPS per disk where you might need to > consult your vendor''s datasheet or lookup the disks here[2]. You''ll > immediately notice that SAS disks offer twice or more IOPS than SATA > drives. > When calculating IOPS you need to specify a workload as well. This means > specify the read/write ratio. Average fileservers have around 80% read > and 20% write. Read and write operations differ in the latency they > have: The more latency a request has the fewer requests can be handled > per second. (This is also the reason why local storage will always bring > more IOPS than network storage: network transport simply adds to > latency.) 2. measure the IOPS you get. I personally prefer using FIO[3] > which is readily available in Debian. FIO is fully configurable; there are > however some reasonable examples which you might use: > /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical > file server workload with 80% read. The IOPS calculator above[1] is only > able to calculate IOPS for a fixed block size where this workload mixes > blocksizes from 512byte to 64k. The result in IOPS cannot be directly > compared. If you want to do so, you need to specify 4k blocks only in > the config. > WARNING: Do not use IOMeter itself on linux: it provides incorrect > results as it cannot use aio on linux and therefor is unable to queue > requests. > Using the stock ''iometer-file-access-server'' profile you should get the > something like: > 3 disks/RAID5: 200-250 IOPS > 4 disks/RAID5: 270-320 IOPS > 5 disks/RAID5: 340-390 IOPS > and so on (for SATA disks with AoE). > 3. find the bottleneck in case you''re not getting what you can expect. > Measure IOPS on the storage server with ''iostat 1'' ("tps" roughly > corresponds to IOPS). > ...ok, writing up how to debug a storage backend will take another > hour... ask me if necessary. > > -- Adi > > [1] http://www.wmarow.com/strcalc/ > [2] http://www.wmarow.com/strdir/hdd/ > [3] http://freshmeat.net/projects/fio > > PS: Maybe there should be a wiki page about how to plan and implement a > storage backend for a xen server? -- then others can add their knowledge > more easily. > ...and the question pops up every once in a while. >Adi, I have been looking at FIO, but what jobfile do you use that you find optimal to test network storage for Xen? cheers, B. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi! [SNIP]> > latency.) 2. measure the IOPS you get. I personally prefer using FIO[3] > > which is readily available in Debian. FIO is fully configurable; there are > > however some reasonable examples which you might use: > > /usr/share/doc/fio/examples/iometer-file-access-server mimiks a typical > > file server workload with 80% read. The IOPS calculator above[1] is only[SNAP]> I have been looking at FIO, but what jobfile do you use that you find optimal > to test network storage for Xen?Actually this is a very hard to answer question! ;-) Short answer: I use iometer-file-access-server with (1) 80% (2) 100% (3) 0% read. Then I have a feeling for what I may expect from the hardware... Long answer: Benchmarking is actually about lying -- depending on who is doing those benchmarks. There is an excellent presentation about benchmarking by Greg Smith of Postgres fame[1] stating some universal truths about how to conduct benchmarks. The most important part is to know what you need: The more details you have the less you''re lying or the less you''re in danger of being lied to. In benchmarking terms this means defining an I/O profile. To get that right you need to monitor your existing infrastructure (eg by collecting the output of ''iostat'') and conducting the very same benchmarks you''re doing on your new hardware on the old one as well. One of my servers reports the following on iostat for a disk containing some mailboxes: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 45.40 356.65 158.90 2005866781 893707736 This means: Since power-on (65 days in this case) the server does 45 IOPS on average per second. Now it is up to me to know if this server does about the same amount of work 24/7 or if it is only busy during the day (which means that you need to calculate two or three times the IOPS for 12 or 8 hours being really busy. (This is a file server scenario in an average office. ie: 45*(24/12) or 45*(24/8)) The next thing I get is the ratio between reads and writes: 1% .............. (2005866781 + 893707736)/100 read percent .... 2005866781/2005866781.0/percent = 69.2 % write percent ... 893707736.0/percent = 30.8 % There is one more very important thing in the output of iostat: (this is actually from a different machine than the above) the average cpu usage: avg-cpu: %user %nice %system %iowait %steal %idle 10.35 2.44 5.69 8.63 0.00 72.89 This shows that the system spends more than 8% of its time on waiting for the completion of outstanding I/Os (which is a bad thing ;-) ). With those numbers one is able to get a better understanding of what servers are doing. Using that information gives the ability to create a set of FIO jobfiles that roughly describe the workload expected and shows wether a storage backend (and in parts the rest of the systems) is able to handle that. When you''re doing benchmarks from time to time you''ll get a feeling for what a storage backend can handle when looking at FIO''s results. Using your own jobfiles with more jobs running in parallel (numjobs=..), using different block sizes (either blocksize=Xk where x is 4,8,... or mixing blocksizes as done in iometer-file-access-server) and finding a balance between read and write transactions you might see more clearly if a storage system can handla your specific workload. Do those benchmarks in your own setup. Do not let someone else do them for you. In case of network storage, be it iSCSI, AoE, NFS or whatever, refrain running the benchmarks on the storage system itself: The results will not reflect the real throughput of the system, numbers will almost always be higher! Uh, quite a lot of text again... ;) Hope this helps! Feedback and discussion apprechiated... -- Adi [1] http://www.pgcon.org/2009/schedule/attachments/123_pg-benchmarking-2.pdf _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thursday 08 July 2010 14:18:00 Adi Kriegisch wrote:> Hi! > > [SNIP] > > > > latency.) 2. measure the IOPS you get. I personally prefer using > > > FIO[3] which is readily available in Debian. FIO is fully configurable; > > > there are however some reasonable examples which you might use: > > > /usr/share/doc/fio/examples/iometer-file-access-server mimiks a > > > typical file server workload with 80% read. The IOPS calculator > > > above[1] is only > > [SNAP] > > > I have been looking at FIO, but what jobfile do you use that you find > > optimal to test network storage for Xen? > > Actually this is a very hard to answer question! ;-) > Short answer: I use iometer-file-access-server with (1) 80% (2) 100% (3) 0% > read. Then I have a feeling for what I may expect from the hardware... > > Long answer: > Benchmarking is actually about lying -- depending on who is doing those > benchmarks. There is an excellent presentation about benchmarking by Greg > Smith of Postgres fame[1] stating some universal truths about how to > conduct benchmarks. > The most important part is to know what you need: The more details you have > the less you''re lying or the less you''re in danger of being lied to. > In benchmarking terms this means defining an I/O profile. To get that right > you need to monitor your existing infrastructure (eg by collecting the > output of ''iostat'') and conducting the very same benchmarks you''re doing on > your new hardware on the old one as well. > One of my servers reports the following on iostat for a disk containing > some mailboxes: > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sda 45.40 356.65 158.90 2005866781 893707736 > > This means: Since power-on (65 days in this case) the server does 45 IOPS > on average per second. Now it is up to me to know if this server does about > the same amount of work 24/7 or if it is only busy during the day (which > means that you need to calculate two or three times the IOPS for 12 or 8 > hours being really busy. (This is a file server scenario in an average > office. ie: 45*(24/12) or 45*(24/8)) > The next thing I get is the ratio between reads and writes: > 1% .............. (2005866781 + 893707736)/100 > read percent .... 2005866781/2005866781.0/percent = 69.2 % > write percent ... 893707736.0/percent = 30.8 % > > There is one more very important thing in the output of iostat: > (this is actually from a different machine than the above) the average cpu > usage: > avg-cpu: %user %nice %system %iowait %steal %idle > 10.35 2.44 5.69 8.63 0.00 72.89 > > This shows that the system spends more than 8% of its time on waiting for > the completion of outstanding I/Os (which is a bad thing ;-) ). > > With those numbers one is able to get a better understanding of what > servers are doing. Using that information gives the ability to create a > set of FIO jobfiles that roughly describe the workload expected and shows > wether a storage backend (and in parts the rest of the systems) is able to > handle that. > When you''re doing benchmarks from time to time you''ll get a feeling for > what a storage backend can handle when looking at FIO''s results. Using your > own jobfiles with more jobs running in parallel (numjobs=..), using > different block sizes (either blocksize=Xk where x is 4,8,... or mixing > blocksizes as done in iometer-file-access-server) and finding a balance > between read and write transactions you might see more clearly if a storage > system can handla your specific workload. > > Do those benchmarks in your own setup. Do not let someone else do them for > you. In case of network storage, be it iSCSI, AoE, NFS or whatever, refrain > running the benchmarks on the storage system itself: The results will not > reflect the real throughput of the system, numbers will almost always be > higher! > > Uh, quite a lot of text again... ;) Hope this helps! Feedback and > discussion apprechiated... > > -- Adi > > [1] > http://www.pgcon.org/2009/schedule/attachments/123_pg-benchmarking-2.pdf >Adi, most useful and elaborate info, thx a mille!!! B. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users