Hi, We have a problem with the MFI raid-driver under FreeBSD 6.1 and 6.2-PRE. Hardware: Dell 1850 Raid adapter: LSI 8480E There is no problem to detect, initialize and configure, but there is a serious problem with performance with Raid-5. Read performance is good, but write performance bottlenecks at 20MB/s. We've tried different setups but hit the same limitation again and again. The strange part is that a Dell raid adapter based on the same chipset, the Perc 5/I, works well with writes achieving 200MB/s. Any ideas would be greatly appreciated. Kind regards, Fredrik Widlund
Fredrik Widlund wrote:> setups but hit the same limitation again and again. The strange part is > that a Dell raid adapter based on the same chipset, the Perc 5/I, works > well with writes achieving 200MB/s. > > Any ideas would be greatly appreciated.Several: - are there cache differences between the controllers (amount of memory, cache policy)? - how does writing directly to the device (bypassing file system) compare?
Ivan Voras wrote:> Several: > > - are there cache differences between the controllers (amount of > memory, cache policy)?Default settings on both.> - how does writing directly to the device (bypassing file system) > compare?Drives are four seagate 7200.10 400GB in a Raid-5 configuration. [/mnt/test (/dev/mfid0p1 mounted)] read: 200MB/s write: 15MB/s [/dev/mfid0p1] read: 200MB/s write: 8MB/s [/dev/mfid0] read: 200MB/s write: 10MB/s Kind regards, Fredrik Widlund
Fredrik Widlund wrote:> Ivan Voras wrote: >> Several: >> >> - are there cache differences between the controllers (amount of >> memory, cache policy)? > Default settings on both.Maybe you should check what the defaults are :) Especially the amount of memory and is there a battery to back the cache.>> - how does writing directly to the device (bypassing file system) >> compare? > Drives are four seagate 7200.10 400GB in a Raid-5 configuration. > > [/mnt/test (/dev/mfid0p1 mounted)] > read: 200MB/s > write: 15MB/s > > [/dev/mfid0p1] > read: 200MB/s > write: 8MB/s > > [/dev/mfid0] > read: 200MB/s > write: 10MB/sThis is bad :( The difference between p0 and raw device might indicate a stripe size misalignment, but the values are too low in any case.
Fredrik Widlund wrote:> Ivan Voras wrote: > >> Several: >> >> - are there cache differences between the controllers (amount of >> memory, cache policy)? >> > Default settings on both. > >> - how does writing directly to the device (bypassing file system) >> compare? >> > Drives are four seagate 7200.10 400GB in a Raid-5 configuration. > > [/mnt/test (/dev/mfid0p1 mounted)] > read: 200MB/s > write: 15MB/s > > [/dev/mfid0p1] > read: 200MB/s > write: 8MB/s > > [/dev/mfid0] > read: 200MB/s > write: 10MB/s > > Kind regards, > Fredrik Widlund > > >Have you tried setting the write cache policy to write back rather than write thru? Tom
256MB cache, no BBU. Tried a lot of different combinations of settings. fbsd 6.2pre writes 220MB/s with raid-0 If I boot windows server 2003 instead, it writes at around 180MB/s with raid-5 (same configs). With fbsd6.2pre, I get the "best" performance with BadBBU/direct io/disabled cache and 8 drives, maybe around 50 MB/s, however I get a zig-zag performance pattern, with the adapter running at 500% (gstat) utilization one second, then down to 0% for maybe 3 seconds, back, etc. Fredrik Ivan Voras wrote:> Fredrik Widlund wrote: >> Ivan Voras wrote: >>> Several: >>> >>> - are there cache differences between the controllers (amount of >>> memory, cache policy)? >> Default settings on both. > > Maybe you should check what the defaults are :) Especially the amount > of memory and is there a battery to back the cache. > >>> - how does writing directly to the device (bypassing file system) >>> compare? >> Drives are four seagate 7200.10 400GB in a Raid-5 configuration. >> >> [/mnt/test (/dev/mfid0p1 mounted)] >> read: 200MB/s >> write: 15MB/s >> >> [/dev/mfid0p1] >> read: 200MB/s >> write: 8MB/s >> >> [/dev/mfid0] >> read: 200MB/s >> write: 10MB/s > > This is bad :( The difference between p0 and raw device might indicate > a stripe size misalignment, but the values are too low in any case. > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Solved my issue with LSI 8480E without BBU. With "write: BadBBU", "cache: enabled", and "io: cached", performance rose to around 200MB/s from 20MB/s. With BadBBU/enabled cache/io cached: [root@ ~/bench]# ./bench /dev/mfid0p1 65536 file /dev/mfid0p1, bs 65536 rate: 228130816 Bps, write: 285 us rate: 206635008 Bps, write: 299 us rate: 206372864 Bps, write: 305 us rate: 206438400 Bps, write: 307 us ... With Write through: [root@ ~/bench]# ./bench /dev/mfid0p1 65536 file /dev/mfid0p1, bs 65536 rate: 40763392 Bps, write: 1609 us rate: 19202048 Bps, write: 2260 us rate: 25231360 Bps, write: 2301 us rate: 19660800 Bps, write: 2418 us ... 10x performance win, so this is one thing off my back. Kind regards, Fredrik Widlund Fredrik Widlund wrote:> 256MB cache, no BBU. Tried a lot of different combinations of settings. > > fbsd 6.2pre writes 220MB/s with raid-0 > > If I boot windows server 2003 instead, it writes at around 180MB/s with > raid-5 (same configs). > > With fbsd6.2pre, I get the "best" performance with BadBBU/direct > io/disabled cache and 8 drives, maybe around 50 MB/s, however I get a > zig-zag performance pattern, with the adapter running at 500% (gstat) > utilization one second, then down to 0% for maybe 3 seconds, back, etc. > > Fredrik > > Ivan Voras wrote: > >> Fredrik Widlund wrote: >> >>> Ivan Voras wrote: >>> >>>> Several: >>>> >>>> - are there cache differences between the controllers (amount of >>>> memory, cache policy)? >>>> >>> Default settings on both. >>> >> Maybe you should check what the defaults are :) Especially the amount >> of memory and is there a battery to back the cache. >> >> >>>> - how does writing directly to the device (bypassing file system) >>>> compare? >>>> >>> Drives are four seagate 7200.10 400GB in a Raid-5 configuration. >>> >>> [/mnt/test (/dev/mfid0p1 mounted)] >>> read: 200MB/s >>> write: 15MB/s >>> >>> [/dev/mfid0p1] >>> read: 200MB/s >>> write: 8MB/s >>> >>> [/dev/mfid0] >>> read: 200MB/s >>> write: 10MB/s >>> >> This is bad :( The difference between p0 and raw device might indicate >> a stripe size misalignment, but the values are too low in any case. >> >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >