Eugene M. Zheganin
2016-Nov-28 17:54 UTC
Sandisk CloudSpeed Gen. II Eco Channel SSD vs ZFS = we're in hell
Hi,
recently we bough a bunch of "Sandisk CloudSpeed Gen. II Eco Channel"
disks (the model name by itself should already made me suspicious) for
using with zfs SAN on FreeBSD, we're plugged them into the LSI SAS3008
and now we are experiencing the performance that I would call "literally
awful". I'm using already some of the zfs SANs on FreeBSD with
Intel/Samsung SSD drives, including the LSI SAS3008 controller, but
never saw anything like this (and yes, these are all SSDs):
dT: 1.004s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
75 472 78 367 104.4 12 1530 94.8 113.4| da0
75 475 81 482 79.2 12 1530 94.5 113.1| da1
69 490 96 626 106.9 12 1530 124.9 149.4| da2
75 400 72 382 51.5 10 1275 93.7 93.4| da3
0 0 0 0 0.0 0 0 0.0 0.0| da4
75 400 72 382 55.0 10 1275 93.9 93.7| da5
2 3975 3975 24020 0.3 0 0 0.0 21.0| da6
0 3967 3967 24144 0.3 0 0 0.0 21.4| da7
1 3929 3929 24259 0.3 0 0 0.0 21.6| da8
0 3998 3998 23933 0.3 0 0 0.0 21.2| da9
0 0 0 0 0.0 0 0 0.0 0.0| da10
0 4037 4037 23710 0.2 0 0 0.0 21.3| da11
0 0 0 0 0.0 0 0 0.0 0.0| da12
0 0 0 0 0.0 0 0 0.0 0.0| da13
0 0 0 0 0.0 0 0 0.0 0.0| da14
0 0 0 0 0.0 0 0 0.0 0.0| da15
0 0 0 0 0.0 0 0 0.0 0.0| da16
Disks are ogranized in the raidz1 pools (which is slower than the raid1
or 10, but, considering the performance of SSDs, we got no problems with
Intel or Samsung drives), the controller is flashed with last firmware
available (identical controller with Samsung drives performs just fine).
Disks are 512e/4K drives, and "diskinfo -v"/"camcontrol
identify" both
report that they have 4K stripersize/physical sector. Pools are
organized using dedicated disks, so, considering all of the above, I
don't see any possiblity to explain this with the alignment errors. No
errors are seen in the dmesg. So, right at this time, I'm out of ideas.
Everything point that these Sandisk drives are the roort of the problem,
but I don't see how this is possible- according to the various
benchmarks (taken, however, with regular drives, not "Channel" ones,
and
so far I haven't figured out what is the difference between
"Channel"
and non-"Channel" ones, but they run different firmware branches) they
have to be okay (or seem so), just the ordinary SSD.
If someone has the explanation of this awful performance, please let me
know.
Thanks.
Eugene.
Steven Hartland
2016-Nov-28 18:07 UTC
Sandisk CloudSpeed Gen. II Eco Channel SSD vs ZFS = we're in hell
Check your gstat with -dp so you also see deletes, it may be that your drives have a very slow TRIM. On 28/11/2016 17:54, Eugene M. Zheganin wrote:> Hi, > > recently we bough a bunch of "Sandisk CloudSpeed Gen. II Eco Channel" > disks (the model name by itself should already made me suspicious) for > using with zfs SAN on FreeBSD, we're plugged them into the LSI SAS3008 > and now we are experiencing the performance that I would call > "literally awful". I'm using already some of the zfs SANs on FreeBSD > with Intel/Samsung SSD drives, including the LSI SAS3008 controller, > but never saw anything like this (and yes, these are all SSDs): > > dT: 1.004s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 75 472 78 367 104.4 12 1530 94.8 113.4| da0 > 75 475 81 482 79.2 12 1530 94.5 113.1| da1 > 69 490 96 626 106.9 12 1530 124.9 149.4| da2 > 75 400 72 382 51.5 10 1275 93.7 93.4| da3 > 0 0 0 0 0.0 0 0 0.0 0.0| da4 > 75 400 72 382 55.0 10 1275 93.9 93.7| da5 > 2 3975 3975 24020 0.3 0 0 0.0 21.0| da6 > 0 3967 3967 24144 0.3 0 0 0.0 21.4| da7 > 1 3929 3929 24259 0.3 0 0 0.0 21.6| da8 > 0 3998 3998 23933 0.3 0 0 0.0 21.2| da9 > 0 0 0 0 0.0 0 0 0.0 0.0| da10 > 0 4037 4037 23710 0.2 0 0 0.0 21.3| da11 > 0 0 0 0 0.0 0 0 0.0 0.0| da12 > 0 0 0 0 0.0 0 0 0.0 0.0| da13 > 0 0 0 0 0.0 0 0 0.0 0.0| da14 > 0 0 0 0 0.0 0 0 0.0 0.0| da15 > 0 0 0 0 0.0 0 0 0.0 0.0| da16 > > Disks are ogranized in the raidz1 pools (which is slower than the > raid1 or 10, but, considering the performance of SSDs, we got no > problems with Intel or Samsung drives), the controller is flashed with > last firmware available (identical controller with Samsung drives > performs just fine). Disks are 512e/4K drives, and "diskinfo > -v"/"camcontrol identify" both report that they have 4K > stripersize/physical sector. Pools are organized using dedicated > disks, so, considering all of the above, I don't see any possiblity to > explain this with the alignment errors. No errors are seen in the > dmesg. So, right at this time, I'm out of ideas. Everything point that > these Sandisk drives are the roort of the problem, but I don't see how > this is possible- according to the various benchmarks (taken, however, > with regular drives, not "Channel" ones, and so far I haven't figured > out what is the difference between "Channel" and non-"Channel" ones, > but they run different firmware branches) they have to be okay (or > seem so), just the ordinary SSD. > > If someone has the explanation of this awful performance, please let > me know. > > Thanks. > > Eugene. > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"