freebsd stable - Nov 2016 - Sandisk CloudSpeed Gen. II Eco Channel SSD vs ZFS = we're in hell

Check your gstat with -dp so you also see deletes, it may be that your 
drives have a very slow TRIM.

On 28/11/2016 17:54, Eugene M. Zheganin wrote:
> Hi,
>
> recently we bough a bunch of "Sandisk CloudSpeed Gen. II Eco
Channel"
> disks (the model name by itself should already made me suspicious) for 
> using with zfs SAN on FreeBSD, we're plugged them into the LSI SAS3008 
> and now we are experiencing the performance that I would call 
> "literally awful". I'm using already some of the zfs SANs on
FreeBSD
> with Intel/Samsung SSD drives, including the LSI SAS3008 controller, 
> but never saw anything like this (and yes, these are all SSDs):
>
> dT: 1.004s  w: 1.000s
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    75    472     78    367  104.4     12   1530   94.8  113.4| da0
>    75    475     81    482   79.2     12   1530   94.5  113.1| da1
>    69    490     96    626  106.9     12   1530  124.9  149.4| da2
>    75    400     72    382   51.5     10   1275   93.7   93.4| da3
>     0      0      0      0    0.0      0      0    0.0    0.0| da4
>    75    400     72    382   55.0     10   1275   93.9   93.7| da5
>     2   3975   3975  24020    0.3      0      0    0.0   21.0| da6
>     0   3967   3967  24144    0.3      0      0    0.0   21.4| da7
>     1   3929   3929  24259    0.3      0      0    0.0   21.6| da8
>     0   3998   3998  23933    0.3      0      0    0.0   21.2| da9
>     0      0      0      0    0.0      0      0    0.0    0.0| da10
>     0   4037   4037  23710    0.2      0      0    0.0   21.3| da11
>     0      0      0      0    0.0      0      0    0.0    0.0| da12
>     0      0      0      0    0.0      0      0    0.0    0.0| da13
>     0      0      0      0    0.0      0      0    0.0    0.0| da14
>     0      0      0      0    0.0      0      0    0.0    0.0| da15
>     0      0      0      0    0.0      0      0    0.0    0.0| da16
>
> Disks are ogranized in the raidz1 pools (which is slower than the 
> raid1 or 10, but, considering the performance of SSDs, we got no 
> problems with Intel or Samsung drives), the controller is flashed with 
> last firmware available (identical controller with Samsung drives 
> performs just fine). Disks are 512e/4K drives, and "diskinfo 
> -v"/"camcontrol identify" both report that they have 4K 
> stripersize/physical sector. Pools are organized using dedicated 
> disks, so, considering all of the above, I don't see any possiblity to 
> explain this with the alignment errors. No errors are seen in the 
> dmesg. So, right at this time, I'm out of ideas. Everything point that 
> these Sandisk drives are the roort of the problem, but I don't see how 
> this is possible- according to the various benchmarks (taken, however, 
> with regular drives, not "Channel" ones, and so far I haven't
figured
> out what is the difference between "Channel" and
non-"Channel" ones,
> but they run different firmware branches) they have to be okay (or 
> seem so), just the ordinary SSD.
>
> If someone has the explanation of this awful performance, please let 
> me know.
>
> Thanks.
>
> Eugene.
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

Hi.

On 28.11.2016 23:07, Steven Hartland wrote:
> Check your gstat with -dp so you also see deletes, it may be that your
> drives have a very slow TRIM.
>
Indeed, I see a bunch of delete operations, and when TRIM disabled my
engineers report that the performance is greatly increasing. Is this it ?

Eugene.