freebsd stable - Apr 2018 - TRIM, iSCSI and %busy waves

That is very hw and use case dependent.

The reason we originally sponsored the project to add TRIM to ZFS was that
in our case without TRIM the performance got so bad that we had to secure
erase disks every couple of weeks as they simply became so slow they where
unusable.

Now admittedly that was a fair few years ago and hw has moved on since
then, but the point remains that it?s not totally true that just not
TRIMing is an option.

On Fri, 6 Apr 2018 at 09:10, Borja Marcos <borjam at sarenet.es> wrote:
>
> > On 5 Apr 2018, at 17:00, Warner Losh <imp at bsdimp.com> wrote:
> >
> > I'm working on trim shaping in -current right now. It's
focused on NVMe,
> > but since I'm doing the bulk of it in cam_iosched.c, it will
eventually
> be
> > available for ada and da. The notion is to measure how long the TRIMs
> take,
> > and only send them at 80% of that rate when there's other traffic
in the
> > queue (so if trims are taking 100ms, send them no faster than 8/s).
While
> > this will allow for better read/write traffic, it does slow the TRIMs
> down
> > which slows down whatever they may be blocking in the upper layers.
Can't
> > speak to ZFS much, but for UFS that's freeing of blocks so things
like
> new
> > block allocation may be delayed if we're almost out of disk (which
we
> have
> > no signal for, so there's no way for the lower layers to
prioritize trims
> > or not).
>
> Have you considered "hard" shaping including discarding TRIMs
when needed?
> Remember that a TRIM is not a write, which is subject to a contract with
> the application,
> but a better-if-you-do-it operation.
>
> Otherwise, as you say, you might be blocking other operations in the upper
> layers.
> I am assuming here that with many devices doing TRIMs is better than not
> doing them.
> And in case of queue congestion doing *some* TRIMs should be better than
> doing
> no TRIMs at all.
>
> Yep, not the first time I propose something of the sort, but my queue of
> suggestions
> to eventually discard TRIMs doesn?s implement the same method ;)
>
>
> Borja.
>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
>

> On 6 Apr 2018, at 10:41, Steven Hartland <killing at multiplay.co.uk>
wrote:
> 
> That is very hw and use case dependent.
> 
> The reason we originally sponsored the project to add TRIM to ZFS was that
in our case without TRIM the performance got so bad that we had to secure erase
disks every couple of weeks as they simply became so slow they where unusable.
> 
> Now admittedly that was a fair few years ago and hw has moved on since
then, but the point remains that it?s not totally true that just not TRIMing is
an option.
As far as I know, letting the device know that you have released a block should
be a Good Thing.
So I prefer to TRIM. I have also seen cases where not doing TRIMs resulted in a
terrible performance
although modern SSDs should have better algorithms to deal with it. 

But I have also seen bad cases of TRIM requests piling up and clogging the
queues for long periods
of time. Admittedly it was a purely synthetic situation (running bonnie++) but,
again, performing
operations like destroying a huge snapshot or dataset could trigger a similar
condition.  It was
especially nasty when I tried NVMEs. I mentioned this here:

https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023134.html


That?s why I think there is a mid point between the radical approach of not
doing TRIMs at all
and clogging the queues with too many of them. 

If the TRIM queue gets very large and the ?mandatory? operations (read, writes,
etc) begin to
get large delays I think it?s safe to assume that *there is* a problem and
discarding at least part
of those TRIMs could help to solve it. 

A ?TRIM when you can? should still be better than a ?Don?t TRIM at all?. 

Cheers,





Borja.

P.S: Attaching the graphs that were lost.