freebsd stable - Apr 2019 - Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o)

Hi all,
> Am 04.04.2019 um 17:11 schrieb Warner Losh <imp at bsdimp.com>:
> There's a request that was sent down to the drive. It took longer than
30s to respond. One of them, at least, was a trim request.
> [?]
Thanks for the explanation.

This further explains why I was seeing a lot more of those and the system
occasionally froze for a couple of seconds after I increased these:

vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_read_max_active: 10

as recommended by Allan Jude reasoning that NVME devices could work on
up to 64 requests in parallel. I have since reverted that change and I am
running with the defaults.

If I understand correctly, this:
>         hw.nvme.per_cpu_io_queues=0
essentially limits the rate at which the system throws commands at the devices.
Correct?

So it?s not a real fix and there?s nothing fundamentally wrong with the per CPU
queue or
interrupt implementation. I will look into new firmware for my Intel devices and
try tweaking the vfs.zfs.vdev.trim_max_active and related parameters.

Out of curiosity: what happens if I disable TRIM? My knowledge is rather
superficial
and I just filed that under ?TRIM is absolutely essential lest performance will
suffer severely and your devices die - plus bad karma, of course ?? ;-)

Kind regards,
Patrick
-- 
punkt.de GmbH			Internet - Dienstleistungen - Beratung
Kaiserallee 13a			Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe			info at punkt.de	http://punkt.de
AG Mannheim 108285		Gf: Juergen Egeling

Hi all,

in addition to the aborted commands every dozen of system boots or so
(this order of magnitude) the kernel simply hangs during initialisation of
one of the NVME devices:

https://cloud.hausen.com/s/TxPTDFJwMe6sJr2

The particular device affected is not constant.

A power cycle fixes it, the system has not shown hangs/freezes during
multiuser operation, yet.


Any ideas?
Patrick
-- 
punkt.de GmbH			Internet - Dienstleistungen - Beratung
Kaiserallee 13a			Tel.: 0721 9109-0 Fax: -100
76133 Karlsruhe			info at punkt.de	http://punkt.de
AG Mannheim 108285		Gf: Juergen Egeling

On Fri, Apr 5, 2019 at 1:33 AM Patrick M. Hausen <hausen at punkt.de>
wrote:
> Hi all,
>
> > Am 04.04.2019 um 17:11 schrieb Warner Losh <imp at bsdimp.com>:
> > There's a request that was sent down to the drive. It took longer
than
> 30s to respond. One of them, at least, was a trim request.
> > [?]
>
> Thanks for the explanation.
>
> This further explains why I was seeing a lot more of those and the system
> occasionally froze for a couple of seconds after I increased these:
>
> vfs.zfs.vdev.async_write_max_active: 10
> vfs.zfs.vdev.async_read_max_active: 3
> vfs.zfs.vdev.sync_write_max_active: 10
> vfs.zfs.vdev.sync_read_max_active: 10
>
> as recommended by Allan Jude reasoning that NVME devices could work on
> up to 64 requests in parallel. I have since reverted that change and I am
> running with the defaults.
>
> If I understand correctly, this:
>
> >         hw.nvme.per_cpu_io_queues=0
>
> essentially limits the rate at which the system throws commands at the
> devices. Correct?
>
Yes. It de-facto limits the number of commands the system can throw at a
nvme drive. Some drives have trouble with multiple CPUs submitting things.
Others just have trouble with the volume of commands sometimes. This limits
both.

> So it?s not a real fix and there?s nothing fundamentally wrong with the
> per CPU queue or
> interrupt implementation. I will look into new firmware for my Intel
> devices and
> try tweaking the vfs.zfs.vdev.trim_max_active and related parameters.
>
Correct. It's a hack-a-around.

> Out of curiosity: what happens if I disable TRIM? My knowledge is rather
> superficial
> and I just filed that under ?TRIM is absolutely essential lest performance
> will
> suffer severely and your devices die - plus bad karma, of course ?? ;-)
>
TRIMs help the drive optimize their garbage collection by giving it a
larger pool of free blocks to work with. This has the effect of reducing
write amplification. Write amp is the measure of the amount of extra work
the drive  has to do for every user write it processes. Ideally, you want
this number to be 1.0. You'll never get to 1.0, but numbers less than 1.5
are common and most of the models drive makers use to rate the lifetime of
their NAND assume a write amp of about 2.

So, if you eliminate the TRIMs you eliminate this optimization and write
amp will increase. This has two bad effects. First, wear and tear on the
NAND. Second, it takes resources away from the user. In practice, however,
the bad effects are quite limited if you don't have a write intensive
workload. Your drive is rated for so many drive writes per day (or
equivalently total data written over the life of the drive). This will be
on the spec sheet somewhere. If you don't have a write intensive workload
(which I'd say is any sustained write load greater than about 1/10th the
datasheet write limit), then if you think TRIMs are causing issues, you
should disable them. The effects of not trimming are likely to be in the
noise on such systems, and the benefits of having things TRIMed will be
less.

At work, for a large video streaming company, we enable the TRIMs, even
though we're on the edge of the rule of thumb since we're unsure how
long
the machines really need to be in the field and don't want to risk it.
Except for the version of Samsung nvme drives (PM963, no longer made) we
got a while ago... those we turn TRIM off on because UFS' machine-gunning
down of TRIMs and nvd's blind pass-through of TRIMs took down the drive.
UFS now combines TRIMs and we've moved to using nda since it also combines
TRIMs and it won't be so bad if we tried again today.

Drive makers optimize different things. Enterprise drives handle TRIMs a
lot better than consumer drives. consumer drives are cheaper (in oh so many
ways), so some care is needed. Intel makes a wide range of drives, from the
super duper awesome (with prices to match) to the somewhat disappointing
(but incredibly cheap and good enough for a lot of applicaitons). Not sure
where on this scale your drives fall on this spectrum.

tl;dr: Unless you are writing the snot out of those Intel drives, disabling
TRIM entirely will likely help avoid pushing so many commands they timeout.

Warner