freebsd stable - Jan 2010 - immense delayed write to file system (ZFS and UFS2), performance issues

I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. 
All boxes have the most recent STABLE. One box is a UP system, two 
others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores 
(Dell Poweredge III).

Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or 
so, sometimes the I/O performance drops massively when doing 'svn 
update', 'make world' or even 'make kernel'. It doesn't
matter what
memory and how many cpu the box has, it get stuck for several seconds 
and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
A very interesting phenomenon is the massively delayed file writing on 
ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm 
and having in another Xterminal my shell for compiling this file, it 
takes sometimes up to 20 seconds to get the file updated after it has 
been written. It's like having an old, slow NFS connection with long 
cache delays.
These massively delayed file transactions are not necessarely under 
heavy load, sometimes they occur in a relaxed situation. They seem to 
occur much more often on the UP box than on the SMP boxes, but this 
strange phenomenon also occur on the Dell Poweredge II, which has 16GB 
RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and 
UFS2 filesystems as well. It is hardly reproducable.

Is there any known issue?

Ragrds,
Oliver

On Mon, Jan 18, 2010 at 09:13:52PM +0100, O. Hartmann
wrote:
> I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64
> boxes. All boxes have the most recent STABLE. One box is a UP
> system, two others SMP boxes, one with a Q6600 4-core, another XEON
> with 2x 4-cores (Dell Poweredge III).
> 
> Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks
> or so, sometimes the I/O performance drops massively when doing 'svn
> update', 'make world' or even 'make kernel'. It
doesn't matter what
> memory and how many cpu the box has, it get stuck for several
> seconds and freezing. On the UP box, this is sometimes for 10 - 20
> seconds.
> A very interesting phenomenon is the massively delayed file writing
> on ZFS filesystems I realise. Editing a file in 'vi' running on one
> XTerm and having in another Xterminal my shell for compiling this
> file, it takes sometimes up to 20 seconds to get the file updated
> after it has been written. It's like having an old, slow NFS
> connection with long cache delays.
> These massively delayed file transactions are not necessarely under
> heavy load, sometimes they occur in a relaxed situation. They seem
> to occur much more often on the UP box than on the SMP boxes, but
> this strange phenomenon also occur on the Dell Poweredge II, which
> has 16GB RAM and summa summarum 16 cores. This phenomenon does occur
> on ZFS- and UFS2 filesystems as well. It is hardly reproducable.
Possibly this is an extreme example of what Garrett Moore et al
have been discussing recently?

http://lists.freebsd.org/pipermail/freebsd-stable/2010-January/053845.html

You might try the force-swap-out approach here to find out if what
you're seeing is identical:

http://lists.freebsd.org/pipermail/freebsd-stable/2010-January/053949.html

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

O. Hartmann wrote:
> I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
> All boxes have the most recent STABLE. One box is a UP system, two
> others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
> (Dell Poweredge III).
> 
> Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
> so, sometimes the I/O performance drops massively when doing 'svn
> update', 'make world' or even 'make kernel'. It
doesn't matter what
> memory and how many cpu the box has, it get stuck for several seconds
> and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
> A very interesting phenomenon is the massively delayed file writing on
> ZFS filesystems I realise. Editing a file in 'vi' running on one
XTerm
> and having in another Xterminal my shell for compiling this file, it
> takes sometimes up to 20 seconds to get the file updated after it has
> been written. It's like having an old, slow NFS connection with long
> cache delays.
> These massively delayed file transactions are not necessarely under
> heavy load, sometimes they occur in a relaxed situation. They seem to
> occur much more often on the UP box than on the SMP boxes, but this
> strange phenomenon also occur on the Dell Poweredge II, which has 16GB
> RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
> UFS2 filesystems as well. It is hardly reproducable.
> 
> Is there any known issue?
> 
> Ragrds,
> Oliver

The disks involved don't happen to be Western Digital Green Power disks,
do they? The Intelli-Park function in these disks are wrecking havoc
with I/O in Linux-land at least, causing massive stalls and iowait
through the roof during the 25-30 seconds it takes for the heads to
unload after parking. I have two of these disks sitting on my desk now
collecting dust...
/Morgan

On 01/18/10 22:13, O. Hartmann wrote:
> Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
> so, sometimes the I/O performance drops massively when doing 'svn
> update', 'make world' or even 'make kernel'. It
doesn't matter what
> memory and how many cpu the box has, it get stuck for several seconds
> and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
> A very interesting phenomenon is the massively delayed file writing on
> ZFS filesystems I realise. Editing a file in 'vi' running on one
XTerm
> and having in another Xterminal my shell for compiling this file, it
> takes sometimes up to 20 seconds to get the file updated after it has
> been written. It's like having an old, slow NFS connection with long
> cache delays.
> These massively delayed file transactions are not necessarely under
> heavy load, sometimes they occur in a relaxed situation. They seem to
> occur much more often on the UP box than on the SMP boxes, but this
> strange phenomenon also occur on the Dell Poweredge II, which has 16GB
> RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
> UFS2 filesystems as well. It is hardly reproducable.
I'm experiencing the same thing, except in my case it's most noticeable 
when writing to a USB flash drive with a FAT32 filesystem.  It slows the 
entire system down, even if the data being written is coming from cache 
or a memory file system.

I don't know if it's related.  I'm running 8-STABLE from about 4
December.


Regards,
Aragon

> You're welcome. I just feel as bad for you as for everyone else who
> has bought these obviously Windoze optimized harddrives. Unfortunately
> neither wdidle3 nor an updated firmware is available or functioning on
> the latest models in the Green series. At least that's what I've
read
> from other people having this issue. WD only claims they don't support
> Linux and they probably have never heard of FreeBSD.
This discussion made me have a look at my 2tb WD Green disks, one of
them is from May 2009, looks pretty reasonable :

Device Model:     WDC WD20EADS-00R6B0
Serial Number:    WD-WCAVY0301430
Firmware Version: 01.00A01

  9 Power_On_Hours          0x0032   093   093   000    Old_age
Always       -       5253
193 Load_Cycle_Count        0x0032   200   200   000    Old_age
Always       -       55

And another is very recent, from December 2009, this does look a bit
worrying in comparison:

Device Model:     WDC WD20EADS-32R6B0
Serial Number:    WD-WCAVY1611513
Firmware Version: 01.00A01

  9 Power_On_Hours          0x0032   100   100   000    Old_age
Always       -       136
193 Load_Cycle_Count        0x0032   199   199   000    Old_age
Always       -       5908

The disks are of exact same model and look to be same firmware. Should
I be worried that the newer disk has, in 136 hours reached a higher
Load Cycle count twice as big as on the disk thats 5253 hours old?


- Sincerely,
Dan Naumov

Can anyone confirm that using the WDIDLE3 utility on the 2TB WD20EADS
discs will not cause any issues if these disks are part of a ZFS
mirror pool? I do have backups of data, but I would rather not spend
the time rebuilding the entire system and restoring enormous amounts
of data over a 100mbit network unless I absolutely have to :)

- Sincerely,
Dan Naumov

Thank you, thank you, thank you!

Now I neither have to worry about premature death of my disks, nor do
I have to endure the loud clicking noises (I have a NAS with these in
my living room)!

If either of you (or both) want me to Paypal you money for a beer,
send me details offlist :)


- Sincerely,
Dan Naumov

>I have a WD2003FYPS sitting in a system, to be used for testing.  Bought it
>just before this thread started, and here's what it looks like right
now:
>
> 9 Power_On_Hours          0x0032   100   100   000    Old_age   Always -   
508
>193 Load_Cycle_Count      0x0032   200   200   000    Old_age   Always -    
2710
>
>This drive is sitting, unused, with no filesystem, and I've performed
>approximately zero writes to the disk.
>
>Having a script kick off and write to a disk will help so long as that
>disk is writable; if it's being used as a hot spare in a raidz array,
it's
>not going to help much.
I wouldn't worry in your particular case. A value of 2710 in 508 hours
is a rate of 5,33/hour. At this rate, it's going to take you 56285
hours or 2345 days to reach 300,000 and most disks will likely
function past 400,000 (over 600,000 all bets are off though). The
people who need(ed) to worry were people like me, who were seeing the
rate increase at a rate of 43+ per hour.

- Sincerely,
Dan Naumov

On 2010-01-27 00:15, Dan Naumov wrote:
> Can anyone confirm that using the WDIDLE3 utility on the 2TB WD20EADS
> discs will not cause any issues if these disks are part of a ZFS
> mirror pool? I do have backups of data, but I would rather not spend
> the time rebuilding the entire system and restoring enormous amounts
> of data over a 100mbit network unless I absolutely have to :)
Sorry to bump into this thread so late, but for some of my servers I
have been using a patch for atacontrol, to turn the APM features of the
disk(s) off, for a long time.  This is mostly noticable with 2.5"
notebook disks, which "click" like crazy all the time. :)

E.g. if you run "atacontrol cap $device", and you see in the output
that
"advanced power management" is supported, you might be able to stop
the
disk from spinning down by turning the APM feature off.

Patch is at <http://www.andric.com/freebsd/atacontrol-apm.diff>.  This
should apply to 8-STABLE, no idea about older branches.