freebsd stable - Mar 2008 - How to debug or explain slow gjournal writes?

I am evaluating using gjournal on my servers.  This one test system is 
running 7.0-RELEASE at the moment on a Dell PE2650 with dual 2ghz xeon, 
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter>, and some seagate 36g 10k 
disks. I had the opportunity to try placing the journal consumer device 
on a dedicated disk.  Whether or not I use a separate journal consumer 
device from the data consumer, I can expect the overall write speed to 
be slower because it is effectively writing everything twice.  However 
when I watch it write just to the journal consumer device, the MB/sec it 
writes to the journal consumer device never seems to exceed around 50% 
of the speed that the disk is capable of.  Its almost as if it is 
competing with IO being written to the provider, but that might be a 
coincidence.  When I see it flush the journal consumer to the data 
consumer, it seems to read from a memory cache and write speed to the 
data consumer is full speed. 

Comparing with a more modern system with a mpt0: <LSILogic SAS/SATA 
Adapter> and two FUJITSU MAY2073RCSUN72G 0401 in a gmirror, I see 
journal write speed approx 40M/sec but data write speed of approx 
45M/sec, which is a lot closer together, but still shows a difference.  
I'm fairly sure a single drive on its own could transfer faster than 
that, but I haven't tested it recently.  I could if it would be helpful.

Example with data,journal together just so I can put the terminology 
together with the device listing below:
 
Writing to da2.journal shows input at approx 35M/sec and writes to the 
da2 journal at approx 35M/sec for a few seconds.  Then the journal 
switches, input speed to da2.journal drops to 0, and apparently an 
in-memory journal gets copied to the data consumer on da2 at 65M/sec.  
Then it goes back to 35M/sec in and 35M/sec to the journal, repeating as 
expected.  It seems to avoid writing to the data and journal at the same 
time, which is probably intentional to avoid head thrashing, especially 
if the journal fits in memory.

Due to the slow journal write speed, and the expected double-writing of 
data, I only see a resulting write speed to da2.journal of approximately 
16-22MB/sec.  There is no improvement in the journal consumer write 
speed if I put it on a separate disk.  The overall write speed is also 
the same.  Additionally, when I have the journal on a gmirror (the data 
is too), the journal write speed is variable between 8 and 35MB/sec, it 
fluxuates pretty wildly second to second. As soon as I deactivate one 
side of the mirror, or get rid of the mirror under the journal, it is a 
consistent 35MB/sec.

I'm looking for input on debugging, tuning, questions, bonehead errors, 
etc because I would like to get the most out of this setup if possible 
and not just settle for an inconsistent 16-22MB/sec.  Thanks.

Geom name: gjournal 170802896
ID: 170802896
Providers:
1. Name: da2.journal
   Mediasize: 35346332672 (33G)
   Sectorsize: 512
   Mode: r1w1e1
Consumers:
1. Name: da2
   Mediasize: 36420075008 (34G)
   Sectorsize: 512
   Mode: r1w1e1
   Jend: 36420074496
   Jstart: 35346332672
   Role: Data,Journal

On Thu, Mar 13, 2008 at 05:57:41PM -0400, Adam McDougall wrote:

  I am evaluating using gjournal on my servers.  This one test system is 
  running 7.0-RELEASE at the moment on a Dell PE2650 with dual 2ghz xeon, 
  ahc0: <Adaptec aic7899 Ultra160 SCSI adapter>, and some seagate 36g 10k 
<snip>
  Additionally, when I have the journal on a gmirror (the data is too), the 
  journal write speed is variable between 8 and 35MB/sec, it fluxuates pretty 
  wildly second to second. As soon as I deactivate one side of the mirror, or 
  get rid of the mirror under the journal, it is a consistent 35MB/sec.

You know, the more I probe, the more it seems to unravel to an issue I have seen
on at least 3 FreeBSD 7.x systems so far:  SOMETIMES writes have inconsistent
speed if they have been used in some sort of raid.  Key words are
"sometimes"
and "have been".  It seems like a disk is more likely to misbehave or
just be
consistently slow if it used to be in a raid like a gmirror.  Sometimes I could
deactivate a disk from a mirror and use it independantly (like for a journal) 
but it would stay slow.  Other times, with a more fresh setup, it would perform 
fine.

- On my brand new desktop, I setup a zfs mirror, my writes went into the toilet.
  I could hear the writes getting interrupted every few seconds while the disk 
  heads seeked.  I deactivated the mirror, and the writes would go to the first
  disk with full speed again.  And when I formatted the second disk with
whatever
  FS I wanted, it would give me full speed.  But mirror them, and performance
would
  drop back down by several factors.  Somehow I avoided this by switching my
desktop
  install to amd64 freebsd and re-setting up the mirror with zfs, but I think
  that was chance.

- On a friend's home server, sometimes when I would setup a multidisk raid
(using
  different zfs methods, or geom raids), I would run gstat and watch the
performance
  bounce all around between the disks in the raid.  It would continue to bounce 
  between 8M/sec and something more reasonable for his ata disks like 40M/sec.  
  If I still had the opportunity to experiment with that again, I would try to
  reproduce it and look at it more closely with gstat -I 100000 or so.  I bet it
  was completely stalling out when I saw it be slow on a one-second average.
  I found a combination of sata controllers that gave fair and consistent
  (but not impressive) performance and called it good because it exceeded what 
  his 1Gbit network could likely push or pull to it for a file server.

- On this Dell 2650, I keep setting up different zfs mirrors, raidz, gmirror,
gstripe,
  and sometimes the write speed is fine, sometimes it keeps bouncing around
  or stalling out for brief periods.  Not always every drive at the same 
  time.  Its too loud in the server room to hear if this one is seeking while
its
  slow.  I can never see anything obvious in top -S, gstat, systat -vmstat that 
  might cause a spike.  I know the scsi bus on this one is capped at 160 so 
  I don't expect to see 3-4 drives run at the same time and get full
performance.

But ALWAYS if you use dd to write directly to a drive, even multiple drives, or 
put a filesystem on a single drive, write speed was fine.


  
  I'm looking for input on debugging, tuning, questions, bonehead errors,
etc
  because I would like to get the most out of this setup if possible and not 
  just settle for an inconsistent 16-22MB/sec.  Thanks.
  
  Geom name: gjournal 170802896
  ID: 170802896
  Providers:
  1. Name: da2.journal
    Mediasize: 35346332672 (33G)
    Sectorsize: 512
    Mode: r1w1e1
  Consumers:
  1. Name: da2
    Mediasize: 36420075008 (34G)
    Sectorsize: 512
    Mode: r1w1e1
    Jend: 36420074496
    Jstart: 35346332672
    Role: Data,Journal
  
  _______________________________________________
  freebsd-stable@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-stable
  To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"