freebsd stable - Oct 2006 - gmirror warnings

Over the weekend and today we got some curious error messages from  
gmirror providers:

Oct  7 03:02:14 dtfe2 kernel: ad6: FAILURE - out of memory in start
Oct  7 04:15:58 dtfe2 kernel: ad4: FAILURE - out of memory in start

Oct 10 03:02:16 dtfe2 kernel: ad4: FAILURE - out of memory in start
Oct 10 03:02:45 dtfe2 kernel: ad6: FAILURE - out of memory in start
Oct 10 03:03:14 dtfe2 kernel: ad6: FAILURE - out of memory in start

Nothing of note on either side of these lines.

Both gmirror status and gmirror list show no issues with the mirror,  
but is there some way to do a full verify?

Should I be worried about these warnings or are they just transient  
failures that correct themselves?

System is FreeBSD 6.1-RELEASE amd64.

Hello!

On Tue, 10 Oct 2006, Vivek Khera wrote:
> Over the weekend and today we got some curious error messages from gmirror 
> providers:
>
> Oct  7 03:02:14 dtfe2 kernel: ad6: FAILURE - out of memory in start
> Oct  7 04:15:58 dtfe2 kernel: ad4: FAILURE - out of memory in start
>
> Should I be worried about these warnings or are they just transient
failures
> that correct themselves?
   Those messages are not directly related to gmirror, they come from the ATA
driver when it can't allocate memory for the structure which describes
I/O request:

ata-disk.c:

static void
ad_strategy(struct bio *bp)
{
     device_t dev =  bp->bio_disk->d_drv1;
     struct ata_device *atadev = device_get_softc(dev);
     struct ata_request *request;

     if (!(request = ata_alloc_request())) {
         device_printf(dev, "FAILURE - out of memory in start\n");
         biofinish(bp, NULL, ENOMEM);
         return;
     }

ata-all.h:

#define ata_alloc_request() uma_zalloc(ata_request_zone, M_NOWAIT | M_ZERO)

Yes, message is somewhat confusing. Those situations will cause original
I/O request to fail with errno = ENOMEM. The cure for the problem is proper
tuning of the KVA-related values - too bad tuning(7) doesn't even mention
KVA or kmem. Off the memory, lowering this:

dmitry@homelynx$ sysctl vm.kmem_size_scale
vm.kmem_size_scale: 3

from the /boot/loader.conf, e.g.:

vm.kmem_size_scale=2

will give you more kernel virtual memory to burn. I'm sure that there are
more options, but they aren't documented ;(

Sincerely, Dmitry
-- 
Atlantis ISP, System Administrator
e-mail:  dmitry@atlantis.dp.ua
nic-hdl: LYNX-RIPE