On Mon, Jan 02, 2012 at 01:25:49PM +0100, Victor Balada Diaz
wrote:> Hello,
>
> First: Happy new year to everybody!
>
> Last year i had a problem with pagedaemon that i reported here[1] but i had
> no replies. The problem went away and now it's back to beat me again.
>
> Seems that mysql creates some bad interactions with pagedaemon and system
starts
> to get slow for disk access. Eg, accessing a man page takes a few seconds.
If
> i stop mysql and start it again, it recovers again. As this is a problem
that have
> appeared just twice (on October and now), it's not easy to reproduce
and test.
>
> Disk I/O as per gstat is not high, not many context switches, etc.
Bottleneck
> seems to be CPU. I've reported more information on october's
mail[1].
>
> What can i do to debug it? Seems that maybe the most interesting thing
would be
> enabling lock profiling and wait for the system to fail again, but is there
anything
> that i can do/report that would be helpful to track the problem down now
that's
> failing?
>
> Thanks a lot.
> Regards.
>
> [1]:
http://lists.freebsd.org/pipermail/freebsd-stable/2011-October/064248.html
The only things I can think of are output from the following:
* vmstat -z
* sysctl -a vm
* top (mainly for mysqld)
* top -H -S
Is this machine in question still running 8.1-RELEASE? If so, have you
tried upgrading it to 8.2-STABLE? I believe there have been some VM
adjustments, but I'd have to go back and examine commits.
Taken from our mysql server. Note that the mysql storage filesystem is
stored on ZFS as well, which has added benefits/complexities yet we see
no issue.
FreeBSD XXXXXXXXXXXXXXXXX 8.2-STABLE FreeBSD 8.2-STABLE #0: Tue Dec 13 00:00:16
PST 2011 root@XXXXXXXXXXXXXXXXX:/usr/obj/usr/src/sys/X7SBI_RELENG_8_amd64 amd64
real memory = 8589934592 (8192 MB)
avail memory = 8250040320 (7867 MB)
Mem: 539M Active, 2559M Inact, 3218M Wired, 4K Cache, 828M Buf, 1603M Free
Swap: 16G Total, 16G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1940 mysql 16 20 0 781M 401M sigwai 1 158:40 0.00% [mysqld]
top -H -S shows no indication pagedaemon is "going crazy".
loader.conf tunings:
# Increase maximum allocatable memory on a process to 2.5GB.
# We don't choose 8GB (our amount of RAM) since that would
# exhaust all memory, and result in a kernel panic. Maximum
# stack size is set to 256MB. One can view these settings
# using limits(1).
#
# dfldsiz = Initial data size limit (bytes)
# maxdsiz = Maximum data size limit (bytes)
# dflssiz = Initial stack size limit (bytes)
# maxssiz = Maximum stack size limit (bytes)
#
kern.maxdsiz="2560M"
kern.dfldsiz="2560M"
kern.maxssiz="256M"
# Limit ZFS ARC maximum.
# NOTE #1: In 8.2-RELEASE and onward, vm.kmem_size_scale defaults to 1,
# which means vm.kmem_size should match the amount of RAM installed
# in the system. If using an earlier FreeBSD release, be sure to set
# vm.kmem_size manually to the amount of RAM you have.
# NOTE #2: Do not set vm.kmem_size to 2x that of physical RAM, otherwise
# vfs.zfs.arc_max effectively becomes halved.
# http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010875.html
vfs.zfs.arc_max="5120M"
my.cnf tunings:
[mysqld]
; http://dev.mysql.com/doc/refman/5.5/en/table-cache.html
table_open_cache = 2048
max_connections = 512
tmp_table_size = 64M
max_allowed_packet = 32M
key_buffer_size = 64M
join_buffer_size = 16M
sort_buffer_size = 16M
read_buffer_size = 16M
query_cache_size = 64M
query_cache_limit = 32M
innodb_file_per_table
innodb_buffer_pool_size = 512M
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_additional_mem_pool_size = 20M
sysctl -a vm output:
vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
==============================================Processes: (RUNQ: 1
Disk Wait: 1 Page Wait: 0 Sleep: 59)
Virtual Memory: (Total: 1075323828K Active: 1483408K)
Real Memory: (Total: 2970948K Active: 585264K)
Shared Virtual Memory: (Total: 17884K Active: 6756K)
Shared Real Memory: (Total: 7500K Active: 6564K)
Free Memory Pages: 1641708K
vm.loadavg: { 0.00 0.00 0.00 }
vm.v_free_min: 12824
vm.v_free_target: 53985
vm.v_free_reserved: 2689
vm.v_inactive_target: 80977
vm.v_cache_min: 53985
vm.v_cache_max: 107970
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.md_malloc_wait: 0
vm.kmem_map_free: 2897080320
vm.kmem_map_size: 2349465600
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 8303726592
vm.nswapdev: 1
vm.dmmax: 32
vm.swap_async_max: 4
vm.overcommit: 0
vm.swap_reserved: 11426652160
vm.swap_total: 17179869184
vm.zone_count: 195
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.kstacks: 475
vm.kstack_cache_size: 128
vm.exec_map_entries: 16
vm.stats.misc.zero_page_count: 158
vm.stats.misc.cnt_prezero: 0
vm.stats.vm.v_kthreadpages: 0
vm.stats.vm.v_rforkpages: 0
vm.stats.vm.v_vforkpages: 35572524
vm.stats.vm.v_forkpages: 131458827
vm.stats.vm.v_kthreads: 22
vm.stats.vm.v_rforks: 0
vm.stats.vm.v_vforks: 159338
vm.stats.vm.v_forks: 711839
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_cache_max: 107970
vm.stats.vm.v_cache_min: 53985
vm.stats.vm.v_cache_count: 1
vm.stats.vm.v_inactive_count: 655146
vm.stats.vm.v_inactive_target: 80977
vm.stats.vm.v_active_count: 137837
vm.stats.vm.v_wire_count: 823816
vm.stats.vm.v_free_count: 410409
vm.stats.vm.v_free_min: 12824
vm.stats.vm.v_free_target: 53985
vm.stats.vm.v_free_reserved: 2689
vm.stats.vm.v_page_count: 2027277
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_tfree: 488602276
vm.stats.vm.v_pfree: 167595871
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_tcached: 42510
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_reactivated: 8878
vm.stats.vm.v_intrans: 19805
vm.stats.vm.v_vnodepgsout: 1
vm.stats.vm.v_vnodepgsin: 58873
vm.stats.vm.v_vnodeout: 1
vm.stats.vm.v_vnodein: 47194
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_ozfod: 511
vm.stats.vm.v_zfod: 231124379
vm.stats.vm.v_cow_optim: 41799
vm.stats.vm.v_cow_faults: 39666798
vm.stats.vm.v_vm_faults: 299438319
vm.stats.sys.v_soft: 411033343
vm.stats.sys.v_intr: 263204059
vm.stats.sys.v_syscall: 3214862746
vm.stats.sys.v_trap: 645468188
vm.stats.sys.v_swtch: 2209989736
vm.stats.object.bypasses: 524363
vm.stats.object.collapses: 3306408
vm.v_free_severe: 7756
vm.old_msync: 0
vm.msync_flush_flags: 0
vm.boot_pages: 48
vm.max_wired: 666530
vm.pageout_lock_miss: 0
vm.disable_swapspace_pageouts: 0
vm.defer_swapspace_pageouts: 0
vm.swap_idle_enabled: 0
vm.pageout_stats_interval: 5
vm.pageout_full_stats_interval: 20
vm.pageout_stats_max: 53985
vm.max_launder: 32
vm.phys_segs:
SEGMENT 0:
start: 0x1000
end: 0x97000
free list: 0xffffffff80718da8
SEGMENT 1:
start: 0x8ad000
end: 0x1000000
free list: 0xffffffff80718da8
SEGMENT 2:
start: 0x1000000
end: 0xcfe70000
free list: 0xffffffff80718a00
SEGMENT 3:
start: 0x100000000
end: 0x21f8b4000
free list: 0xffffffff80718a00
vm.phys_free:
FREE LIST 0:
ORDER (SIZE) | NUMBER
| POOL 0 | POOL 1 | POOL 2
-- -- -- -- -- -- -- --
12 ( 16384K) | 0 | 0 | 0
11 ( 8192K) | 0 | 0 | 0
10 ( 4096K) | 22 | 0 | 0
9 ( 2048K) | 74 | 0 | 0
8 ( 1024K) | 10 | 0 | 0
7 ( 512K) | 15 | 1 | 0
6 ( 256K) | 10 | 1 | 0
5 ( 128K) | 4 | 1 | 0
4 ( 64K) | 1 | 0 | 0
3 ( 32K) | 1 | 0 | 0
2 ( 16K) | 0 | 0 | 0
1 ( 8K) | 0 | 1 | 0
0 ( 4K) | 0 | 1 | 0
FREE LIST 1:
ORDER (SIZE) | NUMBER
| POOL 0 | POOL 1 | POOL 2
-- -- -- -- -- -- -- --
12 ( 16384K) | 0 | 0 | 0
11 ( 8192K) | 0 | 0 | 0
10 ( 4096K) | 0 | 0 | 0
9 ( 2048K) | 0 | 0 | 0
8 ( 1024K) | 0 | 0 | 0
7 ( 512K) | 0 | 0 | 0
6 ( 256K) | 0 | 0 | 0
5 ( 128K) | 0 | 0 | 0
4 ( 64K) | 0 | 0 | 0
3 ( 32K) | 0 | 0 | 0
2 ( 16K) | 1 | 0 | 0
1 ( 8K) | 0 | 0 | 0
0 ( 4K) | 0 | 0 | 0
vm.reserv.reclaimed: 246
vm.reserv.partpopq:
LEVEL SIZE NUMBER
-1: 1377964K, 1226
vm.reserv.freed: 2823528
vm.reserv.broken: 5973
vm.idlezero_enable: 0
vm.kvm_free: 539964207104
vm.kvm_size: 549755809792
vm.pmap.pmap_collect_active: 0
vm.pmap.pmap_collect_inactive: 0
vm.pmap.pv_entry_spare: 5609
vm.pmap.pv_entry_allocs: 932709470
vm.pmap.pv_entry_frees: 932651743
vm.pmap.pc_chunk_tryfail: 0
vm.pmap.pc_chunk_frees: 5562480
vm.pmap.pc_chunk_allocs: 5562857
vm.pmap.pc_chunk_count: 377
vm.pmap.pv_entry_count: 57727
vm.pmap.pdpe.demotions: 0
vm.pmap.pde.promotions: 3234381
vm.pmap.pde.p_failures: 36313949
vm.pmap.pde.mappings: 10185
vm.pmap.pde.demotions: 3202630
vm.pmap.shpgperproc: 200
vm.pmap.pv_entry_max: 3260077
vm.pmap.pg_ps_enabled: 1
vm.pmap.pat_works: 1
vmstat -z output:
ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
UMA Kegs: 208, 0, 193, 11, 193, 0
UMA Zones: 320, 0, 193, 11, 193, 0
UMA Slabs: 568, 0, 91344, 573, 22063422, 0
UMA RCntSlabs: 568, 0, 1652, 0, 1652, 0
UMA Hash: 256, 0, 79, 11, 81, 0
16 Bucket: 152, 0, 198, 2, 198, 0
32 Bucket: 280, 0, 220, 4, 220, 0
64 Bucket: 536, 0, 237, 1, 237, 324
128 Bucket: 1048, 0, 5090, 1, 5090, 3812
VM OBJECT: 216, 0, 112042, 1682, 19218174, 0
MAP: 232, 0, 7, 25, 7, 0
KMAP ENTRY: 120, 278070, 2358, 8802, 56999587, 0
MAP ENTRY: 120, 0, 1254, 2063, 33749125, 0
DP fakepg: 120, 0, 0, 0, 0, 0
SG fakepg: 120, 0, 0, 0, 0, 0
mt_zone: 2056, 0, 203, 34, 203, 0
16: 16, 0, 5939, 4309, 812209016, 0
32: 32, 0, 2778, 9948, 36156302, 0
64: 64, 0, 15479, 124465, 559617065, 0
128: 128, 0, 5998, 6124, 500883081, 0
256: 256, 0, 1542, 105408, 116395928, 0
512: 512, 0, 5900, 5195, 30945149, 0
1024: 1024, 0, 203, 1789, 10984378, 0
2048: 2048, 0, 449, 4339, 14614660, 0
4096: 4096, 0, 789, 784, 11198444, 0
Files: 80, 0, 780, 795, 19941086, 0
TURNSTILE: 136, 0, 778, 102, 778, 0
umtx pi: 96, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0
PROC: 1136, 0, 62, 181, 871274, 0
THREAD: 1096, 0, 475, 302, 1185892, 0
SLEEPQUEUE: 80, 0, 778, 150, 778, 0
VMSPACE: 392, 0, 41, 189, 871256, 0
cpuset: 72, 0, 2, 98, 2, 0
audit_record: 952, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 2046, 770, 119097913, 0
mbuf: 256, 0, 2, 902, 596890710, 0
mbuf_cluster: 2048, 25600, 2816, 116, 2834, 0
mbuf_jumbo_page: 4096, 12800, 0, 186, 8204717, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0
g_bio: 232, 0, 0, 12624, 132751934, 0
ttyinq: 160, 0, 225, 207, 1260, 0
ttyoutq: 256, 0, 117, 153, 663, 0
ata_request: 320, 0, 0, 84, 458, 0
ata_composite: 336, 0, 0, 0, 0, 0
VNODE: 472, 0, 162386, 758, 7441920, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 96, 56473208, 0
S VFS Cache: 108, 0, 178642, 647, 8950044, 0
L VFS Cache: 328, 0, 860, 2428, 128661, 0
DIRHASH: 1024, 0, 930, 134, 1315, 0
NFSMOUNT: 632, 0, 1, 11, 1, 0
NFSNODE: 688, 0, 6, 19, 6, 0
pipe: 728, 0, 40, 115, 292578, 0
ksiginfo: 112, 0, 232, 824, 86359, 0
itimer: 344, 0, 1, 54, 6, 0
KNOTE: 128, 0, 137, 414, 13517474, 0
socket: 680, 25602, 114, 180, 1932153, 0
unpcb: 240, 25600, 49, 207, 121492, 0
ipq: 56, 819, 0, 0, 0, 0
udp_inpcb: 336, 25608, 19, 168, 351143, 0
udpcb: 16, 25704, 19, 821, 351143, 0
tcp_inpcb: 336, 25608, 49, 633, 1226932, 0
tcpcb: 944, 25600, 36, 124, 1226932, 0
tcptw: 72, 5150, 13, 837, 479105, 0
syncache: 144, 15366, 0, 156, 1191373, 0
hostcache: 136, 15372, 2, 166, 267, 0
tcpreass: 40, 1680, 0, 336, 703, 0
sackhole: 32, 0, 0, 505, 725, 0
ripcb: 336, 25608, 8, 146, 232583, 0
rtentry: 200, 0, 12, 64, 12, 0
selfd: 56, 0, 257, 751, 629664344, 0
SWAPMETA: 288, 116519, 0, 0, 0, 0
ip4flow: 56, 99351, 0, 0, 0, 0
ip6flow: 80, 99360, 0, 0, 0, 0
Mountpoints: 752, 0, 9, 16, 9, 0
FFS inode: 168, 0, 157347, 4925, 7264635, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 157347, 4848, 7264635, 0
taskq_zone: 48, 0, 0, 1008, 392677, 0
zio_cache: 880, 0, 1, 23083, 693486370, 0
zio_link_cache: 48, 0, 0, 23472, 267139845, 0
zio_buf_512: 512, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0
sa_cache: 80, 0, 4987, 1988, 176638, 0
dnode_t: 856, 0, 5815, 2089, 44082, 0
dmu_buf_impl_t: 224, 0, 12748, 36144, 1117594, 0
arc_buf_hdr_t: 216, 0, 229221, 243, 1133714, 0
arc_buf_t: 104, 0, 87053, 787, 1243813, 0
zil_lwb_cache: 192, 0, 2, 778, 236435, 0
zfs_znode_cache: 400, 0, 4987, 2195, 176638, 0
pfsrctrpl: 152, 10000, 0, 0, 0, 0
pfrulepl: 912, 0, 34, 78, 124, 0
pfstatepl: 392, 10000, 34, 1866, 1123722, 0
pfaltqpl: 240, 0, 0, 0, 0, 0
pfpooladdrpl: 88, 0, 0, 0, 0, 0
pfrktable: 1296, 1002, 5, 25, 25, 0
pfrkentry: 216, 100008, 658, 890, 2628, 0
pfrkentry2: 216, 0, 0, 0, 0, 0
pffrent: 32, 5050, 0, 0, 0, 0
pffrag: 80, 0, 0, 0, 0, 0
pffrcache: 80, 10035, 0, 0, 0, 0
pffrcent: 24, 50022, 0, 0, 0, 0
pfstatescrub: 40, 0, 0, 0, 0, 0
pfiaddrpl: 120, 0, 0, 0, 0, 0
pfospfen: 112, 0, 700, 92, 2800, 0
pfosfp: 40, 0, 410, 178, 1640, 0
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |