thr3ads.net - freebsd stable - scp: Write Failed: Cannot allocate memory [Jul 2011]

If this information is useful, please help other people find it:
Share via:

Scott Sipe

2011-Jul-01 19:36 UTC

scp: Write Failed: Cannot allocate memory

Hello,

I'm running 8.2-RELEASE and am having new problems with scp. When scping
files to a ZFS directory on the FreeBSD server -- most notably large files
-- the transfer frequently dies after just a few seconds. In my last test, I
tried to scp an 800mb file to the FreeBSD system and the transfer died after
200mb. It completely copied the next 4 times I tried, and then died again on
the next attempt.

On the client side:

"Connection to home closed by remote host.
lost connection"

In /var/log/auth.log:

Jul  1 14:54:42 freebsd sshd[18955]: fatal: Write failed: Cannot allocate
memory

I've never seen this before and have used scp before to transfer large files
without problems. This computer has been used in production for months and
has a current uptime of 36 days. I have not been able to notice any problems
copying files to the server via samba or netatalk, or any problems in
apache.

Uname:

FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19 01:02:54 EST
2011     root@xeon:/usr/obj/usr/src/sys/GENERIC  amd64

I've attached my dmesg and output of vmstat -z.

I have not restarted the sshd daemon or rebooted the computer.

Am glad to provide any other information or test anything else.

Thanks,
Scott
-------------- next part --------------
# vmstat -z
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES

UMA Kegs:                 208,        0,      205,       16,      205,        0
UMA Zones:                704,        0,      205,        0,      205,        0
UMA Slabs:                568,        0,    69503,    32711, 72942509,        0
UMA RCntSlabs:            568,        0,     4667,      583,     5664,        0
UMA Hash:                 256,        0,       79,       11,       81,        0
16 Bucket:                152,        0,       85,      165,      227,        0
32 Bucket:                280,        0,      238,      196,      442,        1
64 Bucket:                536,        0,      359,      222,      721,      126
128 Bucket:              1048,        0,     5867,        1,    45296,   196701
VM OBJECT:                216,        0,    26698,    27140, 40712912,        0
MAP:                      232,        0,        7,       25,        7,        0
KMAP ENTRY:               120,   412734,    15244,     3201, 200399669,        0
MAP ENTRY:                120,        0,    10795,     7557, 64096457,        0
DP fakepg:                120,        0,        0,        0,        0,        0
SG fakepg:                120,        0,        0,        0,        0,        0
mt_zone:                 2056,        0,      278,        1,      278,        0
16:                        16,        0,    88568,   153688, 1913750912,       
0
32:                        32,        0,     3825,    12032, 260796507,        0
64:                        64,        0,    70106,   195950, 21507520870,       
0
128:                      128,        0,   428518,   337836, 3162416717,       
0
256:                      256,        0,    47071,   197099, 517289563,        0
512:                      512,        0,   593921,   656853, 563338488,        0
1024:                    1024,        0,     7190,    18422, 25604261,        0
2048:                    2048,        0,     5271,     5559,  7565792,        0
4096:                    4096,        0,     3344,     2030, 57255230,        0
Files:                     80,        0,     1603,     2312, 52443448,        0
TURNSTILE:                136,        0,     2515,      265,     2545,        0
umtx pi:                   96,        0,        0,        0,        0,        0
MAC labels:                40,        0,        0,        0,        0,        0
PROC:                    1120,        0,      112,     1544,  1117378,        0
THREAD:                  1112,        0,     1932,      582,     3101,        0
SLEEPQUEUE:                80,        0,     2515,      298,     2545,        0
VMSPACE:                  392,        0,       86,      784,  1118412,        0
cpuset:                    72,        0,        2,       98,        2,        0
audit_record:             952,        0,        0,        0,        0,        0
mbuf_packet:              256,        0,     1024,     3584, 218220869,        0
mbuf:                     256,        0,        1,     3461, 562735494,        0
mbuf_cluster:            2048,    25600,     4608,     1008,     5632,        0
mbuf_jumbo_page:         4096,    12800,        0,     1859, 50017085,        0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,     3200,        0,        0,        0,        0
mbuf_ext_refcnt:            4,        0,        0,     3024,     6978,        0
g_bio:                    232,        0,        0,     2480, 492271394,        0
ttyinq:                   160,        0,      135,      801,     2205,        0
ttyoutq:                  256,        0,       72,      483,     1176,        0
ata_request:              320,        0,        0,        0,        0,        0
ata_composite:            336,        0,        0,        0,        0,        0
taskq_zone:                56,        0,        0,     3150,  6008496,        0
VNODE:                    472,        0,    77908,    24228, 175076139,        0
VNODEPOLL:                112,        0,        7,       92,        9,        0
NAMEI:                   1024,        0,        0,      512, 485735231,        0
S VFS Cache:              108,        0,    75050,    32860, 159914963,        0
L VFS Cache:              328,        0,     2079,    36285, 18155578,        0
DIRHASH:                 1024,        0,        0,        0,        0,        0
NFSMOUNT:                 624,        0,        0,        0,        0,        0
NFSNODE:                  688,        0,        0,        0,        0,        0
zio_cache:                864,        0,        1,    50175, 2757424752,       
0
zio_link_cache:            48,        0,        0,    50184, 654814680,        0
zio_buf_512:              512,        0,        0,        0,        0,        0
zio_data_buf_512:         512,        0,        0,        0,        0,        0
zio_buf_1024:            1024,        0,        0,        0,        0,        0
zio_data_buf_1024:       1024,        0,        0,        0,        0,        0
zio_buf_1536:            1536,        0,        0,        0,        0,        0
zio_data_buf_1536:       1536,        0,        0,        0,        0,        0
zio_buf_2048:            2048,        0,        0,        0,        0,        0
zio_data_buf_2048:       2048,        0,        0,        0,        0,        0
zio_buf_2560:            2560,        0,        0,        0,        0,        0
zio_data_buf_2560:       2560,        0,        0,        0,        0,        0
zio_buf_3072:            3072,        0,        0,        0,        0,        0
zio_data_buf_3072:       3072,        0,        0,        0,        0,        0
zio_buf_3584:            3584,        0,        0,        0,        0,        0
zio_data_buf_3584:       3584,        0,        0,        0,        0,        0
zio_buf_4096:            4096,        0,        0,        0,        0,        0
zio_data_buf_4096:       4096,        0,        0,        0,        0,        0
zio_buf_5120:            5120,        0,        0,        0,        0,        0
zio_data_buf_5120:       5120,        0,        0,        0,        0,        0
zio_buf_6144:            6144,        0,        0,        0,        0,        0
zio_data_buf_6144:       6144,        0,        0,        0,        0,        0
zio_buf_7168:            7168,        0,        0,        0,        0,        0
zio_data_buf_7168:       7168,        0,        0,        0,        0,        0
zio_buf_8192:            8192,        0,        0,        0,        0,        0
zio_data_buf_8192:       8192,        0,        0,        0,        0,        0
zio_buf_10240:          10240,        0,        0,        0,        0,        0
zio_data_buf_10240:     10240,        0,        0,        0,        0,        0
zio_buf_12288:          12288,        0,        0,        0,        0,        0
zio_data_buf_12288:     12288,        0,        0,        0,        0,        0
zio_buf_14336:          14336,        0,        0,        0,        0,        0
zio_data_buf_14336:     14336,        0,        0,        0,        0,        0
zio_buf_16384:          16384,        0,        0,        0,        0,        0
zio_data_buf_16384:     16384,        0,        0,        0,        0,        0
zio_buf_20480:          20480,        0,        0,        0,        0,        0
zio_data_buf_20480:     20480,        0,        0,        0,        0,        0
zio_buf_24576:          24576,        0,        0,        0,        0,        0
zio_data_buf_24576:     24576,        0,        0,        0,        0,        0
zio_buf_28672:          28672,        0,        0,        0,        0,        0
zio_data_buf_28672:     28672,        0,        0,        0,        0,        0
zio_buf_32768:          32768,        0,        0,        0,        0,        0
zio_data_buf_32768:     32768,        0,        0,        0,        0,        0
zio_buf_36864:          36864,        0,        0,        0,        0,        0
zio_data_buf_36864:     36864,        0,        0,        0,        0,        0
zio_buf_40960:          40960,        0,        0,        0,        0,        0
zio_data_buf_40960:     40960,        0,        0,        0,        0,        0
zio_buf_45056:          45056,        0,        0,        0,        0,        0
zio_data_buf_45056:     45056,        0,        0,        0,        0,        0
zio_buf_49152:          49152,        0,        0,        0,        0,        0
zio_data_buf_49152:     49152,        0,        0,        0,        0,        0
zio_buf_53248:          53248,        0,        0,        0,        0,        0
zio_data_buf_53248:     53248,        0,        0,        0,        0,        0
zio_buf_57344:          57344,        0,        0,        0,        0,        0
zio_data_buf_57344:     57344,        0,        0,        0,        0,        0
zio_buf_61440:          61440,        0,        0,        0,        0,        0
zio_data_buf_61440:     61440,        0,        0,        0,        0,        0
zio_buf_65536:          65536,        0,        0,        0,        0,        0
zio_data_buf_65536:     65536,        0,        0,        0,        0,        0
zio_buf_69632:          69632,        0,        0,        0,        0,        0
zio_data_buf_69632:     69632,        0,        0,        0,        0,        0
zio_buf_73728:          73728,        0,        0,        0,        0,        0
zio_data_buf_73728:     73728,        0,        0,        0,        0,        0
zio_buf_77824:          77824,        0,        0,        0,        0,        0
zio_data_buf_77824:     77824,        0,        0,        0,        0,        0
zio_buf_81920:          81920,        0,        0,        0,        0,        0
zio_data_buf_81920:     81920,        0,        0,        0,        0,        0
zio_buf_86016:          86016,        0,        0,        0,        0,        0
zio_data_buf_86016:     86016,        0,        0,        0,        0,        0
zio_buf_90112:          90112,        0,        0,        0,        0,        0
zio_data_buf_90112:     90112,        0,        0,        0,        0,        0
zio_buf_94208:          94208,        0,        0,        0,        0,        0
zio_data_buf_94208:     94208,        0,        0,        0,        0,        0
zio_buf_98304:          98304,        0,        0,        0,        0,        0
zio_data_buf_98304:     98304,        0,        0,        0,        0,        0
zio_buf_102400:        102400,        0,        0,        0,        0,        0
zio_data_buf_102400:   102400,        0,        0,        0,        0,        0
zio_buf_106496:        106496,        0,        0,        0,        0,        0
zio_data_buf_106496:   106496,        0,        0,        0,        0,        0
zio_buf_110592:        110592,        0,        0,        0,        0,        0
zio_data_buf_110592:   110592,        0,        0,        0,        0,        0
zio_buf_114688:        114688,        0,        0,        0,        0,        0
zio_data_buf_114688:   114688,        0,        0,        0,        0,        0
zio_buf_118784:        118784,        0,        0,        0,        0,        0
zio_data_buf_118784:   118784,        0,        0,        0,        0,        0
zio_buf_122880:        122880,        0,        0,        0,        0,        0
zio_data_buf_122880:   122880,        0,        0,        0,        0,        0
zio_buf_126976:        126976,        0,        0,        0,        0,        0
zio_data_buf_126976:   126976,        0,        0,        0,        0,        0
zio_buf_131072:        131072,        0,        0,        0,        0,        0
zio_data_buf_131072:   131072,        0,        0,        0,        0,        0
dmu_buf_impl_t:           224,        0,   655561,   709862, 192104158,        0
dnode_t:                  784,        0,   580985,   587305, 162475612,        0
arc_buf_hdr_t:            208,        0,   210039,   281325,  9982125,        0
arc_buf_t:                 72,        0,    80145,   103455, 30682442,        0
zil_lwb_cache:            200,        0,       14,    12165,  1018999,        0
zfs_znode_cache:          352,        0,    77830,    24173, 175074277,        0
pipe:                     728,        0,       65,     1400,   600022,        0
AIO:                      208,        0,       16,      452,      233,        0
AIOP:                      32,        0,        4,      198,        4,        0
AIOCB:                    480,        0,        0,      448,  3955818,        0
AIOL:                     128,        0,        0,        0,        0,        0
AIOLIO:                   272,        0,        0,        0,        0,        0
ksiginfo:                 112,        0,     1645,     1490,   596339,        0
itimer:                   344,        0,        2,       31,        2,        0
KNOTE:                    128,        0,       31,     1129, 13658037,        0
socket:                   680,    25602,      149,     1543,  1355501,        0
ipq:                       56,      819,        0,      252,        3,        0
udp_inpcb:                336,    25608,       19,      663,   478090,        0
udpcb:                     16,    25704,       19,     2333,   478090,        0
tcp_inpcb:                336,    25608,       73,     1280,   357055,        0
tcpcb:                    880,    25600,       72,      772,   357055,        0
tcptw:                     72,     5150,        1,     1799,   104116,        0
syncache:                 144,    15366,        0,      650,   270152,        0
hostcache:                136,    15372,       16,      712,     1606,        0
tcpreass:                  40,     1680,        0,     1680,    29393,        0
sackhole:                  32,        0,        0,     1717,     5335,        0
sctp_ep:                 1272,    25602,        0,        0,        0,        0
sctp_asoc:               2240,    40000,        0,        0,        0,        0
sctp_laddr:                48,    80064,        0,      144,        3,        0
sctp_raddr:               616,    80004,        0,        0,        0,        0
sctp_chunk:               136,   400008,        0,        0,        0,        0
sctp_readq:               104,   400032,        0,        0,        0,        0
sctp_stream_msg_out:       96,   400026,        0,        0,        0,        0
sctp_asconf:               40,   400008,        0,        0,        0,        0
sctp_asconf_ack:           48,   400032,        0,        0,        0,        0
ripcb:                    336,    25608,        0,      110,       61,        0
unpcb:                    240,    25600,       56,     1176,   520291,        0
rtentry:                  200,        0,        9,       48,        9,        0
pfsrctrpl:                152,    10000,        0,        0,        0,        0
pfrulepl:                 912,        0,       80,       72,      152,        0
pfstatepl:                392,    10000,        0,        0,        0,        0
pfaltqpl:                 240,        0,        0,        0,        0,        0
pfpooladdrpl:              88,        0,        0,        0,        0,        0
pfrktable:               1296,     1002,        0,        0,        0,        0
pfrkentry:                216,   100008,        0,        0,        0,        0
pfrkentry2:               216,        0,        0,        0,        0,        0
pffrent:                   32,     5050,        0,        0,        0,        0
pffrag:                    80,        0,        0,        0,        0,        0
pffrcache:                 80,    10035,        0,        0,        0,        0
pffrcent:                  24,    50022,        0,        0,        0,        0
pfstatescrub:              40,        0,        0,        0,        0,        0
pfiaddrpl:                120,        0,        0,        0,        0,        0
pfospfen:                 112,        0,      696,       30,     1392,        0
pfosfp:                    40,        0,      407,       97,      814,        0
selfd:                     56,        0,     1927,     2357, 850297759,        0
SWAPMETA:                 288,   116519,        1,       25,        1,        0
ip4flow:                   56,   394254,       49,     2030,   764004,        0
ip6flow:                   80,   394245,        0,        0,        0,        0
Mountpoints:              752,        0,       24,       31,       26,        0
NetGraph items:            72,     4118,        0,       87,        4,        0
NetGraph data items:       72,      522,        0,      522, 404316704,     1204
-------------- next part --------------
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE #0: Sat Feb 19 01:02:54 EST 2011
    root@xeon:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz (2266.77-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106a5  Family = 6  Model = 1a 
Stepping = 5
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 12884901888 (12288 MB)
avail memory = 12368928768 (11795 MB)
ACPI APIC Table: <031710 APIC1617>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
 cpu8 (AP): APIC ID: 16
 cpu9 (AP): APIC ID: 17
 cpu10 (AP): APIC ID: 18
 cpu11 (AP): APIC ID: 19
 cpu12 (AP): APIC ID: 20
 cpu13 (AP): APIC ID: 21
 cpu14 (AP): APIC ID: 22
 cpu15 (AP): APIC ID: 23
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <031710 XSDT1617> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xAD, should be 0xAA
(20101013/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci10: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci9: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci8: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 8.0 on pci0
pci7: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> at device 9.0 on pci0
pci6: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 10.0 on pci0
pci5: <PCI bus> on pcib6
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver
attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver
attached)
pci0: <base peripheral> at device 22.0 (no driver attached)
pci0: <base peripheral> at device 22.1 (no driver attached)
pci0: <base peripheral> at device 22.2 (no driver attached)
pci0: <base peripheral> at device 22.3 (no driver attached)
pci0: <base peripheral> at device 22.4 (no driver attached)
pci0: <base peripheral> at device 22.5 (no driver attached)
pci0: <base peripheral> at device 22.6 (no driver attached)
pci0: <base peripheral> at device 22.7 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xa400-0xa41f irq
16 at device 26.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xa480-0xa49f irq
21 at device 26.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xa800-0xa81f irq
19 at device 26.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xfbcf4000-0xfbcf43ff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib8
em0: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0xec00-0xec1f mem
0xfbee0000-0xfbefffff,0xfbedc000-0xfbedffff irq 16 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: [ITHREAD]
em0: [ITHREAD]
em0: [ITHREAD]
em0: Ethernet address: snip
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci2: <ACPI PCI bus> on pcib9
em1: <Intel(R) PRO/1000 Network Connection 7.1.9> port 0xdc00-0xdc1f mem
0xfbde0000-0xfbdfffff,0xfbddc000-0xfbddffff irq 17 at device 0.0 on pci2
em1: Using MSIX interrupts with 3 vectors
em1: [ITHREAD]
em1: [ITHREAD]
em1: [ITHREAD]
em1: Ethernet address: snip
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xa880-0xa89f irq
23 at device 29.0 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xac00-0xac1f irq
19 at device 29.1 on pci0
uhci4: [ITHREAD]
uhci4: LegSup = 0x2f00
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xb000-0xb01f irq
18 at device 29.2 on pci0
uhci5: [ITHREAD]
uhci5: LegSup = 0x2f00
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xfbcf6000-0xfbcf63ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xcc00-0xcc7f mem
0xfb000000-0xfb7fffff,0xfafe0000-0xfaffffff irq 16 at device 0.0 on pci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH10 AHCI SATA controller> port
0xb880-0xb887,0xb800-0xb803,0xb480-0xb487,0xb400-0xb403,0xb080-0xb09f mem
0xfbcf8000-0xfbcf87ff irq 19 at device 31.2 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
qpi0: <QPI system bus> on motherboard
pcib11: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib11
pcib12: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci254: <PCI bus> on pcib12
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
p4tcc8: <CPU Frequency Thermal Control> on cpu8
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
p4tcc9: <CPU Frequency Thermal Control> on cpu9
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
p4tcc10: <CPU Frequency Thermal Control> on cpu10
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
p4tcc11: <CPU Frequency Thermal Control> on cpu11
coretemp12: <CPU On-Die Thermal Sensors> on cpu12
est12: <Enhanced SpeedStep Frequency Control> on cpu12
p4tcc12: <CPU Frequency Thermal Control> on cpu12
coretemp13: <CPU On-Die Thermal Sensors> on cpu13
est13: <Enhanced SpeedStep Frequency Control> on cpu13
p4tcc13: <CPU Frequency Thermal Control> on cpu13
coretemp14: <CPU On-Die Thermal Sensors> on cpu14
est14: <Enhanced SpeedStep Frequency Control> on cpu14
p4tcc14: <CPU Frequency Thermal Control> on cpu14
coretemp15: <CPU On-Die Thermal Sensors> on cpu15
est15: <Enhanced SpeedStep Frequency Control> on cpu15
p4tcc15: <CPU Frequency Thermal Control> on cpu15
ZFS filesystem version 4
ZFS storage pool version 15
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x5e8 offMax=0x8ec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD2003FYYS-02W0B0 01.01D01> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD2003FYYS-02W0B0 01.01D01> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #12 Launched!
Root mount waiting for: usbus7 usbus3
Root mount waiting for: usbus7 usbus3
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
ugen7.2: <American Megatrends Inc.> at usbus7
Trying to mount root from zfs:zroot
vboxnet0: Ethernet address: 0a:00:27:00:00:00
em1: link state changed to UP
em1: promiscuous mode enabled

Jeremy Chadwick

2011-Jul-01 22:22 UTC

head link

scp: Write Failed: Cannot allocate memory

On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott Sipe
wrote:> I'm running 8.2-RELEASE and am having new problems with scp. When
scping
> files to a ZFS directory on the FreeBSD server -- most notably large files
> -- the transfer frequently dies after just a few seconds. In my last test,
I
> tried to scp an 800mb file to the FreeBSD system and the transfer died
after
> 200mb. It completely copied the next 4 times I tried, and then died again
on
> the next attempt.
> 
> On the client side:
> 
> "Connection to home closed by remote host.
> lost connection"
> 
> In /var/log/auth.log:
> 
> Jul  1 14:54:42 freebsd sshd[18955]: fatal: Write failed: Cannot allocate
> memory
> 
> I've never seen this before and have used scp before to transfer large
files
> without problems. This computer has been used in production for months and
> has a current uptime of 36 days. I have not been able to notice any
problems
> copying files to the server via samba or netatalk, or any problems in
> apache.
> 
> Uname:
> 
> FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19 01:02:54 EST
> 2011     root@xeon:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> I've attached my dmesg and output of vmstat -z.
> 
> I have not restarted the sshd daemon or rebooted the computer.
> 
> Am glad to provide any other information or test anything else.
>
> {snip vmstat -z and dmesg}
You didn't provide details about your networking setup (rc.conf,
ifconfig -a, etc.).  netstat -m would be useful too.

Next, please see this thread circa September 2010, titled "Network
memory allocation failures":

http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.html#58708

The user in that thread is using rsync, which relies on scp by default.
I believe this problem is similar, if not identical, to yours.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |

Peter Ross

2011-Jul-06 02:23 UTC

head link

scp: Write Failed: Cannot allocate memory

Quoting "Jeremy Chadwick" <freebsd@jdc.parodius.com>:
> On Tue, Jul 05, 2011 at 01:03:20PM -0400, Scott Sipe wrote:
>> I'm running virtualbox 3.2.12_1 if that has anything to do with it.
>>
>> sysctl vfs.zfs.arc_max: 6200000000
>>
>> While I'm trying to scp, kstat.zfs.misc.arcstats.size is hovering  
>> right around that value, sometimes above, sometimes below (that's  
>> as it should be, right?). I don't think that it dies when crossing
>> over arc_max. I can run the same scp 10 times and it might fail 1-3  
>> times, with no correlation to the arcstats.size being above/below  
>> arc_max that I can see.
>>
>> Scott
>>
>> On Jul 5, 2011, at 3:00 AM, Peter Ross wrote:
>>
>>> Hi all,
>>>
>>> just as an addition: an upgrade to last Friday's FreeBSD-Stable
>>> and to VirtualBox 4.0.8 does not fix the problem.
>>>
>>> I will experiment a bit more tomorrow after hours and grab some
statistics.
>>>
>>> Regards
>>> Peter
>>>
>>> Quoting "Peter Ross"
<Peter.Ross@bogen.in-berlin.de>:
>>>
>>>> Hi all,
>>>>
>>>> I noticed a similar problem last week. It is also very similar
to
>>>> one reported last year:
>>>>
>>>>
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html
>>>>
>>>> My server is a Dell T410 server with the same bge card (the
same
>>>> pciconf -lvc output as described by Mahlon:
>>>>
>>>>
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058711.html
>>>>
>>>> Yours, Scott, is a em(4)..
>>>>
>>>> Another similarity: In all cases we are using VirtualBox. I
just
>>>> want to mention it, in case it matters. I am still running  
>>>> VirtualBox 3.2.
>>>>
>>>> Most of the time kstat.zfs.misc.arcstats.size was reaching  
>>>> vfs.zfs.arc_max then, but I could catch one or two cases then
the
>>>> value was still below.
>>>>
>>>> I added vfs.zfs.prefetch_disable=1 to sysctl.conf but it does
not help.
>>>>
>>>> BTW: It looks as ARC only gives back the memory when I destroy
>>>> the ZFS (a cloned snapshot containing virtual machines). Even
if
>>>> nothing happens for hours the buffer isn't released..
>>>>
>>>> My machine was still running 8.2-PRERELEASE so I am upgrading.
>>>>
>>>> I am happy to give information gathered on old/new kernel if it
helps.
>>>>
>>>> Regards
>>>> Peter
>>>>
>>>> Quoting "Scott Sipe" <cscotts@gmail.com>:
>>>>
>>>>>
>>>>> On Jul 2, 2011, at 12:54 AM, jhell wrote:
>>>>>
>>>>>> On Fri, Jul 01, 2011 at 03:22:32PM -0700, Jeremy
Chadwick wrote:
>>>>>>> On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott
Sipe wrote:
>>>>>>>> I'm running 8.2-RELEASE and am having new
problems with scp.
>>>>>>>> When scping
>>>>>>>> files to a ZFS directory on the FreeBSD server
-- most
>>>>>>>> notably large files
>>>>>>>> -- the transfer frequently dies after just a
few seconds. In
>>>>>>>> my last test, I
>>>>>>>> tried to scp an 800mb file to the FreeBSD
system and the
>>>>>>>> transfer died after
>>>>>>>> 200mb. It completely copied the next 4 times I
tried, and
>>>>>>>> then died again on
>>>>>>>> the next attempt.
>>>>>>>>
>>>>>>>> On the client side:
>>>>>>>>
>>>>>>>> "Connection to home closed by remote host.
>>>>>>>> lost connection"
>>>>>>>>
>>>>>>>> In /var/log/auth.log:
>>>>>>>>
>>>>>>>> Jul  1 14:54:42 freebsd sshd[18955]: fatal:
Write failed:
>>>>>>>> Cannot allocate
>>>>>>>> memory
>>>>>>>>
>>>>>>>> I've never seen this before and have used
scp before to
>>>>>>>> transfer large files
>>>>>>>> without problems. This computer has been used
in production
>>>>>>>> for months and
>>>>>>>> has a current uptime of 36 days. I have not
been able to
>>>>>>>> notice any problems
>>>>>>>> copying files to the server via samba or
netatalk, or any problems in
>>>>>>>> apache.
>>>>>>>>
>>>>>>>> Uname:
>>>>>>>>
>>>>>>>> FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE
#0: Sat Feb 19
>>>>>>>> 01:02:54 EST
>>>>>>>> 2011     root@xeon:/usr/obj/usr/src/sys/GENERIC
amd64
>>>>>>>>
>>>>>>>> I've attached my dmesg and output of vmstat
-z.
>>>>>>>>
>>>>>>>> I have not restarted the sshd daemon or
rebooted the computer.
>>>>>>>>
>>>>>>>> Am glad to provide any other information or
test anything else.
>>>>>>>>
>>>>>>>> {snip vmstat -z and dmesg}
>>>>>>>
>>>>>>> You didn't provide details about your
networking setup (rc.conf,
>>>>>>> ifconfig -a, etc.).  netstat -m would be useful
too.
>>>>>>>
>>>>>>> Next, please see this thread circa September 2010,
titled "Network
>>>>>>> memory allocation failures":
>>>>>>>
>>>>>>>
http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.html#58708
>>>>>>>
>>>>>>> The user in that thread is using rsync, which
relies on scp by default.
>>>>>>> I believe this problem is similar, if not
identical, to yours.
>>>>>>>
>>>>>>
>>>>>> Please also provide your output of ( /usr/bin/limits -a
) for the server
>>>>>> end and the client.
>>>>>>
>>>>>> I am not quite sure I agree with the need for ifconfig
-a but some
>>>>>> information about the networking driver your using for
the interface
>>>>>> would be helpful, uptime of the boxes. And
configuration of the pool.
>>>>>> e.g. ( zpool status -a ;zfs get all <poolname> )
You should probably
>>>>>> prop this information up somewhere so you can reference
by URL whenever
>>>>>> needed.
>>>>>>
>>>>>> rsync(1) does not rely on scp(1) whatsoever but
rsync(1) can be made to
>>>>>> use ssh(1) instead of rsh(1) and I believe that is what
Jeremy is
>>>>>> stating here but correct me if I am wrong. It does use
ssh(1) by
>>>>>> default.
>>>>>>
>>>>>> Its a possiblity as well that if using tmpfs(5) or
mdmfs(8) for /tmp
>>>>>> type filesystems that rsync(1) may be just filling up
your temp ram area
>>>>>> and causing the connection abort which would be
expected. ( df
>>>>>> -h ) would
>>>>>> help here.
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm not using tmpfs/mdmfs at all. The clients yesterday
were 3
>>>>> different OSX computers (over gigabit). The FreeBSD server
has
>>>>> 12gb of ram and no bce adapter. For what it's worth,
the server
>>>>> is backed up remotely every night with rsync (remote
FreeBSD
>>>>> uses rsync to pull) to an offsite (slow cable connection)  
>>>>> FreeBSD computer, and I have not seen any errors in the
nightly
>>>>> rsync.
>>>>>
>>>>> Sorry for the omission of networking info, here's the
output of
>>>>> the requested commands and some that popped up in the other
>>>>> thread:
>>>>>
>>>>> http://www.cap-press.com/misc/
>>>>>
>>>>> In rc.conf:  ifconfig_em1="inet 10.1.1.1 netmask
255.255.0.0"
>>>>>
>>>>> Scott
>
> Just to make it crystal clear to everyone:
>
> There is no correlation between this problem and use of ZFS.  People are
> attempting to correlate "cannot allocate memory" messages with
"anything
> on the system that uses memory".  The VM is much more complex than
that.
>
> Given the nature of this problem, it's much more likely the issue is
> "somewhere" within a networking layer within FreeBSD, whether it
be
> driver-level or some sort of intermediary layer.
>
> Two people who have this issue in this thread are both using VirtualBox.
> Can one, or both, of you remove VirtualBox from the configuration
> entirely (kernel, etc. -- not sure what is required) and then see if the
> issue goes away?
On the machine in question I only can do it after hours so I will do  
it tonight.

I was _successfully_ sending the file over the loopback interface using

cat /zpool/temp/zimbra_oldroot.vdi | ssh localhost "cat >
/dev/null"

I did it, btw, with the IPv6 localhost address first (accidently), and  
then using IPv4. Both worked.

It always fails if I am sending it through the bce(4) interface, even  
if my target is the VirtualBox bridged to the bce card (so it does not  
"leave" the computer physically).

Below the uname -a, ifconfig -a, netstat -rn, pciconf -lv and kldstat output.

I have another box where I do not see that problem. It copies files  
happily over the net using ssh.

It is an an older HP ML 150 with 3GB RAM only but with a bge(4) driver  
instead. It runs the same last week's RELENG_8. I installed VirtualBox  
and enabled vboxnet (so it loads the kernel modules). But I do not run  
VirtualBox on it (because it hasn't enough RAM).

Regards
Peter

DellT410one# uname -a
FreeBSD DellT410one.vv.fda 8.2-STABLE FreeBSD 8.2-STABLE #1: Thu Jun  
30 17:07:18 EST 2011      
root@DellT410one.vv.fda:/usr/obj/usr/src/sys/GENERIC  amd64
DellT410one# ifconfig -a
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>  
metric 0 mtu 1500
	options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether 84:2b:2b:68:64:e4
	inet 192.168.50.220 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.221 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.223 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.224 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.225 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.226 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.227 netmask 0xffffff00 broadcast 192.168.50.255
	inet 192.168.50.219 netmask 0xffffff00 broadcast 192.168.50.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
bce1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether 84:2b:2b:68:64:e5
	media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
	inet6 ::1 prefixlen 128
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
vboxnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 0a:00:27:00:00:00
DellT410one# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.50.201     UGS         0    52195   bce0
127.0.0.1          link#11            UH          0        6    lo0
192.168.50.0/24    link#1             U           0  1118212   bce0
192.168.50.219     link#1             UHS         0     9670    lo0
192.168.50.220     link#1             UHS         0     8347    lo0
192.168.50.221     link#1             UHS         0   103024    lo0
192.168.50.223     link#1             UHS         0    43614    lo0
192.168.50.224     link#1             UHS         0     8358    lo0
192.168.50.225     link#1             UHS         0     8438    lo0
192.168.50.226     link#1             UHS         0     8338    lo0
192.168.50.227     link#1             UHS         0     8333    lo0
192.168.165.0/24   192.168.50.200     UGS         0     3311   bce0
192.168.166.0/24   192.168.50.200     UGS         0      699   bce0
192.168.167.0/24   192.168.50.200     UGS         0     3012   bce0
192.168.168.0/24   192.168.50.200     UGS         0      552   bce0

Internet6:
Destination                       Gateway                       Flags   
     Netif Expire
::1                               ::1                           UH      
      lo0
fe80::%lo0/64                     link#11                       U       
      lo0
fe80::1%lo0                       link#11                       UHS     
      lo0
ff01::%lo0/32                     fe80::1%lo0                   U       
      lo0
ff02::%lo0/32                     fe80::1%lo0                   U       
      lo0
DellT410one# kldstat
Id Refs Address            Size     Name
  1   19 0xffffffff80100000 dbf5d0   kernel
  2    3 0xffffffff80ec0000 4c358    vboxdrv.ko
  3    1 0xffffffff81012000 131998   zfs.ko
  4    1 0xffffffff81144000 1ff1     opensolaris.ko
  5    2 0xffffffff81146000 2940     vboxnetflt.ko
  6    2 0xffffffff81149000 8e38     netgraph.ko
  7    1 0xffffffff81152000 153c     ng_ether.ko
  8    1 0xffffffff81154000 e70      vboxnetadp.ko
DellT410one# pciconf -lv
..
bce0@pci0:1:0:0:        class=0x020000 card=0x028d1028 chip=0x163b14e4  
rev=0x20 hdr=0x00
     vendor     = 'Broadcom Corporation'
     class      = network
     subclass   = ethernet
bce1@pci0:1:0:1:        class=0x020000 card=0x028d1028 chip=0x163b14e4  
rev=0x20 hdr=0x00
     vendor     = 'Broadcom Corporation'
     class      = network
     subclass   = ethernet

freebsd stable - Jul 2011 - scp: Write Failed: Cannot allocate memory

scp: Write Failed: Cannot allocate memory

scp: Write Failed: Cannot allocate memory

scp: Write Failed: Cannot allocate memory