thr3ads.net - Lustre discuss - [Lustre-discuss] mkfs.lustre (can''t allocate memory) [Jul 2008]

If this information is useful, please help other people find it:
Share via:

Josephine Palencia

2008-Jul-24 17:36 UTC

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

Hi,

Redoing a new filesystem and I''m getting the error below.
Ps advise.

Thanks,
josephin


[root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6

    Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x74
               (MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:

checking for existing Lustre data: not found
device size = 24121MB
2 6 18
formatting backing filesystem ldiskfs on /dev/cciss/c0d0p6
         target name  MGS
         4k blocks     0
         options        -J size=964 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS  -J size=964 -q -O 
dir_index,uninit_groups -F /dev/cciss/c0d0p6
mkfs.lustre: Unable to mount /dev/cciss/c0d0p6: Cannot allocate memory

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 12 (Cannot allocate memory)

-------------------
modules loaded before mkfs:

ldiskfs               254260  0
crc16                   6272  1 ldiskfs
jbd                    62252  2 ldiskfs,ext3
lustre                962512  0
lov                   458540  1 lustre
mdc                   175900  1 lustre
osc                   236980  1 lustre
ptlrpc               1222556  6 lustre,lov,mdc,fid,lquota,osc
obdclass              815900  7 lustre,lov,mdc,fid,lquota,osc,ptlrpc
lnet                  271500  4 lustre,ksocklnd,ptlrpc,obdclass
lvfs                   70488  8 lustre,lov,mdc,fid,lquota,osc,ptlrpc,obdclass
libcfs                131700  11
lustre,lov,mdc,fid,lquota,osc,ksocklnd,ptlrpc,obdclass,lnet,lvfs

------------------------
strace:


--- SIGCHLD (Child exited) @ 0 (0) ---
unlink("/tmp/run_command_logzB7owp")    = 0
mkdir("/tmp/mntiROBOh", 0700)           = 0
mount("/dev/cciss/c0d0p6", "/tmp/mntiROBOh",
"ldiskfs"..., 0, NULL) = -1
ENOMEM (Cannot allocate memory)
write(2, "mkfs.lustre: Unable to mount /de"..., 71mkfs.lustre: Unable
to mount /dev/cciss/c0d0p6: Cannot allocate memory
) = 71
rmdir("/tmp/mntiROBOh")                 = 0
write(2, "\nmkfs.lustre FATAL: ", 20
mkfs.lustre FATAL: )   = 20
write(2, "failed to write local files\n", 28failed to write local
files) = 28
write(1, "\n   Permanent disk data:\nTarget:"..., 635) = -1 EPIPE
(Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
+++ killed by SIGPIPE +++
-----------------------
dmesg:


Lustre: OBD class driver, info at clusterfs.com
         Lustre Version: 1.9.50
         Build Version:
1.9.50-19691231190000-PRISTINE-.extra.linux-2.6.18-92.1.6-2.6.18-92.1.6-lustre-1.9.50
Lustre: Added LNI 128.182.112.70 at tcp [8/256]
Lustre: Accept secure, port 988
LDISKFS-fs: Unable to create cciss/c0d0p6
LDISKFS-fs: Unable to create cciss/c0d0p6

Josephine Palencia

2008-Jul-24 18:51 UTC

head link

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

Redoing a new filesystem and I''m getting the error below.
Ps advise.  Thanks, josephin

[root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6

    Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x74
               (MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:

checking for existing Lustre data: not found
device size = 24121MB
2 6 18
formatting backing filesystem ldiskfs on /dev/cciss/c0d0p6
         target name  MGS
         4k blocks     0
         options        -J size=964 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS  -J size=964 -q -O 
dir_index,uninit_groups -F /dev/cciss/c0d0p6
mkfs.lustre: Unable to mount /dev/cciss/c0d0p6: Cannot allocate memory

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 12 (Cannot allocate memory)

-------------------
modules loaded before mkfs:

ldiskfs               254260  0
crc16                   6272  1 ldiskfs
jbd                    62252  2 ldiskfs,ext3
lustre                962512  0
lov                   458540  1 lustre
mdc                   175900  1 lustre
osc                   236980  1 lustre
ptlrpc               1222556  6 lustre,lov,mdc,fid,lquota,osc
obdclass              815900  7 lustre,lov,mdc,fid,lquota,osc,ptlrpc
lnet                  271500  4 lustre,ksocklnd,ptlrpc,obdclass
lvfs                   70488  8 lustre,lov,mdc,fid,lquota,osc,ptlrpc,obdclass
libcfs                131700  11 
lustre,lov,mdc,fid,lquota,osc,ksocklnd,ptlrpc,obdclass,lnet,lvfs

------------------------
strace:


--- SIGCHLD (Child exited) @ 0 (0) ---
unlink("/tmp/run_command_logzB7owp")    = 0
mkdir("/tmp/mntiROBOh", 0700)           = 0
mount("/dev/cciss/c0d0p6", "/tmp/mntiROBOh",
"ldiskfs"..., 0, NULL) = -1 ENOMEM (Cannot allocate memory)
write(2, "mkfs.lustre: Unable to mount /de"..., 71mkfs.lustre: Unable
to mount /dev/cciss/c0d0p6: Cannot allocate memory
) = 71
rmdir("/tmp/mntiROBOh")                 = 0
write(2, "\nmkfs.lustre FATAL: ", 20
mkfs.lustre FATAL: )   = 20
write(2, "failed to write local files\n", 28failed to write local
files) = 28
write(1, "\n   Permanent disk data:\nTarget:"..., 635) = -1 EPIPE
(Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
+++ killed by SIGPIPE +++
-----------------------
dmesg:


Lustre: OBD class driver, info at clusterfs.com
         Lustre Version: 1.9.50
         Build Version: 
1.9.50-19691231190000-PRISTINE-.extra.linux-2.6.18-92.1.6-2.6.18-92.1.6-lustre-1.9.50
Lustre: Added LNI 128.182.112.70 at tcp [8/256]
Lustre: Accept secure, port 988
LDISKFS-fs: Unable to create cciss/c0d0p6

Eric Mei

2008-Jul-25 22:34 UTC

head link

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

Forward Josephine''s question. Does anybody have an idea on following
issue?

-------- Original Message --------
Subject: mkfs.lustre (can''t allocate memory)
Date: Wed, 23 Jul 2008 22:00:27 -0400 (EDT)
From: Josephine Palencia <josephin at psc.edu>
To: Eric Mei <Eric.Mei at Sun.COM>


Hi Eric,

Redoing a new filesystem and I''m getting this error below..
Ps advice

Thanks,
josephin


[root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6

    Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x74
               (MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:

checking for existing Lustre data: not found
device size = 24121MB
2 6 18
formatting backing filesystem ldiskfs on /dev/cciss/c0d0p6
         target name  MGS
         4k blocks     0
         options        -J size=964 -q -O dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS  -J size=964 -q -O
dir_index,uninit_groups -F /dev/cciss/c0d0p6
mkfs.lustre: Unable to mount /dev/cciss/c0d0p6: Cannot allocate memory

mkfs.lustre FATAL: failed to write local files
mkfs.lustre: exiting with 12 (Cannot allocate memory)
[root at mgs ~]#


[root at mgs ~]# fdisk -l

Disk /dev/cciss/c0d0: 72.8 GB, 72829501440 bytes
255 heads, 63 sectors/track, 8854 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1         131     1052226   83  Linux
/dev/cciss/c0d0p2             132         914     6289447+  82  Linux swap
/ Solaris
/dev/cciss/c0d0p3             915        4562    29302560   83  Linux
/dev/cciss/c0d0p4            4563        8854    34475490    5  Extended
/dev/cciss/c0d0p5            4563        5779     9775521   83  Linux
/dev/cciss/c0d0p6            5780        8854    24699906   83  Linux
[root at mgs ~]# fdisk -l

Disk /dev/cciss/c0d0: 72.8 GB, 72829501440 bytes
255 heads, 63 sectors/track, 8854 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

            Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1   *           1         131     1052226   83  Linux
/dev/cciss/c0d0p2             132         914     6289447+  82  Linux swap
/ Solaris
/dev/cciss/c0d0p3             915        4562    29302560   83  Linux
/dev/cciss/c0d0p4            4563        8854    34475490    5  Extended
/dev/cciss/c0d0p5            4563        5779     9775521   83  Linux
/dev/cciss/c0d0p6            5780        8854    24699906   83  Linux


-- 
Eric

Andreas Dilger

2008-Jul-26 19:04 UTC

head link

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

Josephine Palencia <josephin at psc.edu> wrote:> /dev/cciss/c0d0p6            5780        8854    24699906   83  Linux
>
> mkfs.ext2 -j -b 4096 -L MGS  -J size=964 -q -O
> 		dir_index,uninit_groups -F /dev/cciss/c0d0p6
> mkfs.lustre: Unable to mount /dev/cciss/c0d0p6: Cannot allocate memory
> 
> mkfs.lustre FATAL: failed to write local files
> mkfs.lustre: exiting with 12 (Cannot allocate memory)
This is unusual, to run out of memory.  I would have said it relates
to having too-large a filesystem, but your filesystem is only 24GB.

Looking at the error messages printed, this ENOMEM (-12) error is coming
from the kernel.  Please check "dmesg" to see if the kernel is failing
some memory allocation.  You may also want to try this with full Lustre
debugging enabled (sysctl -w lnet.debug=-1) and then dump the debug log
(lctl dk /tmp/debug) and search for "-12" errors in the logs.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Andreas Dilger

2008-Jul-28 21:22 UTC

head link

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

On Jul 24, 2008  14:51 -0400, Josephine Palencia wrote:> [root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6
> LDISKFS-fs: Unable to create cciss/c0d0p6
This appears to be an internal problem with creating the directory
"/proc/fs/ldiskfs/cciss/c0d0p6" because the "cciss" part of
the
tree does not yet exist and the "c0d0p6" subdirectory create fails.

As a temporary workaround, can you please try modifying the ldiskfs code:

        sbi->s_dev_proc = proc_mkdir(sb->s_id, proc_root_ext3);
        if (sbi->s_dev_proc == NULL) {
                printk(KERN_ERR "EXT3-fs: Unable to create %s\n",
sb->s_id);
+		/* Don''t fail mounting if the /proc file can''t be created
                sb->s_fs_info = NULL;
                kfree(sbi);
                return -ENOMEM;
+		*/
        }


It looks (though I''m not 100% sure) that the rest of the mballoc code
will deal with s_dev_proc == NULL properly.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Josephine Palencia

2008-Jul-30 06:37 UTC

head link

[Lustre-discuss] Works: Re: mkfs.lustre (can''t allocate memory)

Andreas,

This work-around worked.
Thank you very much.

Josephin



On Mon, 28 Jul 2008, Andreas Dilger wrote:
> On Jul 24, 2008  14:51 -0400, Josephine Palencia wrote:
>> [root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6
>> LDISKFS-fs: Unable to create cciss/c0d0p6
>
> This appears to be an internal problem with creating the directory
> "/proc/fs/ldiskfs/cciss/c0d0p6" because the "cciss"
part of the
> tree does not yet exist and the "c0d0p6" subdirectory create
fails.
>
> As a temporary workaround, can you please try modifying the ldiskfs code:
>
>        sbi->s_dev_proc = proc_mkdir(sb->s_id, proc_root_ext3);
>        if (sbi->s_dev_proc == NULL) {
>                printk(KERN_ERR "EXT3-fs: Unable to create %s\n",
sb->s_id);
> +		/* Don''t fail mounting if the /proc file can''t be
created
>                sb->s_fs_info = NULL;
>                kfree(sbi);
>                return -ENOMEM;
> +		*/
>        }
>
>
> It looks (though I''m not 100% sure) that the rest of the mballoc
code
> will deal with s_dev_proc == NULL properly.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>

Andreas Dilger

2008-Jul-30 11:40 UTC

head link

[Lustre-discuss] Works: Re: mkfs.lustre (can''t allocate memory)

On Jul 30, 2008  02:37 -0400, Josephine Palencia wrote:> This work-around worked.
> Thank you very much.
Can you please file a bug with the details.

A better solution is to replace "/" in s_id with "_" when
creating the
proc file so that the full pathname is available.  Alternately, this
could create each path component separately, and clean it up.
> On Mon, 28 Jul 2008, Andreas Dilger wrote:
>
>> On Jul 24, 2008  14:51 -0400, Josephine Palencia wrote:
>>> [root at mgs ~]# mkfs.lustre --mgs /dev/cciss/c0d0p6
>>> LDISKFS-fs: Unable to create cciss/c0d0p6
>>
>> This appears to be an internal problem with creating the directory
>> "/proc/fs/ldiskfs/cciss/c0d0p6" because the "cciss"
part of the
>> tree does not yet exist and the "c0d0p6" subdirectory create
fails.
>>
>> As a temporary workaround, can you please try modifying the ldiskfs
code:
>>
>>        sbi->s_dev_proc = proc_mkdir(sb->s_id, proc_root_ext3);
>>        if (sbi->s_dev_proc == NULL) {
>>                printk(KERN_ERR "EXT3-fs: Unable to create
%s\n", sb->s_id);
>> +		/* Don''t fail mounting if the /proc file can''t be
created
>>                sb->s_fs_info = NULL;
>>                kfree(sbi);
>>                return -ENOMEM;
>> +		*/
>>        }
>>
>>
>> It looks (though I''m not 100% sure) that the rest of the
mballoc code
>> will deal with s_dev_proc == NULL properly.
>>
>> Cheers, Andreas
>> --
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
>>
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Lustre discuss - Jul 2008 - mkfs.lustre (can't allocate memory)

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

[Lustre-discuss] mkfs.lustre (can''t allocate memory)

[Lustre-discuss] Works: Re: mkfs.lustre (can''t allocate memory)

[Lustre-discuss] Works: Re: mkfs.lustre (can''t allocate memory)