Götz Reinicke - IT-Koordinator
2009-Aug-28 11:03 UTC
[CentOS] Setting up large (12.5 TB) filesystem howto?
Hi, I'm trying to set up an iscsi 12.5 TB storage for some data backup. Doing so, I had some difficulties to find the right tool, maybe it's also a question of the system settings... The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi connection can be establised. fdisk and parted fail to create any information on the device or fail completely. using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly create a logical volume: lvdisplay /dev/VolGroup02/lvol0 --- Logical volume --- LV Name /dev/VolGroup02/lvol0 VG Name VolGroup02 LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x LV Write Access read/write LV Status available # open 0 LV Size 12,73 TB Current LE 3337487 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:4 But, I can't create a filesystem on it: mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 /dev/mapper/VolGroup02-lvol0 mke2fs 1.39 (29-May-2006) mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks (8TB using a blocksize of 4k) are currently supported. The limits information provided by red hat say, that RH EL 5.1 supports 16 TB filesystems: http://www.redhat.com/rhel/compare/ -> Maximum filesystem size (Ext3): 16TB in 5.1 Using a block size of 8192 gives a warning, that this size is to large for that system. So my question: What is my missunderstanding or what's wrong with my system? Where are the real limits? Do I have to switch the OS to 64 Bit? Setting up large Filesystems isn't my staff of life :-) Thanks for amy how to or help of any kind - Best regards, G?tz -- G?tz Reinicke IT-Koordinator Tel. +49 7141 969 420 Fax +49 7141 969 55 420 E-Mail goetz.reinicke at filmakademie.de Filmakademie Baden-W?rttemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzende des Aufsichtsrats: Prof. Dr. Claudia H?bner Staatsr?tin f?r Demographischen Wandel und f?r Senioren im Staatsministerium Gesch?ftsf?hrer: Prof. Thomas Schadt
On Fri, Aug 28, 2009 at 1:03 PM, G?tz Reinicke - IT-Koordinator<goetz.reinicke at filmakademie.de> wrote:> > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. >...snip...> > The limits information provided by red hat say, that RH EL 5.1 supports > 16 TB filesystems: > > http://www.redhat.com/rhel/compare/ > > ? ? ? ?-> Maximum filesystem size (Ext3): 16TB in 5.1 > > Using a block size of 8192 gives a warning, that this size is to large > for that system. > > So my question: What is my missunderstanding or what's wrong with my > system? Where are the real limits? Do I have to switch the OS to 64 Bit? >That was discussed not so long ago. But I can't remember if it was here in the mailing list or on IRC (or somewhere else). But the conclusion was that you are better of using 64 bit and XFS for such large filesystems, and that you need a 64 bit system anyway for ext3/4 to go over 8TB. Well, that is what I remember the conclusion to be :-) Regards, Tim -- Tim Verhoeven - tim.verhoeven.be at gmail.com - 0479 / 88 11 83 Hoping the problem magically goes away by ignoring it is the "microsoft approach to programming" and should never be allowed. (Linus Torvalds)
On Fri, Aug 28, 2009 at 7:03 AM, G?tz Reinicke - IT-Koordinator<goetz.reinicke at filmakademie.de> wrote:> Hi, > > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. > > fdisk and parted fail to create any information on the device or fail > completely. > > using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly > create a logical volume: > > lvdisplay /dev/VolGroup02/lvol0 > ?--- Logical volume --- > ?LV Name ? ? ? ? ? ? ? ?/dev/VolGroup02/lvol0 > ?VG Name ? ? ? ? ? ? ? ?VolGroup02 > ?LV UUID ? ? ? ? ? ? ? ?h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x > ?LV Write Access ? ? ? ?read/write > ?LV Status ? ? ? ? ? ? ?available > ?# open ? ? ? ? ? ? ? ? 0 > ?LV Size ? ? ? ? ? ? ? ?12,73 TB > ?Current LE ? ? ? ? ? ? 3337487 > ?Segments ? ? ? ? ? ? ? 1 > ?Allocation ? ? ? ? ? ? inherit > ?Read ahead sectors ? ? auto > ?- currently set to ? ? 256 > ?Block device ? ? ? ? ? 253:4 > > > But, I can't create a filesystem on it: > > mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 > /dev/mapper/VolGroup02-lvol0You have to smack mkfs around a bit to get it to work. The incantations are listed here, along with the various limitations. The long and short of it is that you have to use -F to tell mkfs that you're really, REALLY sure. http://www.bofh-hunter.com/2008/02/11/large-filesystem-creation/ -- During times of universal deceit, telling the truth becomes a revolutionary act. George Orwell
G?tz Reinicke - IT-Koordinator wrote:> Hi, > > I'm trying to set up an iscsi 12.5 TB storage for some data backup. > > Doing so, I had some difficulties to find the right tool, maybe it's > also a question of the system settings... > > The server is a 32Bit CentOS 5.3 with the recent updates. Ths iscsi > connection can be establised. > > fdisk and parted fail to create any information on the device or fail > completely. > > using the lvm tools (pvcreate, vgcreate, lvcreate), I could finaly > create a logical volume: > > lvdisplay /dev/VolGroup02/lvol0 > --- Logical volume --- > LV Name /dev/VolGroup02/lvol0 > VG Name VolGroup02 > LV UUID h7T6tD-JZw2-UEdb-q1ml-BDqp-9E0u-mAop6x > LV Write Access read/write > LV Status available > # open 0 > LV Size 12,73 TB > Current LE 3337487 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:4 > > > But, I can't create a filesystem on it: > > mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 > /dev/mapper/VolGroup02-lvol0 > > > mke2fs 1.39 (29-May-2006) > mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks > (8TB using a blocksize of 4k) are currently supported. > > > The limits information provided by red hat say, that RH EL 5.1 supports > 16 TB filesystems: > > http://www.redhat.com/rhel/compare/ > > -> Maximum filesystem size (Ext3): 16TB in 5.1 > > Using a block size of 8192 gives a warning, that this size is to large > for that system. > > > So my question: What is my missunderstanding or what's wrong with my > system? Where are the real limits? Do I have to switch the OS to 64 Bit? > > Setting up large Filesystems isn't my staff of life :-) > > > Thanks for amy how to or help of any kind - Best regards, > > G?tz > >Gotz, YMMV, but XFS is what I would use (and do use) as a filesystem in this instance. There are caveats, some of which are listed here: 1) Server/storage must be on stable power, backed up by a ups. I have never dealt with XFS filesystem corruption, but I have read anecdotal horror stories. 2) You need to run 64 bit, if for no other reason than memory requirements. 3) You will need to have a partition that can be used as swap space. I have not found any consistent formula, but I am running multiple servers using XFS, with 12gig of ram, 9TB filesystem, and 12 gigs was not enough memory to run xfs_check, which you should do on occasion. I wound up making a 20gig swap space on a separate partition while I was running xfs_check. That may have been excessively large, but it worked... 4) XFS is not (or has not been) part of the standard RHEL distro, so you lose the 1:1 aspects of running CentOS when you implement XFS. My experience has been that XFS is quite a bit faster than ext3, especially during file manipulation procedures. Others may not agree. It has proven rock-solid for me, both in the linux world and IRIX. Below is a link discussing the need for 64bit vs. 32bit (a bit old but still relevant): http://oss.sgi.com/archives/xfs/2005-08/msg00391.html Hope this helps. Monty
Joshua Baker-LePain
2009-Aug-28 15:36 UTC
[CentOS] Setting up large (12.5 TB) filesystem howto?
On Fri, 28 Aug 2009 at 1:03pm, G?tz Reinicke - IT-Koordinator wrote> fdisk and parted fail to create any information on the device or fail > completely.You can't use fdisk on a volume that large. parted should work fine. What was the error you were getting (exactly)? For a volume that large, you must use a GPT disk label, not the default msdos one.> But, I can't create a filesystem on it: > > mkfs.ext3 -m 2 -j -O dir_index -v -b 4096 -L iscsi2lvol0 > /dev/mapper/VolGroup02-lvol0 > > > mke2fs 1.39 (29-May-2006) > mkfs.ext3: Filesystem too large. No more than 2**31-1 blocks > (8TB using a blocksize of 4k) are currently supported.As has been pointed out, you need to use "-F" to force mkfs.ext3 to make a filesystem bigger than 8TB. IMHO, this is misleading. Filesystems up to 16TB are fully supported in centos >5.1, so I don't see why the upstream vendor left the requirement for "-F" in mkfs.ext3.> So my question: What is my missunderstanding or what's wrong with my > system? Where are the real limits? Do I have to switch the OS to 64 Bit?You do not have to switch to 64bit, and your setup should be fully supported. Other folks have mentioned XFS, and that's an option. But if you want to stay fully compatible with upstream, then ext3 is your only option. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF