On Jul 15, 2005 11:37 +0200, Christian Boehme wrote:> Andreas Dilger schrieb: > >You currently need to stop the whole filesystem, run on the MDS (when > >stopped) "lconf --write-conf config_lustre.xml" and then you can restart. > > OK, I didn''t do that. Could that have caused the problems? Also, the > order of things to do is not clear to me. I assume it should be: > > lconf -d --node client config_lustre.xml #on the clients > lconf -d --force --node gwdu105 config_lustre.xml #on the MDS, gwdu105 > sh config_lustre.sh #after changing > config_lustre.sh, to get new config_lustre.xml > lconf --write_conf config_lustre.xml #on the MDS, gwdu105presumably "lconf -node gwdo002 config_lustre.xml" # on the new OST> lconf --node gwdu105 config_lustre.xml # " > lconf --node client config_lustre.xml #on the clients > > Is that correct?Yes.> >If you revert the config to the old one (without having created new files > >that use the new OST) does the corruption go away? If you run on one or > >two corrupted files "lfs getstripe {filename}" does it show any difference > >before/after adding the new OST? > > The corruption does not go away, and consequently the lfs output is the > same, except for the OBDS-list in the beginning.Ah, the OBDs list is the critical part. Does the new OST appear at the end, and the other OSTs remain in the same order? If something is going wrong, then it is possible the OST is not appearing at the end. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Christian Boehme
2006-May-19 07:36 UTC
[Lustre-discuss] File corruption when adding new OSS
Hello Andreas, Andreas Dilger schrieb:>>lconf -d --node client config_lustre.xml #on the clients >>lconf -d --force --node gwdu105 config_lustre.xml #on the MDS, gwdu105 >>sh config_lustre.sh #after changing >>config_lustre.sh, to get new config_lustre.xml >>lconf --write_conf config_lustre.xml #on the MDS, gwdu105 >> >> > >presumably "lconf -node gwdo002 config_lustre.xml" # on the new OST > > > >>lconf --node gwdu105 config_lustre.xml # " >>lconf --node client config_lustre.xml #on the clients >> >> >>OK, I followed this procedure now, starting with new testfiles. Again, some files became corrupted.>Ah, the OBDs list is the critical part. Does the new OST appear at the end, >and the other OSTs remain in the same order? If something is going wrong, >then it is possible the OST is not appearing at the end. > >The OBD''s remain in the same order, with the new one added after the old. Anything else to blame for the file corruption? Thanks and best wishes Christian
On Jul 14, 2005 14:59 +0200, Christian Boehme wrote:> As a test I tried to add a new OSS (and therefore a new OST) to an > existing lustre file-system. This is how I proceeded: The original > configuration shell script was: > > # Configures OSTs > lmc -m config_lustre.xml --add lov --lov lov-gwdu105 --mds gwdu105 > --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0 > lmc -m config_lustre.xml --add ost --node gwdo201 --lov lov-gwdu105 > --ost ost1-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo202 --lov lov-gwdu105 > --ost ost2-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo203 --lov lov-gwdu105 > --ost ost3-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo204 --lov lov-gwdu105 > --ost ost4-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo205 --lov lov-gwdu105 > --ost ost5-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo206 --lov lov-gwdu105 > --ost ost6-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo207 --lov lov-gwdu105 > --ost ost7-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo208 --lov lov-gwdu105 > --ost ost8-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo001 --lov lov-gwdu105 > --ost ost9-gwdo001 --fstype ldiskfs --dev /scratch/ost9-gwdo001 --size > 10000000 > > I modified it to add the new OSS and OST: > > # Configures OSTs > lmc -m config_lustre.xml --add lov --lov lov-gwdu105 --mds gwdu105 > --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0 > lmc -m config_lustre.xml --add ost --node gwdo201 --lov lov-gwdu105 > --ost ost1-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo202 --lov lov-gwdu105 > --ost ost2-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo203 --lov lov-gwdu105 > --ost ost3-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo204 --lov lov-gwdu105 > --ost ost4-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo205 --lov lov-gwdu105 > --ost ost5-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo206 --lov lov-gwdu105 > --ost ost6-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo207 --lov lov-gwdu105 > --ost ost7-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo208 --lov lov-gwdu105 > --ost ost8-gwdu105 --fstype ldiskfs --dev /dev/dm-1 > lmc -m config_lustre.xml --add ost --node gwdo001 --lov lov-gwdu105 > --ost ost9-gwdo001 --fstype ldiskfs --dev /scratch/ost9-gwdo001 --size > 10000000 > # New OST: > lmc -m config_lustre.xml --add ost --node gwdo002 --lov lov-gwdu105 > --ost osta-gwdo002 --fstype ldiskfs --dev /scratch/osta-gwdo002 --size > 10000000 > > After creating the new config_lustre.xml I did > > lconf -d --node client config_lustre.xml > > and > > lconf --node client config_lustre.xmlYou currently need to stop the whole filesystem, run on the MDS (when stopped) "lconf --write-conf config_lustre.xml" and then you can restart. You correctly added the new OST after all the other OSTs, so that isn''t the problem.> After this df on the lustre filesystem showed the > correct new size on all client nodes, albeit the percentage of used > space did not change, i.e., the used space increased size correspondingly.If the filesystem is virtually empty, then the used space only reflects the size of the ext3 journals.> What''s more important is that the md5sums of the test > files I created showed quite a few files to be corrupted in the process.Adding a new OST shouldn''t affect the existing files. If you revert the config to the old one (without having created new files that use the new OST) does the corruption go away? If you run on one or two corrupted files "lfs getstripe {filename}" does it show any difference before/after adding the new OST? Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Christian Boehme
2006-May-19 07:36 UTC
[Lustre-discuss] File corruption when adding new OSS
Hello! Andreas Dilger schrieb:>You currently need to stop the whole filesystem, run on the MDS (when >stopped) "lconf --write-conf config_lustre.xml" and then you can restart. > > >OK, I didn''t do that. Could that have caused the problems? Also, the order of things to do is not clear to me. I assume it should be: lconf -d --node client config_lustre.xml #on the clients lconf -d --force --node gwdu105 config_lustre.xml #on the MDS, gwdu105 sh config_lustre.sh #after changing config_lustre.sh, to get new config_lustre.xml lconf --write_conf config_lustre.xml #on the MDS, gwdu105 lconf --node gwdu105 config_lustre.xml # " lconf --node client config_lustre.xml #on the clients Is that correct?>If the filesystem is virtually empty, then the used space only reflects the >size of the ext3 journals. > > >That makes sense :)>If you revert the config to the old one (without having created new files >that use the new OST) does the corruption go away? If you run on one or >two corrupted files "lfs getstripe {filename}" does it show any difference >before/after adding the new OST? > > >The corruption does not go away, and consequently the lfs output is the same, except for the OBDS-list in the beginning. Thanks for your support! Best wishes Christian
Christian Boehme
2006-May-19 07:36 UTC
[Lustre-discuss] File corruption when adding new OSS
Hi all! I am using version 1.2.6 of Lustre, as included in the (patched) SuSE SLES 9. As a test I tried to add a new OSS (and therefore a new OST) to an existing lustre file-system. This is how I proceeded: The original configuration shell script was: --- #!/bin/sh # Create nodes rm -f config_lustre.xml lmc -m config_lustre.xml --add net --node gwdu105 --nid gwdu105 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo201 --nid gwdo201 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo202 --nid gwdo202 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo203 --nid gwdo203 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo204 --nid gwdo204 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo205 --nid gwdo205 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo206 --nid gwdo206 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo207 --nid gwdo207 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo208 --nid gwdo208 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo001 --nid gwdo001 --nettype tcp lmc -m config_lustre.xml --add net --node client --nid ''*'' --nettype tcp # Cofigure MDS lmc -m config_lustre.xml --add mds --node gwdu105 --mds gwdu105 --fstype ldiskfs --dev /work/mds_lustre --size 2000000 # Configures OSTs lmc -m config_lustre.xml --add lov --lov lov-gwdu105 --mds gwdu105 --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0 lmc -m config_lustre.xml --add ost --node gwdo201 --lov lov-gwdu105 --ost ost1-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo202 --lov lov-gwdu105 --ost ost2-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo203 --lov lov-gwdu105 --ost ost3-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo204 --lov lov-gwdu105 --ost ost4-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo205 --lov lov-gwdu105 --ost ost5-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo206 --lov lov-gwdu105 --ost ost6-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo207 --lov lov-gwdu105 --ost ost7-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo208 --lov lov-gwdu105 --ost ost8-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo001 --lov lov-gwdu105 --ost ost9-gwdo001 --fstype ldiskfs --dev /scratch/ost9-gwdo001 --size 10000000 # Configure client (this is a ''generic'' client used for all client mounts) lmc -m config_lustre.xml --add mtpt --node client --path /work2 --mds gwdu105 --lov lov-gwdu105 --- I modified it to add the new OSS and OST: --- #!/bin/sh # Create nodes rm -f config_lustre.xml lmc -m config_lustre.xml --add net --node gwdu105 --nid gwdu105 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo201 --nid gwdo201 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo202 --nid gwdo202 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo203 --nid gwdo203 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo204 --nid gwdo204 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo205 --nid gwdo205 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo206 --nid gwdo206 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo207 --nid gwdo207 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo208 --nid gwdo208 --nettype tcp lmc -m config_lustre.xml --add net --node gwdo001 --nid gwdo001 --nettype tcp # New OSS: lmc -m config_lustre.xml --add net --node gwdo002 --nid gwdo002 --nettype tcp lmc -m config_lustre.xml --add net --node client --nid ''*'' --nettype tcp # Cofigure MDS lmc -m config_lustre.xml --add mds --node gwdu105 --mds gwdu105 --fstype ldiskfs --dev /work/mds_lustre --size 2000000 # Configures OSTs lmc -m config_lustre.xml --add lov --lov lov-gwdu105 --mds gwdu105 --stripe_sz 1048576 --stripe_cnt 0 --stripe_pattern 0 lmc -m config_lustre.xml --add ost --node gwdo201 --lov lov-gwdu105 --ost ost1-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo202 --lov lov-gwdu105 --ost ost2-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo203 --lov lov-gwdu105 --ost ost3-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo204 --lov lov-gwdu105 --ost ost4-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo205 --lov lov-gwdu105 --ost ost5-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo206 --lov lov-gwdu105 --ost ost6-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo207 --lov lov-gwdu105 --ost ost7-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo208 --lov lov-gwdu105 --ost ost8-gwdu105 --fstype ldiskfs --dev /dev/dm-1 lmc -m config_lustre.xml --add ost --node gwdo001 --lov lov-gwdu105 --ost ost9-gwdo001 --fstype ldiskfs --dev /scratch/ost9-gwdo001 --size 10000000 # New OST: lmc -m config_lustre.xml --add ost --node gwdo002 --lov lov-gwdu105 --ost osta-gwdo002 --fstype ldiskfs --dev /scratch/osta-gwdo002 --size 10000000 # Configure client (this is a ''generic'' client used for all client mounts) lmc -m config_lustre.xml --add mtpt --node client --path /work2 --mds gwdu105 --lov lov-gwdu105 --- After creating the new config_lustre.xml I did lconf -d --node client config_lustre.xml and lconf --node client config_lustre.xml on all clients. After this df on the lustre filesystem showed the correct new size on all client nodes, albeit the percentage of used space did not change, i.e., the used space increased size correspondingly. What''s more important is that the md5sums of the test files I created showed quite a few files to be corrupted in the process. What went wrong? Thanks for your support. Best wishes Christian Boehme
Christian Boehme wrote:> Hello! > > Christian Boehme schrieb: > >> >>> Ah, the OBDs list is the critical part. Does the new OST appear at >>> the end, >>> and the other OSTs remain in the same order? If something is going >>> wrong, >>> then it is possible the OST is not appearing at the end. >>> >>> >> The OBD''s remain in the same order, with the new one added after the >> old. Anything else to blame for the file corruption? Thanks and best >> wishes > > > > I did not recieve any answers to this, so I guess the file corruption > cannot be easily explained. Is the possiblity to add an OSS to an > existing FS in development or untested? Can I assume that it would work > with a newer version, or is it rather a configuration problem on my > side? I found out that in all cases of corrupt files the files are > changed at the end, which is in some cases overwritten by an earlier > part, sometimes overwritten with garbage. For one corrupted file this is > the lfs getstripe output:[snip] Christian, I have been attempting to reproduce your situation here, with Lustre 1.4.2 - so far I have not seen any corruption. How full is the filesystem when you add the new OST? I created a filesystem, copied files to it, and collected checksums on the files. Then I added an OST, checksum''d all the files again, and compared. Here are the steps i took and some additional data. Stop Lustre ( first clients, then mds, then ods ) lconf --cleanup --node client add_test.xml lconf --cleanup --node netone add_test.xml lconf --cleanup --node tinyone add_test.xml Add new OST at the end of the OST list, re-generate the XML. Re-load the configuration on the MDS. lconf --write_conf --node netone add_test.xml Restart Lustre ( first ods, then mds, then clients ) lconf --node tinyone add_test.xml lconf --node netone add_test.xml lconf --node client add_test.xml ''df'' before the addition: Filesystem 1K-blocks Used Available Use% Mounted on add_test 22287840 9704900 11456732 46% /mnt/lustre ''df'' after: Filesystem 1K-blocks Used Available Use% Mounted on add_test 29674976 9754384 18419132 35% /mnt/lustre sample file before: SHA1 checksum: 7bd6cc1e88e8c5d557cbfd34fc364a234c5a8c5b ./FC2/Fedora/RPMS/zsh-html-4.2.0-1.i386.rpm lfs getstripe: OBDS: 0: ost1-test_UUID 1: ost2-test_UUID 2: ost3-test_UUID /mnt/lustre/old/archive/FC2/Fedora/RPMS/zsh-html-4.2.0-1.i386.rpm obdidx objid objid group 1 47724 0xba6c 0 2 5052 0x13bc 0 0 47724 0xba6c 0 After: SHA1 checksum: 7bd6cc1e88e8c5d557cbfd34fc364a234c5a8c5b ./FC2/Fedora/RPMS/zsh-html-4.2.0-1.i386.rpm lfs getstripe /mnt/lustre/old/archive/FC2/Fedora/RPMS/zsh-html-4.2.0-1.i386.rpm OBDS: 0: ost1-test_UUID 1: ost2-test_UUID 2: ost3-test_UUID 3: ost4-test_UUID /mnt/lustre/old/archive/FC2/Fedora/RPMS/zsh-html-4.2.0-1.i386.rpm obdidx objid objid group 1 47724 0xba6c 0 2 5052 0x13bc 0 0 47724 0xba6c 0 ------------ cliffw> Many thanks and best regards > > Christian Boehme > > >
Christian Boehme
2006-May-19 07:36 UTC
[Lustre-discuss] File corruption when adding new OSS
Hello! Christian Boehme schrieb:> >> Ah, the OBDs list is the critical part. Does the new OST appear at >> the end, >> and the other OSTs remain in the same order? If something is going >> wrong, >> then it is possible the OST is not appearing at the end. >> >> > The OBD''s remain in the same order, with the new one added after the > old. Anything else to blame for the file corruption? Thanks and best > wishesI did not recieve any answers to this, so I guess the file corruption cannot be easily explained. Is the possiblity to add an OSS to an existing FS in development or untested? Can I assume that it would work with a newer version, or is it rather a configuration problem on my side? I found out that in all cases of corrupt files the files are changed at the end, which is in some cases overwritten by an earlier part, sometimes overwritten with garbage. For one corrupted file this is the lfs getstripe output: Before addition: OBDS: 0: ost1-gwdu105_UUID 1: ost2-gwdu105_UUID 2: ost3-gwdu105_UUID 3: ost4-gwdu105_UUID 4: ost5-gwdu105_UUID 5: ost6-gwdu105_UUID 6: ost7-gwdu105_UUID 7: ost8-gwdu105_UUID 8: ost9-gwdo001_UUID /work2/cboehme1/testf.gwdo008 obdidx objid objid group 6 6448 0x1930 0 7 6448 0x1930 0 8 86 0x56 0 0 6448 0x1930 0 1 6448 0x1930 0 2 6448 0x1930 0 3 6410 0x190a 0 4 6448 0x1930 0 5 6448 0x1930 0 After addition: OBDS: 0: ost1-gwdu105_UUID 1: ost2-gwdu105_UUID 2: ost3-gwdu105_UUID 3: ost4-gwdu105_UUID 4: ost5-gwdu105_UUID 5: ost6-gwdu105_UUID 6: ost7-gwdu105_UUID 7: ost8-gwdu105_UUID 8: ost9-gwdo001_UUID 9: osta-gwdo002_UUID /work2/cboehme1/testf.gwdo008 obdidx objid objid group 6 6448 0x1930 0 7 6448 0x1930 0 8 86 0x56 0 0 6448 0x1930 0 1 6448 0x1930 0 2 6448 0x1930 0 3 6410 0x190a 0 4 6448 0x1930 0 5 6448 0x1930 0 Many thanks and best regards Christian Boehme