Hi Franco-- On Oct 31, Franco Broi wrote:> > I''ve just started looking at lustre. I''ve installed all the bits > (linux-2.4.20-20.9_HEAD.200310101535.tar.gz and > lustre-HEAD-lustre23.tar.gz) and got off to a fairly good start with > llmount, but right away I noticed something strange. Doing ls -l would > produce the error "Operation not supported" (or something like it), and > trying to write to /mnt/lustre would return "read-only filesystem".When this happens, do you see anything in the kernel message log? For future reference, this is perhaps the most important thing to always include, even if it is just a statement like "there were no messages on the console." I have never seen this, but if it is so reliably reproducible then we will quickly get to the bottom of it.> I plan to thrash the disk over the weekend to see how stable it is, and > if all goes well I want to move on to using multple OST''s and a separate > MDS talking to several clients over Myrinet - I''ve seen Myrinet > mentioned but no specific instructions on how to link it in.Our Myrinet support is only just being hardened, and our configuration tools, for example, are not yet able to configure such a system. This is maybe something you can help with (the configuration support and some Myrinet testing results), or we can discuss a small support contract to make you happy.> Silly question, probably the first of many. Do I need a separate MDS for > each luster filesystem? I''m assuming the metadata itself is something > you need to keep very safe....You do not need a separate MDS. Each MDS node can serve multiple file systems -- and yes, keep your metadata safe! I hope that helps-- -Phil
On Wed, 2003-11-05 at 01:27, Phil Schwan wrote:> Hi Franco-- > > On Oct 31, Franco Broi wrote: > > > > I''ve just started looking at lustre. I''ve installed all the bits > > (linux-2.4.20-20.9_HEAD.200310101535.tar.gz and > > lustre-HEAD-lustre23.tar.gz) and got off to a fairly good start with > > llmount, but right away I noticed something strange. Doing ls -l would > > produce the error "Operation not supported" (or something like it), and > > trying to write to /mnt/lustre would return "read-only filesystem". > > When this happens, do you see anything in the kernel message log? For > future reference, this is perhaps the most important thing to always > include, even if it is just a statement like "there were no messages > on the console." > > I have never seen this, but if it is so reliably reproducible then we > will quickly get to the bottom of it.I''ve since moved on to using lustre from cvs (b_devel branch) and the patches for the vanilla 2.4.20 kernel. I tried applying patches to the later 2.4.21 and 2.4.22 kernels but there were far too many rejections.> > > I plan to thrash the disk over the weekend to see how stable it is, and > > if all goes well I want to move on to using multple OST''s and a separate > > MDS talking to several clients over Myrinet - I''ve seen Myrinet > > mentioned but no specific instructions on how to link it in. > > Our Myrinet support is only just being hardened, and our configuration > tools, for example, are not yet able to configure such a system. This > is maybe something you can help with (the configuration support and > some Myrinet testing results), or we can discuss a small support > contract to make you happy.I''d be more than happy to test Lustre with our Myrinet network, which currently consists of 20 dual Xeon clients and 7 dual Athlon storage servers (NAS), each with 2TB RAID disk attached. We are running NFS over Myrinet TCP/IP - doesn''t work too well. We also have a 136 node dual Xeon cluster (100Mbit Ethernet). Not sure if we want to use Lustre directly on the cluster., we currently use some of the Myrinet attached Xeons as MPI head nodes to get data to the slaves which suits our cluster application which is CPU intensive. Just let me know what you want me to try, I''ve currently got a 3TB RAID system, 2 storage servers and 1 or more clients to play with. We will also have some more RAID disk arriving shortly. We also have a small Alpha cluster (8 dual 833MHz) running GFS (5.0) over FC SAN, so I can do some direct comparisons of how the 2 shared filesystems measure up.> > > Silly question, probably the first of many. Do I need a separate MDS for > > each luster filesystem? I''m assuming the metadata itself is something > > you need to keep very safe.... > > You do not need a separate MDS. Each MDS node can serve multiple file > systems -- and yes, keep your metadata safe!Would it make sense to serve the metadata over fast Ethernet with the actual data going via GM? I''ve not used GM directly yet so I don''t know how well it''s suited to handling the metadata traffic, as apposed to the large volume data.> > I hope that helps-- > > -Phil > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss@lists.clusterfs.com > https://lists.clusterfs.com/mailman/listinfo/lustre-discuss
Hi Franco-- On Nov 05, Franco Broi wrote:> > I''ve since moved on to using lustre from cvs (b_devel branch) and the > patches for the vanilla 2.4.20 kernel. I tried applying patches to the > later 2.4.21 and 2.4.22 kernels but there were far too many rejections.I take this to mean that the "operation not supported" errors stopped happening when you upgraded? Problem solved, I suppose. We have already begin to put together a patch for 2.4.22. It will be available soon.> I''d be more than happy to test Lustre with our Myrinet network, which > currently consists of 20 dual Xeon clients and 7 dual Athlon storage > servers (NAS), each with 2TB RAID disk attachedI should have mentioned that we only have a network layer for GM 2.0, which I am told is not yet widely deployed. If you are using GM 1.x, we will unfortunately not be able to directly support your Myrinet hardware at this time. TCP over Myrinet is an option which is probably superior to 100bT, if not ideal. You can export NFS or Samba from Lustre, although this is also not ideal. We expect that most users will use Lustre directly from Linux clients, and use NFS and Samba only to support operating systems which do not have native Lustre clients. Lustre is in heavy use in support of very CPU-intensive applications, most of which stop doing computation during periods of I/O. You may find that you save enough I/O time to more than make up for the small CPU utilization of Lustre.> Would it make sense to serve the metadata over fast Ethernet with the > actual data going via GM? I''ve not used GM directly yet so I don''t know > how well it''s suited to handling the metadata traffic, as apposed to the > large volume data.We have considered these heterogeneous network arrangements, but we do not yet support them. So far there has not been a clear need; both metadata and file I/O performance has been very good on our major platforms of gigabit ethernet and Quadrics Elan 3. If you have GM 2.0, we would love for you to test Lustre on that. Otherwise, I recommend Lustre running on TCP over Myrinet. Thanks-- -Phil
Hi I''ve just started looking at lustre. I''ve installed all the bits (linux-2.4.20-20.9_HEAD.200310101535.tar.gz and lustre-HEAD-lustre23.tar.gz) and got off to a fairly good start with llmount, but right away I noticed something strange. Doing ls -l would produce the error "Operation not supported" (or something like it), and trying to write to /mnt/lustre would return "read-only filesystem". I cleaned up and reran llmount.sh and everything then seemed fine, I could write and do ls -l etc. I then tried using a RAID disk, still using a single machine. The disk mounted OK but again there''s something funny going on. [m1@echo5 ~]# df /data20 Filesystem 1K-blocks Used Available Use% Mounted on local 1511898668 32992 1435065676 1% /data20 [m1@echo5 /data20]# cp -r /usr/src/linux-2.4.20-lust . lots of messages like this: cp: cannot create regular file `./linux-2.4.20-lust/include/asm-arm/arch-cl7500/hardware.h'': Operation not supported cp: cannot create regular file `./linux-2.4.20-lust/include/asm-arm/arch-cl7500/ide.h'': Operation not supported cp: cannot create regular file `./linux-2.4.20-lust/include/asm-arm/arch-cl7500/io.h'': Operation not supported So as before I unmounted and started again, and the problem went away.. Anyway doesn''t seem to be a big problem. I plan to thrash the disk over the weekend to see how stable it is, and if all goes well I want to move on to using multple OST''s and a separate MDS talking to several clients over Myrinet - I''ve seen Myrinet mentioned but no specific instructions on how to link it in. Silly question, probably the first of many. Do I need a separate MDS for each luster filesystem? I''m assuming the metadata itself is something you need to keep very safe.... Thanks.