We are looking at implementing a Linux box running samba in the near future with about 1TB of disk online. The purpose of this box will be for basic file and printer sharing needs. I am doing research on the different journaling file systems avaible in RH 7.3 and up (ext3, reiserFS, and JFS) and was wondering if anyone has had any real world experience with them (mostly reiserFS and JFS) and what you would have to say about them. I am mostly looking for cavets or gotchas pertaining to them. -- ---- Corey Hart Systems/Security Analyst St. Edward's University
May I suggest you XFS ? ext3 and XFS have ACLs while, afaik, JFS and reiserFS do not. but ext3 is not really suited for very large file systems. as for testing I use XFS in production and others too and it seem fast and stable. Simo. On Wed, 2002-12-11 at 22:58, Corey Hart wrote:> We are looking at implementing a Linux box running samba in the near > future with about 1TB of disk online. The purpose of this box will be > for basic file and printer sharing needs. I am doing research on the > different journaling file systems avaible in RH 7.3 and up (ext3, > reiserFS, and JFS) and was wondering if anyone has had any real world > experience with them (mostly reiserFS and JFS) and what you would have > to say about them. I am mostly looking for cavets or gotchas > pertaining to them. > > -- > ---- > Corey Hart > Systems/Security Analyst > St. Edward's University-- Simo Sorce - simo.sorce@xsec.it Xsec s.r.l. via Durando 10 Ed. G - 20158 - Milano tel. +39 02 2399 7130 - fax: +39 02 700 442 399 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/samba/attachments/20021211/1bd05a3c/attachment.bin
> ext3 and XFS have ACLs while, > afaik, JFS and reiserFS do not.A dangerous misconception. The best file system around, ReiserFS, can handle ACLs and EAs just beautifully after you enable the features in the kernel, as I learnt from Mr Deschner from SuSE. If you use SuSE Linux 8.1 look at these downloads:>ftp.suse.com/pub/people/mantel/next/RPM/k_deflt-2.4.19-155.i586.rpm >ftp.suse.com/pub/projects/samba/i386/8.1/*After installing I copied 13.6 GB of an ext3 filesystem and got all the bits tucked in 12.6 GB on reiserfs. What a difference in directory manipulation commands! Reiserfs is screaming fast compared to all other fs's out there. Windoze client SMB/CIFS subsystems are very abusive of these commands, so it may mean a lot of difference in performance. I have yet to take it into production (at the moment ext3 rules because of the same initial error of judgement) but tests so far were very encouraging - oh boy, reiser really kicks butts, especially with log on a separate spindle. I tried IBM's JFS and made the mistake of keeping the log on the same medium - it sucked enormously, so I gave up on it. I mean I might be able to fine-tune it but it looks like a lot of hassle if you can just use reiser. My ? 0.02 (2 EuroCents) of thoughts.
On Wed, Dec 11, 2002 at 03:58:15PM -0600, Corey Hart wrote:> We are looking at implementing a Linux box running samba in the near > future with about 1TB of disk online. The purpose of this box will be > for basic file and printer sharing needs. I am doing research on the > different journaling file systems avaible in RH 7.3 and up (ext3, > reiserFS, and JFS) and was wondering if anyone has had any real world > experience with them (mostly reiserFS and JFS) and what you would have > to say about them. I am mostly looking for cavets or gotchas > pertaining to them.I think the main caveat to JFS is that everything I've seen so far says that it is far slower than any of the other common Linux filesystems. It's still not close enough to being a finished product for me to use it, and I'd guess that you probably won't find many people who are using it already. -- Michael Heironimus
On Wed, 2002-12-11 at 16:58, Corey Hart wrote:> We are looking at implementing a Linux box running samba in the near > future with about 1TB of disk online. The purpose of this box will be > for basic file and printer sharing needs. I am doing research on the > different journaling file systems avaible in RH 7.3 and up (ext3, > reiserFS, and JFS) and was wondering if anyone has had any real world > experience with them (mostly reiserFS and JFS) and what you would have > to say about them.I have 5 production servers all running reiserfs with no problems. Three runs SuSE 7.3, one runs SuSE 8.0 and one runs SuSE 8.1. All systems are single servers in small businesses handling all of the normal chores. I also run reiserfs on my main desktop system, Redhat 8.0, also with no problems. The SuSE 7.3 systems have been running non-stop since the release of that version of the OS.
Have you ever tried ACLs with reiserfs? Any opinion on ACL support in reiserfs? Simo. On Thu, 2002-12-12 at 16:07, Chris Smith wrote:> On Wed, 2002-12-11 at 16:58, Corey Hart wrote: > > We are looking at implementing a Linux box running samba in the near > > future with about 1TB of disk online. The purpose of this box will be > > for basic file and printer sharing needs. I am doing research on the > > different journaling file systems avaible in RH 7.3 and up (ext3, > > reiserFS, and JFS) and was wondering if anyone has had any real world > > experience with them (mostly reiserFS and JFS) and what you would have > > to say about them. > > I have 5 production servers all running reiserfs with no problems. Three > runs SuSE 7.3, one runs SuSE 8.0 and one runs SuSE 8.1. All systems are > single servers in small businesses handling all of the normal chores. I > also run reiserfs on my main desktop system, Redhat 8.0, also with no > problems. The SuSE 7.3 systems have been running non-stop since the > release of that version of the OS.-- Simo Sorce - simo.sorce@xsec.it Xsec s.r.l. via Durando 10 Ed. G - 20158 - Milano tel. +39 02 2399 7130 - fax: +39 02 700 442 399 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/samba/attachments/20021212/bca4561e/attachment.bin
Simo Sorce wrote:> ext3 and XFS have ACLs while, afaik, JFS and reiserFS do not.JFS does have acls, and it even uses the same libs as XFS and ext3. See http://acl.bestbits.at
Sime Sorce wrote:> > ext3 and XFS have ACLs while, > > afaik, JFS and reiserFS do not.JFS does have acls now, even uses the same libs as ext3 and XFS. See http://bestbits.acl.at
On Thu, 2002-12-12 at 16:33, Jim McDonough wrote:> Simo Sorce wrote: > > > ext3 and XFS have ACLs while, afaik, JFS and reiserFS do not. > > JFS does have acls, and it even uses the same libs as XFS and ext3. See > http://acl.bestbits.atwow, very nice. -- Simo Sorce - simo.sorce@xsec.it Xsec s.r.l. via Durando 10 Ed. G - 20158 - Milano tel. +39 02 2399 7130 - fax: +39 02 700 442 399 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/samba/attachments/20021212/0a129d35/attachment.bin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> Message: 18 > Reply-To: <dkrnic@t-online.de> > From: dkrnic@t-online.de (Dragan Krnic) > To: <samba@lists.samba.org> > Cc: <chart@acad.stedwards.edu>, "'Simo Sorce'" <simo.sorce@xsec.it> > Subject: AW: [Samba] File Systems - Which one to use? > Date: Thu, 12 Dec 2002 16:54:48 +0100 > Organization: dkdt > > >>>>> > > ext3 and XFS have ACLs while, >>>>> > > afaik, JFS and reiserFS do not. >>> >>>> > A dangerous misconception. The best file system around, ReiserFS, > > canI would be a bit hesitant to make such sweeping statements without at least qualifying the usage the FS is destined for, and what options the FS was mounted with. Benchmarks of ext3 vs ReiserFS have shown similar performance of the two when mounted with equivalent options.> >>>> > handle ACLs and EAs just beautifully after you enable the features > > in > >>>> > the kernel, >> >>> Very nice to know, I was aware about EAs not of ACLs, are tehy in >>> official kernels? I do not use proprietarized distributions so SuSe >>> and such are not an option I consider.Run Mandrake (8.1 or later for XFS, 9.0 or later for ACLs on ext2/3, all works out-the-box via samba). Mandrake 9.0 also has winbind support in the install (if you need that). Or, run RH, and get 3rd-party kernels from SGI for XFS, (or roll your own).>>> >>> Besides that, if it is in an official kernel have you tested them with >>> samba? Are they Posix compliant? Can you give me some more info on > > them?All of them use the draft posix ACLs.> >>> API? >Merged in-kernel with 2.4.19.> > I'm not SuSE. I only use it. Perhaps it's not for you > (proprietarized and all). > > >>>> > What a difference in directory manipulation commands! Reiserfs is >>>> > screaming fast compared to all other fs's out there. >>>> > >>>> > Windoze client SMB/CIFS subsystems are very abusive of >>>> > these commands, so it may mean a lot of difference in performance. I > > >>>> > have yet to take it into production (at the moment ext3 rules > > because > >>>> > of the same initial error of judgement) but tests so far were very >>>> > encouraging - oh boy, reiser really kicks butts, especially with log >>>> > on a separate spindle.Depends of course on the file sizes ... very big files in sparse directories will have a different effect ...>> >>> >>> Maybe, but I would like to see some test before :-) > > > Of course. Before mkreiserfs I copied about 10 GB from the volume, ext3 > to > Ext3, it took 36 minutes. After mkreiserfs I copied back 13,6 GB in > 50 minutes flat. Do ll on a real big directory, the listing just gushes > forth. Sorry, no official benchmark made. >Even "official benchmarks" are difficult to interpret. I have had issues with ReiserFS2.x, but we really need ACLs, so until some distro supports ACLs on ReiserFS, it's not coming near my file servers (mail/news/web maybe). Buchan - -- |--------------Another happy Mandrake Club member--------------| Buchan Milne Mechanical Engineer, Network Manager Cellphone * Work +27 82 472 2231 * +27 21 8828820x121 Stellenbosch Automotive Engineering http://www.cae.co.za GPG Key http://ranger.dnsalias.com/bgmilne.asc 1024D/60D204A7 2919 E232 5610 A038 87B1 72D6 AC92 BA50 60D2 04A7 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQE9+OsorJK6UGDSBKcRAgFfAJ4lIkMXserhMUKRKrVHV9KBhXx6cQCdHQoC 5IHCi2I4wc40+MQKJQzPFNQ=WEgf -----END PGP SIGNATURE-----
> I have had issues with ReiserFS2.x, > but we really need ACLs, so until > some distro supports ACLs on ReiserFS, > it's not coming near my file > servers (mail/news/web maybe).Thanks for moderation and additional answers to Simo, Buchan. I might be a bit overoptimistic about reiserfs. On the other hand, you are talking about problems with Version 2.x, which is quite old. What is now delivered is 3.6 and the author is bringing 4.0 real soon now. I'm sure there will be pathological cases for reiser too, and it might be less efficient than XFS for some uses where XFS shines. But most of the real problems come from bad tuning or quite unrelated issues (e.g. network). That said, I find reiser the most responsive fs to ls and find kinds of commands. I'll certainly learn more and try to put things in right perspective before I take it into production. I might share the results with the list.> Benchmarks of ext3 vs ReiserFS have > shown similar performance of the two > when mounted with equivalent optionsIn another thread John Terpstra rightly positions reiserfs somewhere between ext2 and ext3, which means ahead of ext3. Benchmarks are like statistics - dont't trust'em unless you forged them yourself. They may be tweaked easily to favour one side over the other. Unfortunately there is no consensus as to the weighting one should assign to different numbers that can be obtained by doing this and/or that. In the end your mileage may vary widely.
On Fri, 13 Dec 2002 16:43:40 John H Terpstra wrote:>On Fri, 13 Dec 2002, Dragan Krnic wrote: >> In another thread John Terpstra rightly positions reiserfs >> somewhere between ext2 and ext3, which means ahead of ext3.>And for waht it is worth, even though I found ReiserFS the poorest >performing, I still use it on my file server. So, go figure, not >everyone needs a Maserati!Have I misquoted you, John? In the thread about an ideal samba server (Tyan, Athlons, IDE RAID - as if I wrote it) you said ext2 is by the most efficient and ext3 the least, with reiserfs somewhere between. Now it's the poorest performing. Have you got some new facts? _____________________________________________________________ Get 25MB, POP3, Spam Filtering with LYCOS MAIL PLUS for $19.95/year. http://login.mail.lycos.com/brandPage.shtml?pageId=plus&ref=lmtplus
> I personally prefer dump|restore pipeline. It never goes > much below the theoretical throughput capacity (about 11.5 > MB/s is what I get) and never any swapping. If I dump to > "/dev/null" than the throughput is way beyond that, > although I can't pull the precise figure off the top of my > head right now (I'll check it up and report in a future > installment if necessary).I'm making good on the promise. Here are the test results. They are not as rigourous as they could be but rather more exhaustive than the usual rant: # bdf /data2 /dev/hda3 on /data2 type ext3 (rw,acl,user_xattr) used 13,669,332 KB, available 40,352,944 KB, 26% full # time dump -0uab 64 -f /dev/null /dev/hda3 real 11m59.216s, user 0m13.670, sys 1m1.280s That's 19,462,020 bytes per second on a 60 GB IBM IC35L060AVER07-0> However, using "tar -b 64" I got the same transfer rates > from reiser to ext3 - the limiting factor is fast ether not > fs. I'll see what I get when I redirect to "/dev/null" and > post it later on.Another promise fulfilled: # cd /local # bdf . /dev/hdb1 on /local type reiserfs (rw,acl,user_xattr) used 12,679,580 KB, available 16,614,016 KB, 44% full # time tar -cb 64 -f /dev/null . real 3m14.635s, user 0m4.320s, sys 0m16.600s That's 66,708,916 bytes per second on a 40 GB Maxtor 53073H6. Let's recheck the ext3 with tar: # cd /data2 # bdf . /dev/hda3 on /data2 type ext3 (rw,acl,user_xattr) used 13,669,332 KB, available 40,352,944, KB 26% full # time dump -0uab 64 -f /dev/null /dev/hda3 real 9m55.023s, user 0m4.480, sys 0m15.860s in other words 23,524,126 bytes per second. So it must be the fs that is unnecessarily inefficient. Both IDE disks being rated at 7.2 Krpm, it is a surprising comparison for me too. Just in order to keep things in perspective, here's another dump of an 18 GB SCSI/LVD disk, an IBM DNES-318350W, also 7.2 Krpm, on the same system: # bdf / /dev/sda3 on / type ext3 (rw,data=ordered,acl) used 3,697,568 KB, available 12,611,624 KB, 23% full # time dump -0uab 64 -f /dev/null /dev/sda3 real 7m45,027, user 0m4.650s, sys 0m19.510s In terms of speed paltry 7,211,647 bytes per second. (I must keep the SCSI disk for the system because the IDE DMA part of the i845 chipset freezes the system with a kernel panic every now and then reliably.) It's a further proof that there's something funny about ext3, because if we do a serial read test, we get: # time dd if=/dev/hda3 bs=1024k count=1024 of=/dev/null real 0m33.346s, user 0m0.000s, sys 0m4.460s 32,200,019 bytes per second for IBM IDE (ext3) # time dd if=/dev/hdb1 bs=1024k count=1024 of=/dev/null real 0m45.818s, user 0m0.000s, sys 0m4.170s 23,434,934 bytes per second for Maxtor (reiser) # time dd if=/dev/sda3 bs=1024k count=1024 of=/dev/null real 0m56.250s, user 0m0.010s, sys 0m4.460s 19,088,744 bytes per second for SCSI/LVD IBM (ext3) It's getting curiouser and curiouser! The IBM's IDE disk with ext3 is in fact almost 50% faster than the Maxtor's IDE disk and still it's outperformed by it by a factor of 2.8 using tar on both systems. To really evaluate this data I should turn it around and put reiser on IBM's IDE and ext3 on the Maxtor's disk. Since I'm too lazy I'll just unfairly extrapolate the figures: in that case ext3 should deliver 17,120,684 bytes per second and reiser would beat it by a factor of 5.35 (assuming the IBM's IDE can indeed yield the projected 91,659,245 bytes per second). The IBM's SCSI disk is clearly a lousy oldie - I've had much better experiences with 15 Krpm disks. I've been worried like hell that there might have been an error in the measurements. Especially the tar on reiser was suspect - I mean how can you tar so much faster than you can dd? Simple! The dd utility needs a face-lift. It performs miserably for other people who use it to reblock data too. The next suspicion was that tar on reiser might be skipping something, so I added a pipe to "wc" which showed that tar output 13,100,711,936 bytes, about 3% more than shown by "bdf", not enough to spoil the computation (tar has 512 byte headers, reiser squeezes file tails and is generally more parsimonious in resource wasting than any other system I know). What surprised me was that the pipeline took 23m41.319s (user 2m3.730s, sys 1m24,52s). On more mature comercial *nices this pipe hardly ever makes any difference. Shouldn't someone look into it?> Now, the lack of a decent "dump/restore" for reiser is > another sore fact that might stand in the way of faster > acceptance of this fine fs. I'm all sold on the paradigm > and believe that the authors will make it even better > (read faster) in the coming version 4.Dump didn't exactly pass with flying colors in this test. It was a lot slower (20%) accessing the raw volume than tar which jumped through all the fs hoops of a blocked device. I'll take a closer look at GPL/dump. The results are too paradoxical in my view. My own vxdump for Veritas vxfs (sorry not yet GPLed, not freely available but an excellent alternative for seriously minded - I mean the vxfs - although with reiserfs 4.0 this might change) easily dumps in excess of 50 MB/s to an LTO tape from 80 MB/s SCSI/LVD software striped mirrored disks. I wish more people would pay closer attention to measurable differences in disks and fs's. The tests took what, 35 minutes, annotating the results another hour - everyone can sacrifice so much for computer sciences. I invite everyone to check my timings in the fine tradition of peer review. _____________________________________________________________ Get 25MB, POP3, Spam Filtering with LYCOS MAIL PLUS for $19.95/year. http://login.mail.lycos.com/brandPage.shtml?pageId=plus&ref=lmtplus