(Sorry for cross-posting; I'm not on either ext3-users or linux-xfs, but I thought both lists might find this interesting. CC me with any replies or questions. Thanks.) (The last four paragraphs contain the interesting bits. Basically, XFS hath kick-ed the *ss of ext3 under conditions that are, for our company, critical.) Some listees might be interested in some testing I did the other day, XFS vs. ext3. In our last IMAX film project, right at crunch time, we were getting a whole bunch of dropped frames during compositing. Our compositing program would report "invalid file" partway through writing, and move on to the next frame. A year earlier we had done a very similar project, and stressed the system in very similar ways, but not seen the same problem at all. Difference? Last year we were using XFS, this year we were using ext3. Otherwise, as far as I could tell, the setups were identical. As far as I could tell; film projects differ in subtle ways, and that can have a big impact on how hard the filesystems are stressed. This Sunday I decided to test my hunch. We had to know for sure; if frames were also dropped with XFS under test, or if the test didn't show any dropped frames with either filesystem - in other words, if the problem was a Murphy's Law problem, refusing to show itself until the worst possible moment - we figured we'd be forced to spend a couple of hundred thousand dollars on servers and fabric for our next big IMAX project. The time needed to check for dropped frames and re-render frame-by-frame when a deadline is rushing up - to babysit - is simply too expensive. The test setup: 4 Shake compositing stations running on Win2k, communicating, via Samba, with a 1/2TB Linux server with an IDE software RAID5 setup, over GigE. Shake's job was simply to pump through 48MB Cineon frames from local drives to the server as fast as possible. I ran tests continuously for about 12 hours; I had to be able to guarantee my results. The results were clear and dramatic. Anywhere from 2 to 44 dropped frames out of 200 with ext3. (The worst ext3 numbers came while overwriting already existing files.) Zero dropped frames with XFS. Nadda. None. After the first few clear XFS tests I put extra load on the machine while the tests were running to see if that would make XFS hiccough - copying large files around internally, spawning CPU-eating programs. It didn't. Well... not until I threw a fork bomb at it, anyway. <smirk> But even then, it kept on chuggin' till the load average was somewhere over 900. Conclusion, clear as a bell: XFS for high-bandwidth data transfer over Samba... when running IDE software RAID5, anyway. Oh yeah - another interesting note: There were also dropped frames under ext*2*, which I tried just as a comparison case. XFS truly does, in the patois of the time, r0x0r... Andrew Klaassen
(Sorry for cross-posting; I'm not on either ext3-users or linux-xfs, but I thought both lists might find this interesting. CC me with any replies or questions. Thanks.) (The last four paragraphs contain the interesting bits. Basically, XFS hath kick-ed the *ss of ext3 under conditions that are, for our company, critical.) Some listees might be interested in some testing I did the other day, XFS vs. ext3. In our last IMAX film project, right at crunch time, we were getting a whole bunch of dropped frames during compositing. Our compositing program would report "invalid file" partway through writing, and move on to the next frame. A year earlier we had done a very similar project, and stressed the system in very similar ways, but not seen the same problem at all. Difference? Last year we were using XFS, this year we were using ext3. Otherwise, as far as I could tell, the setups were identical. As far as I could tell; film projects differ in subtle ways, and that can have a big impact on how hard the filesystems are stressed. This Sunday I decided to test my hunch. We had to know for sure; if frames were also dropped with XFS under test, or if the test didn't show any dropped frames with either filesystem - in other words, if the problem was a Murphy's Law problem, refusing to show itself until the worst possible moment - we figured we'd be forced to spend a couple of hundred thousand dollars on servers and fabric for our next big IMAX project. The time needed to check for dropped frames and re-render frame-by-frame when a deadline is rushing up - to babysit - is simply too expensive. The test setup: 4 Shake compositing stations running on Win2k, communicating, via Samba, with a 1/2TB Linux server with an IDE software RAID5 setup, over GigE. Shake's job was simply to pump through 48MB Cineon frames from local drives to the server as fast as possible. I ran tests continuously for about 12 hours; I had to be able to guarantee my results. The results were clear and dramatic. Anywhere from 2 to 44 dropped frames out of 200 with ext3. (The worst ext3 numbers came while overwriting already existing files.) Zero dropped frames with XFS. Nadda. None. After the first few clear XFS tests I put extra load on the machine while the tests were running to see if that would make XFS hiccough - copying large files around internally, spawning CPU-eating programs. It didn't. Well... not until I threw a fork bomb at it, anyway. <smirk> But even then, it kept on chuggin' till the load average was somewhere over 900. Conclusion, clear as a bell: XFS for high-bandwidth data transfer over Samba... when running IDE software RAID5, anyway. Oh yeah - another interesting note: There were also dropped frames under ext*2*, which I tried just as a comparison case. XFS truly does, in the patois of the time, r0x0r... Andrew Klaassen
Veddy Inneresting. Got time to perform a similar test on reiserfs? On Tue, Feb 25, 2003 at 11:43:30PM -0500, Andrew Klaassen wrote:>(Sorry for cross-posting; I'm not on either ext3-users or >linux-xfs, but I thought both lists might find this interesting. >CC me with any replies or questions. Thanks.).... Bill -- INTERNET: bill@Celestial.COM Bill Campbell; Celestial Software LLC UUCP: camco!bill PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676 URL: http://www.celestial.com/ Cutting the space budget really restores my faith in humanity. It eliminates dreams, goals, and ideals and lets us get straight to the business of hate, debauchery, and self-annihilation. -- Johnny Hart
Would you like to elaborate on what these tests were doing, and what exactly a dropped frame is in this context (for those that aren't familiar with this sort of work). I assume you basically have 4 boxes writing lots of data to a server over Samba shares. Are they writing individual files, or components within larger files (ie is (file level) locking involved in this)? Is a dropped frame equivalent to a missing file, or a corrupt file or a hole within a file? Cheers Nigel. -- [ Nigel Metheringham Nigel.Metheringham@InTechnology.co.uk ] [ - Comments in this message are my own and not ITO opinion/policy - ]
Out of curiosity, what was the configuration of the Samba server? How much memory, CPU, disks, etc.? Specifically, was it an SMP machine? - Ted
On Tue, Feb 25, 2003 at 11:43:30PM -0500, Andrew Klaassen wrote:> In our last IMAX film project, right at crunch time, we were > getting a whole bunch of dropped frames during compositing. Our > compositing program would report "invalid file" partway through > writing, and move on to the next frame.I assume you are getting the error on the Windows machine, correct? If that's true, I don't know you can blame ext3. For one thing, you cannot drop a frame or anything due to a particular filesystem unless there is an error; the filesystem code has failed. This should show up as a serious problem or at least a system log entry. Did you check the Linux side of things, in Samba and system logs? I guess I'm trying to figure out how you know it is the filesystem. -- UNIX/Perl/C/Pizza____________________s h a n n o n@wido !SPAM maker.com
Basicali it is normal for XFS to outperform ext2/3 in your case. High badwidth is the main goal in XFS design ( read about it on sgi.com ). Especialy ext3 in ordered mode. XFS can only work in write-back (the fastest) mode.>From all that I am reading about the linux filesystems, and from mypersonal experience, it seems that ext2/3 is the slowest comparing to ReiserFS,JFS,XFS. BUT it is the most robust filesystem that you can find for linux. Try this: ------------- mkfs.ext2 /dev/hd?? about 500-1000 MB mount /dev/hd?? /mnt/???? cp -a /usr/doc /mnt/???? or anything else, just fill it with files umount /mnt/???? dd if=/dev/zero of=/dev/hd?? bs=1024 count=100000 fsck.ext2 -f -y /dev/hd?? mount /dev/hd?? /mnt/???? ------------- Now check what is recovered: Allmost everything, but the blocks that was filled with NULLs. Try this with ReiserFS or XFS :))
Basicali it is normal for XFS to outperform ext2/3 in your case. High badwidth is the main goal in XFS design ( read about it on sgi.com ). Especialy ext3 in ordered mode. XFS can only work in write-back (the fastest) mode.>From all that I am reading about the linux filesystems, and from mypersonal experience, it seems that ext2/3 is the slowest comparing to ReiserFS,JFS,XFS. BUT it is the most robust filesystem that you can find for linux. Try this: ------------- mkfs.ext2 /dev/hd?? about 500-1000 MB mount /dev/hd?? /mnt/???? cp -a /usr/doc /mnt/???? or anything else, just fill it with files umount /mnt/???? dd if=/dev/zero of=/dev/hd?? bs=1024 count=100000 fsck.ext2 -f -y /dev/hd?? mount /dev/hd?? /mnt/???? ------------- Now check what is recovered: Allmost everything, but the blocks that was filled with NULLs. Try this with ReiserFS or XFS :))