Experts, What happens if the disk hosting an external journal of a filesytem running with data=journal goes bust. The Backstory ... I have been batteling with filesystem performance for some time now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems. Recently we added an SSD to our setup and have moved all the journals to this ssd. This has dramatically improved performance and especially reduced the interdependence between performance of different partitions hosted on the same RAID. http://insights.oetiker.ch/linux/external-journal-on-ssd.html I realy like the performance of this new setup, but I am not all that sure about the data security aspects of it. Especially after reading http://www.cs.wisc.edu/adsl/Publications/sfa-dsn05.pdf which suggests that damaged journals are the worst that can happen to ext3. any insights on this? cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900
On Wed, 10 Sep 2008 13:30:45 +0200, Tobias Oetiker wrote:> What happens if the disk hosting an external journal of a filesytem > running with data=journal goes bust.Probably the same as if the journal was on the same disk, going bust. :-) Or rather :-( as this can indeed get pretty ugly. With ext3 you can always fall back to mounting as ext2 and at least try to recover as much as possible.> Recently we added an SSD to our setup and have moved all the journals to > this ssd. This has dramatically improved performance and especially > reduced the interdependence between performance of different partitions > hosted on the same RAID.That is one of the great SSD uses, yes.> http://insights.oetiker.ch/linux/external-journal-on-ssd.htmlVery interesting, thanks! I was planning to do the same but waiting for the Intel SSDs to come to market or the large OZCs to come down in price, whatever happened first..> I realy like the performance of this new setup, but I am not all that sure > about the data security aspects of it. Especially after reading > > http://www.cs.wisc.edu/adsl/Publications/sfa-dsn05.pdf > > which suggests that damaged journals are the worst that can happen to > ext3.True, a borked journal is bad but with the SSD you should actually have *less* chance of corruption (of the type mentioned in the paper), since the wear-leveling should keep the journal blocks alive without the file system/block layer noticing. At least in theory.. :-D You may also find this interesting: http://labs.google.com/papers/disk_failures.html Holger
Tobias Oetiker wrote:> Experts, > > What happens if the disk hosting an external journal of a filesytem > running with data=journal goes bust. > > The Backstory ... > > I have been batteling with filesystem performance for some time > now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems. > > Recently we added an SSD to our setup and have moved all the journals > to this ssd. This has dramatically improved performance and > especially reduced the interdependence between performance of > different partitions hosted on the same RAID. > > http://insights.oetiker.ch/linux/external-journal-on-ssd.htmlHow does this compare to putting journals on a separate non-ssd device? -Eric
Another followup.. On Wed, 10 Sep 2008 13:30:45 +0200, Tobias Oetiker wrote:> Recently we added an SSD to our setup and have moved all the journals to > this ssd. This has dramatically improved performance and especially > reduced the interdependence between performance of different partitions > hosted on the same RAID. > > http://insights.oetiker.ch/linux/external-journal-on-ssd.htmlYou mention that you chose data=journal, i.e. full journaling. Have you tried ordered mode as well? This should still yield a significant performance win because of reduced head movement and faster metadata writes. It may or may not be faster depending on the size of the written data itself..I'm just curious if you tested this. thanks Holger
Hi Chris, Yesterday Chris Worley wrote:> Note that I do have one to experiment with. > What's a good way to measure journal performance, and/or in what cases do > you need a faster journal (i.e. an EXT3 atop an MD device with big block > stripes)? > > ChrisWell, the 'problem' we had to solve was the following: setup: - large HW raid6 array - lvm on top - many ext3 partitions when there was a lot of write or meta data update activity on one partition, performance on all other partitions went to 0. (processes hanging for 10-20 seconds as soon as they accessed the filesystem). I am sure that there is a bad-bad bug in the linux kernel somewhere which is causing this, but all the upgrading and patching did not help, the condition remained. Until we moved the journals off to that external ssd. Now I can copy partition A over to partition B and the server remains nicely responsive. I am atributing that to the external journal. Obviously I would like to know how bad we are going to be had when the ssd dies. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland http://it.oetiker.ch tobi at oetiker.ch ++41 62 775 9902 / sb: -9900