I have an ext3 filesystem mounted r/w on 1 host and r/o on multiple hosts. Dangerous but cost effective. I recently implemented some protection through a fc switch that restricts some hosts to r/o access to the data luns. So if someone types mount -o rw or something, all is not lost. The issue occurs when it's mounted r/w on 1 host and another host attempts to mount it r/o. The mount command takes about a minute to complete, it successfully mounts, and several error messages are reported... Nov 3 12:52:26 lax kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Nov 3 12:52:26 lax kernel: EXT3-fs: write access will be enabled during recovery. Nov 3 12:52:27 lax kernel: cfq: depth 4 reached, tagging now on ...reports this for about 260 different sectors (makes sense, fc switch is preventing write access)... Nov 3 12:52:27 lax kernel: SCSI error : <494 0 0 1> return code 0x8000002 Nov 3 12:52:27 lax kernel: sdl: Current: sense key: Data Protect Nov 3 12:52:27 lax kernel: Additional sense: Logical unit software write protected Nov 3 12:52:27 lax kernel: end_request: I/O error, dev sdl, sector 496 Nov 3 12:52:27 lax kernel: Buffer I/O error on device sdl, logical block 62 Nov 3 12:52:27 lax kernel: lost page write due to I/O error on sdl then completes... Nov 3 12:52:44 laxl kernel: EXT3-fs: recovery complete. (how???) Nov 3 12:52:44 laxl kernel: EXT3-fs: mounted filesystem with ordered data mode. This also happens on other filesystems and other devices under the same circumstances. When the filesystem is umounted from the r/w host, it mounts w/ out error on r/o host. It's interesting to note that after that's done, you can remount the filesystem on the r/w host, and then mount it on the r/o w/ just a few errors and w/ in seconds. My questions are... Should I be concerned by this? Is there a way to automatically skip the recovery attempt, and if so, should I use it? Am I going about this all wrong, is there a better way to do this (other than GFS)? Thanks. - Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/ext3-users/attachments/20051103/9d2c42ce/attachment.htm>
On Thu, 3 Nov 2005, Jeff Dinisco wrote:> I have an ext3 filesystem mounted r/w on 1 host and r/o on multiple > hosts. Dangerous but cost effective. > > My questions are... > Should I be concerned by this? > Is there a way to automatically skip the recovery attempt, and if so, > should I use it? > Am I going about this all wrong, is there a better way to do this (other > than GFS)?Sorry to ask the obvious question, but why not just use NFS? Damian Menscher -- -=#| <menscher at uiuc.edu> www.uiuc.edu/~menscher/ Ofc:(650)273-2757 |#=- -=#| The above opinions are not necessarily those of my employers. |#=-
Performance is the answer. This is streaming media and the throughput is very high. -----Original Message----- From: Wolber, Richard C [mailto:richard.c.wolber at boeing.com] Sent: Thursday, November 03, 2005 5:01 PM To: Damian Menscher; Jeff Dinisco Cc: ext3-users at redhat.com Subject: RE: mount r/w and r/o> > My questions are... > > Should I be concerned by this? > > Is there a way to automatically skip the recovery attempt, and ifso,> > should I use it? > > Am I going about this all wrong, is there a better way to do this > > (other than GFS)? > > Sorry to ask the obvious question, but why not just use NFS?Performance? NFS is a lot of overhead to consider using on something like FC. Mounting r/o seems (and I await the experts opinion) at first glance to be a very effictive way of doing this. ..Chuck..
> > My questions are... > > Should I be concerned by this? > > Is there a way to automatically skip the recovery attempt, and ifso,> > should I use it? > > Am I going about this all wrong, is there a better way to do this > > (other than GFS)? > > Sorry to ask the obvious question, but why not just use NFS?Performance? NFS is a lot of overhead to consider using on something like FC. Mounting r/o seems (and I await the experts opinion) at first glance to be a very effictive way of doing this. ..Chuck..
On Nov 03, 2005 13:18 -0800, Jeff Dinisco wrote:> I have an ext3 filesystem mounted r/w on 1 host and r/o on multiple > hosts. Dangerous but cost effective. I recently implemented some > protection through a fc switch that restricts some hosts to r/o access > to the data luns. So if someone types mount -o rw or something, all is > not lost.This is completely dangerous and should not be done. The FC switch is preventing potentially serious corruption to your filesystem, but is not preventing the r/o clients from getting corrupt/stale data and possibly crashing. There is nothing on those clients to keep their cache up-to-date with what is happening on the r/w server.> Is there a way to automatically skip the recovery attempt, and if so, > should I use it?No.> Am I going about this all wrong, is there a better way to do this (other > than GFS)?As another person suggested, NFS is fine for small-scale usage like this. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
Thanks for the reply. Very interesting. Could you explain how the bsd box read the raw device and built the internal lookup table? The main reason I wrote "not GFS" is because I'm aware of it and that it would take a bit of work to implement. I'm currently looking for a quick fix to give me some time to implement a more robust solution. Also, realizing I had some definite issues w/ my current config, I researched GFS a little while back. It's my understanding that total storage in a GFS cluster cannot exceed 8TB and we have > 12TB. I didn't investigate too much further for a work-around. Andreas suggested lustre which on the surface appears to be viable. -----Original Message----- From: J?r?me Petazzoni [mailto:jp at enix.org] Sent: Friday, November 04, 2005 12:11 PM To: Jeff Dinisco Cc: Wolber, Richard C; Damian Menscher; ext3-users at redhat.com Subject: Re: mount r/w and r/o [one r-w mount, multiple r-o mounts shared thru FC switch]>>>should I use it? >>>Am I going about this all wrong, is there a better way to do this >>>(other than GFS)? >>> >>>I once heard about someone doing something like that for a video farm, intermixing solaris and freebsd servers (so as far as he, and I, knew, there was no easy sharing solution). He did the following : - create the filesystem on the solaris bow - create many 1 GB files, with a specific byte pattern (512 bytes sectors iirc) - the freebsd box would read the raw device, detect the byte patterns and build an internal lookup table, to know that file F, offset O was located on physical sector S - the solaris box would then write data to the 1 GB files, and the freebsd box could then read back the data, thanks to the previously built lookup table (the 1 GB files would only be rewritten to, never truncated or rewritten, AFAIK) IIRC, there was 2 solaris boxen using some HA solution, and many freebsd boxen accessing the data. This worked because the files were smaller than 1 GB (to be honnest, I don't know the exact size he used), and the very impressive performance of the solution balanced the hassle involved in setting up the whole thing. Now, I would not ask "why not NFS?", but "why not GFS?" (and please apologize if it the answer is obvious...)