Ocfs2 users - Mar 2013 - Significant Slowdown when writing and deleting files at the same time

I have been having performance issues from time to time on our 
production ocfs2 volumes, so I set up a test system to try to reproduce 
what I was seeing on the production systems.  This is what I found out:

I have a 2 node test system sharing a 2TB volume with a journal size of 
256MB.  I can easily trigger the slowdown by starting to processes to 
write a 10GB file each, then I delete a different large file (7GB+) 
while the other processes are writing.  The slowdown is significant and 
very disruptive.  Not only did it take over 3 minutes to delete the 
file, every else with pause when entering that directory too.  A du 
command with stop and nfs access to that file system will think the 
server is not responding.  Under heavier amounts of writes, I have had a 
delete takes 13mins for a 8GB file, and NFS mounts return I/O errors. 
We often deal with large files, so this situation above is fairly common.

I would like any ideas that would provide smoother performance of the 
OCFS2 volume and somehow eliminate the long pauses during deletes.

Thanks,

Andy

> I have a 2 node test system sharing a 2TB volume with a
> journal size of 256MB.
> I can easily trigger the slowdown by starting to processes to
> write a 10GB file each, then I delete a different large file
> (7GB+) while the other processes are writing.  The slowdown is
> significant and very disruptive.
> Not only did it take over 3 minutes to delete the file, every
> else with pause when entering that directory too. [ ... ]
Thanks for confirming that the performance of OCFS2 is quite
good even under such demanding conditions.

Data structures with mandatory internal locking with high
latency are very difficult to parallelize, and OCFS2 as you
report does a pretty good job of that despite the contention of
three metadata intensive activities.

[ ... ]
> I would like any ideas that would provide smoother performance
The performance is pretty good and smooth, as both the speed and
latencies are about as good as it can get under such conditions.
> of the OCFS2 volume and somehow eliminate the long pauses
> during deletes.
Fund a large research programme to figure out whether it is
feasible to have large overlapping transactions without
interference in a filesystem with mandatory internal locking
with fairly high latency.

Are you mounting -o writeback?


On Fri, Mar 29, 2013 at 12:28 PM, Andy <aryan1 at allantgroup.com> wrote:
> I have been having performance issues from time to time on our
> production ocfs2 volumes, so I set up a test system to try to reproduce
> what I was seeing on the production systems.  This is what I found out:
>
> I have a 2 node test system sharing a 2TB volume with a journal size of
> 256MB.  I can easily trigger the slowdown by starting to processes to
> write a 10GB file each, then I delete a different large file (7GB+)
> while the other processes are writing.  The slowdown is significant and
> very disruptive.  Not only did it take over 3 minutes to delete the
> file, every else with pause when entering that directory too.  A du
> command with stop and nfs access to that file system will think the
> server is not responding.  Under heavier amounts of writes, I have had a
> delete takes 13mins for a 8GB file, and NFS mounts return I/O errors.
> We often deal with large files, so this situation above is fairly common.
>
> I would like any ideas that would provide smoother performance of the
> OCFS2 volume and somehow eliminate the long pauses during deletes.
>
> Thanks,
>
> Andy
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20130329/a798bf09/attachment.html