Hi, I noticed that the glusterfs client tries to set ulimit -n to 1M. When I run booster with non-privileged user, the following line appears several times in the log file: [2009-09-14 09:15:22] W [client-protocol.c:6010:init] brick-0-0-0: WARNING: Failed to set 'ulimit -n 1M': Operation not permitted When I run it with root, there's no such complaint even though /proc/sys/fs/file-max shows a value of 65536. For normal user, it seems the default maximum of RLIMIT_NOFILE is 4096, which is a much smaller value than that is required by glusterfs. I'm wondering what's the impact of RLIMIT_NOFILE has to the performance and scalability of glusterfs. Does glusterfs in deed needs to open a large number of files when my DHT aggregates ~100 machines and ~500 volumes? I'm not sure if I should increase RLIMIT_NOFILE to a large number for every user when everyone needs to run booster in order to access glusterfs. Thanks, - Wei
> I noticed that the glusterfs client tries to set ulimit -n to 1M. ?When I > run booster with non-privileged user, the following line appears several > times in the log file: > > [2009-09-14 09:15:22] W [client-protocol.c:6010:init] brick-0-0-0: WARNING: > Failed to set 'ulimit -n 1M': Operation not permitted > > When I run it with root, there's no such complaint even though > /proc/sys/fs/file-max shows a value of 65536. ?For normal user, it seems the > default maximum of RLIMIT_NOFILE is 4096, which is a much smaller value than > that is required by glusterfs. > > I'm wondering what's the impact of RLIMIT_NOFILE has to the performance and > scalability of glusterfs. ?Does glusterfs in deed needs to open a large > number of files when my DHT aggregates ~100 machines and ~500 volumes? ?I'm > not sure if I should increase RLIMIT_NOFILE to a large number for every user > when everyone needs to run booster in order to access glusterfs.GlusterFS by itself only keeps the logfile open. Other than this the open fd count limits the sum of total number of client connections (2 sockets per client) and the total number of open file descriptors (1 fd per open file on that server). If you work with a non ulimited mode, you will be limited by 4096 fds for client connections and open fds. On a side note, do you really want glusterfsd to run as non-root? storage/posix expects to be run as root to work properly. Avati