Gluster users - Apr 2015 - strange "permission denied" problem

Hi,

I am new to gluster and would appreciate if someone could help me to
understand what may be wrong.

We have a small filesystem (currently - just one brick) and on the same
client node I have two processes. One is writing files to a specific
glusterfs share and another one is periodically scanning that directory and
loading all the files from there. Files are small, hundreds of bytes at
most. There are not too many files written, maybe one in a couple of
seconds.

Quite often (about 20% of the cases) the "consumer" process fails to
open
the file written by the "producer". The producer does create a
temporary
file first and then renames it to its final name. Both processes are Java
apps.

Producer writes the file as follows:

4957  18:15:37 open("/share/fileXYZ.settings", O_WRONLY|O_CREAT|
O_TRUNC, 0666) = 245 <0.001671>

The consumer does this:

2030  04:26:11 stat("/share/fileXYZ.settings.settings",
{st_mode=S_IFREG|0644, st_size=556, ...}) = 0 <0.000006>
2030  04:26:11 access("/share/fileXYZ.settings.settings", R_OK) = 0
<0.000006>
2030  04:26:11 open("/share/fileXYZ.settings.settings", O_RDONLY) = -1
EACCES (Permission denied) <0.000258>

Interestingly enough, if the consumer retries then the file most likely
will be readable. Very strange - especially considering that access()
returns 0 just before open().

It does look like the problem of this specific filesystem - if the same app
is used against a local directory than it works flawlessly.



-- 
Nikolai Grigoriev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150416/2cd1ff4e/attachment.html>

On 16/04/15 19:46, Nikolai Grigoriev wrote:
> Hi,
>
> I am new to gluster and would appreciate if someone could help me to 
> understand what may be wrong.
>
> We have a small filesystem (currently - just one brick) and on the 
> same client node I have two processes. One is writing files to a 
> specific glusterfs share and another one is periodically scanning that 
> directory and loading all the files from there. Files are small, 
> hundreds of bytes at most. There are not too many files written, maybe 
> one in a couple of seconds.
>
> Quite often (about 20% of the cases) the "consumer" process fails
to
> open the file written by the "producer". The producer does create
a
> temporary file first and then renames it to its final name. Both 
> processes are Java apps.
>
>
I'm seeing a lot of this kind of thing reported on the list lately.

It makes me worry that glusterfs may not be production-ready for 
critical data that requires updates to be visible at the same instant in 
time across multiple clients. Should not updates be atomic and 
consistent across all connected clients? Is there a limitation of the 
present architecture? Or is there something we need to set on the server 
side to enforce such consistency? The available docs certainly don't 
make such things plain.

For our current proposed workload this is not very important but for 
applications like email storage for IMAP and transactional applications 
it's essential.

Thanks.