Gluster users - Jun 2013 - file open/write/move causing blank content in files

Hi

Have found an issue with our installation of gluster, and need some advice
on how to procede.

I wrote out a bug report detailing the issue.

https://bugzilla.redhat.com/show_bug.cgi?id=971630


I'm not sure if its a gluster configuration problem, or an actual bug. Or
even a VM thing. Or combination of both.

Will briefly describe the issue I am seeing (bug report above more
detailed).

In an application I'm developing on a 2 brick gluster installation (2
Centos 6 VM's), occasionally I'd see a file that appeared to exist in
both.

The permissions on both files exactly the same. Also the timestamp. I would
however notice in the application logs a few "permission denied"
errors
occuring, I'm assuming because my application is trying to read a file
thats currently being written.

This is the strange bit. When I cat/less the file, on 1 server its blank.
On the other the content of the file is fine. If I 'touch' the blank
file
the content of the file comes back.

So I wrote a couple of quick & dirty PHP scripts effectively duplicating
what the application tries todo.

write.php - this basically opens and write a file to a directory (in a loop
- 100 files). I read its contents in. I then move the file to another
gluster directory.
read.php -  finds files in both directories & tries to read contents in. In
a loop with no time delay in reading

Every single time, I end up with at least 1 file (mostly more than 1) which
has the characteristics I've described in both the original directory, and
the directory where the file was moved. Indicating that either a
file->open->write or a file->mv AND reading the files at the same time
causes it.

It would be nice if I could figure out what might be happening here. Is it
a bug? Maybe a config issue?

# gluster volume info

Volume Name: prxvol
Type: Replicate
Volume ID: b5879503-3c6b-4556-b6ee-e600208150a7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.178.19.254:/data
Brick2: 10.177.17.23:/data
Options Reconfigured:
auth.allow: 10.178.19.254,10.177.17.23
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
cluster.data-self-heal-algorithm: diff



cheers
Daryn
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130612/443215e6/attachment.html>

Hi!

Just a follow up to this issue. I also noticed if I try to touch the blank
file, sometimes I get a PHP warning

PHP Warning:  touch(): Utime failed: Permission denied etc etc

So I can't even guarantee that touching a file brings its contents back.

Any ideas on what might be happening?

Its quite confusing, because most of the time, everything looks great.
Files get replicated & it just works.

cheers
Daryn


On Wed, Jun 12, 2013 at 2:24 PM, Daryn Hanright <daz at planetnz.com>
wrote:
> Hi
>
> Have found an issue with our installation of gluster, and need some advice
> on how to procede.
>
> I wrote out a bug report detailing the issue.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=971630
>
>
> I'm not sure if its a gluster configuration problem, or an actual bug.
Or
> even a VM thing. Or combination of both.
>
> Will briefly describe the issue I am seeing (bug report above more
> detailed).
>
> In an application I'm developing on a 2 brick gluster installation (2
> Centos 6 VM's), occasionally I'd see a file that appeared to exist
in both.
>
> The permissions on both files exactly the same. Also the timestamp. I
> would however notice in the application logs a few "permission
denied"
> errors occuring, I'm assuming because my application is trying to read
a
> file thats currently being written.
>
> This is the strange bit. When I cat/less the file, on 1 server its blank.
> On the other the content of the file is fine. If I 'touch' the
blank file
> the content of the file comes back.
>
> So I wrote a couple of quick & dirty PHP scripts effectively
duplicating
> what the application tries todo.
>
> write.php - this basically opens and write a file to a directory (in a
> loop - 100 files). I read its contents in. I then move the file to another
> gluster directory.
> read.php -  finds files in both directories & tries to read contents
in.
> In a loop with no time delay in reading
>
> Every single time, I end up with at least 1 file (mostly more than 1)
> which has the characteristics I've described in both the original
> directory, and the directory where the file was moved. Indicating that
> either a file->open->write or a file->mv AND reading the files at
the same
> time causes it.
>
> It would be nice if I could figure out what might be happening here. Is it
> a bug? Maybe a config issue?
>
> # gluster volume info
>
> Volume Name: prxvol
> Type: Replicate
> Volume ID: b5879503-3c6b-4556-b6ee-e600208150a7
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.178.19.254:/data
> Brick2: 10.177.17.23:/data
> Options Reconfigured:
> auth.allow: 10.178.19.254,10.177.17.23
> diagnostics.brick-log-level: INFO
> diagnostics.client-log-level: INFO
> cluster.data-self-heal-algorithm: diff
>
>
>
> cheers
> Daryn
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130613/953fbcd8/attachment.html>