Gluster users - Oct 2010 - GlusterFS behavior with 2 clients

Hello,

I recently discovered GlusterFS and followed the installation instructions and
the relatively brief tutorials that describe setting up "test-volume"
but the behavior I am seeing is odd.  I have 4 machines set up to test
GlusterFS: 2 servers and 2 clients.

All the machines are running RHEL5U4 x86_64. The servers were set up as so:

192.168.56.101   server1
192.168.56.102   server2
192.168.56.103   client1
192.168.56.104   client2

glusterfs-core-3.1.0-1 and glusterfs-fuse-3.1.0-1 were installed on all 4 nodes.
glusterd was started on all 4 nodes (though I expect only the bricks/peers need
it).

The commands executed on server1 were:

gluster peer probe server2
    (no problems here)
gluster peer status
    (it returned the correct results but I don't have the output handy)
gluster volume create test-volume replica 2 transport tcp server1:/exp1
server2:/exp2
    (the command succeeded and it created an /exp1 directory on server1 and
/exp2 on server2)
gluster volume start test-volume
    (again, no problems)


Now, on the clients...

Mount test-volume on the clients:
[root at client1 tmp]# mount -t glusterfs server1:/test-volume /mnt
[root at client2 ~]# mount -t glusterfs server1:/test-volume /mnt

Make a file on test-volume using client1:
[root at client1 tmp]# echo "1234" >/mnt/foo
[root at client1 tmp]# cat /mnt/foo
1234
[root at client1 tmp]# ls -al /mnt/foo
-rw-r--r-- 1 root root 5 Oct 28 19:34 /mnt/foo

Make sure the file is seen on client2:
[root at client2 ~]# cat /mnt/foo
1234
[root at client2 ~]# ls -al /mnt/foo
-rw-r--r-- 1 root root 5 Oct 28 19:34 /mnt/foo

Append to the file using client2:
[root at client2 ~]# echo "5678" >>/mnt/foo
[root at client2 ~]# ls -al /mnt/foo
-rw-r--r-- 1 root root 10 Oct 28 19:35 /mnt/foo
[root at client2 ~]# md5sum /mnt/foo
9053253e972cf40443a4083f452f24d4  /mnt/foo

Now check and see that client1 sees the changes:
[root at client1 tmp]# cat /mnt/foo
1234
[root at client1 tmp]# ls -al /mnt/foo
-rw-r--r-- 1 root root 10 Oct 28 19:35 /mnt/foo
[root at client1 tmp]# md5sum /mnt/foo
e7df7cd2ca07f4f1ab415d457a6e1c13  /mnt/foo

If I check server1:/exp1/foo and server2:/exp2/foo, the files are in agreement
with what client2 thinks the file should look like.

client1 sees the new file size according to "ls" but "cat"
and "md5sum" still only think the file in its original
"1234" state is there.

If I unmount and remount test-volume on client1, it sees the modified file as it
exists on server1 and server2.

I've tried this same test on both RHEL5U4 and Fedora Core 13 with the same
results so I must be doing something wrong.

Can someone please tell me what I have done incorrectly?

Thank you for your patience with a newbie.

-
Matt

On Oct 28, 2010, at 5:00 PM, Matt Sorin wrote:
> Can someone please tell me what I have done incorrectly?

Matt,

You have run into this bug:
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2027

This has been fixed in the repository and will be available in the next release.
Until then, you can set this option on your Gluster volume as a work-around:

# gluster volume set performance.quick-read off

------------------------------
Vikas Gorur
Engineer - Gluster, Inc.
------------------------------

How many people on the list are using Fedora 12 (or 13) in production for
Glusterfs storage servers? I know that Gluster Platform uses Fedora 12 as its OS
- I was thinking of building my new glusterfs storage servers using Fedora, and
was wondering whether Fedora 13 was tested by Gluster for v 3.1.1 and what other
people's experiences were.

One of the reasons for my interest was so that I could use ext4 as the backend
file store, instead of ext3.

Thanks,

James Burnash, Unix Engineering


DISCLAIMER:
This e-mail, and any attachments thereto, is intended only for use by the
addressee(s) named herein and may contain legally privileged and/or confidential
information. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution or copying of this e-mail,
and any attachments thereto, is strictly prohibited. If you have received this
in error, please immediately notify me and permanently delete the original and
any copy of any e-mail and any printout thereof. E-mail transmission cannot be
guaranteed to be secure or error-free. The sender therefore does not accept
liability for any errors or omissions in the contents of this message which
arise as a result of e-mail transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its
discretion, monitor and review the content of all e-mail communications.
http://www.knight.com