Gluster users - Mar 2012 - xattrs errors when creating files on Gluster3.3b2

Hi everyone,

Hope anyone can shed some light on some strange behavior we've just 
having with Gluster.
Last night, while using gluster3.3b2, we came up with some errors, after 
running tests for several days.

We're running the test in a striped 4 disk volume:
# gluster volume info data
Volume Name: data
Type: Stripe
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 192.168.19.103:/gluster/enclusterdata_1
Brick2: 192.168.19.103:/gluster/enclusterdata_2
Brick3: 192.168.19.103:/gluster/enclusterdata_3
Brick4: 192.168.19.103:/gluster/enclusterdata_4

When creating a lot of files, some of the files started to have erratic 
behavior. They are in the file system (ls commands below) but we are not 
being able to read some of the files (file ending with 22413, in the 
command below), 9 in total to be exact.
Both files were created programatically, under the same circumstances at 
nearly the same time. The directory has currently 12668 files inside.

# ls -la fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
-rw-r--r--. 1 1000 1000 41759951 Mar 21 20:34 
fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
# ls -la fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4
-rw-r--r--. 1 1000 1000 40994275 Mar 21 20:34 
fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4

# file fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4: writable, 
regular file, no read permission
# file fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4
fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22412-22412.mp4: ISO Media, 
MPEG v4 system, 3GPP JVT AVC


When we checked the logs of each brick, all of the bricks had these 
errors for the 9 files in question:

[2012-03-21 20:34:36.309525] E [posix-helpers.c:585:posix_handle_pair] 
0-data-posix: 
/gluster/enclusterdata_4/encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4:
key:trusted.data-stripe-0.stripe-size error:File exists
gluster-enclusterdata_4.log:[2012-03-21 20:34:36.309580] E 
[posix.c:1766:posix_create] 0-data-posix: setting xattrs on 
/encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
failed (File exists)
gluster-enclusterdata_4.log:[2012-03-22 08:15:09.205467] I 
[server3_1-fops.c:1493:server_stat_cbk] 0-data-server: 176411: STAT 
/encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
(0) ==> -1 (No such file or directory)
gluster-enclusterdata_4.log:[2012-03-22 08:15:29.287542] I 
[server3_1-fops.c:1493:server_stat_cbk] 0-data-server: 184724: STAT 
/encluster/01c79d00-683a-11e1-a105-003048bfe717/sujwork/fd66e046-6c2f-11e1-8e8d-003048bfe717-segment22413-22413.mp4
(0) ==> -1 (No such file or directory)

Whenever we try to read the file, the "No such file or directory"
error
crops up in the logs.
Any idea what could be going wrong? In total we have 9 files that show 
this erratic behavior, all with similar information on the brick logs.

Thanks in advance,
Daniel

On 03/22/2012 08:51 AM, Daniel Pereira wrote:
> Hi everyone,
>
> Hope anyone can shed some light on some strange behavior we've just
> having with Gluster.
> Last night, while using gluster3.3b2, we came up with some errors, after
> running tests for several days.
>
Hi Daniel,

Can you consider testing 3.3.0qa32+ versions untill 3.3.0beta3 is made 
available? Because 3.3.0beta2 was pretty old and we have made 
significant changes/bug fixes after the beta2 release.

> We're running the test in a striped 4 disk volume:
> # gluster volume info data
> Volume Name: data
> Type: Stripe
Considering this is a 'stripe' volume, what is the backend filesystem 
you are using? Because with XFS there seems to be some issues with data 
block pre-allocation algorithm, and glusterfs's logic of leaving holes 
for striped blocks. We are evaluating better ways of solving the stripe 
issue, till then, I recommend ext4 or other filesystems for you as backend.

Regards,
Amar