Gluster users - Jan 2011 - No space left on device

Dear all,

today, we started using GlusterFS at our institute at a large scale. A 
problem occurred for the first migrated users, that was already 
discussed on this mailing list, but with an older version of glusterFS 
and, as far as I could tell, without a solution.

When users wants to create or move a file, they sometimes get the error 
message:

mv: cannot create regular file `/storage/cluster/<etc...>': No space 
left on device

Doing df -h tells me, however:

glusterfs#192.168.101.247:/lemmy
                      104T   69T   36T  66% /storage/cluster

It may be of importance that one brick in the cluster is actually 100% 
used. Others are almost completely empty. I am using GlusterFS.3.1.1, 
the file servers are running debian lenny or ubuntu server 10.04, 
clients are SLC4, SLC5, CentOS and ubuntu server 10.04.

The migration of the users was done in the following way:

The following volume was created:

Volume Name: lemmy
Type: Distribute
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: 192.168.101.249:/storage/4/cluster
Brick2: 192.168.101.248:/storage/5/cluster
Brick3: 192.168.101.250:/storage/6/cluster
Brick4: 192.168.101.247:/storage/7/cluster
Brick5: 192.168.101.246:/storage/8/cluster

The user directories were moved into one of the above folders 
(whichever required the least amount of network traffic), using "find 
-print" a file list was created and an "ls" was done on each
line. All
the data can be read by the users, and for the most part, they can also 
write to their directories. Not always, however, as described above.

Any advice is very much appreciated,
Best Regards,
Daniel

Assuming you are doing a straight distribute there(?), if the user in 
question is hashed onto the brick that is 100% full you'll get a space 
error. Not sure I followed your migration details though, when you say 
"user directories were moved into one of the above folders" do you
mean
copied directly onto the individual storage bricks?

On 01/19/2011 05:01 AM, zander at ekp.uni-karlsruhe.de
wrote:
> mv: cannot create regular file `/storage/cluster/<etc...>': No
space
> left on device
>
> Doing df -h tells me, however:
>
> glusterfs#192.168.101.247:/lemmy
>                        104T   69T   36T  66% /storage/cluster
>
> It may be of importance that one brick in the cluster is actually 100%
> used. Others are almost completely empty. I am using GlusterFS.3.1.1,
> the file servers are running debian lenny or ubuntu server 10.04,
> clients are SLC4, SLC5, CentOS and ubuntu server 10.04.
>