thr3ads.net - Gluster users - [Gluster-users] "No space left on device" because one brick is full, rebalance is stopped [Jan 2013]

If this information is useful, please help other people find it:
Share via:

Homer Li

2013-Jan-31 02:34 UTC

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped

Hello ;
    I use rsync backup some files.  rsync get a error, it show "No space
left on device (28)".
    There is 4.4TB free space in the glusterfs,  how can I use these free
space ?


I found one brick is full.

#ssh -n 10.0.10.89 df -h /export/data1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb               17T   17T  9.9M 100% /export/data1


Glusterfs mount info

# df -h /mnt2
Filesystem            Size  Used Avail Use% Mounted on
10.0.10.88:/sz_vol01   82T   78T  4.4T  95% /mnt2


I try rebalance command.
Rebalance was stopped. it could not balance the disk space.

Rebalance info:
#gluster volume rebalance sz_vol01 status
                                    Node Rebalanced-files
size       scanned      failures         status
                               ---------      -----------   -----------
-----------   -----------   ------------
                               localhost                0
0Bytes       4350872           192      completed
                              10.0.10.86                7
11.5GB       4350879         32741      completed
                              10.0.10.90                0
0Bytes       4350872         14022      completed
                              10.0.10.89                0
0Bytes       2221881             1        stopped
                              10.0.10.87               91
9.4GB       4350872         46517      completed



#gluster volume info

Volume Name: sz_vol01
Type: Distribute
Volume ID: 6b331704-b89c-4bc9-bdc8-561c9eea1e03
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: 10.0.10.86:/export/data1
Brick2: 10.0.10.87:/export/data1
Brick3: 10.0.10.89:/export/data1
Brick4: 10.0.10.90:/export/data1
Brick5: 10.0.10.88:/export/data1
Options Reconfigured:
auth.allow: 10.*
nfs.disable: off

gluster version:
# rpm -qa | grep gluster
glusterfs-server-3.3.1-3.el6.x86_64
glusterfs-fuse-3.3.1-3.el6.x86_64
glusterfs-geo-replication-3.3.1-3.el6.x86_64
glusterfs-debuginfo-3.3.1-3.el6.x86_64
glusterfs-3.3.1-3.el6.x86_64
glusterfs-rdma-3.3.1-3.el6.x86_64

# cat /etc/redhat-release
CentOS release 6.3 (Final)



-- 
Best Regards
Homer Li
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130131/4f33d66c/attachment.html>

Homer Li

2013-Jan-31 02:35 UTC

head link

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped

Hello ;
    I use rsync backup some files.  rsync get a error, it show "No space
left on device (28)".
    There is 4.4TB free space in the glusterfs,  how can I use these free
space ?


I found one brick is full.

#ssh -n 10.0.10.89 df -h /export/data1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb               17T   17T  9.9M 100% /export/data1


Glusterfs mount info

# df -h /mnt2
Filesystem            Size  Used Avail Use% Mounted on
10.0.10.88:/sz_vol01   82T   78T  4.4T  95% /mnt2


I try rebalance command.
Rebalance was stopped. it could not balance the disk space.

Rebalance info:
#gluster volume rebalance sz_vol01 status
                                    Node Rebalanced-files
size       scanned      failures         status
                               ---------      -----------   -----------
-----------   -----------   ------------
                               localhost                0
0Bytes       4350872           192      completed
                              10.0.10.86                7
11.5GB       4350879         32741      completed
                              10.0.10.90                0
0Bytes       4350872         14022      completed
                              10.0.10.89                0
0Bytes       2221881             1        stopped
                              10.0.10.87               91
9.4GB       4350872         46517      completed



#gluster volume info

Volume Name: sz_vol01
Type: Distribute
Volume ID: 6b331704-b89c-4bc9-bdc8-561c9eea1e03
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: 10.0.10.86:/export/data1
Brick2: 10.0.10.87:/export/data1
Brick3: 10.0.10.89:/export/data1
Brick4: 10.0.10.90:/export/data1
Brick5: 10.0.10.88:/export/data1
Options Reconfigured:
auth.allow: 10.*
nfs.disable: off

gluster version:
# rpm -qa | grep gluster
glusterfs-server-3.3.1-3.el6.x86_64
glusterfs-fuse-3.3.1-3.el6.x86_64
glusterfs-geo-replication-3.3.1-3.el6.x86_64
glusterfs-debuginfo-3.3.1-3.el6.x86_64
glusterfs-3.3.1-3.el6.x86_64
glusterfs-rdma-3.3.1-3.el6.x86_64

# cat /etc/redhat-release
CentOS release 6.3 (Final)



-- 
Best Regards
Homer Li
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130131/6f1f1c2f/attachment.html>

Homer Li

2013-Jan-31 02:39 UTC

head link

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped

Hello ;
    I use rsync backup some files.  rsync get a error, it show "No space
left on device (28)".
    There is 4.4TB free space in the glusterfs,  how can I use these free
space ?


I found one brick is full.

#ssh -n 10.0.10.89 df -h /export/data1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb               17T   17T  9.9M 100% /export/data1


Glusterfs mount info

# df -h /mnt2
Filesystem            Size  Used Avail Use% Mounted on
10.0.10.88:/sz_vol01   82T   78T  4.4T  95% /mnt2


I try rebalance command.
Rebalance was stopped. it could not balance the disk space.

Rebalance info:
#gluster volume rebalance sz_vol01 status

      Node Rebalanced-files          size       scanned
failures         status
                               ---------      -----------   -----------
-----------   -----------   ------------
                               localhost                0
0Bytes       4350872           192      completed
                              10.0.10.86                7
11.5GB       4350879         32741      completed
                              10.0.10.90                0
0Bytes       4350872         14022      completed
                              10.0.10.89                0
0Bytes       2221881             1        stopped
                              10.0.10.87               91
9.4GB       4350872         46517      completed



#gluster volume info

Volume Name: sz_vol01
Type: Distribute
Volume ID: 6b331704-b89c-4bc9-bdc8-561c9eea1e03
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: 10.0.10.86:/export/data1
Brick2: 10.0.10.87:/export/data1
Brick3: 10.0.10.89:/export/data1
Brick4: 10.0.10.90:/export/data1
Brick5: 10.0.10.88:/export/data1
Options Reconfigured:
auth.allow: 10.*
nfs.disable: off

gluster version:
# rpm -qa | grep gluster
glusterfs-server-3.3.1-3.el6.x86_64
glusterfs-fuse-3.3.1-3.el6.x86_64
glusterfs-geo-replication-3.3.1-3.el6.x86_64
glusterfs-debuginfo-3.3.1-3.el6.x86_64
glusterfs-3.3.1-3.el6.x86_64
glusterfs-rdma-3.3.1-3.el6.x86_64

# cat /etc/redhat-release
CentOS release 6.3 (Final)


-- 
Best Regards
Homer Li
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130131/1fb113cc/attachment.html>

Gluster users - Jan 2013 - "No space left on device" because one brick is full, rebalance is stopped

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped

[Gluster-users] "No space left on device" because one brick is full, rebalance is stopped