Gluster users - Mar 2011 - GlusterFS 3.1.2 - df and du irregularities

Hi

I'm having a weird issue where my gluster volumes are being reported as full
by df but du shows only 18GB of actual data in them (which is less than 1% of
the capacity of the volumes).  The client shows 2/3 of the volume is full, only
replicate-3 is currently not "full" and given how this problem started
I expect it to be full by the time I come into the office tomorrow morning.  I
am runnung glusterfs 3.1.2 on Ubuntu 10.10. Does anyone have any idea why this
might be happening?

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md3              1.7T  1.7T     0 100% /export/glusterfs

Output of dh in the exported folder on my first gluster host

# du -h
..
...
....
18G	total

# cat /etc/glusterd/vols/backup-volume/backup-volume-fuse.vol

volume backup-volume-client-0
    type protocol/client
    option remote-host gluster01
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-1
    type protocol/client
    option remote-host gluster02
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-2
    type protocol/client
    option remote-host gluster03
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-3
    type protocol/client
    option remote-host gluster04
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-4
    type protocol/client
    option remote-host gluster05
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-5
    type protocol/client
    option remote-host gluster06
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-6
    type protocol/client
    option remote-host gluster07
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-7
    type protocol/client
    option remote-host gluster08
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-client-8
    type protocol/client
    option remote-host gluster09
    option remote-subvolume /export/glusterfs
    option transport-type tcp
end-volume

volume backup-volume-replicate-0
    type cluster/replicate
    subvolumes backup-volume-client-0 backup-volume-client-1
backup-volume-client-2
end-volume

volume backup-volume-replicate-1
    type cluster/replicate
    subvolumes backup-volume-client-3 backup-volume-client-4
backup-volume-client-5
end-volume

volume backup-volume-replicate-2
    type cluster/replicate
    subvolumes backup-volume-client-6 backup-volume-client-7
backup-volume-client-8
end-volume

volume backup-volume-dht
    type cluster/distribute
    subvolumes backup-volume-replicate-0 backup-volume-replicate-1
backup-volume-replicate-2
end-volume

volume backup-volume-write-behind
    type performance/write-behind
    subvolumes backup-volume-dht
end-volume

volume backup-volume-read-ahead
    type performance/read-ahead
    subvolumes backup-volume-write-behind
end-volume

volume backup-volume-io-cache
    type performance/io-cache
    subvolumes backup-volume-read-ahead
end-volume

volume backup-volume-quick-read
    type performance/quick-read
    subvolumes backup-volume-io-cache
end-volume

volume backup-volume-stat-prefetch
    type performance/stat-prefetch
    subvolumes backup-volume-quick-read
end-volume

volume backup-volume
    type debug/io-stats
    subvolumes backup-volume-stat-prefetch
end-volume

Hi Nic,

Answers inline.

> I'm having a weird issue where my gluster volumes are being reported as
> full by df but du shows only 18GB of actual data in them (which is less
than
> 1% of the capacity of the volumes).  The client shows 2/3 of the volume is
> full, only replicate-3 is currently not "full" and given how this
problem
> started I expect it to be full by the time I come into the office tomorrow
> morning.  I am runnung glusterfs 3.1.2 on Ubuntu 10.10. Does anyone have
any
> idea why this might be
happening?
>
>
GlusterFS exports in a thin provision way by design, which causes this
behavior. You are seeing this behavior because of the fundamental difference
in the implementation of the two commands. (df internally uses
'statvfs(2)',
and du uses 'readdir(), stat()' recursively to find out the size under a
given directory).

To understand it better, I will give you an example from my laptop

root at supernova:~# gluster volume create test-df
supernova:/tmp/export/test-df
Creation of volume test-df has been successful. Please start the volume to
access data.
root at supernova:~# gluster volume start test-df
Starting volume test-df has been successful
root at supernova:~# df -h /tmp/export/test-df
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.2G  6.3G  2.5G  72% /
root at supernova:~# mount -t glusterfs localhost:/test-df /mnt/glusterfs
root at supernova:~# df -h /mnt/glusterfs
Filesystem            Size  Used Avail Use% Mounted on
localhost:/test-df    9.2G  6.3G  2.5G  72% /mnt/glusterfs
root at supernova:~# du -sh /mnt/glusterfs/
4.0K /mnt/glusterfs/

That means the export directory used for the volume resides on a filesystem
which is already 72% full. So, when the filesystem is mounted, the 'df'
of
the filesystem shows 72% full, and there are no files in glusterfs volume.

Hope that explains. So, to set up glusterfs properly, make sure your backend
exports are dedicated to GlusterFS and some other process is not filling it
up.

Regards,
Amar

PS: this should be to our FAQ in wiki, so people will be aware of these type
of issues due to thin provision factors.