Cory Sanders
2016-Oct-18 04:57 UTC
[Gluster-users] gluster0:group1 not matching up with mounted directory
I have volumes set up like this: gluster> volume info Volume Name: machines0 Type: Distribute Volume ID: f602dd45-ddab-4474-8308-d278768f1e00 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: gluster4:/data/brick1/machines0 Volume Name: group1 Type: Distribute Volume ID: cb64c8de-1f76-46c8-8136-8917b1618939 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: gluster1:/data/brick1/group1 Volume Name: backups Type: Replicate Volume ID: d7cb93c4-4626-46fd-b638-65fd244775ae Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gluster3:/data/brick1/backups Brick2: gluster4:/data/brick1/backups Volume Name: group0 Type: Distribute Volume ID: 0c52b522-5b04-480c-a058-d863df9ee949 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: gluster0:/data/brick1/group0 My problem is that when I do a disk free, group1 is filled up: root at node0:~# df -h Filesystem Size Used Avail Use% Mounted on udev 10M 0 10M 0% /dev tmpfs 3.2G 492K 3.2G 1% /run /dev/mapper/pve-root 24G 12G 11G 52% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 6.3G 56M 6.3G 1% /run/shm /dev/mapper/pve-data 48G 913M 48G 2% /var/lib/vz /dev/sda1 495M 223M 248M 48% /boot /dev/sdb1 740G 382G 359G 52% /data/brick1 /dev/fuse 30M 64K 30M 1% /etc/pve gluster0:group0 740G 382G 359G 52% /mnt/pve/group0 16.xx.xx.137:backups 1.9T 1.6T 233G 88% /mnt/pve/backups node4:machines0 7.3T 5.1T 2.3T 70% /mnt/pve/machines0 gluster0:group1 740G 643G 98G 87% /mnt/pve/group1 gluster2:/var/lib/vz 1.7T 182G 1.5T 11% /mnt/pve/node2local When I do a du -h in the respective directories, this is what I get. They don't match up with what a df -h shows. Gluster0:group0 shows the right amount of disk free, but gluster0:group1 is too fat and does not correspond to what is in /mnt/pve/group1 root at node0:/mnt/pve/group0# du -h -d 2 0 ./images/2134 0 ./images/8889 6.3G ./images/134 56G ./images/140 31G ./images/153 9.9G ./images/144 0 ./images/166 29G ./images/141 9.9G ./images/152 22G ./images/142 0 ./images/155 0 ./images/145 18G ./images/146 25G ./images/148 24G ./images/151 0 ./images/156 11G ./images/143 0 ./images/157 0 ./images/158 0 ./images/159 0 ./images/160 0 ./images/161 0 ./images/162 0 ./images/164 0 ./images/9149 0 ./images/7186 0 ./images/9150 9.7G ./images/149 29G ./images/150 0 ./images/9100 0 ./images/9145 17G ./images/147 51G ./images/187 12G ./images/9142 0 ./images/186 0 ./images/184 0 ./images/9167 0 ./images/102 0 ./images/99102 30G ./images/9153 382G ./images 0 ./template/iso 0 ./template 0 ./dump 382G . root at node0:/mnt/pve/group1/images# du -h -d 2 2.7G ./9153 9.7G ./162 9.9G ./164 11G ./166 9.6G ./161 0 ./146 9.8G ./155 9.8G ./156 9.9G ./157 9.7G ./159 9.9G ./160 9.9G ./158 21G ./185 11G ./165 0 ./153 11G ./154 0 ./9167 11G ./168 11G ./169 11G ./167 0 ./9165 11G ./171 0 ./9171 182G . root at node0:/data/brick1# du -h -d2 382G ./group0/.glusterfs 8.0K ./group0/images 0 ./group0/template 0 ./group0/dump 382G ./group0 0 ./group1/.glusterfs 0 ./group1/images 0 ./group1/template 0 ./group1/dump 0 ./group1 382G . root at node0:/data/brick1# gluster> peer status Number of Peers: 3 Hostname: 10.0.0.137 Uuid: 92071298-6809-49ff-9d6c-3761c01039ea State: Peer in Cluster (Connected) Hostname: 10.0.0.138 Uuid: 040a3b67-c516-4c9b-834b-f7f7470e8dfd State: Peer in Cluster (Connected) Hostname: gluster1 Uuid: 71cbcefb-0aea-4414-b88f-11f8954a8be2 State: Peer in Cluster (Connected) gluster> gluster> pool list UUID Hostname State 92071298-6809-49ff-9d6c-3761c01039ea 10.0.0.137 Connected 040a3b67-c516-4c9b-834b-f7f7470e8dfd 10.0.0.138 Connected 71cbcefb-0aea-4414-b88f-11f8954a8be2 gluster1 Connected 398228da-2300-4bc9-8e66-f4ae06a7c98e localhost Connected gluster> There are 5 nodes in a ProxMox cluster. Node0 has a 900GB RAID1 and is primarily responsible for running VMs from gluster0:group0 /mnt/pve/group0 Node1 has a 900GB RAID1 and is primarily responsible for running VMS from gluster0:group1 /mnt/pve/group1 Node2 is a development machine: gluster2:/var/lib/vz /mnt/pve/node2local Node3 has backups: /mnt/pve/backups Node4 has backups and also is supposed to mirror gluster0:group0 and group1 I think things are off on the configs. Thanks, I'm a bit of a newbie at gluster. Wanting to learn. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161018/b45069b5/attachment.html>
Niels de Vos
2016-Oct-18 08:28 UTC
[Gluster-users] gluster0:group1 not matching up with mounted directory
On Tue, Oct 18, 2016 at 04:57:29AM +0000, Cory Sanders wrote:> I have volumes set up like this: > gluster> volume info > > Volume Name: machines0 > Type: Distribute > Volume ID: f602dd45-ddab-4474-8308-d278768f1e00 > Status: Started > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: gluster4:/data/brick1/machines0 > > Volume Name: group1 > Type: Distribute > Volume ID: cb64c8de-1f76-46c8-8136-8917b1618939 > Status: Started > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: gluster1:/data/brick1/group1 > > Volume Name: backups > Type: Replicate > Volume ID: d7cb93c4-4626-46fd-b638-65fd244775ae > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gluster3:/data/brick1/backups > Brick2: gluster4:/data/brick1/backups > > Volume Name: group0 > Type: Distribute > Volume ID: 0c52b522-5b04-480c-a058-d863df9ee949 > Status: Started > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: gluster0:/data/brick1/group0 > > My problem is that when I do a disk free, group1 is filled up: > > root at node0:~# df -h > Filesystem Size Used Avail Use% Mounted on > udev 10M 0 10M 0% /dev > tmpfs 3.2G 492K 3.2G 1% /run > /dev/mapper/pve-root 24G 12G 11G 52% / > tmpfs 5.0M 0 5.0M 0% /run/lock > tmpfs 6.3G 56M 6.3G 1% /run/shm > /dev/mapper/pve-data 48G 913M 48G 2% /var/lib/vz > /dev/sda1 495M 223M 248M 48% /boot > /dev/sdb1 740G 382G 359G 52% /data/brick1 > /dev/fuse 30M 64K 30M 1% /etc/pve > gluster0:group0 740G 382G 359G 52% /mnt/pve/group0 > 16.xx.xx.137:backups 1.9T 1.6T 233G 88% /mnt/pve/backups > node4:machines0 7.3T 5.1T 2.3T 70% /mnt/pve/machines0 > gluster0:group1 740G 643G 98G 87% /mnt/pve/group1 > gluster2:/var/lib/vz 1.7T 182G 1.5T 11% /mnt/pve/node2local > > When I do a du -h in the respective directories, this is what I get. > They don't match up with what a df -h shows. Gluster0:group0 shows > the right amount of disk free, but gluster0:group1 is too fat and does > not correspond to what is in /mnt/pve/group1du and df work a little different: - du: crawl the directory structure and calculate the size - df: call the statfs() function that resturns information directly from the (superblock of the) filesystem This means, that all 'df' calls are routed to the bricks that are used for the Gluster volume. Those bricks then call statfs() on behalf of the Gluster client (fuse mountpoint), and the Gluster client uses the values returned by the bricks to calculate the 'fake' output for 'df'. Now, on your environment you seem to have the RAID1 filesystem mounted on /data/brick1 (/dev/sdb1 in the above 'df' output). All of the bricks are also located under /data/brick1/<volume>. This means that all 'df' commands will execute statfs() on the same filesystem hosting all of the bricks. Because statfs() returns the statistics over the whole filesystem (/data/brick1), the used and available size of /data/brick1 will be used in the calculations by the Gluster client to return the statistics to 'df'. With this understanding, you should be able to verify the size of the filesystems used for the bricks, and combine them per Gluster volume. Any of the /data/brick1 filesystems that host multiple bricks will likely have an 'unexpected' difference in available/used size. HTH, Niels -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161018/07702549/attachment.sig>