Pat Haley
2020-Mar-10 15:51 UTC
[Gluster-users] Erroneous "No space left on device." messages
Hi, We have developed a problem with Gluster reporting "No space left on device." even though "df" of both the gluster filesystem and the underlying bricks show space available (details below).? Our inode usage is between 1-3%.? We are running gluster 3.7.11 in a distributed volume across 2 servers (2 bricks each). We have followed the thread https://lists.gluster.org/pipermail/gluster-users/2020-March/037821.html but haven't found a solution yet. Last night we ran a rebalance which appeared successful (and have since cleared up some more space which seems to have mainly been on one brick).? There were intermittent erroneous "No space..." messages last night, but they have become much more frequent today. Any help would be greatly appreciated. Thanks --------------------------- [root at mseas-data2 ~]# df -h --------------------------- Filesystem????? Size? Used Avail Use% Mounted on /dev/sdb??????? 164T? 164T? 324G 100% /mnt/brick2 /dev/sda??????? 164T? 164T? 323G 100% /mnt/brick1 --------------------------- [root at mseas-data2 ~]# df -i --------------------------- Filesystem???????? Inodes??? IUsed????? IFree IUse% Mounted on /dev/sdb?????? 1375470800 31207165 1344263635??? 3% /mnt/brick2 /dev/sda?????? 1384781520 28706614 1356074906??? 3% /mnt/brick1 --------------------------- [root at mseas-data3 ~]# df -h --------------------------- /dev/sda?????????????? 91T?? 91T? 323G 100% /export/sda/brick3 /dev/mapper/vg_Data4-lv_Data4 ?????????????????????? 91T?? 88T? 3.4T? 97% /export/sdc/brick4 --------------------------- [root at mseas-data3 ~]# df -i --------------------------- /dev/sda????????????? 679323496? 9822199? 669501297??? 2% /export/sda/brick3 /dev/mapper/vg_Data4-lv_Data4 ???????????????????? 3906272768 11467484 3894805284??? 1% /export/sdc/brick4 --------------------------------------- [root at mseas-data2 ~]# gluster --version --------------------------------------- glusterfs 3.7.11 built on Apr 27 2016 14:09:22 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. ----------------------------------------- [root at mseas-data2 ~]# gluster volume info ----------------------------------------- Volume Name: data-volume Type: Distribute Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: mseas-data2:/mnt/brick1 Brick2: mseas-data2:/mnt/brick2 Brick3: mseas-data3:/export/sda/brick3 Brick4: mseas-data3:/export/sdc/brick4 Options Reconfigured: nfs.export-volumes: off nfs.disable: on performance.readdir-ahead: on diagnostics.brick-sys-log-level: WARNING nfs.exports-auth-enable: on server.allow-insecure: on auth.allow: * disperse.eager-lock: off performance.open-behind: off performance.md-cache-timeout: 60 network.inode-lru-limit: 50000 diagnostics.client-log-level: ERROR -------------------------------------------------------------- [root at mseas-data2 ~]# gluster volume status data-volume detail -------------------------------------------------------------- Status of volume: data-volume ------------------------------------------------------------------------------ Brick??????????????? : Brick mseas-data2:/mnt/brick1 TCP Port???????????? : 49154 RDMA Port??????????? : 0 Online?????????????? : Y Pid????????????????? : 4601 File System????????? : xfs Device?????????????? : /dev/sda Mount Options??????? : rw Inode Size?????????? : 256 Disk Space Free????? : 318.8GB Total Disk Space???? : 163.7TB Inode Count????????? : 1365878288 Free Inodes????????? : 1337173596 ------------------------------------------------------------------------------ Brick??????????????? : Brick mseas-data2:/mnt/brick2 TCP Port???????????? : 49155 RDMA Port??????????? : 0 Online?????????????? : Y Pid????????????????? : 7949 File System????????? : xfs Device?????????????? : /dev/sdb Mount Options??????? : rw Inode Size?????????? : 256 Disk Space Free????? : 319.8GB Total Disk Space???? : 163.7TB Inode Count????????? : 1372421408 Free Inodes????????? : 1341219039 ------------------------------------------------------------------------------ Brick??????????????? : Brick mseas-data3:/export/sda/brick3 TCP Port???????????? : 49153 RDMA Port??????????? : 0 Online?????????????? : Y Pid????????????????? : 4650 File System????????? : xfs Device?????????????? : /dev/sda Mount Options??????? : rw Inode Size?????????? : 512 Disk Space Free????? : 325.3GB Total Disk Space???? : 91.0TB Inode Count????????? : 692001992 Free Inodes????????? : 682188893 ------------------------------------------------------------------------------ Brick??????????????? : Brick mseas-data3:/export/sdc/brick4 TCP Port???????????? : 49154 RDMA Port??????????? : 0 Online?????????????? : Y Pid????????????????? : 23772 File System????????? : xfs Device?????????????? : /dev/mapper/vg_Data4-lv_Data4 Mount Options??????? : rw Inode Size?????????? : 256 Disk Space Free????? : 3.4TB Total Disk Space???? : 90.9TB Inode Count????????? : 3906272768 Free Inodes????????? : 3894809903 -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
Pat Haley
2020-Mar-10 18:14 UTC
[Gluster-users] Erroneous "No space left on device." messages
HI,
After some more poking around in the logs (specifically the brick logs)
* brick1 & brick2 have both been recording "No space left on
device"
messages today (as recently at 15 minutes ago)
* brick3 last recorded a "No space left on device" message last
night
around 10:30pm
* brick4 has no such messages in its log file
Note brick1 & brick2 are on one server, brick3 and brick4 are on the
second server.
Pat
On 3/10/20 11:51 AM, Pat Haley wrote:>
> Hi,
>
> We have developed a problem with Gluster reporting "No space left on
> device." even though "df" of both the gluster filesystem and
the
> underlying bricks show space available (details below).? Our inode
> usage is between 1-3%.? We are running gluster 3.7.11 in a distributed
> volume across 2 servers (2 bricks each). We have followed the thread
> https://lists.gluster.org/pipermail/gluster-users/2020-March/037821.html
> but haven't found a solution yet.
>
> Last night we ran a rebalance which appeared successful (and have
> since cleared up some more space which seems to have mainly been on
> one brick).? There were intermittent erroneous "No space..."
messages
> last night, but they have become much more frequent today.
>
> Any help would be greatly appreciated.
>
> Thanks
>
> ---------------------------
> [root at mseas-data2 ~]# df -h
> ---------------------------
> Filesystem????? Size? Used Avail Use% Mounted on
> /dev/sdb??????? 164T? 164T? 324G 100% /mnt/brick2
> /dev/sda??????? 164T? 164T? 323G 100% /mnt/brick1
> ---------------------------
> [root at mseas-data2 ~]# df -i
> ---------------------------
> Filesystem???????? Inodes??? IUsed????? IFree IUse% Mounted on
> /dev/sdb?????? 1375470800 31207165 1344263635??? 3% /mnt/brick2
> /dev/sda?????? 1384781520 28706614 1356074906??? 3% /mnt/brick1
>
> ---------------------------
> [root at mseas-data3 ~]# df -h
> ---------------------------
> /dev/sda?????????????? 91T?? 91T? 323G 100% /export/sda/brick3
> /dev/mapper/vg_Data4-lv_Data4
> ?????????????????????? 91T?? 88T? 3.4T? 97% /export/sdc/brick4
> ---------------------------
> [root at mseas-data3 ~]# df -i
> ---------------------------
> /dev/sda????????????? 679323496? 9822199? 669501297??? 2%
> /export/sda/brick3
> /dev/mapper/vg_Data4-lv_Data4
> ???????????????????? 3906272768 11467484 3894805284??? 1%
> /export/sdc/brick4
>
>
>
> ---------------------------------------
> [root at mseas-data2 ~]# gluster --version
> ---------------------------------------
> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> You may redistribute copies of GlusterFS under the terms of the GNU
> General Public License.
>
>
>
> -----------------------------------------
> [root at mseas-data2 ~]# gluster volume info
> -----------------------------------------
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 4
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Brick3: mseas-data3:/export/sda/brick3
> Brick4: mseas-data3:/export/sdc/brick4
> Options Reconfigured:
> nfs.export-volumes: off
> nfs.disable: on
> performance.readdir-ahead: on
> diagnostics.brick-sys-log-level: WARNING
> nfs.exports-auth-enable: on
> server.allow-insecure: on
> auth.allow: *
> disperse.eager-lock: off
> performance.open-behind: off
> performance.md-cache-timeout: 60
> network.inode-lru-limit: 50000
> diagnostics.client-log-level: ERROR
>
>
>
> --------------------------------------------------------------
> [root at mseas-data2 ~]# gluster volume status data-volume detail
> --------------------------------------------------------------
> Status of volume: data-volume
>
------------------------------------------------------------------------------
>
> Brick??????????????? : Brick mseas-data2:/mnt/brick1
> TCP Port???????????? : 49154
> RDMA Port??????????? : 0
> Online?????????????? : Y
> Pid????????????????? : 4601
> File System????????? : xfs
> Device?????????????? : /dev/sda
> Mount Options??????? : rw
> Inode Size?????????? : 256
> Disk Space Free????? : 318.8GB
> Total Disk Space???? : 163.7TB
> Inode Count????????? : 1365878288
> Free Inodes????????? : 1337173596
>
------------------------------------------------------------------------------
>
> Brick??????????????? : Brick mseas-data2:/mnt/brick2
> TCP Port???????????? : 49155
> RDMA Port??????????? : 0
> Online?????????????? : Y
> Pid????????????????? : 7949
> File System????????? : xfs
> Device?????????????? : /dev/sdb
> Mount Options??????? : rw
> Inode Size?????????? : 256
> Disk Space Free????? : 319.8GB
> Total Disk Space???? : 163.7TB
> Inode Count????????? : 1372421408
> Free Inodes????????? : 1341219039
>
------------------------------------------------------------------------------
>
> Brick??????????????? : Brick mseas-data3:/export/sda/brick3
> TCP Port???????????? : 49153
> RDMA Port??????????? : 0
> Online?????????????? : Y
> Pid????????????????? : 4650
> File System????????? : xfs
> Device?????????????? : /dev/sda
> Mount Options??????? : rw
> Inode Size?????????? : 512
> Disk Space Free????? : 325.3GB
> Total Disk Space???? : 91.0TB
> Inode Count????????? : 692001992
> Free Inodes????????? : 682188893
>
------------------------------------------------------------------------------
>
> Brick??????????????? : Brick mseas-data3:/export/sdc/brick4
> TCP Port???????????? : 49154
> RDMA Port??????????? : 0
> Online?????????????? : Y
> Pid????????????????? : 23772
> File System????????? : xfs
> Device?????????????? : /dev/mapper/vg_Data4-lv_Data4
> Mount Options??????? : rw
> Inode Size?????????? : 256
> Disk Space Free????? : 3.4TB
> Total Disk Space???? : 90.9TB
> Inode Count????????? : 3906272768
> Free Inodes????????? : 3894809903
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200310/661dab82/attachment.html>