Muslim Adel Abu Taha
2012-Nov-18 21:08 UTC
[Gluster-users] peer not online again after reboot
Hello there, I've 6 servers creating a 50TB storage. My setup is Distribution + Replication. My servers are named: storage01 - storage06, and they are running Ubuntu 12.04 64-bit. Today I decided to install available packages updates on the servers. So I started with storage06. After the installation of new packages and a new kernel, I decided to reboot it to use the newly installed kernel. I issued the reboot and withing minutes, the server was back again. But gluster on it seems to be offline: When I run the below command on storage01 (The server where I created all the volumes) I get the below: root at storage01:~# gluster peer status Number of Peers: 5 Hostname: storage05 Uuid: 9274b525-8089-4cc7-afd8-5031b7e2774e State: Peer in Cluster (Connected) Hostname: storage02 Uuid: a0d68c85-b34a-4052-9d45-cbdd9b08ad18 State: Peer in Cluster (Connected) Hostname: storage06 Uuid: 5aadb3d0-d7bc-447f-83a7-368104b4ddc4 State: Peer in Cluster (Connected) Hostname: storage03 Uuid: 94d0375a-31f2-4bb1-9a5d-647c20eedbcf State: Peer in Cluster (Connected) Hostname: storage04 Uuid: 888cb603-ffbf-4f09-a0a4-890170519eb9 State: Peer in Cluster (Connected) But when I run volume info, I get: gluster volume status all detail Status of volume: orig ------------------------------------------------------------------------------ Brick : Brick storage01:/mnt/storage/orig Port : 24012 Online : Y Pid : 775 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.5TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864568149 ------------------------------------------------------------------------------ Brick : Brick storage02:/mnt/storage/orig Port : 24012 Online : Y Pid : 22838 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 8.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864445198 ------------------------------------------------------------------------------ Brick : Brick storage03:/mnt/storage/orig Port : 24012 Online : Y Pid : 11342 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561821 ------------------------------------------------------------------------------ Brick : Brick storage04:/mnt/storage/orig Port : 24012 Online : Y Pid : 10887 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864557582 ------------------------------------------------------------------------------ Brick : Brick storage05:/mnt/storage/orig Port : 24012 Online : Y Pid : 32098 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561308 ------------------------------------------------------------------------------ Brick : Brick storage06:/mnt/storage/orig Port : 24012 Online : N Pid : N/A File System : N/A Device : N/A Mount Options : N/A Inode Size : N/A Disk Space Free : 0Bytes Total Disk Space : 0Bytes Inode Count : N/A Free Inodes : N/A Status of volume: mp3 ------------------------------------------------------------------------------ Brick : Brick storage01:/mnt/storage/mp3 Port : 24009 Online : Y Pid : 737 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.5TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864568149 ------------------------------------------------------------------------------ Brick : Brick storage02:/mnt/storage/mp3 Port : 24009 Online : Y Pid : 22802 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 8.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864445198 ------------------------------------------------------------------------------ Brick : Brick storage03:/mnt/storage/mp3 Port : 24009 Online : Y Pid : 11306 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561821 ------------------------------------------------------------------------------ Brick : Brick storage04:/mnt/storage/mp3 Port : 24009 Online : Y Pid : 10853 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864557582 ------------------------------------------------------------------------------ Brick : Brick storage05:/mnt/storage/mp3 Port : 24009 Online : Y Pid : 32062 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561308 ------------------------------------------------------------------------------ Brick : Brick storage06:/mnt/storage/mp3 Port : 24009 Online : N Pid : N/A File System : N/A Device : N/A Mount Options : N/A Inode Size : N/A Disk Space Free : 0Bytes Total Disk Space : 0Bytes Inode Count : N/A Free Inodes : N/A Status of volume: flv ------------------------------------------------------------------------------ Brick : Brick storage01:/mnt/storage/flv Port : 24011 Online : Y Pid : 435 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.5TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864568149 ------------------------------------------------------------------------------ Brick : Brick storage02:/mnt/storage/flv Port : 24011 Online : Y Pid : 22544 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 8.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864445198 ------------------------------------------------------------------------------ Brick : Brick storage03:/mnt/storage/flv Port : 24011 Online : Y Pid : 11051 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561821 ------------------------------------------------------------------------------ Brick : Brick storage04:/mnt/storage/flv Port : 24011 Online : Y Pid : 10595 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864557582 ------------------------------------------------------------------------------ Brick : Brick storage05:/mnt/storage/flv Port : 24011 Online : Y Pid : 31798 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561308 ------------------------------------------------------------------------------ Brick : Brick storage06:/mnt/storage/flv Port : 24011 Online : N Pid : N/A File System : N/A Device : N/A Mount Options : N/A Inode Size : N/A Disk Space Free : 0Bytes Total Disk Space : 0Bytes Inode Count : N/A Free Inodes : N/A Status of volume: mp4 ------------------------------------------------------------------------------ Brick : Brick storage01:/mnt/storage/mp4 Port : 24010 Online : Y Pid : 698 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.5TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864568149 ------------------------------------------------------------------------------ Brick : Brick storage02:/mnt/storage/mp4 Port : 24010 Online : Y Pid : 22756 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 8.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864445198 ------------------------------------------------------------------------------ Brick : Brick storage03:/mnt/storage/mp4 Port : 24010 Online : Y Pid : 11272 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561821 ------------------------------------------------------------------------------ Brick : Brick storage04:/mnt/storage/mp4 Port : 24010 Online : Y Pid : 10818 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.3TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864557582 ------------------------------------------------------------------------------ Brick : Brick storage05:/mnt/storage/mp4 Port : 24010 Online : Y Pid : 32027 File System : xfs Device : /dev/mapper/storage_vg-videos Mount Options : rw Inode Size : 256 Disk Space Free : 11.4TB Total Disk Space : 18.0TB Inode Count : 3865470528 Free Inodes : 3864561308 ------------------------------------------------------------------------------ Brick : Brick storage06:/mnt/storage/mp4 Port : 24010 Online : N Pid : N/A File System : N/A Device : N/A Mount Options : N/A Inode Size : N/A Disk Space Free : 0Bytes Total Disk Space : 0Bytes Inode Count : N/A Free Inodes : N/A As you can see, all bricks from storage06 are offiline. I tried to restart gluster daemon many times, I rebooted, and I even stopped all volumes and started them again but the same result. My main issue is that this is a production environment, and the whole load is going to storage05 (load 35 now and all CPUs 100% full) and I can't do a lot of trials on it. Your advice please? Thanks & Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121118/ad9887e4/attachment.html>