thr3ads.net - Ocfs2 users - [Ocfs2-users] Checking on the nature of these messages / errors [Oct 2015]

If this information is useful, please help other people find it:
Share via:

Jonathan Ramsay

2015-Oct-29 12:39 UTC

[Ocfs2-users] Checking on the nature of these messages / errors

Previous history : This cluster has been stable etc for months .

On two occasions this week , we have experienced one server lock up
resulting in many D state processes - primarily nfs - requiring a reboot .

26337 D    lookup_slow                              smbd26372 D
lookup_slow                              smbd26381 D
dlm_wait_for_lock_mastery                smbd26406 D    iterate_dir
                          smbd26417 D    iterate_dir
           bash26530 D    iterate_dir
smbd26557 D    iterate_dir                              smbd26761 D
iterate_dir                              smbd


Oct 26 12:05:01 server1 CRON[60021]: (root) CMD (command -v debian-sa1 >
/dev/null && debian-sa1 1 1)
Oct 26 12:12:27 server1 kernel: [1633038.930488] o2net: Connection to node
server2  (num 0) at 10.0.0.11:7777 has been idle for 30.5 secs, shutting it
down.
Oct 26 12:12:27 server1 kernel: [1633038.930579] o2net: No longer connected
to node server2  (num 0) at 10.0.0.11:7777
Oct 26 12:12:27 server1 kernel: [1633039.018762] o2net: Connection to node
server3  (num 2) at 10.0.0.30:7777 shutdown, state 8
Oct 26 12:12:27 server1 kernel: [1633039.018846] o2net: No longer connected
to node server3  (num 2) at 10.0.0.30:7777
Oct 26 12:12:27 server1 kernel: [1633039.018987]
(kworker/u192:2,59600,0):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
Oct 26 12:12:32 server1 kernel: [1633043.622980] o2net: Accepted connection
from node server3  (num 2) at 10.0.0.30:7777
Oct 26 12:12:32 server1 kernel: [1633043.623052] o2net: Connected to node
server2  (num 0) at 10.0.0.11:7777


Oct 28 12:01:35 server2  kernel: [158857.315968]
(kworker/u64:0,25700,8):dlm_send_remote_convert_request:392 ERROR: Error
-107 when sending message 504 (key 0x276c0b15) to node 1
Oct 28 12:01:35 server2  kernel: [158857.317060] o2dlm: Waiting on the
death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB
Oct 28 12:01:40 server2  kernel: [158862.288279]
(nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Error -107 when
sending message 504 (key 0xbd7d045e) to node 1
Oct 28 12:01:40 server2  kernel: [158862.289730] o2dlm: Waiting on the
death of node 1 in domain 71DEDAB8A10A4F849C60F454E41418FB
Oct 28 12:01:40 server2  kernel: [158862.420276]
(kworker/u64:0,25700,10):dlm_send_remote_convert_request:392 ERROR: Error
-107 when sending message 504 (key 0x276c0b15) to node 1
Oct 28 12:01:40 server2  kernel: [158862.421855] o2dlm: Waiting on the
death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB
Oct 28 12:01:45 server2  kernel: [158866.795872]
(smbd,6126,8):dlm_do_master_request:1344 ERROR: link to 1 went down!
Oct 28 12:01:45 server2  kernel: [158866.796643]
(smbd,6126,8):dlm_get_lock_resource:929 ERROR: status = -107
Oct 28 12:01:45 server2  kernel: [158867.392484]
(nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Er


Oct 26 12:12:46 server2  kernel: [441237.624180]
(smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!
Oct 26 12:12:46 server2  kernel: [441237.624181]
(smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107
Oct 26 12:12:46 server2  kernel: [441237.624832]
(smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!
Oct 26 12:12:46 server2  kernel: [441237.624833]
(smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107
Oct 26 12:12:46 server2  kernel: [441237.914234]
(smbd,16974,11):dlm_get_lock_resource:929 ERROR: status = -112
Oct 26 12:12:50 server2  kernel: [441242.308190] o2net: Accepted connection
from node server1  (num 1) at 10.0.0.80:7777
Oct 26 12:15:01 server2  CRON[17178]: (root) CMD (command -v debian-sa1 >
/dev/null && debian-sa1 1 1)

Oct 28 12:01:04 server1  kernel: [1805223.124152] o2net: Connection to node
server2  (num 0) at 10.0.0.11:7777 shutdown, state 8
Oct 28 12:01:04 server1  kernel: [1805223.124252] o2net: No longer
connected to node server2  (num 0) at 10.0.0.11:7777
Oct 28 12:01:04 server1  kernel: [1805223.124285]
(python,7983,0):dlm_do_master_request:1344 ERROR: link to 0 went down!
Oct 28 12:01:04 server1  kernel: [1805223.124563]
(kworker/u192:1,9377,4):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
Oct 28 12:01:04 server1  kernel: [1805223.145675]
(python,7983,0):dlm_get_lock_resource:929 ERROR: status = -112
Oct 28 12:01:04 server1  kernel: [1805223.237950] o2net: Connection to node
server3  (num 2) at 10.0.0.30:7777 shutdown, state 8
Oct 28 12:01:04 server1  kernel: [1805223.237993] o2net: No longer
connected to node server3  (num 2) at 10.0.0.30:7777
Oct 28 12:01:04 server1  kernel: [1805223.238067]
(python,7983,2):dlm_do_master_request:1344 ERROR: link to 2 went down!
Oct 28 12:01:04 server1  kernel: [1805223.238078]
(kworker/u192:1,9377,12):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
Oct 28 12:01:04 server1  kernel: [1805223.238118]
(dlm_thread,4881,6):dlm_send_proxy_ast_msg:482 ERROR:
71DEDAB8A10A4F849C60F454E41418FB: res P000000000000000000000000000000,
error -112 send AST to node 2
Oct 28 12:01:04 server1  kernel: [1805223.238121]
(dlm_thread,4881,6):dlm_flush_asts:596 ERROR: status = -112


I am investigating a perc controller as we had a battery fail message on
server1 during reboot , but wanted some more input on these errors as well.

Thank you for your time ,

Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20151029/8296c8fd/attachment.html

Joseph Qi

2015-Oct-30 00:56 UTC

head link

[Ocfs2-users] Checking on the nature of these messages / errors

Hi Jon,>From the information you provided, I can see:Your ocfs2 cluster has 3 nodes configured, which are node 0, 1, 2.
And the network of node 1 (host name is server1) has been down and then
result in the -ENOTCONN(-107) and -EHOSTDOWN(-112) errors when sending
dlm messages.

On 2015/10/29 20:39, Jonathan Ramsay wrote:> Previous history : This cluster has been stable etc for months . 
> 
> On two occasions this week , we have experienced one server lock up
resulting in many D state processes - primarily nfs - requiring a reboot .
> 
> 26337 D    lookup_slow                              smbd
> 26372 D    lookup_slow                              smbd
> 26381 D    dlm_wait_for_lock_mastery                smbd
> 26406 D    iterate_dir                              smbd
> 26417 D    iterate_dir                              bash
> 26530 D    iterate_dir                              smbd
> 26557 D    iterate_dir                              smbd
> 26761 D    iterate_dir                              smbd
> 
> 
> Oct 26 12:05:01 server1 CRON[60021]: (root) CMD (command -v debian-sa1 >
/dev/null && debian-sa1 1 1)
> Oct 26 12:12:27 server1 kernel: [1633038.930488] o2net: Connection to node
server2  (num 0) at 10.0.0.11:7777 <http://10.0.0.11:7777> has been idle
for 30.5 secs, shutting it down.
> Oct 26 12:12:27 server1 kernel: [1633038.930579] o2net: No longer connected
to node server2  (num 0) at 10.0.0.11:7777 <http://10.0.0.11:7777>
> Oct 26 12:12:27 server1 kernel: [1633039.018762] o2net: Connection to node
server3  (num 2) at 10.0.0.30:7777 <http://10.0.0.30:7777> shutdown, state
8
> Oct 26 12:12:27 server1 kernel: [1633039.018846] o2net: No longer connected
to node server3  (num 2) at 10.0.0.30:7777 <http://10.0.0.30:7777>
> Oct 26 12:12:27 server1 kernel: [1633039.018987]
(kworker/u192:2,59600,0):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
> Oct 26 12:12:32 server1 kernel: [1633043.622980] o2net: Accepted connection
from node server3  (num 2) at 10.0.0.30:7777 <http://10.0.0.30:7777>
> Oct 26 12:12:32 server1 kernel: [1633043.623052] o2net: Connected to node
server2  (num 0) at 10.0.0.11:7777 <http://10.0.0.11:7777>
> 
> 
> Oct 28 12:01:35 server2  kernel: [158857.315968]
(kworker/u64:0,25700,8):dlm_send_remote_convert_request:392 ERROR: Error -107
when sending message 504 (key 0x276c0b15) to node 1
> Oct 28 12:01:35 server2  kernel: [158857.317060] o2dlm: Waiting on the
death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB
> Oct 28 12:01:40 server2  kernel: [158862.288279]
(nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Error -107 when sending
message 504 (key 0xbd7d045e) to node 1
> Oct 28 12:01:40 server2  kernel: [158862.289730] o2dlm: Waiting on the
death of node 1 in domain 71DEDAB8A10A4F849C60F454E41418FB
> Oct 28 12:01:40 server2  kernel: [158862.420276]
(kworker/u64:0,25700,10):dlm_send_remote_convert_request:392 ERROR: Error -107
when sending message 504 (key 0x276c0b15) to node 1
> Oct 28 12:01:40 server2  kernel: [158862.421855] o2dlm: Waiting on the
death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB
> Oct 28 12:01:45 server2  kernel: [158866.795872]
(smbd,6126,8):dlm_do_master_request:1344 ERROR: link to 1 went down!
> Oct 28 12:01:45 server2  kernel: [158866.796643]
(smbd,6126,8):dlm_get_lock_resource:929 ERROR: status = -107
> Oct 28 12:01:45 server2  kernel: [158867.392484]
(nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Er
> 
> 
> Oct 26 12:12:46 server2  kernel: [441237.624180]
(smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!
> Oct 26 12:12:46 server2  kernel: [441237.624181]
(smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107
> Oct 26 12:12:46 server2  kernel: [441237.624832]
(smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!
> Oct 26 12:12:46 server2  kernel: [441237.624833]
(smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107
> Oct 26 12:12:46 server2  kernel: [441237.914234]
(smbd,16974,11):dlm_get_lock_resource:929 ERROR: status = -112
> Oct 26 12:12:50 server2  kernel: [441242.308190] o2net: Accepted connection
from node server1  (num 1) at 10.0.0.80:7777 <http://10.0.0.80:7777>
> Oct 26 12:15:01 server2  CRON[17178]: (root) CMD (command -v debian-sa1
> /dev/null && debian-sa1 1 1)
> 
> Oct 28 12:01:04 server1  kernel: [1805223.124152] o2net: Connection to node
server2  (num 0) at 10.0.0.11:7777 <http://10.0.0.11:7777> shutdown, state
8
> Oct 28 12:01:04 server1  kernel: [1805223.124252] o2net: No longer
connected to node server2  (num 0) at 10.0.0.11:7777
<http://10.0.0.11:7777>
> Oct 28 12:01:04 server1  kernel: [1805223.124285]
(python,7983,0):dlm_do_master_request:1344 ERROR: link to 0 went down!
> Oct 28 12:01:04 server1  kernel: [1805223.124563]
(kworker/u192:1,9377,4):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
> Oct 28 12:01:04 server1  kernel: [1805223.145675]
(python,7983,0):dlm_get_lock_resource:929 ERROR: status = -112
> Oct 28 12:01:04 server1  kernel: [1805223.237950] o2net: Connection to node
server3  (num 2) at 10.0.0.30:7777 <http://10.0.0.30:7777> shutdown, state
8
> Oct 28 12:01:04 server1  kernel: [1805223.237993] o2net: No longer
connected to node server3  (num 2) at 10.0.0.30:7777
<http://10.0.0.30:7777>
> Oct 28 12:01:04 server1  kernel: [1805223.238067]
(python,7983,2):dlm_do_master_request:1344 ERROR: link to 2 went down!
> Oct 28 12:01:04 server1  kernel: [1805223.238078]
(kworker/u192:1,9377,12):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32
instead of 24
> Oct 28 12:01:04 server1  kernel: [1805223.238118]
(dlm_thread,4881,6):dlm_send_proxy_ast_msg:482 ERROR:
71DEDAB8A10A4F849C60F454E41418FB: res P000000000000000000000000000000, error
-112 send AST to node 2
> Oct 28 12:01:04 server1  kernel: [1805223.238121]
(dlm_thread,4881,6):dlm_flush_asts:596 ERROR: status = -112
> 
> 
> I am investigating a perc controller as we had a battery fail message on
server1 during reboot , but wanted some more input on these errors as well.
> 
> Thank you for your time , 
> 
> Jon 
> 
> 
> 
> 
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-users
>

Ocfs2 users - Oct 2015 - Checking on the nature of these messages / errors

[Ocfs2-users] Checking on the nature of these messages / errors

[Ocfs2-users] Checking on the nature of these messages / errors