search for: libglusterf

...o happens when volumes started state. Below is the 5 stack traces taken by 10 min intervals and volumes stopped state. --1-- Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007f4146312d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f413e9a6700 (LWP 104250)): #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000040643b in glusterfs_sigwaiter () #2 0...

...too many nodes, the glusterd takes a quite long amount of time to finish the handshake. > > --1-- > Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)): > #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 > #1 0x00007f4146312d57 in gf_timer_proc () from > /usr/lib64/libglusterfs.so.0 > #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 > #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f413e9a6700 (LWP 104250)): > #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 > #1 0x000000000040643b in glu...

...the first stack trace is here: > >> >>> > >> >>> Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): > >> >>> #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 > >> >>> #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 > >> >>> #2 0x0000003aa5c07aa1 in start_thread () from > /lib64/libpthread.so.0 > >> >>> #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 > >> >>> Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): > >> >>> #0 0x0...

...seconds between each trace. https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 Content of the first stack trace is here: Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 #2 0x0000003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): #0 0x0000003aa5c0f585 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000040643b in glusterfs_sigwaiter () #2 0x...

...u can find pstack sampes here: https://www.dropbox.com/s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 Here is the first one: Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 #1 0x000000310fe37d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9286fad700 (LWP 78910)): #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000040643b in glusterfs_sigwaiter () #2 0x...

...https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 > > Content of the first stack trace is here: > > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): > #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 > #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 > #2 0x0000003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0 > #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): > #0 0x0000003aa5c0f585 in sigwait () from /lib64/libpthread.so.0 > #1 0x000000000040643b in glus...

...: > https://www.dropbox.com/s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 > > Here is the first one: > Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): > #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 > #1 0x000000310fe37d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 > #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 > #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f9286fad700 (LWP 78910)): > #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 > #1 0x000000000040643b in glus...

.../s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 > > > > Here is the first one: > > Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): > > #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 > > #1 0x000000310fe37d57 in gf_timer_proc () from > /usr/lib64/libglusterfs.so.0 > > #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 > > #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7f9286fad700 (LWP 78910)): > > #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 > > #1 0...

...?oban <cobanserkan at gmail.com> wrote: > Hi Atin, > > Do you have time to check the logs? > > On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> > wrote: > > Same thing happens with 3.12.rc0. This time perf top shows hanging in > > libglusterfs.so and below is the glusterd logs, which are different > > from 3.10. > > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see > > brick processes come online and system starts to answer commands like > > "gluster peer status".. > > > >...

...goq5xn3p1yt/glusterd_pstack.zip?dl=0 > > > > Content of the first stack trace is here: > > > > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): > > #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 > > #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 > > #2 0x0000003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0 > > #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): > > #0 0x0000003aa5c0f585 in sigwait () from /lib64/libpthread.so.0 > > #1 0...

Same thing happens with 3.12.rc0. This time perf top shows hanging in libglusterfs.so and below is the glusterd logs, which are different from 3.10. With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see brick processes come online and system starts to answer commands like "gluster peer status".. [2017-08-23 06:46:02.150472] E [client_t.c:324:gf_client_r...

...https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0 > > Content of the first stack trace is here: > > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): > #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 > #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 > #2 0x0000003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0 > #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): > #0 0x0000003aa5c0f585 in sigwait () from /lib64/libpthread.so.0 > #1 0x000000000040643b in glus...

...s the qemu log of instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (-- > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(...

...; > >> > Content of the first stack trace is here: > >> > > >> > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): > >> > #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 > >> > #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 > >> > #2 0x0000003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0 > >> > #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 > >> > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): > >> > #0 0x0000003aa5c0f585 in sigwait () from /l...

...gt; [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] > (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] > (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] > (-- >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--&gt...

...gt; wrote: > >> Hi Atin, >> >> Do you have time to check the logs? >> >> On Wed, Aug 23, 2017 at 10:02 AM, Serkan ?oban <cobanserkan at gmail.com> >> wrote: >> > Same thing happens with 3.12.rc0. This time perf top shows hanging in >> > libglusterfs.so and below is the glusterd logs, which are different >> > from 3.10. >> > With 3.10.5, after 60-70 minutes CPU usage becomes normal and we see >> > brick processes come online and system starts to answer commands like >> > "gluster peer status".. &gt...

...the Gluster run xfs_repair and bring it back, that would solve the issue. Any other suggestions? [2023-07-21 18:51:38.260507] W [inode.c:1638:inode_table_prune] (-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/features/shard.so(+0x21b47) [0x7fb261c13b47] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) [0x7fb26947f416] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) [0x7fb26947f37a] ) 0-GLB1image-shard: Empty inode lru list found but with (-2) lru_size [2023-07-21 18:51:38.261231] W [inode.c:1638:inode_table_prune] (-->/usr/lib/x86_64-linux-gnu/glusterf...

...uginfo.zip?dl=0 > > > > > > Here is the first one: > > > Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): > > > #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 > > > #1 0x000000310fe37d57 in gf_timer_proc () from > > > /usr/lib64/libglusterfs.so.0 > > > #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 > > > #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 > > > Thread 7 (Thread 0x7f9286fad700 (LWP 78910)): > > > #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so....

...ox.com/s/6gw8b6tng8puiox/pstack_with_debuginfo.zip?dl=0 > > > > Here is the first one: > > Thread 8 (Thread 0x7f92879ae700 (LWP 78909)): > > #0 0x0000003d99c0f00d in nanosleep () from /lib64/libpthread.so.0 > > #1 0x000000310fe37d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 > > #2 0x0000003d99c07aa1 in start_thread () from /lib64/libpthread.so.0 > > #3 0x0000003d998e8bbd in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7f9286fad700 (LWP 78910)): > > #0 0x0000003d99c0f585 in sigwait () from /lib64/libpthread.so.0 > > #1 0x000...

...the first stack trace is here: >>> >> > >>> >> > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)): >>> >> > #0 0x0000003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0 >>> >> > #1 0x000000303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0 >>> >> > #2 0x0000003aa5c07aa1 in start_thread () from >>> >> > /lib64/libpthread.so.0 >>> >> > #3 0x0000003aa58e8bbd in clone () from /lib64/libc.so.6 >>> >> > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)): >>>...