Gluster users - Jul 2013 - new glusterfs logging framework

Hi All,

Recently new logging framework was introduced [1][2][3] in glusterfs master
branch.  You could read more about this on doc/logging.txt.  In brief, current
log target is moved to syslog and user has an option to this new logging at
compile time (passing '--disable-syslog' to ./configure or
'--without syslog' to rpmbuild) and run time (having a file
/var/log/glusterd/logger.conf and restarting gluster services).

As rsyslog is used as syslog server in Fedora and CentOS/RHEL and default
configuration of rsyslog does not have any rule specific to gluster logs, you
see all logs are in /var/log/messages in JSON format.

Below is the way to make them neat and clean.

For fedora users:
1. It requires to install rsyslog-mmjsonparse rpm (yum -y install
rsyslog-mmjsonparse)
2. Place below configuration under /etc/rsyslog.d/gluster.conf file.

#$RepeatedMsgReduction on

$ModLoad mmjsonparse
*.* :mmjsonparse:

template (name="GlusterLogFile" type="string"
string="/var/log/gluster/%app-name%.log")
template (name="GlusterPidLogFile" type="string"
string="/var/log/gluster/%app-name%-%procid%.log")

template(name="GLFS_template" type="list") {
   property(name="$!mmcount")
   constant(value="/")
   property(name="syslogfacility-text"
caseConversion="upper")
   constant(value="/")
   property(name="syslogseverity-text"
caseConversion="upper")
   constant(value=" ")
   constant(value="[")
   property(name="timereported" dateFormat="rfc3339")
   constant(value="] ")
   constant(value="[")
   property(name="$!gf_code")
   constant(value="] ")
   constant(value="[")
   property(name="$!gf_message")
   constant(value="] ")
   property(name="$!msg")
   constant(value="\n")
}

if $app-name == 'gluster' or $app-name == 'glusterd' then {
   action(type="omfile"
          DynaFile="GlusterLogFile"
          Template="GLFS_template")
   stop
}

if $app-name contains 'gluster' then {
   action(type="omfile"
          DynaFile="GlusterPidLogFile"
          Template="GLFS_template")
   stop
}


3. Restart rsyslog (service rsyslog restart)
4. Done. All gluster process specific logs are separated into /var/log/gluster/
directory


Note: Fedora 19 users
There is a bug in rsyslog of fedora 19 [4], so its required to recompile rsyslog
source rpm downloaded from fedora repository ('rpmbuild --rebuild
rsyslog-7.2.6-1.fc19.src.rpm' works fine) and use generated rsyslog and
rsyslog-mmjsonparse binary rpms

For CentOS/RHEL users:
Current rsyslog available in CentOS/RHEL does not have json support.  I have
added the support which requires some testing.  I will update once done.


TODO:
1. need to add volume:brick specific tag to logging so that those logs can be
separated out than pid.
2. enable gfapi to use this logging framework


I would like to get feedback/suggestion about this logging framework


Regards,
Bala


[1] http://review.gluster.org/4977
[2] http://review.gluster.org/5002
[3] http://review.gluster.org/4915
[4] https://bugzilla.redhat.com/show_bug.cgi?id=989886

I think that adding all that 'rsyslog' configuration only to see logs is
too much. (I admit it, I don't know how to configure rsyslog at that 
level so that may influence my opinion)

Regards,


El 30/07/2013 06:29 a.m., Balamurugan Arumugam escribi?:
> Hi All,
>
> Recently new logging framework was introduced [1][2][3] in glusterfs master
branch.  You could read more about this on doc/logging.txt.  In brief, current
log target is moved to syslog and user has an option to this new logging at
compile time (passing '--disable-syslog' to ./configure or
'--without syslog' to rpmbuild) and run time (having a file
/var/log/glusterd/logger.conf and restarting gluster services).
>
> As rsyslog is used as syslog server in Fedora and CentOS/RHEL and default
configuration of rsyslog does not have any rule specific to gluster logs, you
see all logs are in /var/log/messages in JSON format.
>
> Below is the way to make them neat and clean.
>
> For fedora users:
> 1. It requires to install rsyslog-mmjsonparse rpm (yum -y install
rsyslog-mmjsonparse)
> 2. Place below configuration under /etc/rsyslog.d/gluster.conf file.
>
> #$RepeatedMsgReduction on
>
> $ModLoad mmjsonparse
> *.* :mmjsonparse:
>
> template (name="GlusterLogFile" type="string"
string="/var/log/gluster/%app-name%.log")
> template (name="GlusterPidLogFile" type="string"
string="/var/log/gluster/%app-name%-%procid%.log")
>
> template(name="GLFS_template" type="list") {
>     property(name="$!mmcount")
>     constant(value="/")
>     property(name="syslogfacility-text"
caseConversion="upper")
>     constant(value="/")
>     property(name="syslogseverity-text"
caseConversion="upper")
>     constant(value=" ")
>     constant(value="[")
>     property(name="timereported" dateFormat="rfc3339")
>     constant(value="] ")
>     constant(value="[")
>     property(name="$!gf_code")
>     constant(value="] ")
>     constant(value="[")
>     property(name="$!gf_message")
>     constant(value="] ")
>     property(name="$!msg")
>     constant(value="\n")
> }
>
> if $app-name == 'gluster' or $app-name == 'glusterd' then {
>     action(type="omfile"
>            DynaFile="GlusterLogFile"
>            Template="GLFS_template")
>     stop
> }
>
> if $app-name contains 'gluster' then {
>     action(type="omfile"
>            DynaFile="GlusterPidLogFile"
>            Template="GLFS_template")
>     stop
> }
>
>
> 3. Restart rsyslog (service rsyslog restart)
> 4. Done. All gluster process specific logs are separated into
/var/log/gluster/ directory
>
>
> Note: Fedora 19 users
> There is a bug in rsyslog of fedora 19 [4], so its required to recompile
rsyslog source rpm downloaded from fedora repository ('rpmbuild --rebuild
rsyslog-7.2.6-1.fc19.src.rpm' works fine) and use generated rsyslog and
rsyslog-mmjsonparse binary rpms
>
> For CentOS/RHEL users:
> Current rsyslog available in CentOS/RHEL does not have json support.  I
have added the support which requires some testing.  I will update once done.
>
>
> TODO:
> 1. need to add volume:brick specific tag to logging so that those logs can
be separated out than pid.
> 2. enable gfapi to use this logging framework
>
>
> I would like to get feedback/suggestion about this logging framework
>
>
> Regards,
> Bala
>
>
> [1] http://review.gluster.org/4977
> [2] http://review.gluster.org/5002
> [3] http://review.gluster.org/4915
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=989886
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>

On Tue, Jul 30, 2013 at 05:29:43AM -0400, Balamurugan Arumugam
wrote:
> 
> Hi All,
> 
> Recently new logging framework was introduced [1][2][3] in glusterfs 
> master branch.  You could read more about this on doc/logging.txt.  In 
> brief, current log target is moved to syslog and user has an option to 
> this new logging at compile time (passing '--disable-syslog' to 
> ./configure or '--without syslog' to rpmbuild) and run time (having
> a file /var/log/glusterd/logger.conf and restarting gluster services).
> 
> As rsyslog is used as syslog server in Fedora and CentOS/RHEL and 
> default configuration of rsyslog does not have any rule specific to 
> gluster logs, you see all logs are in /var/log/messages in JSON 
> format.
As the below does not function on RHEL-6 with its own rsyslog (no 
rsyslog-mmjsonparse available), I am creating an empty 
/var/lib/glusterd/logger.conf which (after a restart) makes glusterd and 
friends to log to their standard files again.

Do you have some progress on the rsyslog-mmjsonparse for rhel5 and 
rhel6?

Also, it would really be nice to include that 
/etc/rsyslog.d/gluster.conf in the rpms and add a dependency on the 
rsyslog-mmjsonparse package. Or, provide a /var/lib/glusterd/logger.conf 
for the distributions that do not have rsyslog-mmjsonparse.

Thanks,
Niels
> 
> Below is the way to make them neat and clean.
> 
> For fedora users:
> 1. It requires to install rsyslog-mmjsonparse rpm (yum -y install
rsyslog-mmjsonparse)
> 2. Place below configuration under /etc/rsyslog.d/gluster.conf file.
> 
> #$RepeatedMsgReduction on
> 
> $ModLoad mmjsonparse
> *.* :mmjsonparse:
> 
> template (name="GlusterLogFile" type="string"
string="/var/log/gluster/%app-name%.log")
> template (name="GlusterPidLogFile" type="string"
string="/var/log/gluster/%app-name%-%procid%.log")
> 
> template(name="GLFS_template" type="list") {
>    property(name="$!mmcount")
>    constant(value="/")
>    property(name="syslogfacility-text"
caseConversion="upper")
>    constant(value="/")
>    property(name="syslogseverity-text"
caseConversion="upper")
>    constant(value=" ")
>    constant(value="[")
>    property(name="timereported" dateFormat="rfc3339")
>    constant(value="] ")
>    constant(value="[")
>    property(name="$!gf_code")
>    constant(value="] ")
>    constant(value="[")
>    property(name="$!gf_message")
>    constant(value="] ")
>    property(name="$!msg")
>    constant(value="\n")
> }
> 
> if $app-name == 'gluster' or $app-name == 'glusterd' then {
>    action(type="omfile"
>           DynaFile="GlusterLogFile"
>           Template="GLFS_template")
>    stop
> }
> 
> if $app-name contains 'gluster' then {
>    action(type="omfile"
>           DynaFile="GlusterPidLogFile"
>           Template="GLFS_template")
>    stop
> }
> 
> 
> 3. Restart rsyslog (service rsyslog restart)
> 4. Done. All gluster process specific logs are separated into
/var/log/gluster/ directory
> 
> 
> Note: Fedora 19 users
> There is a bug in rsyslog of fedora 19 [4], so its required to recompile
rsyslog source rpm downloaded from fedora repository ('rpmbuild --rebuild
rsyslog-7.2.6-1.fc19.src.rpm' works fine) and use generated rsyslog and
rsyslog-mmjsonparse binary rpms
> 
> For CentOS/RHEL users:
> Current rsyslog available in CentOS/RHEL does not have json support.  I
have added the support which requires some testing.  I will update once done.
> 
> 
> TODO:
> 1. need to add volume:brick specific tag to logging so that those logs can
be separated out than pid.
> 2. enable gfapi to use this logging framework
> 
> 
> I would like to get feedback/suggestion about this logging framework
> 
> 
> Regards,
> Bala
> 
> 
> [1] http://review.gluster.org/4977
> [2] http://review.gluster.org/5002
> [3] http://review.gluster.org/4915
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=989886
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
-- 
Niels de Vos
Sr. Software Maintenance Engineer
Support Engineering Group
Red Hat Global Support Services

----- Original Message -----
> From: "Niels de Vos"
<ndevos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Balamurugan Arumugam"
<bala-+FkPdpiNhgJBDgjK7y7TUQ@public.gmane.org>
> Cc: gluster-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org,
gluster-users-+FkPdpiNhgJAfugRpC6u6w@public.gmane.org
> Sent: Tuesday, August 6, 2013 8:48:58 PM
> Subject: Re: [Gluster-devel] new glusterfs logging framework
> 
> On Tue, Jul 30, 2013 at 05:29:43AM -0400, Balamurugan Arumugam wrote:
> > 
> > Hi All,
> > 
> > Recently new logging framework was introduced [1][2][3] in glusterfs
> > master branch.  You could read more about this on doc/logging.txt.  In
> > brief, current log target is moved to syslog and user has an option to
> > this new logging at compile time (passing
''--disable-syslog'' to
> > ./configure or ''--without syslog'' to rpmbuild) and
run time (having
> > a file /var/log/glusterd/logger.conf and restarting gluster services).
> > 
> > As rsyslog is used as syslog server in Fedora and CentOS/RHEL and
> > default configuration of rsyslog does not have any rule specific to
> > gluster logs, you see all logs are in /var/log/messages in JSON
> > format.
> 
> As the below does not function on RHEL-6 with its own rsyslog (no
> rsyslog-mmjsonparse available), I am creating an empty
> /var/lib/glusterd/logger.conf which (after a restart) makes glusterd and
> friends to log to their standard files again.
> 
> Do you have some progress on the rsyslog-mmjsonparse for rhel5 and
> rhel6?
> 
mmcount is a rsyslog plugin (developed by me) available rsyslog upstream which
tracks each gluster log using error codes.  The error code is the entity enabled
by gluster logging framework.  This module is back ported to rhel6 which does
tracking and json parsing.  I am in the process of submitting this to rhel 6
downstream.  I haven''t tried rhel5 yet and I will do that too.

> Also, it would really be nice to include that
> /etc/rsyslog.d/gluster.conf in the rpms and add a dependency on the
> rsyslog-mmjsonparse package. Or, provide a /var/lib/glusterd/logger.conf
> for the distributions that do not have rsyslog-mmjsonparse.
> 
Yes.  I had similar thought on this to do so.  I will send the fix soon.

Thanks,

Regards,
Bala

> Thanks,
> Niels
> 
> > 
> > Below is the way to make them neat and clean.
> > 
> > For fedora users:
> > 1. It requires to install rsyslog-mmjsonparse rpm (yum -y install
> > rsyslog-mmjsonparse)
> > 2. Place below configuration under /etc/rsyslog.d/gluster.conf file.
> > 
> > #$RepeatedMsgReduction on
> > 
> > $ModLoad mmjsonparse
> > *.* :mmjsonparse:
> > 
> > template (name="GlusterLogFile" type="string"
> > string="/var/log/gluster/%app-name%.log")
> > template (name="GlusterPidLogFile" type="string"
> > string="/var/log/gluster/%app-name%-%procid%.log")
> > 
> > template(name="GLFS_template" type="list") {
> >    property(name="$!mmcount")
> >    constant(value="/")
> >    property(name="syslogfacility-text"
caseConversion="upper")
> >    constant(value="/")
> >    property(name="syslogseverity-text"
caseConversion="upper")
> >    constant(value=" ")
> >    constant(value="[")
> >    property(name="timereported"
dateFormat="rfc3339")
> >    constant(value="] ")
> >    constant(value="[")
> >    property(name="$!gf_code")
> >    constant(value="] ")
> >    constant(value="[")
> >    property(name="$!gf_message")
> >    constant(value="] ")
> >    property(name="$!msg")
> >    constant(value="\n")
> > }
> > 
> > if $app-name == ''gluster'' or $app-name ==
''glusterd'' then {
> >    action(type="omfile"
> >           DynaFile="GlusterLogFile"
> >           Template="GLFS_template")
> >    stop
> > }
> > 
> > if $app-name contains ''gluster'' then {
> >    action(type="omfile"
> >           DynaFile="GlusterPidLogFile"
> >           Template="GLFS_template")
> >    stop
> > }
> > 
> > 
> > 3. Restart rsyslog (service rsyslog restart)
> > 4. Done. All gluster process specific logs are separated into
> > /var/log/gluster/ directory
> > 
> > 
> > Note: Fedora 19 users
> > There is a bug in rsyslog of fedora 19 [4], so its required to
recompile
> > rsyslog source rpm downloaded from fedora repository
(''rpmbuild --rebuild
> > rsyslog-7.2.6-1.fc19.src.rpm'' works fine) and use generated
rsyslog and
> > rsyslog-mmjsonparse binary rpms
> > 
> > For CentOS/RHEL users:
> > Current rsyslog available in CentOS/RHEL does not have json support. 
I
> > have added the support which requires some testing.  I will update
once
> > done.
> > 
> > 
> > TODO:
> > 1. need to add volume:brick specific tag to logging so that those logs
can
> > be separated out than pid.
> > 2. enable gfapi to use this logging framework
> > 
> > 
> > I would like to get feedback/suggestion about this logging framework
> > 
> > 
> > Regards,
> > Bala
> > 
> > 
> > [1] http://review.gluster.org/4977
> > [2] http://review.gluster.org/5002
> > [3] http://review.gluster.org/4915
> > [4] https://bugzilla.redhat.com/show_bug.cgi?id=989886
> > 
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org
> > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> 
> --
> Niels de Vos
> Sr. Software Maintenance Engineer
> Support Engineering Group
> Red Hat Global Support Services
>