Attempting to lookup why rsyslogd is listening on the high port UDP/51427. Have not succeeded in what this port is used for and what directive controls what interface it binds to. [root at bedrock ~]# netstat --listen --inet --program --numeric | grep syslog udp??0??0 0.0.0.0:51427??0.0.0.0:*???66655/rsyslogd? -- Adam Tauno Williams <mailto:awilliam at whitemice.org> GPG D95ED383 OpenGroupware Developer <http://www.opengroupware.us/>
On Thu, 6 Sep 2018 at 14:11, Adam Tauno Williams <awilliam at whitemice.org> wrote:> > Attempting to lookup why rsyslogd is listening on the high port > UDP/51427. Have not succeeded in what this port is used for and what > directive controls what interface it binds to. > > [root at bedrock ~]# netstat --listen --inet --program --numeric | grep syslog > udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd >I don't think that is really rsyslogd running.. first rsyslogd usually is running with a much lower pid as it starts early on. All the boxes I saw had it running under 1000. If the system is set up to actually listen to the internet it will look like [root at log01 ~][PROD]# netstat --list --inet --program --numeric | grep rsys tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 1078/rsyslogd tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 1078/rsyslogd udp 0 0 0.0.0.0:514 0.0.0.0:* 1078/rsyslogd and those are set in /etc/rsyslogd.conf [root at log01 ~][PROD]# egrep '514|5000' /etc/rsyslog.conf $UDPServerRun 514 $InputTCPServerRun 514 $InputTCPServerRun 5000 I would do a ps auxww | grep 66655 and see what is running and then check to see if that binary is what it is supposed to be.> -- > Adam Tauno Williams <mailto:awilliam at whitemice.org> GPG D95ED383 > OpenGroupware Developer <http://www.opengroupware.us/> > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- Stephen J Smoogen.
On 2018-09-06 14:06, Adam Tauno Williams wrote:> Attempting to lookup why rsyslogd is listening on the high port > UDP/51427. Have not succeeded in what this port is used for and what > directive controls what interface it binds to. > > [root at bedrock ~]# netstat --listen --inet --program --numeric | grep > syslog > udp??0??0 0.0.0.0:51427??0.0.0.0:*???66655/rsyslogd?Adam, You might want to try running: lsof -i -P | grep LISTEN | grep :51427 to determine what process is actually listening to that port. -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
On Thu, 2018-09-06 at 15:10 -0400, Mike Burger wrote:> On 2018-09-06 14:06, Adam Tauno Williams wrote: > > Attempting to lookup why rsyslogd is listening on the high port > > UDP/51427.????Have not succeeded in what this port is used for and > > what directive controls what interface it binds to. > > [root at bedrock ~]# netstat --listen --inet --program --numeric | > > grep?syslog > > udp??0??0 0.0.0.0:51427??0.0.0.0:*???66655/rsyslogd? > You might want to try running: > lsof -i -P | grep LISTEN | grep :51427 > to determine what process is actually listening to that port.That is what is strange; lsof does *not* see the port as listening, but it is visible if I search by inode [and it is clearly rsyslogd - but I cannot find any documentation indicating what this is]. [root at bedrock ~]# netstat --inet --program --numeric --listen | grep syslog udp????????0??????0 0.0.0.0:51427???????????0.0.0.0:*????????66655/rsyslogd [root at bedrock ~]# lsof -i -P | grep LISTEN? rpcbind?????810??????rpc????8u??IPv4???19806??????0t0??TCP *:111 (LISTEN) rpcbind?????810??????rpc???11u??IPv6???19809??????0t0??TCP *:111 (LISTEN) sshd???????1156?????root????3u??IPv4???23045??????0t0??TCP *:22 (LISTEN) sshd???????1156?????root????4u??IPv6???23047??????0t0??TCP *:22 (LISTEN) snmpd??????1158?????root????8u??IPv4???26937??????0t0??TCP localhost:199 (LISTEN) rpc.statd??1196??rpcuser????9u??IPv4???24024??????0t0??TCP *:662 (LISTEN) rpc.statd??1196??rpcuser???11u??IPv6???24030??????0t0??TCP *:662 (LISTEN) avagent.b??1431?????root????4u??IPv6???26892??????0t0??TCP *:28002 (LISTEN) avagent.b??1431?????root????6u??IPv6???28867??????0t0??TCP localhost:38061 (LISTEN) master?????1535?????root???13u??IPv4???26579??????0t0??TCP localhost:25 (LISTEN) master?????1535?????root???14u??IPv6???26580??????0t0??TCP localhost:25 (LISTEN) smbd???????1663?????root???35u??IPv6???28676??????0t0??TCP *:445 (LISTEN) smbd???????1663?????root???36u??IPv4???28677??????0t0??TCP *:445 (LISTEN) [root at bedrock ~]# netstat -e --inet --program --numeric --listen | grep syslog udp????????0??????0 0.0.0.0:51427???????????0.0.0.0:*???0????5032773????66655/rsyslogd [root at bedrock ~]# lsof | awk 'NR==1 || /5032773/' COMMAND?????PID???TID?????USER???FD???TYPE????????DEVICE??SIZE/OFF??NODE NAME rsyslogd??66655???????????root????4u??IPv4???????5032773???????0t0??UDP *:51427? in:imjour 66655 66657?????root????4u??IPv4???????5032773???????0t0??UDP *:51427? rs:main???66655 66658?????root????4u??IPv4???????5032773???????0t0??UDP *:51427? The port is closed on the firewall; but I file a report which includes every listening port and its justification. NOTE: rsyslog exhibits this behavior on every CentOS6 & CentOS7 host. And all our host log via rsyslogd via UDP/514 to a central NMS with a syslog receiver. -- Adam Tauno Williams <mailto:awilliam at whitemice.org> GPG D95ED383 OpenGroupware Developer <http://www.opengroupware.us/>
On Thu, Sep 06, 2018 at 02:06:37PM -0400, Adam Tauno Williams wrote:> > Attempting to lookup why rsyslogd is listening on the high port > UDP/51427. Have not succeeded in what this port is used for and what > directive controls what interface it binds to. > > [root at bedrock ~]# netstat --listen --inet --program --numeric | grep syslog > udp??0??0 0.0.0.0:51427??0.0.0.0:*???66655/rsyslogd?The 51427 is the ephemeral port on the client side of the UDP session. You can verify this by running tcpdump to capture traffic when a syslog message is passed. I can report that I also see this netstat (and similar with ss) state for systems with rsyslog set up to send to a remote log server, where ss reports that the process has UNCONN state on high UDP ports. I suspect it's part of the UDP handshake that rsyslog uses for sending syslogs, but I'm not familiar enough with how it works to say definitively. Since it's UDP, it's a sessionless protocol, so it's not strictly LISTENing, but with ss you can see it's UNCONN, which other daemons that *are* listening for UDP traffic also report. It is quite interesting to me, and if anyone knows why this works this way, I'd be happy to hear. I did some tests with 'nc -u' and I couldn't get similar results. -- Jonathan Billings <billings at negate.org>
On Fri, 2018-09-07 at 09:20 -0400, Jonathan Billings wrote:> > [root at bedrock ~]# netstat --listen --inet --program --numeric | > > grep syslog > > udp??0??0 0.0.0.0:51427??0.0.0.0:*???66655/rsyslogd? > The 51427 is the ephemeral port on the client side of the UDP > session.??You can verify this by running tcpdump to capture traffic > when a syslog message is passed.Thanks, I was suspecting something like this as the only way to make that port disappear was to disable remote logging.> I suspect it's part of the UDP handshake that rsyslog uses for > sending syslogs, but I'm not familiar enough with how it works to say > definitively.?It was puzzling because I can't find any reference to this behavior in any documentation.> ?Since it's UDP, it's a sessionless protocol, so it's > not strictly LISTENing, but with ss you can see it's UNCONN, which > other daemons that *are* listening for UDP traffic also report.Right, distinguishing between the listening and open in UDP is always somewhere between tedious and impossible. Perhaps I should investigate logging over TCP! :) Anyway, I have something to write in the report now. -- Meetings Coordinator, Michigan Association of Railroad Passengers 537 Shirley St NE Grand Rapids, MI 49503-1754 Phone: 616.581.8010 E-mail: awilliam at whitemice.org GPG#D95ED383 Web: http://www.marp.org
On Fri, 7 Sep 2018 at 09:20, Jonathan Billings <billings at negate.org> wrote:> > On Thu, Sep 06, 2018 at 02:06:37PM -0400, Adam Tauno Williams wrote: > > > > Attempting to lookup why rsyslogd is listening on the high port > > UDP/51427. Have not succeeded in what this port is used for and what > > directive controls what interface it binds to. > > > > [root at bedrock ~]# netstat --listen --inet --program --numeric | grep syslog > > udp 0 0 0.0.0.0:51427 0.0.0.0:* 66655/rsyslogd > > The 51427 is the ephemeral port on the client side of the UDP > session. You can verify this by running tcpdump to capture traffic > when a syslog message is passed. >That makes perfect sense. I had checked several boxes on our side and had not found any with that port being shown but we use TCP for logging so I expect that is why. My apologies to Mr Billings for going for the worst possible scenario.> I can report that I also see this netstat (and similar with ss) state > for systems with rsyslog set up to send to a remote log server, where > ss reports that the process has UNCONN state on high UDP ports. > > I suspect it's part of the UDP handshake that rsyslog uses for sending > syslogs, but I'm not familiar enough with how it works to say > definitively. Since it's UDP, it's a sessionless protocol, so it's > not strictly LISTENing, but with ss you can see it's UNCONN, which > other daemons that *are* listening for UDP traffic also report. > > It is quite interesting to me, and if anyone knows why this works this > way, I'd be happy to hear. I did some tests with 'nc -u' and I > couldn't get similar results. > > > -- > Jonathan Billings <billings at negate.org> > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- Stephen J Smoogen.