CentOS - Apr 2015 - Access Problem after update to CentOS 7.1

Everyone,

This morning I did a manual yum update on our a mail server to 7.1
without any incident or problems.  A new kernel was installed, and I
rebooted after the update.  

When I rebooted the machine I could not gain ssh access to it from an
external ip address.  I was able to ssh to this mail server through a
different machine on the local network.  

At first I thought the problem was related to the firewall.  I stopped
firewalld, and fail2ban, and clear all firewall rules without being able
to gain access.

I disabled firewalld, and fail2ban.  I enabled iptables and started it
without a problem, but I could still not gain access.  I removed all
entries in the host.allow and host.deny files, and this did not make a
difference either.  

On one of the various reboots I tried to use the previous kernel before
today's update, but there was no success.

I can scan the mail server and reach it without a problem from the
internal network but I am not able to reach it from outside the local
network.  I have the mail server behind a Centso 5.11 machine that is
the gateway router for the internal network, and the mail server is nat
addressed with it's external ip address to the internal machine.  I have
had this configuration set up for over 7 years.  I tweaked the Gateway
router to nat address the mail server's ip address to a different
machine inside the network and everything worked perfectly like it
should, and then re-adjusted the gateway router again back to the mail
server and am not able to gain access from outside the local network.

"traceroute" does not get to the mail server from outside the local
network, but works fine inside the local network.

Bottom line, this does not look like a host.deny, host.allow problem,
nor does it look like a firewalld or iptables problem.  And it does not
appear to be a problem with the gateway server.  

Is there another feature of CentOs 7.1 that I need to evaluate?  Has
anyone else had this problem after the 7.1 update?

Thank you for your help!!!!

Greg Ennis

On Sat, 2015-04-04 at 16:47 -0500, Gregory P. Ennis
wrote:
> Everyone,
> 
> This morning I did a manual yum update on our a mail server to 7.1
> without any incident or problems.  A new kernel was installed, and I
> rebooted after the update.  
> 
> When I rebooted the machine I could not gain ssh access to it from an
> external ip address.  I was able to ssh to this mail server through a
> different machine on the local network.  
> 
> At first I thought the problem was related to the firewall.  I stopped
> firewalld, and fail2ban, and clear all firewall rules without being able
> to gain access.
> 
> I disabled firewalld, and fail2ban.  I enabled iptables and started it
> without a problem, but I could still not gain access.  I removed all
> entries in the host.allow and host.deny files, and this did not make a
> difference either.  
> 
> On one of the various reboots I tried to use the previous kernel before
> today's update, but there was no success.
> 
> I can scan the mail server and reach it without a problem from the
> internal network but I am not able to reach it from outside the local
> network.  I have the mail server behind a Centso 5.11 machine that is
> the gateway router for the internal network, and the mail server is nat
> addressed with it's external ip address to the internal machine.  I
have
> had this configuration set up for over 7 years.  I tweaked the Gateway
> router to nat address the mail server's ip address to a different
> machine inside the network and everything worked perfectly like it
> should, and then re-adjusted the gateway router again back to the mail
> server and am not able to gain access from outside the local network.
> 
> "traceroute" does not get to the mail server from outside the
local
> network, but works fine inside the local network.
> 
> Bottom line, this does not look like a host.deny, host.allow problem,
> nor does it look like a firewalld or iptables problem.  And it does not
> appear to be a problem with the gateway server.  
> 
> Is there another feature of CentOs 7.1 that I need to evaluate?  Has
> anyone else had this problem after the 7.1 update?
> 
> Thank you for your help!!!!
> 
> Greg Ennis
> 
---------------------------------------------------------------------

I sure need some help on this one, if any of you have ideas of what to
do next I would surely appreciate it.  An additional aspect of this
scenario is that when I have used ssh to connect to this mail server via
the internal network, I am able to ssh out of the machine to one of the
internal networks or remotely to a different network.   If no one else
has had this problem with 7.1 then it is obviously something I have
done, but right now I am at a loss.

Greg

> On 05 Apr 2015, at 14:35, Gregory P. Ennis <PoMec at PoMec.Net>
wrote:
> I sure need some help on this one, if any of you have ideas of what to
> do next I would surely appreciate it.  An additional aspect of this
> scenario is that when I have used ssh to connect to this mail server via
> the internal network, I am able to ssh out of the machine to one of the
> internal networks or remotely to a different network.   If no one else
> has had this problem with 7.1 then it is obviously something I have
> done, but right now I am at a loss.
Assuming that the mail server?s routing table is correct, you will need some
tcpdump to understand if the packets from outside reach the server (and then it
discards them).

I would do this:
1. Ensure that the mail server still has a valid default gateway and a correct
routing table
2. start tcpdump on the gateway
3. start tcpdump on the mail server
4 Try to connect to the mail server from outside.
> Greg
Ciao,
Andrea (just upgraded some servers, no problems)

--
Andrea Dell'Amico
http://adellam.sevenseas.org/



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.centos.org/pipermail/centos/attachments/20150405/0c9355c0/attachment-0001.sig>

On Sat, 2015-04-04 at 16:47 -0500, Gregory P. Ennis
wrote:
> Everyone,
> 
> This morning I did a manual yum update on our a mail server to 7.1
> without any incident or problems.  A new kernel was installed, and I
> rebooted after the update.  
> 
> When I rebooted the machine I could not gain ssh access to it from an
> external ip address.  I was able to ssh to this mail server through a
> different machine on the local network.  
> 
> At first I thought the problem was related to the firewall.  I stopped
> firewalld, and fail2ban, and clear all firewall rules without being able
> to gain access.
> 
> I disabled firewalld, and fail2ban.  I enabled iptables and started it
> without a problem, but I could still not gain access.  I removed all
> entries in the host.allow and host.deny files, and this did not make a
> difference either.  
> 
> On one of the various reboots I tried to use the previous kernel before
> today's update, but there was no success.
> 
> I can scan the mail server and reach it without a problem from the
> internal network but I am not able to reach it from outside the local
> network.  I have the mail server behind a Centso 5.11 machine that is
> the gateway router for the internal network, and the mail server is nat
> addressed with it's external ip address to the internal machine.  I
have
> had this configuration set up for over 7 years.  I tweaked the Gateway
> router to nat address the mail server's ip address to a different
> machine inside the network and everything worked perfectly like it
> should, and then re-adjusted the gateway router again back to the mail
> server and am not able to gain access from outside the local network.
> 
> "traceroute" does not get to the mail server from outside the
local
> network, but works fine inside the local network.
> 
> Bottom line, this does not look like a host.deny, host.allow problem,
> nor does it look like a firewalld or iptables problem.  And it does not
> appear to be a problem with the gateway server.  
> 
> Is there another feature of CentOs 7.1 that I need to evaluate?  Has
> anyone else had this problem after the 7.1 update?
> 
> Thank you for your help!!!!
> 
> Greg Ennis
> 
-----------------------------------------------------------------------

I am still having difficulty with this problem.  Everything worked
perfectly until the upgrade from 7.0 to 7.1.    I attempted to use
wireshark and tcpdump to analyze the packets but I do not have
sufficient experience with these tools to be helpful yet.

The mail server works fine with local network access, but intermittently
blocks access from eternal ip addresses.  Is there a way to back down
7.1 to 7.0.  When I get to the mail server by using one of the local
machines to ssh in to the mail server, I am able to perform any outbound
function from the mail server.

Greg

On 04/09/2015 03:50 PM, Gregory P. Ennis wrote:
> I am still having difficulty with this problem.
# ip addr show

# ip route show

# iptables -L -n -v

# tcpdump -nn -i <your primary interface> host <ip.ad.dr.es>

Watch tcpdump's output and try to connect from the IP address you name 
on the command line.  Don't ssh to that host and then back, or tcpdump 
will show you your outbound session.  Connect to the remote host from 
some other system.

Tail /var/log/messages and /var/log/secure while you connect.

On 04/04/2015 04:47 PM, Gregory P. Ennis wrote:
> Everyone,
> 
> This morning I did a manual yum update on our a mail server to 7.1
> without any incident or problems.  A new kernel was installed, and I
> rebooted after the update.  
> 
> When I rebooted the machine I could not gain ssh access to it from an
> external ip address.  I was able to ssh to this mail server through a
> different machine on the local network.  
> 
> At first I thought the problem was related to the firewall.  I stopped
> firewalld, and fail2ban, and clear all firewall rules without being able
> to gain access.
> 
> I disabled firewalld, and fail2ban.  I enabled iptables and started it
> without a problem, but I could still not gain access.  I removed all
> entries in the host.allow and host.deny files, and this did not make a
> difference either.  
> 
> On one of the various reboots I tried to use the previous kernel before
> today's update, but there was no success.
> 
> I can scan the mail server and reach it without a problem from the
> internal network but I am not able to reach it from outside the local
> network.  I have the mail server behind a Centso 5.11 machine that is
> the gateway router for the internal network, and the mail server is nat
> addressed with it's external ip address to the internal machine.  I
have
> had this configuration set up for over 7 years.  I tweaked the Gateway
> router to nat address the mail server's ip address to a different
> machine inside the network and everything worked perfectly like it
> should, and then re-adjusted the gateway router again back to the mail
> server and am not able to gain access from outside the local network.
> 
> "traceroute" does not get to the mail server from outside the
local
> network, but works fine inside the local network.
> 
> Bottom line, this does not look like a host.deny, host.allow problem,
> nor does it look like a firewalld or iptables problem.  And it does not
> appear to be a problem with the gateway server.  
> 
> Is there another feature of CentOs 7.1 that I need to evaluate?  Has
> anyone else had this problem after the 7.1 update?
> 
> Thank you for your help!!!!
> 
> Greg Ennis
Greg, do you have access to a console for that machine .. the mechanism
in RHEL (and therefore CentOS) to accept licenses changed from 7.0 to
7.1 .. before it was all firstboot, now it is a combination of firstboot
and initial-setup.

What may be happening is that you may need to be on the console and
accept the license on the first reboot after the update.

We tried to turn this off for CLI only installs, but in some
combinations of software, you may still get the acceptance screen and
have to complete it.

We know this is suboptimal, but it is exactly the same is in RHEL .. we
may try to remove these from CLI only machines in the future.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20150410/3d5db6b3/attachment-0001.sig>

On 04/04/2015 04:47 PM, Gregory P. Ennis wrote:
> Everyone,
> 
> This morning I did a manual yum update on our a mail server to 7.1
> without any incident or problems.  A new kernel was installed, and I
> rebooted after the update.  
> 
> When I rebooted the machine I could not gain ssh access to it from an
> external ip address.  I was able to ssh to this mail server through a
> different machine on the local network.  
> 
> At first I thought the problem was related to the firewall.  I stopped
> firewalld, and fail2ban, and clear all firewall rules without being able
> to gain access.
> 
> I disabled firewalld, and fail2ban.  I enabled iptables and started it
> without a problem, but I could still not gain access.  I removed all
> entries in the host.allow and host.deny files, and this did not make a
> difference either.  
> 
> On one of the various reboots I tried to use the previous kernel before
> today's update, but there was no success.
> 
> I can scan the mail server and reach it without a problem from the
> internal network but I am not able to reach it from outside the local
> network.  I have the mail server behind a Centso 5.11 machine that is
> the gateway router for the internal network, and the mail server is nat
> addressed with it's external ip address to the internal machine.  I
have
> had this configuration set up for over 7 years.  I tweaked the Gateway
> router to nat address the mail server's ip address to a different
> machine inside the network and everything worked perfectly like it
> should, and then re-adjusted the gateway router again back to the mail
> server and am not able to gain access from outside the local network.
> 
> "traceroute" does not get to the mail server from outside the
local
> network, but works fine inside the local network.
> 
> Bottom line, this does not look like a host.deny, host.allow problem,
> nor does it look like a firewalld or iptables problem.  And it does not
> appear to be a problem with the gateway server.  
> 
> Is there another feature of CentOs 7.1 that I need to evaluate?  Has
> anyone else had this problem after the 7.1 update?
> 
> Thank you for your help!!!!
> 
> Greg Ennis
Greg, do you have access to a console for that machine .. the mechanism
in RHEL (and therefore CentOS) to accept licenses changed from 7.0 to
7.1 .. before it was all firstboot, now it is a combination of firstboot
and initial-setup.

What may be happening is that you may need to be on the console and
accept the license on the first reboot after the update.

We tried to turn this off for CLI only installs, but in some
combinations of software, you may still get the acceptance screen and
have to complete it.

We know this is suboptimal, but it is exactly the same is in RHEL .. we
may try to remove these from CLI only machines in the future.

----------------------------------------------------------------

Johnny,

It is about 30 miles away from my location today.  I did take a look at
the console when the problem first started, but could not log in because
of the 7.1 problem related to multiple users on the log in screen
without the ability to scroll through the users.  I switched to a
terminal interface to try to solve the problem, and did not try to log
in via the gui.  

I'll take a look latter tonight to see if that will make a difference.

Thanks,

Greg

On Fri, Apr 10, 2015 at 06:33:27AM -0500, Johnny Hughes wrote:
> What may be happening is that you may need to be on the console and
> accept the license on the first reboot after the update.
> 
> We tried to turn this off for CLI only installs, but in some
> combinations of software, you may still get the acceptance screen and
> have to complete it.
So just to be clear, some of us who installed 7.0 servers in the GUI
and then carted them to a remotely colo might be screwed if the
machine reboots after updating to 7.1?  Are there some files I can
touch (or whatever) to prevent this from happening? Or is the best
solution to go to the colo and reboot?

I have consoles for all of my professional servers, but not my hobby
server! Fun fun! And I feel for you guys, given that upstream was the
main cause.

-- greg