Hi List, We have a CentOS VPS running a web site in a DC far away. The chap that dev's this site told me he couldn't SFTP in yesterday, his password was being rejected (I went to his desk to confirm and saw it was telling him the password was incorrect but neither him nor me had changed it and we are the only two with access to this VPS). So I logged in as root and reset his password, be he still couldn't log in (same problem, claiming the password was wrong). [root at server ~]# passwd webdevuser Changing password for user webdevuser. New UNIX password: Retype new UNIX password: passwd: all authentication tokens updates successfully. I tried to SSH in as the web dev user and it wouldn't let me in. Returning back to my root console window; [root at server ~]# su - webdevuser [webdevuser at server ~]# passwd Changing password for user webdevuser. Changing password for webdevuser. (current) UNIX password: passwd: Authentication token manipulation error Firstly; I am stracthing my head as to why his password was no longer working in the first place? Secondly; Why I can't reset it? Googling around many people suggest there is a discrepancy between the /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using pwconv to recreate shadow and the same for /etc/groups, deleting gshadow recreating it with grpconv will solve the problem but I still can't login as the web dev user. Any ideas anyone? -- James. http://www.jamesbensley.co.cc/ There are 10 kinds of people in the world; Those who understand Vigesimal, and J others...? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20110216/d685c097/attachment.html>
On Wed, Feb 16, 2011 at 7:28 AM, James Bensley <jwbensley at gmail.com> wrote:> Hi List, > > We have a CentOS VPS running a web site in a DC far away. The chap that > dev's this site told me he couldn't SFTP in yesterday, his password was > being rejected (I went to his desk to confirm and saw it was telling him the > password was incorrect but neither him nor me had changed it and we are the > only two with access to this VPS). So I logged in as root and reset his > password, be he still couldn't log in (same problem, claiming the password > was wrong). > > [root at server ~]# passwd webdevuser > Changing password for user webdevuser. > New UNIX password: > Retype new UNIX password: > passwd: all authentication tokens updates successfully. > > I tried to SSH in as the web dev user and it wouldn't let me in. Returning > back to my root console window; > > [root at server ~]# su - webdevuser > [webdevuser at server ~]# passwd > Changing password for user webdevuser. > Changing password for webdevuser. > (current) UNIX password: > passwd: Authentication token manipulation error > > Firstly; I am stracthing my head as to why his password was no longer > working in the first place? > > Secondly; Why I can't reset it? > > Googling around many people suggest there is a discrepancy between the > /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using > pwconv to recreate shadow and the same for /etc/groups, deleting gshadow > recreating it with grpconv will solve the problem but I still can't login as > the web dev user. > > Any ideas anyone?Uh-oh. Has your developer, or you, been editing the /etc/passwd, /etc/shadow, /etc/group, or /etc/gshadow files manually? And do you use NIS or LDAP for authentication? And this is a publicly exposed webserver, right? How fast can you rebuild it if it's been rootkitted? Check the /etc/shadow and /etc/group for consistent numbers of entries, and /etc/group and /etc/gshadow. Do you have other users who can still log in or not?
On 16 Feb 2011 12:34, "Nico Kadel-Garcia" <nkadel at gmail.com> wrote:> > Uh-oh. Has your developer, or you, been editing the /etc/passwd, > /etc/shadow, /etc/group, or /etc/gshadow files manually?Nope.> And do you > use NIS or LDAP for authentication?Nope.> And this is a publicly exposed > webserver, right? How fast can you rebuild it if it's been rootkitted?How long is a peice of string? As quick as I can reupload the data, but thats another issue for another day.> Check the /etc/shadow and /etc/group for consistent numbers of > entries, and /etc/group and /etc/gshadow.Do you mean duplicate entries? If so there are none of those.> Do you have other users who > can still log in or not?There is only the root and web dev user on this box. Thanks for your input Nico :) --James. (This email was sent from a mobile device) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20110216/64172e9e/attachment.html>
On Wed, Feb 16, 2011 at 7:43 AM, James Bensley <jwbensley at gmail.com> wrote:> On 16 Feb 2011 12:34, "Nico Kadel-Garcia" <nkadel at gmail.com> wrote: >> >> Uh-oh. Has your developer, or you, been editing the /etc/passwd, >> /etc/shadow, /etc/group, or /etc/gshadow files manually? > > Nope. > >> And do you >> use NIS or LDAP for authentication? > > Nope. > >> And this is a publicly exposed >> webserver, right? How fast can you rebuild it if it's been rootkitted? > > How long is a peice of string? As quick as I can reupload the data, but > thats another issue for another day. > >> Check the /etc/shadow and /etc/group for consistent numbers of >> entries, and /etc/group and /etc/gshadow. > > Do you mean duplicate entries? If so there are none of those.No, I mean the sam enumber of entries. wc /etc/shadow /etc/passwd cut -f1 -d: /etc/shasow /etc/passwd | sort | uniq -c And actually go line by line down these files, checking for matching usernames, correct layout of ':' separated entries, correct numbers of entries, and blank lines. I've seen serous problems where one or ther other of these files were corrupted by something, especially badly written installer scripts that only edited /etc/passwd directly and ignored /etc/shadow, or which mishandled "$" entries in newly created encrypted passwords.>> Do you have other users who >> can still log in or not? > > There is only the root and web dev user on this box. > > Thanks for your input Nico :) > > --James. (This email was sent from a mobile device)Are you *sure*? Can you back this thing up for review and rebuilding? It might be safest to image it for analysis and simply rebuild it.
On 16/02/11 13:28, James Bensley wrote:> Hi List, > > We have a CentOS VPS running a web site in a DC far away. The chap that > dev's this site told me he couldn't SFTP in yesterday, his password was > being rejected (I went to his desk to confirm and saw it was telling him > the password was incorrect but neither him nor me had changed it and we are > the only two with access to this VPS). So I logged in as root and reset his > password, be he still couldn't log in (same problem, claiming the password > was wrong). > > [root at server ~]# passwd webdevuser > Changing password for user webdevuser. > New UNIX password: > Retype new UNIX password: > passwd: all authentication tokens updates successfully. > > I tried to SSH in as the web dev user and it wouldn't let me in. Returning > back to my root console window; > > [root at server ~]# su - webdevuser > [webdevuser at server ~]# passwd > Changing password for user webdevuser. > Changing password for webdevuser. > (current) UNIX password: > passwd: Authentication token manipulation error > > Firstly; I am stracthing my head as to why his password was no longer > working in the first place? > > Secondly; Why I can't reset it? > > Googling around many people suggest there is a discrepancy between the > /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using > pwconv to recreate shadow and the same for /etc/groups, deleting gshadow > recreating it with grpconv will solve the problem but I still can't login > as the web dev user. > > Any ideas anyone?- Could the account have become locked somehow? (passwd -u $user) Or could the account have become expired? - Are the permissions strict on the users ~/.ssh? (0700 on the directory, and 0600 on any files inside that directory - like authorized_keys ...) - Is SELinux in Enforced mode and are the SELinux file context correct on /home? (restorecon -rv /home) Also double check /var/log/messages, /var/log/secure and /var/log/audit/audit.log carefully when trying to log in as that user. kind regards, David Sommerseth
On Wed, Feb 16, 2011 at 7:28 AM, James Bensley <jwbensley at gmail.com> wrote:> [root at server ~]# su - webdevuser > [webdevuser at server ~]# passwd > Changing password for user webdevuser. > Changing password for webdevuser. > (current) UNIX password: > passwd: Authentication token manipulation errorA lot of things can cause this, including a full /var filesystem :/
On Wed, Feb 16, 2011 at 6:28 AM, James Bensley <jwbensley at gmail.com> wrote:> Hi List, > > We have a CentOS VPS running a web site in a DC far away. The chap that > dev's this site told me he couldn't SFTP in yesterday, his password was > being rejected (I went to his desk to confirm and saw it was telling him the > password was incorrect but neither him nor me had changed it and we are the > only two with access to this VPS). So I logged in as root and reset his > password, be he still couldn't log in (same problem, claiming the password > was wrong). > > [root at server ~]# passwd webdevuser > Changing password for user webdevuser. > New UNIX password: > Retype new UNIX password: > passwd: all authentication tokens updates successfully. > > I tried to SSH in as the web dev user and it wouldn't let me in. Returning > back to my root console window; > > [root at server ~]# su - webdevuser > [webdevuser at server ~]# passwd > Changing password for user webdevuser. > Changing password for webdevuser. > (current) UNIX password: > passwd: Authentication token manipulation error > > Firstly; I am stracthing my head as to why his password was no longer > working in the first place? > > Secondly; Why I can't reset it? > > Googling around many people suggest there is a discrepancy between the > /etc/passwd and /etc/shadow files and by deleting /etc/shadow and using > pwconv to recreate shadow and the same for /etc/groups, deleting gshadow > recreating it with grpconv will solve the problem but I still can't login as > the web dev user. > > Any ideas anyone?What does /etc/nsswitch.conf look like? Anything other than "files" for passwd, shadow and group? If that's OK, I would start comparing files in /etc/pam.d to a known-good system. -- Jeff