No, we haven't been hacked. ;) We have a prospective client who is asking us what our policy is in the event of unauthorized access. Obviously you fix the system(s) that have been compromised, but what steps do you take to mitigate the effects of a breach? What is industry best practice? So far, searches haven't produced anything that looks consistent, except maybe identity monitoring for financial data. (EG: Target breach) We host a significant amount of educational data, but no financial information. How would we even respond to this question? I've also posted this question at https://www.reddit.com/r/linuxadmin/comments/42mi1r/what_to_do_when_youve_been_hacked/ Thanks, Ben
On Mon, January 25, 2016 12:04 pm, Benjamin Smith wrote:> No, we haven't been hacked. ;) > We have a prospective client who is asking us what our policy is in the > event > of unauthorized access. Obviously you fix the system(s) that have been > compromised, but what steps do you take to mitigate the effects of a > breach? > What is industry best practice? So far, searches haven't produced anything > that looks consistent, except maybe identity monitoring for financial > data. > (EG: Target breach) > We host a significant amount of educational data, but no financial > information. > How would we even respond to this question? > I've also posted this question at > https://www.reddit.com/r/linuxadmin/comments/42mi1r/what_to_do_when_youve_been_hacked/Of course, mine will by no means be comprehensive or detailed. The order may change when you start investigatinge it. But just in general: 1. It already happened, do not rush to do anything, change anything, reboot, kill processes. If you can keep it for some time connected to the network, you have chance to have more leak or deeper damage, but simultaneously you may loose some of the tracks if you disconnect (thing self sweeping up malicious things when network connection is lost). They usually suggest yank from network, but it is your decision. Rebooting will wipe content of /dev/shm and similar places used by bad guys to an advantage of disappearing tracks 2. You should have backups, but you may need to make a copy of stuff that might have changed after last backup 3. damage check 4. finding out level of compromise and the way compromise happened. Was it root compromise and they got the whole system? Or some regular user account was compromised (password or secret key was stolen). In last case: were then attempts of elevation of privileges (they usually are)? Were these attempts successful? (this is why you keep your system patched - it is rare that there is hole on fully patched system known to bad guys, but not software developers/system vendors). Was it through some service? 5. what have they done to come back or use your system otherwise? This usually is parallel to investigating potential system level compromise. Namely: scan open ports internally, and compare to scan from external box (though some malicious may listen to their master box only, so this may not necessarily reveal everything). Check if any of binaries or libraries were modified (you should be prepared for that, look for integrity tools like afick, eics, I used tripwire before it went commercial). 6. Preserve hacked system drive(s) (or their images) for further forensics, which may take two weeks and longer... Build new system, patch, enable the services you need and restore, make sure you don't open the hole they came through. Restore files. 7. If it was system level compromise (root) sysadmin is obliged to contact all users about it, stressing that their secret keys and passwords to machines they were logging from compromised machine should be considered compromised. 8. After you feel you recovered from compromise, and have freshly build system with all necessary services running, do not drop forensics, it is tedious but necessary thing. Process accounting may be awfully helpful (you do have process accounting enabled, right?) SANS is one of great resources everybody will probably mention. But the best is to avoid system level compromise, so: patch, patch, patch... (or: update, update, update...). As far as user level compromises are concerned: with 100+ users it is just a question of time when someone's account will be compromised (often in a system compromise elsewhere where user has an account). The best you can do is to run your systems with an assumption that bad guys are already inside. Make sure they can not do damage (even DOS, not to mention elevation of privileges). I hope, this helps. Valeri ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On Jan 25, 2016, at 11:04 AM, Benjamin Smith <lists at benjamindsmith.com> wrote:> > We have a prospective client who is asking us what our policy is in the event > of unauthorized access.Tell them you use the Mr. Miyagi defense: ?Don?t get hit.? Your prospective client sounds like they?re expecting someone to have established procedures to deal with breaches. You know who has established procedures? Organizations that see the same problems again and again. Selecting an information service provider based on which one is best at recovering from a hack attack is like hiring a football coach based on how skilled he is at setting bones or selecting a cargo ship captain based on how good he is at patching hull breaches. Why is ?We?ve been at this for 20 years and have never *had* to clean up after a hacking incident? not an excellent rejoinder?> what steps do you take to mitigate the effects of a breach? > What is industry best practice?You should not have to ask this. You should know it, because you are a professional and have been in this industry long enough. Since you don?t, maybe you shouldn?t be bidding on this job. I don?t mean to make this sound cabalistic, where only insiders know the secret handshakes, but rather exactly the opposite: this is information you should have been slowly absorbing for years: - SSH instead of telnet and FTP - HTTPS wherever possible over HTTP - Always enable SELinux - Prefer to surf default SELinux policies rather than override or custom-craft - Know in your heart that deny-by-default firewalls are a good thing - Turn off unnecessary services? - ?then run ?netstat -na | grep LISTEN? and justify each output line - Understand chown and chmod effects implicitly - Be able to read ls -l output at a blink And much more. All of this will be covered in any decent text on Unix/Linux security. Sorry, I can?t give recommendations since I got past the book learnin? stage long ago, and have been accreting such things ever since. Coming back to martial arts, at some point you get past the point of conscious action and react implicitly. The equivalent in security is recognizing risks and mitigating against them before they become NY Times headlines.
On Monday, January 25, 2016 11:56:19 AM Warren Young wrote:> On Jan 25, 2016, at 11:04 AM, Benjamin Smith <lists at benjamindsmith.com>wrote:> > We have a prospective client who is asking us what our policy is in the > > event of unauthorized access. > > Tell them you use the Mr. Miyagi defense: ?Don?t get hit.? > > Your prospective client sounds like they?re expecting someone to have > established procedures to deal with breaches. You know who has established > procedures? Organizations that see the same problems again and again. > > Selecting an information service provider based on which one is best at > recovering from a hack attack is like hiring a football coach based on how > skilled he is at setting bones or selecting a cargo ship captain based on > how good he is at patching hull breaches. > > Why is ?We?ve been at this for 20 years and have never *had* to clean up > after a hacking incident? not an excellent rejoinder?Agreed! (although for us it has been 15 years.> > what steps do you take to mitigate the effects of a breach? > > What is industry best practice? > > You should not have to ask this. You should know it, because you are a > professional and have been in this industry long enough. > > Since you don?t, maybe you shouldn?t be bidding on this job. > > I don?t mean to make this sound cabalistic, where only insiders know the > secret handshakes, but rather exactly the opposite: this is information you > should have been slowly absorbing for years: > > - SSH instead of telnet and FTP > - HTTPS wherever possible over HTTP > - Always enable SELinux > - Prefer to surf default SELinux policies rather than override or > custom-craft - Know in your heart that deny-by-default firewalls are a good > thing - Turn off unnecessary services? > - ?then run ?netstat -na | grep LISTEN? and justify each output line > - Understand chown and chmod effects implicitly > - Be able to read ls -l output at a blink > > And much more.Which I'd consider "best practices" and we do them. They are specifically asking about what to do *after* a breach. Despite all the best practices in place, there's *still* some risk.