On Sun, Aug 06, 2017 at 01:01:56PM -0700, wk wrote:> I'm not sure what you mean by saying "NFS is available by anyone"? > > Are your gluster nodes physically isolated on their own network/switch?Nope, impossible to do for us> > In other words can an outsider access them directly without having to > compromise a NFS client machine first? >Yes, but we don't have any NFS client, only libgfapi. I added a bunch of iptables rules to prevent that from happening, if they did use NFS which I am unsure of. If they used something else to access the volume though, who knows .. It hasn't been re-hacked since so that's a good sign.> -bill > > > On 8/6/2017 7:57 AM, lemonnierk at ulrar.net wrote: > > Hi, > > > > This morning one of our cluster was hacked, all the VM disks were > > deleted and a file README.txt was left with inside just > > "http://virtualisan.net/contactus.php :D" > > > > I don't speak the language but with google translete it looks like it's > > just a webdev company or something like that, a bit surprised .. > > In any case, we'd really like to know how that happened. > > > > I realised NFS is accessible by anyone (sigh), is there a way to check > > if that is what they used ? I tried reading the nfs.log but it's not > > really clear if someone used it or not. What do I need to look for in > > there to see if someone mounted the volume ? > > There are stuff in the log on one of the bricks (only one), > > and as we aren't using NFS for that volume that in itself seems > > suspicious. > > > > Thanks > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/885c73a9/attachment.sig>
On 8/6/2017 1:09 PM, lemonnierk at ulrar.net wrote:> >> Are your gluster nodes physically isolated on their own network/switch? > Nope, impossible to do for usok, yes, that makes it much harder to secure. You should add VLANS, and/or overlay networks and/or Mac Address filtering/locking/security which raises the bar quite a bit for hackers. Perhaps your provider can help you with that. Then there is the Gluster Auth stuff, which is cert based as I recall. Unfortunately, I don't have any experience with it as we have relied on unique seperate physical networks for our clusters. Hackers (and us) can't even get to our Gluster boxes except via IP/KVM or the client itself. I'm now curious as to what you find and am thinking we should be looking at the Gluster Auth protocols as well.>> In other words can an outsider access them directly without having to >> compromise a NFS client machine first? >> > Yes, but we don't have any NFS client, only libgfapi. > I added a bunch of iptables rules to prevent that from happening, if > they did use NFS which I am unsure of. If they used something else to > access the volume though, who knows .. It hasn't been re-hacked since so > that's a good sign.Well if you aren't using it, then turn NFS off. I think NFS is turned off by default in the new versions anyway in favor of NFS-Ganesha. But the original question remains, did they get into just the Gluster boxes or are they in the Client already? Unless they rooted the boxes and cleaned the logs, there should be some traces of activity in the various system and gluster logs. The various root kit checker programs may find something (chkrootkit) -bill -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/25261f39/attachment.html>
> You should add VLANS, and/or overlay networks and/or Mac Address > filtering/locking/security which raises the bar quite a bit for hackers. > Perhaps your provider can help you with that. >Gluster already uses a vlan, the problem is that there is no easy way that I know of to tell gluster not to listen on an interface, and I can't not have a public IP on the server. I really wish ther was a simple "listen only on this IP/interface" option for this> Then there is the Gluster Auth stuff, which is cert based as I recall. > Unfortunately, I don't have any experience with it as we have relied on > unique seperate physical networks for our clusters. > Hackers (and us) can't even get to our Gluster boxes except via IP/KVM > or the client itself. >Well never used it, but I never thought I needed that since the vlan gluster uses is private so outside users can't reach it. Didn't realise NFS works with access to any one node since we don't use it.> > Well if you aren't using it, then turn NFS off. I think NFS is turned > off by default in the new versions anyway in favor of NFS-Ganesha.Yeah, we are still on 3.7 for now, I haven't taken the time to test newer versions yet. Since 3.7.15 does everything we need pretty well, not really felt the need for that.> > But the original question remains, did they get into just the Gluster > boxes or are they in the Client already? > > Unless they rooted the boxes and cleaned the logs, there should be some > traces of activity in the various system and gluster logs. The various > root kit checker programs may find something (chkrootkit) >Well it's one and the same, gluster is installed on the proxmox servers so the VM are just using localhost as their disk storage. So either they got into the volume itself (NFS or some other way I haven't thought of), or they got root on the hypervisors but in that case why f*ck up with the volume instead of everything else. Since everything else looks okay, I think they just had access to the volume, and the only way I can think of is NFS. But I don't see anything really suspicious in nfs.log, it seems to me like only normal glusterd restart logs I'll be sure to scan for rootkits tomorrow just in case, but I assume they would have re-wiped everything if they still had access. Googling the link they left I found a forum where some guy got his hard drive wiped in a similar manner on his router a few days ago, it looks like someone having fun wiping unsecured NAS .. What a great way to spend your free time :( -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170807/e6a35d0d/attachment.sig>