Hi, This morning one of our cluster was hacked, all the VM disks were deleted and a file README.txt was left with inside just "http://virtualisan.net/contactus.php :D" I don't speak the language but with google translete it looks like it's just a webdev company or something like that, a bit surprised .. In any case, we'd really like to know how that happened. I realised NFS is accessible by anyone (sigh), is there a way to check if that is what they used ? I tried reading the nfs.log but it's not really clear if someone used it or not. What do I need to look for in there to see if someone mounted the volume ? There are stuff in the log on one of the bricks (only one), and as we aren't using NFS for that volume that in itself seems suspicious. Thanks -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/1a8f66a1/attachment.sig>
Thinking about it, is it even normal they managed to delete the VM disks? Shoudn't they have gotten "file in use" errors ? Or does libgfapi not lock the access files ? On Sun, Aug 06, 2017 at 03:57:06PM +0100, lemonnierk at ulrar.net wrote:> Hi, > > This morning one of our cluster was hacked, all the VM disks were > deleted and a file README.txt was left with inside just > "http://virtualisan.net/contactus.php :D" > > I don't speak the language but with google translete it looks like it's > just a webdev company or something like that, a bit surprised .. > In any case, we'd really like to know how that happened. > > I realised NFS is accessible by anyone (sigh), is there a way to check > if that is what they used ? I tried reading the nfs.log but it's not > really clear if someone used it or not. What do I need to look for in > there to see if someone mounted the volume ? > There are stuff in the log on one of the bricks (only one), > and as we aren't using NFS for that volume that in itself seems > suspicious. > > Thanks> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/c8209e0c/attachment.sig>
I'm not sure what you mean by saying "NFS is available by anyone"? Are your gluster nodes physically isolated on their own network/switch? In other words can an outsider access them directly without having to compromise a NFS client machine first? -bill On 8/6/2017 7:57 AM, lemonnierk at ulrar.net wrote:> Hi, > > This morning one of our cluster was hacked, all the VM disks were > deleted and a file README.txt was left with inside just > "http://virtualisan.net/contactus.php :D" > > I don't speak the language but with google translete it looks like it's > just a webdev company or something like that, a bit surprised .. > In any case, we'd really like to know how that happened. > > I realised NFS is accessible by anyone (sigh), is there a way to check > if that is what they used ? I tried reading the nfs.log but it's not > really clear if someone used it or not. What do I need to look for in > there to see if someone mounted the volume ? > There are stuff in the log on one of the bricks (only one), > and as we aren't using NFS for that volume that in itself seems > suspicious. > > Thanks > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/6ee6e282/attachment.html>
On Sun, Aug 06, 2017 at 01:01:56PM -0700, wk wrote:> I'm not sure what you mean by saying "NFS is available by anyone"? > > Are your gluster nodes physically isolated on their own network/switch?Nope, impossible to do for us> > In other words can an outsider access them directly without having to > compromise a NFS client machine first? >Yes, but we don't have any NFS client, only libgfapi. I added a bunch of iptables rules to prevent that from happening, if they did use NFS which I am unsure of. If they used something else to access the volume though, who knows .. It hasn't been re-hacked since so that's a good sign.> -bill > > > On 8/6/2017 7:57 AM, lemonnierk at ulrar.net wrote: > > Hi, > > > > This morning one of our cluster was hacked, all the VM disks were > > deleted and a file README.txt was left with inside just > > "http://virtualisan.net/contactus.php :D" > > > > I don't speak the language but with google translete it looks like it's > > just a webdev company or something like that, a bit surprised .. > > In any case, we'd really like to know how that happened. > > > > I realised NFS is accessible by anyone (sigh), is there a way to check > > if that is what they used ? I tried reading the nfs.log but it's not > > really clear if someone used it or not. What do I need to look for in > > there to see if someone mounted the volume ? > > There are stuff in the log on one of the bricks (only one), > > and as we aren't using NFS for that volume that in itself seems > > suspicious. > > > > Thanks > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170806/885c73a9/attachment.sig>
On Sun, Aug 06, 2017 at 08:54:33PM +0100, lemonnierk at ulrar.net wrote:> Thinking about it, is it even normal they managed to delete the VM disks? > Shoudn't they have gotten "file in use" errors ? Or does libgfapi not > lock the access files ?It really depends on the application if locks are used. Most (Linux) applications will use advisory locks. This means that locking is only effective when all participating applications use and honour the locks. If one application uses (advisory) locks, and an other application now, well, then all bets are off. It is also possible to delete files that are in active use. The contens will still be served by the filesystem, but there is no accessible filename anymore. If the VMs using those files are still running, there might be a way to create a new filename for the data. If the VMs have been stopped, and the file-descriptior has been closed, the data will be gone :-/ Niels> > > On Sun, Aug 06, 2017 at 03:57:06PM +0100, lemonnierk at ulrar.net wrote: > > Hi, > > > > This morning one of our cluster was hacked, all the VM disks were > > deleted and a file README.txt was left with inside just > > "http://virtualisan.net/contactus.php :D" > > > > I don't speak the language but with google translete it looks like it's > > just a webdev company or something like that, a bit surprised .. > > In any case, we'd really like to know how that happened. > > > > I realised NFS is accessible by anyone (sigh), is there a way to check > > if that is what they used ? I tried reading the nfs.log but it's not > > really clear if someone used it or not. What do I need to look for in > > there to see if someone mounted the volume ? > > There are stuff in the log on one of the bricks (only one), > > and as we aren't using NFS for that volume that in itself seems > > suspicious. > > > > Thanks > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users