On Sun, Aug 06, 2017 at 08:54:33PM +0100, lemonnierk at ulrar.net wrote:> Thinking about it, is it even normal they managed to delete the VM disks? > Shoudn't they have gotten "file in use" errors ? Or does libgfapi not > lock the access files ?It really depends on the application if locks are used. Most (Linux) applications will use advisory locks. This means that locking is only effective when all participating applications use and honour the locks. If one application uses (advisory) locks, and an other application now, well, then all bets are off. It is also possible to delete files that are in active use. The contens will still be served by the filesystem, but there is no accessible filename anymore. If the VMs using those files are still running, there might be a way to create a new filename for the data. If the VMs have been stopped, and the file-descriptior has been closed, the data will be gone :-/ Niels> > > On Sun, Aug 06, 2017 at 03:57:06PM +0100, lemonnierk at ulrar.net wrote: > > Hi, > > > > This morning one of our cluster was hacked, all the VM disks were > > deleted and a file README.txt was left with inside just > > "http://virtualisan.net/contactus.php :D" > > > > I don't speak the language but with google translete it looks like it's > > just a webdev company or something like that, a bit surprised .. > > In any case, we'd really like to know how that happened. > > > > I realised NFS is accessible by anyone (sigh), is there a way to check > > if that is what they used ? I tried reading the nfs.log but it's not > > really clear if someone used it or not. What do I need to look for in > > there to see if someone mounted the volume ? > > There are stuff in the log on one of the bricks (only one), > > and as we aren't using NFS for that volume that in itself seems > > suspicious. > > > > Thanks > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users >> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
> It really depends on the application if locks are used. Most (Linux) > applications will use advisory locks. This means that locking is only > effective when all participating applications use and honour the locks. > If one application uses (advisory) locks, and an other application now, > well, then all bets are off. > > It is also possible to delete files that are in active use. The contens > will still be served by the filesystem, but there is no accessible > filename anymore. If the VMs using those files are still running, there > might be a way to create a new filename for the data. If the VMs have > been stopped, and the file-descriptior has been closed, the data will be > gone :-/ >Oh the data was gone long before I stopped the VM, every binary was doing I/O errors when accessed, only whatever was in ram (ssh ..) when the disk got suppressed was still working. I'm a bit surpised they could be deleted, but I imagine qemu through libgfapi doesn't really access the file as a whole, maybe just the part it needs when it needs it. In any case the gluster logs show clearly file descriptor errors from 8h47 AM UTC, which seems to match our first monitoring alerts. I assume that's when the deletion happened. Now I just need to figure out what they used to access the volume, I hope it's just NFS since that's the only thing I can think of. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170807/c83c50fc/attachment.sig>
check out the syslogs of iptables logs on ip address access during that time. maybe you should move in the future to the centralised logging independent of vm infrastructure Am 07.08.2017 2:20 nachm. schrieb <lemonnierk at ulrar.net>:> > It really depends on the application if locks are used. Most (Linux) > > applications will use advisory locks. This means that locking is only > > effective when all participating applications use and honour the locks. > > If one application uses (advisory) locks, and an other application now, > > well, then all bets are off. > > > > It is also possible to delete files that are in active use. The contens > > will still be served by the filesystem, but there is no accessible > > filename anymore. If the VMs using those files are still running, there > > might be a way to create a new filename for the data. If the VMs have > > been stopped, and the file-descriptior has been closed, the data will be > > gone :-/ > > > > Oh the data was gone long before I stopped the VM, every binary was > doing I/O errors when accessed, only whatever was in ram (ssh ..) when > the disk got suppressed was still working. > > I'm a bit surpised they could be deleted, but I imagine qemu through > libgfapi doesn't really access the file as a whole, maybe just the part > it needs when it needs it. In any case the gluster logs show clearly > file descriptor errors from 8h47 AM UTC, which seems to match our first > monitoring alerts. I assume that's when the deletion happened. > > Now I just need to figure out what they used to access the volume, I > hope it's just NFS since that's the only thing I can think of. > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170809/25d47a61/attachment.html>