Gluster users - May 2018 - Usage monitoring per user

Hi

Is there any easy way to find usage per user in Gluster? We have 300TB
storage with almost 100 million files. Running du take too much time. Are
people aware of any other tool which can be used to break up storage per
user?

Thanks

Kashif
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180501/cdd33e5e/attachment.html>

Hi,

There are several programs that will basically take the outputs of your
scans and store the results in a database. If you size the database
appropriately, then querying that database will be much quicker than
querying the filesystem.  But of course the results will be a little bit
outdated.

One such project is robinhood. https://github.com/cea-hpc/robinhood/wiki

A simpler way might be to just have daily/weekly cron jobs that output text
reports, without maintaining a separate database.

But there is no way to avoid doing a recursive POSIX tree traversal, since
that is how you get your info out of your filesystem.

Regards,
Alex

On Tue, May 1, 2018 at 5:30 AM, mohammad kashif <kashif.alig at gmail.com>
wrote:
> Hi
>
> Is there any easy way to find usage per user in Gluster? We have 300TB
> storage with almost 100 million files. Running du take too much time. Are
> people aware of any other tool which can be used to break up storage per
> user?
>
> Thanks
>
> Kashif
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180501/b580e5ea/attachment.html>

I rather like agedu    It probably does what you want.

But as Mohammad says you do have to traverse your filesystem.


https://www.chiark.greenend.org.uk/~sgtatham/agedu/

agedu: track down wasted disk space - chiark home
page<https://www.chiark.greenend.org.uk/~sgtatham/agedu/>
www.chiark.greenend.org.uk
agedu. a Unix utility for tracking down wasted disk space Introduction. Suppose
you're running low on disk space. You need to free some up, by finding
something that's a waste of space and deleting it (or moving it to an
archive medium).






________________________________
From: gluster-users-bounces at gluster.org <gluster-users-bounces at
gluster.org> on behalf of Alex Chekholko <alex at calicolabs.com>
Sent: 01 May 2018 18:45
To: mohammad kashif
Cc: gluster-users
Subject: Re: [Gluster-users] Usage monitoring per user

Hi,

There are several programs that will basically take the outputs of your scans
and store the results in a database. If you size the database appropriately,
then querying that database will be much quicker than querying the filesystem. 
But of course the results will be a little bit outdated.

One such project is robinhood.
https://github.com/cea-hpc/robinhood/wiki<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcea-hpc%2Frobinhood%2Fwiki&data=01%7C01%7Cjohe%40novozymes.com%7Cb280d3cf0fde4c28124c08d5af82fa19%7C43d5f49ee03a4d22a2285684196bb001%7C0&sdata=m5s1l7DvoKiXlAhSP8eGlu9lYjtlkIM5k3CTGt6zoH0%3D&reserved=0>

A simpler way might be to just have daily/weekly cron jobs that output text
reports, without maintaining a separate database.

But there is no way to avoid doing a recursive POSIX tree traversal, since that
is how you get your info out of your filesystem.

Regards,
Alex

On Tue, May 1, 2018 at 5:30 AM, mohammad kashif <kashif.alig at
gmail.com<mailto:kashif.alig at gmail.com>> wrote:
Hi

Is there any easy way to find usage per user in Gluster? We have 300TB storage
with almost 100 million files. Running du take too much time. Are people aware
of any other tool which can be used to break up storage per user?

Thanks

Kashif

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.gluster.org%2Fmailman%2Flistinfo%2Fgluster-users&data=01%7C01%7Cjohe%40novozymes.com%7Cb280d3cf0fde4c28124c08d5af82fa19%7C43d5f49ee03a4d22a2285684196bb001%7C0&sdata=P%2Bkfpz8XaI7j7PM9%2FocnBkf0Wop%2BLZDAbgnvNpIcdd0%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180502/6f4bd540/attachment.html>