Gluster users - Aug 2012 - Problem with too many small files

I am not sure how it works on Gluster but to mitigate the problem with listing a
lot of small files wouldn't it be suitable to keep on every node a copy of
the directory tree. I think Isilon does that and there is probably a lot to be
learned from them which seems quiet mature technology. Could also have another
interesting thing added in the future, local SSD to keep the file system
metadata for faster access.

Regards,
Fernando Frediani
Lead Systems Engineer
[Description: Description: Description: Description: Description:
cid:image003.png at 01CD637B.AB2C8040]

260-266 Goswell Road, London, EC1V 7EB, United Kingdom
sales: +44 (0) 20 7150 3800
ddi:     +44 (0) 20 7150 3803
fax:    +44 (0) 20 7336 8420
web:   http://www.qubenet.net/

Qube Managed Services Limited
Company Number: 6215769 (Registered in England and Wales)
VAT Registration Number: GB 933 8400 27

This e-mail and the information it contains are confidential.
If you have received this e-mail in error please notify the sender immediately.
You should not copy it for any purpose or disclose its contents to any other
person.
The contents of this e-mail do not necessarily reflect the views of the company.
E&OE

P Please consider the environment before printing this email

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120813/1798a74c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2373 bytes
Desc: image001.png
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120813/1798a74c/attachment.png>

On August 13, 2012 5:52:26 AM "Fernando Frediani (Qube)" 
<fernando.frediani at qubenet.net> wrote:
> I am not sure how it works on Gluster but to mitigate the problem with 
> listing a lot of small files wouldn't it be suitable to keep on every 
> node a copy of the directory tree. I think Isilon does that and there 
> is probably a lot to be learned from them which seems quiet mature 
> technology. Could also have another interesting thing added in the 
> future, local SSD to keep the file system metadata for faster access.
We could do that, in fact I've been an advocate for it, but it must be 
understood that there's no such thing as a free lunch.  Once you're 
caching directory structures on clients, you either have to give up a 
certain amount of consistency or make the entire protocol much more 
complex to perform cache invalidations etc.  Who's volunteering to do 
that work?  Who's even asking us to do that in the core team, once they 
understand that it means taking resources away from other priorities 
and permanently slowing down development because of that complexity?  
Nobody.  At least, unlike Isilon, there's the possibility that somebody 
could take a stab at reducing consistency for the sake of performance 
themselves (as I myself have done e.g. with negative-lookup caching and 
replication bypass).  There's not really all that much to be learned 
from a closed-source system that's not even described in papers.  In 
fact, I *know* that they learn more from us than vice versa.