Gluster users - Sep 2008 - Gluster update | need your support

Dear Members,
Even though Gluster team is growing at a steady phase, our aggressive
development
schedule out phases our resources. We need to expand and also maintain a 1:1
developer /
QA engineer ratio. Our major development focus in the next 8 months will be
towards:

  * Large scale regression tests (24/7/365)
  * Web based monitoring and management
  * Hot upgrade/add/remove of storage nodes
  * Gluster Embedded Storage Operating System ( GESOS )
  * and of course more translators: snapshot, dedup, compression, encryption..

In a month, we will also open up a 100TB Infiniband and GigE based clustered
storage for
community evaluation/testing.

We are in the middle of raising a venture capital investment. We need
testimonials,
use-cases and active references. It will be a phenomenal help, if you all help
us cross this bridge.

How can you help:
* Add yourself to the "Who's using GlusterFS page":
   http://www.gluster.org/docs/index.php/Who%27s_using_GlusterFS

* Provide testimonials:
   http://www.gluster.org/docs/index.php/GlusterFS_Testimonials

* Volunteer as a reference for venture capital technical due-diligence:
   Please write to Anand Babu <ab at zresearch.com> your name, email and
phone number.
   We promise, your information will be kept strictly confidential.

Thanks in advance,
Happy Hacking!
--
Gluster Team

I have an application which stores old copies of files whenever new 
ones are uploaded.

it does this by checking if file_exists in the archive folder
this has roughly 30000 items in it.

I''ve got an AFR configuration.

so, when someone uploads 10 files, it checks for the file existing, 
which does a directory scan which causes afr to check the other 
server, however this takes a reallllly long time.
and it does this for each file, which is a problem.

which performance translator would help the most with this.

the AFR configuration uses local read-volume.

will any performance translators help given it''s an AFR config?  I 
understand afr wants to maintain the directory integrity, but it''s 
kind of irritating.

the directory filesize is 2.2 mb so I''m not sure if it''s
actually
moving this volume of data constantly or not, but it''s painfully slow 
and things seem to timeout constantly.

any thoughts are appreciated.

Keith

I have an application which stores old copies of files whenever new 
ones are uploaded.

it does this by checking if file_exists in the archive folder
this has roughly 30000 items in it.

I''ve got an AFR configuration.

so, when someone uploads 10 files, it checks for the file existing, 
which does a directory scan which causes afr to check the other 
server, however this takes a reallllly long time.
and it does this for each file, which is a problem.

which performance translator would help the most with this.

the AFR configuration uses local read-volume.

will any performance translators help given it''s an AFR config?  I 
understand afr wants to maintain the directory integrity, but it''s 
kind of irritating.

the directory filesize is 2.2 mb so I''m not sure if it''s
actually
moving this volume of data constantly or not, but it''s painfully slow 
and things seem to timeout constantly.

any thoughts are appreciated.

Keith

I have an application which stores old copies of files whenever new 
ones are uploaded.

it does this by checking if file_exists in the archive folder
this has roughly 30000 items in it.

I've got an AFR configuration.

so, when someone uploads 10 files, it checks for the file existing, 
which does a directory scan which causes afr to check the other 
server, however this takes a reallllly long time.
and it does this for each file, which is a problem.

which performance translator would help the most with this.

the AFR configuration uses local read-volume.

will any performance translators help given it's an AFR config?  I 
understand afr wants to maintain the directory integrity, but it's 
kind of irritating.

the directory filesize is 2.2 mb so I'm not sure if it's actually 
moving this volume of data constantly or not, but it's painfully slow 
and things seem to timeout constantly.

any thoughts are appreciated.

Keith

I have an application which stores old copies of files whenever new 
ones are uploaded.

it does this by checking if file_exists in the archive folder
this has roughly 30000 items in it.

I''ve got an AFR configuration.

so, when someone uploads 10 files, it checks for the file existing, 
which does a directory scan which causes afr to check the other 
server, however this takes a reallllly long time.
and it does this for each file, which is a problem.

which performance translator would help the most with this.

the AFR configuration uses local read-volume.

will any performance translators help given it''s an AFR config?  I 
understand afr wants to maintain the directory integrity, but it''s 
kind of irritating.

the directory filesize is 2.2 mb so I''m not sure if it''s
actually
moving this volume of data constantly or not, but it''s painfully slow 
and things seem to timeout constantly.

any thoughts are appreciated.

Keith

Hi Keith,

Comments are inlined.

On Mon, Sep 8, 2008 at 3:11 AM, Keith Freedman <freedman at
freeformit.com>wrote:
> I have an application which stores old copies of files whenever new
> ones are uploaded.
>
> it does this by checking if file_exists in the archive folder
> this has roughly 30000 items in it.

How are you checking whether a file exists or not? do you  use  ls? Can you
make sure, that  ls doesn't  send a stat call on each of the files in the
directory (as in ls -l )? if ls just sends readdir calls and still the
performance is low, let us know about it.

If you need stat calls, you might need stat-prefetch, which is currently not
funcitonal.

>
> I've got an AFR configuration.
>
> so, when someone uploads 10 files, it checks for the file existing,
> which does a directory scan which causes afr to check the other
> server, however this takes a reallllly long time.
> and it does this for each file, which is a problem.
>
> which performance translator would help the most with this.
>
> the AFR configuration uses local read-volume.
>
> will any performance translators help given it's an AFR config?  I
> understand afr wants to maintain the directory integrity, but it's
> kind of irritating.
>
> the directory filesize is 2.2 mb so I'm not sure if it's actually
> moving this volume of data constantly or not, but it's painfully slow
> and things seem to timeout constantly.
>
> any thoughts are appreciated.
>
> Keith
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>


-- 
Raghavendra G

A centipede was happy quite, until a toad in fun,
Said, "Prey, which leg comes after which?",
This raised his doubts to such a pitch,
He fell flat into the ditch,
Not knowing how to run.
-Anonymous
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080908/b186d09d/attachment.html>

I just wanted to report back.

I''m now running 1.4pre5 on this set of AFRed servers
it seems since the installation it''s doing a LOT less chatter.
I haven''t done any thorough testing but the quick tests I did seems 
to indicate this situation is much improved.

I''ll report back if I find this not to be the case with 1.4

Keith

At 09:29 PM 9/7/2008, Raghavendra G wrote:
>Hi Keith,
>
>Comments are inlined.
>
>On Mon, Sep 8, 2008 at 3:11 AM, Keith Freedman 
><<mailto:freedman at freeformit.com>freedman at freeformit.com>
wrote:
>I have an application which stores old copies of files whenever new
>ones are uploaded.
>
>it does this by checking if file_exists in the archive folder
>this has roughly 30000 items in it.
>
>
>How are you checking whether a file exists or not? do you  use  ls? 
>Can you make sure, that  ls doesn''t  send a stat call on each of
the
>files in the directory (as in ls -l )? if ls just sends readdir 
>calls and still the performance is low, let us know about it.
>
>If you need stat calls, you might need stat-prefetch, which is 
>currently not funcitonal.
>
>
>
>I''ve got an AFR configuration.
>
>so, when someone uploads 10 files, it checks for the file existing,
>which does a directory scan which causes afr to check the other
>server, however this takes a reallllly long time.
>and it does this for each file, which is a problem.
>
>which performance translator would help the most with this.
>
>the AFR configuration uses local read-volume.
>
>will any performance translators help given it''s an AFR config?  I
>understand afr wants to maintain the directory integrity, but it''s
>kind of irritating.
>
>the directory filesize is 2.2 mb so I''m not sure if it''s
actually
>moving this volume of data constantly or not, but it''s painfully
slow
>and things seem to timeout constantly.
>
>any thoughts are appreciated.
>
>Keith
>
>
>_______________________________________________
>Gluster-users mailing list
><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org
>http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
>
>
>--
>Raghavendra G
>
>A centipede was happy quite, until a toad in fun,
>Said, "Prey, which leg comes after which?",
>This raised his doubts to such a pitch,
>He fell flat into the ditch,
>Not knowing how to run.
>-Anonymous

I just wanted to report back.

I''m now running 1.4pre5 on this set of AFRed servers
it seems since the installation it''s doing a LOT less chatter.
I haven''t done any thorough testing but the quick tests I did seems 
to indicate this situation is much improved.

I''ll report back if I find this not to be the case with 1.4

Keith

At 09:29 PM 9/7/2008, Raghavendra G wrote:
>Hi Keith,
>
>Comments are inlined.
>
>On Mon, Sep 8, 2008 at 3:11 AM, Keith Freedman 
><<mailto:freedman at freeformit.com>freedman at freeformit.com>
wrote:
>I have an application which stores old copies of files whenever new
>ones are uploaded.
>
>it does this by checking if file_exists in the archive folder
>this has roughly 30000 items in it.
>
>
>How are you checking whether a file exists or not? do you  use  ls? 
>Can you make sure, that  ls doesn''t  send a stat call on each of
the
>files in the directory (as in ls -l )? if ls just sends readdir 
>calls and still the performance is low, let us know about it.
>
>If you need stat calls, you might need stat-prefetch, which is 
>currently not funcitonal.
>
>
>
>I''ve got an AFR configuration.
>
>so, when someone uploads 10 files, it checks for the file existing,
>which does a directory scan which causes afr to check the other
>server, however this takes a reallllly long time.
>and it does this for each file, which is a problem.
>
>which performance translator would help the most with this.
>
>the AFR configuration uses local read-volume.
>
>will any performance translators help given it''s an AFR config?  I
>understand afr wants to maintain the directory integrity, but it''s
>kind of irritating.
>
>the directory filesize is 2.2 mb so I''m not sure if it''s
actually
>moving this volume of data constantly or not, but it''s painfully
slow
>and things seem to timeout constantly.
>
>any thoughts are appreciated.
>
>Keith
>
>
>_______________________________________________
>Gluster-users mailing list
><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org
>http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
>
>
>--
>Raghavendra G
>
>A centipede was happy quite, until a toad in fun,
>Said, "Prey, which leg comes after which?",
>This raised his doubts to such a pitch,
>He fell flat into the ditch,
>Not knowing how to run.
>-Anonymous

I just wanted to report back.

I''m now running 1.4pre5 on this set of AFRed servers
it seems since the installation it''s doing a LOT less chatter.
I haven''t done any thorough testing but the quick tests I did seems 
to indicate this situation is much improved.

I''ll report back if I find this not to be the case with 1.4

Keith

At 09:29 PM 9/7/2008, Raghavendra G wrote:
>Hi Keith,
>
>Comments are inlined.
>
>On Mon, Sep 8, 2008 at 3:11 AM, Keith Freedman 
><<mailto:freedman at freeformit.com>freedman at freeformit.com>
wrote:
>I have an application which stores old copies of files whenever new
>ones are uploaded.
>
>it does this by checking if file_exists in the archive folder
>this has roughly 30000 items in it.
>
>
>How are you checking whether a file exists or not? do you  use  ls? 
>Can you make sure, that  ls doesn''t  send a stat call on each of
the
>files in the directory (as in ls -l )? if ls just sends readdir 
>calls and still the performance is low, let us know about it.
>
>If you need stat calls, you might need stat-prefetch, which is 
>currently not funcitonal.
>
>
>
>I''ve got an AFR configuration.
>
>so, when someone uploads 10 files, it checks for the file existing,
>which does a directory scan which causes afr to check the other
>server, however this takes a reallllly long time.
>and it does this for each file, which is a problem.
>
>which performance translator would help the most with this.
>
>the AFR configuration uses local read-volume.
>
>will any performance translators help given it''s an AFR config?  I
>understand afr wants to maintain the directory integrity, but it''s
>kind of irritating.
>
>the directory filesize is 2.2 mb so I''m not sure if it''s
actually
>moving this volume of data constantly or not, but it''s painfully
slow
>and things seem to timeout constantly.
>
>any thoughts are appreciated.
>
>Keith
>
>
>_______________________________________________
>Gluster-users mailing list
><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org
>http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
>
>
>--
>Raghavendra G
>
>A centipede was happy quite, until a toad in fun,
>Said, "Prey, which leg comes after which?",
>This raised his doubts to such a pitch,
>He fell flat into the ditch,
>Not knowing how to run.
>-Anonymous