Raghavendra Gowdappa
2018-Jul-06 05:27 UTC
[Gluster-users] Slow write times to gluster disk
On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <phaley at mit.edu> wrote:> > Hi Raghavendra, > > Our technician may have some time to look at this issue tomorrow. Are > there any tests that you'd like to see? >Sorry. I've been busy with other things and was away from work for couple of days. It'll take me another 2 days to work on this issue again. So, most likely you'll have an update on this next week.> Thanks > > Pat > > > > On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote: > > > > On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu> wrote: > >> >> Hi Raghavendra, >> >> We ran the tests (write tests) and I copied the log files for both the >> server and the client to http://mseas.mit.edu/download/ >> phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace >> information you need? (If so, where should I look for it?) >> > > Nothing for now. I can see from logs that workaround is not helping. fstat > requests are not absorbed by md-cache and read-ahead is witnessing them and > flushing its read-ahead cache. I am investigating more on md-cache (It also > seems to be invalidating inodes quite frequently which actually might be > the root cause of seeing so many fstat requests from kernel). Will post > when I find anything relevant. > > >> Also the volume information you requested >> >> [root at mseas-data2 ~]# gluster volume info data-volume >> >> Volume Name: data-volume >> Type: Distribute >> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: mseas-data2:/mnt/brick1 >> Brick2: mseas-data2:/mnt/brick2 >> Options Reconfigured: >> diagnostics.client-log-level: TRACE >> network.inode-lru-limit: 50000 >> performance.md-cache-timeout: 60 >> performance.open-behind: off >> disperse.eager-lock: off >> auth.allow: * >> server.allow-insecure: on >> nfs.exports-auth-enable: on >> diagnostics.brick-sys-log-level: WARNING >> performance.readdir-ahead: on >> nfs.disable: on >> nfs.export-volumes: off >> [root at mseas-data2 ~]# >> >> >> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote: >> >> >> >> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu> wrote: >> >>> >>> Hi Raghavendra, >>> >>> Our technician was able to try the manual setting today. He found that >>> our upper limit for performance.md-cache-timeout was 60 not 600, so he >>> used that value, along with the network.inode-lru-limit=50000. >>> >>> The result was another small (~1%) increase in speed. Does this suggest >>> some addition tests/changes we could try? >>> >> >> Can you set gluster option diagnostics.client-log-level to TRACE and run >> sequential read tests again (with md-cache-timeout value of 60)? >> >> #gluster volume set <volname> diagnostics.client-log-level TRACE >> >> Also are you sure that open-behind was turned off? Can you give the >> output of, >> >> # gluster volume info <volname> >> >> >>> Thanks >>> >>> Pat >>> >>> >>> >>> >>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote: >>> >>> >>> >>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <phaley at mit.edu> wrote: >>> >>>> >>>> Hi Raghavendra, >>>> >>>> Setting the performance.write-behind off had a small improvement on the >>>> write speed (~3%), >>>> >>>> We were unable to turn on "group metadata-cache". When we try get >>>> errors like >>>> >>>> # gluster volume set data-volume group metadata-cache >>>> '/var/lib/glusterd/groups/metadata-cache' file format not valid. >>>> >>>> Was metadata-cache available for gluster 3.7.11? We ask because the >>>> release notes for 3.11 mentions ?Feature for metadata-caching/small file >>>> performance is production ready.? (https://gluster.readthedocs.i >>>> o/en/latest/release-notes/3.11.0/). >>>> >>>> Do any of these results suggest anything? If not, what further tests >>>> would be useful? >>>> >>> >>> Group metadata-cache is just a bunch of options one sets on a volume. >>> So, You can set them manually using gluster cli. Following are the options >>> and their values: >>> >>> performance.md-cache-timeout=600 >>> network.inode-lru-limit=50000 >>> >>> >>> >>>> Thanks >>>> >>>> Pat >>>> >>>> >>>> >>>> >>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote: >>>> >>>> >>>> >>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <phaley at mit.edu> wrote: >>>> >>>>> >>>>> Hi Raghavendra, >>>>> >>>>> Thanks for the suggestions. Our technician will be in on Monday. >>>>> We'll test then and let you know the results. >>>>> >>>>> One question I have, is the "group metadata-cache" option supposed to >>>>> directly impact the performance or is it to help collect data? If the >>>>> latter, where will the data be located? >>>>> >>>> >>>> It impacts performance. >>>> >>>> >>>>> Thanks again. >>>>> >>>>> Pat >>>>> >>>>> >>>>> >>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote: >>>>> >>>>> >>>>> >>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa < >>>>> rgowdapp at redhat.com> wrote: >>>>> >>>>>> For the case of writes to glusterfs mount, >>>>>> >>>>>> I saw in earlier conversations that there are too many lookups, but >>>>>> small number of writes. Since writes cached in write-behind would >>>>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am >>>>>> wondering what would results look like if we turn off >>>>>> performance.write-behind. >>>>>> >>>>>> @Pat, >>>>>> >>>>>> Can you set, >>>>>> >>>>>> # gluster volume set <volname> performance.write-behind off >>>>>> >>>>> >>>>> Please turn on "group metadata-cache" for write tests too. >>>>> >>>>> >>>>>> and redo the tests writing to glusterfs mount? Let us know about the >>>>>> results you see. >>>>>> >>>>>> regards, >>>>>> Raghavendra >>>>>> >>>>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa < >>>>>> rgowdapp at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa < >>>>>>> rgowdapp at redhat.com> wrote: >>>>>>> >>>>>>>> For the case of reading from Glusterfs mount, read-ahead should >>>>>>>> help. However, we've known issues with read-ahead[1][2]. To work around >>>>>>>> these, can you try with, >>>>>>>> >>>>>>>> 1. Turn off performance.open-behind >>>>>>>> #gluster volume set <volname> performance.open-behind off >>>>>>>> >>>>>>>> 2. enable group meta metadata-cache >>>>>>>> # gluster volume set <volname> group metadata-cache >>>>>>>> >>>>>>> >>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508 >>>>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <phaley at mit.edu> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> We were recently revisiting our problems with the slowness of >>>>>>>>> gluster writes (http://lists.gluster.org/pipe >>>>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we were >>>>>>>>> testing the suggestions in a recent post ( >>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March >>>>>>>>> /033699.html). The first two suggestions (specifying a >>>>>>>>> negative-timeout in the mount settings or adding rpc-auth-allow-insecure to >>>>>>>>> glusterd.vol) did not improve our performance, while setting >>>>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up. >>>>>>>>> >>>>>>>>> Some of the various tests we have tried earlier can be seen in the >>>>>>>>> links below. Do any of the above observations suggest what we could try >>>>>>>>> next to either improve the speed or debug the issue? Thanks >>>>>>>>> >>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0 >>>>>>>>> 31565.html >>>>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03 >>>>>>>>> 0937.html >>>>>>>>> >>>>>>>>> Pat >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>>>>> Pat Haley Email: phaley at mit.edu >>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>>>>>> 77 Massachusetts Avenue >>>>>>>>> Cambridge, MA 02139-4301 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>> Pat Haley Email: phaley at mit.edu >>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>> 77 Massachusetts Avenue >>>>> Cambridge, MA 02139-4301 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>>> -- >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>> Pat Haley Email: phaley at mit.edu >>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>> 77 Massachusetts Avenue >>>> Cambridge, MA 02139-4301 >>>> >>>> >>> >>> -- >>> >>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>> Pat Haley Email: phaley at mit.edu >>> Center for Ocean Engineering Phone: (617) 253-6824 >>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>> 77 Massachusetts Avenue >>> Cambridge, MA 02139-4301 >>> >>> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: phaley at mit.edu >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> > > -- > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email: phaley at mit.edu > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180706/370baa7c/attachment.html>
Hi Raghavendra, We were wondering if you have had a chance to look at this again, and if so, did you have any further suggestions? Thanks Pat On 07/06/2018 01:27 AM, Raghavendra Gowdappa wrote:> > > On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <phaley at mit.edu > <mailto:phaley at mit.edu>> wrote: > > > Hi Raghavendra, > > Our technician may have some time to look at this issue tomorrow.? > Are there any tests that you'd like to see? > > > Sorry. I've been busy with other things and was away from work for > couple of days. It'll take me another 2 days to work on this issue > again. So, most likely you'll have an update on this next week. > > > Thanks > > Pat > > > > On 06/29/2018 11:25 PM, Raghavendra Gowdappa wrote: >> >> >> On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <phaley at mit.edu >> <mailto:phaley at mit.edu>> wrote: >> >> >> Hi Raghavendra, >> >> We ran the tests (write tests) and I copied the log files for >> both the server and the client to >> http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/ >> <http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/> >> .? Is there any additional trace information you need?? (If >> so, where should I look for it?) >> >> >> Nothing for now. I can see from logs that workaround is not >> helping. fstat requests are not absorbed by md-cache and >> read-ahead is witnessing them and flushing its read-ahead cache. >> I am investigating more on md-cache (It also seems to be >> invalidating inodes quite frequently which actually might be the >> root cause of seeing so many fstat requests from kernel). Will >> post when I find anything relevant. >> >> >> Also the volume information you requested >> >> [root at mseas-data2 ~]# gluster volume info data-volume >> >> Volume Name: data-volume >> Type: Distribute >> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 >> Status: Started >> Number of Bricks: 2 >> Transport-type: tcp >> Bricks: >> Brick1: mseas-data2:/mnt/brick1 >> Brick2: mseas-data2:/mnt/brick2 >> Options Reconfigured: >> diagnostics.client-log-level: TRACE >> network.inode-lru-limit: 50000 >> performance.md-cache-timeout: 60 >> performance.open-behind: off >> disperse.eager-lock: off >> auth.allow: * >> server.allow-insecure: on >> nfs.exports-auth-enable: on >> diagnostics.brick-sys-log-level: WARNING >> performance.readdir-ahead: on >> nfs.disable: on >> nfs.export-volumes: off >> [root at mseas-data2 ~]# >> >> >> On 06/29/2018 12:28 PM, Raghavendra Gowdappa wrote: >>> >>> >>> On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <phaley at mit.edu >>> <mailto:phaley at mit.edu>> wrote: >>> >>> >>> Hi Raghavendra, >>> >>> Our technician was able to try the manual setting >>> today.? He found that our upper limit for >>> performance.md-cache-timeout was 60 not 600, so he used >>> that value, along with the network.inode-lru-limit=50000. >>> >>> The result was another small (~1%) increase in speed.? >>> Does this suggest some addition tests/changes we could try? >>> >>> >>> Can you set gluster option diagnostics.client-log-level to >>> TRACE? and run sequential read tests again (with >>> md-cache-timeout value of 60)? >>> >>> #gluster volume set <volname> diagnostics.client-log-level TRACE >>> >>> Also are you sure that open-behind was turned off? Can you >>> give the output of, >>> >>> # gluster volume info <volname> >>> >>> >>> Thanks >>> >>> Pat >>> >>> >>> >>> >>> On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote: >>>> >>>> >>>> On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley >>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote: >>>> >>>> >>>> Hi Raghavendra, >>>> >>>> Setting the performance.write-behind off had a >>>> small improvement on the write speed (~3%), >>>> >>>> We were unable to turn on "group metadata-cache". >>>> When we try get errors like >>>> >>>> # gluster volume set data-volume group metadata-cache >>>> '/var/lib/glusterd/groups/metadata-cache' file >>>> format not valid. >>>> >>>> Was metadata-cache available for gluster 3.7.11? We >>>> ask because the release notes for 3.11 mentions >>>> ?Feature for metadata-caching/small file >>>> performance is production ready.? >>>> (https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/ >>>> <https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>). >>>> >>>> Do any of these results suggest anything?? If not, >>>> what further tests would be useful? >>>> >>>> >>>> Group metadata-cache is just a bunch of options one >>>> sets on a volume. So, You can set them manually using >>>> gluster cli. Following are the options and their values: >>>> >>>> performance.md-cache-timeout=600 >>>> network.inode-lru-limit=50000 >>>> >>>> >>>> >>>> Thanks >>>> >>>> Pat >>>> >>>> >>>> >>>> >>>> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote: >>>>> >>>>> >>>>> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley >>>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote: >>>>> >>>>> >>>>> Hi Raghavendra, >>>>> >>>>> Thanks for the suggestions. Our technician >>>>> will be in on Monday.? We'll test then and let >>>>> you know the results. >>>>> >>>>> One question I have, is the "group >>>>> metadata-cache" option supposed to directly >>>>> impact the performance or is it to help >>>>> collect data? If the latter, where will the >>>>> data be located? >>>>> >>>>> >>>>> It impacts performance. >>>>> >>>>> >>>>> Thanks again. >>>>> >>>>> Pat >>>>> >>>>> >>>>> >>>>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa >>>>> wrote: >>>>>> >>>>>> >>>>>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra >>>>>> Gowdappa <rgowdapp at redhat.com >>>>>> <mailto:rgowdapp at redhat.com>> wrote: >>>>>> >>>>>> For the case of writes to glusterfs mount, >>>>>> >>>>>> I saw in earlier conversations that there >>>>>> are too many lookups, but small number of >>>>>> writes. Since writes cached in >>>>>> write-behind would invalidate metadata >>>>>> cache, lookups won't be absorbed by >>>>>> md-cache. I am wondering what would >>>>>> results look like if we turn off >>>>>> performance.write-behind. >>>>>> >>>>>> @Pat, >>>>>> >>>>>> Can you set, >>>>>> >>>>>> # gluster volume set <volname> >>>>>> performance.write-behind off >>>>>> >>>>>> >>>>>> Please turn on "group metadata-cache" for >>>>>> write tests too. >>>>>> >>>>>> >>>>>> and redo the tests writing to glusterfs >>>>>> mount? Let us know about the results you see. >>>>>> >>>>>> regards, >>>>>> Raghavendra >>>>>> >>>>>> On Thu, Jun 21, 2018 at 8:33 AM, >>>>>> Raghavendra Gowdappa <rgowdapp at redhat.com >>>>>> <mailto:rgowdapp at redhat.com>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 21, 2018 at 8:32 AM, >>>>>> Raghavendra Gowdappa >>>>>> <rgowdapp at redhat.com >>>>>> <mailto:rgowdapp at redhat.com>> wrote: >>>>>> >>>>>> For the case of reading from >>>>>> Glusterfs mount, read-ahead >>>>>> should help. However, we've known >>>>>> issues with read-ahead[1][2]. To >>>>>> work around these, can you try with, >>>>>> >>>>>> 1. Turn off performance.open-behind >>>>>> #gluster volume set <volname> >>>>>> performance.open-behind off >>>>>> >>>>>> 2. enable group meta metadata-cache >>>>>> # gluster volume set <volname> >>>>>> group metadata-cache >>>>>> >>>>>> >>>>>> [1] >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1084508 >>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1084508> >>>>>> [2] >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1214489 >>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1214489> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 21, 2018 at 5:00 AM, >>>>>> Pat Haley <phaley at mit.edu >>>>>> <mailto:phaley at mit.edu>> wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> We were recently revisiting >>>>>> our problems with the >>>>>> slowness of gluster writes >>>>>> (http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html >>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>). >>>>>> Specifically we were testing >>>>>> the suggestions in a recent >>>>>> post >>>>>> (http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html >>>>>> <http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>). >>>>>> The first two suggestions >>>>>> (specifying a >>>>>> negative-timeout in the mount >>>>>> settings or adding >>>>>> rpc-auth-allow-insecure to >>>>>> glusterd.vol) did not improve >>>>>> our performance, while >>>>>> setting "disperse.eager-lock >>>>>> off" provided a tiny (5%) >>>>>> speed-up. >>>>>> >>>>>> Some of the various tests we >>>>>> have tried earlier can be >>>>>> seen in the links below. Do >>>>>> any of the above observations >>>>>> suggest what we could try >>>>>> next to either improve the >>>>>> speed or debug the issue? Thanks >>>>>> >>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html >>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html> >>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html >>>>>> <http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html> >>>>>> >>>>>> Pat >>>>>> >>>>>> -- >>>>>> >>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>> Pat Haley ? ? ? Email: >>>>>> phaley at mit.edu >>>>>> <mailto:phaley at mit.edu> >>>>>> Center for Ocean Engineering >>>>>> ? ?Phone: (617) 253-6824 >>>>>> Dept. of Mechanical >>>>>> Engineering Fax:? ? (617) >>>>>> 253-8125 >>>>>> MIT, Room 5-213 >>>>>> http://web.mit.edu/phaley/www/ >>>>>> 77 Massachusetts Avenue >>>>>> Cambridge, MA 02139-4301 >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> <mailto:Gluster-users at gluster.org> >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> >>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/ >>>>> 77 Massachusetts Avenue >>>>> Cambridge, MA 02139-4301 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> <mailto:Gluster-users at gluster.org> >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>>>> >>>>> >>>> >>>> -- >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> >>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>> MIT, Room 5-213http://web.mit.edu/phaley/www/ >>>> 77 Massachusetts Avenue >>>> Cambridge, MA 02139-4301 >>>> >>>> >>> >>> -- >>> >>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> >>> Center for Ocean Engineering Phone: (617) 253-6824 >>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>> MIT, Room 5-213http://web.mit.edu/phaley/www/ >>> 77 Massachusetts Avenue >>> Cambridge, MA 02139-4301 >>> >>> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> > > -- > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > >-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180712/39aa640a/attachment.html>