Lustre discuss - Feb 2011 - advice on OST journals in OSS failover pairs

We are planning OSS failover pairs for a lustre deployment, the question of
external journals comes up.
testing showed that for our applications having an external journal for the OSTs
provides a worthwhile increase in performance.

For a failover pair this presumably means that the OST journals should have to
reside on an external storage target that each OSSs pair can see?
Or would the MDS journal take over in the case of an OSS switchover?

thanks 

sam aparicio

Professor Samuel Aparicio BM BCh PhD FRCPath
Nan and Lorraine Robertson Chair UBC/BC Cancer Agency

Hello!

On Feb 13, 2011, at 6:54 PM, Samuel Aparicio wrote:
> We are planning OSS failover pairs for a lustre deployment, the question of
external journals comes up.
> testing showed that for our applications having an external journal for the
OSTs provides a worthwhile increase in performance.
> 
> For a failover pair this presumably means that the OST journals should have
to reside on an external storage target that each OSSs pair can see?
Yes, basically you need two LUNs visible on both nodes per OST. One for the data
and one for the journal.
> Or would the MDS journal take over in the case of an OSS switchover?
MDS journal is just for the MDS, it has nothing to do with OSTs.

Also you might want try async journal commits (+ make your journals big) to see
if that improves your performance in a similar way to an external journal.

Bye,
    Oleg

Professor Samuel Aparicio BM BCh PhD FRCPath
Nan and Lorraine Robertson Chair UBC/BC Cancer Agency
> 
> Also you might want try async journal commits (+ make your journals big) to
see if that improves your performance in a similar way to an external journal.
> 
> Bye,
>    Oleg
we thought about testing this but I also saw a number of bug reports around
async commits in failover configurations. Is this stable/available for 1.8.5?

Hello!

On Feb 13, 2011, at 7:18 PM, Samuel Aparicio wrote:
>> Also you might want try async journal commits (+ make your journals
big) to see if that improves your performance in a similar way to an external
journal.
> we thought about testing this but I also saw a number of bug reports around
async commits in failover configurations. Is this stable/available for 1.8.5?
It is believed that all bugs are currently fixed.
I believe that Oak Ridge National Laboratory uses async journals for quite a
while on their systems.
There are some other big users, I think.

Bye,
    Oleg

Indeed. I believe that I am right in saying that LLNL and all DDN sites 
have async journal commits on by default. The issues that I am aware of 
were reported against the pre-release version of the code while it was 
still in development.

On 11-02-13 4:24 PM, Oleg Drokin wrote:
> Hello!
>
> On Feb 13, 2011, at 7:18 PM, Samuel Aparicio wrote:
>
>>> Also you might want try async journal commits (+ make your journals
big) to see if that improves your performance in a similar way to an external
journal.
>> we thought about testing this but I also saw a number of bug reports
around async commits in failover configurations. Is this stable/available for
1.8.5?
> It is believed that all bugs are currently fixed.
> I believe that Oak Ridge National Laboratory uses async journals for quite
a while on their systems.
> There are some other big users, I think.
>
> Bye,
>      Oleg
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>

The external journal is an important part of whatever filesystem is using it, so
it needs to be available on any node that is serving this filesystem (i.e. on
the primary or failover OSS for each OST).

Cheers, Andreas

On 2011-02-13, at 16:54, Samuel Aparicio <saparicio at bccrc.ca> wrote:
> We are planning OSS failover pairs for a lustre deployment, the question of
external journals comes up.
> testing showed that for our applications having an external journal for the
OSTs provides a worthwhile increase in performance.
> 
> For a failover pair this presumably means that the OST journals should have
to reside on an external storage target that each OSSs pair can see?
> Or would the MDS journal take over in the case of an OSS switchover?
> 
> thanks 
> 
> sam aparicio
> 
> Professor Samuel Aparicio BM BCh PhD FRCPath
> Nan and Lorraine Robertson Chair UBC/BC Cancer Agency
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

One known issue is that the memory requirements on the clients is increased with
async journals in the case of few clients writing to the filesystem.

Cheers, Andreas

On 2011-02-13, at 17:24, Oleg Drokin <green at whamcloud.com> wrote:
> Hello!
> 
> On Feb 13, 2011, at 7:18 PM, Samuel Aparicio wrote:
> 
>>> Also you might want try async journal commits (+ make your journals
big) to see if that improves your performance in a similar way to an external
journal.
>> we thought about testing this but I also saw a number of bug reports
around async commits in failover configurations. Is this stable/available for
1.8.5?
> 
> It is believed that all bugs are currently fixed.
> I believe that Oak Ridge National Laboratory uses async journals for quite
a while on their systems.
> There are some other big users, I think.
> 
> Bye,
>    Oleg
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

True. Each client might pin memory amount equal to amount of written data from
this client for up to 7 seconds.

On Feb 13, 2011, at 9:07 PM, Andreas Dilger wrote:
> One known issue is that the memory requirements on the clients is increased
with async journals in the case of few clients writing to the filesystem.
> 
> Cheers, Andreas
> 
> On 2011-02-13, at 17:24, Oleg Drokin <green at whamcloud.com> wrote:
> 
>> Hello!
>> 
>> On Feb 13, 2011, at 7:18 PM, Samuel Aparicio wrote:
>> 
>>>> Also you might want try async journal commits (+ make your
journals big) to see if that improves your performance in a similar way to an
external journal.
>>> we thought about testing this but I also saw a number of bug
reports around async commits in failover configurations. Is this
stable/available for 1.8.5?
>> 
>> It is believed that all bugs are currently fixed.
>> I believe that Oak Ridge National Laboratory uses async journals for
quite a while on their systems.
>> There are some other big users, I think.
>> 
>> Bye,
>>   Oleg
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss

You will find that async jounral commits don''t help O_DIRECT I/O
though.
 So, maybe you should think about using both.

Thanks,
-Cory

On 02/13/2011 06:04 PM, Oleg Drokin wrote:
> Hello!
> 
> On Feb 13, 2011, at 6:54 PM, Samuel Aparicio wrote:
> 
>> We are planning OSS failover pairs for a lustre deployment, the
question of external journals comes up.
>> testing showed that for our applications having an external journal for
the OSTs provides a worthwhile increase in performance.
>>
>> For a failover pair this presumably means that the OST journals should
have to reside on an external storage target that each OSSs pair can see?
> 
> Yes, basically you need two LUNs visible on both nodes per OST. One for the
data and one for the journal.
> 
>> Or would the MDS journal take over in the case of an OSS switchover?
> 
> MDS journal is just for the MDS, it has nothing to do with OSTs.
> 
> Also you might want try async journal commits (+ make your journals big) to
see if that improves your performance in a similar way to an external journal.
> 
> Bye,
>     Oleg
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

On Mon, 2011-02-14 at 09:46 -0600, Cory Spitz wrote:
> You will find that async jounral commits don''t help O_DIRECT I/O
though.
>  So, maybe you should think about using both.
It definitely depends on your workload.

It''s not in the paper at
http://www.usenix.org/events/fast10/tech/full_papers/oral.pdf

but I seem to recall that using external journals in conjunction with
async commits reduced the performance benefits from the async commits.
There does seem to be some bottlenecks with the external journalling
code paths, but since async worked so well for our workloads, we never
tried to run them down.

-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office