llvm dev - Sep 2016 - [XRay][RFC] Tooling for XRay Trace Analysis

On Thu, Sep 8, 2016 at 11:34 PM Dean Michael Berris <dean.berris at
gmail.com>
wrote:
>
> > On 9 Sep 2016, at 12:35, Dean Michael Berris <dean.berris at
gmail.com>
> wrote:
> >
> >
> >
> >> On 7 Sep 2016, at 01:21, David Blaikie <dblaikie at
gmail.com> wrote:
> >>
> >> But I take it you mean (as detailed later) to have a separate
format
> (could be a portable binary format, but currently discussing it as
> JSON/YAML/etc) that things are converted from that makes them portable?
> >>
>
> One thing worth mentioning on this regard is maybe we can use Flatbuffers (
> https://google.github.io/flatbuffers/) for the XRay log.
>
For the secondary form (instead of YAML/JSON) or the primary form (the
thing compiler-rt writes out)? Or as a form that would cover both use cases
without the need to convert from one format to another?

>
> Flatbuffers is Apache 2 licensed though, and if we're going to use it
in
> compiler-rt, whether that raises issues if embedded in other people's
> applications.
>
> This might potentially get us around the non-sharing of code if we're
able
> to keep the flatbuffer definitions in sync at least across compiler-rt and
> LLVM. Also, it might be useful to use flatbuffers for other things in LLVM
> as well so that might be something worth exploring (the bit code comes to
> mind, somehow being discussed on IRC recently).
>
> I am not a lawyer nor do I play one on the Internet, so I'll let Danny
> Berlin chime in on this one on whether/how we can use Flatbuffers for XRay.
>
> -- Dean
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160909/ba892013/attachment.html>

> On 10 Sep 2016, at 02:05, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> 
> 
> On Thu, Sep 8, 2016 at 11:34 PM Dean Michael Berris <dean.berris at
gmail.com> wrote:
> 
> > On 9 Sep 2016, at 12:35, Dean Michael Berris <dean.berris at
gmail.com> wrote:
> >
> >
> >
> >> On 7 Sep 2016, at 01:21, David Blaikie <dblaikie at
gmail.com> wrote:
> >>
> >> But I take it you mean (as detailed later) to have a separate
format (could be a portable binary format, but currently discussing it as
JSON/YAML/etc) that things are converted from that makes them portable?
> >>
> 
> One thing worth mentioning on this regard is maybe we can use Flatbuffers
(https://google.github.io/flatbuffers/) for the XRay log.
> 
> For the secondary form (instead of YAML/JSON) or the primary form (the
thing compiler-rt writes out)? Or as a form that would cover both use cases
without the need to convert from one format to another?
>  
The primary form -- I suspect the costs here would be acceptable it if it means
that we're able to write it once in a single format and make that
automatically be durable and portable. I suspect the bit swizzling is going to
be dwarfed by the cost of writing to stable media anyway that it could be worth
it.

Now this is still to be measured, but according to the benchmarks
(https://google.github.io/flatbuffers/flatbuffers_benchmarks.html) encoding 1M
times takes just 3.2 seconds -- ~3 microseconds each, right in the acceptable
range of performance for I/O.

This allows us to write the tool to just deal with flat buffers that just works
across platforms.

Does that make sense?

-- Dean

On Sun, Sep 11, 2016 at 10:24 PM Dean Michael Berris <dean.berris at
gmail.com>
wrote:
>
> > On 10 Sep 2016, at 02:05, David Blaikie <dblaikie at gmail.com>
wrote:
> >
> >
> >
> > On Thu, Sep 8, 2016 at 11:34 PM Dean Michael Berris <
> dean.berris at gmail.com> wrote:
> >
> > > On 9 Sep 2016, at 12:35, Dean Michael Berris <dean.berris at
gmail.com>
> wrote:
> > >
> > >
> > >
> > >> On 7 Sep 2016, at 01:21, David Blaikie <dblaikie at
gmail.com> wrote:
> > >>
> > >> But I take it you mean (as detailed later) to have a separate
format
> (could be a portable binary format, but currently discussing it as
> JSON/YAML/etc) that things are converted from that makes them portable?
> > >>
> >
> > One thing worth mentioning on this regard is maybe we can use
> Flatbuffers (https://google.github.io/flatbuffers/) for the XRay log.
> >
> > For the secondary form (instead of YAML/JSON) or the primary form (the
> thing compiler-rt writes out)? Or as a form that would cover both use cases
> without the need to convert from one format to another?
> >
>
> The primary form -- I suspect the costs here would be acceptable it if it
> means that we're able to write it once in a single format and make that
> automatically be durable and portable. I suspect the bit swizzling is going
> to be dwarfed by the cost of writing to stable media anyway that it could
> be worth it.
>
> Now this is still to be measured, but according to the benchmarks (
> https://google.github.io/flatbuffers/flatbuffers_benchmarks.html)
> encoding 1M times takes just 3.2 seconds -- ~3 microseconds each, right in
> the acceptable range of performance for I/O.
>
> This allows us to write the tool to just deal with flat buffers that just
> works across platforms.
>
> Does that make sense?
>
Yep. sounds good - though I assume you want to make progress in the interim
before any decision on flatbuffers is made?

>
> -- Dean
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160912/bcdc482f/attachment.html>