llvm dev - Aug 2016 - [RFC] GitHub Survey - Please review

Hi Chris,

Bear in mind that the more questions we have, the harder it will be to
interpret the results. If we have 20+ questions, it'll be impossible
to understand anything.

Also, the multiple choice questions are meant as a guide to understand
"how many" people fall into one or another category, while the free
text ones are meant to complement and give technical reasons for their
answers.

So, we should focus our multiple choice questions on divisive topics
and let everything else to the free text-fields.


On 19 August 2016 at 18:06, Chris Bieneman <beanz at apple.com>
wrote:
> (1) Which project(s) do you contribute to?
> (2) Which project(s) do you regularly use?
I've added these two as one. I know they're slightly different, but so
will be the the answer to the first question, which will work to
disambiguate this one.

> (3) How often do you bisect LLVM with one or more subproject?
I understand that this is a contentious issue around the git move, but
we should focus on the bigger picture, which is day to day usage as
well as infrastructure.

> (4) Do you use any of the llvm.org projects without LLVM or with
out-of-sync
> LLVM (i.e. trunk libunwind with an old LLVM)?
This looks very specific to me, I'm trying to avoid side questions
here and let people write up on the free text areas what their usage
is.

> (5) In which ways do you get LLVM sources from LLVM.org?
>    (a) SVN
>    (b) llvm.org Git mirrors
>    (c) Git-SVN
>    (d) GitHub Git mirrors
>    (e) Other
I don't think that previous usage is relevant. It may be relevant to
the people doing it and to their responses on how hard it will be, but
this should be encoded in the other questions. Some of that already
is.

> (6) Do you, or an organization you are affiliated with, maintain tooling or
> infrastructure that interacts with llvm.org and is not public?
This is a topic for the free-text fields.

 cheers,
--renato

> On Aug 19, 2016, at 11:04 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> 
> Hi Chris,
> 
> Bear in mind that the more questions we have, the harder it will be to
> interpret the results. If we have 20+ questions, it'll be impossible
> to understand anything.
> 
> Also, the multiple choice questions are meant as a guide to understand
> "how many" people fall into one or another category, while the
free
> text ones are meant to complement and give technical reasons for their
> answers.
> 
> So, we should focus our multiple choice questions on divisive topics
> and let everything else to the free text-fields.
> 
> 
> On 19 August 2016 at 18:06, Chris Bieneman <beanz at apple.com>
wrote:
>> (1) Which project(s) do you contribute to?
>> (2) Which project(s) do you regularly use?
> 
> I've added these two as one. I know they're slightly different, but
so
> will be the the answer to the first question, which will work to
> disambiguate this one.
> 
> 
>> (3) How often do you bisect LLVM with one or more subproject?
> 
> I understand that this is a contentious issue around the git move, but
> we should focus on the bigger picture, which is day to day usage as
> well as infrastructure.
I disagree. I think this is a really important thing to understand how people
interact with the SCM system. I think one part of the benefit of doing a survey
is that we can gather user data about how common specific workflows are.
> 
> 
>> (4) Do you use any of the llvm.org projects without LLVM or with
out-of-sync
>> LLVM (i.e. trunk libunwind with an old LLVM)?
> 
> This looks very specific to me, I'm trying to avoid side questions
> here and let people write up on the free text areas what their usage
> is.
Fair enough. I do think that relying on the write-up fields is potentially
difficult. If everyone who contributed to LLVM projects in the last year
responds to the survey you’re talking about somewhere on the order of 500
responses. More if non-contributors respond (and I hope they do). Weeding
through lots of write-in data could be come a monumental task. I think it is
better to have more yes/no and multiple choice questions to help collate the
data. Admittedly wording them neutrally will be a challenge, but I think it is
something we should do.
> 
> 
>> (5) In which ways do you get LLVM sources from LLVM.org?
>>   (a) SVN
>>   (b) llvm.org Git mirrors
>>   (c) Git-SVN
>>   (d) GitHub Git mirrors
>>   (e) Other
> 
> I don't think that previous usage is relevant. It may be relevant to
> the people doing it and to their responses on how hard it will be, but
> this should be encoded in the other questions. Some of that already
> is.
I also disagree on this. I think it is useful to know which workflows are common
in the community. It will help make informed decisions about the direction of
the infrastructure.
> 
> 
>> (6) Do you, or an organization you are affiliated with, maintain
tooling or
>> infrastructure that interacts with llvm.org and is not public?
> 
> This is a topic for the free-text fields.
I feel less invested in this question than in some of the others, because I
think we know that this is a fairly common situation (especially with the large
corporate contributors), but I still think relying on free-text fields for
questions that can be yes/no or multiple choice will degrade our ability to
process the information.

In general I think there is a sweet-spot on the length of the survey. We don’t
want to go overboard with too many questions, but I think that relying on text
fields could turn out to make the data very difficult to process if we get a lot
of respondents.

-Chris
> 
> cheers,
> --renato

On Fri, Aug 19, 2016 at 8:04 PM Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Chris,
>
> Bear in mind that the more questions we have, the harder it will be to
> interpret the results. If we have 20+ questions, it'll be impossible
> to understand anything.
>
> Also, the multiple choice questions are meant as a guide to understand
> "how many" people fall into one or another category, while the
free
> text ones are meant to complement and give technical reasons for their
> answers.
>
> So, we should focus our multiple choice questions on divisive topics
> and let everything else to the free text-fields.
>
>
> On 19 August 2016 at 18:06, Chris Bieneman <beanz at apple.com>
wrote:
> > (1) Which project(s) do you contribute to?
> > (2) Which project(s) do you regularly use?
>
> I've added these two as one. I know they're slightly different, but
so
> will be the the answer to the first question, which will work to
> disambiguate this one.
>
Wouldn't this question catch cases where someone uses prebuilt clang as
C/C++ compiler?

>
>
> > (3) How often do you bisect LLVM with one or more subproject?
>
> I understand that this is a contentious issue around the git move, but
> we should focus on the bigger picture, which is day to day usage as
> well as infrastructure.
>
>
> > (4) Do you use any of the llvm.org projects without LLVM or with
> out-of-sync
> > LLVM (i.e. trunk libunwind with an old LLVM)?
>
> This looks very specific to me, I'm trying to avoid side questions
> here and let people write up on the free text areas what their usage
> is.
>
>
> > (5) In which ways do you get LLVM sources from LLVM.org?
> >    (a) SVN
> >    (b) llvm.org Git mirrors
> >    (c) Git-SVN
> >    (d) GitHub Git mirrors
> >    (e) Other
>
> I don't think that previous usage is relevant. It may be relevant to
> the people doing it and to their responses on how hard it will be, but
> this should be encoded in the other questions. Some of that already
> is.
>
>
> > (6) Do you, or an organization you are affiliated with, maintain
tooling
> or
> > infrastructure that interacts with llvm.org and is not public?
>
> This is a topic for the free-text fields.
>
>  cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160819/35386952/attachment.html>

On 19 August 2016 at 20:35, Paweł Bylica <chfast at gmail.com>
wrote:
> Wouldn't this question catch cases where someone uses prebuilt clang as
> C/C++ compiler?
Possibly. But not really relevant to the survey.

--renato

> On Aug 19, 2016, at 11:19 AM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> 
>> On Aug 19, 2016, at 11:04 AM, Renato Golin <renato.golin at
linaro.org> wrote:
>> 
>> Hi Chris,
>> 
>> Bear in mind that the more questions we have, the harder it will be to
>> interpret the results. If we have 20+ questions, it'll be
impossible
>> to understand anything.
>> 
>> Also, the multiple choice questions are meant as a guide to understand
>> "how many" people fall into one or another category, while
the free
>> text ones are meant to complement and give technical reasons for their
>> answers.
>> 
>> So, we should focus our multiple choice questions on divisive topics
>> and let everything else to the free text-fields.
>> 
>> 
>> On 19 August 2016 at 18:06, Chris Bieneman <beanz at apple.com>
wrote:
>>> (1) Which project(s) do you contribute to?
>>> (2) Which project(s) do you regularly use?
>> 
>> I've added these two as one. I know they're slightly different,
but so
>> will be the the answer to the first question, which will work to
>> disambiguate this one.
>> 
>> 
>>> (3) How often do you bisect LLVM with one or more subproject?
>> 
>> I understand that this is a contentious issue around the git move, but
>> we should focus on the bigger picture, which is day to day usage as
>> well as infrastructure.
> 
> I disagree. I think this is a really important thing to understand how
people interact with the SCM system. I think one part of the benefit of doing a
survey is that we can gather user data about how common specific workflows are.
The data you’ll get with this question is biased because the answer will be
constrained by current limitations: what if I don’t bisect because the current
structure does not help? If you want to go this route, you should also ask
something like "how much would you bisect if it was made super easy to do
so” for instance.

— 
Mehdi

> 
>> 
>> 
>>> (4) Do you use any of the llvm.org <http://llvm.org/>
projects without LLVM or with out-of-sync
>>> LLVM (i.e. trunk libunwind with an old LLVM)?
>> 
>> This looks very specific to me, I'm trying to avoid side questions
>> here and let people write up on the free text areas what their usage
>> is.
> 
> Fair enough. I do think that relying on the write-up fields is potentially
difficult. If everyone who contributed to LLVM projects in the last year
responds to the survey you’re talking about somewhere on the order of 500
responses. More if non-contributors respond (and I hope they do). Weeding
through lots of write-in data could be come a monumental task. I think it is
better to have more yes/no and multiple choice questions to help collate the
data. Admittedly wording them neutrally will be a challenge, but I think it is
something we should do.
> 
>> 
>> 
>>> (5) In which ways do you get LLVM sources from LLVM.org
<http://llvm.org/>?
>>>  (a) SVN
>>>  (b) llvm.org <http://llvm.org/> Git mirrors
>>>  (c) Git-SVN
>>>  (d) GitHub Git mirrors
>>>  (e) Other
>> 
>> I don't think that previous usage is relevant. It may be relevant
to
>> the people doing it and to their responses on how hard it will be, but
>> this should be encoded in the other questions. Some of that already
>> is.
> 
> I also disagree on this. I think it is useful to know which workflows are
common in the community. It will help make informed decisions about the
direction of the infrastructure.
> 
>> 
>> 
>>> (6) Do you, or an organization you are affiliated with, maintain
tooling or
>>> infrastructure that interacts with llvm.org
<http://llvm.org/> and is not public?
>> 
>> This is a topic for the free-text fields.
> 
> I feel less invested in this question than in some of the others, because I
think we know that this is a fairly common situation (especially with the large
corporate contributors), but I still think relying on free-text fields for
questions that can be yes/no or multiple choice will degrade our ability to
process the information.
> 
> In general I think there is a sweet-spot on the length of the survey. We
don’t want to go overboard with too many questions, but I think that relying on
text fields could turn out to make the data very difficult to process if we get
a lot of respondents.
> 
> -Chris
> 
>> 
>> cheers,
>> --renato
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160819/27f7685a/attachment.html>