Nouveau - Jul 2014 - [PATCH 09/17] drm/radeon: use common fence implementation for fences

On Tue, Jul 22, 2014 at 02:19:57PM +0200, Christian K?nig
wrote:
> Am 22.07.2014 13:57, schrieb Daniel Vetter:
> >On Tue, Jul 22, 2014 at 01:46:07PM +0200, Daniel Vetter wrote:
> >>On Tue, Jul 22, 2014 at 10:43:13AM +0200, Christian K?nig wrote:
> >>>Am 22.07.2014 06:05, schrieb Dave Airlie:
> >>>>On 9 July 2014 22:29, Maarten Lankhorst
<maarten.lankhorst at canonical.com> wrote:
> >>>>>Signed-off-by: Maarten Lankhorst <maarten.lankhorst
at canonical.com>
> >>>>>---
> >>>>>  drivers/gpu/drm/radeon/radeon.h        |   15 +-
> >>>>>  drivers/gpu/drm/radeon/radeon_device.c |   60
++++++++-
> >>>>>  drivers/gpu/drm/radeon/radeon_fence.c  |  223
++++++++++++++++++++++++++------
> >>>>>  3 files changed, 248 insertions(+), 50 deletions(-)
> >>>>>
> >>>> From what I can see this is still suffering from the
problem that we
> >>>>need to find a proper solution to,
> >>>>
> >>>>My summary of the issues after talking to Jerome and Ben
and
> >>>>re-reading things is:
> >>>>
> >>>>We really need to work out a better interface into the
drivers to be
> >>>>able to avoid random atomic entrypoints,
> >>>Which is exactly what I criticized from the very first
beginning. Good to
> >>>know that I'm not the only one thinking that this isn't
such a good idea.
> >>I guess I've lost context a bit, but which atomic entry point
are we
> >>talking about? Afaics the only one that's mandatory is the is
> >>fence->signaled callback to check whether a fence really has
been
> >>signalled. It's used internally by the fence code to avoid
spurious
> >>wakeups. Afaik that should be doable already on any hardware. If
that's
> >>not the case then we can always track the signalled state in
software and
> >>double-check in a worker thread before updating the sw state. And
wrap
> >>this all up into a special fence class if there's more than one
driver
> >>needing this.
> >One thing I've forgotten: The i915 scheduler that's floating
around runs
> >its bottom half from irq context. So I really want to be able to check
> >fence state from irq context and I also want to make it possible
> >(possible! not mandatory) to register callbacks which are run from any
> >context asap after the fence is signalled.
> 
> NAK, that's just the bad design I've talked about. Checking fence
state
> inside the same driver from interrupt context is OK, because it's the
> drivers interrupt that we are talking about here.
> 
> Checking fence status from another drivers interrupt context is what really
> concerns me here, cause your driver doesn't have the slightest idea if
the
> called driver is really capable of checking the fence right now.
I guess my mail hasn't been clear then. If you don't like it we could
add
a bit of glue to insulate the madness and bad design i915 might do from
radeon. That imo doesn't invalidate the overall fence interfaces.

So what about the following:
- fence->enabling_signaling is restricted to be called from process
  context. We don't use any different yet, so would boild down to adding a
  WARN_ON(in_interrupt) or so to fence_enable_sw_signalling.

- Make fence->signaled optional (already the case) and don't implement it
  in readon (i.e. reduce this patch here). Only downside is that radeon
  needs to correctly (i.e. without races or so) call fence_signal. And the
  cross-driver synchronization might be a bit less efficient. Note that
  you can call fence_signal from wherever you want to, so hopefully that
  doesn't restrict your implementation. 

End result: No one calls into radeon from interrupt context, and this is
guaranteed.

Would that be something you can agree to?

Like I've said I think restricting the insanity other people are willing
to live with just because you don't like it isn't right. But it is
certainly right for you to insist on not being forced into any such
design. I think the above would achieve this.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Am 22.07.2014 15:26, schrieb Daniel Vetter:
> On Tue, Jul 22, 2014 at 02:19:57PM +0200, Christian K?nig wrote:
>> Am 22.07.2014 13:57, schrieb Daniel Vetter:
>>> On Tue, Jul 22, 2014 at 01:46:07PM +0200, Daniel Vetter wrote:
>>>> On Tue, Jul 22, 2014 at 10:43:13AM +0200, Christian K?nig
wrote:
>>>>> Am 22.07.2014 06:05, schrieb Dave Airlie:
>>>>>> On 9 July 2014 22:29, Maarten Lankhorst
<maarten.lankhorst at canonical.com> wrote:
>>>>>>> Signed-off-by: Maarten Lankhorst
<maarten.lankhorst at canonical.com>
>>>>>>> ---
>>>>>>>   drivers/gpu/drm/radeon/radeon.h        |   15 +-
>>>>>>>   drivers/gpu/drm/radeon/radeon_device.c |   60
++++++++-
>>>>>>>   drivers/gpu/drm/radeon/radeon_fence.c  |  223
++++++++++++++++++++++++++------
>>>>>>>   3 files changed, 248 insertions(+), 50
deletions(-)
>>>>>>>
>>>>>>  From what I can see this is still suffering from the
problem that we
>>>>>> need to find a proper solution to,
>>>>>>
>>>>>> My summary of the issues after talking to Jerome and
Ben and
>>>>>> re-reading things is:
>>>>>>
>>>>>> We really need to work out a better interface into the
drivers to be
>>>>>> able to avoid random atomic entrypoints,
>>>>> Which is exactly what I criticized from the very first
beginning. Good to
>>>>> know that I'm not the only one thinking that this
isn't such a good idea.
>>>> I guess I've lost context a bit, but which atomic entry
point are we
>>>> talking about? Afaics the only one that's mandatory is the
is
>>>> fence->signaled callback to check whether a fence really has
been
>>>> signalled. It's used internally by the fence code to avoid
spurious
>>>> wakeups. Afaik that should be doable already on any hardware.
If that's
>>>> not the case then we can always track the signalled state in
software and
>>>> double-check in a worker thread before updating the sw state.
And wrap
>>>> this all up into a special fence class if there's more than
one driver
>>>> needing this.
>>> One thing I've forgotten: The i915 scheduler that's
floating around runs
>>> its bottom half from irq context. So I really want to be able to
check
>>> fence state from irq context and I also want to make it possible
>>> (possible! not mandatory) to register callbacks which are run from
any
>>> context asap after the fence is signalled.
>> NAK, that's just the bad design I've talked about. Checking
fence state
>> inside the same driver from interrupt context is OK, because it's
the
>> drivers interrupt that we are talking about here.
>>
>> Checking fence status from another drivers interrupt context is what
really
>> concerns me here, cause your driver doesn't have the slightest idea
if the
>> called driver is really capable of checking the fence right now.
> I guess my mail hasn't been clear then. If you don't like it we
could add
> a bit of glue to insulate the madness and bad design i915 might do from
> radeon. That imo doesn't invalidate the overall fence interfaces.
>
> So what about the following:
> - fence->enabling_signaling is restricted to be called from process
>    context. We don't use any different yet, so would boild down to
adding a
>    WARN_ON(in_interrupt) or so to fence_enable_sw_signalling.
>
> - Make fence->signaled optional (already the case) and don't
implement it
>    in readon (i.e. reduce this patch here). Only downside is that radeon
>    needs to correctly (i.e. without races or so) call fence_signal. And the
>    cross-driver synchronization might be a bit less efficient. Note that
>    you can call fence_signal from wherever you want to, so hopefully that
>    doesn't restrict your implementation.
>
> End result: No one calls into radeon from interrupt context, and this is
> guaranteed.
>
> Would that be something you can agree to?
No, the whole enable_signaling stuff should go away. No callback from 
the driver into the fence code, only the other way around.

fence->signaled as well as fence->wait should become mandatory and only 
called from process context without holding any locks, neither atomic 
nor any mutex/semaphore (rcu might be ok).
> Like I've said I think restricting the insanity other people are
willing
> to live with just because you don't like it isn't right. But it is
> certainly right for you to insist on not being forced into any such
> design. I think the above would achieve this.
I don't think so. If it's just me I would say that I'm just to
cautious
and the idea is still save to apply to the whole kernel.

But since Dave, Jerome and Ben seems to have similar concerns I think we 
need to agree to a minimum and save interface for all drivers.

Christian.

op 22-07-14 15:45, Christian K?nig schreef:
> Am 22.07.2014 15:26, schrieb Daniel Vetter:
>> On Tue, Jul 22, 2014 at 02:19:57PM +0200, Christian K?nig wrote:
>>> Am 22.07.2014 13:57, schrieb Daniel Vetter:
>>>> On Tue, Jul 22, 2014 at 01:46:07PM +0200, Daniel Vetter wrote:
>>>>> On Tue, Jul 22, 2014 at 10:43:13AM +0200, Christian K?nig
wrote:
>>>>>> Am 22.07.2014 06:05, schrieb Dave Airlie:
>>>>>>> On 9 July 2014 22:29, Maarten Lankhorst
<maarten.lankhorst at canonical.com> wrote:
>>>>>>>> Signed-off-by: Maarten Lankhorst
<maarten.lankhorst at canonical.com>
>>>>>>>> ---
>>>>>>>>   drivers/gpu/drm/radeon/radeon.h        |   15
+-
>>>>>>>>   drivers/gpu/drm/radeon/radeon_device.c |   60
++++++++-
>>>>>>>>   drivers/gpu/drm/radeon/radeon_fence.c  |  223
++++++++++++++++++++++++++------
>>>>>>>>   3 files changed, 248 insertions(+), 50
deletions(-)
>>>>>>>>
>>>>>>>  From what I can see this is still suffering from
the problem that we
>>>>>>> need to find a proper solution to,
>>>>>>>
>>>>>>> My summary of the issues after talking to Jerome
and Ben and
>>>>>>> re-reading things is:
>>>>>>>
>>>>>>> We really need to work out a better interface into
the drivers to be
>>>>>>> able to avoid random atomic entrypoints,
>>>>>> Which is exactly what I criticized from the very first
beginning. Good to
>>>>>> know that I'm not the only one thinking that this
isn't such a good idea.
>>>>> I guess I've lost context a bit, but which atomic entry
point are we
>>>>> talking about? Afaics the only one that's mandatory is
the is
>>>>> fence->signaled callback to check whether a fence really
has been
>>>>> signalled. It's used internally by the fence code to
avoid spurious
>>>>> wakeups. Afaik that should be doable already on any
hardware. If that's
>>>>> not the case then we can always track the signalled state
in software and
>>>>> double-check in a worker thread before updating the sw
state. And wrap
>>>>> this all up into a special fence class if there's more
than one driver
>>>>> needing this.
>>>> One thing I've forgotten: The i915 scheduler that's
floating around runs
>>>> its bottom half from irq context. So I really want to be able
to check
>>>> fence state from irq context and I also want to make it
possible
>>>> (possible! not mandatory) to register callbacks which are run
from any
>>>> context asap after the fence is signalled.
>>> NAK, that's just the bad design I've talked about. Checking
fence state
>>> inside the same driver from interrupt context is OK, because
it's the
>>> drivers interrupt that we are talking about here.
>>>
>>> Checking fence status from another drivers interrupt context is
what really
>>> concerns me here, cause your driver doesn't have the slightest
idea if the
>>> called driver is really capable of checking the fence right now.
>> I guess my mail hasn't been clear then. If you don't like it we
could add
>> a bit of glue to insulate the madness and bad design i915 might do from
>> radeon. That imo doesn't invalidate the overall fence interfaces.
>>
>> So what about the following:
>> - fence->enabling_signaling is restricted to be called from process
>>    context. We don't use any different yet, so would boild down to
adding a
>>    WARN_ON(in_interrupt) or so to fence_enable_sw_signalling.
>>
>> - Make fence->signaled optional (already the case) and don't
implement it
>>    in readon (i.e. reduce this patch here). Only downside is that
radeon
>>    needs to correctly (i.e. without races or so) call fence_signal. And
the
>>    cross-driver synchronization might be a bit less efficient. Note
that
>>    you can call fence_signal from wherever you want to, so hopefully
that
>>    doesn't restrict your implementation.
>>
>> End result: No one calls into radeon from interrupt context, and this
is
>> guaranteed.
>>
>> Would that be something you can agree to?
>
> No, the whole enable_signaling stuff should go away. No callback from the
driver into the fence code, only the other way around.
>
> fence->signaled as well as fence->wait should become mandatory and
only called from process context without holding any locks, neither atomic nor
any mutex/semaphore (rcu might be ok).
fence->wait is mandatory, and already requires sleeping.

If .signaled is not implemented there is no guarantee the fence will be
signaled sometime soon, this is also why enable_signaling exists, to
allow the driver to flush. I get it that it doesn't apply to radeon and
nouveau,
but for other drivers that could be necessary, like vmwgfx.

Ironically that is also a part of the ttm fence, except it was called flush
there.

I would also like to note that ttm_bo_wait currently is also a function that
currently uses is_signaled from atomic_context...

For the more complicated locking worries: Lockdep is your friend, use
PROVE_LOCKING and find bugs before they trigger. ;-)
>> Like I've said I think restricting the insanity other people are
willing
>> to live with just because you don't like it isn't right. But it
is
>> certainly right for you to insist on not being forced into any such
>> design. I think the above would achieve this.
>
> I don't think so. If it's just me I would say that I'm just to
cautious and the idea is still save to apply to the whole kernel.
>
> But since Dave, Jerome and Ben seems to have similar concerns I think we
need to agree to a minimum and save interface for all drivers.
>
> Christian.
>

On Tue, Jul 22, 2014 at 3:45 PM, Christian K?nig
<deathsimple at vodafone.de> wrote:
>> Would that be something you can agree to?
>
>
> No, the whole enable_signaling stuff should go away. No callback from the
> driver into the fence code, only the other way around.
>
> fence->signaled as well as fence->wait should become mandatory and
only
> called from process context without holding any locks, neither atomic nor
> any mutex/semaphore (rcu might be ok).
So for the enable_signaling, that's optional already. It's only for
drivers that don't want to keep interrupts enabled all the time. You
can opt out of that easily.

Wrt holding no locks at all while calling into any fence functions,
that's just not going to work out. The point here is to make different
drivers work together and we can rework all the ttm and i915 code to
work locklessly in all cases where they need to wait for someone to
complete rendering. Or at least I don't think that's feasible. So if
you insist that no one might call into radeon code then we simply need
to exclude radeon from participating in any shared fencing. But that's
a bit pointless.
>> Like I've said I think restricting the insanity other people are
willing
>> to live with just because you don't like it isn't right. But it
is
>> certainly right for you to insist on not being forced into any such
>> design. I think the above would achieve this.
>
>
> I don't think so. If it's just me I would say that I'm just to
cautious and
> the idea is still save to apply to the whole kernel.
>
> But since Dave, Jerome and Ben seems to have similar concerns I think we
> need to agree to a minimum and save interface for all drivers.
Well I haven't yet seen a proposal that actually works. From an intel
pov I don't care that much since we don't care about desktop prime, so
if radeon/nouveau don't want to do that, meh. Imo the design as-is is
fairly sound, and as simple as it can get given the requirements. I
haven't heard an  argument convincing me otherwise, so I guess we
won't have prime support on linux that actually works, ever.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch