I spent some time this week trying to add support for input transformations to the server. I tried a few different approaches. The patches I've attached are what's currently working best for me. In this approach I allow every window to have it's own root window coordinate space. A window's root window coordinate space is affected by all it's ancestors. Composite clients can set a triangular mesh that maps root window coordinates from a composite window to one of its redirected child-windows. Event coordinates are generated in the event windows root coordinate space. Simple cases where the same root coordinate space is used for all windows, like when zooming or rotating the desktop works just fine. More complex cases where each redirected window has a different root coordinate space like the scale plugin in compiz, which transforms and tiles all windows so that they don't overlap, also work surprisingly well but there are some obvious issues. Most of them seem to be things that need to be fixed on the client side, though. E.g. there's no way to know which coordinate space to use for override-redirect windows like tool-tips and menus. The compositing manager can make a decent guess from the window location but without any hint telling the compositing manager which top-level window it's related to, it's hard. Anyhow, I'm not sure my current approach is the best way to allow interaction with transformed windows but it seems to work pretty well with what we want to do in compiz. I have a local branch of compiz with some input transformation hacks, I'll clean it up and push it out in the next couple of days. It seems to me like we need to experiment a bit before we can decide what's the best solution. My current patches are at least something to start with and I look forward to discussing it more next week at xdevconf. - David -------------- next part -------------- A non-text attachment was scrubbed... Name: compositeproto-input-transform-1.patch Type: text/x-patch Size: 1337 bytes Desc: not available Url : http://lists.freedesktop.org/archives/compiz/attachments/20070201/15dbfab9/compositeproto-input-transform-1.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: libXcomposite-input-transform-1.patch Type: text/x-patch Size: 3622 bytes Desc: not available Url : http://lists.freedesktop.org/archives/compiz/attachments/20070201/15dbfab9/libXcomposite-input-transform-1.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: xserver-input-transform-1.patch Type: text/x-patch Size: 17476 bytes Desc: not available Url : http://lists.freedesktop.org/archives/compiz/attachments/20070201/15dbfab9/xserver-input-transform-1.bin
On Sun, 2007-02-04 at 03:59 +1100, Daniel Stone wrote:> On Thu, Feb 01, 2007 at 06:26:13PM -0500, David Reveman wrote: > > I spent some time this week trying to add support for input > > transformations to the server. I tried a few different approaches. The > > patches I've attached are what's currently working best for me. > > Hi David, > This looks largely fine to me: my only comments are that adding your > request in the middle of the request numbering breaks the wire protocol, > and that you need to account for Device* events as well, including > making whatever modifications are necessary to the DeviceValuator > events.Yes, the request numbering was sort of a hack as I didn't know what we're going to do with those last two composite requests that as far as I know never got implemented on client or server side. Are we keeping those or was adding them to protocol spec a mistake? I'll look at the Device* events next. - David
On Sun, 2007-02-04 at 12:13 -0800, Keith Packard wrote:> On Thu, 2007-02-01 at 18:26 -0500, David Reveman wrote: > > I spent some time this week trying to add support for input > > transformations to the server. I tried a few different approaches. The > > patches I've attached are what's currently working best for me. > > Thanks for getting started on this; I believe this is the largest > remaining problem with a composited desktop. > > > In this approach I allow every window to have it's own root window > > coordinate space. A window's root window coordinate space is affected by > > all it's ancestors. Composite clients can set a triangular mesh that > > maps root window coordinates from a composite window to one of its > > redirected child-windows. Event coordinates are generated in the event > > windows root coordinate space. > > I was thinking we'd want a quad mesh instead of a tri mesh so we could > represent projective transforms, but otherwise, this stuff looks like > it's moving in the right direction. Note that Deron Johnson is working > in this space from an entirely different direction (for looking glass); > I asked him to try and construct an interface layer within the server > that would allow both the polygonal decomposition mechansim and their > own java adventures.Why do we need a quad mesh to represent projective transforms? As far as I could understand all we would get out of using a quad mesh instead of a tri mesh would be that we could do some fancier interpolation. But I though that most OpenGL implementations were decomposing all GL primitives to triangles anyhow and it seemed most useful to have the input match the output. We could have an additional request for setting a quad mesh if that's desirable of course. Am I missing something here?> > > The compositing manager can make a decent guess > > from the window location but without any hint telling the compositing > > manager which top-level window it's related to, it's hard. > > Let's work out what the problems are and start experimenting with > changing toolkits to solve this correctly. Obviously having heuristics > which work for legacy apps will be necessary, but I don't want to rely > on guess work when applications can provide definitive answers.Exactly. - David
On Thu, Feb 01, 2007 at 06:26:13PM -0500, David Reveman wrote:> I spent some time this week trying to add support for input > transformations to the server. I tried a few different approaches. The > patches I've attached are what's currently working best for me.Hi David, This looks largely fine to me: my only comments are that adding your request in the middle of the request numbering breaks the wire protocol, and that you need to account for Device* events as well, including making whatever modifications are necessary to the DeviceValuator events. Other than that, looks good; thanks for knocking this one up. Cheers, Daniel -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://lists.freedesktop.org/archives/compiz/attachments/20070204/e0a75060/attachment.pgp
On Thu, 2007-02-01 at 18:26 -0500, David Reveman wrote:> I spent some time this week trying to add support for input > transformations to the server. I tried a few different approaches. The > patches I've attached are what's currently working best for me.Thanks for getting started on this; I believe this is the largest remaining problem with a composited desktop.> In this approach I allow every window to have it's own root window > coordinate space. A window's root window coordinate space is affected by > all it's ancestors. Composite clients can set a triangular mesh that > maps root window coordinates from a composite window to one of its > redirected child-windows. Event coordinates are generated in the event > windows root coordinate space.I was thinking we'd want a quad mesh instead of a tri mesh so we could represent projective transforms, but otherwise, this stuff looks like it's moving in the right direction. Note that Deron Johnson is working in this space from an entirely different direction (for looking glass); I asked him to try and construct an interface layer within the server that would allow both the polygonal decomposition mechansim and their own java adventures.> The compositing manager can make a decent guess > from the window location but without any hint telling the compositing > manager which top-level window it's related to, it's hard.Let's work out what the problems are and start experimenting with changing toolkits to solve this correctly. Obviously having heuristics which work for legacy apps will be necessary, but I don't want to rely on guess work when applications can provide definitive answers. -- keith.packard@intel.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://lists.freedesktop.org/archives/compiz/attachments/20070204/3e5c30dc/attachment.pgp
On Mon, 2007-02-05 at 11:11 -0800, Deron Johnson wrote:> It's hard for me to evaluate this without some higher level context. > What sort of window transformations are you aiming to support > 3D perspective affine transformations? What sort of 3D objects will the > windows be mapped onto? There are a variety of possibilities: > a quad of infinite thinness, a flat slab (rectangular parallelipiped) or > any arbitrarily shaped 3D object. And, do you want to permit interaction > with transformed windows or use transformation only for transitional > effects?I'd like us to be able to support an arbitrarily shaped 3D object even though I don't have any good use cases for that yet. The most important use cases right now are scaling, translating and duplicating windows but we'll definitely use this for more complex transformations soon.> If you want the former you will need to perform mouse-over picking with > input pointer events. If you merely want the latter then it simplifies the > problem considerably. If you can give me a brief summary of what > effects you are trying to achieve I can be in a better position to > help evaluate this proposal. Thanks.We've been doing the later in compiz for quite a while now. You can check out some of the videos here: http://www.go-compiz.org/index.php?title=Video if you want to get up to date on the kind of window transformations that compiz is currently doing. However, I'm not just interested in solving the current cases but providing a solution that will work for years to come. - David
On Mon, 2007-02-05 at 23:56 +0100, Xavier Bestel wrote:> Le lundi 05 f?vrier 2007 ? 16:10 -0500, David Reveman a ?crit : > > On Mon, 2007-02-05 at 11:11 -0800, Deron Johnson wrote: > > > It's hard for me to evaluate this without some higher level context. > > > What sort of window transformations are you aiming to support > > > 3D perspective affine transformations? What sort of 3D objects will the > > > windows be mapped onto? There are a variety of possibilities: > > > a quad of infinite thinness, a flat slab (rectangular parallelipiped) or > > > any arbitrarily shaped 3D object. And, do you want to permit interaction > > > with transformed windows or use transformation only for transitional > > > effects? > > > > I'd like us to be able to support an arbitrarily shaped 3D object even > > though I don't have any good use cases for that yet. The most important > > use cases right now are scaling, translating and duplicating windows but > > we'll definitely use this for more complex transformations soon. > > Then take into account "non-contiguous windows" (imagine a compiz/beryl > plugin which "explodes" a window into many pieces).My current implementation allows that. - David
On Mon, 2007-02-05 at 15:22 -0800, Deron Johnson wrote:> Xavier Bestel wrote: > > >Le lundi 05 f?vrier 2007 ? 16:10 -0500, David Reveman a ?crit : > > > > > >>On Mon, 2007-02-05 at 11:11 -0800, Deron Johnson wrote: > >> > >> > >>>It's hard for me to evaluate this without some higher level context. > >>>What sort of window transformations are you aiming to support > >>>3D perspective affine transformations? What sort of 3D objects will the > >>>windows be mapped onto? There are a variety of possibilities: > >>>a quad of infinite thinness, a flat slab (rectangular parallelipiped) or > >>>any arbitrarily shaped 3D object. And, do you want to permit interaction > >>>with transformed windows or use transformation only for transitional > >>>effects? > >>> > >>> > >>I'd like us to be able to support an arbitrarily shaped 3D object even > >>though I don't have any good use cases for that yet. The most important > >>use cases right now are scaling, translating and duplicating windows but > >>we'll definitely use this for more complex transformations soon. > >> > >> > > > >Then take into account "non-contiguous windows" (imagine a compiz/beryl > >plugin which "explodes" a window into many pieces). > > > > Xav > > > > > > > > > If you want complete generality and full access to 3D effects then I > would recommend > that you route the X events through an external picker which has the > scene graph > in its address space (like LG does).>From the talks keithp has given on the subject I've understood thatimplementing this properly is very hard, if not impossible. Is that not the case anymore?> But if you really want to have the > scene graph inside > the X server I would recommend keeping it very simple and to just > provide a single > contiguous tri-mesh for a window. This would cover 99% of the forseeable > use cases > and you could always extend it later on.Non-contiguous windows are important so I wouldn't want an implementation that didn't support that. It didn't add any complexity to support it in my implementation but I can't be 100% sure of that yet as it's not fully complete though. The only problem I see with a tri-mesh approach is that you'll have to increase the resolution of the mesh to have it match the output in some cases. - David
Xavier Bestel wrote:> On Mon, 2007-02-05 at 18:06 -0800, Keith Packard wrote: >> On Mon, 2007-02-05 at 20:04 -0500, David Reveman wrote: >>> Non-contiguous windows are important so I wouldn't want an >>> implementation that didn't support that. It didn't add any complexity to >>> support it in my implementation but I can't be 100% sure of that yet as >>> it's not fully complete though. >> Yeah, I can't imagine that will be all that complicated. How about the >> ability to map from multiple representations of the same window? > >>From the look of it Mandriva's X server has that too.Not sure what you are saying here as this whole thread, while very interesting to read, is completely over my head. I will say that in Mandriva, our X server is just the stock 1.2 release albeit with a couple of patches to make it behave better with compiz/beryl etc. (that is for cooker, for 2007.0, we are using 1.1.1) Col.
On 2/6/07, Dan Nicholson <dbn.lists@gmail.com> wrote:> On 2/6/07, Keith Packard <keithp@keithp.com> wrote: > > On Tue, 2007-02-06 at 14:12 +0100, Xavier Bestel wrote: > > > > > >From the look of it Mandriva's X server has that too. > > > > Yeah, I haven't heard anything about this and had to go looking on the > > net. Anyone know what this looks like? > > Here's a little more info and some videos: > > http://www.mandriva.com/projects/metisse > http://www.mandriva.com/projects/metisse/download >Here's more: http://dev.beryl-project.org/~cyberorg/uncategorized/13/metisse-on-opensuse/ I tried to build packages, but somehow they are segfaulting here, local build works though, may be someone could give them a try, you need nucleo, metisse, and metisse-fvwm to run: http://software.opensuse.org/download/home:/cyberorg/openSUSE_10.2/i586/ Cheers -J
On 2/6/07, Deron Johnson <Deron.Johnson@sun.com> wrote:> David Reveman wrote: > > >>From the talks keithp has given on the subject I've understood that > >implementing this properly is very hard, if not impossible. Is that not > >the case anymore? > > In December I came up with an idea which I discussed with Keith which > may finally solve the problem of > correctly routing events between the X server and an external "picker" > client which owns the scene graph. At that time > he was advocating for the idea of a simple in-server picker. (By > "picking" I mean using 3D geometry to map transform > the event). So together we came up with an idea which I call "plug-in > pickers."Hi Deron, I understand that the pluggable picker is a very flexible approach, but what does it actually acheive that you can't do with a tri-mesh? What kind of geomerty or picking methods do you have in mind that can not be describe as a mesh of triangles? There has to be a number of really good use cases that can't be expressed as a tri-mesh to warrant this level of complexity. The only idea that I can think of is that a pluggable picker can convert the coordinates corresponding to an exact model of the scene graph. For example if a window is wrapped around a perfect sphere, the corresponding picker can use the exact formula for mapping the screen coordinates into window coordinates. But the key here is that since you have to decompose your objects into triangles to render them anyway, this extra precision doesn't buy you anything. Even if the input remapping is "better" in some sense, it deviates from the on-screen geometry and is effectively less precise. That's all I can think of, and it doesn't really justify the big design you sketched out above. I'm sure you have a number of more realistic use cases since you've been working with this for a while, and I'm curious as to what they are. thanks, Kristian
On Mon, 2007-02-05 at 18:06 -0800, Keith Packard wrote:> On Mon, 2007-02-05 at 20:04 -0500, David Reveman wrote: > > > From the talks keithp has given on the subject I've understood that > > implementing this properly is very hard, if not impossible. Is that not > > the case anymore? > > The hard part was making it asynchronous and routing it through the X > protocol itself. I did a small fraction of that implementation before > abandoning it. Creating a synchronous back-channel would work, although > I have concerns about performance in that case.Thanks for the explanation, I remember now. Deron, are you using a synchronous back-channel for LG then? I would also be very concerned about the performance in that case, how is it working for you? I guess it might make sense to support both a tri-mesh and an external picker approach. Most of the dix changes to support either approach should be the same.> > > Non-contiguous windows are important so I wouldn't want an > > implementation that didn't support that. It didn't add any complexity to > > support it in my implementation but I can't be 100% sure of that yet as > > it's not fully complete though. > > Yeah, I can't imagine that will be all that complicated. How about the > ability to map from multiple representations of the same window?Yes, that works fine too and I haven't seen any potential issues with it yet. - David
On Tue, 2007-02-06 at 11:36 -0800, Deron Johnson wrote:> David Reveman wrote: > > I'm having trouble understanding this terminology. > > >In this approach I allow every window to have it's own root window > >coordinate space. > > > In the X server, "root window coordinate space" means something very > specific. > It means the coordinate space of the WindowTable[pScreen->myNum] window, > or, the root window. Saying that each window has its own root window > coordinate > space is going to get confusing. Is there some other term you can use?Each window got its own coordinate space and the tri-mesh for a window maps root coordinates from the parent window. I called it "root coordinate space" because it's root coordinates we're mapping but I understand that that must have been confusing.> > And, is the coordinate space a 2D or 3D frame of reference?2D.> > > A window's root window coordinate space is affected by > >all it's ancestors. > > > How do these ancestors affect the coordinate space? Are their transforms > concatenated?Yes.> > > Composite clients can set a triangular mesh that > >maps root window coordinates from a composite window to one of its > >redirected child-windows. > > > Specifically how is the triangular mesh used to map the coordinates? > What is the algorithm?It's just pairs of triangles, one triangle in parent space paired with a triangle in child space. Basic uv-mapping is used to transform a coordinate from parent -> child or child -> parent. Implementation doesn't care of triangles in the mesh overlap or if holes exist in the mesh. Looking at the code in my patch might also help understand what I'm trying to explain. - David
On Wed, 2007-02-07 at 00:01 -0800, Keith Packard wrote:> On Wed, 2007-02-07 at 00:25 -0500, David Reveman wrote: > > > It's just pairs of triangles, one triangle in parent space paired with a > > triangle in child space. Basic uv-mapping is used to transform a > > coordinate from parent -> child or child -> parent. > > How are you doing picking? Using the original stacking order? Or is > there a tag on each triangle in the mapping which indicates whether that > space is mapped to visible window pixels?Using the original stacking order. Just making a triangle map all coordinates outside the border clip of the window will give you the same result as such a tag. - David
On Thu, 2007-02-01 at 18:26 -0500, David Reveman wrote:> It seems to me like we need to experiment a bit before we can decide > what's the best solution. My current patches are at least something to > start with and I look forward to discussing it more next week at > xdevconf.The compositor decides what goes where on the screen. As a result, it always has the information needed to perform input transformation. Locating input transformation within the server has to involve the transfer of that information from the compositor to the server. Whatever protocol you establish, it is likely to stifle the development of better solutions in the future and will prove a long term hassle to all involved. Furthermore, transfering the essential details of a complex and rapidly changing scene graph might easily prove expensive. User inputs are simple and infrequent. Their transfer costs are low and will remain low. The argument against placing input transformation within the compositor is founded on the assumption that it introduces an unnecessary round trip into the process: input device event -> server -> compositor -> server -> client. versus input device event -> server -> client However, some input device events might be directed at the compositor itself or at a non-X client of the compositor. In these cases the routes become identical and the server becomes nothing more than a hinderance to input transformation: input device event -> server -> compositor (-> non X client) Furthermore, the compositor will almost always reside on the same host as the input device. This provides a better solution to the round trip problem - establish a direct link from the input device to the compositor. This mirrors the direct rendering approach to cutting out the X server. The longest path then becomes: input device event -> compositor -> server -> client. whilst the shorter paths become input device event -> compositor (-> non X client) The X server changes required to implement compositor based input transformations are relatively simple. A partially working model can be implemented entirely within the compositor with an unpatched server: 1) capture X input events by selecting them on a child of the overlay. 2) pick the target client window and transform to its texel coordinates 3) send the transformed event to the target window using XSendEvent. This works with sufficiently simple X clients. However, there remain two obstacles to this approach: 1) Client pointer grabs are unavailable. 2) Clients typically ignore synthetic events. Both of these obstacles could probably be removed reasonably easily within the server. It just needs a means for the compositor to hide its role in transforming the pointer from the X clients. Felix
On Wed, 2007-02-28 at 19:04 +0200, Daniel Stone wrote:> On Wed, Feb 28, 2007 at 11:07:15AM +0000, Felix Bellaby wrote: > > The argument against placing input transformation within the compositor > > is founded on the assumption that it introduces an unnecessary round > > trip into the process: > > > > input device event -> server -> compositor -> server -> client. > > > > versus > > > > input device event -> server -> client > > Yes, that would be bad, and require some kind of SIGIO-style preemption > (possibly a backchannel) for the compositor to hand the events back, > pre-empting rendering.Later in my post, I suggested the possibility of passing the input device events through the compositor before they even get to the server. This would avoid the need to send every input event over a back channel.> But the really problematic part is the semantics. Right now, when an > event's left the server, it's left the server. But we need the ability > to freeze the input queue pending transformations. If you transform a > mouse, then all subsequent relative events need to be dealt with > relative to the transformed co-ordinates, and so on.True, but when the compositor alters the arrangement of the windows on the screen then the basis on which you have to transform the pointer coordinates also changes. You can not expect to transform the pointer coordinates within the server independently of what is going on in the compositor and get the right results. The server and compositor have to agree on how each and every event is transformed before either one can hope to process the event. What we need is the least bad solution to a horrible coordination problem.> What exactly do you do with grabs? > > It's this part that's quite complex to deal with. Anyone that has a > roughly-working implementation that works with all kinds of grabs (e.g. > popup menus) is a king. :)Let us assume that the compositor receives all events directly from the input device, transforms them and sends the results onto the server, whilst simultaneously placing a copy of the untransformed event into an internal queue. When it recieves an event from the server then it would flush input events occuring before that event from its input queue. I think that the server would normally be able to pass the transformed coordinates onto the targeted clients immediately without any worries. However, there would be occasions when the basis for transforming the input events would need to be altered following the receipt of a request by the server. Grabs would be one example, window maps would be another. When these requests were received then the server would flush any input events from its input queue, send an event relaying the request to the compositor and wait for a response. The compositor would then flush all input events occuring before the relayed request from its input queue, alter the scene graph to reflect the event, retransform all of the events in its queue based on the new graph and resend the events to the server (keeping copies as before). The server would recommence processing immediately after recieving the first of the newly transformed input events from the compositor. I think that this might work, but it would mean delegating the input device event processing to the compositor and suspending the server whenever it received a request that would alter the basis for transforming the input events. I can not see an alternative to suspending input processing in the server while the consequences of a request that alters the basis for input processing is calculated. Nor can I see how the server can be expected to calculate the consequences of such a request independently from the compositor. Therefore, I do not think that this protocol would impose any costs over and above the those that are unavoidable. I am not sure at present which requests would need to be relayed to the compositor so that it could change the input transformation basis, but this will need to be resolved before the true costs of any form of input transformation can be properly assessed. Felix
tor 2007-02-01 klockan 18:26 -0500 skrev David Reveman:> I spent some time this week trying to add support for input > transformations to the server. I tried a few different approaches. The > patches I've attached are what's currently working best for me. > > In this approach I allow every window to have it's own root window > coordinate space. A window's root window coordinate space is affected by > all it's ancestors. Composite clients can set a triangular mesh that > maps root window coordinates from a composite window to one of its > redirected child-windows. Event coordinates are generated in the event > windows root coordinate space.Reading extension documentation are not that easy. When I read about the composite extension it says nothing about events. I assume this means that when you redirect a window the pixels are off-screen the event handling in the server still works as if the redirected windows was on screen with the same place and size? Using the composite extension I could get some of the simple things that was possible in NeWS. I could scale the window to half size (in NeWS you applied a transformation matrix on the window). This would mean that the window now is much smaller on screen. If event handling still thinks the window is full size, mouse events will go to the window event when mouse is not over the window (as seen on screen). I do not know if the above input transformation fully fixes this. It is not just that each window can have its own coordinate space (which could be 3D), it will due to its transformation matrix also have a different size/shape - that is, it is a different area on screen that events will hit than the standard case. I assume you want to have the server do the event distribution, then the server need to handle not just coordinate transformation, it also need to handle the effects a window's transformation matrix have on its positition, shape and size, on screen. Hope the above gives some ideas on how to handle events. Dan