Linus Torvalds
2022-Mar-01 00:41 UTC
[Nouveau] [PATCH 2/6] treewide: remove using list iterator after loop body as a ptr
On Mon, Feb 28, 2022 at 1:47 PM Jakob Koschel <jakobkoschel at gmail.com> wrote:> > The goal of this is to get compiler warnings right? This would indeed be great.Yes, so I don't mind having a one-time patch that has been gathered using some automated checker tool, but I don't think that works from a long-term maintenance perspective. So if we have the basic rule being "don't use the loop iterator after the loop has finished, because it can cause all kinds of subtle issues", then in _addition_ to fixing the existing code paths that have this issue, I really would want to (a) get a compiler warning for future cases and (b) make it not actually _work_ for future cases. Because otherwise it will just happen again.> Changing the list_for_each_entry() macro first will break all of those cases > (e.g. the ones using 'list_entry_is_head()).So I have no problems with breaking cases that we basically already have a patch for due to your automated tool. There were certainly more than a handful, but it didn't look _too_ bad to just make the rule be "don't use the iterator after the loop". Of course, that's just based on that patch of yours. Maybe there are a ton of other cases that your patch didn't change, because they didn't match your trigger case, so I may just be overly optimistic here. But basically to _me_, the important part is that the end result is maintainable longer-term. I'm more than happy to have a one-time patch to fix a lot of dubious cases if we can then have clean rules going forward.> I assumed it is better to fix those cases first and then have a simple > coccinelle script changing the macro + moving the iterator into the scope > of the macro.So that had been another plan of mine, until I actually looked at changing the macro. In the one case I looked at, it was ugly beyond belief. It turns out that just syntactically, it's really nice to give the type of the iterator from outside the way we do now. Yeah, it may be a bit odd, and maybe it's partly because I'm so used to the "list_for_each_list_entry()" syntax, but moving the type into the loop construct really made it nasty - either one very complex line, or having to split it over two lines which was even worse. Maybe the place I looked at just happened to have a long typename, but it's basically always going to be a struct, so it's never a _simple_ type. And it just looked very odd adn unnatural to have the type as one of the "arguments" to that list_for_each_entry() macro. So yes, initially my idea had been to just move the iterator entirely inside the macro. But specifying the type got so ugly that I think that typeof (pos) pos trick inside the macro really ends up giving us the best of all worlds: (a) let's us keep the existing syntax and code for all the nice cases that did everything inside the loop anyway (b) gives us a nice warning for any normal use-after-loop case (unless you explicitly initialized it like that sgx_mmu_notifier_release() function did for no good reason (c) also guarantees that even if you don't get a warning, non-converted (or newly written) bad code won't actually _work_ so you end up getting the new rules without any ambiguity or mistaken> With this you are no longer able to set the 'outer' pos within the list > iterator loop body or am I missing something?Correct. Any assignment inside the loop will be entirely just to the local loop case. So any "break;" out of the loop will have to set another variable - like your updated patch did.> I fail to see how this will make most of the changes in this > patch obsolete (if that was the intention).I hope my explanation above clarifies my thinking: I do not dislike your patch, and in fact your patch is indeed required to make the new semantics work. What I disliked was always the maintainability of your patch - making the rules be something that isn't actually visible in the source code, and letting the old semantics still work as well as they ever did, and having to basically run some verification pass to find bad users. (I also disliked your original patch that mixed up the "CPU speculation type safety" with the actual non-speculative problems, but that was another issue). Linus
Jakub Kicinski
2022-Mar-01 06:32 UTC
[Nouveau] [PATCH 2/6] treewide: remove using list iterator after loop body as a ptr
On Mon, 28 Feb 2022 16:41:04 -0800 Linus Torvalds wrote:> So yes, initially my idea had been to just move the iterator entirely > inside the macro. But specifying the type got so ugly that I think > that > > typeof (pos) pos > > trick inside the macro really ends up giving us the best of all worlds: > > (a) let's us keep the existing syntax and code for all the nice cases > that did everything inside the loop anyway > > (b) gives us a nice warning for any normal use-after-loop case > (unless you explicitly initialized it like that > sgx_mmu_notifier_release() function did for no good reason > > (c) also guarantees that even if you don't get a warning, > non-converted (or newly written) bad code won't actually _work_ > > so you end up getting the new rules without any ambiguity or mistakenI presume the goal is that we can do this without changing existing code? Otherwise actually moving the iterator into the loop body would be an option, by creating a different hidden variable: #define list_iter(head) \ for (struct list head *_l = (head)->next; _l != (head); _l = _l->next) #define list_iter_entry(var, member) \ list_entry(_l, typeof(*var), member) list_iter(&p->a_head) { struct entry *e = list_iter_entry(e, a_member); /* use e->... */ } Or we can slide into soft insanity and exploit one of Kees'es tricks to encode the type of the entries "next to" the head: #define LIST_HEAD_MEM(name, type) \ union { \ struct list_head name; \ type *name ## _entry; \ } struct entry { struct list_head a_member; }; struct parent { LIST_HEAD_MEM(a_head, struct entry); }; #define list_for_each_magic(pos, head, member) \ for (typeof(**(head ## _entry)) *pos = list_first_entry(head, typeof(**(head ## _entry)), member); \ &pos->member != (head); \ pos = list_next_entry(pos, member)) list_for_each_magic(e, &p->a_head, a_member) { /* use e->... */ } I'll show myself out...
Jakob Koschel
2022-Mar-01 11:28 UTC
[Nouveau] [PATCH 2/6] treewide: remove using list iterator after loop body as a ptr
> On 1. Mar 2022, at 01:41, Linus Torvalds <torvalds at linux-foundation.org> wrote: > > On Mon, Feb 28, 2022 at 1:47 PM Jakob Koschel <jakobkoschel at gmail.com> wrote: >> >> The goal of this is to get compiler warnings right? This would indeed be great. > > Yes, so I don't mind having a one-time patch that has been gathered > using some automated checker tool, but I don't think that works from a > long-term maintenance perspective. > > So if we have the basic rule being "don't use the loop iterator after > the loop has finished, because it can cause all kinds of subtle > issues", then in _addition_ to fixing the existing code paths that > have this issue, I really would want to (a) get a compiler warning for > future cases and (b) make it not actually _work_ for future cases. > > Because otherwise it will just happen again. > >> Changing the list_for_each_entry() macro first will break all of those cases >> (e.g. the ones using 'list_entry_is_head()). > > So I have no problems with breaking cases that we basically already > have a patch for due to your automated tool. There were certainly > more than a handful, but it didn't look _too_ bad to just make the > rule be "don't use the iterator after the loop". > > Of course, that's just based on that patch of yours. Maybe there are a > ton of other cases that your patch didn't change, because they didn't > match your trigger case, so I may just be overly optimistic here.Based on the coccinelle script there are ~480 cases that need fixing in total. I'll now finish all of them and then split them by submodules as Greg suggested and repost a patch set per submodule. Sounds good?> > But basically to _me_, the important part is that the end result is > maintainable longer-term. I'm more than happy to have a one-time patch > to fix a lot of dubious cases if we can then have clean rules going > forward. > >> I assumed it is better to fix those cases first and then have a simple >> coccinelle script changing the macro + moving the iterator into the scope >> of the macro. > > So that had been another plan of mine, until I actually looked at > changing the macro. In the one case I looked at, it was ugly beyond > belief. > > It turns out that just syntactically, it's really nice to give the > type of the iterator from outside the way we do now. Yeah, it may be a > bit odd, and maybe it's partly because I'm so used to the > "list_for_each_list_entry()" syntax, but moving the type into the loop > construct really made it nasty - either one very complex line, or > having to split it over two lines which was even worse. > > Maybe the place I looked at just happened to have a long typename, but > it's basically always going to be a struct, so it's never a _simple_ > type. And it just looked very odd adn unnatural to have the type as > one of the "arguments" to that list_for_each_entry() macro. > > So yes, initially my idea had been to just move the iterator entirely > inside the macro. But specifying the type got so ugly that I think > that > > typeof (pos) pos > > trick inside the macro really ends up giving us the best of all worlds: > > (a) let's us keep the existing syntax and code for all the nice cases > that did everything inside the loop anyway > > (b) gives us a nice warning for any normal use-after-loop case > (unless you explicitly initialized it like that > sgx_mmu_notifier_release() function did for no good reason > > (c) also guarantees that even if you don't get a warning, > non-converted (or newly written) bad code won't actually _work_ > > so you end up getting the new rules without any ambiguity or mistaken > >> With this you are no longer able to set the 'outer' pos within the list >> iterator loop body or am I missing something? > > Correct. Any assignment inside the loop will be entirely just to the > local loop case. So any "break;" out of the loop will have to set > another variable - like your updated patch did. > >> I fail to see how this will make most of the changes in this >> patch obsolete (if that was the intention). > > I hope my explanation above clarifies my thinking: I do not dislike > your patch, and in fact your patch is indeed required to make the new > semantics work.ok it's all clear now, thanks for clarifying. I've defined all the 'tmp' iterator variables uninitialized so applying your patch on top of that later will just give the nice compiler warning if they are used past the loop body.> > What I disliked was always the maintainability of your patch - making > the rules be something that isn't actually visible in the source code, > and letting the old semantics still work as well as they ever did, and > having to basically run some verification pass to find bad users.Since this patch is not a complete list of cases that need fixing (30%) I haven't included the actual change of moving the iterator variable into the loop and thought that would be a second step coming after this is merged. With these changes alone, yes you still rely on manual verification passes.> > (I also disliked your original patch that mixed up the "CPU > speculation type safety" with the actual non-speculative problems, but > that was another issue). > > Linus- Jakob
Xiaomeng Tong
2022-Mar-02 09:31 UTC
[Nouveau] [PATCH 2/6] treewide: remove using list iterator after loop body as a ptr
On Mon, 28 Feb 2022 16:41:04 -0800, Linus Torvalds <torvalds at linux-foundation.org> wrote:> > But basically to _me_, the important part is that the end result is > maintainable longer-term.I couldn't agree more. And because of that, I stick with the following approach because it's maintainable longer-term than "type(pos) pos" one: Implements a new macro for each list_for_each_entry* with _inside suffix. #define list_for_each_entry_inside(pos, type, head, member) I have posted a patch series here to demonstrate this approach: https://lore.kernel.org/lkml/20220301075839.4156-3-xiam0nd.tong at gmail.com/ Although we need replace all the use of list_for_each_entry* (15000+) with list_for_each_entry*_inside, the work can be done gradually rather than all at once. We can incrementally replace these callers until all these in the kernel are completely updated with *_inside* one. At that time, we can just remove the implements of origin macros and rename the *_inside* macro back to the origin name just in one single patch. And the "type(pos) pos" approach need teach developers to "not initialize the iterator variable, otherwise the use-after-loop will not be reported by compiler", which is unreasonable and impossible for all developers. And it will mess up the following code logic and no warnning reported by compiler, even without initializing "ext" at the beginning: void foo(struct mem_extent *arg) { struct mem_extent *ext; // used both for iterator and normal ptr ... ext = arg; // this assignment can alse be done in another bar() func ... list_for_each_entry(ext, head, member) { if (found(ext)) break; } ... // use ext after the loop ret = ext; } If the loop hit the break, the last "ret" will be the found ext iterator. However, if the "type(pos) pos" approach applied, the last "ret" will be "arg" which is not the intention of the developers, because the "ext" is two different variables inside and outside the loop. Thus, my idea is *better a finger off than always aching*, let's choose the "list_for_each_entry_inside(pos, type, head, member)" approach.> It turns out that just syntactically, it's really nice to give the > type of the iterator from outside the way we do now. Yeah, it may be a > bit odd, and maybe it's partly because I'm so used to the > "list_for_each_list_entry()" syntax, but moving the type into the loop > construct really made it nasty - either one very complex line, or > having to split it over two lines which was even worse. > > Maybe the place I looked at just happened to have a long typename, but > it's basically always going to be a struct, so it's never a _simple_ > type. And it just looked very odd adn unnatural to have the type as > one of the "arguments" to that list_for_each_entry() macro.we can pass a shorter type name to list_for_each_entry_inside, thus no need to split it over two lines. Actually it is not a big problem. + #define t struct sram_bank_info - list_for_each_entry(pos, head, member) { + list_for_each_entry_inside(pos, t, head, member) { I put the type at the second argument not the first to avoid messing up the pattern match in some coccinelle scripts.> (b) gives us a nice warning for any normal use-after-loop case > (unless you explicitly initialized it like that > sgx_mmu_notifier_release() function did for no good reasonsometimes developers can be confused by the reported warnning: "used without having been initialized", and can not figure out immediately that "oh, now i am using another different variable but with the same name of the loop iterator variable", which has changed the programming habits of developers.> (c) also guarantees that even if you don't get a warning, > non-converted (or newly written) bad code won't actually _work_ > > so you end up getting the new rules without any ambiguity or mistakenIt will lead to a wrong/NULL pointer dereference if the pointer is used anywhere else, depend on which value is used to initialized with. Best regard, -- Xiaomeng Tong