search for: setlexmod

...nce that is done, we can easily add a `nextArithmeticToken` or something > like that which just lexes with different rules. I like that idea. I first thought of always having '*' as a token, but then space has to be a token, which is an incredible pain. I then thought of having a "setLexMode" method, but the lex mode can always be implicit from where we are in the parser. The parser should always know if it should call next or nextArithmetic. And I agree we should probably implement this. Even if it is not common, it looks pretty silly to not be able to handle 2*5. Cheers, Rafa...

...r >> something >> > like that which just lexes with different rules. >> >> I like that idea. I first thought of always having '*' as a token, but >> then space has to be a token, which is an incredible pain. >> >> I then thought of having a "setLexMode" method, but the lex mode can >> always be implicit from where we are in the parser. The parser should >> always know if it should call next or nextArithmetic. >> >> And I agree we should probably implement this. Even if it is not common, >> it looks pretty silly...

On Tue, Jan 24, 2017 at 11:29 AM, Rui Ueyama <ruiu at google.com> wrote: > Well, maybe, we should just change the Linux kernel instead of tweaking > our tokenizer too hard. > This is silly. Writing a simple and maintainable lexer is not hard (look e.g. at https://reviews.llvm.org/D10817). There are some complicated context-sensitive cases in linker scripts that break our approach

...ily add a `nextArithmeticToken` or something > > like that which just lexes with different rules. > > I like that idea. I first thought of always having '*' as a token, but > then space has to be a token, which is an incredible pain. > > I then thought of having a "setLexMode" method, but the lex mode can > always be implicit from where we are in the parser. The parser should > always know if it should call next or nextArithmetic. > > And I agree we should probably implement this. Even if it is not common, > it looks pretty silly to not be able to h...