search for: nextarithmet

...ake the body of the for loop inside ScriptParserBase::tokenize and add a > helper which does that on the fly and is called by consume/next/etc. > Instead of an index into a token vector, just keep a `const char *` pointer > that we advance. > > Once that is done, we can easily add a `nextArithmeticToken` or something > like that which just lexes with different rules. I like that idea. I first thought of always having '*' as a token, but then space has to be a token, which is an incredible pain. I then thought of having a "setLexMode" method, but the lex mode can alway...

...and add >> a >> > helper which does that on the fly and is called by consume/next/etc. >> > Instead of an index into a token vector, just keep a `const char *` >> pointer >> > that we advance. >> > >> > Once that is done, we can easily add a `nextArithmeticToken` or >> something >> > like that which just lexes with different rules. >> >> I like that idea. I first thought of always having '*' as a token, but >> then space has to be a token, which is an incredible pain. >> >> I then thought of havin...

On Tue, Jan 24, 2017 at 11:29 AM, Rui Ueyama <ruiu at google.com> wrote: > Well, maybe, we should just change the Linux kernel instead of tweaking > our tokenizer too hard. > This is silly. Writing a simple and maintainable lexer is not hard (look e.g. at https://reviews.llvm.org/D10817). There are some complicated context-sensitive cases in linker scripts that break our approach

...nside ScriptParserBase::tokenize and add a > > helper which does that on the fly and is called by consume/next/etc. > > Instead of an index into a token vector, just keep a `const char *` > pointer > > that we advance. > > > > Once that is done, we can easily add a `nextArithmeticToken` or something > > like that which just lexes with different rules. > > I like that idea. I first thought of always having '*' as a token, but > then space has to be a token, which is an incredible pain. > > I then thought of having a "setLexMode" method...