search for: 2af5c409

Our tokenizer recognize [A-Za-z0-9_.$/\\~=+[]*?\-:!<>]+ as a token. gold uses more complex rules to tokenize. I don't think we need that much complex rules, but there seems to be room to improve our tokenizer. In particular, I believe we can parse the Linux's linker script by changing the tokenizer rules as follows.

...if that a file name we are parsing or a math expression. May be worth trying to handle this on higher level, during evaluation of expressions ? George. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170124/2af5c409/attachment.html>