Course of Raku / Regexes and grammars / Grammars / Tokens and rules / Exercises / Regex versus token

Solution: Regex versus token

Here is a possible solution to the task.

Code

my regex r { \d+ '5' }
my token t { \d+ '5' }

say so '12345' ~~ / <r> /;
say so '12345' ~~ / <t> /;

🦋 You can find the source code in the file token-instead.raku.

Output

True
False

Comments

In the regex, \d+ first grabs all of 12345, then backtracks — giving back the final 5 so the literal 5 can match. So the regex succeeds.
The token refuses to give anything back: \d+ keeps all of 12345, the literal 5 finds nothing left, and the token fails. This non-backtracking behaviour is exactly why token is the right default inside grammars.

← Regex versus token | A spaced rule →