Course of Raku / Regexes and grammars / Grammars / Tokens and rules

regex, token, and rule

The three keywords build on one another:

• regex — backtracks, like the patterns in / … /
• token — does not backtrack
• rule — like token, but spaces in the pattern are significant

Backtracking means that when a later part of the pattern fails, the engine goes back and tries a shorter match for an earlier part. A regex does this:

my regex r { \w+ 'b' }
say so 'aaab' ~~ / <r> /; # True

\w+ first grabs all of aaab, then has to give back the last b so the literal b can match. A token refuses to give anything back:

my token t { \w+ 'b' }
say so 'aaab' ~~ / <t> /; # False

Here \w+ takes all of aaab, the literal b finds nothing left, and the token simply fails instead of backtracking.

That sounds like a limitation, but for grammars it is exactly what you want: each token should match one clean thing and commit to it. This makes parsing faster and the result predictable. Use token by default; reach for regex only on the rare occasions you truly need backtracking.

Course navigation

← Tokens and rules   |   Quiz — Backtracking →

💪 Or jump directly to the exercises in this section.