Course of Raku / Regexes and grammars / Grammars / Tokens and rules

Significant space in rules

A rule is a token with one extra feature: whitespace in the pattern is significant, exactly as if the :s adverb were switched on. This is what you want whenever the text you parse has spaces between its parts.

Compare the two. With a token, the spaces in the pattern mean nothing, so no whitespace is allowed in the input:

grammar WithToken {
    token TOP    { <first> <second> }
    token first  { \w+ }
    token second { \w+ }
}

say WithToken.parse('foo bar').defined; # False

With a rule, each space in the pattern matches the whitespace in the input:

grammar WithRule {
    rule TOP     { <first> <second> }
    token first  { \w+ }
    token second { \w+ }
}

say WithRule.parse('foo bar').defined; # True

A common pattern is to use rule for the higher-level structure — where the parts are separated by spaces — and token for the small pieces like names and numbers, which contain no spaces. That keeps your grammar both correct and easy to read.

Course navigation

← Quiz — Backtracking   |   Quiz — Tokens and rules →

💪 Or jump directly to the exercises in this section.