Public Member Functions | |
TZToken | NextToken (TStream Stream, Char FirstChar, TZTokenizer Tokenizer) |
SetWordChars (Char FromChar, Char ToChar, Boolean Enable) | |
TZWordState () | |
Private Attributes | |
Boolean | FWordChars [255] |
Like other states, a tokenizer transfers the job of reading to this state, depending on an initial character. Thus, the tokenizer decides which characters may begin a word, and this state determines which characters may appear as a second or later character in a word. These are typically different sets of characters; in particular, it is typical for digits to appear as parts of a word, but not as the initial character of a word.
By default, the following characters may appear in a word. The method setWordChars()
allows customizing this. <blockquote>
From To 'a', 'z' 'A', 'Z' '0', '9'
as well as: minus sign, underscore, and apostrophe.</blockquote>
Definition at line 517 of file ZTokenizer.pas.
TZWordState::TZWordState | ( | ) |
TZToken TZWordState::NextToken | ( | TStream | Stream, | |
Char | FirstChar, | |||
TZTokenizer | Tokenizer | |||
) |
Reimplemented from TZTokenizerState.
Reimplemented in TZExpressionWordState, and TZGenericSQLWordState.
TZWordState::SetWordChars | ( | Char | FromChar, | |
Char | ToChar, | |||
Boolean | Enable | |||
) |
Boolean TZWordState::FWordChars[255] [private] |
[0..255]
Definition at line 522 of file ZTokenizer.pas.