Public Member Functions | |
Add (const string Value) | |
TZToken | NextToken (TStream Stream, Char FirstChar, TZTokenizer Tokenizer) |
TZSymbolState () | |
~TZSymbolState () | |
Protected Attributes | |
TZSymbolRootNode | Symbols |
Private Attributes | |
TZSymbolRootNode | FSymbols |
For example, when tokenizing the expression (isReady)& (isWilling)
, a typical tokenizer would return 7 tokens, including one for each parenthesis and one for the ampersand. Thus a series of symbols such as )&(
becomes three tokens, while a series of letters such as isReady
becomes a single word token.
Multi-character symbols are an exception to the rule that a symbol is a standalone character. For example, a tokenizer may want less-than-or-equals to tokenize as a single token. This class provides a method for establishing which multi-character symbols an object of this class should treat as single symbols. This allows, for example, "cat <= dog"
to tokenize as three tokens, rather than splitting the less-than and equals symbols into separate tokens.
By default, this state recognizes the following multi- character symbols: !=, :-, <=, >=
Definition at line 430 of file ZTokenizer.pas.
TZSymbolState::TZSymbolState | ( | ) |
TZSymbolState::~TZSymbolState | ( | ) |
TZSymbolState::Add | ( | const string | Value | ) |
TZToken TZSymbolState::NextToken | ( | TStream | Stream, | |
Char | FirstChar, | |||
TZTokenizer | Tokenizer | |||
) |
Reimplemented from TZTokenizerState.
TZSymbolRootNode TZSymbolState::FSymbols [private] |
Definition at line 435 of file ZTokenizer.pas.
TZSymbolRootNode TZSymbolState::Symbols [protected] |