TZTokenizerState Class Reference

A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader. More...

Inheritance diagram for TZTokenizerState:

TZCommentState TZNumberState TZQuoteState TZSymbolState TZWhitespaceState TZWordState TZCppCommentState TZExpressionNumberState TZMySQLNumberState TZPostgreSQLNumberState TZSQLiteNumberState TZSybaseNumberState TZExpressionQuoteState TZGenericSQLQuoteState TZMySQLQuoteState TZSQLiteQuoteState TZSybaseQuoteState TZExpressionSymbolState TZGenericSQLSymbolState TZInterbaseSymbolState TZMySQLSymbolState TZOracleSymbolState TZPostgreSQLSymbolState TZSQLiteSymbolState TZSybaseSymbolState TZExpressionWordState TZGenericSQLWordState

List of all members.

Public Member Functions

TZToken NextToken (TStream Stream, Char FirstChar, TZTokenizer Tokenizer)


Detailed Description

A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader.

The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the nextToken() method has a tokenizer argument.

Definition at line 139 of file ZTokenizer.pas.


Member Function Documentation

TZToken TZTokenizerState::NextToken ( TStream  Stream,
Char  FirstChar,
TZTokenizer  Tokenizer 
)


The documentation for this class was generated from the following file:

Generated on Wed Dec 30 08:42:58 2009 for zeoslib by  doxygen 1.5.7.1