gate.creole.annic.apache.lucene.analysis.standard

Class StandardTokenizerTokenManager

java.lang.Object

gate.creole.annic.apache.lucene.analysis.standard.StandardTokenizerTokenManager

All Implemented Interfaces:

StandardTokenizerConstants

public class StandardTokenizerTokenManager
extends Object
implements StandardTokenizerConstants

Field Summary

Fields

Modifier and Type	Field and Description
protected char	curChar
PrintStream	debugStream
protected CharStream	input_stream
static String[]	jjstrLiteralImages
static String[]	lexStateNames

Fields inherited from interface gate.creole.annic.apache.lucene.analysis.standard.StandardTokenizerConstants

ACRONYM, ALPHA, ALPHANUM, APOSTROPHE, CJK, COMPANY, DEFAULT, DIGIT, EMAIL, EOF, HAS_DIGIT, HOST, LETTER, NOISE, NUM, P, tokenImage

Constructor Summary

Constructors

Constructor and Description
StandardTokenizerTokenManager(CharStream stream)
StandardTokenizerTokenManager(CharStream stream, int lexState)

Method Summary

Modifier and Type	Method and Description
Token	getNextToken()
protected Token	jjFillToken()
void	ReInit(CharStream stream)
void	ReInit(CharStream stream, int lexState)
void	setDebugStream(PrintStream ds)
void	SwitchTo(int lexState)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

debugStream

public PrintStream debugStream

jjstrLiteralImages

public static final String[] jjstrLiteralImages

lexStateNames

public static final String[] lexStateNames

input_stream

protected CharStream input_stream

curChar

protected char curChar

Constructor Detail

StandardTokenizerTokenManager

public StandardTokenizerTokenManager(CharStream stream)

StandardTokenizerTokenManager

public StandardTokenizerTokenManager(CharStream stream,
                                     int lexState)

Method Detail

setDebugStream

public void setDebugStream(PrintStream ds)

ReInit

public void ReInit(CharStream stream)

ReInit

public void ReInit(CharStream stream,
                   int lexState)

SwitchTo

public void SwitchTo(int lexState)

jjFillToken

protected Token jjFillToken()

getNextToken

public Token getNextToken()