com.atilika.kuromoji

Class TokenBase

java.lang.Object

com.atilika.kuromoji.TokenBase

Direct Known Subclasses:

Token

public abstract class TokenBase
extends java.lang.Object

Abstract token class with features shared by all tokens produced by all tokenizers

Constructor Summary

Constructors

Constructor and Description
TokenBase(int wordId, java.lang.String surface, ViterbiNode.Type type, int position, Dictionary dictionary)

Method Summary

Modifier and Type	Method and Description
java.lang.String	getAllFeatures() Gets all features for this token as a comma-separated String
java.lang.String[]	getAllFeaturesArray() Gets all features for this token as a String array
protected java.lang.String	getFeature(int feature) Gets a numbered feature for this token
int	getPosition() Gets the position/start index where this token is found in the input text
java.lang.String	getSurface() Gets the surface form of this token (表層形)
boolean	isKnown() Predicate indicating whether this token is known (contained in the standard dictionary)
boolean	isUser() Predicate indicating whether this token is included is from the user dictionary
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TokenBase

public TokenBase(int wordId,
                 java.lang.String surface,
                 ViterbiNode.Type type,
                 int position,
                 Dictionary dictionary)

Method Detail

getSurface

public java.lang.String getSurface()

Gets the surface form of this token (表層形)

Returns:

surface form, not null

isKnown

public boolean isKnown()

Predicate indicating whether this token is known (contained in the standard dictionary)

Returns:

true if the token is known, otherwise false

isUser

public boolean isUser()

Predicate indicating whether this token is included is from the user dictionary

If a token is contained both in the user dictionary and standard dictionary, this method will return true

Returns:

true if this token is in user dictionary. false if not.

getPosition

public int getPosition()

Gets the position/start index where this token is found in the input text

Returns:

token position

getAllFeatures

public java.lang.String getAllFeatures()

Gets all features for this token as a comma-separated String

Returns:

token features, not null

getAllFeaturesArray

public java.lang.String[] getAllFeaturesArray()

Gets all features for this token as a String array

Returns:

token feature array, not null

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

getFeature

protected java.lang.String getFeature(int feature)

Gets a numbered feature for this token

Parameters:

feature - feature number

Returns:

token feature, not null