com.atilika.kuromoji

Class TokenizerBase

java.lang.Object

com.atilika.kuromoji.TokenizerBase

Direct Known Subclasses:

Tokenizer

public abstract class TokenizerBase
extends java.lang.Object

TokenizerBase main class

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TokenizerBase.Builder Abstract Builder shared by all tokenizers
static class	TokenizerBase.Mode

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.EnumMap<ViterbiNode.Type,Dictionary>	dictionaryMap
protected TokenFactory	tokenFactory

Constructor Summary

Constructors

Constructor and Description
TokenizerBase()

Method Summary

Modifier and Type	Method and Description
protected void	configure(TokenizerBase.Builder builder)
protected <T extends TokenBase> java.util.List<T>	createTokenList(java.lang.String text) Tokenizes the provided text and returns a list of tokens with various feature information
void	debugLattice(java.io.OutputStream outputStream, java.lang.String text) Writes the Viterbi lattice for the provided text to an output stream
void	debugTokenize(java.io.OutputStream outputStream, java.lang.String text) Tokenizes the provided text and outputs the corresponding Viterbi lattice and the Viterbi path to the provided output stream
java.util.List<? extends TokenBase>	tokenize(java.lang.String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenFactory

protected TokenFactory tokenFactory

dictionaryMap

protected java.util.EnumMap<ViterbiNode.Type,Dictionary> dictionaryMap

Constructor Detail

TokenizerBase

public TokenizerBase()

Method Detail

configure

protected void configure(TokenizerBase.Builder builder)

tokenize

public java.util.List<? extends TokenBase> tokenize(java.lang.String text)

createTokenList

protected <T extends TokenBase> java.util.List<T> createTokenList(java.lang.String text)

Tokenizes the provided text and returns a list of tokens with various feature information

This method is thread safe

Type Parameters:

T - token type

Parameters:

text - text to tokenize

Returns:

list of Token, not null

debugTokenize

public void debugTokenize(java.io.OutputStream outputStream,
                          java.lang.String text)
                   throws java.io.IOException

Tokenizes the provided text and outputs the corresponding Viterbi lattice and the Viterbi path to the provided output stream

The output is written in DOT format.

This method is not thread safe

Parameters:

outputStream - output stream to write to

text - text to tokenize

Throws:

java.io.IOException - if an error occurs when writing the lattice and path

debugLattice

public void debugLattice(java.io.OutputStream outputStream,
                         java.lang.String text)
                  throws java.io.IOException

Writes the Viterbi lattice for the provided text to an output stream

The output is written in DOT format.

This method is not thread safe

Parameters:

outputStream - output stream to write to

text - text to create lattice for

Throws:

java.io.IOException - if an error occurs when writing the lattice