com.atilika.kuromoji

Class TokenizerBase.Builder

java.lang.Object

com.atilika.kuromoji.TokenizerBase.Builder

Direct Known Subclasses:

Tokenizer.Builder

Enclosing class:

TokenizerBase

public abstract static class TokenizerBase.Builder
extends java.lang.Object

Abstract Builder shared by all tokenizers

Field Summary

Fields

Modifier and Type	Field and Description
protected CharacterDefinitions	characterDefinitions
protected ConnectionCosts	connectionCosts
protected DoubleArrayTrie	doubleArrayTrie
protected InsertedDictionary	insertedDictionary
protected TokenizerBase.Mode	mode
protected int	partOfSpeechFeature
protected java.util.List<java.lang.Integer>	penalties
protected int	readingFeature
protected ResourceResolver	resolver
protected boolean	split
protected TokenFactory	tokenFactory
protected TokenInfoDictionary	tokenInfoDictionary
protected int	totalFeatures
protected UnknownDictionary	unknownDictionary
protected UserDictionary	userDictionary

Constructor Summary

Constructors

Constructor and Description
Builder()

Method Summary

Modifier and Type	Method and Description
abstract <T extends TokenizerBase> T	build() Creates a Tokenizer instance defined by this Builder
protected void	loadDictionaries()
TokenizerBase.Builder	userDictionary(java.io.InputStream input) Sets an optional user dictionary as an input stream
TokenizerBase.Builder	userDictionary(java.lang.String filename) Sets an optional user dictionary filename

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

doubleArrayTrie

protected DoubleArrayTrie doubleArrayTrie

connectionCosts

protected ConnectionCosts connectionCosts

tokenInfoDictionary

protected TokenInfoDictionary tokenInfoDictionary

unknownDictionary

protected UnknownDictionary unknownDictionary

characterDefinitions

protected CharacterDefinitions characterDefinitions

insertedDictionary

protected InsertedDictionary insertedDictionary

userDictionary

protected UserDictionary userDictionary

mode

protected TokenizerBase.Mode mode

split

protected boolean split

penalties

protected java.util.List<java.lang.Integer> penalties

totalFeatures

protected int totalFeatures

readingFeature

protected int readingFeature

partOfSpeechFeature

protected int partOfSpeechFeature

resolver

protected ResourceResolver resolver

tokenFactory

protected TokenFactory tokenFactory

Constructor Detail

Builder

public Builder()

Method Detail

loadDictionaries

protected void loadDictionaries()

build

public abstract <T extends TokenizerBase> T build()

Creates a Tokenizer instance defined by this Builder

Type Parameters:

T - token type

Returns:

Tokenizer instance

userDictionary

public TokenizerBase.Builder userDictionary(java.io.InputStream input)
                                     throws java.io.IOException

Sets an optional user dictionary as an input stream

The inpuut stream provided is not closed by this method

Parameters:

input - user dictionary as an input stream

Returns:

this builder

Throws:

java.io.IOException - if an error occurs when reading the user dictionary

userDictionary

public TokenizerBase.Builder userDictionary(java.lang.String filename)
                                     throws java.io.IOException

Sets an optional user dictionary filename

Parameters:

filename - user dictionary filename

Returns:

this builder

Throws:

java.io.IOException - if an error occurs when reading the user dictionary