com.atilika.kuromoji.ipadic

Class Tokenizer

java.lang.Object

com.atilika.kuromoji.TokenizerBase

com.atilika.kuromoji.ipadic.Tokenizer

public class Tokenizer
extends TokenizerBase

A tokenizer based on the IPADIC dictionary

See Token for details on the morphological features produced by this tokenizer

The following code example demonstrates how to use the Kuromoji tokenizer:

 package com.atilika.kuromoji.example;
 import com.atilika.kuromoji.ipadic.Token;
 import com.atilika.kuromoji.ipadic.Tokenizer;
 import java.util.List;

 public class KuromojiExample {
     public static void main(String[] args) {
         Tokenizer tokenizer = new Tokenizer() ;
         List<Token> tokens = tokenizer.tokenize("お寿司が食べたい。");
         for (Token token : tokens) {
             System.out.println(token.getSurface() + "\t" + token.getAllFeatures());
         }
     }
 }
 
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Tokenizer.Builder Builder class for creating a customized tokenizer instance

Nested classes/interfaces inherited from class com.atilika.kuromoji.TokenizerBase

TokenizerBase.Mode

Field Summary

Fields inherited from class com.atilika.kuromoji.TokenizerBase

dictionaryMap, tokenFactory

Constructor Summary

Constructors

Constructor and Description
Tokenizer() Construct a default tokenizer

Method Summary

Modifier and Type	Method and Description
java.util.List<Token>	tokenize(java.lang.String text) Tokenizes the provided text and returns a list of tokens with various feature information

Methods inherited from class com.atilika.kuromoji.TokenizerBase

configure, createTokenList, debugLattice, debugTokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer()

Construct a default tokenizer

Method Detail

tokenize

public java.util.List<Token> tokenize(java.lang.String text)

Tokenizes the provided text and returns a list of tokens with various feature information

This method is thread safe

Overrides:

tokenize in class TokenizerBase

Parameters:

text - text to tokenize

Returns:

list of Token, not null