Tokenizer API

TokenizerBuilder

Lindera::TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

Lindera::TokenizerBuilder.new

Creates a new builder with default configuration.

require 'lindera'

builder = Lindera::TokenizerBuilder.new

Lindera::TokenizerBuilder.from_file(file_path)

Loads configuration from a JSON file and returns a new builder. This is a class method, not chained off an existing instance.

builder = Lindera::TokenizerBuilder.from_file('config.json')

Configuration Methods

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode('normal')

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary('embedded://ipadic')

# Use an external dictionary
builder.set_dictionary('/path/to/dictionary')

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary('/path/to/user_dictionary')

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(true)

append_character_filter(kind, args)

Appends a character filter to the preprocessing pipeline. The args parameter is a hash with string keys.

builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })

append_token_filter(kind, args)

Appends a token filter to the postprocessing pipeline. The args parameter is a hash with string keys, or nil if the filter requires no arguments.

builder.append_token_filter('lowercase', nil)

Build

build

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build

Tokenizer

Lindera::Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Lindera::Tokenizer.new(dictionary, mode, user_dictionary)

Creates a tokenizer directly from a loaded dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('embedded://ipadic')
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', nil)

With a user dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

tokens = tokenizer.tokenize('形態素解析')

Parameters:

Returns: Array<Token>

tokenize_nbest(text, n, unique, cost_threshold)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)
results.each do |tokens, cost|
  puts "#{cost}: #{tokens.map(&:surface).inspect}"
end

Parameters:

Returns: Array<Array(Array<Token>, Integer)>

Mode

Lindera::Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder#set_mode and Tokenizer.new currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Lindera::Mode.new(mode_str)

Creates a Mode. The argument is required, but may be nil. Accepts "normal" / "Normal" (used when mode_str is nil) or "decompose" / "Decompose"; any other value raises ArgumentError.

require 'lindera'

mode = Lindera::Mode.new('normal')
mode = Lindera::Mode.new('decompose')
mode = Lindera::Mode.new(nil)  # defaults to "normal"

Mode Methods

mode = Lindera::Mode.new('decompose')
mode.to_s        # "decompose"
mode.normal?      # false
mode.decompose?   # true

Penalty

Lindera::Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Lindera::Penalty.new(kanji_penalty_length_threshold, kanji_penalty_length_penalty, other_penalty_length_threshold, other_penalty_length_penalty)

All four positional arguments are required, but each may be nil to fall back to its default (shown below).

require 'lindera'

penalty = Lindera::Penalty.new(2, 3000, 7, 1700)
penalty = Lindera::Penalty.new(nil, nil, nil, nil)  # uses all defaults

Penalty Properties

All properties are read-only (there are no setter methods):

penalty = Lindera::Penalty.new(nil, nil, nil, nil)
penalty.kanji_penalty_length_threshold  # 2

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode and Tokenizer.new only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Additionally, the predicate method unknown? returns true if the word is not in the dictionary:

token.unknown?  # => false

Token Methods

get_detail(index)

Returns the detail string at the specified index, or nil if the index is out of range.

token = tokenizer.tokenize('東京')[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: String or nil

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Schema

Lindera::Schema holds an ordered list of field names and provides lookups between field name and index. It is used by Metadata#dictionary_schema and Metadata#user_dictionary_schema (see Dictionary Management).

Creating a Schema

Lindera::Schema.new(fields)

Creates a schema from an array of field names.

require 'lindera'

schema = Lindera::Schema.new(%w[
  surface
  left_context_id
  right_context_id
  cost
  major_pos
  reading
])

Lindera::Schema.create_default

Returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Lindera::Schema.create_default

Schema Methods

schema = Lindera::Schema.create_default

schema.field_count                  # 13
schema.get_field_index('cost')      # 3
schema.get_field_name(0)            # "surface"
schema.get_custom_fields            # ["major_pos", "pos_detail_1", ..., "pronunciation"]

field = schema.get_field_by_name('surface')
puts "#{field.index} #{field.name} #{field.field_type}"  # 0 surface surface

schema.validate_record([
  '東京', '1288', '1288', '100',
  '名詞', '固有名詞', '地域', '一般', '*', '*',
  '東京', 'トウキョウ', 'トーキョー'
])

FieldDefinition

Lindera::FieldDefinition describes a single field within a Schema. Instances are only obtained from Schema#get_field_by_name -- there is no public constructor (Lindera::FieldDefinition.new raises TypeError).

FieldDefinition Properties

schema = Lindera::Schema.create_default
field = schema.get_field_by_name('surface')

field.index         # 0
field.name          # "surface"
field.field_type    # #<Lindera::FieldType: surface>
field.description    # nil (the default schema does not set descriptions)

FieldType

Lindera::FieldType enumerates the category of a single field. Like FieldDefinition, instances are only obtained from a Schema (via FieldDefinition#field_type) -- there is no public constructor.

to_s (and inspect) return one of:

• "surface" -- surface form (word text)
• "left_context_id" -- left context ID
• "right_context_id" -- right context ID
• "cost" -- word cost
• "custom" -- any other, dictionary-specific field

field = Lindera::Schema.create_default.get_field_by_name('surface')
field.field_type.to_s  # "surface"