Tokenizer API

TokenizerBuilder

Lindera::TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

Lindera::TokenizerBuilder.new

Creates a new builder with default configuration.

require 'lindera'

builder = Lindera::TokenizerBuilder.new

Lindera::TokenizerBuilder.from_file(file_path)

Loads configuration from a JSON file and returns a new builder. This is a class method, not chained off an existing instance.

builder = Lindera::TokenizerBuilder.from_file('config.json')

Configuration Methods

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode('normal')

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary('embedded://ipadic')

# Use an external dictionary
builder.set_dictionary('/path/to/dictionary')

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary('/path/to/user_dictionary')

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(true)

set_space_penalty(value)

Sets the left-space penalty for Korean: a candidate that starts right after whitespace and whose part-of-speech tag normally attaches to the preceding word (particles, endings, ...) gets a cost added. value has the same meaning as segmenter.space_penalty in a configuration file:

• nil -- The default: the rules the dictionary ships in its metadata.json, if any (ko-dic ships mecab-ko-dic's rules; the other bundled dictionaries ship none). Use it to undo an earlier call.
• false -- Turns the penalty off.
• true -- Requires the dictionary's rules; build raises RuntimeError for a dictionary that ships none.
• A hash with string keys, { 'rules' => [{ 'pos' => [...], 'cost' => n }, ...] } -- Applies these rules instead. A candidate whose first part-of-speech tag is listed in pos gets cost added; the first matching rule wins.

A value that is not one of these forms, such as 1, 'false' or { 'rules' => 'JKS' }, raises RuntimeError. A value that cannot be converted raises TypeError: a Symbol (also as a hash key or an element, so write %w[JKS] rather than %i[JKS]), a non-finite Float (NaN or Infinity), an Integer outside the 64-bit range, or an object other than nil, true, false, an Integer, a Float, a String, an Array or a Hash.

# Turn the penalty off
builder.set_space_penalty(false)

# Explicit rules
builder.set_space_penalty({
  'rules' => [
    { 'pos' => %w[EC EF EP ETM ETN VCP XSA XSN XSV], 'cost' => 3000 },
    { 'pos' => %w[JC JKB JKC JKG JKO JKQ JKS JKV JX], 'cost' => 6000 }
  ]
})

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, as mecab-ko does. For example, 서울 시 에서 now reads 시 as the noun NNG rather than the ending EP. Call builder.set_space_penalty(false) to turn it off. A tokenizer created directly with Lindera::Tokenizer.new(dictionary, ...) always uses the default, so use TokenizerBuilder to turn it off. Only a ko-dic built by Lindera 6.1.0 or later ships the rules; with a ko-dic from the v6.0.0 release the penalty stays off and true fails until the dictionary is rebuilt. See Segmenter for details.

append_character_filter(kind, args)

Appends a character filter to the preprocessing pipeline. The args parameter is a hash with string keys.

builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })

append_token_filter(kind, args)

Appends a token filter to the postprocessing pipeline. The args parameter is a hash with string keys, or nil if the filter requires no arguments.

builder.append_token_filter('lowercase', nil)

The arguments of set_space_penalty, append_character_filter and append_token_filter may nest arrays and hashes at most 128 levels deep. A deeper value, or one that contains itself, raises ArgumentError.

Build

build

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build

Tokenizer

Lindera::Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Lindera::Tokenizer.new(dictionary, mode = nil, user_dictionary = nil, space_penalty: nil)

Creates a tokenizer directly from a loaded dictionary. mode defaults to 'normal', and user_dictionary is optional.

The space_penalty: keyword takes the same values as TokenizerBuilder#set_space_penalty. nil (the default) uses the left-space penalty rules the dictionary ships, so with ko-dic the penalty is on; pass false to turn it off. An invalid setting raises the same errors as set_space_penalty and build.

require 'lindera'

dictionary = Lindera.load_dictionary('embedded://ipadic')
tokenizer = Lindera::Tokenizer.new(dictionary)

# ko-dic without the left-space penalty
ko_tokenizer = Lindera::Tokenizer.new(Lindera.load_dictionary('embedded://ko-dic'), space_penalty: false)

With a user dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

tokens = tokenizer.tokenize('形態素解析')

Parameters:

Returns: Array<Token>

tokenize_surfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no Token objects are created and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map(&:surface).

surfaces = tokenizer.tokenize_surfaces('形態素解析')
# ["形態素", "解析"]

Parameters:

Returns: Array<String>

tokenize_nbest(text, n, unique, cost_threshold)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)
results.each do |tokens, cost|
  puts "#{cost}: #{tokens.map(&:surface).inspect}"
end

Parameters:

Returns: Array<Array(Array<Token>, Integer)>

Mode

Lindera::Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder#set_mode and Tokenizer.new currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Lindera::Mode.new(mode_str)

Creates a Mode. The argument is required, but may be nil. Accepts "normal" / "Normal" (used when mode_str is nil) or "decompose" / "Decompose"; any other value raises ArgumentError.

require 'lindera'

mode = Lindera::Mode.new('normal')
mode = Lindera::Mode.new('decompose')
mode = Lindera::Mode.new(nil)  # defaults to "normal"

Mode Methods

mode = Lindera::Mode.new('decompose')
mode.to_s        # "decompose"
mode.normal?      # false
mode.decompose?   # true

Penalty

Lindera::Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Lindera::Penalty.new(kanji_penalty_length_threshold, kanji_penalty_length_penalty, other_penalty_length_threshold, other_penalty_length_penalty)

All four positional arguments are required, but each may be nil to fall back to its default (shown below).

require 'lindera'

penalty = Lindera::Penalty.new(2, 3000, 7, 1700)
penalty = Lindera::Penalty.new(nil, nil, nil, nil)  # uses all defaults

Penalty Properties

All properties are read-only (there are no setter methods):

penalty = Lindera::Penalty.new(nil, nil, nil, nil)
penalty.kanji_penalty_length_threshold  # 2

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode and Tokenizer.new only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Additionally, the predicate method unknown? returns true if the word is not in the dictionary:

token.unknown?  # => false

Token Methods

get_detail(index)

Returns the detail string at the specified index, or nil if the index is out of range.

token = tokenizer.tokenize('東京')[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: String or nil

to_h / to_hash

Returns the token as a plain Hash keyed by Symbols, keeping each field's natural Ruby type, so it serializes without a custom encoder. to_h and to_hash are the same method.

require 'json'

data = tokenizer.tokenize('東京')[0].to_h
# {surface: "東京", byte_start: 0, byte_end: 6, position: 0,
#  word_id: 12345, is_unknown: false, details: [...]}

JSON.generate(tokens.map(&:to_h))

Returns: Hash with the keys :surface, :byte_start, :byte_end, :position, :word_id, :is_unknown, and :details.

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Schema

Lindera::Schema holds an ordered list of field names and provides lookups between field name and index. It is used by Metadata#dictionary_schema and Metadata#user_dictionary_schema (see Dictionary Management).

Creating a Schema

Lindera::Schema.new(fields)

Creates a schema from an array of field names.

require 'lindera'

schema = Lindera::Schema.new(%w[
  surface
  left_context_id
  right_context_id
  cost
  major_pos
  reading
])

Lindera::Schema.create_default

Returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Lindera::Schema.create_default

Schema Methods

schema = Lindera::Schema.create_default

schema.field_count                  # 13
schema.get_field_index('cost')      # 3
schema.get_field_name(0)            # "surface"
schema.get_custom_fields            # ["major_pos", "pos_detail_1", ..., "pronunciation"]

field = schema.get_field_by_name('surface')
puts "#{field.index} #{field.name} #{field.field_type}"  # 0 surface surface

schema.validate_record([
  '東京', '1288', '1288', '100',
  '名詞', '固有名詞', '地域', '一般', '*', '*',
  '東京', 'トウキョウ', 'トーキョー'
])

FieldDefinition

Lindera::FieldDefinition describes a single field within a Schema. Instances are only obtained from Schema#get_field_by_name -- there is no public constructor (Lindera::FieldDefinition.new raises TypeError).

FieldDefinition Properties

schema = Lindera::Schema.create_default
field = schema.get_field_by_name('surface')

field.index         # 0
field.name          # "surface"
field.field_type    # #<Lindera::FieldType: surface>
field.description    # nil (the default schema does not set descriptions)

FieldType

Lindera::FieldType enumerates the category of a single field. Like FieldDefinition, instances are only obtained from a Schema (via FieldDefinition#field_type) -- there is no public constructor.

to_s (and inspect) return one of:

• "surface" -- surface form (word text)
• "left_context_id" -- left context ID
• "right_context_id" -- right context ID
• "cost" -- word cost
• "custom" -- any other, dictionary-specific field

field = Lindera::Schema.create_default.get_field_by_name('surface')
field.field_type.to_s  # "surface"