Lindera

A morphological analysis library in Rust. Lindera is forked from kuromoji-rs and aims to provide easy installation and concise APIs for tokenizing text in multiple languages.

Key Features

Tokenization Flow

graph LR
    subgraph Your Application
        T["Text"]
    end
    subgraph Lindera
        CF["Character Filters"]
        SEG["Segmenter\n(Dictionary + Viterbi)"]
        TF["Token Filters"]
    end
    T --> CF --> SEG --> TF --> R["Tokens"]

Document Map

Performance

Lindera tokenizes Japanese text (IPADIC) at roughly 10-20 MB/s single-threaded, depending on hardware and whether the Viterbi lattice is reused across calls. Run make bench to reproduce the benchmark suite (lindera/benches/) on your own machine; see #875 for detailed methodology and comparison numbers.

Quick Example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = tokenizer.tokenize(text)?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Run the example:

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

Output:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

License

Lindera is released under the MIT License.

Architecture

Lindera is organized as a Cargo workspace comprising multiple crates. Each crate has a focused responsibility, from low-level CRF computation to high-level CLI and language bindings.

Crate Dependency Graph

graph TB
    CRF["lindera-crf\n(CRF Engine)"]
    DICT["lindera-dictionary\n(Dictionary Base)"]
    TRAINER["lindera-trainer\n(CRF Training)"]
    IPADIC["lindera-ipadic"]
    UNIDIC["lindera-unidic"]
    SUDACHIDICT["lindera-sudachidict"]
    KODIC["lindera-ko-dic"]
    CCCEDICT["lindera-cc-cedict"]
    JIEBA["lindera-jieba"]
    NEOLOGD["lindera-ipadic-neologd"]
    SEGMENTER["lindera-segmenter\n(Segmenter)"]
    ANALYSIS["lindera-analysis\n(Analysis Chain)"]
    LIB["lindera\n(Facade)"]
    CLI["lindera-cli\n(CLI)"]
    BINDING["lindera-binding"]
    PY["lindera-python"]
    NODEJS["lindera-nodejs"]
    RUBY["lindera-ruby"]
    PHP["lindera-php"]
    WASM["lindera-wasm"]

    CRF --> TRAINER
    DICT --> TRAINER
    TRAINER -.->|"train feature"| SEGMENTER
    DICT --> IPADIC
    DICT --> UNIDIC
    DICT --> SUDACHIDICT
    DICT --> KODIC
    DICT --> CCCEDICT
    DICT --> JIEBA
    DICT --> NEOLOGD
    DICT --> SEGMENTER
    IPADIC --> SEGMENTER
    UNIDIC --> SEGMENTER
    SUDACHIDICT --> SEGMENTER
    KODIC --> SEGMENTER
    CCCEDICT --> SEGMENTER
    JIEBA --> SEGMENTER
    NEOLOGD --> SEGMENTER
    SEGMENTER --> ANALYSIS
    SEGMENTER --> LIB
    ANALYSIS -.->|"analysis feature"| LIB
    LIB --> CLI
    LIB --> BINDING
    BINDING --> PY
    BINDING --> NODEJS
    BINDING --> RUBY
    BINDING --> PHP
    BINDING --> WASM

Crate Overview

Tokenization Pipeline

Lindera processes text through a multi-stage pipeline:

Input Text
  |
  v
Character Filters    -- Normalize characters (e.g., Unicode normalization, mapping)
  |
  v
Segmenter            -- Segment text into tokens using a dictionary and the Viterbi algorithm
  |
  v
Token Filters        -- Transform tokens (e.g., POS filtering, stop words, stemming)
  |
  v
Output Tokens

The Segmenter is the core component. It builds a lattice of candidate tokens from the dictionary, then applies the Viterbi algorithm to find the lowest-cost path, producing the most likely segmentation. Dictionary lookups walk a char-wise double-array trie (built with crawdad) in place over the serialized bytes of dict.trie, rather than deserializing into an owned structure. For repeated tokenization, Segmenter::new_worker() returns a SegmentWorker that reuses lattice and scratch-buffer allocations across calls, with an automatic shrink policy to bound retained memory.

Feature Flags

Learn More

• Getting Started -- Installation and first steps
• Core Concepts -- Dictionaries, tokenization, and filters
• Lindera Library -- The lindera facade and the segmenter API
• Lindera Analysis -- Character filters, token filters, and the Tokenizer
• Lindera Dictionary -- Dictionary loading and building
• Lindera Trainer -- CRF-based dictionary training
• Lindera CRF -- The CRF engine
• Lindera CLI -- Command-line interface
• Development Guide -- Build, test, and contribute

Getting Started

This section will guide you through installing Lindera and running your first morphological analysis.

• Installation -- Add Lindera to your project and configure environment variables
• Quick Start -- Tokenize your first text in just a few lines of code
• Examples -- Explore example programs for common use cases

Installation

Requirements

Rust 1.88 or newer. The workspace declares this as rust-version, so an older toolchain reports it as a cargo error rather than failing to compile.

Adding the Dependency

Put the following in Cargo.toml:

[dependencies]
lindera = "7"

[!NOTE] This single dependency also provides the analysis chain (character filters, token filters, and the Tokenizer) as lindera::analysis; use default-features = false if you only need the segmenter. Upgrading from v6? See Migration v6 to v7 for details.

Dictionary Setup

Lindera requires a pre-built dictionary at runtime. Download a dictionary from GitHub Releases and specify its path when loading:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

#![allow(unused)]
fn main() {
// Cargo.toml: lindera = { version = "7", features = ["embed-ipadic"] }
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

Environment Variables

LINDERA_BUILD_DICTIONARY_CACHE_DIR

The LINDERA_BUILD_DICTIONARY_CACHE_DIR environment variable designates a build-time cache directory for the embedded-dictionary build pipeline. It is read only by the dictionary crates' build scripts and has no effect at runtime.

When set, each build stores two kinds of files under $LINDERA_BUILD_DICTIONARY_CACHE_DIR/<version>-fmt<format>/ (where <version> is the dictionary crate version and <format> is the dictionary format version):

• the downloaded distribution archive (validated with MD5; invalid files are re-downloaded)
• the built binary dictionary that gets embedded into the crate

The format version is part of the path so that a cache written by a build with a different on-disk dictionary layout is a miss rather than a stale hit. A cached directory is also re-checked against its recorded format version before being reused, which covers entries left behind by an interrupted build. Directories from older format versions are not deleted automatically; remove them by hand if the cache grows.

This enables:

• Offline builds: once cached, subsequent builds need no network access
• Faster builds: download and dictionary build are skipped when valid cached files exist
• Reproducible builds: consistent dictionary versions across builds

Usage:

export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache
cargo build --features=embed-ipadic

Notes:

• The directory is managed automatically and is safe to delete; contents are re-downloaded and rebuilt as needed
• Version subdirectories accumulate across upgrades and are not garbage-collected; old ones can be removed freely
• Setting this variable causes dictionary crates to download and build their dictionaries even when no embed-* feature is enabled (useful for pre-populating the cache)

LINDERA_CONFIG_PATH

The LINDERA_CONFIG_PATH environment variable specifies the path to a YAML configuration file for the tokenizer. This allows you to configure tokenizer behavior without modifying Rust code.

export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

See the Configuration section for details on the configuration format.

DOCS_RS

The DOCS_RS environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.

This is primarily used internally by docs.rs and typically doesn't need to be set by users.

LINDERA_WORKDIR

The LINDERA_WORKDIR environment variable is automatically set during the build process by the lindera-dictionary crate. It points to the directory containing the built dictionary data files and is used internally by dictionary crates to locate their data files.

This variable is set automatically and should not be modified by users.

Quick Start

This example covers the basic usage of Lindera.

It will:

• Create a segmenter in normal mode
• Segment the input text
• Output the tokens

This example uses the embed-ipadic feature, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

use std::borrow::Cow;

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = segmenter.segment(Cow::Borrowed(text))?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

The above example can be run as follows:

% cargo run --features embed-ipadic --example=segment

[!TIP] If you prefer not to embed the dictionary into the binary, download a pre-built IPADIC dictionary from GitHub Releases, extract it to a local directory (e.g., /path/to/ipadic), and call load_dictionary("/path/to/ipadic") instead — no embed-ipadic feature needed in that case. See Feature Flags for details.

You can see the result as follows:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

[!NOTE] Character filters, token filters, and the Tokenizer API are available as lindera::analysis (the analysis feature, enabled by default), so lindera = "7" already covers the whole analysis chain — no extra dependency is needed. With default-features = false you get the segmenter alone. See Lindera Analysis for details, and Migration v6 to v7 if you are upgrading from v6.

Examples

Lindera includes several example programs that demonstrate common use cases. The source code is available in the examples directory on GitHub (the tokenize* examples are in lindera-analysis/examples).

The tokenize* examples use the Tokenizer and filter APIs from lindera::analysis. They live in the lindera-analysis crate, hence -p lindera-analysis below.

All examples below are run with the embed-ipadic feature enabled, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

Available Examples

segment

Basic morphological segmentation with the Segmenter API — it needs only the segmenter, so it also works with default-features = false.

cargo run -p lindera --features=embed-ipadic --example=segment

tokenize

Basic tokenization using an external IPADIC dictionary. Segments input text and prints each token with its part-of-speech details.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

tokenize_with_user_dict

Tokenization with a user dictionary. Shows how to supplement the dictionary with custom entries for domain-specific terms.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_user_dict

tokenize_with_filters

Tokenization with character filters and token filters. Demonstrates the text processing pipeline, including Unicode normalization, part-of-speech filtering, and other transformations.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_filters

tokenize_with_config

Tokenization using a YAML configuration file. Shows how to configure the tokenizer declaratively instead of programmatically.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_config

Core Concepts

This section explains the fundamental concepts behind Lindera's morphological analysis system.

• Morphological Analysis - How Lindera segments and analyzes text.
• Dictionaries - Dictionary formats supported by Lindera.
• Tokenization - Tokenization modes and N-Best analysis.
• User Dictionary - Adding custom words with user dictionaries.
• Filters - Pre-processing text and post-processing tokens with character and token filters.

Morphological Analysis

What is morphological analysis?

Morphological analysis is the process of breaking down text into its smallest meaningful units (morphemes) and identifying their grammatical properties. For languages like Japanese, Chinese, and Korean -- where words are not separated by spaces -- morphological analysis is an essential first step for natural language processing tasks such as search indexing, text classification, and machine translation.

How Lindera works

Lindera is a dictionary-based morphological analyzer. It uses a pre-compiled system dictionary containing known words along with their costs, and applies the Viterbi algorithm to find the optimal segmentation of input text.

The analysis process works as follows:

1. Sentence splitting: Before building the lattice, Lindera splits the input text into sentences at delimiter characters (\n, \t, 。, 、) and processes one sentence at a time. If no delimiter appears within roughly 32 KiB of a sentence's start, Lindera forces a sentence boundary there anyway and logs a warning, to bound the lattice's memory and CPU cost. Up to the next \n or \t, each sentence continues from the words that end the previous one, so the result is, with few exceptions, the one a single lattice over that text gives (see Sentence Splitting). The forced cut can affect tokenization at the artificial cut point for pathological delimiter-free input, such as minified text or base64-encoded blobs.
2. Lattice construction: Lindera scans each sentence and looks up all possible words in the dictionary at every position, building a directed acyclic graph (lattice) of candidate segmentations.
3. Cost assignment: Each candidate word has an associated word cost (from the dictionary), and each pair of adjacent words has a connection cost (from the connection cost matrix).
4. Optimal path search: The Viterbi algorithm finds the path through the lattice with the minimum total cost, producing the best segmentation.

Key terminology

Cost-based segmentation

The Viterbi algorithm selects the segmentation path with the minimum total cost. The total cost of a path is the sum of:

• Word costs: Each word in the dictionary has an associated cost. Lower cost means the word is more likely to appear. Common words tend to have lower costs, while rare words have higher costs.
• Connection costs: The cost of connecting two adjacent words, determined by the right context ID of the left word and the left context ID of the right word.

The algorithm computes:

Total cost = sum of word costs + sum of connection costs

By minimizing this total cost, Lindera finds the most natural segmentation of the input text.

Connection cost matrix

The connection cost matrix stores the cost of transitioning from one word to another. It is a two-dimensional matrix indexed by:

• The right context ID of the preceding word
• The left context ID of the following word

These context IDs encode grammatical information about word boundaries. For example, the connection cost between a noun and a particle is typically low (natural sequence), while the connection cost between two verbs in base form might be high (unnatural sequence).

The connection cost matrix is compiled into binary format as part of the dictionary build process and is loaded at runtime for efficient lookup.

Dictionaries

Lindera supports various dictionaries for Japanese, Korean, and Chinese morphological analysis. Each dictionary is provided as a separate crate.

Obtaining Dictionaries

Pre-built dictionaries are available for download from GitHub Releases. Download the dictionary archive for your target language and extract it to a local directory.

#![allow(unused)]
fn main() {
// Load an external dictionary from a local path
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you need a self-contained binary without external dictionary files, you can embed dictionaries using the embed-* feature flags and load them using the embedded:// scheme:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

See each dictionary crate's documentation for format details, build instructions, and usage examples.

Tokenization

Lindera provides multiple tokenization modes and supports N-Best analysis for enumerating alternative segmentation candidates.

Tokenization modes

Normal mode

Normal mode performs standard tokenization based on dictionary entries. Compound words that exist as single entries in the dictionary are kept as-is.

Example -- tokenizing "関西国際空港限定トートバッグ" in Normal mode:

関西国際空港 | 限定 | トートバッグ

The compound noun "関西国際空港" (Kansai International Airport) is preserved as a single token because it exists as one entry in the dictionary.

Decompose mode

Decompose mode further breaks down compound nouns into their constituent parts, even when the compound exists as a dictionary entry.

Example -- tokenizing "関西国際空港限定トートバッグ" in Decompose mode:

関西 | 国際 | 空港 | 限定 | トートバッグ

The compound "関西国際空港" is decomposed into "関西", "国際", and "空港".

Selecting a mode

In Rust, specify the mode when creating a Segmenter:

#![allow(unused)]
fn main() {
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;

// Normal mode
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

// Decompose mode
let segmenter = Segmenter::new(Mode::Decompose(Default::default()), dictionary, None);
}

With the CLI, use the --mode flag:

echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode normal
echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode decompose

[!NOTE] The embedded:// scheme requires lindera-cli to be built with the corresponding embed-* feature (e.g. cargo install lindera-cli --features=embed-ipadic). A default build of lindera-cli does not enable any embed-* feature, and passing --dict embedded://ipadic without it fails with Invalid dictionary scheme: embedded.

N-Best tokenization

N-Best tokenization enumerates the top N tokenization candidates ordered by total path cost (lower cost = better segmentation). This is useful when the best result is ambiguous, or when you want to explore alternative interpretations of the input text.

Algorithm

N-Best tokenization is based on the Forward-DP Backward-A* algorithm, which is compatible with MeCab's N-Best implementation. The forward pass computes optimal costs using dynamic programming, and the backward pass uses A* search to enumerate paths in order of increasing total cost.

As in 1-best tokenization, the input is first split into sentences, and the sentences up to each \n or \t form a segment (see Sentence Splitting). Each segment is searched on its own: within it, the context is carried across 、 and 。, so its paths are the N best paths of one lattice over the segment. Every candidate segments the whole input: it takes one path in each segment, and its cost is the sum of those paths' costs. The candidates are the N cheapest of these combinations.

Parameters

The tokenize_nbest method accepts the following parameters:

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "すもももももももものうち";

    // Get top 3 tokenization results
    let results = tokenizer.tokenize_nbest(text, 3, false, None)?;

    for (rank, (tokens, cost)) in results.iter().enumerate() {
        println!("--- NBEST {} (cost={}) ---", rank + 1, cost);
        for token in tokens {
            let details = token.details().join(",");
            println!("{}\t{}", token.surface.as_ref(), details);
        }
    }

    Ok(())
}

Output:

--- NBEST 1 (cost=21245) ---
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
--- NBEST 2 (cost=24541) ---
...

CLI example

echo "すもももももももものうち" | lindera tokenize --dict embedded://ipadic -N 3

Lattice reuse

For repeated tokenization, you can reuse a Lattice to reduce memory allocations:

#![allow(unused)]
fn main() {
use lindera::dictionary::Lattice;

let mut lattice = Lattice::default();
let results = tokenizer.tokenize_nbest_with_lattice(text, &mut lattice, 3, false, None)?;
}

At the segmenter level, SegmentWorker (created via Segmenter::new_worker) packages this pattern into a session object that owns the lattice and scratch buffers, and additionally bounds retained memory with an automatic shrink policy. See the Segmenter page for details.

User Dictionary

A user dictionary is a supplementary dictionary that allows you to register custom words alongside the system dictionary. This is useful for domain-specific terms, brand names, proper nouns, or any words that are not in the default system dictionary.

CSV format

The simplest user dictionary format is a CSV file with three columns:

<surface>,<part_of_speech>,<reading>

Example CSV content

東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ

The surface (the first column) is used exactly as written: whitespace at its start or end is part of the word, and a surface may consist of whitespace only (IPADIC registers U+3000 this way). Only an empty surface is skipped. The same applies to the lexicon CSV files of a system dictionary built with lindera build.

An entry that ends with whitespace keeps it in its token, as in MeCab, even when whitespace is skipped in the lattice (the default for every bundled dictionary except SudachiDict, see Whitespace Handling). With whitespace skipped, however, an entry that starts with whitespace of the dictionary's SPACE category, or consists of such whitespace only, never matches, because no word starts on skipped whitespace. U+3000 is not in the SPACE category, so IPADIC's U+3000 entry still matches.

Each dictionary type (IPADIC, UniDic, ko-dic, etc.) also supports a detailed CSV format with full control over context IDs, costs, and all feature fields. See the Dictionaries section for the detailed format of each dictionary type.

Rust API example

use std::fs::File;
use std::path::PathBuf;

use lindera::dictionary::{Metadata, load_dictionary, load_user_dictionary};
use lindera::error::LinderaErrorKind;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let user_dict_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("user_dict")
        .join("ipadic_simple_userdic.csv");

    let metadata_file = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../lindera-ipadic")
        .join("metadata.json");
    let metadata: Metadata = serde_json::from_reader(
        File::open(metadata_file)
            .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
            .unwrap(),
    )
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
    .unwrap();

    let dictionary = load_dictionary("embedded://ipadic")?;
    let user_dictionary = load_user_dictionary(user_dict_path.to_str().unwrap(), &metadata)?;
    let segmenter = Segmenter::new(
        Mode::Normal,
        dictionary,
        Some(user_dictionary), // Using the loaded user dictionary
    );

    // Create a tokenizer.
    let tokenizer = Tokenizer::new(segmenter);

    // Tokenize a text.
    let text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です";
    let mut tokens = tokenizer.tokenize(text)?;

    // Print the text and tokens.
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Output:

text:   東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です
token:  東京スカイツリー        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリー,*
token:  の      助詞,連体化,*,*,*,*,の,ノ,ノ
token:  最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
token:  は      助詞,係助詞,*,*,*,*,は,ハ,ワ
token:  とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリーエキ,*
token:  です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス

Note that the base_form field (the 7th CSV/detail field) is * rather than the surface form. The simple three-column user-dictionary schema only supplies surface, part_of_speech, and reading — every other field, including base_form, is filled with the dictionary's metadata.default_field_value (* for IPADIC).

Building a user dictionary with CLI

You can build a user dictionary from CSV to binary format using the CLI:

lindera build --src <source_dir> --dest <dest_dir> --metadata <metadata.json> --user

Binary vs CSV user dictionary

• CSV format: Loaded and parsed at runtime. Convenient for development and small dictionaries.
• Binary format: Pre-compiled for faster loading. Recommended for production use with large user dictionaries.

Both formats can be specified when creating a Segmenter. The binary format skips the CSV parsing step, resulting in faster startup times.

Filters

Lindera's analysis pipeline (the Tokenizer in lindera::analysis, provided by the lindera-analysis crate) has two extension points around the segmenter: character filters, which transform the raw input text before tokenization, and token filters, which transform the list of tokens after tokenization.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character filters

Character filters preprocess the input text before it reaches the segmenter. They're typically used to normalize text so that tokenization is more consistent -- for example, converting full-width characters to half-width, canonicalizing Unicode representations, or expanding Japanese iteration marks into their repeated form. Because a character filter can change the length of the text, Lindera tracks every transformation it makes and corrects each token's byte offsets back to positions in the original, unfiltered text.

Token filters

Token filters postprocess the list of tokens produced by the segmenter. They're typically used to normalize or reduce the token list for search and analysis -- for example, replacing a token with its base (dictionary) form, converting a token's surface text between hiragana and katakana, removing tokens by part-of-speech tag, or removing stop words.

Configuring filters

Every filter -- character or token -- is identified by a kind string and configured with a JSON args object of filter-specific parameters. Filters run in the order they're added, and there are two ways to add them:

• YAML configuration file: list filters under the character_filters and token_filters keys.

  character_filters:
    - kind: unicode_normalize
      args:
        kind: nfkc
  
  token_filters:
    - kind: japanese_base_form
  

• Rust API: append filters to a Tokenizer, in order, with append_character_filter and append_token_filter.

  #![allow(unused)]
  fn main() {
  // Character filters run first, transforming the raw input text;
  // token filters run last, transforming the resulting token list.
  tokenizer
      .append_character_filter(BoxCharacterFilter::from(unicode_normalize_char_filter))
      .append_token_filter(BoxTokenFilter::from(japanese_base_form_filter));
  }

Available filters

Lindera ships 4 character filters and 21 token filters, covering Japanese, Korean, and general-purpose text normalization:

See the Filters reference for what each filter does, its parameters, and runnable YAML and Rust API examples.

Lindera CRF

Lindera CRF is a pure Rust implementation of Conditional Random Fields (CRFs), forked from rucrf. It provides a trainer and an estimator for CRFs with support for lattice structures.

Key Features

• Lattices with variable length edges
• L1, L2, and Elastic Net regularization
• Multi-threaded training
• Zero-copy deserialization with rkyv
• no_std support via --no-default-features --features alloc

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Changes from rucrf

• Serialization backend: Switched from bincode to rkyv for zero-copy deserialization
• Elastic Net regularization: Added Regularization::ElasticNet combining L1 and L2 penalties
• Rust 2024 edition: Updated to Rust 2024 edition
• Dependency updates: Updated argmin, argmin-math, hashbrown, etc.

Architecture

Module Structure

lindera-crf/src/
├── lib.rs                # Public API re-exports
├── feature.rs            # FeatureSet, FeatureProvider
├── lattice.rs            # Edge, Node, Lattice
├── model.rs              # RawModel, MergedModel, Model trait
├── trainer.rs            # Trainer, Regularization enum
├── errors.rs             # Error types
├── forward_backward.rs   # Forward-backward algorithm
├── math.rs               # Mathematical utilities (logsumexp)
├── optimizers.rs         # Declares the optimizers module
├── optimizers/
│   └── lbfgs.rs          # L-BFGS optimization
└── utils.rs              # Utility traits

Key Components

FeatureProvider / FeatureSet

Manage per-label feature sets. Each FeatureSet holds unigram features and left/right bigram features for a given label (feature IDs only, no weights). FeatureProvider maps label IDs to FeatureSet instances. Weights are held separately, on RawModel (weights, unigram_weight_indices, bigram_weight_indices).

Lattice / Edge / Node

Lattice structure with variable-length edges for sequence labeling. Edge represents a candidate span with a label, while Node aggregates edges at a given position. The Lattice is constructed from input data and used by the model to find the best path.

Trainer

Trains a CRF model using L-BFGS optimization with configurable regularization. The trainer accepts labeled lattice examples, computes gradients via the forward-backward algorithm, and iteratively updates model weights.

Regularization

Configurable regularization strategies:

• L1: Sparse models via L1 penalty
• L2: Smooth models via L2 penalty
• ElasticNet: Combines L1 and L2 with a configurable l1_ratio

Model (trait)

Interface for searching the best path through a lattice. Two implementations are provided:

• RawModel: Stores weights in a flat vector indexed by feature ID
• MergedModel: Optimized for inference; merges feature weights into a compact representation serializable with rkyv, using MergedFeatureSet as its per-label element type

MergedFeatureSet holds the pre-summed unigram weight for a label, along with its left_id/right_id bigram connection IDs into MergedModel's bigram weight matrix.

Forward-backward Algorithm

Computes alpha (forward) and beta (backward) values over the lattice. Used during training to calculate expected feature counts and gradients.

Feature Flags

* Not listed directly in default, but transitively enabled by train (which is on by default and enables std, which in turn enables alloc) -- so alloc/std are active in a default build.

API Reference

The API reference is available. Please see following URL:

• lindera-crf

Lindera Dictionary

Lindera Dictionary is the base library for morphological analysis dictionaries. It provides dictionary loading, building, and Viterbi-based segmentation. CRF-based dictionary training lives in the separate lindera-trainer crate. Through the lindera facade its modules are available under lindera::dictionary — lindera::dictionary::core (the dictionary data structures), lindera::dictionary::builder, lindera::dictionary::viterbi, and so on — so a direct lindera-dictionary dependency is rarely needed.

Key Features

• Dictionary loading from filesystem or embedded data
• Dictionary building from MeCab-format CSV source files
• Viterbi algorithm for optimal segmentation
• N-best path generation (Forward-DP Backward-A*)
• Memory-mapped file support

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-dictionary/src/
├── lib.rs               # Public API
├── dictionary.rs        # Dictionary, UserDictionary
├── builder.rs           # DictionaryBuilder
├── loader.rs            # DictionaryLoader trait, FSDictionaryLoader
├── viterbi.rs           # Lattice, Edge, Viterbi segmentation
├── nbest.rs             # NBestGenerator (Forward-DP Backward-A*)
├── mode.rs              # Mode (Normal/Decompose), Penalty
├── error.rs             # LinderaError, LinderaErrorKind
├── assets.rs            # Download and file management
├── macros.rs            # embedded_dictionary! macro shared by dictionary crates
├── dictionary/
│   ├── character_definition.rs    # Character type definitions
│   ├── connection_cost_matrix.rs  # Connection cost matrix
│   ├── context_id_map.rs          # ContextIdMap: connection-cost context-ID remap
│   ├── prefix_dictionary.rs       # Prefix dictionary (char-wise double-array trie)
│   ├── unknown_dictionary.rs      # Unknown word handling
│   ├── metadata.rs                # Dictionary metadata
│   └── schema.rs                  # Schema definitions

Key Components

Dictionary / UserDictionary

Main data structures holding the compiled dictionary data. A Dictionary contains the character definitions, connection cost matrix, prefix dictionary, and unknown word dictionary. The prefix dictionary is a char-wise double-array trie, built with crawdad at build time and walked in place over its serialized bytes at runtime — loading it is an O(1) header check, not a deserialization pass. UserDictionary allows users to add custom vocabulary on top of the system dictionary; unlike the system dictionary, it still uses a daachorse automaton inside its rkyv-archived .bin files.

DictionaryBuilder

Fluent API for building dictionaries from source CSV files. It compiles MeCab-format dictionary sources into the binary format used at runtime. Its four build stages (metadata, unknown dictionary, prefix dictionary, connection cost matrix) run concurrently on scoped threads on non-wasm targets, falling back to a sequential build on wasm (which has no OS threads); the concurrent path has higher peak memory, since all four stages' working sets are held at once.

DictionaryLoader / FSDictionaryLoader

DictionaryLoader is a trait for loading compiled dictionaries. FSDictionaryLoader is the filesystem-based implementation that reads dictionary files from a directory, with optional memory-mapped file support.

Embedded Dictionary Macro

The embedded_dictionary! macro (lindera-dictionary/src/macros.rs, #[macro_export]) generates the boilerplate each dictionary crate needs to bake its compiled dictionary into the binary: a load() function that reads the dictionary components via include_bytes! and a loader struct implementing DictionaryLoader. Every dictionary crate's embedded.rs (lindera-ipadic, lindera-ipadic-neologd, lindera-unidic, lindera-ko-dic, lindera-cc-cedict, lindera-jieba) invokes this macro instead of duplicating the loading logic.

Viterbi (Lattice, Edge)

Builds a lattice of candidate tokens from the input text and finds the optimal segmentation path using the Viterbi algorithm. Each Edge in the lattice represents a candidate token with associated costs (word cost + connection cost).

NBestGenerator

Generates N-best segmentation paths using the Forward-DP Backward-A* algorithm. This enables applications to consider alternative segmentations beyond the single best path.

Mode

Controls tokenization behavior:

• Normal: Standard tokenization using the optimal Viterbi path
• Decompose: Further splits compound nouns based on configurable Penalty thresholds

Dictionary Format Version

A built dictionary directory records the on-disk layout it was written in, as format_version in metadata.json. DictionaryBuilder::build_metadata stamps the value from DICTIONARY_FORMAT_VERSION (lindera-dictionary/src/dictionary/metadata.rs) — the builder is the only thing that can state truthfully what it just wrote, so a value in the source metadata.json is ignored. Loading a dictionary directory checks the recorded version and fails with an actionable message if it does not match.

The check matters because most artifacts have no header of their own: matrix.mtx, dict.vals and dict.words are raw arrays, so a dictionary from an older layout that happens to be a plausible length decodes into garbage rather than failing. A source metadata.json — the hand-written kind checked into each dictionary crate — describes build inputs, carries no format_version, and is never format-checked.

Bump DICTIONARY_FORMAT_VERSION on any change to the bytes of a built artifact. That includes upgrading a dependency whose serialized form is written verbatim (crawdad for dict.trie, rkyv for char_def.bin and unk.bin, daachorse inside user dictionary .bin files), which is easy to overlook because no line of this crate changes. The build cache is keyed on the format version for exactly that reason.

The current format version is 2. Version 2 replaced the system prefix dictionary's byte-wise daachorse Aho-Corasick automaton (dict.da) with a char-wise double-array trie (dict.trie) plus a u32 prefix-sum index (dict.valsidx). Dictionaries at version 1 or earlier are rejected at load time with an actionable error. User dictionary .bin files were not affected by that change — but they are written by daachorse, so a daachorse upgrade does invalidate them, which is exactly why it belongs in the list above.

Context ID Remapping

Dictionary metadata (lindera-dictionary/src/dictionary/metadata.rs) carries a connection_id_mapping: bool flag and an optional context_id_map: Option<ContextIdMap>. When a dictionary crate (e.g. lindera-unidic) enables connection_id_mapping, DictionaryBuilder relabels the connection matrix's left/right context IDs by access frequency at build time, so that frequently-used connection-cost cells cluster together for better cache locality; DictionaryBuilder::with_context_id_freq attaches an optional bundled frequency histogram used to rank the IDs, and the remap itself is computed in lindera-dictionary/src/builder/context_id_remap.rs. ContextIdMap (lindera-dictionary/src/dictionary/context_id_map.rs) holds the resulting left/right permutation and is persisted into the built metadata.json so the same mapping can be reapplied later. Because user dictionaries are always compiled in the original, un-remapped ID space, UserDictionary::remap_context_ids (lindera-dictionary/src/dictionary.rs) uses the persisted ContextIdMap to relabel a user dictionary's context IDs into the same space as the system dictionary it is attached to. The remap is a bijective relabeling, so it does not change tokenization output, only lookup locality.

Training

The CRF-based dictionary training pipeline lives in the separate lindera-trainer crate, which builds on this crate's runtime types. See the training pipeline documentation for details.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-dictionary

Lindera Trainer

Lindera Trainer implements the CRF-based dictionary training pipeline (lindera train), turning an annotated corpus and a seed lexicon into a trained model that can be exported to MeCab-format dictionary source files and compiled into a binary dictionary. It builds on the runtime types of lindera-dictionary and the CRF core of lindera-crf. The crate can be used directly, or through the lindera facade's train feature, which re-exports it as lindera::dictionary::trainer.

Key Features

• CRF-based weight learning via lindera-crf, with L1, L2, and Elastic Net regularization
• MeCab-compatible feature template parsing (feature.def: %F[n], %L[n], %R[n], %w, %u, %l, %r, and their optional ? variants)
• MeCab-compatible 3-section feature rewriting (rewrite.def: unigram / left / right rewrite rules)
• Dictionary-format agnostic: works with any lexicon whose columns follow surface,left_id,right_id,cost,feature... (IPADIC, UniDic, ko-dic, CC-CEDICT, etc.)
• Automatic unknown-word categorization driven by char.def character categories
• Connection cost matrix and MeCab-compatible cost conversion (tocost) derived from learned CRF weights
• Zero-copy rkyv binary model serialization, with legacy JSON fallback on read
• Direct export of Lindera/MeCab dictionary source files (lex.csv, matrix.def, unk.def, char.def, feature.def, rewrite.def, left-id.def, right-id.def)

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-trainer/src/
├── lib.rs                # Trainer: CRF training orchestration; public API re-exports
├── config.rs             # TrainerConfig: parses seed lexicon, char.def, feature.def, rewrite.def
├── corpus.rs              # Corpus, Example, Word: training data representation
├── feature_extractor.rs  # FeatureExtractor: feature template parsing and feature ID management
├── feature_rewriter.rs   # DictionaryRewriter: MeCab-compatible 3-section rewrite
└── model.rs               # Model, SerializableModel: trained model storage, serialization, dictionary output

Key Components

TrainerConfig

Parses the five training input files -- seed lexicon (lex.csv), character definition (char.def), unknown-word definition (unk.def), feature template (feature.def), and rewrite rules (rewrite.def) -- into the configuration consumed by Trainer. TrainerConfig::from_readers (or from_paths, its file-path convenience wrapper) extracts the surface/feature vocabulary from the seed lexicon, builds a FeatureExtractor from the parsed feature templates, builds a DictionaryRewriter from the rewrite rules, and assembles a minimal in-memory lindera_dictionary::dictionary::Dictionary from the character definition. char.def is parsed by the dictionary builder (CharacterDefinitionBuilder::build_from_str), so training sees the same character categories as the built dictionary; the default category of each character, which the %t feature uses, comes from the builder's build_default_category_table. Note: the Dictionary's prefix_dictionary (system lexicon) and unknown_dictionary fields are built as empty/stub placeholders -- the real seed-lexicon vocabulary and unk-category features used during training are held separately in TrainerConfig's own surfaces/features/unk_categories/unk_costs fields, not in this Dictionary object. It also exposes surfaces(), surface_features(), get_features(), a user lexicon accessible via user_lexicon() / add_user_lexicon_entry() / load_user_lexicon_from_content(), and metadata().

Corpus / Example / Word

Represent annotated training data. Word pairs a surface form with its (comma-joined) feature string. Example is one training sentence, built from a Vec<Word> whose surfaces are concatenated to reconstruct the original sentence text. Corpus is a collection of Examples; Corpus::from_reader parses tab-separated surface<TAB>features lines, treating blank lines or literal EOS lines as sentence boundaries.

FeatureExtractor

Parses MeCab-compatible feature templates and manages the mapping from generated feature strings to interned, NonZeroU32 feature IDs. Supported template placeholders are %F[n] / %F?[n] (feature field at index n, the ? form skipped when the value is *), %t (character category), %w (surface form, unigram only), %u / %l / %r (full rewritten unigram/left/right feature string), and %L[n] / %L?[n] / %R[n] / %R?[n] for bigram left/right context fields. extract_unigram_feature_ids[_with_ctx], extract_left_feature_ids[_with_ctx], and extract_right_feature_ids[_with_ctx] apply the parsed templates to a feature vector (plus, for bigram extraction, an optional TemplateContext carrying surface/ufeature/lfeature/rfeature) and return the resulting feature IDs, creating new IDs on first use.

DictionaryRewriter

Implements MeCab's 3-section rewrite.def format: [unigram rewrite], [left rewrite], and [right rewrite] sections, each holding an ordered list of pattern<TAB>replacement rules built into a FeatureRewriter prefix trie (via the internal FeatureRewriterBuilder). DictionaryRewriter::from_reader parses all three sections (falling back to treating an unsectioned file as legacy right-rewrite-only content, for backward compatibility with older Lindera rewrite.def files). rewrite() applies all three rewriters to a feature string and returns (ufeature, lfeature, rfeature), passing a section through unchanged when no rule matches. (An earlier rewrite_cached() memoized per full feature string, but real lexicon features are unique per row, so the cache never hit and was removed — #975.)

Model / SerializableModel

Model is the trained model produced by Trainer::train. It owns the trained lindera_crf::RawModel, the TrainerConfig used to train it, the extracted feature weights and labels, and any user lexicon entries added afterward via read_user_lexicon. It converts learned CRF weights into MeCab-compatible integer costs (tocost(weight, cost_factor) = clamp(round(-weight * cost_factor), i16::MIN, i16::MAX), i.e. clamped to [-32768, 32767]), with the cost factor computed to make full use of the i16 range (calculate_cost_factor). Model can serialize itself (write_model) and export dictionary source files directly.

SerializableModel is the plain-data, rkyv-serializable counterpart returned by Model::read_model when loading a previously trained model back from a reader (as done by the lindera export CLI command). It carries the same trained information as Model -- feature weights, labels, POS info, connection cost matrix, unknown-word categories, preserved char.def/feature.def/rewrite.def content, cost factor, and left/right context ID maps -- as owned data, without a TrainerConfig, and exposes its own writer methods for producing the dictionary export files.

Model public methods

SerializableModel public methods

Available on the value returned by Model::read_model, for re-exporting dictionary source files from a previously trained and serialized model (used by the lindera export CLI command):

ModelMetadata and FeatureSetInfo

ModelMetadata records summary information about a training run: version, regularization (the cost coefficient used), iterations (max iterations configured), feature_count, and label_count. It is embedded in SerializableModel::metadata.

FeatureSetInfo holds the per-label connection information extracted from the merged CRF model after training: left_id, right_id, and weight. SerializableModel::feature_sets stores one entry per label, in the same order as labels.

API Reference

The API reference is available. Please see following URL:

• lindera-trainer

Lindera Library

The lindera crate is the facade of the Rust API: it re-exports the morphological segmenter from lindera-segmenter (lindera::segmenter, lindera::dictionary, lindera::mode, ...) and, with the default analysis feature, the analysis chain from lindera-analysis as lindera::analysis. lindera = "7" is the only dependency you need; default-features = false gives you the segmenter alone. This section covers segmentation, error handling, and API reference.

If you need the Tokenizer, character filters, or token filters (a Lucene-style analysis chain built on top of Segmenter), use lindera::analysis — see the Lindera Analysis chapter, including its Configuration and Filters pages.

• Segmenter - Core segmentation component using the Viterbi algorithm
• Error Handling - Error types and handling patterns
• API Reference - Links to generated API documentation

Segmenter

The Segmenter is the core component that performs morphological analysis. It uses the Viterbi algorithm to find the optimal segmentation of input text based on a dictionary and cost model.

Creating a Segmenter

A Segmenter requires three components:

• Mode - the tokenization strategy (Normal or Decompose)
• Dictionary - a system dictionary for morphological analysis
• UserDictionary (optional) - a supplementary dictionary for custom words

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
}

Tokenization Modes

Mode::Normal

Standard tokenization based on the dictionary entries. Words are segmented faithfully according to what is registered in the dictionary.

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Normal;
}

Mode::Decompose

Decomposes compound nouns into their constituent parts. This mode applies a configurable penalty to long compound words, encouraging the segmenter to split them into shorter components.

For example, with Mode::Normal, the compound word "関西国際空港" in the sentence "関西国際空港限定トートバッグ" remains part of a single token, while with Mode::Decompose, it is split into "関西", "国際", and "空港".

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Decompose(Default::default());
}

Dictionary Loading

Lindera provides the load_dictionary function to load dictionaries from various sources.

Embedded Dictionaries

When built with the appropriate feature flag (e.g., embed-ipadic), dictionaries can be loaded directly from the binary:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;
}

Available embedded dictionary URIs:

• embedded://ipadic - IPADIC (Japanese)
• embedded://ipadic-neologd - IPADIC NEologd (Japanese)
• embedded://unidic - UniDic (Japanese)
• embedded://ko-dic - ko-dic (Korean)
• embedded://cc-cedict - CC-CEDICT (Chinese)
• embedded://jieba - Jieba (Chinese)

External Dictionaries

Pre-built dictionary directories can be loaded from the filesystem:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("/path/to/dictionary")?;
}

Using with Tokenizer

The Segmenter is typically used through the Tokenizer, which adds support for character filters and token filters:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let tokens = tokenizer.tokenize(text)?;

    for mut token in tokens {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Note the mut binding on token: Token::details takes &mut self, so iterating with a plain for token in tokens fails to compile (E0596: cannot borrow as mutable).

Building from Config

Segmenter::from_config builds a Segmenter from a SegmenterConfig (a serde_json::Value), the same configuration format used by Tokenizer/TokenizerBuilder (see Configuration) but scoped to just the segmenter: section:

#![allow(unused)]
fn main() {
use serde_json::json;
use lindera::segmenter::{Segmenter, SegmenterConfig};

let config: SegmenterConfig = json!({
    "mode": "normal",
    "dictionary": "embedded://ipadic",
    "keep_whitespace": false,
    "use_mmap": false
});
let segmenter = Segmenter::from_config(&config)?;
}

Note: use_mmap here is set to false only to illustrate the option explicitly; omitting it entirely gives the same default (true, see below).

Memory-Mapped Loading

For a filesystem-based (not embedded://) dictionary, use_mmap defaults to true whenever the mmap Cargo feature is compiled in (the default) — memory-mapped reads are used automatically. Set it to false to force eager, non-memory-mapped file reads instead. Every large component stays lazily paged this way: the word-list files, the connection-cost matrix, and the prefix-dictionary trie, which is walked in place over its serialized bytes. use_mmap is silently ignored for embedded:// dictionaries, since their data is already a static, zero-copy byte slice. Requires the mmap cargo feature (enabled by default).

Whitespace Handling

By default, whitespace is skipped in the lattice, as MeCab does: no word starts on a whitespace character, and the word after a run of whitespace connects directly to the word before it, with the connection cost the dictionary was trained with. Skipped whitespace is not output and is left out of token surfaces and offsets; whitespace that belongs to a dictionary entry, inside it (IPADIC-NEologd GeForce GTX, a user entry 해운대 해수욕장) or at its end (ko-dic 에듀 followed by a space), stays in that entry's token, as in MeCab. "Whitespace" is the dictionary's SPACE character category (char.def): U+0020, U+0009, U+000A, U+000B and U+000D for ko-dic, and U+0020, U+0009, U+000A and U+000B for IPADIC, IPADIC-NEologd, UniDic, SudachiDict, CC-CEDICT and Jieba. Those char.def files also map U+00D0 (Ð) to SPACE, a typo for U+000D, but a later 0x00C0..0x00FF ALPHA line takes precedence, so Ð is a letter, as in MeCab.

The default comes from the dictionary's metadata.json (skip_whitespace). SudachiDict sets it to false: Sudachi keeps whitespace in the lattice and outputs it as 空白 tokens, and SudachiDict's costs are tuned for that, so skipping would, for example, split the entry caramel man into two common nouns. With SudachiDict, whitespace therefore stays in the lattice as a SPACE node by default, though it is still dropped from the output. The other bundled dictionaries leave the setting out, which means skipping, and so does a dictionary built before the setting existed (a SudachiDict built by Lindera 6.x included, until it is rebuilt).

Call keep_whitespace(true) on a Segmenter to output whitespace tokens. Whitespace then stays in the lattice as SPACE unknown-word nodes:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).keep_whitespace(true);
}

Before v7, Lindera kept whitespace in the lattice even when it dropped it from the output. Dictionaries trained with MeCab never see a connection to or from the SPACE unknown word: in ko-dic its row and column of the connection matrix are all zero, so every space reset the context and a word between spaces was chosen by its word cost alone. ko-dic read 전 in 2년 전 대회 as 저/NP + ㄴ/JX instead of the noun NNG, and IPADIC, whose SPACE entry shares the context of the full-width space, read が in Google が 新しい as a conjunction. CC-CEDICT and Jieba have no connection costs at all, so their output does not depend on this. skip_whitespace overrides the dictionary's default: skip_whitespace(false) keeps whitespace nodes in the lattice as before, while still dropping whitespace from the output, and skip_whitespace(true) turns skipping on for SudachiDict. In a SegmenterConfig the skip_whitespace key takes true or false, and leaving it out, or null, keeps the dictionary's default; on the CLI, --disable-skip-whitespace turns skipping off:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).skip_whitespace(false);
}

A \n or a \t still ends a segment (see Sentence Splitting), so the context is not carried across a newline or a tab. mecab-ko analyzes a line at a time and skips a tab inside it like any other whitespace.

Unknown-Word Grouping

A word that is not in the dictionary is read as an unknown word, built from the categories that the dictionary's char.def gives its characters (katakana, Latin letters, digits, symbols and so on). At every position that a path through the lattice reaches, Lindera creates unknown-word candidates for each category of the character there if the category is set to always create them (INVOKE in char.def) or if no dictionary word starts at that position, as MeCab does. For a category set to group (GROUP), one candidate is a grouped unknown word: the whole run of characters of that category from that position on, such as a run of katakana.

An unknown word can start at any such position, also inside a run that a grouped unknown word from an earlier position covers. IPADIC's char.def puts ・ in the katakana range, so in ジョン・レノンが歌う the first position has the grouped unknown word ジョン・レノン. The dictionary words ジョン and ・ reach the position of レ, where the unknown word レノン starts, and the result is ジョン, ・, レノン, が and 歌う, as in MeCab. Before v7, normal mode created no unknown words at the positions inside the last grouped unknown word, a shortcut inherited from Kuromoji, so it returned ジョン・レノン as one unknown word; Decompose mode never took this shortcut.

Unknown-word grouping is unbounded by default. max_grouping_len(Some(n)) caps it with MeCab's max-grouping-size semantics, counting characters beyond the first (MeCab defaults to 24):

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).max_grouping_len(Some(24));
}

The cap is applied at each lattice position rather than to a run as a whole. When the same-category run starting at a position is longer than the cap, the grouped candidate is not emitted there; the single-character candidate (plus the length-ladder candidates described below and any dictionary words) remains, and the remaining tail is grouped again once it fits. No unknown token is therefore longer than n + 1 characters, but an over-long run does not turn into single characters throughout. With IPADIC, the out-of-vocabulary katakana run ヷヸヹヺヷ and max_grouping_len(Some(2)) segment as ヷ / ヸ / ヹヺヷ when the length ladder is off: the first two positions see runs of 5 and 4 characters (over the cap) and keep only the single-character candidate, while the 3-character tail fits the cap and groups. With the ladder on (the default), the 2-character ladder candidate wins at the first position and the output is ヷヸ / ヹヺヷ.

Some(0) never groups a run of two or more characters. The max_grouping_len config key and the CLI flag --max-grouping-len treat 0 as unbounded instead.

Independently of grouping, Lindera also generates a MeCab/Vibrato-inspired "length ladder" of shorter unknown-word candidates (up to each category's char.def LENGTH field) by default, so the Viterbi search can pick whichever length scores lowest. Call unknown_word_ladder(false) to disable it: each category then gets only the grouped or the single-character candidate at a position, as before v6. The output still differs from that of Lindera before v6 wherever later changes apply, such as unknown words that start inside a grouped run (see Migrating from v6 to v7):

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).unknown_word_ladder(false);
}

Left-Space Penalty (Korean)

MeCab-based Korean analyzers (mecab-ko with mecab-ko-dic, Lucene's nori) add a cost to a candidate that starts right after whitespace when its part-of-speech tag is one that attaches to the preceding word without a space: particles (J*), endings (E*), the copula (VCP) and derivational suffixes (XS*). Without it, 검색 이 잘 된다 reads 이 as the subject particle JKS rather than the interjection IC. Lindera applies the same penalty by default for dictionaries that ship the rules in their metadata (ko-dic does); other dictionaries are unaffected. Opt out with Segmenter::space_penalty(None), "space_penalty": false in the config, or --disable-space-penalty on the CLI. Together with skip_whitespace(false), this turns off both changes since v6.0 in how Korean is read around spaces (this penalty and whitespace skipping), but not the other output changes since then, such as the split of punctuation runs like ." (see Migrating from v6 to v7).

SpacePenaltyConfig is a list of rules, each pairing first part-of-speech tags with a cost. A candidate is matched by the part of its tag before the first + (for ko-dic Inflect rows, the first_part_of_speech column); the first matching rule wins and unlisted tags cost nothing. mecab-ko-dic's dicrc rules translate to:

#![allow(unused)]
fn main() {
use lindera::space_penalty::{SpacePenaltyConfig, SpacePenaltyRule};

let rules = SpacePenaltyConfig::new(vec![
    SpacePenaltyRule::new(["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], 3000),
    SpacePenaltyRule::new(["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], 6000),
]);
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).space_penalty(Some(rules))?;
}

"Whitespace" means a character of the dictionary's SPACE category (char.def), the same set keep_whitespace filters on; a dictionary without that category falls back to Unicode White_Space. The penalty applies in both modes and in N-best search, to system, user and unknown-word entries alike. space_penalty builds a per-word-id lookup once (a few tens of milliseconds for ko-dic), and returns an error when the dictionary schema has neither a part_of_speech_tag nor a part_of_speech field.

A dictionary can ship its default rules in metadata.json under space_penalty; ko-dic does, with exactly the rules above, and Segmenter::new applies them automatically. space_penalty_from_dictionary() re-enables them after an opt-out and returns an error for a dictionary that ships none. Shipped rules that cannot be applied, because the schema has no part-of-speech field, only log a warning in Segmenter::new:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).space_penalty_from_dictionary()?;
}

Only a ko-dic built by Lindera 6.1.0 or later carries these rules. A ko-dic downloaded from the v6.0.0 release ships none, so with it the penalty silently stays off, and space_penalty_from_dictionary() (or "space_penalty": true) returns an error until the dictionary is rebuilt. Explicit rules (space_penalty, --space-penalty-rules) work with any ko-dic.

In a SegmenterConfig, the space_penalty key takes an object for explicit rules, false for off, and true for the dictionary's rules (an error when it ships none). Leaving the key out, or null, keeps the default: the dictionary's rules when it ships any.

{
  "dictionary": "embedded://ko-dic",
  "space_penalty": {
    "rules": [
      { "pos": ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], "cost": 3000 },
      { "pos": ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], "cost": 6000 }
    ]
  }
}

The penalty keys on the character before a candidate, which is mecab-ko's test as well, so it applies the same way whether whitespace is skipped in the lattice (the default, see Whitespace Handling) or kept as nodes. With whitespace skipped, the connection costs on their own already rule out some readings the penalty targets (시 in 서울 시 에서 reads as NNG without it), but not all of them (이 in 검색 이 잘 된다).

N-Best Segmentation

segment_nbest returns the top-n segmentations ordered by total path cost, each paired with its cost. Set unique to deduplicate results that share the same word boundaries but differ only in POS tags, and cost_threshold to discard paths whose cost exceeds best_cost + threshold:

#![allow(unused)]
fn main() {
let results = segmenter.segment_nbest(Cow::Borrowed("すもももももももものうち"), 3, false, None)?;
for (tokens, cost) in results {
    println!("cost={cost}");
    for token in tokens {
        println!("  {}", token.surface.as_ref());
    }
}
}

Every result segments the whole input. The input is split into segments as described in Sentence Splitting, and each segment is searched on its own. Within a segment, the context is carried across 、, 。 and forced cuts, so the paths of a segment are the n best paths of one lattice over the whole segment, with the exceptions listed there; unique compares the word boundaries of the whole segment. A result takes one path in each segment, its cost is the sum of those paths' costs, and the results are the n cheapest of these combinations. So a result can differ from the best one in a single segment. best_cost is the cost of the first result, so cost_threshold applies to the whole input. The first result is the output of segment, unless several paths have exactly the same cost.

segment_nbest_with_lattice is the same operation but lets you pass in a reusable Lattice buffer to avoid reallocating one per call.

Sentence Splitting

Before building the lattice, Lindera splits the input into sentences at delimiter characters (\n, \t, 。, 、) and builds one Viterbi lattice per sentence. If no delimiter appears within roughly 32 KiB of a sentence's start, Lindera forces a sentence boundary there anyway and logs a warning. The cuts bound the size — and therefore the memory and CPU cost — of each lattice.

A \n, a \t or the end of the input ends a segment, the run of sentences up to it. Segments are independent of each other. Within a segment, the cuts at 、 and 。 and the forced cuts carry the context: the next sentence starts from the words that end the previous one (from their right context IDs, each with the cost of the best path to it) rather than from the beginning of a sentence (BOS), and the connection to the end of the sentence (EOS) is paid only at the end of the segment. So the best path of a segment, and its cost, are those of one lattice over the whole segment, as MeCab, which does not cut inside a line, finds them. Before v7, every sentence started from BOS and ended with EOS, so the word after 、 lost its context: with IPADIC, ああ in 彼は、ああ言った was an interjection, and it is now an adverb, as in MeCab.

The result can still differ from one lattice over the segment:

• No word spans a cut: a dictionary entry that contains 、 or 。, such as UniDic's 一、二塁, never matches, an unknown word is not grouped across a cut, and a word that would span a forced cut is split there.
• The left-space penalty (see Left-Space Penalty (Korean)) does not apply to a word that starts right at a forced cut, even when whitespace precedes the cut.

Ordinary text never reaches the forced cut, but for pathological delimiter-free input (e.g. minified text or base64-encoded blobs), it can affect tokenization at the artificial cut point.

N-best search uses the same segments; see N-Best Segmentation.

Reusable Worker

SegmentWorker is a reusable segmentation session that owns the Viterbi lattice and the scratch buffers of the segmentation, so repeated calls avoid the per-call allocations segment pays. Create one with new_worker (clones the segmenter) or into_worker (consumes it, avoiding a user-dictionary copy), then call segment/segment_nbest on it:

#![allow(unused)]
fn main() {
let mut worker = segmenter.new_worker();
for line in lines {
    let tokens = worker.segment(line)?;
    for token in &tokens {
        println!("{}", token.surface.as_ref());
    }
}
}

The returned tokens borrow the worker, so they must be consumed before the next call (the usual per-line loop above compiles as-is). set_mode and set_keep_whitespace switch the configuration between calls. segmenter() returns a shared reference to the underlying segmenter; no &mut accessor is provided, since swapping the dictionary out from under the reused lattice would break the dictionary-lattice pairing the worker exists to guarantee.

A worker also bounds retained memory: one delimiter-free maximum-length sentence grows the lattice to several MB (roughly 18 MB for a 32 KiB ASCII sentence, 6 MB for a 32 KiB CJK one, whose character-indexed lattice has a third of the slots), and a plain Lattice keeps that forever. The worker automatically shrinks its lattice, and releases its scratch buffers, once a window of calls shows the capacity is oversized, and shrink_to(text_len_hint) forces a shrink immediately. reset() discards the internal buffers outright and replaces them with fresh ones; it is intended for recovery paths (e.g. after a panic poisoned a mutex holding the worker) where the buffers may hold an inconsistent intermediate state — the segmenter configuration itself is preserved.

The worker is permanently bound to the dictionary of the segmenter that created it; there is no way to swap dictionaries under a live worker, which rules out a class of lattice-reuse bugs by construction. For multi-threaded use, create one worker per thread from a shared Segmenter.

Error Handling

Lindera uses a structured error system based on anyhow and thiserror for ergonomic error handling throughout the library.

LinderaResult

The LinderaResult<T> type alias is the standard return type for fallible operations in Lindera:

#![allow(unused)]
fn main() {
pub type LinderaResult<T> = Result<T, LinderaError>;
}

LinderaError

LinderaError is the main error type, containing an error kind and a source error with full context:

#![allow(unused)]
fn main() {
pub struct LinderaError {
    pub kind: LinderaErrorKind,
    source: anyhow::Error,
}
}

The add_context method allows attaching additional context to an error:

#![allow(unused)]
fn main() {
let error = error.add_context("failed to load dictionary from /path/to/dict");
}

LinderaErrorKind

LinderaErrorKind is an enum that categorizes errors:

Creating Errors

Use LinderaErrorKind::with_error to create an error from a kind and a source:

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let error = LinderaErrorKind::Io.with_error(anyhow::anyhow!("file not found: config.yml"));
}

Using the ? Operator

Since Lindera functions return LinderaResult, the ? operator can propagate errors naturally:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn analyze(text: &str) -> LinderaResult<Vec<String>> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let tokens = tokenizer.tokenize(text)?;
    Ok(tokens.iter().map(|t| t.surface.as_ref().to_string()).collect())
}
}

Error Handling Patterns

Matching on Error Kind

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::error::LinderaErrorKind;

match load_dictionary("/path/to/dictionary") {
    Ok(dict) => { /* use dictionary */ }
    Err(e) if e.kind() == LinderaErrorKind::NotFound => {
        eprintln!("Dictionary not found: {}", e);
    }
    Err(e) if e.kind() == LinderaErrorKind::Io => {
        eprintln!("I/O error loading dictionary: {}", e);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}
}

Converting from External Errors

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let content = std::fs::read_to_string("config.yml")
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))?;
}

API Reference

The API reference is available. Please see following URL:

• lindera
• lindera-segmenter
• lindera-analysis

Lindera Analysis

Lindera Analysis layers a Lucene-style text analysis chain on top of the morphological segmenter provided by the lindera-segmenter crate. It is normally used through the lindera facade as lindera::analysis (enabled by the default analysis feature). It composes character filters, a Segmenter, and token filters into a single Tokenizer pipeline that can be built programmatically or loaded entirely from a YAML configuration file.

Key Features

• Character filters that transform input text before segmentation, with automatic byte-offset correction back to the original text
• Token filters that transform, merge, filter, or reorder the tokens produced by the segmenter
• Tokenizer / TokenizerBuilder to assemble a full analysis pipeline, either in Rust code or from a YAML file (LINDERA_CONFIG_PATH)
• Built-in filters covering Japanese, Korean, and general-purpose text normalization

Dependency

Through the facade (recommended):

[dependencies]
lindera = "7"

#![allow(unused)]
fn main() {
use lindera::analysis::tokenizer::Tokenizer;
}

As a direct dependency. It is the same crate, so both paths name the same types; keep lindera and lindera-analysis at the same major version (see Migration v6 to v7):

[dependencies]
lindera = "7"
lindera-analysis = "7"

#![allow(unused)]
fn main() {
use lindera_analysis::tokenizer::Tokenizer;
}

Contents

• Configuration -- YAML configuration file format for the Tokenizer
• Filters -- Reference for all character filters and token filters
• Architecture -- Internal structure and key components
• API Reference -- API documentation

Configuration

Lindera is able to read YAML format configuration files. Specify the path to the following file in the environment variable LINDERA_CONFIG_PATH. You can use it easily without having to code the behavior of the tokenizer in Rust code.

segmenter:
  mode: "normal"
  dictionary: "embedded://ipadic"
  # user_dictionary: "./resources/user_dict/ipadic_simple_userdic.csv"
  # keep_whitespace: false
  # skip_whitespace: false # skip whitespace in the lattice, as MeCab does; on by default except for SudachiDict. false keeps whitespace nodes (pre-v7)
  # use_mmap: false # only meaningful for filesystem (non-embedded://) dictionaries
  # max_grouping_len: 24 # cap on unknown-word grouping; omit or 0 for unbounded
  # unknown_word_ladder: true # emit shorter unknown-word candidates (default: true)
  # space_penalty: false # left-space penalty (Korean); on by default for dictionaries that ship rules (ko-dic). false turns it off, an object gives explicit rules

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true
  - kind: mapping
    args:
       mapping:
         リンデラ: Lindera

token_filters:
  - kind: "japanese_compound_word"
    args:
      # Merge the numeral tokens only. A counter merged into the number token keeps japanese_number from converting it.
      tags:
        - "名詞,数"
      new_tag: "名詞,数"
  - kind: "japanese_number"
    args:
      tags:
        - "名詞,数"
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "接続詞"
        - "助詞"
        - "助詞,格助詞"
        - "助詞,格助詞,一般"
        - "助詞,格助詞,引用"
        - "助詞,格助詞,連語"
        - "助詞,係助詞"
        - "助詞,副助詞"
        - "助詞,間投助詞"
        - "助詞,並立助詞"
        - "助詞,終助詞"
        - "助詞,副助詞／並立助詞／終助詞"
        - "助詞,連体化"
        - "助詞,副詞化"
        - "助詞,特殊"
        - "助動詞"
        - "記号"
        - "記号,一般"
        - "記号,読点"
        - "記号,句点"
        - "記号,空白"
        - "記号,括弧閉"
        - "その他,間投"
        - "フィラー"
        - "非言語音"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "remove_diacritical_mark"
    args:
      japanese: false

Segmenter Options

% export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

use std::path::PathBuf;

use lindera::analysis::tokenizer::TokenizerBuilder;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    // Load tokenizer configuration from file
    let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("config")
        .join("lindera.yml");

    let builder = TokenizerBuilder::from_file(&path)?;

    let tokenizer = builder.build()?;

    let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。".to_string();
    println!("text: {text}");

    let tokens = tokenizer.tokenize(&text)?;

    for token in tokens {
        println!(
            "token: {:?}, start: {:?}, end: {:?}, details: {:?}",
            token.surface, token.byte_start, token.byte_end, token.details
        );
    }

    Ok(())
}

Filters

Character filters and token filters are the two pre/post-processing stages of the lindera-analysis Tokenizer pipeline.

• Character filters transform the input text before segmentation. Byte offsets are corrected automatically so that the resulting tokens still report positions relative to the original, unfiltered text.
• Token filters transform the list of tokens produced by the segmenter after segmentation.

Both kinds of filters are configured the same way: a kind string (also used with the CLI's --character-filter / --token-filter flags in the form kind:{"json": "args"}) and a JSON args object with filter-specific parameters.

Character Filters

Character filters are configured under the character_filters key of the YAML configuration file. Each entry is applied in order, and each filter's output feeds into the next.

unicode_normalize

Normalizes the input text using one of the four standard Unicode normalization forms.

Parameters:

Example:

{
  "kind": "unicode_normalize",
  "args": {
    "kind": "nfkc"
  }
}

japanese_iteration_mark

Normalizes Japanese iteration marks (々, ゝ, ゞ, ヽ, ヾ) by replacing each mark with the character it repeats, adding or removing the voiced sound mark (dakuten) as needed for the hiragana/katakana variants.

Parameters:

Example:

{
  "kind": "japanese_iteration_mark",
  "args": {
    "normalize_kanji": true,
    "normalize_kana": true
  }
}

mapping (character filter)

Replaces occurrences of the keys in mapping with their corresponding values, using longest-match search over the input text (built with an Aho-Corasick automaton).

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "リンデラ": "Lindera"
    }
  }
}

regex

Replaces every match of a regular expression in the input text with a literal replacement string. Capture groups are not interpolated into the replacement.

Parameters:

Example:

{
  "kind": "regex",
  "args": {
    "pattern": "\\s{2,}",
    "replacement": " "
  }
}

Token Filters

Token filters are configured under the token_filters key of the YAML configuration file. Each filter is applied in order to the token list produced by the segmenter.

japanese_base_form

Replaces the token's surface text with its base (dictionary) form, as registered in the base_form or orthographic_base_form field of the dictionary. Acts as a lemmatizer for verbs and adjectives. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_base_form"
}

japanese_compound_word

Merges consecutive tokens whose part-of-speech tag matches one of tags into a single compound token.

The merged token's details carry new_tag at the part-of-speech position resolved from the dictionary schema; any leading non-part-of-speech field (SudachiDict's display_surface) and every remaining field are set to *.

Parameters:

Example:

{
  "kind": "japanese_compound_word",
  "args": {
    "tags": [
      "名詞,数",
      "名詞,接尾,助数詞"
    ],
    "new_tag": "名詞,数"
  }
}

japanese_kana

Converts token text between hiragana and katakana.

Parameters:

Example:

{
  "kind": "japanese_kana",
  "args": {
    "kind": "hiragana"
  }
}

japanese_katakana_stem

Removes a trailing prolonged sound mark (ー, U+30FC) from katakana tokens, but only when the token is longer than min characters.

Parameters:

Example:

{
  "kind": "japanese_katakana_stem",
  "args": {
    "min": 3
  }
}

japanese_keep_tags

Keeps only tokens whose part-of-speech tag matches one of tags, removing all others.

Parameters:

Example:

{
  "kind": "japanese_keep_tags",
  "args": {
    "tags": [
      "名詞,一般",
      "名詞,固有名詞,一般"
    ]
  }
}

japanese_number

Converts Japanese numeral representations (kanji numerals, formal/legal kanji numerals, and fullwidth digits) in the token's surface text to Arabic numerals.

A token is converted only when every one of its characters takes part in a numeral, the rule Lucene's JapaneseNumberFilter applies. An ordinary word that merely begins with a numeral character is therefore returned unchanged: 一部 stays 一部 rather than becoming 1部.

When this filter follows japanese_compound_word, merge only the 名詞,数 tokens. A counter (名詞,接尾,助数詞) merged into the number token makes the merged token something other than a numeral, so 一万円 is left alone instead of being converted.

Parameters:

Example:

{
  "kind": "japanese_number",
  "args": {
    "tags": [
      "名詞,数"
    ]
  }
}

japanese_reading_form

Replaces the token's surface text with its reading, in katakana, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_reading_form"
}

japanese_stop_tags

Removes tokens whose part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "japanese_stop_tags",
  "args": {
    "tags": [
      "助詞,格助詞,一般",
      "助詞,係助詞",
      "助詞,連体化",
      "助動詞"
    ]
  }
}

Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*). The same matching rule applies to japanese_keep_tags, japanese_compound_word, and the tags option of japanese_number. Because the position is resolved through the schema, the rule also holds for dictionaries whose schema has leading non-part-of-speech columns, such as SudachiDict's display_surface.

keep_words

Keeps only tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "keep_words",
  "args": {
    "words": [
      "すもも",
      "もも"
    ]
  }
}

korean_compound_word

Merges consecutive Korean tokens whose first part-of-speech tag matches one of tags into a single compound token. The merged token keeps the first token's byte_start and position, takes the last token's byte_end, and its position_length is the number of tokens merged. A matching token whose neighbours do not match is left as it is.

ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so this filter is what turns 이/NR 천/NR 이/NR 십/NR 육/NR back into one token before korean_number converts it:

Parameters:

Example:

{
  "kind": "korean_compound_word",
  "args": {
    "tags": [
      "SN",
      "NR"
    ],
    "new_tag": "NR"
  }
}

만 right after a space is NR under the default left-space penalty, but at the start of the text or right after another numeral ko-dic tags it JX (만 원, 십만 원), which stays outside ["SN", "NR"].

korean_decompound

Splits ko-dic compound, inflected and pre-analyzed tokens into the morphemes their expression field lists, reproducing Lucene nori's DecompoundMode. This is the inverse of korean_compound_word, which merges neighbouring tokens. Lindera's Mode::Decompose applies a length penalty tuned for Japanese kanji and does not read these fields, so 무궁화 stays whole in every mode without this filter.

Fragment offsets follow nori: only Compound fragments get their own byte range, each covering the text it spells. Inflect and Preanalysis fragments share the whole token's span, because an inflected form's fragments need not be substrings of it (갔 decomposes to 가 + 았).

A Compound token is carved only when its fragments spell its surface exactly; otherwise it shares its span too. That covers the 12 bundled ko-dic entries whose fragments are not the surface's own text (안지랑역 = 안지 + 역, 고춧값 = 고추 + 값), where nori's offsets point some fragments at text they do not spell, as well as a user-dictionary entry such as 프린터3D = 프린터 + 쓰리디 and a surface an earlier filter has rewritten. For every other bundled entry the offsets are nori's, and every offset is safe to slice the text with.

In discard the fragments take consecutive positions starting at the compound's own; in mixed the compound keeps that position with a position_length spanning its fragments, which are then emitted from the same position onward. Tokens after the expansion shift by the positions it added, and only by those, so a gap left by an earlier filter such as korean_stop_tags survives.

Only tokens spanning a single position are expanded, so the compound a mixed pass keeps is not split again. Put the filter in a chain once all the same: a single-fragment entry kept by mixed still spans one position. When combining it with korean_compound_word, place the merge first; merging never joins tokens that overlap in byte range, so the reverse order does not corrupt anything, but merging first lets it see the segmenter's own tokens.

A fragment takes its own part-of-speech tag and semantic class and blanks the fields that describe the compound (type, expression, first_part_of_speech, last_part_of_speech, reading, presence_absence). Blanking the reading matches nori, whose DecompoundToken.getReading() returns null; korean_reading_form skips *, so a fragment keeps its own surface.

Applied to a dictionary whose schema has no type/expression fields, the filter warns once per configured filter and passes every token through unchanged.

Parameters:

Example:

{
  "kind": "korean_decompound",
  "args": {
    "mode": "discard",
    "types": [
      "Compound",
      "Inflect",
      "Preanalysis"
    ]
  }
}

korean_keep_tags

Keeps only Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_keep_tags",
  "args": {
    "tags": [
      "NNG"
    ]
  }
}

korean_number

Converts Sino-Korean numeral representations (Hangul numerals, Hanja numerals, and fullwidth digits) in the token's surface text to Arabic numerals. Native Korean numerals (하나, 둘, 열, 스물, ...) have no positional structure and are left unchanged.

Each token is converted on its own. ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so the tokens of a single number do not add up to one Arabic numeral:

Put korean_compound_word in front of this filter to merge those tokens first, the way japanese_compound_word does for IPADIC. Its new_tag must be one of this filter's tags for the merged token to be converted:

[
  {"kind": "korean_compound_word", "args": {"tags": ["SN", "NR"], "new_tag": "NR"}},
  {"kind": "korean_number", "args": {"tags": ["SN", "NR"]}}
]

With that chain 이천이십육년 becomes 2026 년, 10만 becomes 100000 and 2천26 becomes 2026. A single token that holds the whole numeral, such as the Hanja 二千二十六/SH, is converted in full by this filter alone.

Parameters:

A token is converted only when every one of its characters is a numeral, the rule Lucene's KoreanNumberFilter applies. That is what keeps native Korean numerals intact: 일곱 ("seven") starts with the Sino-Korean 일, and a character-by-character conversion would give 1곱.

Hangul numerals are also homographs of very common morphemes (이 is a subject particle, 만 an auxiliary particle), and those are single characters that the all-numeral rule cannot separate from a numeral. That is why the default is restricted to the numeral tags.

Add SH to convert Hanja numerals as well. The coverage is partial in two ways: ko-dic tags whole Hanja words as NNG (參加, 萬歲 are untouched), while a single Hanja character it cannot attach to a word is SH on its own, so 十字架 becomes 10 字 架 and 百貨店 becomes 100 貨 店.

Example:

{
  "kind": "korean_number",
  "args": {
    "tags": [
      "SN",
      "NR"
    ]
  }
}

korean_reading_form

Replaces the token's surface text with its reading, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "korean_reading_form"
}

korean_stop_tags

Removes Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_stop_tags",
  "args": {
    "tags": [
      "EP",
      "EF",
      "JKG"
    ]
  }
}

length

Keeps only tokens whose surface text length (in characters) falls within [min, max].

Parameters:

Example:

{
  "kind": "length",
  "args": {
    "min": 2,
    "max": 3
  }
}

lowercase

Converts token surface text to lowercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "lowercase"
}

mapping (token filter)

Replaces occurrences of the keys in mapping with their corresponding values in each token's surface text, using longest-match search (built with an Aho-Corasick automaton). This is the token-level counterpart of the mapping character filter.

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "籠": "篭"
    }
  }
}

remove_diacritical_mark

Removes diacritical (combining) marks from token surface text, re-applying the text's original Unicode normalization form afterward.

Parameters:

Example:

{
  "kind": "remove_diacritical_mark",
  "args": {
    "japanese": false
  }
}

stop_words

Removes tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "stop_words",
  "args": {
    "words": [
      "も",
      "の"
    ]
  }
}

uppercase

Converts token surface text to uppercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "uppercase"
}

YAML Configuration

Character filters and token filters are configured together with the segmenter in a single YAML file. See Configuration for the full file format; the relevant excerpt looks like this:

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true

token_filters:
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "助詞,格助詞,一般"
        - "助詞,係助詞"
        - "助詞,連体化"
        - "助動詞"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "lowercase"
  - kind: "length"
    args:
      min: 2

Rust API

Character filters and token filters can also be created and applied programmatically:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::character_filter::BoxCharacterFilter;
use lindera::analysis::character_filter::unicode_normalize::{
    UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera::analysis::token_filter::BoxTokenFilter;
use lindera::analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera::analysis::token_filter::japanese_katakana_stem::JapaneseKatakanaStemTokenFilter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let mut tokenizer = Tokenizer::new(segmenter);

    // Add a character filter
    let normalize_filter = UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);
    tokenizer.append_character_filter(BoxCharacterFilter::from(normalize_filter));

    // Add token filters
    let stop_tags_filter = JapaneseStopTagsTokenFilter::new(
        vec![
            "助詞,格助詞,一般".to_string(),
            "助詞,係助詞".to_string(),
            "助詞,連体化".to_string(),
            "助動詞".to_string(),
        ]
        .into_iter()
        .collect(),
    );
    tokenizer.append_token_filter(BoxTokenFilter::from(stop_tags_filter));

    let katakana_stem_filter =
        JapaneseKatakanaStemTokenFilter::new(std::num::NonZeroUsize::new(3).unwrap());
    tokenizer.append_token_filter(BoxTokenFilter::from(katakana_stem_filter));

    // Tokenize with filters applied
    let tokens = tokenizer.tokenize("Linderaは形態素解析エンジンです。")?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

The append_character_filter and append_token_filter methods add filters in order. Character filters are applied sequentially to the text before segmentation; token filters are applied sequentially to the token list after segmentation.

Architecture

Module Structure

lindera-analysis/src/
├── lib.rs                          # Public API re-exports, CLI flag parsing helper
├── character_filter.rs             # CharacterFilter trait, OffsetMapping, CharacterFilterLoader
├── character_filter/
│   ├── unicode_normalize.rs        # Unicode normalization (NFC/NFD/NFKC/NFKD)
│   ├── japanese_iteration_mark.rs  # Japanese iteration mark normalization
│   ├── mapping.rs                  # Mapping-based text replacement
│   └── regex.rs                    # Regex-based text replacement
├── token_filter.rs                 # TokenFilter trait, TokenFilterLoader
├── token_filter/
│   ├── compound.rs                 # Shared compound-word merge helper (private)
│   ├── japanese_base_form.rs
│   ├── japanese_compound_word.rs
│   ├── japanese_kana.rs
│   ├── japanese_katakana_stem.rs
│   ├── japanese_keep_tags.rs
│   ├── japanese_number.rs
│   ├── japanese_reading_form.rs
│   ├── japanese_stop_tags.rs
│   ├── keep_words.rs
│   ├── korean_compound_word.rs
│   ├── korean_decompound.rs
│   ├── korean_keep_tags.rs
│   ├── korean_number.rs
│   ├── korean_reading_form.rs
│   ├── korean_stop_tags.rs
│   ├── length.rs
│   ├── lowercase.rs
│   ├── mapping.rs
│   ├── numeral.rs                  # Shared numeral conversion engine (private)
│   ├── remove_diacritical_mark.rs
│   ├── stop_words.rs
│   ├── tags.rs                     # Shared keep/stop-tag filtering helpers (private)
│   └── uppercase.rs
└── tokenizer.rs                     # Tokenizer, TokenizerBuilder

Key Components

CharacterFilter

A trait for filters that preprocess text before segmentation. Each implementation provides a name() and an apply(&self, text: &mut String) -> LinderaResult<OffsetMapping> method that rewrites text in place and returns an OffsetMapping describing every transformation it performed.

The OffsetMapping (built from a list of Transformation records) lets the Tokenizer translate token byte offsets computed against the filtered text back to byte offsets in the original input, even after multiple filters have run in sequence. BoxCharacterFilter wraps any CharacterFilter implementation as a boxed, cloneable trait object, and CharacterFilterLoader builds one from a kind string plus a serde_json::Value of arguments (used both by YAML configuration loading and by CLI flag parsing).

TokenFilter

A trait for filters that post-process the tokens produced by the segmenter. Each implementation provides a name() and an apply(&self, tokens: &mut Vec<Token<'_>>) -> LinderaResult<()> method that modifies, merges, reorders, or removes tokens in place. BoxTokenFilter wraps any TokenFilter implementation as a boxed, cloneable trait object, and TokenFilterLoader builds one from a kind string plus a serde_json::Value of arguments, mirroring CharacterFilterLoader.

Tokenizer / TokenizerBuilder

Tokenizer composes character filters, a lindera::segmenter::Segmenter, and token filters into a single analysis pipeline. Calling tokenize runs the character filters over the input text, segments the filtered text, applies the token filters to the resulting tokens, and finally corrects each token's byte offsets back to the original text via the recorded OffsetMappings.

TokenizerBuilder assembles a Tokenizer from a TokenizerConfig (a serde_json::Value), which can be constructed programmatically, loaded from a YAML file (via TokenizerBuilder::from_file, or automatically from the LINDERA_CONFIG_PATH environment variable via TokenizerBuilder::new), or built up incrementally with set_segmenter_mode, set_segmenter_dictionary, append_character_filter, and append_token_filter. See Configuration for the YAML file format and Filters for the full filter reference.

AnalysisWorker

AnalysisWorker (created with Tokenizer::new_worker or Tokenizer::into_worker) is a reusable session over the full analysis chain. It owns every per-call buffer — the Viterbi lattice and backtrace scratch (via SegmentWorker), the normalized-text buffer the character filters operate on, and the offset-mapping scratch — so repeated tokenize calls avoid the per-call allocations Tokenizer::tokenize pays. When character filters are configured, token surfaces borrow the worker's buffer instead of being copied into per-token Strings. Returned tokens borrow the worker and must be consumed before the next call; for multi-threaded use, create one worker per thread (or guard one with a Mutex, as lindera-binding does). See the Segmenter page for the underlying SegmentWorker and its automatic memory-shrink policy.

AnalysisWorker's public methods:

• tokenize(&mut self, text: &str) — tokenizes text through the full analysis chain, reusing the worker's internal buffers; produces exactly the same tokens as Tokenizer::tokenize.
• tokenize_nbest(&mut self, text: &str, n, unique, cost_threshold) — tokenizes text and returns the top-N results with costs, reusing the worker's internal buffers; produces exactly the same results as Tokenizer::tokenize_nbest.
• set_mode(&mut self, mode: Mode) — sets the segmentation mode for subsequent calls.
• set_keep_whitespace(&mut self, keep: bool) — sets whether whitespace tokens are kept in the output for subsequent calls.
• shrink_to(&mut self, text_len_hint: usize) — immediately shrinks the worker's internal buffers to what an input of text_len_hint bytes needs.
• reset(&mut self) — discards all internal buffers, replacing them with fresh ones; intended for recovery paths (e.g. after a panic poisoned a mutex holding the worker), with configuration (dictionary, filters, mode) preserved.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-analysis

Lindera CLI

A morphological analysis command-line interface for Lindera.

• Installation - Install or build the CLI
• Commands - Command reference for list, tokenize, build, download, train, and export
• Tutorial - Step-by-step guide to get started

Installation

Install via Cargo

You can install the binary via cargo:

% cargo install lindera-cli

Download from GitHub Releases

Alternatively, you can download a pre-built binary from the release page:

• https://github.com/lindera/lindera/releases

Obtaining Dictionaries

Lindera does not bundle dictionaries with the binary. The easiest way to obtain one is the download subcommand, which fetches the pre-built dictionary matching the CLI version and installs it under the OS-standard application data directory:

% lindera download ipadic

After downloading, the dictionary can be referenced by name:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict ipadic

See Commands for the available dictionary names and storage locations.

Alternatively, you can download a pre-built dictionary manually from the GitHub Releases page:

# Example: download and extract the IPADIC dictionary
% curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
% unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Then specify the extracted dictionary path when using the CLI (note that the archive contains a lindera-ipadic directory):

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict /path/to/ipadic/lindera-ipadic

Build from Source

Build without dictionaries (default)

Build a binary containing only the tokenizer and trainer without embedded dictionaries:

% cargo build --release

Build with all features

% cargo build --release --all-features

Build with Embedded Dictionaries (Advanced)

For advanced users who want to embed dictionaries directly into the binary, use the embed-* feature flags. This eliminates the need for external dictionary files at runtime but increases the binary size.

IPADIC (Japanese dictionary)

% cargo build --release --features=embed-ipadic

IPADIC NEologd (Japanese dictionary)

% cargo build --release --features=embed-ipadic-neologd

UniDic (Japanese dictionary)

% cargo build --release --features=embed-unidic

SudachiDict (Japanese dictionary)

% cargo build --release --features=embed-sudachidict

ko-dic (Korean dictionary)

% cargo build --release --features=embed-ko-dic

CC-CEDICT (Chinese dictionary)

% cargo build --release --features=embed-cc-cedict

Jieba (Chinese dictionary)

% cargo build --release --features=embed-jieba

[!TIP] After building with an embed-* feature flag, use the embedded:// scheme to load the embedded dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic

See Feature Flags for details.

Commands

The Lindera CLI provides six main commands:

• list - List the morphological analysis dictionaries and their status
• tokenize - Perform morphological analysis on text
• build - Build a dictionary from source CSV files
• download - Download a pre-built dictionary from the GitHub releases page
• train - Train a CRF model from annotated corpus data
• export - Export a trained model to dictionary format

list

List all known morphological analysis dictionaries and show, for each one, whether it is embedded in the binary (via the embed-* feature flags) and whether a pre-built copy has been downloaded locally with lindera download.

List parameters

This command takes no arguments. The LINDERA_DATA_DIR environment variable overrides the application data directory searched for downloaded dictionaries, exactly as it does for lindera download.

List usage

% lindera list

NAME            EMBEDDED  DOWNLOADED  PATH
ipadic          yes       yes         /home/user/.local/share/lindera/dictionaries/5.1.0/lindera-ipadic
ipadic-neologd  no        no          -
unidic          no        incomplete  /home/user/.local/share/lindera/dictionaries/5.1.0/lindera-unidic
sudachidict     no        no          -
ko-dic          no        no          -
cc-cedict       no        no          -
jieba           no        no          -

The output contains one dictionary per line:

• EMBEDDED is yes when the dictionary was embedded at build time (e.g. --features=embed-ipadic).
• DOWNLOADED is yes when a complete dictionary installed by lindera download exists for the running CLI version, and incomplete when the installation directory exists but required files are missing (re-download it with lindera download <name> --force).
• PATH shows the installation directory when it exists, and - otherwise.

tokenize

Perform morphological analysis (tokenization) on Japanese, Chinese, or Korean text using various dictionaries.

Parameters

• --dict / -d: Dictionary path, URI, or downloaded dictionary name (required)
  • File path: /path/to/dictionary
  • Embedded: embedded://ipadic, embedded://unidic, etc.
  • Downloaded dictionary name: ipadic, unidic, etc. — resolved to the dictionary installed by lindera download. An existing filesystem path with the same name takes precedence.
• --output / -o: Output format (default: mecab)
  • mecab: MeCab-compatible format with part-of-speech info
  • wakati: Space-separated tokens only
  • json: Detailed JSON format with all token information
• --user-dict / -u: User dictionary path (optional)
• --mode / -m: Tokenization mode (default: normal)
  • normal: Standard tokenization
  • decompose: Decompose compound words
• --char-filter / -c: Character filter configuration (JSON)
• --token-filter / -t: Token filter configuration (JSON)
• --keep-whitespace: Keep whitespace tokens in the output (by default, whitespace is skipped in the lattice and dropped from the output, as MeCab does)
• --disable-skip-whitespace: Keep whitespace in the lattice as SPACE unknown-word nodes instead of skipping it, while still dropping it from the output (the behavior before v7). Every space then resets the connection context. Skipping is on by default for every dictionary whose metadata.json does not turn it off (SudachiDict does). No effect with --keep-whitespace. See the Segmenter documentation
• --max-grouping-len: Cap on unknown-word grouping, in characters beyond the first (MeCab's max-grouping-size; MeCab defaults to 24). The cap is applied at each position: a same-category run longer than the cap is not grouped there, the single-character candidate (plus the length ladder and dictionary words) remains, and the remaining tail is grouped again once it fits, so no unknown token exceeds cap+1 characters. 0 or omitted: unbounded. See the Segmenter documentation for an example
• --disable-unknown-word-ladder: Disable the MeCab/Vibrato-inspired unknown-word length ladder (char.def's LENGTH field). Enabled by default. Disabling it leaves out these candidates, as before v6; other changes since then, such as unknown words that start inside a grouped run, still apply. See the Segmenter documentation
• --disable-space-penalty: Disable the left-space penalty (mecab-ko's left-space-penalty-factor) that a dictionary shipping rules in its metadata.json applies by default (ko-dic ships mecab-ko-dic's). A candidate that starts right after whitespace and whose first part-of-speech tag is listed gets the cost added; disabling it together with --disable-skip-whitespace turns off both changes since v6.0 in how Korean is read around spaces (this penalty and whitespace skipping), but not the other output changes since then, such as the split of punctuation runs like .". See the Segmenter documentation
• --space-penalty-rules: Use explicit left-space penalty rules as JSON instead of the dictionary's, e.g. '{"rules":[{"pos":["JKS","JX"],"cost":6000}]}'. Mutually exclusive with --disable-space-penalty
• --mmap: Use memory-mapped file loading for the dictionary directory's word list. Ignored for embedded:// dictionaries and when the mmap feature is disabled. Rebuilding or truncating the dictionary directory while a process holds it mapped can cause a SIGBUS on the next lookup.
• --nbest / -N: Number of N-best results to return (default: 1). When set to 2 or more, N-best output is enabled.
• --nbest-unique: Deduplicate N-best results by removing paths that produce the same segmentation.
• --nbest-cost-threshold: Maximum cost difference from the best result. Only results with cost within best_cost + threshold are returned; both costs are totals over the whole input line.
• Input file: Optional file path (default: stdin)

Basic usage

# Tokenize text using a dictionary directory
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /path/to/dictionary

# Tokenize text using embedded dictionary
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

# Tokenize with different output format
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

# Tokenize text from file
lindera tokenize \
  --dict /path/to/dictionary \
  --output wakati \
  input.txt

Examples with external dictionaries

Tokenize with external IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external IPADIC Neologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-neologd-0.0.7-20200820

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-unidic-2.1.2

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

Tokenize with external ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict /tmp/lindera-ko-dic-2.1.1-20180720

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

Tokenize with external CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-cc-cedict-0.1.0-20200409

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

Tokenize with external Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-jieba-0.1.1

Examples with embedded dictionaries

Lindera can include dictionaries directly in the binary when built with specific feature flags. This allows tokenization without external dictionary files.

Tokenize with embedded IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC dictionary in the binary, you must build with the --features=embed-ipadic option.

Tokenize with embedded UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://unidic

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-unidic option.

Tokenize with embedded IPADIC NEologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic-neologd

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-ipadic-neologd option.

Tokenize with embedded ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict embedded://ko-dic

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

NOTE: To include ko-dic dictionary in the binary, you must build with the --features=embed-ko-dic option.

Tokenize with embedded CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://cc-cedict

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

NOTE: To include CC-CEDICT dictionary in the binary, you must build with the --features=embed-cc-cedict option.

Tokenize with embedded Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://jieba

NOTE: To include Jieba dictionary in the binary, you must build with the --features=embed-jieba option.

User dictionary examples

Lindera supports user dictionaries to add custom words alongside system dictionaries. User dictionaries can be in CSV or binary format.

Use user dictionary (CSV format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Use user dictionary (Binary format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920 \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.bin

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Tokenization modes

Lindera provides two tokenization modes: normal and decompose.

Normal mode (default)

Tokenizes faithfully based on words registered in the dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode normal

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Decompose mode

Tokenizes compound noun words additionally:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Output formats

Lindera provides three output formats: mecab, wakati and json.

MeCab format (default)

Outputs results in MeCab-compatible format with part-of-speech information:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output mecab

お待ち  名詞,サ変接続,*,*,*,*,お待ち,オマチ,オマチ
し  動詞,自立,*,*,サ変・スル,連用形,する,シ,シ
て  助詞,接続助詞,*,*,*,*,て,テ,テ
おり  動詞,非自立,*,*,五段・ラ行,連用形,おる,オリ,オリ
ます  助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。  記号,句点,*,*,*,*,。,。,。
EOS

Wakati format

Outputs only the token text separated by spaces:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

お待ち し て おり ます 。

JSON format

Outputs detailed token information in JSON format:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

[
  {
    "base_form": "お待ち",
    "byte_end": 9,
    "byte_start": 0,
    "conjugation_form": "*",
    "conjugation_type": "*",
    "part_of_speech": "名詞",
    "part_of_speech_subcategory_1": "サ変接続",
    "part_of_speech_subcategory_2": "*",
    "part_of_speech_subcategory_3": "*",
    "pronunciation": "オマチ",
    "reading": "オマチ",
    "surface": "お待ち",
    "word_id": 14698
  },
  ...
]

N-Best tokenization

Lindera supports N-Best tokenization, which returns the top N tokenization candidates ordered by cost (lower cost = better). This is based on the Forward-DP Backward-A* algorithm, compatible with MeCab's N-Best implementation.

A line is cut into sentences at \t, 。 and 、, but the context is carried across 、 and 。, so the candidates for a line without a tab are the N best segmentations of the whole line (for the few exceptions, see Sentence Splitting). A tab ends a segment: a line with tabs is searched segment by segment, and its candidates are the N cheapest combinations of one path per segment. Every candidate segments the whole line.

Basic N-Best example

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  -N 3

NBEST 1 (cost=7546)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 2 (cost=7914)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 3 (cost=10060)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
も      助詞,係助詞,*,*,*,*,も,モ,モ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

N-Best with unique results

When the same segmentation appears in multiple paths (differing only in internal Viterbi states), use --nbest-unique to deduplicate:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 5 --nbest-unique -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です
NBEST 4 (cost=19320)
営業 部長 谷川 で す
NBEST 5 (cost=20814)
営業 部 長谷 川 です

N-Best with cost threshold

Use --nbest-cost-threshold to limit results to paths within a certain cost range of the best path:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 10 --nbest-unique --nbest-cost-threshold 5000 -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です

Only 3 results are returned because the remaining candidates exceed 15760 + 5000 = 20760.

Advanced tokenization with filters

Lindera provides an analytical framework that combines character filters, tokenizers, and token filters for advanced text processing. Filters are configured using JSON.

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般"]}'

すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
EOS

build

Build (compile) a morphological analysis dictionary from source CSV files for use with Lindera.

Build parameters

• --src / -s: Source directory containing dictionary CSV files (or single CSV file for user dictionary)
• --dest / -d: Destination directory for compiled dictionary output
• --metadata / -m: Metadata configuration file (metadata.json) that defines dictionary structure
• --user / -u: Build user dictionary instead of system dictionary (optional flag)
• --context-id-freq / -f: Context-ID access-frequency file used to reorder connection-cost IDs (optional; only meaningful when the dictionary's metadata.json sets connection_id_mapping: true)

Dictionary types

System dictionary

A full morphological analysis dictionary containing:

• Lexicon entries (word definitions)
• Connection cost matrix
• Unknown word handling rules
• Character type definitions

User dictionary

A supplementary dictionary for custom words that works alongside a system dictionary.

Examples

Build IPADIC (Japanese dictionary)

# Download and extract IPADIC source files
% curl -L -o /tmp/mecab-ipadic-2.7.0-20250920.tar.gz "https://Lindera.dev/mecab-ipadic-2.7.0-20250920.tar.gz"
% tar zxvf /tmp/mecab-ipadic-2.7.0-20250920.tar.gz -C /tmp

# Build the dictionary
% lindera build \
  --src /tmp/mecab-ipadic-2.7.0-20250920 \
  --dest /tmp/lindera-ipadic-2.7.0-20250920 \
  --metadata ./lindera-ipadic/metadata.json

Build IPADIC NEologd (Japanese dictionary)

% curl -L -o /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz "https://lindera.dev/mecab-ipadic-neologd-0.0.7-20200820.tar.gz"
% tar zxvf /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ipadic-neologd-0.0.7-20200820 \
  --dest /tmp/lindera-ipadic-neologd-0.0.7-20200820 \
  --metadata ./lindera-ipadic-neologd/metadata.json

Build UniDic (Japanese dictionary)

% curl -L -o /tmp/unidic-mecab-2.1.2.tar.gz "https://Lindera.dev/unidic-mecab-2.1.2.tar.gz"
% tar zxvf /tmp/unidic-mecab-2.1.2.tar.gz -C /tmp

% lindera build \
  --src /tmp/unidic-mecab-2.1.2 \
  --dest /tmp/lindera-unidic-2.1.2 \
  --metadata ./lindera-unidic/metadata.json

Build CC-CEDICT (Chinese dictionary)

% curl -L -o /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz "https://lindera.dev/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz"
% tar zxvf /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz -C /tmp

% lindera build \
  --src /tmp/CC-CEDICT-MeCab-0.1.0-20200409 \
  --dest /tmp/lindera-cc-cedict-0.1.0-20200409 \
  --metadata ./lindera-cc-cedict/metadata.json

Build Jieba (Chinese dictionary)

% curl -L -o /tmp/mecab-jieba-0.1.1.tar.gz "https://lindera.dev/mecab-jieba-0.1.1.tar.gz"
% tar zxvf /tmp/mecab-jieba-0.1.1.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-jieba-0.1.1/dict-src \
  --dest /tmp/lindera-jieba-0.1.1 \
  --metadata ./lindera-jieba/metadata.json

Build ko-dic (Korean dictionary)

% curl -L -o /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz "https://Lindera.dev/mecab-ko-dic-2.1.1-20180720.tar.gz"
% tar zxvf /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ko-dic-2.1.1-20180720 \
  --dest /tmp/lindera-ko-dic-2.1.1-20180720 \
  --metadata ./lindera-ko-dic/metadata.json

Build user dictionaries

Build IPADIC user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera IPADIC Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ipadic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ipadic/metadata.json \
  --user

Build UniDic user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera UniDic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/unidic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-unidic/metadata.json \
  --user

Build SudachiDict user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera SudachiDict Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/sudachidict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-sudachidict/metadata.json \
  --user

Build CC-CEDICT user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera CC-CEDICT Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/cc-cedict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-cc-cedict/metadata.json \
  --user

Build Jieba user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera Jieba Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/jieba_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-jieba/metadata.json \
  --user

Build ko-dic user dictionary (Korean)

For more details about user dictionary format please refer to the following URL:

• Lindera ko-dic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ko-dic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ko-dic/metadata.json \
  --user

download

Download the pre-built dictionary archive matching the CLI version from the GitHub releases page and install it under the OS-standard application data directory.

Download parameters

• Dictionary name (required): one of ipadic, ipadic-neologd, unidic, sudachidict, ko-dic, cc-cedict, jieba
• --force: Re-download and replace an existing dictionary

Storage location

Dictionaries are installed under a versioned layout <data dir>/dictionaries/<version>/lindera-<name>/. The default base directory per OS:

Set the LINDERA_DATA_DIR environment variable to override the base directory. This variable is unrelated to the build-time source cache variable LINDERA_BUILD_DICTIONARY_CACHE_DIR.

Download usage

% lindera download ipadic

Downloading https://github.com/lindera/lindera/releases/download/v5.1.0/lindera-ipadic-5.1.0.zip
100% (15.1/15.1 MiB)
Downloaded dictionary 'ipadic'
/home/user/.local/share/lindera/dictionaries/5.1.0/lindera-ipadic

The installed path is printed to stdout (progress and notices go to stderr), so it can be captured in scripts:

% DICT_DIR=$(lindera download ipadic)

After downloading, the dictionary can be referenced by name in tokenize:

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize --dict ipadic

Notes:

• The archive version always matches the CLI version; running a development build whose version has no published release yet fails with HTTP 404.
• Integrity is verified via the HTTPS transport and the ZIP archive's built-in per-file CRC32 checksums.
• The largest dictionary (ipadic-neologd) is a ~305 MB download and requires roughly twice that amount of free disk space during extraction.

train

Train a new morphological analysis model from annotated corpus data. To use this feature, you must build with the train feature flag enabled. (The train feature flag is enabled by default.)

Train parameters

• --seed / -s: Seed lexicon file (CSV format) to be weighted
• --corpus / -c: Training corpus (annotated text)
• --char-def / -C: Character definition file (char.def)
• --unk-def / -u: Unknown word definition file (unk.def) to be weighted
• --feature-def / -f: Feature definition file (feature.def)
• --rewrite-def / -r: Rewrite rule definition file (rewrite.def)
• --output / -o: Output model file
• --lambda / -l: Regularization coefficient (0.0-1.0) (default: 0.01)
• --regularization / -R: Regularization type: l1, l2, or elasticnet (default: l1)
• --elastic-net-l1-ratio: L1 ratio for Elastic Net regularization (0.0-1.0), only used with --regularization elasticnet (default: 0.5)
• --max-iterations / -i: Maximum number of iterations for training (default: 100)
• --max-threads / -t: Maximum number of threads (defaults to CPU core count; affects only speed, never the trained model)

Basic workflow

1. Prepare training files

Seed lexicon file (seed.csv):

The seed lexicon file contains initial dictionary entries used for training the CRF model. Each line represents a word entry with comma-separated fields:

• Surface
• Left context ID
• Right context ID
• Word cost
• Part-of-speech tags (multiple fields)
• Base form
• Reading (katakana)
• Pronunciation

Note: The exact field definitions differ between dictionary formats (IPADIC, UniDic, ko-dic, CC-CEDICT). Please refer to each dictionary's format specification for details.

外国,0,0,0,名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人,0,0,0,名詞,接尾,一般,*,*,*,人,ジン,ジン

Training corpus (corpus.txt):

The training corpus file contains annotated text data used to train the CRF model. Each line consists of:

• A surface form (word) followed by a tab character
• Comma-separated morphological features (part-of-speech tags, base form, reading, pronunciation)
• Sentences are separated by "EOS" (End Of Sentence) markers

外国	名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人	名詞,接尾,一般,*,*,*,人,ジン,ジン
参政	名詞,サ変接続,*,*,*,*,参政,サンセイ,サンセイ
権	名詞,接尾,一般,*,*,*,権,ケン,ケン
EOS

For detailed information about file formats and advanced features, see Training Pipeline.

2. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

3. Training results

The trained model will contain:

• Existing words: All seed dictionary records with newly learned weights
• New words: Words from the corpus not in the seed dictionary, added with appropriate weights

export

Export a trained model file to Lindera dictionary format files. This feature requires building with the train feature flag enabled.

Export parameters

• --model / -m: Path to the trained model file (.dat format)
• --output / -o: Directory to output the dictionary files
• --metadata: Optional metadata.json file to update with trained model information
• --cost-factor: Override cost factor for weight-to-cost conversion (default: value from trained model, typically 700)

Output files

The export command creates the following dictionary files in the output directory:

• lex.csv: Lexicon file with learned weights (MeCab-compatible cost via tocost())
• matrix.def: Dense connection cost matrix covering all (right_id, left_id) pairs
• unk.def: Unknown word definitions
• char.def: Character type definitions
• feature.def: Feature template definitions (copied from trained model)
• rewrite.def: Feature rewrite rules (copied from trained model)
• left-id.def: Left context ID to feature string mapping
• right-id.def: Right context ID to feature string mapping
• metadata.json: Updated metadata file (if --metadata option is provided)

Complete workflow example

1. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

2. Export to dictionary format

lindera export \
  --model /tmp/lindera/training/model.dat \
  --metadata ./resources/training/metadata.json \
  --output /tmp/lindera/training/dictionary

3. Build dictionary

lindera build \
  --src /tmp/lindera/training/dictionary \
  --dest /tmp/lindera/training/compiled_dictionary \
  --metadata /tmp/lindera/training/dictionary/metadata.json

4. Use trained dictionary

echo "これは外国人参政権です。" | lindera tokenize \
  -d /tmp/lindera/training/compiled_dictionary

Metadata update feature

When the --metadata option is provided, the export command will:

1. Read the base metadata.json file to preserve existing configuration
2. Update specific fields with values from the trained model:
   • default_word_cost: Calculated from feature weight median
   • model_info: Training statistics including feature count, label count, matrix size, iterations, regularization, and version. No timestamp is written, so exporting the same model twice produces identical output
3. Preserve existing settings such as dictionary name, character encoding, schema definitions, and other user-defined configuration

Tutorial

This tutorial walks you through the basic usage of the Lindera CLI, from installation to advanced text processing.

1. Install the CLI

Install Lindera CLI:

% cargo install lindera-cli

Verify the installation:

% lindera --help

2. Download a dictionary

Download the pre-built IPADIC dictionary from the GitHub releases page. It is installed under the OS-standard application data directory:

% lindera download ipadic

See Commands for the available dictionary names and storage locations.

3. Basic tokenization

Tokenize Japanese text using the downloaded IPADIC dictionary, referencing it by name:

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic

Expected output:

東京    名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
日本    名詞,固有名詞,地域,国,*,*,日本,ニホン,ニホン
の      助詞,連体化,*,*,*,*,の,ノ,ノ
首都    名詞,一般,*,*,*,*,首都,シュト,シュト
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。      記号,句点,*,*,*,*,。,。,。
EOS

4. Try different output formats

Wakati format (word segmentation only)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic \
  --output wakati

Expected output:

東京 は 日本 の 首都 です 。

JSON format (detailed information)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic \
  --output json

This produces a JSON array with detailed token information including byte offsets, part-of-speech tags, readings, and more.

5. Use decompose mode

Decompose mode splits compound nouns into their constituent parts:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict ipadic \
  --mode decompose

Expected output:

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Compare with normal mode, where "関西国際空港" remains as a single token.

6. Apply character and token filters

Use Unicode normalization and keep only common nouns:

% echo "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。" | lindera tokenize \
  --dict ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般","名詞,サ変接続","名詞,固有名詞,組織"]}'

Expected output:

Lindera 名詞,固有名詞,組織,*,*,*,*,*,*
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
エンジン        名詞,一般,*,*,*,*,エンジン,エンジン,エンジン
EOS

The Unicode normalization converts full-width characters to half-width, and the token filter keeps only tokens matching the specified part-of-speech tags.

You can also combine multiple filters:

% echo "すもももももももものうち" | lindera tokenize \
  --dict ipadic \
  --token-filter 'japanese_stop_tags:{"tags":["助詞,格助詞,一般","助詞,係助詞","助詞,連体化","助動詞"]}'

7. Use user dictionary

Create a CSV file with custom word entries (e.g., my_dict.csv):

東京スカイツリー,カスタム名詞,トウキョウスカイツリー

Tokenize with the user dictionary:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict ipadic \
  --user-dict ./my_dict.csv

Without the user dictionary, "東京スカイツリー" would be split into multiple tokens. With the user dictionary, it is recognized as a single token.

For pre-built user dictionary examples, see:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

Expected output:

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Lindera Python

Lindera Python provides Python bindings for the Lindera morphological analysis engine, built with PyO3. It brings Lindera's high-performance tokenization capabilities to the Python ecosystem with support for Python 3.10 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from PyPI

Pre-built wheels are available on PyPI:

pip install lindera

The wheels cover CPython 3.10 to 3.14 on Linux (x86_64 and aarch64, glibc), macOS (x86_64 and arm64), and Windows (x64 and arm64). On any other platform, such as musl-based Linux (Alpine), other CPU architectures, or free-threaded CPython, pip install lindera builds the package from the source distribution instead. That build needs the Rust toolchain 1.88 or later (install it via rustup) and a C linker such as cc, and takes several minutes.

[!NOTE] The PyPI package does not include dictionaries. See Obtaining Dictionaries below.

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Building from Source

If you need to build from source (e.g., to enable specific feature flags), the following prerequisites are required:

• Python 3.10 or later (up to 3.14)
• Rust toolchain -- Install via rustup
• maturin -- Python package for building Rust-based Python extensions

Install maturin with pip:

pip install maturin

Development Build

Build and install lindera-python in development mode:

cd lindera-python
maturin develop

Or use the project Makefile:

make python-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

maturin develop --features train

Feature Flags

Multiple features can be combined:

maturin develop --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Python:

import lindera

print(lindera.version())

Quick Start

This guide shows how to tokenize text using lindera-python.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

from lindera import TokenizerBuilder

builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("/path/to/ipadic")
tokenizer = builder.build()

tokens = tokenizer.tokenize("関西国際空港限定トートバッグ")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("/path/to/ipadic")
    .build()
)

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    print(f"{token.surface}\t{token.get_detail(0)}")

Accessing Token Properties

Each token exposes the following properties:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
tokens = tokenizer.tokenize("東京タワー")

for token in tokens:
    print(f"Surface: {token.surface}")
    print(f"Byte range: {token.byte_start}..{token.byte_end}")
    print(f"Position: {token.position}")
    print(f"Word ID: {token.word_id}")
    print(f"Unknown: {token.is_unknown}")
    print(f"Details: {token.details}")
    print()

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)

for tokens, cost in results:
    surfaces = [t.surface for t in tokens]
    print(f"Cost {cost}: {' / '.join(surfaces)}")

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

TokenizerBuilder()

Creates a new builder with default configuration.

from lindera import TokenizerBuilder

builder = TokenizerBuilder()

TokenizerBuilder().from_file(file_path)

Loads configuration from a YAML file and returns a new builder. See lindera-python/resources/lindera.yml for a complete example covering segmenter, character_filters, and token_filters.

builder = TokenizerBuilder().from_file("lindera.yml")

Configuration Methods

All setter methods return self for method chaining.

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode("normal")

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary("embedded://ipadic")

# Use an external dictionary
builder.set_dictionary("/path/to/dictionary")

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary("/path/to/user_dictionary")

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(True)

set_space_penalty(value)

Sets the left-space penalty for Korean: a candidate that starts right after whitespace and whose part-of-speech tag normally attaches to the preceding word (particles, endings, ...) gets a cost added. value has the same meaning as segmenter.space_penalty in a configuration file:

• None -- The default: the rules the dictionary ships in its metadata.json, if any (ko-dic ships mecab-ko-dic's rules; the other bundled dictionaries ship none). Use it to undo an earlier call.
• False -- Turns the penalty off.
• True -- Requires the dictionary's rules; build() raises ValueError for a dictionary that ships none.
• A dict of the form {"rules": [{"pos": [...], "cost": n}, ...]} -- Applies these rules instead. A candidate whose first part-of-speech tag is listed in pos gets cost added; the first matching rule wins.

Any other value raises ValueError (a value with no JSON equivalent, such as an arbitrary object, raises TypeError, as in append_token_filter).

# Turn the penalty off
builder.set_space_penalty(False)

# Explicit rules
builder.set_space_penalty({
    "rules": [
        {"pos": ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], "cost": 3000},
        {"pos": ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], "cost": 6000},
    ]
})

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, as mecab-ko does. For example, 서울 시 에서 now reads 시 as the noun NNG rather than the ending EP. Call builder.set_space_penalty(False) to turn it off. Only a ko-dic built by Lindera 6.1.0 or later ships the rules; with a ko-dic from the v6.0.0 release the penalty stays off and True fails until the dictionary is rebuilt. See Segmenter for details.

append_character_filter(kind, args=None)

Appends a character filter to the preprocessing pipeline.

builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

append_token_filter(kind, args=None)

Appends a token filter to the postprocessing pipeline.

builder.append_token_filter("lowercase", {})

The arguments of set_space_penalty, append_character_filter and append_token_filter may nest lists and dicts at most 128 levels deep. A deeper value, or one that contains itself, raises ValueError.

Build

build()

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build()

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Tokenizer(dictionary, mode="normal", user_dictionary=None, space_penalty=None)

Creates a tokenizer directly from a loaded dictionary.

space_penalty takes the same values as TokenizerBuilder.set_space_penalty. None (the default) uses the left-space penalty rules the dictionary ships, so with ko-dic the penalty is on; pass False to turn it off. An invalid setting raises the same errors as set_space_penalty and build().

from lindera import Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode="normal")

# ko-dic without the left-space penalty
ko_tokenizer = Tokenizer(load_dictionary("embedded://ko-dic"), space_penalty=False)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns a list of Token objects.

tokens = tokenizer.tokenize("形態素解析")

Parameters:

Returns: list[Token]

tokenize_surfaces(text)

Tokenizes the input text and returns only the token surfaces, as a list of strings. This is the fast path for wakati-style use: no Token objects are created and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals [t.surface for t in tokenizer.tokenize(text)].

surfaces = tokenizer.tokenize_surfaces("形態素解析")
# ["形態素", "解析"]

Parameters:

Returns: list[str]

tokenize_nbest(text, n, unique=False, cost_threshold=None)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)
for tokens, cost in results:
    print(cost, [t.surface for t in tokens])

Parameters:

Returns: list[tuple[list[Token], int]]

Mode

Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder.set_mode() and the Tokenizer constructor currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Mode(mode_str=None)

Creates a Mode. Accepts "normal" / "Normal" (the default when omitted) or "decompose" / "Decompose"; any other value raises ValueError.

from lindera import Mode

mode = Mode("normal")
mode = Mode("decompose")
mode = Mode()  # defaults to "normal"

Methods

mode = Mode("decompose")
str(mode)            # "decompose"
repr(mode)           # "Mode.Decompose"
mode.is_normal()      # False
mode.is_decompose()   # True

Penalty

Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Penalty(kanji_penalty_length_threshold=2, kanji_penalty_length_penalty=3000, other_penalty_length_threshold=7, other_penalty_length_penalty=1700)

All arguments are optional and default to the values shown above.

from lindera import Penalty

penalty = Penalty(
    kanji_penalty_length_threshold=2,
    kanji_penalty_length_penalty=3000,
    other_penalty_length_threshold=7,
    other_penalty_length_penalty=1700,
)

Penalty Properties

All four fields support both getting and setting:

penalty.kanji_penalty_length_threshold = 3
print(penalty.kanji_penalty_length_threshold)  # 3

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode() and the Tokenizer constructor only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Token Methods

get_detail(index)

Returns the detail string at the specified index, or None if the index is out of range.

token = tokenizer.tokenize("東京")[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: str or None

to_dict()

Returns the token as a plain dict, keeping each field's natural Python type, so it serializes without a custom encoder.

import json

data = tokenizer.tokenize("東京")[0].to_dict()
# {'surface': '東京', 'byte_start': 0, 'byte_end': 6, 'position': 0,
#  'word_id': 12345, 'is_unknown': False, 'details': [...]}

json.dumps([token.to_dict() for token in tokens])

Returns: dict with the keys surface, byte_start, byte_end, position, word_id, is_unknown, and details.

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Error Handling

Lindera Python functions raise standard Python exceptions rather than a custom exception type:

• IOError (an alias of OSError) -- for I/O-related failures, such as a missing or unreadable file
• ValueError -- for everything else, such as invalid configuration, parse errors, or tokenization failures

from lindera import load_dictionary

try:
    dictionary = load_dictionary("/path/that/does/not/exist")
except ValueError as e:
    print(f"Failed to load dictionary: {e}")

A LinderaError class is also registered as lindera.LinderaError, but no function in this crate currently raises it -- it can only be constructed and raised manually. Catch IOError/ValueError (or the general Exception) when handling errors from this library, not LinderaError.

Dictionary Management

Lindera Python provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

from lindera import load_dictionary

dictionary = load_dictionary("/path/to/ipadic")

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = load_dictionary("embedded://ipadic")

A loaded Dictionary also exposes its own metadata:

print(dictionary.metadata_name())      # e.g. "ipadic"
print(dictionary.metadata_encoding())  # e.g. "UTF-8"
metadata = dictionary.metadata()       # the full Metadata object

This is useful for loading a user dictionary that must share the same metadata (schema, encoding, etc.) as the system dictionary it augments, as in lindera-python/examples/tokenize_with_userdict.py:

from lindera import Tokenizer, load_dictionary, load_user_dictionary

dictionary = load_dictionary("embedded://ipadic")
metadata = dictionary.metadata()
user_dictionary = load_user_dictionary("/path/to/user_dictionary.csv", metadata)

tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dictionary)

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

from lindera import load_user_dictionary, Metadata

metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

Pass the user dictionary when building a tokenizer:

from lindera import Tokenizer, load_dictionary, load_user_dictionary, Metadata

dictionary = load_dictionary("/path/to/ipadic")
metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dict)

Or via the builder:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("/path/to/ipadic")
    .set_user_dictionary("/path/to/user_dictionary")
    .build()
)

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

from lindera import build_dictionary, Metadata

metadata = Metadata(name="custom", encoding="UTF-8")
build_dictionary("/path/to/input_dir", "/path/to/output_dir", metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

Metadata loaded with Metadata.from_json_file keeps the space_penalty rules of the metadata.json, and the built dictionary ships them. For ko-dic this means the left-space penalty stays on by default in the dictionary you build; turn it off per tokenizer with TokenizerBuilder.set_space_penalty(False) (see Tokenizer API). A Metadata created with the constructor carries no rules.

User Dictionary

Build a user dictionary from a CSV file:

from lindera import build_user_dictionary, Metadata

metadata = Metadata()
build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir")

Note: the first argument ("ipadic" above) is currently unused and reserved for future use -- it does not select or configure the build in any way. The build behavior is controlled entirely by metadata (in particular metadata.user_dictionary_schema). Any string may be passed today.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

from lindera import Metadata

# Default metadata
metadata = Metadata()

# Custom metadata
metadata = Metadata(
    name="my_dictionary",
    encoding="UTF-8",
    default_word_cost=-10000,
)

Loading from JSON

metadata = Metadata.from_json_file("metadata.json")

Properties

All properties support both getting and setting:

metadata = Metadata()
metadata.name = "custom_dict"
metadata.encoding = "EUC-JP"
print(metadata.name)  # "custom_dict"

to_dict()

Returns the metadata as a dict of strings: the properties above, with dictionary_schema_fields and user_dictionary_schema_fields (comma-separated field names) in place of the two schemas.

metadata = Metadata(name="test")
print(metadata.to_dict())

When the metadata carries left-space penalty rules, as ko-dic's does, the dict also has a space_penalty entry holding them as a JSON string; the key is absent otherwise:

metadata = Metadata.from_json_file("/path/to/ko-dic/metadata.json")
metadata.to_dict()["space_penalty"]
# '{"rules":[{"pos":["EC","EF",...],"cost":3000},{"pos":["JC","JKB",...],"cost":6000}]}'

Schema

Schema, FieldDefinition, and FieldType describe the field layout of a dictionary entry. A schema is used by Metadata.dictionary_schema and Metadata.user_dictionary_schema (see the table above).

FieldType

FieldType enumerates the category of a single field:

• FieldType.Surface -- surface form (word text)
• FieldType.LeftContextId -- left context ID
• FieldType.RightContextId -- right context ID
• FieldType.Cost -- word cost
• FieldType.Custom -- any other, dictionary-specific field

FieldDefinition

FieldDefinition describes a single field within a schema.

FieldDefinition(index, name, field_type, description=None)

from lindera import FieldDefinition, FieldType

field = FieldDefinition(0, "surface", FieldType.Surface, "Surface form")

Properties (read-only):

Creating a Schema

Schema holds an ordered list of field names and provides lookups between field name and index.

Schema(fields)

Creates a schema from a list of field names.

from lindera import Schema

schema = Schema([
    "surface",
    "left_context_id",
    "right_context_id",
    "cost",
    "major_pos",
    "reading",
])

Schema.create_default()

A static method that returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Schema.create_default()

Schema Methods and Properties

schema = Schema.create_default()

schema.field_count()               # 13
schema.get_field_index("cost")     # 3
schema.get_field_name(0)           # "surface"
schema.get_custom_fields()         # ["major_pos", "pos_detail_1", ..., "pronunciation"]
len(schema)                        # 13

field = schema.get_field_by_name("surface")
print(field.index, field.name, field.field_type)  # 0 surface FieldType.Surface

schema.validate_record([
    "東京", "1288", "1288", "100",
    "名詞", "固有名詞", "地域", "一般", "*", "*",
    "東京", "トウキョウ", "トーキョー",
])

Text Processing Pipeline

Lindera Python supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .build()
)

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("mapping", {
        "mapping": {
            "\u30fc": "-",
            "\uff5e": "~",
        }
    })
    .build()
)

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    .build()
)

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("lowercase", {})
    .build()
)

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_base_form", {})
    .build()
)

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞"],
    })
    .build()
)

Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*).

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed. The following keeps only general nouns:

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_keep_tags", {
        "tags": ["名詞,一般"],
    })
    .build()
)

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("embedded://ipadic")
    # Preprocessing
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    # Postprocessing
    .append_token_filter("japanese_base_form", {})
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞", "記号,句点", "記号,読点"],
    })
    .append_token_filter("lowercase", {})
    .build()
)

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and punctuation
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Python supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-python with the train feature enabled (enabled by default):

maturin develop --features train

Training a Model

Use lindera.train() to train a CRF model from a seed lexicon and annotated corpus:

import lindera

lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
    max_threads=4,
)

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using lindera.export():

import lindera

lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

import lindera

# Step 1: Train the CRF model
lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
)

# Step 2: Export to dictionary source files
lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

# Step 3: Build the dictionary from exported source files
metadata = lindera.Metadata.from_json_file("/tmp/dictionary_source/metadata.json")
lindera.build_dictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata)

# Step 4: Use the trained dictionary
tokenizer = (
    lindera.TokenizerBuilder()
    .set_dictionary("/tmp/dictionary")
    .set_mode("normal")
    .build()
)

tokens = tokenizer.tokenize("形態素解析のテスト")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Lindera Node.js

Lindera Node.js provides Node.js bindings for the Lindera morphological analysis engine, built with NAPI-RS. It brings Lindera's high-performance tokenization capabilities to the Node.js ecosystem with support for Node.js 18 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary
• TypeScript support: Full type definitions included out of the box

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from npm

Pre-built packages will be available on npm:

npm install lindera

[!NOTE] The npm package does not include dictionaries. See Obtaining Dictionaries below. For browser/WASM usage, see lindera-wasm.

Building from Source

Prerequisites

• Node.js 18 or later (LTS versions recommended)
• Rust toolchain -- Install via rustup
• NAPI-RS CLI -- CLI tool for building native Node.js addons in Rust

Install the NAPI-RS CLI globally:

npm install -g @napi-rs/cli

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build lindera-nodejs in development mode:

cd lindera-nodejs
npm install
npm run build

Or use the project Makefile:

make nodejs-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

npm run build -- --features train

Feature Flags

Multiple features can be combined:

npm run build -- --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

const dictionary = loadDictionary("embedded://ipadic");

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Node.js:

const lindera = require("lindera");

console.log(lindera.version());

[!NOTE] The lindera npm package's exports map declares types, import, and require conditions, so the package loads from CommonJS (const { TokenizerBuilder } = require("lindera")) and ES modules (import { TokenizerBuilder } from "lindera") alike:

import { TokenizerBuilder } from "lindera";

console.log(typeof TokenizerBuilder); // "function"

Quick Start

This guide shows how to tokenize text using lindera-nodejs.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setMode("normal");
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Method Chaining

Every setter returns the builder itself (this), so the configuration can be chained:

const { TokenizerBuilder } = require("lindera");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("すもももももももものうち");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details[0]}`);
}

Calling the setters one statement at a time works just as well — both styles configure the same builder.

Accessing Token Properties

Each token exposes the following properties:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("東京タワー");
for (const token of tokens) {
  console.log(`Surface: ${token.surface}`);
  console.log(`Byte range: ${token.byteStart}..${token.byteEnd}`);
  console.log(`Position: ${token.position}`);
  console.log(`Word ID: ${token.wordId}`);
  console.log(`Unknown: ${token.isUnknown}`);
  console.log(`Details: ${token.details}`);
  console.log();
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  const surfaces = tokens.map((t) => t.surface);
  console.log(`Cost ${cost}: ${surfaces.join(" / ")}`);
}

TypeScript

Lindera Node.js includes TypeScript type definitions, so all classes and functions are fully typed. The package's exports map serves both require and import, and the following sample compiles under moduleResolution: node16 with strict enabled:

import type { Token } from "lindera";
import { TokenizerBuilder } from "lindera";

const builder = new TokenizerBuilder();
builder.setMode("normal");
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens: Token[] = tokenizer.tokenize("形態素解析");
for (const token of tokens) {
  console.log(`${token.surface}: ${token.details.join(",")}`);
}

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return the builder itself (this), so they can be chained or called one statement at a time.

setMode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.setMode("normal");

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.setKeepWhitespace(true);

setSpacePenalty(value)

Sets the left-space penalty for Korean (mecab-ko's left-space-penalty-factor): a candidate that starts right after whitespace and whose first part-of-speech tag is listed in a rule gets that rule's cost added. The value means the same as segmenter.space_penalty in a configuration file.

Any other value throws an Error.

// Turn the penalty off
builder.setSpacePenalty(false);

// Explicit rules (these are the ones ko-dic ships)
builder.setSpacePenalty({
  rules: [
    { pos: ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], cost: 3000 },
    { pos: ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], cost: 6000 },
  ],
});

// Back to the dictionary's default
builder.setSpacePenalty(null);

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, because its metadata.json ships mecab-ko-dic's rules. This changes the Korean output compared with v6.0 (for example, 시 in 서울 시 에서 becomes the noun NNG instead of the ending EP). Call setSpacePenalty(false) to turn it off. For a Tokenizer created directly, pass false as the fourth argument: new Tokenizer(dictionary, "normal", null, false). A ko-dic downloaded from the v6.0.0 release ships no rules, so with it the penalty stays off and setSpacePenalty(true) makes build() throw. See Segmenter for details.

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

The arguments of setSpacePenalty, appendCharacterFilter and appendTokenFilter may nest arrays and objects at most 128 levels deep. A deeper value, or one that contains itself, throws an Error.

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?, spacePenalty?)

Creates a tokenizer directly from a loaded dictionary.

spacePenalty takes the same values as setSpacePenalty. null or undefined (the default) uses the left-space penalty rules the dictionary ships. An invalid setting throws, as setSpacePenalty and build() do.

const { Tokenizer, loadDictionary } = require("lindera");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

// ko-dic without the left-space penalty
const koTokenizer = new Tokenizer(loadDictionary("embedded://ko-dic"), "normal", null, false);

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of plain token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

Tokens are plain JavaScript objects, not class instances, so they are reclaimed by ordinary garbage collection and pass through JSON.stringify, structuredClone, and worker transfer without conversion. See Token.

tokenizeSurfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no token objects are built and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map((t) => t.surface).

const surfaces = tokenizer.tokenizeSurfaces("形態素解析");
// ["形態素", "解析"]

Parameters:

Returns: string[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: NbestResult[], where each NbestResult is { tokens: Token[], cost: number }

Token

Token is a plain object describing a single morphological token. It is a TypeScript interface, not a class: it has no methods and no prototype, and every field is read directly.

Properties

Reading Details

Index details directly. Out-of-range indexes yield undefined.

const token = tokenizer.tokenize("東京")[0];
const pos = token.details[0];      // e.g., "名詞"
const subpos = token.details[1];   // e.g., "固有名詞"
const reading = token.details[7];  // e.g., "トウキョウ"

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

Mode and Penalty are exported from the lindera npm package, but they are not currently wired into any public API: TokenizerBuilder.setMode() / Tokenizer's constructor accept only a plain mode string ("normal" or "decompose"), and decompose mode always uses the default penalty configuration internally. These types are documented here for completeness; they cannot yet be used to customize penalty behavior from JavaScript.

Mode

A string enum with two values:

• Mode.Normal -- standard dictionary-cost-based tokenization
• Mode.Decompose -- penalty-based decomposition of compound words

Penalty

An object shape describing the penalty parameters used by decompose mode:

Dictionary Management

Lindera Node.js provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use loadDictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

const { loadDictionary } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

const dictionary = loadDictionary("embedded://ipadic");

Dictionary exposes a few read-only accessors for inspecting the loaded dictionary's metadata:

console.log(dictionary.metadataName());     // e.g. "ipadic"
console.log(dictionary.metadataEncoding()); // e.g. "UTF-8"

const metadata = dictionary.metadata(); // full Metadata object
console.log(metadata.defaultWordCost);

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

const { loadUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

Pass the user dictionary when building a tokenizer:

const { Tokenizer, loadDictionary, loadUserDictionary, Metadata } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

const tokenizer = new Tokenizer(dictionary, "normal", userDict);

Or via the builder:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
builder.setUserDictionary("/path/to/user_dictionary");
const tokenizer = builder.build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

const { buildDictionary, Metadata } = require("lindera");

const metadata = new Metadata({ name: "custom", encoding: "UTF-8" });
buildDictionary("/path/to/input_dir", "/path/to/output_dir", metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

To build a dictionary with the settings its metadata.json ships, load the file with Metadata.fromJsonFile() and pass the result. The file's space_penalty rules are kept, so a ko-dic built this way applies the left-space penalty by default (see setSpacePenalty). A Metadata created with new Metadata(...) carries no rules.

const metadata = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
buildDictionary("/path/to/mecab-ko-dic", "/path/to/output_dir", metadata);

User Dictionary

Build a user dictionary from a CSV file:

const { buildUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata);

The metadata parameter is optional. When omitted, default metadata values are used:

buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir");

[!NOTE] The first argument (kind) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

const { Metadata } = require("lindera");

// Default metadata
const metadata = new Metadata();

// Custom metadata
const metadata = new Metadata({
  name: "my_dictionary",
  encoding: "UTF-8",
  defaultWordCost: -10000,
});

Loading from JSON

const metadata = Metadata.fromJsonFile("metadata.json");

Properties

[!NOTE] Schema information (dictionary/user-dictionary field layout) is not exposed on this binding's Metadata object -- there are no dictionarySchema / userDictionarySchema properties. Use the standalone Schema class instead.

All properties support both getting and setting:

const metadata = new Metadata();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

toObject()

Returns a plain object representation of the metadata, with every value as a string:

const metadata = new Metadata({ name: "test" });
console.log(metadata.toObject());

When the metadata carries left-space penalty rules (for example, when it was loaded from ko-dic's metadata.json), the object also has a spacePenalty entry holding the rules as a JSON string. Otherwise the key is absent.

const koDic = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
const { rules } = JSON.parse(koDic.toObject().spacePenalty);
console.log(rules[1].pos); // ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"]

Schema

The Schema class defines the field structure of dictionary entries.

Creating a Schema

const { Schema } = require("lindera");

// Default IPADIC-compatible schema
const schema = Schema.createDefault();

// Custom schema
const custom = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Methods

const schema = Schema.createDefault();

console.log(schema.fieldCount());           // 13 (IPADIC format)
console.log(schema.getFieldIndex("pos1"));  // e.g., 4
console.log(schema.getAllFields());          // ["surface", "left_id", ...]
console.log(schema.getCustomFields());      // Fields from index 4 onward

FieldDefinition

FieldType

Text Processing Pipeline

Lindera Node.js supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. See Filters for the full, authoritative catalogue of every character and token filter that lindera-analysis ships, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
const tokenizer = builder.build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("mapping", {
  mapping: {
    "\u30fc": "-",
    "\uff5e": "~",
  },
});
const tokenizer = builder.build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("japanese_iteration_mark", {
  normalize_kanji: true,
  normalize_kana: true,
});
const tokenizer = builder.build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("lowercase", {});
const tokenizer = builder.build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_base_form", {});
const tokenizer = builder.build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_stop_tags", {
  tags: ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞"],
});
const tokenizer = builder.build();

[!NOTE] Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*).

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed. The following keeps only general nouns:

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_keep_tags", {
  tags: ["名詞,一般"],
});
const tokenizer = builder.build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setMode("normal");
builder.setDictionary("embedded://ipadic");
// Preprocessing
builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
builder.appendCharacterFilter("japanese_iteration_mark", {
  normalize_kanji: true,
  normalize_kana: true,
});
// Postprocessing
builder.appendTokenFilter("japanese_base_form", {});
builder.appendTokenFilter("japanese_stop_tags", {
  tags: ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞", "記号,句点", "記号,読点"],
});
builder.appendTokenFilter("lowercase", {});
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and punctuation
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Node.js supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-nodejs with the train feature enabled (enabled by default):

npm run build -- --features train

Training a Model

[!NOTE] The file paths below (resources/training/*.csv, *.def) are illustrative placeholders — this repository does not ship those exact files. For a complete, runnable example that generates a seed lexicon, corpus, and definition files on the fly and then trains/exports/builds a real dictionary, see lindera-nodejs/examples/train_and_export.js.

Use train() to train a CRF model from a seed lexicon and annotated corpus:

const { train } = require("lindera");

train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
  maxThreads: 4,
});

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using exportModel():

const { exportModel } = require("lindera");

exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

const {
  train,
  exportModel,
  buildDictionary,
  Metadata,
  TokenizerBuilder,
} = require("lindera");

// Step 1: Train the CRF model
train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
});

// Step 2: Export to dictionary source files
exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

// Step 3: Build the dictionary from exported source files
const metadata = Metadata.fromJsonFile("/tmp/dictionary_source/metadata.json");
buildDictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata);

// Step 4: Use the trained dictionary
const builder = new TokenizerBuilder();
builder.setDictionary("/tmp/dictionary");
builder.setMode("normal");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("形態素解析のテスト");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Lindera Ruby

Lindera Ruby provides Ruby bindings for the Lindera morphological analysis engine, built with Magnus and rb-sys. It brings Lindera's high-performance tokenization capabilities to the Ruby ecosystem with support for Ruby 3.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

From v6.1.0, lindera-ruby installs from RubyGems as lindera. It is a source gem: gem install compiles the native extension from the bundled Rust source, which takes a few minutes.

Prerequisites

• Ruby 3.1 or later, with the tools needed to build native gems (a C compiler and make)
• Rust toolchain 1.88 or later -- Install via rustup
• libclang -- libclang-dev on Debian/Ubuntu; the Xcode command line tools on macOS
• Bundler -- only to build from source (gem install bundler)

Install from RubyGems

gem install lindera

Or add it to your Gemfile:

gem "lindera"

[!NOTE] Use lindera v6.1.0 or later. The lindera gems published before v6.1.0 fail to compile during installation; for those versions, build from source as described below.

The gem compiles against the crates.io releases of lindera and lindera-binding of the same version, which it pins exactly. It ships no Cargo.lock, so their dependencies, including the other lindera-* crates, resolve to the latest compatible releases at install time. It builds with the default features (train) and no embedded dictionary; see Obtaining Dictionaries. To enable other feature flags, set LINDERA_FEATURES when installing:

LINDERA_FEATURES="embed-ipadic" gem install lindera

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build and install lindera-ruby in development mode:

cd lindera-ruby
bundle install
bundle exec rake compile

Or use the project Makefile:

make build-lindera-ruby

Run make test-lindera-ruby to run both the Rust unit tests and the Ruby minitest suite.

Building the Gem

bundle exec rake build (or make package-lindera-ruby) writes the source gem to lindera-ruby/pkg/. It packages the crate as cargo package normalizes it, so the gem compiles outside this repository, against the crates.io releases of lindera and lindera-binding of the same version. Do not run gem build directly: the crate's own Cargo.toml only resolves inside the Cargo workspace.

make test-lindera-ruby-gem (requires Docker) builds the gem, installs it with gem install in a clean ruby container and tokenizes text with it. The test compiles the lindera crates from the checkout, so it also works for versions not yet released on crates.io.

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

LINDERA_FEATURES="train" bundle exec rake compile

Feature Flags

Features are specified through the LINDERA_FEATURES environment variable as a comma-separated list, both for bundle exec rake compile and for gem install.

Multiple features can be combined:

LINDERA_FEATURES="train,embed-ipadic,embed-ko-dic" bundle exec rake compile

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = Lindera.load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Ruby:

require 'lindera'

puts Lindera.version

Quick Start

This guide shows how to tokenize text using lindera-ruby.

Basic Tokenization

The recommended way to create a tokenizer is through Lindera::TokenizerBuilder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('関西国際空港限定トートバッグ')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Sequential Configuration

TokenizerBuilder is configured through sequential method calls:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('すもももももももものうち')
tokens.each do |token|
  puts "#{token.surface}\t#{token.get_detail(0)}"
end

Accessing Token Properties

Each token exposes the following properties:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('東京タワー')

tokens.each do |token|
  puts "Surface: #{token.surface}"
  puts "Byte range: #{token.byte_start}..#{token.byte_end}"
  puts "Position: #{token.position}"
  puts "Word ID: #{token.word_id}"
  puts "Unknown: #{token.unknown?}"
  puts "Details: #{token.details}"
  puts
end

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)

results.each do |tokens, cost|
  surfaces = tokens.map(&:surface)
  puts "Cost #{cost}: #{surfaces.join(' / ')}"
end

Tokenizer API

TokenizerBuilder

Lindera::TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

Lindera::TokenizerBuilder.new

Creates a new builder with default configuration.

require 'lindera'

builder = Lindera::TokenizerBuilder.new

Lindera::TokenizerBuilder.from_file(file_path)

Loads configuration from a JSON file and returns a new builder. This is a class method, not chained off an existing instance.

builder = Lindera::TokenizerBuilder.from_file('config.json')

Configuration Methods

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode('normal')

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary('embedded://ipadic')

# Use an external dictionary
builder.set_dictionary('/path/to/dictionary')

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary('/path/to/user_dictionary')

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(true)

set_space_penalty(value)

Sets the left-space penalty for Korean: a candidate that starts right after whitespace and whose part-of-speech tag normally attaches to the preceding word (particles, endings, ...) gets a cost added. value has the same meaning as segmenter.space_penalty in a configuration file:

• nil -- The default: the rules the dictionary ships in its metadata.json, if any (ko-dic ships mecab-ko-dic's rules; the other bundled dictionaries ship none). Use it to undo an earlier call.
• false -- Turns the penalty off.
• true -- Requires the dictionary's rules; build raises RuntimeError for a dictionary that ships none.
• A hash with string keys, { 'rules' => [{ 'pos' => [...], 'cost' => n }, ...] } -- Applies these rules instead. A candidate whose first part-of-speech tag is listed in pos gets cost added; the first matching rule wins.

A value that is not one of these forms, such as 1, 'false' or { 'rules' => 'JKS' }, raises RuntimeError. A value that cannot be converted raises TypeError: a Symbol (also as a hash key or an element, so write %w[JKS] rather than %i[JKS]), a non-finite Float (NaN or Infinity), an Integer outside the 64-bit range, or an object other than nil, true, false, an Integer, a Float, a String, an Array or a Hash.

# Turn the penalty off
builder.set_space_penalty(false)

# Explicit rules
builder.set_space_penalty({
  'rules' => [
    { 'pos' => %w[EC EF EP ETM ETN VCP XSA XSN XSV], 'cost' => 3000 },
    { 'pos' => %w[JC JKB JKC JKG JKO JKQ JKS JKV JX], 'cost' => 6000 }
  ]
})

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, as mecab-ko does. For example, 서울 시 에서 now reads 시 as the noun NNG rather than the ending EP. Call builder.set_space_penalty(false) to turn it off. A tokenizer created directly with Lindera::Tokenizer.new(dictionary, ...) always uses the default, so use TokenizerBuilder to turn it off. Only a ko-dic built by Lindera 6.1.0 or later ships the rules; with a ko-dic from the v6.0.0 release the penalty stays off and true fails until the dictionary is rebuilt. See Segmenter for details.

append_character_filter(kind, args)

Appends a character filter to the preprocessing pipeline. The args parameter is a hash with string keys.

builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })

append_token_filter(kind, args)

Appends a token filter to the postprocessing pipeline. The args parameter is a hash with string keys, or nil if the filter requires no arguments.

builder.append_token_filter('lowercase', nil)

The arguments of set_space_penalty, append_character_filter and append_token_filter may nest arrays and hashes at most 128 levels deep. A deeper value, or one that contains itself, raises ArgumentError.

Build

build

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build

Tokenizer

Lindera::Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Lindera::Tokenizer.new(dictionary, mode = nil, user_dictionary = nil, space_penalty: nil)

Creates a tokenizer directly from a loaded dictionary. mode defaults to 'normal', and user_dictionary is optional.

The space_penalty: keyword takes the same values as TokenizerBuilder#set_space_penalty. nil (the default) uses the left-space penalty rules the dictionary ships, so with ko-dic the penalty is on; pass false to turn it off. An invalid setting raises the same errors as set_space_penalty and build.

require 'lindera'

dictionary = Lindera.load_dictionary('embedded://ipadic')
tokenizer = Lindera::Tokenizer.new(dictionary)

# ko-dic without the left-space penalty
ko_tokenizer = Lindera::Tokenizer.new(Lindera.load_dictionary('embedded://ko-dic'), space_penalty: false)

With a user dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

tokens = tokenizer.tokenize('形態素解析')

Parameters:

Returns: Array<Token>

tokenize_surfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no Token objects are created and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map(&:surface).

surfaces = tokenizer.tokenize_surfaces('形態素解析')
# ["形態素", "解析"]

Parameters:

Returns: Array<String>

tokenize_nbest(text, n, unique, cost_threshold)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)
results.each do |tokens, cost|
  puts "#{cost}: #{tokens.map(&:surface).inspect}"
end

Parameters:

Returns: Array<Array(Array<Token>, Integer)>

Mode

Lindera::Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder#set_mode and Tokenizer.new currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Lindera::Mode.new(mode_str)

Creates a Mode. The argument is required, but may be nil. Accepts "normal" / "Normal" (used when mode_str is nil) or "decompose" / "Decompose"; any other value raises ArgumentError.

require 'lindera'

mode = Lindera::Mode.new('normal')
mode = Lindera::Mode.new('decompose')
mode = Lindera::Mode.new(nil)  # defaults to "normal"

Mode Methods

mode = Lindera::Mode.new('decompose')
mode.to_s        # "decompose"
mode.normal?      # false
mode.decompose?   # true

Penalty

Lindera::Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Lindera::Penalty.new(kanji_penalty_length_threshold, kanji_penalty_length_penalty, other_penalty_length_threshold, other_penalty_length_penalty)

All four positional arguments are required, but each may be nil to fall back to its default (shown below).

require 'lindera'

penalty = Lindera::Penalty.new(2, 3000, 7, 1700)
penalty = Lindera::Penalty.new(nil, nil, nil, nil)  # uses all defaults

Penalty Properties

All properties are read-only (there are no setter methods):

penalty = Lindera::Penalty.new(nil, nil, nil, nil)
penalty.kanji_penalty_length_threshold  # 2

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode and Tokenizer.new only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Additionally, the predicate method unknown? returns true if the word is not in the dictionary:

token.unknown?  # => false

Token Methods

get_detail(index)

Returns the detail string at the specified index, or nil if the index is out of range.

token = tokenizer.tokenize('東京')[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: String or nil

to_h / to_hash

Returns the token as a plain Hash keyed by Symbols, keeping each field's natural Ruby type, so it serializes without a custom encoder. to_h and to_hash are the same method.

require 'json'

data = tokenizer.tokenize('東京')[0].to_h
# {surface: "東京", byte_start: 0, byte_end: 6, position: 0,
#  word_id: 12345, is_unknown: false, details: [...]}

JSON.generate(tokens.map(&:to_h))

Returns: Hash with the keys :surface, :byte_start, :byte_end, :position, :word_id, :is_unknown, and :details.

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Schema

Lindera::Schema holds an ordered list of field names and provides lookups between field name and index. It is used by Metadata#dictionary_schema and Metadata#user_dictionary_schema (see Dictionary Management).

Creating a Schema

Lindera::Schema.new(fields)

Creates a schema from an array of field names.

require 'lindera'

schema = Lindera::Schema.new(%w[
  surface
  left_context_id
  right_context_id
  cost
  major_pos
  reading
])

Lindera::Schema.create_default

Returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Lindera::Schema.create_default

Schema Methods

schema = Lindera::Schema.create_default

schema.field_count                  # 13
schema.get_field_index('cost')      # 3
schema.get_field_name(0)            # "surface"
schema.get_custom_fields            # ["major_pos", "pos_detail_1", ..., "pronunciation"]

field = schema.get_field_by_name('surface')
puts "#{field.index} #{field.name} #{field.field_type}"  # 0 surface surface

schema.validate_record([
  '東京', '1288', '1288', '100',
  '名詞', '固有名詞', '地域', '一般', '*', '*',
  '東京', 'トウキョウ', 'トーキョー'
])

FieldDefinition

Lindera::FieldDefinition describes a single field within a Schema. Instances are only obtained from Schema#get_field_by_name -- there is no public constructor (Lindera::FieldDefinition.new raises TypeError).

FieldDefinition Properties

schema = Lindera::Schema.create_default
field = schema.get_field_by_name('surface')

field.index         # 0
field.name          # "surface"
field.field_type    # #<Lindera::FieldType: surface>
field.description    # nil (the default schema does not set descriptions)

FieldType

Lindera::FieldType enumerates the category of a single field. Like FieldDefinition, instances are only obtained from a Schema (via FieldDefinition#field_type) -- there is no public constructor.

to_s (and inspect) return one of:

• "surface" -- surface form (word text)
• "left_context_id" -- left context ID
• "right_context_id" -- right context ID
• "cost" -- word cost
• "custom" -- any other, dictionary-specific field

field = Lindera::Schema.create_default.get_field_by_name('surface')
field.field_type.to_s  # "surface"

Dictionary Management

Lindera Ruby provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera.load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

Pass the user dictionary when building a tokenizer:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Or via the builder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
builder.set_user_dictionary('/path/to/user_dictionary')
tokenizer = builder.build

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_dictionary('/path/to/input_dir', '/path/to/output_dir', metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

Metadata loaded with Lindera::Metadata.from_json_file keeps the space_penalty rules of the metadata.json, and the built dictionary ships them. For ko-dic this means the left-space penalty stays on by default in the dictionary you build; turn it off per tokenizer with TokenizerBuilder#set_space_penalty(false) (see Tokenizer API).

User Dictionary

Build a user dictionary from a CSV file:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', nil)

[!NOTE] The first argument (kind, 'ipadic' above) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today.

Metadata

The Lindera::Metadata class configures dictionary parameters.

Creating Metadata

require 'lindera'

# Create default metadata with standard settings
metadata = Lindera::Metadata.create_default

Lindera::Metadata.new takes all eight properties as required positional arguments (each may be nil to fall back to its default) -- use it only when you need to override specific values:

metadata = Lindera::Metadata.new(
  'my_dict', # name
  'UTF-8',   # encoding
  -10_000,   # default_word_cost
  1288,      # default_left_context_id
  1288,      # default_right_context_id
  '*',       # default_field_value
  false,     # flexible_csv
  false      # skip_invalid_cost_or_id
)

Loading from JSON

metadata = Lindera::Metadata.from_json_file('metadata.json')

Getting Metadata from a Dictionary

A loaded dictionary's metadata can be retrieved directly:

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata

Properties

to_h (and its alias to_hash) returns these properties as a Hash of strings, together with dictionary_schema_fields and user_dictionary_schema_fields (comma-separated field names). When the metadata carries left-space penalty rules, as ko-dic's does, the hash also has a space_penalty entry holding them as a JSON string:

metadata = Lindera::Metadata.from_json_file('/path/to/ko-dic/metadata.json')
metadata.to_h['space_penalty']
# '{"rules":[{"pos":["EC","EF",...],"cost":3000},{"pos":["JC","JKB",...],"cost":6000}]}'

Text Processing Pipeline

Lindera Ruby supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. See Filters for the full, authoritative catalogue of every character and token filter that lindera-analysis ships, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
tokenizer = builder.build

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('mapping', {
  'mapping' => {
    "\u30fc" => '-',
    "\uff5e" => '~'
  }
})
tokenizer = builder.build

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => true,
  'normalize_kana' => true
})
tokenizer = builder.build

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('lowercase', nil)
tokenizer = builder.build

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_base_form', nil)
tokenizer = builder.build

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞,格助詞,一般', '助詞,係助詞', '助詞,連体化', '助動詞']
})
tokenizer = builder.build

[!NOTE] Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*).

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed. The following keeps only general nouns:

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_keep_tags', {
  'tags' => ['名詞,一般']
})
tokenizer = builder.build

japanese_katakana_stem

Removes trailing prolonged sound marks from katakana tokens that exceed a minimum length.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_katakana_stem', { 'min' => 3 })
tokenizer = builder.build

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('embedded://ipadic')

# Preprocessing
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => true,
  'normalize_kana' => true
})

# Postprocessing
builder.append_token_filter('japanese_base_form', nil)
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞,格助詞,一般', '助詞,係助詞', '助詞,連体化', '助動詞', '記号,句点', '記号,読点']
})
builder.append_token_filter('lowercase', nil)

tokenizer = builder.build

tokens = tokenizer.tokenize('Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and punctuation
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Ruby supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-ruby with the train feature enabled:

LINDERA_FEATURES="embed-ipadic,train" bundle exec rake compile

Training a Model

Use Lindera.train to train a CRF model from a seed lexicon and annotated corpus:

require 'lindera'

Lindera.train(
  'resources/training/seed.csv',
  'resources/training/corpus.txt',
  'resources/training/char.def',
  'resources/training/unk.def',
  'resources/training/feature.def',
  'resources/training/rewrite.def',
  '/tmp/model.dat',
  0.01,  # lambda (L1 regularization)
  100,   # max_iter
  nil    # max_threads (nil = auto-detect CPU cores)
)

Training Parameters

Parameters are passed as positional arguments in the following order: