Lindera

A morphological analysis library in Rust. Lindera is forked from kuromoji-rs and aims to provide easy installation and concise APIs for tokenizing text in multiple languages.

Key Features

Tokenization Flow

graph LR
    subgraph Your Application
        T["Text"]
    end
    subgraph Lindera
        CF["Character Filters"]
        SEG["Segmenter\n(Dictionary + Viterbi)"]
        TF["Token Filters"]
    end
    T --> CF --> SEG --> TF --> R["Tokens"]

Document Map

Performance

Lindera tokenizes Japanese text (IPADIC) at roughly 10-20 MB/s single-threaded, depending on hardware and whether the Viterbi lattice is reused across calls. Run make bench to reproduce the benchmark suite (lindera/benches/) on your own machine; see #875 for detailed methodology and comparison numbers.

Quick Example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = tokenizer.tokenize(text)?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Run the example:

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

Output:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

License

Lindera is released under the MIT License.

Architecture

Lindera is organized as a Cargo workspace comprising multiple crates. Each crate has a focused responsibility, from low-level CRF computation to high-level CLI and language bindings.

Crate Dependency Graph

graph TB
    CRF["lindera-crf\n(CRF Engine)"]
    DICT["lindera-dictionary\n(Dictionary Base)"]
    TRAINER["lindera-trainer\n(CRF Training)"]
    IPADIC["lindera-ipadic"]
    UNIDIC["lindera-unidic"]
    SUDACHIDICT["lindera-sudachidict"]
    KODIC["lindera-ko-dic"]
    CCCEDICT["lindera-cc-cedict"]
    JIEBA["lindera-jieba"]
    NEOLOGD["lindera-ipadic-neologd"]
    SEGMENTER["lindera-segmenter\n(Segmenter)"]
    ANALYSIS["lindera-analysis\n(Analysis Chain)"]
    LIB["lindera\n(Facade)"]
    CLI["lindera-cli\n(CLI)"]
    BINDING["lindera-binding"]
    PY["lindera-python"]
    NODEJS["lindera-nodejs"]
    RUBY["lindera-ruby"]
    PHP["lindera-php"]
    WASM["lindera-wasm"]

    CRF --> TRAINER
    DICT --> TRAINER
    TRAINER -.->|"train feature"| SEGMENTER
    DICT --> IPADIC
    DICT --> UNIDIC
    DICT --> SUDACHIDICT
    DICT --> KODIC
    DICT --> CCCEDICT
    DICT --> JIEBA
    DICT --> NEOLOGD
    DICT --> SEGMENTER
    IPADIC --> SEGMENTER
    UNIDIC --> SEGMENTER
    SUDACHIDICT --> SEGMENTER
    KODIC --> SEGMENTER
    CCCEDICT --> SEGMENTER
    JIEBA --> SEGMENTER
    NEOLOGD --> SEGMENTER
    SEGMENTER --> ANALYSIS
    SEGMENTER --> LIB
    ANALYSIS -.->|"analysis feature"| LIB
    LIB --> CLI
    LIB --> BINDING
    BINDING --> PY
    BINDING --> NODEJS
    BINDING --> RUBY
    BINDING --> PHP
    BINDING --> WASM

Crate Overview

Tokenization Pipeline

Lindera processes text through a multi-stage pipeline:

Input Text
  |
  v
Character Filters    -- Normalize characters (e.g., Unicode normalization, mapping)
  |
  v
Segmenter            -- Segment text into tokens using a dictionary and the Viterbi algorithm
  |
  v
Token Filters        -- Transform tokens (e.g., POS filtering, stop words, stemming)
  |
  v
Output Tokens

The Segmenter is the core component. It builds a lattice of candidate tokens from the dictionary, then applies the Viterbi algorithm to find the lowest-cost path, producing the most likely segmentation. Dictionary lookups walk a char-wise double-array trie (built with crawdad) in place over the serialized bytes of dict.trie, rather than deserializing into an owned structure. For repeated tokenization, Segmenter::new_worker() returns a SegmentWorker that reuses lattice and scratch-buffer allocations across calls, with an automatic shrink policy to bound retained memory.

Feature Flags

Learn More

• Getting Started -- Installation and first steps
• Core Concepts -- Dictionaries, tokenization, and filters
• Lindera Library -- The lindera facade and the segmenter API
• Lindera Analysis -- Character filters, token filters, and the Tokenizer
• Lindera Dictionary -- Dictionary loading and building
• Lindera Trainer -- CRF-based dictionary training
• Lindera CRF -- The CRF engine
• Lindera CLI -- Command-line interface
• Development Guide -- Build, test, and contribute

Getting Started

This section will guide you through installing Lindera and running your first morphological analysis.

• Installation -- Add Lindera to your project and configure environment variables
• Quick Start -- Tokenize your first text in just a few lines of code
• Examples -- Explore example programs for common use cases

Installation

Requirements

Rust 1.89 or newer. The workspace declares this as rust-version, so an older toolchain reports it as a cargo error rather than failing to compile.

Adding the Dependency

Put the following in Cargo.toml:

[dependencies]
lindera = "7"

[!NOTE] This single dependency also provides the analysis chain (character filters, token filters, and the Tokenizer) as lindera::analysis; use default-features = false if you only need the segmenter. Upgrading from v6? See Migration v6 to v7 for details.

Dictionary Setup

Lindera requires a pre-built dictionary at runtime. Download a dictionary from GitHub Releases and specify its path when loading:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

#![allow(unused)]
fn main() {
// Cargo.toml: lindera = { version = "7", features = ["embed-ipadic"] }
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

Environment Variables

LINDERA_BUILD_DICTIONARY_CACHE_DIR

The LINDERA_BUILD_DICTIONARY_CACHE_DIR environment variable designates a build-time cache directory for the embedded-dictionary build pipeline. It is read only by the dictionary crates' build scripts and has no effect at runtime.

When set, each build stores two kinds of files under $LINDERA_BUILD_DICTIONARY_CACHE_DIR/<version>-fmt<format>/ (where <version> is the dictionary crate version and <format> is the dictionary format version):

• the downloaded distribution archive (validated with MD5; invalid files are re-downloaded)
• the built binary dictionary that gets embedded into the crate, with a build_inputs.json that records what it was built from

The format version is part of the path so that a cache written by a build with a different on-disk dictionary layout is a miss rather than a stale hit. A cached dictionary is reused only when all of the following match the current build, and is rebuilt otherwise:

• the dictionary format version and the contents of the crate's metadata.json
• the lindera-dictionary crate that builds it: its version and a digest of its source, so that any change to the dictionary builder, an uncommitted one included, rebuilds the dictionary
• the source archive (file name and MD5) and the directory the build reads in it
• the context-id frequency histogram the build reads (context_id_freq.txt, or the file named by LINDERA_CTX_FREQ_FILE) when connection_id_mapping is enabled

A dictionary left behind by an interrupted build, or cached by a release from before build_inputs.json, is rebuilt. The versions of the builder's dependencies are not compared: after upgrading a dependency of lindera-dictionary without a format version change, clear the cache if the built dictionaries may differ. Directories from older format versions are not deleted automatically; remove them by hand if the cache grows.

cargo build -vv shows, for each dictionary crate, whether the cached dictionary was used and why or why not, on lines that start with lindera dictionary cache:.

This enables:

• Offline builds: once cached, subsequent builds need no network access
• Faster builds: download and dictionary build are skipped when valid cached files exist
• Reproducible builds: consistent dictionary versions across builds

Usage:

export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache
cargo build --features=embed-ipadic

Notes:

• The directory is managed automatically and is safe to delete; contents are re-downloaded and rebuilt as needed
• Version subdirectories accumulate across upgrades and are not garbage-collected; old ones can be removed freely
• Setting this variable causes dictionary crates to download and build their dictionaries even when no embed-* feature is enabled (useful for pre-populating the cache). Such a build embeds nothing, so it keeps a dictionary already in the cache without checking it; a build with the embed-* feature checks it and rebuilds it if needed
• Use one cache directory per checkout. Two checkouts with different builder code that share a directory rebuild each other's dictionaries, and a crate whose build script does not rerun can embed the dictionary the other checkout built. In this repository, a relative path resolves under the workspace root, which gives each checkout its own cache
• Builds that share the directory, such as cargo build alongside rust-analyzer with its own target directory, write each dictionary one at a time: a build that finds another build writing the same dictionary waits for it, and then uses the dictionary it left (cargo build -vv shows lindera dictionary cache: waiting for another build). The lock is an empty <dictionary>.lock file next to the dictionary, held while a build writes it. It works between builds on one machine, and on a network file system only as far as the file system supports file locks. Where the file cannot be locked, the build prints a warning and goes on without the lock, as builds before 7.0.0 did, so two such builds can still remove each other's files
• Builds with DOCS_RS set ignore this variable (see DOCS_RS below)

LINDERA_CONFIG_FILE

The LINDERA_CONFIG_FILE environment variable names a YAML configuration file for the tokenizer, which TokenizerBuilder::new() reads. This allows you to configure tokenizer behavior without modifying Rust code.

export LINDERA_CONFIG_FILE=./resources/config/lindera.yml

LINDERA_CONFIG_PATH, the name of this variable up to v6, still works until v8.0.0, with a deprecation warning on standard error. When both are set, LINDERA_CONFIG_FILE is used.

See the Configuration section for details on the configuration format.

DOCS_RS

The DOCS_RS environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.

Such a build ignores LINDERA_BUILD_DICTIONARY_CACHE_DIR: the dummy dictionaries are built in the build script's output directory, never in the cache, so a later build without DOCS_RS does not embed them.

This is primarily used internally by docs.rs and typically doesn't need to be set by users.

Quick Start

This example covers the basic usage of Lindera.

It will:

• Create a segmenter in normal mode
• Segment the input text
• Output the tokens

This example uses the embed-ipadic feature, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

use std::borrow::Cow;

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = segmenter.segment(Cow::Borrowed(text))?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

The above example can be run as follows:

% cargo run --features embed-ipadic --example=segment

[!TIP] If you prefer not to embed the dictionary into the binary, download a pre-built IPADIC dictionary from GitHub Releases, extract it to a local directory (e.g., /path/to/ipadic), and call load_dictionary("/path/to/ipadic") instead — no embed-ipadic feature needed in that case. See Feature Flags for details.

You can see the result as follows:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

[!NOTE] Character filters, token filters, and the Tokenizer API are available as lindera::analysis (the analysis feature, enabled by default), so lindera = "7" already covers the whole analysis chain — no extra dependency is needed. With default-features = false you get the segmenter alone. See Lindera Analysis for details, and Migration v6 to v7 if you are upgrading from v6.

Examples

Lindera includes several example programs that demonstrate common use cases. The source code is available in the examples directory on GitHub (the tokenize* examples are in lindera-analysis/examples).

The tokenize* examples use the Tokenizer and filter APIs from lindera::analysis. They live in the lindera-analysis crate, hence -p lindera-analysis below.

All examples below are run with the embed-ipadic feature enabled, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

Available Examples

segment

Basic morphological segmentation with the Segmenter API — it needs only the segmenter, so it also works with default-features = false.

cargo run -p lindera --features=embed-ipadic --example=segment

tokenize

Basic tokenization using an external IPADIC dictionary. Segments input text and prints each token with its part-of-speech details.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

tokenize_with_user_dict

Tokenization with a user dictionary. Shows how to supplement the dictionary with custom entries for domain-specific terms.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_user_dict

tokenize_with_filters

Tokenization with character filters and token filters. Demonstrates the text processing pipeline, including Unicode normalization, part-of-speech filtering, and other transformations.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_filters

tokenize_with_config

Tokenization using a YAML configuration file. Shows how to configure the tokenizer declaratively instead of programmatically.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_config

Core Concepts

This section explains the fundamental concepts behind Lindera's morphological analysis system.

• Morphological Analysis - How Lindera segments and analyzes text.
• Dictionaries - Dictionary formats supported by Lindera.
• Tokenization - Tokenization modes and N-Best analysis.
• User Dictionary - Adding custom words with user dictionaries.
• Filters - Pre-processing text and post-processing tokens with character and token filters.

Morphological Analysis

What is morphological analysis?

Morphological analysis is the process of breaking down text into its smallest meaningful units (morphemes) and identifying their grammatical properties. For languages like Japanese, Chinese, and Korean -- where words are not separated by spaces -- morphological analysis is an essential first step for natural language processing tasks such as search indexing, text classification, and machine translation.

How Lindera works

Lindera is a dictionary-based morphological analyzer. It uses a pre-compiled system dictionary containing known words along with their costs, and applies the Viterbi algorithm to find the optimal segmentation of input text.

The analysis process works as follows:

1. Sentence splitting: Before building the lattice, Lindera splits the input text into sentences at delimiter characters (\n, \t, 。, 、) and processes one sentence at a time. If no delimiter appears within roughly 32 KiB of a sentence's start, Lindera forces a sentence boundary there anyway and logs a warning, to bound the lattice's memory and CPU cost. Up to the next \n or \t, each sentence continues from the words that end the previous one, so the result is, with few exceptions, the one a single lattice over that text gives (see Sentence Splitting). The forced cut can affect tokenization at the artificial cut point for pathological delimiter-free input, such as minified text or base64-encoded blobs.
2. Lattice construction: Lindera scans each sentence and looks up all possible words in the dictionary at every position, building a directed acyclic graph (lattice) of candidate segmentations.
3. Cost assignment: Each candidate word has an associated word cost (from the dictionary), and each pair of adjacent words has a connection cost (from the connection cost matrix).
4. Optimal path search: The Viterbi algorithm finds the path through the lattice with the minimum total cost, producing the best segmentation.

Key terminology

Cost-based segmentation

The Viterbi algorithm selects the segmentation path with the minimum total cost. The total cost of a path is the sum of:

• Word costs: Each word in the dictionary has an associated cost. Lower cost means the word is more likely to appear. Common words tend to have lower costs, while rare words have higher costs.
• Connection costs: The cost of connecting two adjacent words, determined by the right context ID of the left word and the left context ID of the right word.

The algorithm computes:

Total cost = sum of word costs + sum of connection costs

By minimizing this total cost, Lindera finds the most natural segmentation of the input text.

Connection cost matrix

The connection cost matrix stores the cost of transitioning from one word to another. It is a two-dimensional matrix indexed by:

• The right context ID of the preceding word
• The left context ID of the following word

These context IDs encode grammatical information about word boundaries. For example, the connection cost between a noun and a particle is typically low (natural sequence), while the connection cost between two verbs in base form might be high (unnatural sequence).

The connection cost matrix is compiled into binary format as part of the dictionary build process and is loaded at runtime for efficient lookup.

Dictionaries

Lindera supports various dictionaries for Japanese, Korean, and Chinese morphological analysis. Each dictionary is provided as a separate crate.

Obtaining Dictionaries

Pre-built dictionaries are available for download from GitHub Releases. Download the dictionary archive for your target language and extract it to a local directory.

#![allow(unused)]
fn main() {
// Load an external dictionary from a local path
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you need a self-contained binary without external dictionary files, you can embed dictionaries using the embed-* feature flags and load them using the embedded:// scheme:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

See each dictionary crate's documentation for format details, build instructions, and usage examples.

Tokenization

Lindera provides multiple tokenization modes and supports N-Best analysis for enumerating alternative segmentation candidates.

Tokenization modes

Normal mode

Normal mode performs standard tokenization based on dictionary entries. Compound words that exist as single entries in the dictionary are kept as-is.

Example -- tokenizing "関西国際空港限定トートバッグ" in Normal mode:

関西国際空港 | 限定 | トートバッグ

The compound noun "関西国際空港" (Kansai International Airport) is preserved as a single token because it exists as one entry in the dictionary.

Decompose mode

Decompose mode further breaks down compound nouns into their constituent parts, even when the compound exists as a dictionary entry.

Example -- tokenizing "関西国際空港限定トートバッグ" in Decompose mode:

関西 | 国際 | 空港 | 限定 | トートバッグ

The compound "関西国際空港" is decomposed into "関西", "国際", and "空港".

Selecting a mode

In Rust, specify the mode when creating a Segmenter:

#![allow(unused)]
fn main() {
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;

// Normal mode
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

// Decompose mode
let segmenter = Segmenter::new(Mode::Decompose(Default::default()), dictionary, None);
}

With the CLI, use the --mode flag:

echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode normal
echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode decompose

[!NOTE] The embedded:// scheme requires lindera-cli to be built with the corresponding embed-* feature (e.g. cargo install lindera-cli --features=embed-ipadic). A default build of lindera-cli does not enable any embed-* feature, and passing --dict embedded://ipadic without it fails with Invalid dictionary scheme: embedded.

N-Best tokenization

N-Best tokenization enumerates the top N tokenization candidates ordered by total path cost (lower cost = better segmentation). This is useful when the best result is ambiguous, or when you want to explore alternative interpretations of the input text.

Algorithm

N-Best tokenization is based on the Forward-DP Backward-A* algorithm, which is compatible with MeCab's N-Best implementation. The forward pass computes optimal costs using dynamic programming, and the backward pass uses A* search to enumerate paths in order of increasing total cost.

As in 1-best tokenization, the input is first split into sentences, and the sentences up to each \n or \t form a segment (see Sentence Splitting). Each segment is searched on its own: within it, the context is carried across 、 and 。, so its paths are the N best paths of one lattice over the segment. Every candidate segments the whole input: it takes one path in each segment, and its cost is the sum of those paths' costs. The candidates are the N cheapest of these combinations.

With unique, the backward search runs over word boundaries instead of paths: a segmentation has one path per choice of dictionary entry for each of its words, which can be millions for a short line, and the search takes each segmentation once, with its cheapest path, without going through the others.

Parameters

The tokenize_nbest method accepts the following parameters:

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "すもももももももものうち";

    // Get top 3 tokenization results
    let results = tokenizer.tokenize_nbest(text, 3, false, None)?;

    for (rank, (tokens, cost)) in results.iter().enumerate() {
        println!("--- NBEST {} (cost={}) ---", rank + 1, cost);
        for token in tokens {
            let details = token.details().join(",");
            println!("{}\t{}", token.surface.as_ref(), details);
        }
    }

    Ok(())
}

Output:

--- NBEST 1 (cost=21245) ---
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
--- NBEST 2 (cost=24541) ---
...

CLI example

echo "すもももももももものうち" | lindera tokenize --dict embedded://ipadic -N 3

Lattice reuse

For repeated tokenization, you can reuse a Lattice to reduce memory allocations:

#![allow(unused)]
fn main() {
use lindera::dictionary::Lattice;

let mut lattice = Lattice::default();
let results = tokenizer.tokenize_nbest_with_lattice(text, &mut lattice, 3, false, None)?;
}

At the segmenter level, SegmentWorker (created via Segmenter::new_worker) packages this pattern into a session object that owns the lattice and scratch buffers, and additionally bounds retained memory with an automatic shrink policy. See the Segmenter page for details.

User Dictionary

A user dictionary is a supplementary dictionary that allows you to register custom words alongside the system dictionary. This is useful for domain-specific terms, brand names, proper nouns, or any words that are not in the default system dictionary.

CSV format

The simplest user dictionary format is a CSV file with three columns:

<surface>,<part_of_speech>,<reading>

Example CSV content

東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ

The surface (the first column) is used exactly as written: whitespace at its start or end is part of the word, and a surface may consist of whitespace only (IPADIC registers U+3000 this way). Only an empty surface is skipped. The same applies to the lexicon CSV files of a system dictionary built with lindera build.

An entry that ends with whitespace keeps it in its token, as in MeCab, even when whitespace is skipped in the lattice (the default for every bundled dictionary except SudachiDict, see Whitespace Handling). With whitespace skipped, however, an entry that starts with whitespace of the dictionary's SPACE category, or consists of such whitespace only, never matches, because no word starts on skipped whitespace. U+3000 is not in the SPACE category, so IPADIC's U+3000 entry still matches.

Each dictionary type (IPADIC, UniDic, ko-dic, etc.) also supports a detailed CSV format with full control over context IDs, costs, and all feature fields. See the Dictionaries section for the detailed format of each dictionary type.

Tied entries

Two entries tie when they have the same surface, the same left and right context IDs and the same cost: every path costs the same with either of them. A user entry that ties with a system entry is chosen over it, so a detailed entry that copies the surface, the context IDs and the cost of a system entry, for example to give the word another reading, takes its place in the output. MeCab chooses the system entry instead. Among tied user entries, the first row is chosen, as in MeCab (before v7, the last row was; see Migrating from v6 to v7). Simple entries get the cost and the context IDs from the dictionary's metadata.json (default_word_cost, default_left_context_id and default_right_context_id: -10000, 0 and 0 for IPADIC), so they practically never tie with a system entry.

Context IDs

The left and right context IDs of a detailed entry must address a cell of the connection matrix of the system dictionary it is used with: the left context ID must be below the second value of that dictionary's matrix.def header, and the right context ID below the first, the bounds lindera build applies to the system dictionary itself. IPADIC's matrix is 1316 x 1316, for example.

A user dictionary built from CSV together with its system dictionary is checked row by row as it is built, and the first row outside the matrix fails with an error that names its CSV line and surface:

• lindera build --user when --system-dict names the system dictionary (DictionaryBuilder::with_system_matrix in the Rust API; see Context ID check)
• a .csv user dictionary that the configuration names (Segmenter::from_config, TokenizerBuilder), that lindera tokenize --user-dict names, or that the TokenizerBuilder of the Python, Node.js, Ruby and PHP bindings is given by path
• the loaders that load a user dictionary for a loaded system dictionary: load_user_dictionary_for in the Rust API, Python and Ruby, loadUserDictionaryFor in Node.js, Lindera\Dictionary::loadUserFor in PHP, and loadUserDictionaryFromBytesFor in WebAssembly
• the builders that build a user dictionary for a loaded system dictionary: build_user_dictionary_for in Python and Ruby, buildUserDictionaryFor in Node.js, and Lindera\Dictionary::buildUserFor in PHP

For example, with a userdic.csv whose first row lies inside IPADIC's matrix and whose second row has the left context ID 60000:

$ echo 東京スカイツリーへ行く | lindera tokenize -d embedded://ipadic -u userdic.csv
Error: LinderaError { kind: Args, source: LinderaError(kind=Parse, source=failed to build tokenizer: LinderaError(kind=Content, source=user dictionary line 2 (surface: '東京スカイツリー'): left context id 60000 is out of range for the connection matrix (matrix.def size 1316 1316; left context ids must be below 1316)))

Otherwise the user dictionary is built without the system dictionary: a .bin that lindera build --user wrote without --system-dict or that a builder of the bindings wrote with metadata only (build_user_dictionary in Python and Ruby, buildUserDictionary in Node.js, Lindera\Dictionary::buildUser in PHP), or a user dictionary loaded with metadata only (load_user_dictionary in the Rust API, Python and Ruby, loadUserDictionary in Node.js, Lindera\Dictionary::loadUser in PHP, loadUserDictionaryFromBytes in WebAssembly). Its IDs are checked when the two dictionaries are combined: Segmenter::try_new, the configuration path, lindera tokenize --user-dict, and the Tokenizer constructors and builders of the language bindings return an error for the first entry outside the matrix. The error names the entry by its word id, which counts the rows from 0 in the order of their surfaces, and by its detail fields. The .bin built from the userdic.csv above gives:

$ echo 東京スカイツリーへ行く | lindera tokenize -d embedded://ipadic -u user_dict/userdic.bin
Error: LinderaError { kind: Args, source: LinderaError(kind=Parse, source=failed to build tokenizer: LinderaError(kind=Content, source=user dictionary entry 1 (名詞,固有名詞,一般,*,*,*,東京スカイツリー,トウキョウスカイツリー,トウキョウスカイツリー): left context id 60000 is out of range for the connection matrix (matrix.def size 1316 1316; left context ids must be below 1316)))

Segmenter::new does not check the IDs: an entry outside the matrix then makes segmentation panic when the entry matches the input. Before v7.0.0, every path behaved this way. Simple entries are checked too; they get the context IDs that the metadata.json they are built with sets (default_left_context_id and default_right_context_id, 0 in every bundled dictionary).

Rust API example

use std::fs::File;
use std::path::PathBuf;

use lindera::dictionary::{Metadata, load_dictionary, load_user_dictionary};
use lindera::error::LinderaErrorKind;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let user_dict_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("user_dict")
        .join("ipadic_simple_userdic.csv");

    let metadata_file = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../lindera-ipadic")
        .join("metadata.json");
    let metadata: Metadata = serde_json::from_reader(
        File::open(metadata_file)
            .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
            .unwrap(),
    )
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
    .unwrap();

    let dictionary = load_dictionary("embedded://ipadic")?;
    let user_dictionary = load_user_dictionary(user_dict_path.to_str().unwrap(), &metadata)?;
    // `try_new` checks the user dictionary's context IDs against the
    // dictionary's connection matrix (see "Context IDs" above).
    let segmenter = Segmenter::try_new(
        Mode::Normal,
        dictionary,
        Some(user_dictionary), // Using the loaded user dictionary
    )?;

    // Create a tokenizer.
    let tokenizer = Tokenizer::new(segmenter);

    // Tokenize a text.
    let text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です";
    let mut tokens = tokenizer.tokenize(text)?;

    // Print the text and tokens.
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Output:

text:   東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です
token:  東京スカイツリー        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリー,*
token:  の      助詞,連体化,*,*,*,*,の,ノ,ノ
token:  最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
token:  は      助詞,係助詞,*,*,*,*,は,ハ,ワ
token:  とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリーエキ,*
token:  です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス

Note that the base_form field (the 7th CSV/detail field) is * rather than the surface form. The simple three-column user-dictionary schema only supplies surface, part_of_speech, and reading — every other field, including base_form, is filled with the dictionary's metadata.default_field_value (* for IPADIC).

Building a user dictionary with CLI

You can build a user dictionary from CSV to binary format using the CLI:

lindera build --src <user_dictionary.csv> --dest <dest_dir> --metadata <metadata.json> --user [--system-dict <system_dictionary>]

With --system-dict, the build checks the context IDs against the system dictionary the user dictionary is for (see Context IDs).

Binary vs CSV user dictionary

• CSV format: Loaded and parsed at runtime. Convenient for development and small dictionaries.
• Binary format: Pre-compiled for faster loading. Recommended for production use with large user dictionaries.

Both formats can be specified when creating a Segmenter. The binary format skips the CSV parsing step, resulting in faster startup times.

Filters

Lindera's analysis pipeline (the Tokenizer in lindera::analysis, provided by the lindera-analysis crate) has two extension points around the segmenter: character filters, which transform the raw input text before tokenization, and token filters, which transform the list of tokens after tokenization.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character filters

Character filters preprocess the input text before it reaches the segmenter. They're typically used to normalize text so that tokenization is more consistent -- for example, converting full-width characters to half-width, canonicalizing Unicode representations, or expanding Japanese iteration marks into their repeated form. Because a character filter can change the length of the text, Lindera tracks every transformation it makes and corrects each token's byte offsets back to positions in the original, unfiltered text.

Token filters

Token filters postprocess the list of tokens produced by the segmenter. They're typically used to normalize or reduce the token list for search and analysis -- for example, replacing a token with its base (dictionary) form, converting a token's surface text between hiragana and katakana, removing tokens by part-of-speech tag, or removing stop words.

Configuring filters

Every filter -- character or token -- is identified by a kind string and configured with a JSON args object of filter-specific parameters. Filters run in the order they're added, and there are two ways to add them:

• YAML configuration file: list filters under the character_filters and token_filters keys.

  character_filters:
    - kind: unicode_normalize
      args:
        kind: nfkc
  
  token_filters:
    - kind: japanese_base_form
  

• Rust API: append filters to a Tokenizer, in order, with append_character_filter and append_token_filter.

  #![allow(unused)]
  fn main() {
  // Character filters run first, transforming the raw input text;
  // token filters run last, transforming the resulting token list.
  tokenizer
      .append_character_filter(BoxCharacterFilter::from(unicode_normalize_char_filter))
      .append_token_filter(BoxTokenFilter::from(japanese_base_form_filter));
  }

Available filters

Lindera ships 4 character filters and 21 token filters, covering Japanese, Korean, and general-purpose text normalization:

See the Filters reference for what each filter does, its parameters, and runnable YAML and Rust API examples.

Lindera CRF

Lindera CRF is a pure Rust implementation of Conditional Random Fields (CRFs), forked from rucrf. It provides a trainer and an estimator for CRFs with support for lattice structures.

Key Features

• Lattices with variable length edges
• L1, L2, and Elastic Net regularization
• Multi-threaded training
• Zero-copy deserialization with rkyv
• no_std support via --no-default-features --features alloc

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Changes from rucrf

• Serialization backend: Switched from bincode to rkyv for zero-copy deserialization
• Elastic Net regularization: Added Regularization::ElasticNet combining L1 and L2 penalties
• Default regularization: Trainer::new regularizes with L2 and DEFAULT_LAMBDA (1.0), the objective of MeCab's learner with its default -c 1.0; rucrf's default is L1 with 0.1
• Rust 2024 edition: Updated to Rust 2024 edition
• Dependency updates: Updated argmin, argmin-math, hashbrown, etc.

Architecture

Module Structure

lindera-crf/src/
├── lib.rs                # Public API re-exports
├── bigram_table.rs       # Bigram weight indices of the training lattices
├── feature.rs            # FeatureSet, FeatureProvider
├── lattice.rs            # Edge, Node, Lattice
├── model.rs              # RawModel, MergedModel, Model trait
├── trainer.rs            # Trainer, Regularization, StopReason, TrainingStats
├── errors.rs             # Error types
├── forward_backward.rs   # Forward-backward algorithm
├── math.rs               # Mathematical utilities (logsumexp)
├── optimizers.rs         # Declares the optimizers module
├── optimizers/
│   └── lbfgs.rs          # L-BFGS optimization
└── utils.rs              # Utility traits

Key Components

FeatureProvider / FeatureSet

Manage per-label feature sets. Each FeatureSet holds unigram features and left/right bigram features for a given label (feature IDs only, no weights). FeatureProvider maps label IDs to FeatureSet instances. Weights are held separately, on RawModel (weights, unigram_weight_indices, bigram_weight_indices).

The bigram feature IDs are held in Arc<[Option<NonZeroU32>]>, so labels with the same bigram features can share one list: FeatureSet::with_shared_bigrams takes the lists as Arcs, while FeatureSet::new copies the slices it is given. When training drops the unused features, it filters each shared list once, and the labels that shared the list share the filtered one. lindera-trainer shares the lists among the labels whose rewritten left and right features are the same (#1263).

Lattice / Edge / Node

Lattice structure with variable-length edges for sequence labeling. Edge represents a candidate span with a label, while Node aggregates edges at a given position. The Lattice is constructed from input data and used by the model to find the best path.

Trainer

Trains a CRF model using L-BFGS optimization (OWL-QN when the regularization has an L1 part) with configurable regularization. As in MeCab's learner and libLBFGS, the first line search starts at a step of 1 / |d|, d being the first search direction, and every later one at a step of 1. The trainer accepts labeled lattice examples, computes the loss and its gradient via the forward-backward algorithm, and iteratively updates model weights. Trainer::train returns the trained RawModel; Trainer::train_with_stats also returns a TrainingStats with:

• iterations: the number of iterations that ran, which is the maximum set with Trainer::max_iter when that stopped training, and fewer when the optimizer stopped earlier.
• evaluations: the number of evaluations of the loss and its gradient, each one forward-backward pass over every lattice, line-search trials included.
• stop_reason: why the optimizer stopped, a StopReason.

The optimizer stops at the first of these:

• StopReason::Converged: the relative change of the objective, |previous - current| / |previous|, stayed below the tolerance set with Trainer::eta for 3 iterations in a row, or the optimizer stopped near the optimum, where the norm of the gradient (the pseudo-gradient for OWL-QN) is under 1e-5 times its norm at the initial weights, or vanishes. The default tolerance, DEFAULT_ETA (5e-5), is the default eta of MeCab's learner, which applies the same test to every evaluation, line-search trials included, where Lindera counts iterations. Trainer::eta(0.0) turns the test off. Near the optimum the objective changes by less than its rounding, and whether an iteration then leaves it unchanged or a line search fails depends on the platform; either is Converged there.
• StopReason::NoProgress: an iteration left the objective unchanged far from the optimum, where the norm of the gradient was still at least 1e-5 times its norm at the initial weights. When this happens after a few iterations, the model is barely trained.
• StopReason::MaxIterations: the cap set with Trainer::max_iter.
• StopReason::LineSearchFailed: the line search of an iteration failed far from the optimum, where the norm of the gradient at the best weights of the run was still at least 1e-5 times its norm at the initial weights. The weights are the best ones of the run. A search direction with a component that is not finite, which the optimizer builds from a step that left the gradient unchanged, fails the line search too.

Regularization

Configurable regularization strategies:

• L1: Sparse models via L1 penalty
• L2: Smooth models via L2 penalty
• ElasticNet: Combines L1 and L2 with a configurable l1_ratio

Trainer::new uses L2 with DEFAULT_LAMBDA (1.0): the objective of MeCab's learner with its default -c 1.0. MeCab adds ‖w‖² / (2C), so the coefficient is 1 / C in general.

Model (trait)

Interface for searching the best path through a lattice. Two implementations are provided:

• RawModel: Stores weights in a flat vector indexed by feature ID
• MergedModel: Optimized for inference; merges feature weights into a compact representation serializable with rkyv, using MergedFeatureSet as its per-label element type

MergedFeatureSet holds the pre-summed unigram weight for a label, along with its left_id/right_id bigram connection IDs into MergedModel's bigram weight matrix.

RawModel::unigram_weight computes that sum for a list of unigram feature IDs, and merge calls it for every label. A feature without a weight adds nothing, so it also weighs the features of an entry that was not a label in training: lindera-trainer uses it for the cost of a user lexicon entry.

Forward-backward Algorithm

Computes alpha (forward) and beta (backward) values over the lattice. Used during training to calculate the loss and the expected feature counts. One pass over a lattice yields both its loss and its share of the gradient. The optimizer keeps the result for the last parameter vector, because argmin's L-BFGS asks for the cost and the gradient of the same point in two calls.

Before the optimizer starts, a bigram table records, for every pair of edges that meet in the lattices, the weight indices of their bigram features, in the order the evaluation adds them; labels with the same bigram feature IDs share the entries. The evaluations read the indices from the table instead of looking each bigram feature up in a map, and add the same weights in the same order, so the trained weights do not change.

Feature Flags

* Not listed directly in default, but transitively enabled by train (which is on by default and enables std, which in turn enables alloc) -- so alloc/std are active in a default build.

API Reference

The API reference is available. Please see following URL:

• lindera-crf

Lindera Dictionary

Lindera Dictionary is the base library for morphological analysis dictionaries. It provides dictionary loading, building, and Viterbi-based segmentation. CRF-based dictionary training lives in the separate lindera-trainer crate. Through the lindera facade its modules are available under lindera::dictionary — lindera::dictionary::core (the dictionary data structures), lindera::dictionary::builder, lindera::dictionary::viterbi, and so on — so a direct lindera-dictionary dependency is rarely needed.

Key Features

• Dictionary loading from filesystem or embedded data
• Dictionary building from MeCab-format CSV source files
• Viterbi algorithm for optimal segmentation
• N-best path generation (Forward-DP Backward-A*)
• Memory-mapped file support

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-dictionary/src/
├── lib.rs               # Public API
├── dictionary.rs        # Dictionary, UserDictionary
├── builder.rs           # DictionaryBuilder
├── loader.rs            # DictionaryLoader trait, FSDictionaryLoader
├── viterbi.rs           # Lattice, Edge, Viterbi segmentation
├── nbest.rs             # NBestGenerator, UniqueNBestGenerator (Forward-DP Backward-A*)
├── mode.rs              # Mode (Normal/Decompose), Penalty
├── error.rs             # LinderaError, LinderaErrorKind
├── assets.rs            # Download and file management
├── macros.rs            # embedded_dictionary! macro shared by dictionary crates
├── dictionary/
│   ├── character_definition.rs    # Character type definitions
│   ├── connection_cost_matrix.rs  # Connection cost matrix
│   ├── context_id_map.rs          # ContextIdMap: connection-cost context-ID remap
│   ├── prefix_dictionary.rs       # Prefix dictionary (char-wise double-array trie)
│   ├── unknown_dictionary.rs      # Unknown word handling
│   ├── metadata.rs                # Dictionary metadata
│   └── schema.rs                  # Schema definitions

Key Components

Dictionary / UserDictionary

Main data structures holding the compiled dictionary data. A Dictionary contains the character definitions, connection cost matrix, prefix dictionary, and unknown word dictionary. The prefix dictionary is a char-wise double-array trie, built with crawdad at build time and walked in place over its serialized bytes at runtime — loading it is an O(1) header check, not a deserialization pass. UserDictionary allows users to add custom vocabulary on top of the system dictionary; unlike the system dictionary, it still uses a daachorse automaton inside its rkyv-archived .bin files.

DictionaryBuilder

Fluent API for building dictionaries from source CSV files. It compiles MeCab-format dictionary sources into the binary format used at runtime. Its four build stages (metadata, unknown dictionary, prefix dictionary, connection cost matrix) run concurrently on scoped threads on non-wasm targets, falling back to a sequential build on wasm (which has no OS threads); the concurrent path has higher peak memory, since all four stages' working sets are held at once.

DictionaryLoader / FSDictionaryLoader

DictionaryLoader is a trait for loading compiled dictionaries. FSDictionaryLoader is the filesystem-based implementation that reads dictionary files from a directory, with optional memory-mapped file support.

Embedded Dictionary Macro

The embedded_dictionary! macro (lindera-dictionary/src/macros.rs, #[macro_export]) generates the boilerplate each dictionary crate needs to bake its compiled dictionary into the binary: a load() function that reads the dictionary components via include_bytes! and a loader struct implementing DictionaryLoader. Every dictionary crate's embedded.rs (lindera-ipadic, lindera-ipadic-neologd, lindera-unidic, lindera-ko-dic, lindera-cc-cedict, lindera-jieba) invokes this macro instead of duplicating the loading logic.

Viterbi (Lattice, Edge)

Builds a lattice of candidate tokens from the input text and finds the optimal segmentation path using the Viterbi algorithm. Each Edge in the lattice represents a candidate token with associated costs (word cost + connection cost).

NBestGenerator

Generates N-best segmentation paths using the Forward-DP Backward-A* algorithm. This enables applications to consider alternative segmentations beyond the single best path.

UniqueNBestGenerator

Generates the N-best distinct segmentations: for each sequence of word boundaries, its cheapest path, in ascending order of cost. Unique N-best (unique, --nbest-unique) uses it. A segmentation has one path per choice of dictionary entry for each of its words, which can be millions of paths for a short line, so the search runs backward over suffixes of word boundaries instead of paths: one search node holds every candidate of a span, and its cost is exact, so each segmentation comes out once and the work grows with the number of results and the length of the line, not with the number of paths. The first result is the 1-best path, and of identical entries (same surface, context ids and cost) every result holds the first CSV row, as the 1-best path does.

Mode

Controls tokenization behavior:

• Normal: Standard tokenization using the optimal Viterbi path
• Decompose: Further splits compound nouns based on configurable Penalty thresholds

Dictionary Format Version

A built dictionary directory records the on-disk layout it was written in, as format_version in metadata.json. DictionaryBuilder::build_metadata stamps the value from DICTIONARY_FORMAT_VERSION (lindera-dictionary/src/dictionary/metadata.rs) — the builder is the only thing that can state truthfully what it just wrote, so a value in the source metadata.json is ignored. Loading a dictionary directory checks the recorded version and fails with an actionable message if it does not match.

The check matters because most artifacts have no header of their own: matrix.mtx, dict.vals and dict.words are raw arrays, so a dictionary from an older layout that happens to be a plausible length decodes into garbage rather than failing. A source metadata.json — the hand-written kind checked into each dictionary crate — describes build inputs, carries no format_version, and is never format-checked.

Bump DICTIONARY_FORMAT_VERSION on any change to the bytes of a built artifact. That includes upgrading a dependency whose serialized form is written verbatim (crawdad for dict.trie, rkyv for char_def.bin and unk.bin, daachorse inside user dictionary .bin files), which is easy to overlook because no line of this crate changes. The build cache is keyed on the format version for exactly that reason.

The current format version is 3. Version 2 (v6.0.0) replaced the system prefix dictionary's byte-wise daachorse Aho-Corasick automaton (dict.da) with a char-wise double-array trie (dict.trie) plus a u32 prefix-sum index (dict.valsidx). Version 3 (v7.0.0) lists each code point's default category first in its char_def.bin row, followed by its other categories in category id order, because unknown words come from the default category only, as in MeCab; a version 2 row would silently give the wrong default category. Version 3 also covers the earlier 7.0.0 changes to what the builder writes (see Migrating from v6 to v7). Dictionaries at version 2 or earlier are rejected at load time with an actionable error. User dictionary .bin files were not affected by that change — but they are written by daachorse, so a daachorse upgrade does invalidate them, which is exactly why it belongs in the list above.

Context ID Remapping

Dictionary metadata (lindera-dictionary/src/dictionary/metadata.rs) carries a connection_id_mapping: bool flag and an optional context_id_map: Option<ContextIdMap>. When a dictionary crate (e.g. lindera-unidic) enables connection_id_mapping, DictionaryBuilder relabels the connection matrix's left/right context IDs by access frequency at build time, so that frequently-used connection-cost cells cluster together for better cache locality; DictionaryBuilder::with_context_id_freq attaches an optional bundled frequency histogram used to rank the IDs, and the remap itself is computed in lindera-dictionary/src/builder/context_id_remap.rs. ContextIdMap (lindera-dictionary/src/dictionary/context_id_map.rs) holds the resulting left/right permutation and is persisted into the built metadata.json so the same mapping can be reapplied later. Because user dictionaries are always compiled in the original, un-remapped ID space, UserDictionary::remap_context_ids (lindera-dictionary/src/dictionary.rs) uses the persisted ContextIdMap to relabel a user dictionary's context IDs into the same space as the system dictionary it is attached to. The remap is a bijective relabeling, so it does not change tokenization output, only lookup locality.

Training

The CRF-based dictionary training pipeline lives in the separate lindera-trainer crate, which builds on this crate's runtime types. See the training pipeline documentation for details.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-dictionary

Lindera Trainer

Lindera Trainer implements the CRF-based dictionary training pipeline (lindera train), turning an annotated corpus and a seed lexicon into a trained model that can be exported to MeCab-format dictionary source files and compiled into a binary dictionary. It builds on the runtime types of lindera-dictionary and the CRF core of lindera-crf. The crate can be used directly, or through the lindera facade's train feature, which re-exports it as lindera::dictionary::trainer.

Key Features

• CRF-based weight learning via lindera-crf, with L1, L2, and Elastic Net regularization
• MeCab-compatible feature template parsing (feature.def: %F[n], %L[n], %R[n], %w, %u, %l, %r, and their optional ? variants)
• MeCab-compatible 3-section feature rewriting (rewrite.def: unigram / left / right rewrite rules)
• Dictionary-format agnostic: works with any lexicon whose columns follow surface,left_id,right_id,cost,feature... (IPADIC, UniDic, ko-dic, CC-CEDICT, etc.)
• Training lattices built as MeCab's learner builds them: the seed words at every position, and the unknown words that the runtime creates from char.def and unk.def, with one CRF label per unk.def entry
• Connection cost matrix and MeCab-compatible cost conversion (tocost) derived from learned CRF weights
• Zero-copy rkyv binary model serialization with a format version, with legacy JSON fallback on read
• Direct export of Lindera/MeCab dictionary source files (lex.csv, matrix.def, unk.def, char.def, feature.def, rewrite.def, left-id.def, right-id.def)

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-trainer/src/
├── lib.rs                # Trainer: CRF training orchestration; public API re-exports
├── config.rs             # TrainerConfig: parses seed lexicon, char.def, unk.def, feature.def, rewrite.def
├── corpus.rs              # Corpus, Example, Word: training data representation
├── feature_extractor.rs  # FeatureExtractor: feature template parsing and feature ID management
├── feature_rewriter.rs   # DictionaryRewriter: MeCab-compatible 3-section rewrite
└── model.rs               # Model, SerializableModel: trained model storage, serialization, dictionary output

Key Components

TrainerConfig

Parses the five training input files -- seed lexicon (lex.csv), character definition (char.def), unknown-word definition (unk.def), feature template (feature.def), and rewrite rules (rewrite.def) -- into the configuration consumed by Trainer. TrainerConfig::from_readers (or from_paths, its file-path convenience wrapper) extracts the surface/feature vocabulary from the seed lexicon, builds a FeatureExtractor from the parsed feature templates, builds a DictionaryRewriter from the rewrite rules, and assembles an in-memory lindera_dictionary::dictionary::Dictionary from the seed lexicon, the character definition and the unknown-word definition. char.def is parsed by the dictionary builder (CharacterDefinitionBuilder::build_from_str), so training sees the same character categories as the built dictionary; the default category of each character, which the %t feature uses, comes from the builder's build_default_category_table. The Dictionary (dict()) holds what the training lattices are built from: its prefix_dictionary (system_lexicon()) is the seed lexicon, a prefix dictionary over the seed surfaces whose word ID is the row index, and its unknown_dictionary (unk_handler()) holds the unk.def entries, read with the dictionary builder's parse_unk, one entry per line in file order. Their costs and context IDs are not used in training, and the Dictionary's connection matrix is a placeholder. TrainerConfig also keeps the seed feature strings and the unk.def entries as (category, feature string) pairs in file order, from which the labels and the exported unk.def are made. In unk.def, lines that start with # and empty lines are skipped, and a line with fewer than five fields, or whose context IDs or cost are not numbers, is an error that names the line. As mecab-dict-index and lindera build do, from_readers also rejects a char.def range line that names a category not defined on an earlier line, a category defined twice, an unk.def line whose category char.def does not define, and a char.def category without an unk.def line (lindera_dictionary::dictionary::unknown_dictionary::check_unknown_word_categories), so that such input fails before training rather than when the exported files are built. It also exposes surfaces() and features(), the surfaces and the feature strings of the seed rows in row order, get_features(), a user lexicon accessible via user_lexicon() / add_user_lexicon_entry() / load_user_lexicon_from_content(), and metadata().

Corpus / Example / Word

Represent annotated training data. Word pairs a surface form with its (comma-joined) feature string. Example is one training sentence, built from a Vec<Word> whose surfaces are concatenated to reconstruct the original sentence text. Corpus is a collection of Examples; Corpus::from_reader parses tab-separated surface<TAB>features lines, treating blank lines or literal EOS lines as sentence boundaries.

FeatureExtractor

Parses MeCab-compatible feature templates and manages the mapping from generated feature strings to interned, NonZeroU32 feature IDs. Supported template placeholders are %F[n] / %F?[n] (feature field at index n, the ? form skipped when the value is *), %t (character category), %w (surface form, unigram only), %u / %l / %r (full rewritten unigram/left/right feature string), and %L[n] / %L?[n] / %R[n] / %R?[n] for bigram left/right context fields. A template whose %F[n], %L[n] or %R[n] reads past the last column gives no feature, with or without ?, as in MeCab's learner. extract_unigram_feature_ids[_with_ctx], extract_left_feature_ids[_with_ctx], and extract_right_feature_ids[_with_ctx] apply the parsed templates to a feature vector (plus, for bigram extraction, an optional TemplateContext carrying surface/ufeature/lfeature/rfeature) and return the resulting feature IDs, creating new IDs on first use. The IDs live in three FeatureIdMaps, read through unigram_feature_ids(), left_feature_ids() and right_feature_ids(): the k-th string a map meets gets ID k, an ID is never reused (a feature that training removed and that comes again gets a new one), and the strings are kept back to back in one buffer with only the IDs in the hash table, since the unigram map of a full dictionary holds millions of strings (#1286).

The trainer takes the left and right feature IDs of a label from a cache keyed by the label's rewritten left and right features: the left and right templates read nothing else, so the pair decides the IDs. The first label with a pair makes them with extract_left_feature_ids_with_ctx and extract_right_feature_ids_with_ctx, so new IDs are minted in the order they would be without the cache, and the labels with the same pair share the lists (the 347,344 rows of mecab-ipadic have 1,329 pairs). Training forgets the cache once it removes the unused features. A template is expanded into a buffer that the extractor keeps, so a feature string is allocated only when it is new (#1263).

DictionaryRewriter

Implements MeCab's 3-section rewrite.def format: [unigram rewrite], [left rewrite], and [right rewrite] sections, each holding an ordered list of pattern<TAB>replacement rules built into a FeatureRewriter prefix trie (via the internal FeatureRewriterBuilder). DictionaryRewriter::from_reader parses all three sections (falling back to treating an unsectioned file as legacy right-rewrite-only content, for backward compatibility with older Lindera rewrite.def files). rewrite() applies all three rewriters to a feature string and returns (ufeature, lfeature, rfeature), passing a section through unchanged when no rule matches. As in MeCab, the feature and the pattern and replacement of each rule are split into columns with the rules of MeCab's tokenizeCSV (the crate-private feature_csv::split_feature), so a quoted column with , in it is one column, and rewrite() joins the rewritten columns with ,, quoting a column that contains , or ". The trainer splits the three results the same way for the feature templates, and has them written into buffers it reuses from one label to the next, so that the rewrite allocates only when a buffer grows (#1286). (An earlier rewrite_cached() memoized per full feature string, but real lexicon features are unique per row, so the cache never hit and was removed — #975.)

Trainer

Trainer::new creates the CRF labels: one per seed row, in row order (label row + 1, so that a match in the training lexicon, whose word ID is the row index, maps to its label), then one per unk.def entry, in file order, whose %t is the ID of the entry's category. Trainer::train builds one lindera_crf::Lattice per example, as MeCab's learner does:

• The candidates at each position are the seed words that start there and the unknown words that the runtime creates there, listed by lindera_dictionary::viterbi::Lattice::unknown_word_candidates: those of the character's default category, with its INVOKE, GROUP and LENGTH settings, for every unk.def entry of the category.
• Each gold token gets the label of the first seed row with the same span and the same whole feature string; else that of the first unk.def entry, in file order, of an unknown-word candidate with the same span whose feature columns other than * all equal the gold token's (both split as rewrite() splits a feature); else a virtual label built from the gold feature string (%t 0, no %w, one per distinct feature string), as MeCab's learner adds a virtual node. Virtual labels follow the other labels, are trained, and are never exported.
• Every other candidate becomes a negative edge.

Once the lattices are built, it prints a Lattice statistics: line with the number of gold tokens labelled each way (gold_lexicon, gold_unknown, gold_virtual), the number of virtual labels (virtual_labels), and the negative edges from the seed lexicon and from the unknown-word candidates (negative_lexicon, negative_unknown). When there are gold tokens and none is a seed word, or the gold tokens whose surface has seed rows but none with their feature string outnumber the seed words, it then writes a warning on stderr with eprintln!, not through the log crate, for which the bindings install no logger. The warning gives the first such gold token next to the feature strings of the first five seed rows with its surface. Trainer::max_grouping_len caps unknown-word grouping with the runtime's meaning (characters beyond the first; None, the default, leaves it unbounded); the other builder methods set the regularization (L2 with DEFAULT_LAMBDA, 1.0, by default; use_l2(false) selects L1), the iteration cap, the tolerance of the convergence test (Trainer::eta, default DEFAULT_ETA, 5e-5; see the lindera-crf architecture) and the thread count. After training it prints why the optimizer stopped, with the number of iterations and of evaluations of the loss and its gradient. When an iteration left the objective unchanged or the line search failed far from the optimum, it then writes a warning on stderr, with eprintln! as for the warning about seed words.

Model / SerializableModel

Model is the trained model produced by Trainer::train. It owns the trained lindera_crf::RawModel, the TrainerConfig used to train it, the extracted feature weights and labels, the feature strings of the virtual labels, and any user lexicon entries added afterward via read_user_lexicon. It converts learned CRF weights into MeCab-compatible integer costs (tocost(weight, cost_factor) = clamp(round(-weight * cost_factor), i16::MIN, i16::MAX), i.e. clamped to [-32768, 32767]), with the cost factor computed to make full use of the i16 range (calculate_cost_factor). Model can serialize itself (write_model) and export dictionary source files directly.

SerializableModel is the plain-data, rkyv-serializable counterpart returned by Model::read_model when loading a previously trained model back from a reader (as done by the lindera export CLI command). It carries the same trained information as Model -- feature weights, labels, POS info, connection cost matrix, the unk.def entries (unk_entries, (category, feature string) pairs in file order), preserved char.def/feature.def/rewrite.def content, cost factor, and left/right context ID maps -- as owned data, without a TrainerConfig, and exposes its own writer methods for producing the dictionary export files.

Model public methods

SerializableModel public methods

Available on the value returned by Model::read_model, for re-exporting dictionary source files from a previously trained and serialized model (used by the lindera export CLI command and the export functions of the Python, Node.js, Ruby and PHP bindings):

ModelMetadata and FeatureSetInfo

ModelMetadata records summary information about a training run: version (the model format version, MODEL_FORMAT_VERSION), regularization (the cost coefficient used), regularization_type (the RegularizationType used, exported as model_info.regularization_type), elastic_net_l1_ratio (the L1 ratio, None unless the type is Elastic Net; exported for Elastic Net only), iterations (the number of L-BFGS iterations that ran, at most the maximum set with Trainer::max_iter; exported as model_info.training_iterations), feature_count, and label_count. It is embedded in SerializableModel::metadata.

FeatureSetInfo holds the per-label connection information extracted from the merged CRF model after training: left_id, right_id, and weight. SerializableModel::feature_sets stores one entry per CRF label: one for each entry of labels, in the same order, then one for each virtual label, which is not exported.

API Reference

The API reference is available. Please see following URL:

• lindera-trainer

Lindera Library

The lindera crate is the facade of the Rust API: it re-exports the morphological segmenter from lindera-segmenter (lindera::segmenter, lindera::dictionary, lindera::mode, ...) and, with the default analysis feature, the analysis chain from lindera-analysis as lindera::analysis. lindera = "7" is the only dependency you need; default-features = false gives you the segmenter alone. This section covers segmentation, error handling, and API reference.

If you need the Tokenizer, character filters, or token filters (a Lucene-style analysis chain built on top of Segmenter), use lindera::analysis — see the Lindera Analysis chapter, including its Configuration and Filters pages.

• Segmenter - Core segmentation component using the Viterbi algorithm
• Error Handling - Error types and handling patterns
• API Reference - Links to generated API documentation

Segmenter

The Segmenter is the core component that performs morphological analysis. It uses the Viterbi algorithm to find the optimal segmentation of input text based on a dictionary and cost model. Among dictionary entries that share the surface, the context IDs and the cost, and so tie, it chooses the first CSV row, as MeCab does, and a user-dictionary entry over a tied system entry (see Tied entries). Of two segmentations of equal cost that end at the same position, it chooses the one whose last word starts later, as MeCab does.

Creating a Segmenter

A Segmenter requires three components:

• Mode - the tokenization strategy (Normal or Decompose)
• Dictionary - a system dictionary for morphological analysis
• UserDictionary (optional) - a supplementary dictionary for custom words

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
}

Segmenter::new does not check a user dictionary against the system dictionary. Segmenter::try_new takes the same arguments and returns an error when an entry's context ID lies outside the system dictionary's connection matrix; Segmenter::new accepts such an entry, and segmentation then panics when the entry matches the input (see Context IDs). Building from a configuration checks too. load_user_dictionary_for loads a user dictionary for a loaded system dictionary and checks a .csv already as it builds it, so the error names the row's CSV line and surface; try_new names an entry of a .bin, or of a user dictionary loaded with load_user_dictionary and metadata only, by its word id.

#![allow(unused)]
fn main() {
use lindera::dictionary::{load_dictionary, load_user_dictionary_for};
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("embedded://ipadic")?;
let user_dictionary = load_user_dictionary_for("userdic.csv", &dictionary)?;
let segmenter = Segmenter::try_new(Mode::Normal, dictionary, Some(user_dictionary))?;
}

Tokenization Modes

Mode::Normal

Standard tokenization based on the dictionary entries. Words are segmented faithfully according to what is registered in the dictionary.

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Normal;
}

Mode::Decompose

Decomposes compound nouns into their constituent parts. This mode applies a configurable penalty to long compound words, encouraging the segmenter to split them into shorter components.

For example, with Mode::Normal, the compound word "関西国際空港" in the sentence "関西国際空港限定トートバッグ" remains part of a single token, while with Mode::Decompose, it is split into "関西", "国際", and "空港".

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Decompose(Default::default());
}

Dictionary Loading

Lindera provides the load_dictionary function to load dictionaries from various sources.

Embedded Dictionaries

When built with the appropriate feature flag (e.g., embed-ipadic), dictionaries can be loaded directly from the binary:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;
}

Available embedded dictionary URIs:

• embedded://ipadic - IPADIC (Japanese)
• embedded://ipadic-neologd - IPADIC NEologd (Japanese)
• embedded://unidic - UniDic (Japanese)
• embedded://ko-dic - ko-dic (Korean)
• embedded://cc-cedict - CC-CEDICT (Chinese)
• embedded://jieba - Jieba (Chinese)

External Dictionaries

Pre-built dictionary directories can be loaded from the filesystem:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("/path/to/dictionary")?;
}

Using with Tokenizer

The Segmenter is typically used through the Tokenizer, which adds support for character filters and token filters:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let tokens = tokenizer.tokenize(text)?;

    for mut token in tokens {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Note the mut binding on token: Token::details takes &mut self, so iterating with a plain for token in tokens fails to compile (E0596: cannot borrow as mutable).

Building from Config

Segmenter::from_config builds a Segmenter from a SegmenterConfig (a serde_json::Value), the same configuration format used by Tokenizer/TokenizerBuilder (see Configuration) but scoped to just the segmenter: section:

#![allow(unused)]
fn main() {
use serde_json::json;
use lindera::segmenter::{Segmenter, SegmenterConfig};

let config: SegmenterConfig = json!({
    "mode": "normal",
    "dictionary": "embedded://ipadic",
    "keep_whitespace": false,
    "use_mmap": false
});
let segmenter = Segmenter::from_config(&config)?;
}

Note: use_mmap here is set to false only to illustrate the option explicitly; omitting it entirely gives the same default (true, see below).

Segmenter::from_config_with_dictionaries applies a configuration to dictionaries that are already loaded, such as ones loaded from bytes. A dictionary passed as Some is used instead of the one the dictionary key names, and the key may then be absent (use_mmap does not apply); a user dictionary passed as Some is used instead of the one the user_dictionary key names. For None, the dictionary is loaded from the configuration as from_config loads it, and a configured user dictionary is loaded for the system dictionary in use with load_user_dictionary_for: a .csv is built with its metadata and checked against its connection matrix. Every other key applies as with from_config:

#![allow(unused)]
fn main() {
use serde_json::json;
use lindera::dictionary::load_dictionary;
use lindera::segmenter::{Segmenter, SegmenterConfig};

let dictionary = load_dictionary("embedded://ipadic")?;
let config: SegmenterConfig = json!({
    "mode": "normal",
    "keep_whitespace": true
});
let segmenter = Segmenter::from_config_with_dictionaries(&config, Some(dictionary), None)?;
}

Tokenizer::from_config_with_dictionaries and TokenizerBuilder::build_with_dictionaries do the same for a tokenizer configuration, filters included (see Architecture).

Memory-Mapped Loading

For a filesystem-based (not embedded://) dictionary, use_mmap defaults to true whenever the mmap Cargo feature is compiled in (the default) — memory-mapped reads are used automatically. Set it to false to force eager, non-memory-mapped file reads instead. Every large component stays lazily paged this way: the word-list files, the connection-cost matrix, and the prefix-dictionary trie, which is walked in place over its serialized bytes. use_mmap is silently ignored for embedded:// dictionaries, since their data is already a static, zero-copy byte slice. Requires the mmap cargo feature (enabled by default).

Whitespace Handling

By default, whitespace is skipped in the lattice, as MeCab does: no word starts on a whitespace character, and the word after a run of whitespace connects directly to the word before it, with the connection cost the dictionary was trained with. Skipped whitespace is not output and is left out of token surfaces and offsets; whitespace that belongs to a dictionary entry, inside it (IPADIC-NEologd GeForce GTX, a user entry 해운대 해수욕장) or at its end (ko-dic 에듀 followed by a space), stays in that entry's token, as in MeCab. "Whitespace" is the dictionary's SPACE character category (char.def): U+0020, U+0009, U+000A, U+000B and U+000D for ko-dic, and U+0020, U+0009, U+000A and U+000B for IPADIC, IPADIC-NEologd, UniDic, SudachiDict, CC-CEDICT and Jieba. Those char.def files also map U+00D0 (Ð) to SPACE, a typo for U+000D, but a later 0x00C0..0x00FF ALPHA line takes precedence, so Ð is a letter, as in MeCab.

The default comes from the dictionary's metadata.json (skip_whitespace). SudachiDict sets it to false: Sudachi keeps whitespace in the lattice and outputs it as 空白 tokens, and SudachiDict's costs are tuned for that, so skipping would, for example, split the entry caramel man into two common nouns. With SudachiDict, whitespace therefore stays in the lattice by default, as a node of its own (a 空白 dictionary entry, such as the one for U+0020, or the SPACE unknown word), though it is still dropped from the output. The other bundled dictionaries leave the setting out, which means skipping, and so does a dictionary built before the setting existed (a SudachiDict built by Lindera 6.x included, until it is rebuilt).

Call keep_whitespace(true) on a Segmenter to output whitespace tokens. Whitespace then also stays in the lattice, as a node of its own (a whitespace dictionary entry, such as IPADIC's or UniDic's for U+3000, or the SPACE unknown word), so the other tokens can differ from those of the default output, which skips whitespace:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).keep_whitespace(true);
}

Before v7, Lindera kept whitespace in the lattice even when it dropped it from the output, so the words on either side of a space connected to the whitespace node instead of to each other. How much that mattered depends on the dictionary's connection costs for the SPACE unknown word. In ko-dic its row and column of the connection matrix are all zero, so every space reset the context and a word between spaces was chosen by its word cost alone: ko-dic read 전 in 2년 전 대회 as 저/NP + ㄴ/JX instead of the noun NNG. In IPADIC and UniDic the costs are not zero (IPADIC's SPACE unknown word shares its context with the full-width space entry 記号,空白), and IPADIC read が in Google が 新しい as a conjunction. CC-CEDICT has no connection costs at all, so its segmentation does not depend on this, although the path costs that N-best search reports do. Jieba's connection costs are trained (mecab-jieba 0.1.3), but on the 247 sentences with a space of UD Chinese GSD and of the GSDSimp dev and test splits, its segmentation is the same either way. skip_whitespace overrides the dictionary's default: skip_whitespace(false) keeps whitespace nodes in the lattice as before, while still dropping whitespace from the output, and skip_whitespace(true) turns skipping on for SudachiDict. In a SegmenterConfig the skip_whitespace key takes true or false, and leaving it out, or null, keeps the dictionary's default; on the CLI, --disable-skip-whitespace turns skipping off:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).skip_whitespace(false);
}

A \n or a \t still ends a segment (see Sentence Splitting), so the context is not carried across a newline or a tab. mecab-ko analyzes a line at a time and skips a tab inside it like any other whitespace.

Unknown-Word Grouping

A word that is not in the dictionary is read as an unknown word, built from the categories that the dictionary's char.def gives its characters (katakana, Latin letters, digits, symbols and so on). A character can have more than one category, such as a kanji numeral (KANJINUMERIC KANJI in IPADIC); the first category of the char.def line that decides its categories is its default category. At every position that a path through the lattice reaches, Lindera creates unknown-word candidates from the default category of the character there, if that category is set to always create them (INVOKE in char.def) or if no dictionary word starts at that position, as MeCab does. If the category is set to group (GROUP), one candidate is a grouped unknown word: the run of characters from that position on in which each character shares a category with the one before it, such as a run of katakana.

With ko-dic, the numeral 三 is HANJANUMERIC HANJA (default category HANJANUMERIC, which groups) and the other hanja are HANJA, so the grouped unknown word that starts at 三 runs through the hanja after it: 三國史記 is one unknown word (SH), as in MeCab. Before v7, Lindera created candidates for every category of a character and went on with a run only while the next character had the same category at the same position of its category list, so it returned 三 / 國史 / 記.

An unknown word can start at any such position, also inside a run that a grouped unknown word from an earlier position covers. IPADIC's char.def puts ・ in the katakana range, so in ジョン・レノンが歌う the first position has the grouped unknown word ジョン・レノン. The dictionary words ジョン and ・ reach the position of レ, where the unknown word レノン starts, and the result is ジョン, ・, レノン, が and 歌う, as in MeCab. Before v7, normal mode created no unknown words at the positions inside the last grouped unknown word, a shortcut inherited from Kuromoji, so it returned ジョン・レノン as one unknown word; Decompose mode never took this shortcut.

Unknown-word grouping is unbounded by default. max_grouping_len(Some(n)) caps it with MeCab's max-grouping-size semantics, counting characters beyond the first (MeCab defaults to 24):

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).max_grouping_len(Some(24));
}

The cap is applied at each lattice position rather than to a run as a whole. When the run starting at a position is longer than the cap, the grouped candidate is not emitted there; the single-character candidate (plus the length-ladder candidates described below and any dictionary words) remains, and the remaining tail is grouped again once it fits. No unknown token is therefore longer than n + 1 characters, but an over-long run does not turn into single characters throughout. With IPADIC, the out-of-vocabulary katakana run ヷヸヹヺヷ and max_grouping_len(Some(2)) segment as ヷ / ヸ / ヹヺヷ when the length ladder is off: the first two positions see runs of 5 and 4 characters (over the cap) and keep only the single-character candidate, while the 3-character tail fits the cap and groups. With the ladder on (the default), the 2-character ladder candidate wins at the first position and the output is ヷヸ / ヹヺヷ.

Some(0) never groups a run of two or more characters. The max_grouping_len config key and the CLI flag --max-grouping-len treat 0 as unbounded instead.

Independently of grouping, Lindera also generates a MeCab/Vibrato-inspired "length ladder" of shorter unknown-word candidates by default, so the Viterbi search can pick whichever length scores lowest. The candidates are one character long and longer, up to the char.def LENGTH field of the default category, over the characters that share a category with the first one; as in MeCab, each character is compared with the first one, not with the one before it. Call unknown_word_ladder(false) to disable it: a position then gets only the grouped or the single-character candidate, as before v6. The output still differs from that of Lindera before v6 wherever later changes apply, such as unknown words that start inside a grouped run (see Migrating from v6 to v7):

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).unknown_word_ladder(false);
}

Left-Space Penalty (Korean)

MeCab-based Korean analyzers (mecab-ko with mecab-ko-dic, Lucene's nori) add a cost to a candidate that starts right after whitespace when its part-of-speech tag is one that attaches to the preceding word without a space: particles (J*), endings (E*), the copula (VCP) and derivational suffixes (XS*). Without it, 검색 이 잘 된다 reads 이 as the subject particle JKS rather than the interjection IC. Lindera applies the same penalty by default for dictionaries that ship the rules in their metadata (ko-dic does); other dictionaries are unaffected. Opt out with Segmenter::space_penalty(None), "space_penalty": false in the config, or --disable-space-penalty on the CLI. Together with skip_whitespace(false), this turns off both changes since v6.0 in how Korean is read around spaces (this penalty and whitespace skipping), but not the other output changes since then, such as the split of punctuation runs like ." (see Migrating from v6 to v7).

SpacePenaltyConfig is a list of rules, each pairing first part-of-speech tags with a cost. A candidate is matched by the part of its tag before the first + (for ko-dic Inflect rows, the first_part_of_speech column); the first matching rule wins and unlisted tags cost nothing. mecab-ko-dic's dicrc rules translate to:

#![allow(unused)]
fn main() {
use lindera::space_penalty::{SpacePenaltyConfig, SpacePenaltyRule};

let rules = SpacePenaltyConfig::new(vec![
    SpacePenaltyRule::new(["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], 3000),
    SpacePenaltyRule::new(["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], 6000),
]);
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).space_penalty(Some(rules))?;
}

"Whitespace" means a character of the dictionary's SPACE category (char.def), the same set that is skipped in the lattice and that keep_whitespace filters on; a dictionary without that category falls back to Unicode White_Space. The penalty applies in both modes and in N-best search, to system, user and unknown-word entries alike. space_penalty builds a per-word-id lookup once (a few tens of milliseconds for ko-dic), and returns an error when the dictionary schema has neither a part_of_speech_tag nor a part_of_speech field.

A dictionary can ship its default rules in metadata.json under space_penalty; ko-dic does, with exactly the rules above, and Segmenter::new applies them automatically. space_penalty_from_dictionary() re-enables them after an opt-out and returns an error for a dictionary that ships none. Shipped rules that cannot be applied, because the schema has no part-of-speech field, only log a warning in Segmenter::new:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).space_penalty_from_dictionary()?;
}

Only a ko-dic built by Lindera 6.1.0 or later carries these rules. A ko-dic downloaded from the v6.0.0 release ships none, so with it the penalty silently stays off, and space_penalty_from_dictionary() (or "space_penalty": true) returns an error until the dictionary is rebuilt. Explicit rules (space_penalty, --space-penalty-rules) work with any ko-dic.

In a SegmenterConfig, the space_penalty key takes an object for explicit rules, false for off, and true for the dictionary's rules (an error when it ships none). Leaving the key out, or null, keeps the default: the dictionary's rules when it ships any.

{
  "dictionary": "embedded://ko-dic",
  "space_penalty": {
    "rules": [
      { "pos": ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], "cost": 3000 },
      { "pos": ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], "cost": 6000 }
    ]
  }
}

A candidate pays the penalty when whitespace that is not part of the word before it separates the two, which is mecab-ko's test (it penalizes a node that skipped whitespace, rlength > length). So a word after leading whitespace pays it, and a word right after an entry whose surface ends with the whitespace does not: ko-dic has four such entries (내셔날, 에듀, 캘리 and 콜라비, each followed by a space), and 에듀 이 reads as that entry and 이/VCP, as in mecab-ko, while 에듀 이, with a second space the entry does not end with, reads 이/NR. The test is the same whether whitespace is skipped in the lattice (the default, see Whitespace Handling) or kept as nodes, where a word after a whitespace node pays the penalty. With whitespace skipped, the connection costs on their own already rule out some readings the penalty targets (시 in 서울 시 에서 reads as NNG without it), but not all of them (이 in 검색 이 잘 된다).

N-Best Segmentation

segment_nbest returns the top-n segmentations ordered by total path cost, each paired with its cost. Set unique to deduplicate results that share the same word boundaries but differ only in POS tags (each result is the cheapest path of its word boundaries, and the search does not go through the others, so it stays fast when the words of a line have many entries), and cost_threshold to discard paths whose cost exceeds best_cost + threshold:

#![allow(unused)]
fn main() {
let results = segmenter.segment_nbest(Cow::Borrowed("すもももももももものうち"), 3, false, None)?;
for (tokens, cost) in results {
    println!("cost={cost}");
    for token in tokens {
        println!("  {}", token.surface.as_ref());
    }
}
}

Every result segments the whole input. The input is split into segments as described in Sentence Splitting, and each segment is searched on its own. Within a segment, the context is carried across 、, 。 and forced cuts, so the paths of a segment are the n best paths of one lattice over the whole segment, with the exceptions listed there; unique compares the word boundaries of the whole segment. A result takes one path in each segment, its cost is the sum of those paths' costs, and the results are the n cheapest of these combinations. So a result can differ from the best one in a single segment. best_cost is the cost of the first result, so cost_threshold applies to the whole input. The first result is always the output of segment. An input without words gives one result without tokens, with the connection cost from BOS to EOS, as in MeCab: an input of skipped whitespace and, since v7, an empty input (which gave no result before).

segment_nbest_with_lattice is the same operation but lets you pass in a reusable Lattice buffer to avoid reallocating one per call.

Sentence Splitting

Before building the lattice, Lindera splits the input into sentences at delimiter characters (\n, \t, 。, 、) and builds one Viterbi lattice per sentence. If no delimiter appears within roughly 32 KiB of a sentence's start, Lindera forces a sentence boundary there anyway and logs a warning. The cuts bound the size — and therefore the memory and CPU cost — of each lattice.

A \n, a \t or the end of the input ends a segment, the run of sentences up to it. Segments are independent of each other. Within a segment, the cuts at 、 and 。 and the forced cuts carry the context: the next sentence starts from the words that end the previous one (from their right context IDs, each with the cost of the best path to it) rather than from the beginning of a sentence (BOS), and the connection to the end of the sentence (EOS) is paid only at the end of the segment. So the best path of a segment, and its cost, are those of one lattice over the whole segment, as MeCab, which does not cut inside a line, finds them. Before v7, every sentence started from BOS and ended with EOS, so the word after 、 lost its context: with IPADIC, ああ in 彼は、ああ言った was an interjection, and it is now an adverb, as in MeCab.

The result can still differ from one lattice over the segment:

• No word spans a cut: a dictionary entry that contains 、 or 。, such as UniDic's 一、二塁, never matches, an unknown word is not grouped across a cut, and a word that would span a forced cut is split there.
• The left-space penalty (see Left-Space Penalty (Korean)) does not apply to a word that starts right at a forced cut, even when whitespace precedes the cut.
• A sentence without a path, which only a dictionary that leaves some character without an unknown-word candidate gives, ends the segment before it: the best path up to the cut is kept as a path that ends there, the EOS connection included, and the sentence is segmented on its own. The first N-best result keeps the same path.

Ordinary text never reaches the forced cut, but for pathological delimiter-free input (e.g. minified text or base64-encoded blobs), it can affect tokenization at the artificial cut point.

N-best search uses the same segments; see N-Best Segmentation.

Reusable Worker

SegmentWorker is a reusable segmentation session that owns the Viterbi lattice and the scratch buffers of the segmentation, so repeated calls avoid the per-call allocations segment pays. Create one with new_worker (clones the segmenter) or into_worker (consumes it, avoiding a user-dictionary copy), then call segment/segment_nbest on it:

#![allow(unused)]
fn main() {
let mut worker = segmenter.new_worker();
for line in lines {
    let tokens = worker.segment(line)?;
    for token in &tokens {
        println!("{}", token.surface.as_ref());
    }
}
}

The returned tokens borrow the worker, so they must be consumed before the next call (the usual per-line loop above compiles as-is). set_mode, set_keep_whitespace, set_skip_whitespace, set_max_grouping_len, set_unknown_word_ladder, set_space_penalty and set_space_penalty_from_dictionary switch the configuration between calls. segmenter() returns a shared reference to the underlying segmenter; no &mut accessor is provided, since swapping the dictionary out from under the reused lattice would break the dictionary-lattice pairing the worker exists to guarantee.

A worker also bounds retained memory: one delimiter-free maximum-length sentence grows the lattice to several MB (roughly 18 MB for a 32 KiB ASCII sentence, 6 MB for a 32 KiB CJK one, whose character-indexed lattice has a third of the slots), and a plain Lattice keeps that forever. The worker automatically shrinks its lattice, and releases its scratch buffers, once a window of calls shows the capacity is oversized, and shrink_to(text_len_hint) forces a shrink immediately. reset() discards the internal buffers outright and replaces them with fresh ones; it is intended for recovery paths (e.g. after a panic poisoned a mutex holding the worker) where the buffers may hold an inconsistent intermediate state — the segmenter configuration itself is preserved.

The worker is permanently bound to the dictionary of the segmenter that created it; there is no way to swap dictionaries under a live worker, which rules out a class of lattice-reuse bugs by construction. For multi-threaded use, create one worker per thread from a shared Segmenter.

Error Handling

Lindera uses a structured error system based on anyhow and thiserror for ergonomic error handling throughout the library.

LinderaResult

The LinderaResult<T> type alias is the standard return type for fallible operations in Lindera:

#![allow(unused)]
fn main() {
pub type LinderaResult<T> = Result<T, LinderaError>;
}

LinderaError

LinderaError is the main error type, containing an error kind and a source error with full context:

#![allow(unused)]
fn main() {
pub struct LinderaError {
    pub kind: LinderaErrorKind,
    source: anyhow::Error,
}
}

The add_context method allows attaching additional context to an error:

#![allow(unused)]
fn main() {
let error = error.add_context("failed to load dictionary from /path/to/dict");
}

LinderaErrorKind

LinderaErrorKind is an enum that categorizes errors:

Creating Errors

Use LinderaErrorKind::with_error to create an error from a kind and a source:

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let error = LinderaErrorKind::Io.with_error(anyhow::anyhow!("file not found: config.yml"));
}

Using the ? Operator

Since Lindera functions return LinderaResult, the ? operator can propagate errors naturally:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn analyze(text: &str) -> LinderaResult<Vec<String>> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let tokens = tokenizer.tokenize(text)?;
    Ok(tokens.iter().map(|t| t.surface.as_ref().to_string()).collect())
}
}

Error Handling Patterns

Matching on Error Kind

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::error::LinderaErrorKind;

match load_dictionary("/path/to/dictionary") {
    Ok(dict) => { /* use dictionary */ }
    Err(e) if e.kind() == LinderaErrorKind::NotFound => {
        eprintln!("Dictionary not found: {}", e);
    }
    Err(e) if e.kind() == LinderaErrorKind::Io => {
        eprintln!("I/O error loading dictionary: {}", e);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}
}

Converting from External Errors

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let content = std::fs::read_to_string("config.yml")
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))?;
}

API Reference

The API reference is available. Please see following URL:

• lindera
• lindera-segmenter
• lindera-analysis

Lindera Analysis

Lindera Analysis layers a Lucene-style text analysis chain on top of the morphological segmenter provided by the lindera-segmenter crate. It is normally used through the lindera facade as lindera::analysis (enabled by the default analysis feature). It composes character filters, a Segmenter, and token filters into a single Tokenizer pipeline that can be built programmatically or loaded entirely from a YAML configuration file.

Key Features

• Character filters that transform input text before segmentation, with automatic byte-offset correction back to the original text
• Token filters that transform, merge, filter, or reorder the tokens produced by the segmenter
• Tokenizer / TokenizerBuilder to assemble a full analysis pipeline, either in Rust code or from a YAML file (LINDERA_CONFIG_FILE)
• Built-in filters covering Japanese, Korean, and general-purpose text normalization

Dependency

Through the facade (recommended):

[dependencies]
lindera = "7"

#![allow(unused)]
fn main() {
use lindera::analysis::tokenizer::Tokenizer;
}

As a direct dependency. It is the same crate, so both paths name the same types; keep lindera and lindera-analysis at the same major version (see Migration v6 to v7):

[dependencies]
lindera = "7"
lindera-analysis = "7"

#![allow(unused)]
fn main() {
use lindera_analysis::tokenizer::Tokenizer;
}

Contents

• Configuration -- YAML configuration file format for the Tokenizer
• Filters -- Reference for all character filters and token filters
• Architecture -- Internal structure and key components
• API Reference -- API documentation

Configuration

Lindera is able to read YAML format configuration files. Specify the path to the following file in the environment variable LINDERA_CONFIG_FILE. You can use it easily without having to code the behavior of the tokenizer in Rust code.

segmenter:
  mode: "normal"
  dictionary: "embedded://ipadic"
  # user_dictionary: "./resources/user_dict/ipadic_simple_userdic.csv"
  # keep_whitespace: false
  # skip_whitespace: false # skip whitespace in the lattice, as MeCab does; on by default except for SudachiDict. false keeps whitespace nodes (pre-v7)
  # use_mmap: false # only meaningful for filesystem (non-embedded://) dictionaries
  # max_grouping_len: 24 # cap on unknown-word grouping; omit or 0 for unbounded
  # unknown_word_ladder: true # emit shorter unknown-word candidates (default: true)
  # space_penalty: false # left-space penalty (Korean); on by default for dictionaries that ship rules (ko-dic). false turns it off, an object gives explicit rules

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true
  - kind: mapping
    args:
       mapping:
         リンデラ: Lindera

token_filters:
  - kind: "japanese_compound_word"
    args:
      # Merge the numeral tokens only. A counter merged into the number token keeps japanese_number from converting it.
      tags:
        - "名詞,数"
      new_tag: "名詞,数"
  - kind: "japanese_number"
    args:
      tags:
        - "名詞,数"
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "接続詞"
        - "助詞"
        - "助詞,格助詞"
        - "助詞,格助詞,一般"
        - "助詞,格助詞,引用"
        - "助詞,格助詞,連語"
        - "助詞,係助詞"
        - "助詞,副助詞"
        - "助詞,間投助詞"
        - "助詞,並立助詞"
        - "助詞,終助詞"
        - "助詞,副助詞／並立助詞／終助詞"
        - "助詞,連体化"
        - "助詞,副詞化"
        - "助詞,特殊"
        - "助動詞"
        - "記号"
        - "記号,一般"
        - "記号,読点"
        - "記号,句点"
        - "記号,空白"
        - "記号,括弧閉"
        - "その他,間投"
        - "フィラー"
        - "非言語音"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "remove_diacritical_mark"
    args:
      japanese: false

Segmenter Options

% export LINDERA_CONFIG_FILE=./resources/config/lindera.yml

use std::path::PathBuf;

use lindera::analysis::tokenizer::TokenizerBuilder;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    // Load tokenizer configuration from file
    let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("config")
        .join("lindera.yml");

    let builder = TokenizerBuilder::from_file(&path)?;

    let tokenizer = builder.build()?;

    let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。".to_string();
    println!("text: {text}");

    let tokens = tokenizer.tokenize(&text)?;

    for token in tokens {
        println!(
            "token: {:?}, start: {:?}, end: {:?}, details: {:?}",
            token.surface, token.byte_start, token.byte_end, token.details
        );
    }

    Ok(())
}

Filters

Character filters and token filters are the two pre/post-processing stages of the lindera-analysis Tokenizer pipeline.

• Character filters transform the input text before segmentation. Byte offsets are corrected automatically so that the resulting tokens still report positions relative to the original, unfiltered text.
• Token filters transform the list of tokens produced by the segmenter after segmentation.

Both kinds of filters are configured the same way: a kind string (also used with the CLI's --character-filter / --token-filter flags in the form kind:{"json": "args"}) and a JSON args object with filter-specific parameters.

Character Filters

Character filters are configured under the character_filters key of the YAML configuration file. Each entry is applied in order, and each filter's output feeds into the next.

unicode_normalize

Normalizes the input text using one of the four standard Unicode normalization forms.

Parameters:

Example:

{
  "kind": "unicode_normalize",
  "args": {
    "kind": "nfkc"
  }
}

japanese_iteration_mark

Normalizes Japanese iteration marks (々, ゝ, ゞ, ヽ, ヾ) by replacing each mark with the character it repeats, adding or removing the voiced sound mark (dakuten) as needed for the hiragana/katakana variants.

Parameters:

Example:

{
  "kind": "japanese_iteration_mark",
  "args": {
    "normalize_kanji": true,
    "normalize_kana": true
  }
}

mapping (character filter)

Replaces occurrences of the keys in mapping with their corresponding values, using longest-match search over the input text (built with an Aho-Corasick automaton).

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "リンデラ": "Lindera"
    }
  }
}

regex

Replaces every match of a regular expression in the input text with a literal replacement string. Capture groups are not interpolated into the replacement.

Parameters:

Example:

{
  "kind": "regex",
  "args": {
    "pattern": "\\s{2,}",
    "replacement": " "
  }
}

Token Filters

Token filters are configured under the token_filters key of the YAML configuration file. Each filter is applied in order to the token list produced by the segmenter.

japanese_base_form

Replaces the token's surface text with its base (dictionary) form, as registered in the base_form or orthographic_base_form field of the dictionary. Acts as a lemmatizer for verbs and adjectives. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_base_form"
}

japanese_compound_word

Merges consecutive tokens whose part-of-speech tag matches one of tags into a single compound token.

The merged token's details carry new_tag at the part-of-speech position resolved from the dictionary schema; any leading non-part-of-speech field (SudachiDict's display_surface) and every remaining field are set to *.

Parameters:

Example:

{
  "kind": "japanese_compound_word",
  "args": {
    "tags": [
      "名詞,数",
      "名詞,接尾,助数詞"
    ],
    "new_tag": "名詞,数"
  }
}

japanese_kana

Converts token text between hiragana and katakana.

Parameters:

Example:

{
  "kind": "japanese_kana",
  "args": {
    "kind": "hiragana"
  }
}

japanese_katakana_stem

Removes a trailing prolonged sound mark (ー, U+30FC) from katakana tokens, but only when the token is longer than min characters.

Parameters:

Example:

{
  "kind": "japanese_katakana_stem",
  "args": {
    "min": 3
  }
}

japanese_keep_tags

Keeps only tokens whose part-of-speech tag matches one of tags, removing all others.

Parameters:

Example:

{
  "kind": "japanese_keep_tags",
  "args": {
    "tags": [
      "名詞,一般",
      "名詞,固有名詞,一般"
    ]
  }
}

japanese_number

Converts Japanese numeral representations (kanji numerals, formal/legal kanji numerals, and fullwidth digits) in the token's surface text to Arabic numerals.

A token is converted only when every one of its characters takes part in a numeral, the rule Lucene's JapaneseNumberFilter applies. An ordinary word that merely begins with a numeral character is therefore returned unchanged: 一部 stays 一部 rather than becoming 1部.

When this filter follows japanese_compound_word, merge only the 名詞,数 tokens. A counter (名詞,接尾,助数詞) merged into the number token makes the merged token something other than a numeral, so 一万円 is left alone instead of being converted.

Parameters:

Example:

{
  "kind": "japanese_number",
  "args": {
    "tags": [
      "名詞,数"
    ]
  }
}

japanese_reading_form

Replaces the token's surface text with its reading, in katakana, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_reading_form"
}

japanese_stop_tags

Removes tokens whose part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "japanese_stop_tags",
  "args": {
    "tags": [
      "助詞,格助詞,一般",
      "助詞,係助詞",
      "助詞,連体化",
      "助動詞"
    ]
  }
}

Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*). The same matching rule applies to japanese_keep_tags, japanese_compound_word, and the tags option of japanese_number. Because the position is resolved through the schema, the rule also holds for dictionaries whose schema has leading non-part-of-speech columns, such as SudachiDict's display_surface.

keep_words

Keeps only tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "keep_words",
  "args": {
    "words": [
      "すもも",
      "もも"
    ]
  }
}

korean_compound_word

Merges consecutive Korean tokens whose first part-of-speech tag matches one of tags into a single compound token. The merged token keeps the first token's byte_start and position, takes the last token's byte_end, and its position_length is the number of tokens merged. A matching token whose neighbours do not match is left as it is.

ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so this filter is what turns 이/NR 천/NR 이/NR 십/NR 육/NR back into one token before korean_number converts it:

Parameters:

Example:

{
  "kind": "korean_compound_word",
  "args": {
    "tags": [
      "SN",
      "NR"
    ],
    "new_tag": "NR"
  }
}

With the default settings, ko-dic tags 만 in 만 원 and 십만 원 as the numeral NR, so 십만 원 becomes 십만/NR 원/NNBC. With whitespace skipping turned off (skip_whitespace: false), the word after 만 connects to the whitespace node instead, and ko-dic tags 만 there as the particle JX (만/JX 원/NNG), which stays outside ["SN", "NR"].

korean_decompound

Splits ko-dic compound, inflected and pre-analyzed tokens into the morphemes their expression field lists, reproducing Lucene nori's DecompoundMode. This is the inverse of korean_compound_word, which merges neighbouring tokens. Lindera's Mode::Decompose applies a length penalty tuned for Japanese kanji and does not read these fields, so 무궁화 stays whole in every mode without this filter.

Fragment offsets follow nori: only Compound fragments get their own byte range, each covering the text it spells. Inflect and Preanalysis fragments share the whole token's span, because an inflected form's fragments need not be substrings of it (갔 decomposes to 가 + 았).

A Compound token is carved only when its fragments spell its surface exactly; otherwise it shares its span too. That covers the 12 bundled ko-dic entries whose fragments are not the surface's own text (안지랑역 = 안지 + 역, 고춧값 = 고추 + 값), where nori's offsets point some fragments at text they do not spell, as well as a user-dictionary entry such as 프린터3D = 프린터 + 쓰리디 and a surface an earlier filter has rewritten. For every other bundled entry the offsets are nori's, and every offset is safe to slice the text with.

In discard the fragments take consecutive positions starting at the compound's own; in mixed the compound keeps that position with a position_length spanning its fragments, which are then emitted from the same position onward. Tokens after the expansion shift by the positions it added, and only by those, so a gap left by an earlier filter such as korean_stop_tags survives.

Only tokens spanning a single position are expanded, so the compound a mixed pass keeps is not split again. Put the filter in a chain once all the same: a single-fragment entry kept by mixed still spans one position. When combining it with korean_compound_word, place the merge first; merging never joins tokens that overlap in byte range, so the reverse order does not corrupt anything, but merging first lets it see the segmenter's own tokens.

A fragment takes its own part-of-speech tag and semantic class and blanks the fields that describe the compound (type, expression, first_part_of_speech, last_part_of_speech, reading, presence_absence). Blanking the reading matches nori, whose DecompoundToken.getReading() returns null; korean_reading_form skips *, so a fragment keeps its own surface.

Applied to a dictionary whose schema has no type/expression fields, the filter warns once per configured filter and passes every token through unchanged.

Parameters:

Example:

{
  "kind": "korean_decompound",
  "args": {
    "mode": "discard",
    "types": [
      "Compound",
      "Inflect",
      "Preanalysis"
    ]
  }
}

korean_keep_tags

Keeps only Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_keep_tags",
  "args": {
    "tags": [
      "NNG"
    ]
  }
}

korean_number

Converts Sino-Korean numeral representations (Hangul numerals, Hanja numerals, and fullwidth digits) in the token's surface text to Arabic numerals. Native Korean numerals (하나, 둘, 열, 스물, ...) have no positional structure and are left unchanged.

Each token is converted on its own. ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so the tokens of a single number do not add up to one Arabic numeral:

Put korean_compound_word in front of this filter to merge those tokens first, the way japanese_compound_word does for IPADIC. Its new_tag must be one of this filter's tags for the merged token to be converted:

[
  {"kind": "korean_compound_word", "args": {"tags": ["SN", "NR"], "new_tag": "NR"}},
  {"kind": "korean_number", "args": {"tags": ["SN", "NR"]}}
]

With that chain 이천이십육년 becomes 2026 년, 10만 becomes 100000 and 2천26 becomes 2026. A single token that holds the whole numeral, such as the Hanja 二千二十六/SH, is converted in full by this filter alone.

Parameters:

A token is converted only when every one of its characters is a numeral, the rule Lucene's KoreanNumberFilter applies. That is what keeps native Korean numerals intact: 일곱 ("seven") starts with the Sino-Korean 일, and a character-by-character conversion would give 1곱.

Hangul numerals are also homographs of very common morphemes (이 is a subject particle, 만 an auxiliary particle), and those are single characters that the all-numeral rule cannot separate from a numeral. That is why the default is restricted to the numeral tags.

Add SH to convert Hanja numerals as well. The coverage is partial in two ways: ko-dic tags whole Hanja words as NNG (參加, 萬歲 are untouched), while a single Hanja character it cannot attach to a word is SH on its own, so 十字架 becomes 10 字 架 and 百貨店 becomes 100 貨 店.

Example:

{
  "kind": "korean_number",
  "args": {
    "tags": [
      "SN",
      "NR"
    ]
  }
}

korean_reading_form

Replaces the token's surface text with its reading, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "korean_reading_form"
}

korean_stop_tags

Removes Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_stop_tags",
  "args": {
    "tags": [
      "EP",
      "EF",
      "JKG"
    ]
  }
}

length

Keeps only tokens whose surface text length (in characters) falls within [min, max].

Parameters:

Example:

{
  "kind": "length",
  "args": {
    "min": 2,
    "max": 3
  }
}

lowercase

Converts token surface text to lowercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "lowercase"
}

mapping (token filter)

Replaces occurrences of the keys in mapping with their corresponding values in each token's surface text, using longest-match search (built with an Aho-Corasick automaton). This is the token-level counterpart of the mapping character filter.

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "籠": "篭"
    }
  }
}

remove_diacritical_mark

Removes diacritical (combining) marks from token surface text, re-applying the text's original Unicode normalization form afterward.

Parameters:

Example:

{
  "kind": "remove_diacritical_mark",
  "args": {
    "japanese": false
  }
}

stop_words

Removes tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "stop_words",
  "args": {
    "words": [
      "も",
      "の"
    ]
  }
}

uppercase

Converts token surface text to uppercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "uppercase"
}

YAML Configuration

Character filters and token filters are configured together with the segmenter in a single YAML file. See Configuration for the full file format; the relevant excerpt looks like this:

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true

token_filters:
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "助詞,格助詞,一般"
        - "助詞,係助詞"
        - "助詞,連体化"
        - "助動詞"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "lowercase"
  - kind: "length"
    args:
      min: 2

Rust API

Character filters and token filters can also be created and applied programmatically:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::character_filter::BoxCharacterFilter;
use lindera::analysis::character_filter::unicode_normalize::{
    UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera::analysis::token_filter::BoxTokenFilter;
use lindera::analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera::analysis::token_filter::japanese_katakana_stem::JapaneseKatakanaStemTokenFilter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let mut tokenizer = Tokenizer::new(segmenter);

    // Add a character filter
    let normalize_filter = UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);
    tokenizer.append_character_filter(BoxCharacterFilter::from(normalize_filter));

    // Add token filters
    let stop_tags_filter = JapaneseStopTagsTokenFilter::new(
        vec![
            "助詞,格助詞,一般".to_string(),
            "助詞,係助詞".to_string(),
            "助詞,連体化".to_string(),
            "助動詞".to_string(),
        ]
        .into_iter()
        .collect(),
    );
    tokenizer.append_token_filter(BoxTokenFilter::from(stop_tags_filter));

    let katakana_stem_filter =
        JapaneseKatakanaStemTokenFilter::new(std::num::NonZeroUsize::new(3).unwrap());
    tokenizer.append_token_filter(BoxTokenFilter::from(katakana_stem_filter));

    // Tokenize with filters applied
    let tokens = tokenizer.tokenize("Linderaは形態素解析エンジンです。")?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

The append_character_filter and append_token_filter methods add filters in order. Character filters are applied sequentially to the text before segmentation; token filters are applied sequentially to the token list after segmentation.

Architecture

Module Structure

lindera-analysis/src/
├── lib.rs                          # Public API re-exports, CLI flag parsing helper
├── character_filter.rs             # CharacterFilter trait, OffsetMapping, CharacterFilterLoader
├── character_filter/
│   ├── unicode_normalize.rs        # Unicode normalization (NFC/NFD/NFKC/NFKD)
│   ├── japanese_iteration_mark.rs  # Japanese iteration mark normalization
│   ├── mapping.rs                  # Mapping-based text replacement
│   └── regex.rs                    # Regex-based text replacement
├── token_filter.rs                 # TokenFilter trait, TokenFilterLoader
├── token_filter/
│   ├── compound.rs                 # Shared compound-word merge helper (private)
│   ├── japanese_base_form.rs
│   ├── japanese_compound_word.rs
│   ├── japanese_kana.rs
│   ├── japanese_katakana_stem.rs
│   ├── japanese_keep_tags.rs
│   ├── japanese_number.rs
│   ├── japanese_reading_form.rs
│   ├── japanese_stop_tags.rs
│   ├── keep_words.rs
│   ├── korean_compound_word.rs
│   ├── korean_decompound.rs
│   ├── korean_keep_tags.rs
│   ├── korean_number.rs
│   ├── korean_reading_form.rs
│   ├── korean_stop_tags.rs
│   ├── length.rs
│   ├── lowercase.rs
│   ├── mapping.rs
│   ├── numeral.rs                  # Shared numeral conversion engine (private)
│   ├── remove_diacritical_mark.rs
│   ├── stop_words.rs
│   ├── tags.rs                     # Shared keep/stop-tag filtering helpers (private)
│   └── uppercase.rs
└── tokenizer.rs                     # Tokenizer, TokenizerBuilder

Key Components

CharacterFilter

A trait for filters that preprocess text before segmentation. Each implementation provides a name() and an apply(&self, text: &mut String) -> LinderaResult<OffsetMapping> method that rewrites text in place and returns an OffsetMapping describing every transformation it performed.

The OffsetMapping (built from a list of Transformation records) lets the Tokenizer translate token byte offsets computed against the filtered text back to byte offsets in the original input, even after multiple filters have run in sequence. BoxCharacterFilter wraps any CharacterFilter implementation as a boxed, cloneable trait object, and CharacterFilterLoader builds one from a kind string plus a serde_json::Value of arguments (used both by YAML configuration loading and by CLI flag parsing).

TokenFilter

A trait for filters that post-process the tokens produced by the segmenter. Each implementation provides a name() and an apply(&self, tokens: &mut Vec<Token<'_>>) -> LinderaResult<()> method that modifies, merges, reorders, or removes tokens in place. BoxTokenFilter wraps any TokenFilter implementation as a boxed, cloneable trait object, and TokenFilterLoader builds one from a kind string plus a serde_json::Value of arguments, mirroring CharacterFilterLoader.

Tokenizer / TokenizerBuilder

Tokenizer composes character filters, a lindera::segmenter::Segmenter, and token filters into a single analysis pipeline. Calling tokenize runs the character filters over the input text, segments the filtered text, applies the token filters to the resulting tokens, and finally corrects each token's byte offsets back to the original text via the recorded OffsetMappings.

TokenizerBuilder assembles a Tokenizer from a TokenizerConfig (a serde_json::Value), which can be constructed programmatically, loaded from a YAML file (via TokenizerBuilder::from_file, or automatically from the LINDERA_CONFIG_FILE environment variable via TokenizerBuilder::new), or built up incrementally with set_segmenter_mode, set_segmenter_dictionary, append_character_filter, and append_token_filter. See Configuration for the YAML file format and Filters for the full filter reference.

TokenizerBuilder::build loads the dictionaries the configuration names. TokenizerBuilder::build_with_dictionaries (and Tokenizer::from_config_with_dictionaries) takes already-loaded dictionaries instead, such as ones loaded from bytes, and applies every other setting of the configuration to them, filters included; see Building from Config for how the given dictionaries replace the configured ones. The WASM binding's TokenizerBuilder builds its tokenizers this way, with the dictionaries set by setDictionaryInstance() and setUserDictionaryInstance().

AnalysisWorker

AnalysisWorker (created with Tokenizer::new_worker or Tokenizer::into_worker) is a reusable session over the full analysis chain. It owns every per-call buffer — the Viterbi lattice and backtrace scratch (via SegmentWorker), the normalized-text buffer the character filters operate on, and the offset-mapping scratch — so repeated tokenize calls avoid the per-call allocations Tokenizer::tokenize pays. When character filters are configured, token surfaces borrow the worker's buffer instead of being copied into per-token Strings. Returned tokens borrow the worker and must be consumed before the next call; for multi-threaded use, create one worker per thread (or guard one with a Mutex, as lindera-binding does). See the Segmenter page for the underlying SegmentWorker and its automatic memory-shrink policy.

AnalysisWorker's public methods:

• tokenize(&mut self, text: &str) — tokenizes text through the full analysis chain, reusing the worker's internal buffers; produces exactly the same tokens as Tokenizer::tokenize.
• tokenize_nbest(&mut self, text: &str, n, unique, cost_threshold) — tokenizes text and returns the top-N results with costs, reusing the worker's internal buffers; produces exactly the same results as Tokenizer::tokenize_nbest.
• set_mode(&mut self, mode: Mode) — sets the segmentation mode for subsequent calls.
• set_keep_whitespace(&mut self, keep: bool) — sets whether whitespace tokens are kept in the output for subsequent calls; kept whitespace also stays in the lattice, so the other tokens can differ.
• set_skip_whitespace(&mut self, skip: bool) — sets whether whitespace is skipped in the lattice for subsequent calls when whitespace tokens are not kept (by default the dictionary's setting).
• set_max_grouping_len(&mut self, max_grouping_len: Option<usize>), set_unknown_word_ladder(&mut self, unknown_word_ladder: bool) — set the unknown-word grouping cap and the length ladder for subsequent calls.
• set_space_penalty(&mut self, space_penalty: Option<SpacePenaltyConfig>), set_space_penalty_from_dictionary(&mut self) — set the left-space penalty rules for subsequent calls, or restore the dictionary's.
• shrink_to(&mut self, text_len_hint: usize) — immediately shrinks the worker's internal buffers to what an input of text_len_hint bytes needs.
• reset(&mut self) — discards all internal buffers, replacing them with fresh ones; intended for recovery paths (e.g. after a panic poisoned a mutex holding the worker), with configuration (dictionary, filters, mode) preserved.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-analysis

Lindera CLI

A morphological analysis command-line interface for Lindera.

• Installation - Install or build the CLI
• Commands - Command reference for list, tokenize, build, download, train, and export
• Tutorial - Step-by-step guide to get started

Installation

Install via Cargo

You can install the binary via cargo:

% cargo install lindera-cli

Download from GitHub Releases

Alternatively, you can download a pre-built binary from the release page:

• https://github.com/lindera/lindera/releases

Obtaining Dictionaries

Lindera does not bundle dictionaries with the binary. The easiest way to obtain one is the download subcommand, which fetches the pre-built dictionary matching the CLI version and installs it under the OS-standard application data directory:

% lindera download ipadic

After downloading, the dictionary can be referenced by name:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict ipadic

See Commands for the available dictionary names and storage locations.

Alternatively, you can download a pre-built dictionary manually from the GitHub Releases page:

# Example: download and extract the IPADIC dictionary
% curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
% unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Then specify the extracted dictionary path when using the CLI (note that the archive contains a lindera-ipadic directory):

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict /path/to/ipadic/lindera-ipadic

Build from Source

Build without dictionaries (default)

Build a binary containing only the tokenizer and trainer without embedded dictionaries:

% cargo build --release

Build with all features

% cargo build --release --all-features

Build with Embedded Dictionaries (Advanced)

For advanced users who want to embed dictionaries directly into the binary, use the embed-* feature flags. This eliminates the need for external dictionary files at runtime but increases the binary size.

IPADIC (Japanese dictionary)

% cargo build --release --features=embed-ipadic

IPADIC NEologd (Japanese dictionary)

% cargo build --release --features=embed-ipadic-neologd

UniDic (Japanese dictionary)

% cargo build --release --features=embed-unidic

SudachiDict (Japanese dictionary)

% cargo build --release --features=embed-sudachidict

ko-dic (Korean dictionary)

% cargo build --release --features=embed-ko-dic

CC-CEDICT (Chinese dictionary)

% cargo build --release --features=embed-cc-cedict

Jieba (Chinese dictionary)

% cargo build --release --features=embed-jieba

[!TIP] After building with an embed-* feature flag, use the embedded:// scheme to load the embedded dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic

See Feature Flags for details.

Commands

The Lindera CLI provides six main commands:

• list - List the morphological analysis dictionaries and their status
• tokenize - Perform morphological analysis on text
• build - Build a dictionary from source CSV files
• download - Download a pre-built dictionary from the GitHub releases page
• train - Train a CRF model from annotated corpus data
• export - Export a trained model to dictionary format

Warnings and log messages

Every command writes its results to standard output and the warnings of the Lindera libraries to standard error. For example, when tokenize reads more than 32 KiB of text without a sentence delimiter, it cuts the sentence there (see Sentence Splitting) and warns:

% head -c 40000 /dev/zero | tr '\0' a | lindera tokenize --dict embedded://ipadic -o wakati > /dev/null
[WARN  lindera_segmenter::segmenter] no sentence delimiter (\n, \t, 。, 、) found within 32768 bytes from offset 0; forcing a sentence boundary to bound lattice size (see https://github.com/lindera/lindera/issues/871)

The RUST_LOG environment variable selects the messages, in the syntax of env_logger:

• Not set: warnings and errors.
• RUST_LOG=error: errors only, which hides the warnings. RUST_LOG=off writes no log messages at all.
• RUST_LOG=debug: also debugging messages, such as the files lindera build reads and writes.

The setting does not change standard output, and the error of a command that fails is written whatever the setting. The messages are colored when standard error is a terminal; RUST_LOG_STYLE=never turns the colors off.

list

List all known morphological analysis dictionaries and show, for each one, whether it is embedded in the binary (via the embed-* feature flags) and whether a pre-built copy has been downloaded locally with lindera download.

List parameters

This command takes no arguments. The LINDERA_DATA_DIR environment variable overrides the application data directory searched for downloaded dictionaries, exactly as it does for lindera download.

List usage

% lindera list

NAME            EMBEDDED  DOWNLOADED  PATH
ipadic          yes       yes         /home/user/.local/share/lindera/dictionaries/5.1.0/lindera-ipadic
ipadic-neologd  no        no          -
unidic          no        incomplete  /home/user/.local/share/lindera/dictionaries/5.1.0/lindera-unidic
sudachidict     no        no          -
ko-dic          no        no          -
cc-cedict       no        no          -
jieba           no        no          -

The output contains one dictionary per line:

• EMBEDDED is yes when the dictionary was embedded at build time (e.g. --features=embed-ipadic).
• DOWNLOADED is yes when a complete dictionary installed by lindera download exists for the running CLI version, and incomplete when the installation directory exists but required files are missing (re-download it with lindera download <name> --force).
• PATH shows the installation directory when it exists, and - otherwise.

tokenize

Perform morphological analysis (tokenization) on Japanese, Chinese, or Korean text using various dictionaries.

Parameters

• --dict / -d: Dictionary path, URI, or downloaded dictionary name (required)
  • File path: /path/to/dictionary
  • Embedded: embedded://ipadic, embedded://unidic, etc.
  • Downloaded dictionary name: ipadic, unidic, etc. — resolved to the dictionary installed by lindera download. An existing filesystem path with the same name takes precedence.
• --output / -o: Output format (default: mecab)
  • mecab: MeCab-compatible format with part-of-speech info
  • wakati: Space-separated tokens only, one line per input line (an empty line for a line without tokens)
  • json: Detailed JSON format with all token information
• --user-dict / -u: User dictionary path (optional). An entry whose context ID lies outside the system dictionary's connection matrix is an error (see Context IDs).
• --mode / -m: Tokenization mode (default: normal)
  • normal: Standard tokenization
  • decompose: Decompose compound words
• --char-filter / -c: Character filter configuration (JSON)
• --token-filter / -t: Token filter configuration (JSON)
• --keep-whitespace: Keep whitespace tokens in the output. Whitespace then also stays in the lattice, so the other tokens can differ from those of the default output, in which whitespace is dropped and, unless the dictionary's metadata.json turns it off (SudachiDict does), skipped in the lattice as MeCab does
• --disable-skip-whitespace: Keep whitespace in the lattice as a node of its own (a whitespace dictionary entry or the SPACE unknown word) instead of skipping it, while still dropping it from the output (the behavior before v7). With ko-dic, whose connection costs for that node are all zero, every space then resets the connection context. Skipping is on by default for every dictionary whose metadata.json does not turn it off (SudachiDict does). No effect with --keep-whitespace. See the Segmenter documentation
• --max-grouping-len: Cap on unknown-word grouping, in characters beyond the first (MeCab's max-grouping-size; MeCab defaults to 24). The cap is applied at each position: a run (characters that each share a char.def category with the one before, as in MeCab) longer than the cap is not grouped there, the single-character candidate (plus the length ladder and dictionary words) remains, and the remaining tail is grouped again once it fits, so no unknown token exceeds cap+1 characters. 0 or omitted: unbounded. See the Segmenter documentation for an example
• --disable-unknown-word-ladder: Disable the MeCab/Vibrato-inspired unknown-word length ladder (char.def's LENGTH field). Enabled by default. Disabling it leaves out these candidates, as before v6; other changes since then, such as unknown words that start inside a grouped run, still apply. See the Segmenter documentation
• --disable-space-penalty: Disable the left-space penalty (mecab-ko's left-space-penalty-factor) that a dictionary shipping rules in its metadata.json applies by default (ko-dic ships mecab-ko-dic's). A candidate that starts right after whitespace and whose first part-of-speech tag is listed gets the cost added; disabling it together with --disable-skip-whitespace turns off both changes since v6.0 in how Korean is read around spaces (this penalty and whitespace skipping), but not the other output changes since then, such as the split of punctuation runs like .". See the Segmenter documentation
• --space-penalty-rules: Use explicit left-space penalty rules as JSON instead of the dictionary's, e.g. '{"rules":[{"pos":["JKS","JX"],"cost":6000}]}'. Mutually exclusive with --disable-space-penalty
• --mmap: Use memory-mapped file loading for the dictionary directory's word list. Ignored for embedded:// dictionaries and when the mmap feature is disabled. Rebuilding or truncating the dictionary directory while a process holds it mapped can cause a SIGBUS on the next lookup.
• --nbest / -N: Number of N-best results to return (default: 1). When set to 2 or more, N-best output is enabled.
• --nbest-unique: Deduplicate N-best results by removing paths that produce the same segmentation: each result is the cheapest path of its word boundaries.
• --nbest-cost-threshold: Maximum cost difference from the best result. Only results with cost within best_cost + threshold are returned; both costs are totals over the whole input line.
• Input file: Optional file path (default: stdin). The input is read line by line, and each line is tokenized on its own. Only the line terminator (\n, or \r\n) is removed: whitespace at the start or end of a line is handled as elsewhere in the line (spaces and tabs, which are SPACE characters, are skipped and dropped from the output; U+3000 is a token with IPADIC, IPADIC-NEologd and UniDic), and the byte offsets in the JSON output index the line. Unlike MeCab, which outputs the \r of a CRLF line as an unknown word, Lindera removes it

Basic usage

# Tokenize text using a dictionary directory
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /path/to/dictionary

# Tokenize text using embedded dictionary
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

# Tokenize with different output format
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

# Tokenize text from file
lindera tokenize \
  --dict /path/to/dictionary \
  --output wakati \
  input.txt

Examples with external dictionaries

Tokenize with external IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external IPADIC Neologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-neologd-0.0.7-20200820

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-unidic-2.1.2

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

Tokenize with external ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict /tmp/lindera-ko-dic-2.1.1-20180720

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

Tokenize with external CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-cc-cedict-0.1.0-20261021

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

Tokenize with external Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-jieba-0.1.3

Examples with embedded dictionaries

Lindera can include dictionaries directly in the binary when built with specific feature flags. This allows tokenization without external dictionary files.

Tokenize with embedded IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC dictionary in the binary, you must build with the --features=embed-ipadic option.

Tokenize with embedded UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://unidic

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-unidic option.

Tokenize with embedded IPADIC NEologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic-neologd

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-ipadic-neologd option.

Tokenize with embedded ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict embedded://ko-dic

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

NOTE: To include ko-dic dictionary in the binary, you must build with the --features=embed-ko-dic option.

Tokenize with embedded CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://cc-cedict

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

NOTE: To include CC-CEDICT dictionary in the binary, you must build with the --features=embed-cc-cedict option.

Tokenize with embedded Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://jieba

NOTE: To include Jieba dictionary in the binary, you must build with the --features=embed-jieba option.

User dictionary examples

Lindera supports user dictionaries to add custom words alongside system dictionaries. User dictionaries can be in CSV or binary format.

Use user dictionary (CSV format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Use user dictionary (Binary format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920 \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.bin

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Tokenization modes

Lindera provides two tokenization modes: normal and decompose.

Normal mode (default)

Tokenizes faithfully based on words registered in the dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode normal

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Decompose mode

Tokenizes compound noun words additionally:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Output formats

Lindera provides three output formats: mecab, wakati and json.

MeCab format (default)

Outputs results in MeCab-compatible format with part-of-speech information:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output mecab

お待ち  名詞,サ変接続,*,*,*,*,お待ち,オマチ,オマチ
し  動詞,自立,*,*,サ変・スル,連用形,する,シ,シ
て  助詞,接続助詞,*,*,*,*,て,テ,テ
おり  動詞,非自立,*,*,五段・ラ行,連用形,おる,オリ,オリ
ます  助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。  記号,句点,*,*,*,*,。,。,。
EOS

Wakati format

Outputs only the token text separated by spaces, one line per input line (or per result with -N). A line without tokens, such as an empty line or a line of spaces, gives an empty line, as in MeCab:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

お待ち し て おり ます 。

JSON format

Outputs detailed token information in JSON format:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

[
  {
    "base_form": "お待ち",
    "byte_end": 9,
    "byte_start": 0,
    "conjugation_form": "*",
    "conjugation_type": "*",
    "part_of_speech": "名詞",
    "part_of_speech_subcategory_1": "サ変接続",
    "part_of_speech_subcategory_2": "*",
    "part_of_speech_subcategory_3": "*",
    "pronunciation": "オマチ",
    "reading": "オマチ",
    "surface": "お待ち",
    "word_id": 14698
  },
  ...
]

N-Best tokenization

Lindera supports N-Best tokenization, which returns the top N tokenization candidates ordered by cost (lower cost = better). This is based on the Forward-DP Backward-A* algorithm, compatible with MeCab's N-Best implementation.

A line is cut into sentences at \t, 。 and 、, but the context is carried across 、 and 。, so the candidates for a line without a tab are the N best segmentations of the whole line (for the few exceptions, see Sentence Splitting). A tab ends a segment: a line with tabs is searched segment by segment, and its candidates are the N cheapest combinations of one path per segment. Every candidate segments the whole line.

Basic N-Best example

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  -N 3

NBEST 1 (cost=21245)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 2 (cost=24541)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 3 (cost=24541)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

Results 2 and 3 are different segmentations with the same cost (24541).

N-Best with unique results

When the same segmentation appears in multiple paths (differing only in the dictionary entries of their words, such as their parts of speech), use --nbest-unique to deduplicate. Each result is then the cheapest path of its word boundaries, and the search does not go through the other paths, which can be millions for a short line:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 5 --nbest-unique -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17747)
営業 部 長谷川 です
NBEST 3 (cost=18365)
営業 部 長谷 川 です
NBEST 4 (cost=21125)
営業 部 長 谷川 です
NBEST 5 (cost=22259)
営業 部長 谷 川 です

N-Best with cost threshold

Use --nbest-cost-threshold to limit results to paths within a certain cost range of the best path:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 10 --nbest-unique --nbest-cost-threshold 5000 -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17747)
営業 部 長谷川 です
NBEST 3 (cost=18365)
営業 部 長谷 川 です

Only 3 results are returned: the 4th result of the example above costs 21125, which exceeds 15760 + 5000 = 20760.

Advanced tokenization with filters

Lindera provides an analytical framework that combines character filters, tokenizers, and token filters for advanced text processing. Filters are configured using JSON.

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般"]}'

すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
EOS

build

Build (compile) a morphological analysis dictionary from source CSV files for use with Lindera.

Build parameters

• --src / -s: Source directory containing dictionary CSV files (or single CSV file for user dictionary)
• --dest / -d: Destination directory for compiled dictionary output
• --metadata / -m: Metadata configuration file (metadata.json) that defines dictionary structure
• --user / -u: Build user dictionary instead of system dictionary (optional flag)
• --system-dict: The system dictionary the user dictionary is for, as a directory path, URI, or downloaded dictionary name, the forms tokenize --dict accepts (optional; only with --user). The build then checks every row's context IDs against its connection matrix (see Context ID check)
• --context-id-freq / -f: Context-ID access-frequency file used to reorder connection-cost IDs (optional; only meaningful when the dictionary's metadata.json sets connection_id_mapping: true)

Dictionary types

System dictionary

A full morphological analysis dictionary containing:

• Lexicon entries (word definitions)
• Connection cost matrix
• Unknown word handling rules
• Character type definitions

User dictionary

A supplementary dictionary for custom words that works alongside a system dictionary.

Context ID check

When it builds a system dictionary, lindera build checks that every context ID lies inside the connection matrix, as MeCab's mecab-dict-index does. In every lex.csv row and unk.def line, the right context ID must be below the first value of the matrix.def header, and the left context ID below the second. A dictionary that breaks this fails to build, with an error that names the file, the line, the ID and the matrix size. For example, with a 2 x 2 matrix.def and a lex.csv whose second row has the right context ID 2:

Error: LinderaError { kind: Content, source: ./dict-source/lex.csv: line 2: right context id 2 is out of range for the connection matrix (matrix.def size 2 2; right context ids must be below 2) }

• If the dictionary's metadata.json sets skip_invalid_cost_or_id to true, such a lex.csv row is dropped instead, with a warning on standard error (see Warnings and log messages). An unk.def line is always checked.
• A matrix.def row whose IDs fall outside its axes is an error too, and so is an unk.def line with fewer than four fields.
• A user dictionary (--user) is checked only when --system-dict names the system dictionary it is for, as described below.

With --system-dict, lindera build --user checks every row of the user dictionary against the connection matrix of that system dictionary, with the same bounds. A simple row is checked with the default context IDs of metadata.json (default_left_context_id and default_right_context_id). The first row outside the matrix fails the build, with an error that names its CSV line and surface, and no .bin file is written. For example, against IPADIC's 1316 x 1316 matrix, with a userdic.csv whose second row has the left context ID 60000:

% lindera build --src ./userdic.csv --dest ./user_dict --metadata ./lindera-ipadic/metadata.json --user --system-dict embedded://ipadic
Error: LinderaError { kind: Content, source: user dictionary line 2 (surface: '東京スカイツリー'): left context id 60000 is out of range for the connection matrix (matrix.def size 1316 1316; left context ids must be below 1316) }

Without --system-dict, the build does not know the system dictionary, and the user dictionary builds. lindera tokenize --user-dict then checks it against the system dictionary and fails with an error that names the first entry outside the matrix by its word id (see Context IDs). A .csv given to lindera tokenize --user-dict is built for the system dictionary of --dict as it is loaded, so its error names the CSV line and surface, as with --system-dict.

Cells missing from matrix.def

A matrix.def does not have to list every cell. lindera build gives a cell that it leaves out the cost 0, as MeCab's mecab-dict-index does. Up to v6.2.0, such a cell cost 32767 (i16::MAX), which made the connection very expensive. For example, with the following matrix.def, the connections from the right context ID 1 to the left context IDs 0 and 1 cost 10 and 20, and the other seven cells cost 0:

3 3
1 0 10
1 1 20

The bundled dictionaries and the matrix.def that lindera export writes list every cell, so this rule does not change them.

char.def and unk.def check

When it builds a system dictionary, lindera build also checks char.def and unk.def against each other, as mecab-dict-index does:

• A range line of char.def names only categories defined on an earlier line.
• Each category is defined once.
• Every unk.def line names a category that char.def defines.
• Every char.def category has at least one unk.def line.

A dictionary that breaks one of these fails to build, with an error that names the file, the category and, except for a category without an unk.def line, the line. Such a dictionary used to build, and then lindera tokenize panicked on a character of an undefined category, gave some categories the settings of others, stopped its output before a character that no word or unknown word covers, or ignored the unk.def line. For example, with a char.def that defines the SPACE category and an unk.def that has no SPACE line:

Error: LinderaError { kind: Content, source: ./dict-source/unk.def: no line for category SPACE, which char.def defines; every char.def category needs at least one unk.def line }

lindera train checks its --char-def and --unk-def in the same way before it trains (see Training Pipeline).

Examples

Build IPADIC (Japanese dictionary)

# Download and extract IPADIC source files
% curl -L -o /tmp/mecab-ipadic-2.7.0-20250920.tar.gz "https://Lindera.dev/mecab-ipadic-2.7.0-20250920.tar.gz"
% tar zxvf /tmp/mecab-ipadic-2.7.0-20250920.tar.gz -C /tmp

# Build the dictionary
% lindera build \
  --src /tmp/mecab-ipadic-2.7.0-20250920 \
  --dest /tmp/lindera-ipadic-2.7.0-20250920 \
  --metadata ./lindera-ipadic/metadata.json

Build IPADIC NEologd (Japanese dictionary)

% curl -L -o /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz "https://lindera.dev/mecab-ipadic-neologd-0.0.7-20200820.tar.gz"
% tar zxvf /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ipadic-neologd-0.0.7-20200820 \
  --dest /tmp/lindera-ipadic-neologd-0.0.7-20200820 \
  --metadata ./lindera-ipadic-neologd/metadata.json

Build UniDic (Japanese dictionary)

% curl -L -o /tmp/unidic-mecab-2.1.2.tar.gz "https://Lindera.dev/unidic-mecab-2.1.2.tar.gz"
% tar zxvf /tmp/unidic-mecab-2.1.2.tar.gz -C /tmp

% lindera build \
  --src /tmp/unidic-mecab-2.1.2 \
  --dest /tmp/lindera-unidic-2.1.2 \
  --metadata ./lindera-unidic/metadata.json \
  --context-id-freq ./lindera-unidic/context_id_freq.txt

Build CC-CEDICT (Chinese dictionary)

% curl -L -o /tmp/CC-CEDICT-MeCab-0.1.0-20261021.tar.gz "https://lindera.dev/CC-CEDICT-MeCab-0.1.0-20261021.tar.gz"
% tar zxvf /tmp/CC-CEDICT-MeCab-0.1.0-20261021.tar.gz -C /tmp

% lindera build \
  --src /tmp/CC-CEDICT-MeCab-0.1.0-20261021 \
  --dest /tmp/lindera-cc-cedict-0.1.0-20261021 \
  --metadata ./lindera-cc-cedict/metadata.json

Build Jieba (Chinese dictionary)

% curl -L -o /tmp/mecab-jieba-0.1.3.tar.gz "https://lindera.dev/mecab-jieba-0.1.3.tar.gz"
% tar zxvf /tmp/mecab-jieba-0.1.3.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-jieba-0.1.3/dict-src \
  --dest /tmp/lindera-jieba-0.1.3 \
  --metadata ./lindera-jieba/metadata.json

Build ko-dic (Korean dictionary)

% curl -L -o /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz "https://Lindera.dev/mecab-ko-dic-2.1.1-20180720.tar.gz"
% tar zxvf /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ko-dic-2.1.1-20180720 \
  --dest /tmp/lindera-ko-dic-2.1.1-20180720 \
  --metadata ./lindera-ko-dic/metadata.json \
  --context-id-freq ./lindera-ko-dic/context_id_freq.txt

Build user dictionaries

Build IPADIC user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera IPADIC Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ipadic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ipadic/metadata.json \
  --user

Add --system-dict embedded://ipadic, or the path of a built IPADIC, to check the rows' context IDs against IPADIC's connection matrix (see Context ID check).

Build UniDic user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera UniDic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/unidic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-unidic/metadata.json \
  --user

Build SudachiDict user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera SudachiDict Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/sudachidict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-sudachidict/metadata.json \
  --user

Build CC-CEDICT user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera CC-CEDICT Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/cc-cedict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-cc-cedict/metadata.json \
  --user

Build Jieba user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera Jieba Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/jieba_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-jieba/metadata.json \
  --user

Build ko-dic user dictionary (Korean)

For more details about user dictionary format please refer to the following URL:

• Lindera ko-dic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ko-dic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ko-dic/metadata.json \
  --user

download

Download the pre-built dictionary archive matching the CLI version from the GitHub releases page and install it under the OS-standard application data directory.

Download parameters

• Dictionary name (required): one of ipadic, ipadic-neologd, unidic, sudachidict, ko-dic, cc-cedict, jieba
• --force: Re-download and replace an existing dictionary

Storage location

Dictionaries are installed under a versioned layout <data dir>/dictionaries/<version>/lindera-<name>/. The default base directory per OS:

Set the LINDERA_DATA_DIR environment variable to override the base directory. This variable is unrelated to the build-time source cache variable LINDERA_BUILD_DICTIONARY_CACHE_DIR.

Download usage

% lindera download ipadic

Downloading https://github.com/lindera/lindera/releases/download/v5.1.0/lindera-ipadic-5.1.0.zip
100% (15.1/15.1 MiB)
Downloaded dictionary 'ipadic'
/home/user/.local/share/lindera/dictionaries/5.1.0/lindera-ipadic

The installed path is printed to stdout (progress and notices go to stderr), so it can be captured in scripts:

% DICT_DIR=$(lindera download ipadic)

After downloading, the dictionary can be referenced by name in tokenize:

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize --dict ipadic

Notes:

• The archive version always matches the CLI version; running a development build whose version has no published release yet fails with HTTP 404.
• Integrity is verified via the HTTPS transport and the ZIP archive's built-in per-file CRC32 checksums.
• The largest dictionary (ipadic-neologd) is a ~305 MB download and requires roughly twice that amount of free disk space during extraction.

train

Train a new morphological analysis model from annotated corpus data. To use this feature, you must build with the train feature flag enabled. (The train feature flag is enabled by default.)

Train parameters

• --seed / -s: Seed lexicon file (CSV format) to be weighted
• --corpus / -c: Training corpus (annotated text)
• --char-def / -C: Character definition file (char.def)
• --unk-def / -u: Unknown word definition file (unk.def) to be weighted
• --feature-def / -f: Feature definition file (feature.def)
• --rewrite-def / -r: Rewrite rule definition file (rewrite.def)
• --output / -o: Output model file
• --lambda / -l: Regularization coefficient, finite and >= 0 (0 disables regularization); with l2, MeCab's learner's -c C is 1 / C (default: 1.0, MeCab's -c 1.0)
• --regularization / -R: Regularization type: l1, l2, or elasticnet, in any case; elastic_net and elastic-net also select Elastic Net (default: l2)
• --elastic-net-l1-ratio: L1 ratio for Elastic Net regularization (0.0-1.0), only used with --regularization elasticnet; 0 uses only L2, 1 uses only L1 (default: 0.5)
• --max-iterations / -i: Maximum number of iterations for training, 1 or more (default: 100)
• --eta / -e: Tolerance of the convergence test: training stops before --max-iterations once the relative change of the objective stays below this value for 3 iterations in a row; 0 disables the test (default: 0.00005, the default of MeCab's learner)
• --max-threads / -t: Maximum number of threads, 1 or more (defaults to CPU core count; affects only speed, never the trained model)

Basic workflow

1. Prepare training files

Seed lexicon file (seed.csv):

The seed lexicon file contains initial dictionary entries used for training the CRF model. Each line represents a word entry with comma-separated fields:

• Surface
• Left context ID
• Right context ID
• Word cost
• Part-of-speech tags (multiple fields)
• Base form
• Reading (katakana)
• Pronunciation

Note: The exact field definitions differ between dictionary formats (IPADIC, UniDic, ko-dic, CC-CEDICT). Please refer to each dictionary's format specification for details.

外国,0,0,0,名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人,0,0,0,名詞,接尾,一般,*,*,*,人,ジン,ジン

The file is read as CSV, as lindera build reads lex.csv: a field may be quoted (",",0,0,0,...), and a line that starts with # is an entry, not a comment. See Seed Lexicon for the details.

Training corpus (corpus.txt):

The training corpus file contains annotated text data used to train the CRF model. Each line consists of:

• A surface form (word) followed by a tab character
• Comma-separated morphological features (part-of-speech tags, base form, reading, pronunciation)
• Sentences are separated by "EOS" (End Of Sentence) markers

外国	名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人	名詞,接尾,一般,*,*,*,人,ジン,ジン
参政	名詞,サ変接続,*,*,*,*,参政,サンセイ,サンセイ
権	名詞,接尾,一般,*,*,*,権,ケン,ケン
EOS

For detailed information about file formats and advanced features, see Training Pipeline.

2. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --max-iterations 100

3. Training results

lindera train writes the model to the --output file. Once it has built the training lattices, it prints how the gold tokens of the corpus were labelled (see How training works). With the files in ./resources/training/:

Lattice statistics: gold_lexicon=14 gold_unknown=0 gold_virtual=0 virtual_labels=0 negative_lexicon=0 negative_unknown=17

When no gold token is a seed word, or fewer than half of the gold tokens whose surface has seed rows are, it then writes a warning on stderr, with the first gold token whose surface has seed rows but none with its feature string, next to those rows (see How training works).

Once training stops, it prints why, with the number of iterations and of evaluations of the loss and its gradient (see --eta above):

L-BFGS stopped after 8 iterations and 9 evaluations of the loss and its gradient: converged

When an iteration left the objective unchanged or the line search failed far from the optimum, it then writes a warning on stderr (see How training works).

The trained model contains:

• Seed words: every seed lexicon row, with a learned weight and context IDs
• Unknown words: every unk.def entry, in file order, with a learned weight and context IDs

Corpus words that are not in the seed lexicon are not added as words. They are learned through the unk.def entries, or through virtual labels, which are trained but not exported.

export

Export a trained model file to Lindera dictionary format files. This feature requires building with the train feature flag enabled.

Export parameters

• --model / -m: Path to the trained model file (.dat format)
• --output / -o: Directory to output the dictionary files
• --metadata: Optional metadata.json file to update with trained model information
• --cost-factor: Override cost factor for weight-to-cost conversion (default: value from trained model, typically 700)

Output files

The export command creates the following dictionary files in the output directory:

• lex.csv: Lexicon file with learned weights (MeCab-compatible cost via tocost())
• matrix.def: Dense connection cost matrix covering all (right_id, left_id) pairs of the model's context IDs, plus 0 for BOS/EOS. A pair without a trained weight costs 0, as in training and in MeCab
• unk.def: Unknown word definitions, one line for each unk.def entry of the training input, in the same order
• char.def: Character type definitions
• feature.def: Feature template definitions (copied from trained model)
• rewrite.def: Feature rewrite rules (copied from trained model)
• left-id.def: Left context ID to feature string mapping
• right-id.def: Right context ID to feature string mapping
• metadata.json: Updated metadata file (if --metadata option is provided)

Model format version

lindera export reads a model of the current format version (3.0.0) only. A model.dat written by an earlier version, which includes every model.dat written before 7.0.0, is rejected, and has to be trained again with lindera train. For a model trained with the 6.1.0 CLI:

Error: LinderaError { kind: Deserialize, source: failed to deserialize model: failed without error information; enable debug assertions and the `alloc` feature in rancor for error information. The file is not a legacy JSON model either, so it was most likely written by a different version of lindera-trainer; re-run `lindera train` to regenerate it. }

Complete workflow example

1. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --max-iterations 100

2. Export to dictionary format

lindera export \
  --model /tmp/lindera/training/model.dat \
  --metadata ./resources/training/metadata.json \
  --output /tmp/lindera/training/dictionary

3. Build dictionary

lindera build \
  --src /tmp/lindera/training/dictionary \
  --dest /tmp/lindera/training/compiled_dictionary \
  --metadata /tmp/lindera/training/dictionary/metadata.json

4. Use trained dictionary

echo "これは外国人参政権です。" | lindera tokenize -d /tmp/lindera/training/compiled_dictionary

これ	名詞,代名詞,一般,*,*,*,これ,コレ,コレ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
外国	名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人	名詞,接尾,一般,*,*,*,人,ジン,ジン
参政	名詞,サ変接続,*,*,*,*,参政,サンセイ,サンセイ
権	名詞,接尾,一般,*,*,*,権,ケン,ケン
です	助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。	記号,句点,*,*,*,*,。,。,。
EOS

Metadata update feature

When the --metadata option is provided, the export command will:

1. Read the base metadata.json file to preserve existing configuration
2. Update specific fields with values from the trained model:
   • default_word_cost: Calculated from feature weight median
   • model_info: Training statistics including feature count, label count, matrix size (connection_matrix_size, the size of matrix.def as rows x columns), the number of training iterations that ran (training_iterations, at most --max-iterations), the regularization coefficient (regularization), its type (regularization_type: l1, l2 or elasticnet) and, for Elastic Net only, its L1 ratio (elastic_net_l1_ratio), and the model format version. No timestamp is written, so exporting the same model twice produces identical output
3. Preserve existing settings such as dictionary name, character encoding, schema definitions, and other user-defined configuration

Tutorial

This tutorial walks you through the basic usage of the Lindera CLI, from installation to advanced text processing.

1. Install the CLI

Install Lindera CLI:

% cargo install lindera-cli

Verify the installation:

% lindera --help

2. Download a dictionary

Download the pre-built IPADIC dictionary from the GitHub releases page. It is installed under the OS-standard application data directory:

% lindera download ipadic

See Commands for the available dictionary names and storage locations.

3. Basic tokenization

Tokenize Japanese text using the downloaded IPADIC dictionary, referencing it by name:

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic

Expected output:

東京    名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
日本    名詞,固有名詞,地域,国,*,*,日本,ニホン,ニホン
の      助詞,連体化,*,*,*,*,の,ノ,ノ
首都    名詞,一般,*,*,*,*,首都,シュト,シュト
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。      記号,句点,*,*,*,*,。,。,。
EOS

4. Try different output formats

Wakati format (word segmentation only)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic \
  --output wakati

Expected output:

東京 は 日本 の 首都 です 。

JSON format (detailed information)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict ipadic \
  --output json

This produces a JSON array with detailed token information including byte offsets, part-of-speech tags, readings, and more.

5. Use decompose mode

Decompose mode splits compound nouns into their constituent parts:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict ipadic \
  --mode decompose

Expected output:

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Compare with normal mode, where "関西国際空港" remains as a single token.

6. Apply character and token filters

Use Unicode normalization and keep only common nouns:

% echo "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。" | lindera tokenize \
  --dict ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般","名詞,サ変接続","名詞,固有名詞,組織"]}'

Expected output:

Lindera 名詞,固有名詞,組織,*,*,*,*,*,*
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
エンジン        名詞,一般,*,*,*,*,エンジン,エンジン,エンジン
EOS

The Unicode normalization converts full-width characters to half-width, and the token filter keeps only tokens matching the specified part-of-speech tags.

You can also combine multiple filters:

% echo "すもももももももものうち" | lindera tokenize \
  --dict ipadic \
  --token-filter 'japanese_stop_tags:{"tags":["助詞,格助詞,一般","助詞,係助詞","助詞,連体化","助動詞"]}'

7. Use user dictionary

Create a CSV file with custom word entries (e.g., my_dict.csv):

東京スカイツリー,カスタム名詞,トウキョウスカイツリー

Tokenize with the user dictionary:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict ipadic \
  --user-dict ./my_dict.csv

Without the user dictionary, "東京スカイツリー" would be split into multiple tokens. With the user dictionary, it is recognized as a single token.

For pre-built user dictionary examples, see:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

Expected output:

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Lindera Python

Lindera Python provides Python bindings for the Lindera morphological analysis engine, built with PyO3. It brings Lindera's high-performance tokenization capabilities to the Python ecosystem with support for Python 3.10 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from PyPI

Pre-built wheels are available on PyPI:

pip install lindera

The wheels cover CPython 3.10 to 3.15 on Linux (x86_64 and aarch64, glibc), macOS (x86_64 and arm64), and Windows (x64 and arm64). On any other platform, such as musl-based Linux (Alpine), other CPU architectures, or free-threaded CPython, pip install lindera builds the package from the source distribution instead. That build needs the Rust toolchain 1.89 or later (install it via rustup) and a C linker such as cc, and takes several minutes.

[!NOTE] The PyPI package does not include dictionaries. See Obtaining Dictionaries below.

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Building from Source

If you need to build from source (e.g., to enable specific feature flags), the following prerequisites are required:

• Python 3.10 or later (up to 3.15)
• Rust toolchain -- Install via rustup
• maturin -- Python package for building Rust-based Python extensions

Install maturin with pip:

pip install maturin

Development Build

Build and install lindera-python in development mode:

cd lindera-python
maturin develop

Or let the project Makefile create the virtual environment that maturin develop installs into: make setup-venv creates lindera-python/.venv with maturin and pytest installed:

make setup-venv
source lindera-python/.venv/bin/activate
cd lindera-python
maturin develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

maturin develop --features train

Feature Flags

Multiple features can be combined:

maturin develop --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Python:

import lindera

print(lindera.version())

Quick Start

This guide shows how to tokenize text using lindera-python.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

from lindera import Mode, TokenizerBuilder

builder = TokenizerBuilder()
builder.set_mode(Mode.normal())
builder.set_dictionary("/path/to/ipadic")
tokenizer = builder.build()

tokens = tokenizer.tokenize("関西国際空港限定トートバッグ")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

from lindera import Mode, TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode(Mode.normal())
    .set_dictionary("/path/to/ipadic")
    .build()
)

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    print(f"{token.surface}\t{token.get_detail(0)}")

Accessing Token Properties

Each token exposes the following properties:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
tokens = tokenizer.tokenize("東京タワー")

for token in tokens:
    print(f"Surface: {token.surface}")
    print(f"Byte range: {token.byte_start}..{token.byte_end}")
    print(f"Position: {token.position}")
    print(f"Word ID: {token.word_id}")
    print(f"Unknown: {token.is_unknown}")
    print(f"Details: {token.details}")
    print()

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)

for tokens, cost in results:
    surfaces = [t.surface for t in tokens]
    print(f"Cost {cost}: {' / '.join(surfaces)}")

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

TokenizerBuilder()

Creates a new builder with default configuration.

from lindera import TokenizerBuilder

builder = TokenizerBuilder()

TokenizerBuilder().from_file(file_path)

Loads configuration from a YAML file and returns a new builder. See lindera-python/resources/lindera.yml for a complete example covering segmenter, character_filters, and token_filters.

builder = TokenizerBuilder().from_file("lindera.yml")

Configuration Methods

All setter methods return self for method chaining.

set_mode(mode)

Sets the tokenization mode. mode is a Mode:

• Mode.normal() -- Standard tokenization (default)
• Mode.decompose(penalty=None) -- Decomposes compound words into smaller units, with the length penalties of a Penalty; None uses the default penalty

A value that is not a Mode, such as the string "normal", raises TypeError. To set a mode from its name, such as a configuration value, create it with Mode.from_str().

from lindera import Mode, Penalty

builder.set_mode(Mode.normal())
builder.set_mode(Mode.decompose(Penalty(kanji_penalty_length_penalty=0)))
builder.set_mode(Mode.from_str("decompose"))

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary("embedded://ipadic")

# Use an external dictionary
builder.set_dictionary("/path/to/dictionary")

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary("/path/to/user_dictionary.csv")

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output. Kept whitespace also stays in the lattice, so the other tokens can differ from those of the default output, which skips whitespace in the lattice as MeCab does. To keep whitespace in the lattice without outputting it, use set_skip_whitespace(False) instead.

builder.set_keep_whitespace(True)

set_skip_whitespace(skip)

Sets whether whitespace is skipped in the lattice. skip has the same meaning as segmenter.skip_whitespace in a configuration file:

• True -- Skips whitespace, so the words on either side of it connect directly, as MeCab does.
• False -- Keeps whitespace in the lattice as a node of its own (a whitespace dictionary entry or the SPACE unknown word), as before v7. Whitespace is still dropped from the output.
• None -- The default of the dictionary's metadata.json: skipping, except with SudachiDict. Use it to undo an earlier call.

A value set here overrides the dictionary's default and a skip_whitespace loaded with from_file. It has no effect while whitespace tokens are kept (set_keep_whitespace(True)), because kept whitespace always stays in the lattice. Any other value raises TypeError.

With IPADIC, for example, Google が 新しい reads が as a particle (助詞) by default and as a conjunction (接続詞) with set_skip_whitespace(False). See Segmenter for details.

# Keep whitespace in the lattice, as before v7
builder.set_skip_whitespace(False)

set_space_penalty(value)

Sets the left-space penalty for Korean: a candidate that starts right after whitespace and whose part-of-speech tag normally attaches to the preceding word (particles, endings, ...) gets a cost added. value has the same meaning as segmenter.space_penalty in a configuration file:

• None -- The default: the rules the dictionary ships in its metadata.json, if any (ko-dic ships mecab-ko-dic's rules; the other bundled dictionaries ship none). Use it to undo an earlier call.
• False -- Turns the penalty off.
• True -- Requires the dictionary's rules; build() raises ValueError for a dictionary that ships none.
• A dict of the form {"rules": [{"pos": [...], "cost": n}, ...]} -- Applies these rules instead. A candidate whose first part-of-speech tag is listed in pos gets cost added; the first matching rule wins.

Any other value raises ValueError (a value with no JSON equivalent, such as an arbitrary object, raises TypeError, as in append_token_filter).

# Turn the penalty off
builder.set_space_penalty(False)

# Explicit rules
builder.set_space_penalty({
    "rules": [
        {"pos": ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], "cost": 3000},
        {"pos": ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], "cost": 6000},
    ]
})

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, as mecab-ko does. For example, 검색 이 잘 된다 reads 이 as the interjection IC; without the penalty, as the subject particle JKS. Call builder.set_space_penalty(False) to turn it off. Only a ko-dic built by Lindera 6.1.0 or later ships the rules; with a ko-dic from the v6.0.0 release the penalty stays off and True fails until the dictionary is rebuilt. See Segmenter for details.

append_character_filter(kind, args=None)

Appends a character filter to the preprocessing pipeline.

builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

append_token_filter(kind, args=None)

Appends a token filter to the postprocessing pipeline.

builder.append_token_filter("lowercase", {})

The arguments of set_space_penalty, append_character_filter and append_token_filter may nest lists and dicts at most 128 levels deep. A deeper value, or one that contains itself, raises ValueError.

Build

build()

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build()

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Tokenizer(dictionary, mode=None, user_dictionary=None, space_penalty=None)

Creates a tokenizer directly from a loaded dictionary.

mode takes a Mode, as TokenizerBuilder.set_mode does; None (the default) is normal mode. Any other value, such as the string "normal", raises TypeError.

space_penalty takes the same values as TokenizerBuilder.set_space_penalty. None (the default) uses the left-space penalty rules the dictionary ships, so with ko-dic the penalty is on; pass False to turn it off. An invalid setting raises the same errors as set_space_penalty and build().

from lindera import Mode, Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode=Mode.normal())

# ko-dic without the left-space penalty
ko_tokenizer = Tokenizer(load_dictionary("embedded://ko-dic"), space_penalty=False)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns a list of Token objects.

tokens = tokenizer.tokenize("形態素解析")

Parameters:

Returns: list[Token]

tokenize_surfaces(text)

Tokenizes the input text and returns only the token surfaces, as a list of strings. This is the fast path for wakati-style use: no Token objects are created and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals [t.surface for t in tokenizer.tokenize(text)].

surfaces = tokenizer.tokenize_surfaces("形態素解析")
# ["形態素", "解析"]

Parameters:

Returns: list[str]

tokenize_nbest(text, n, unique=False, cost_threshold=None)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)
for tokens, cost in results:
    print(cost, [t.surface for t in tokens])

Parameters:

Returns: list[tuple[list[Token], int]]

Mode

Mode is the tokenization mode that TokenizerBuilder.set_mode() and the Tokenizer constructor take. It is the counterpart of the Rust lindera::mode::Mode: normal mode, or decompose mode with its Penalty.

Creating a Mode

Mode has no constructor, so Mode() raises TypeError, and it has no Mode.Normal or Mode.Decompose attributes. Create a mode with one of the following static methods.

Mode.normal()

Returns normal mode: standard tokenization.

Mode.decompose(penalty=None)

Returns decompose mode, which adds the length penalties of penalty to the cost of long words, so that compound words are decomposed into smaller units. None (the default) uses the default penalty, Penalty(). A value that is not a Penalty raises TypeError.

Mode.from_str(name)

Returns the mode named name, as the Rust Mode parses a name: "normal" for normal mode, and "decompose" for decompose mode with the default penalty. Use it for a mode name read at run time, such as a configuration value. The name is case sensitive: any other name, such as "Normal", raises ValueError with the message Invalid mode. A value that is not a str raises TypeError.

from lindera import Mode, Penalty

normal = Mode.normal()
decompose = Mode.decompose()
whole_words = Mode.decompose(Penalty(kanji_penalty_length_penalty=0, other_penalty_length_penalty=0))
from_name = Mode.from_str("decompose")  # equal to Mode.decompose()

Mode Properties

Both properties are read-only:

Methods

mode = Mode.decompose()
mode.name              # 'decompose'
mode.penalty           # Penalty(kanji_threshold=2, kanji_penalty=3000, other_threshold=7, other_penalty=1700)
str(mode)              # 'decompose'
repr(mode)             # 'Mode.decompose(Penalty(kanji_threshold=2, kanji_penalty=3000, other_threshold=7, other_penalty=1700))'
mode.is_normal()       # False
mode.is_decompose()    # True
Mode.normal().penalty  # None
repr(Mode.normal())    # 'Mode.normal()'

Two modes are equal when their names and penalties are, as the Rust Mode compares them. A Mode is never equal to a value of another type, such as the string "normal":

Mode.from_str("decompose") == Mode.decompose()                               # True
Mode.decompose() == Mode.decompose(Penalty(kanji_penalty_length_penalty=0))  # False
Mode.normal() == "normal"                                                    # False

A Mode cannot be changed. Setting a property raises AttributeError, and penalty returns a copy, so changing the returned Penalty leaves the mode as it was; Mode.decompose() keeps a copy of the Penalty it is given as well. A Mode is not hashable, as the Rust Mode implements Eq but not Hash: hash(mode) raises TypeError, so a Mode cannot be a dict key or a set member.

mode = Mode.decompose()
penalty = mode.penalty
penalty.kanji_penalty_length_penalty = 0
mode.penalty.kanji_penalty_length_penalty  # 3000

Penalty

Penalty configures the length-based penalty thresholds used by decompose mode segmentation.

Creating a Penalty

Penalty(kanji_penalty_length_threshold=2, kanji_penalty_length_penalty=3000, other_penalty_length_threshold=7, other_penalty_length_penalty=1700)

All arguments are optional and default to the values shown above.

from lindera import Penalty

penalty = Penalty(
    kanji_penalty_length_threshold=2,
    kanji_penalty_length_penalty=3000,
    other_penalty_length_threshold=7,
    other_penalty_length_penalty=1700,
)

Penalty Properties

All four fields support both getting and setting:

penalty.kanji_penalty_length_threshold = 3
print(penalty.kanji_penalty_length_threshold)  # 3

Using a Penalty

Pass a Penalty to Mode.decompose(), and the mode to TokenizerBuilder.set_mode() or to the Tokenizer constructor. Without length penalties, decompose mode keeps long words whole:

from lindera import Mode, Penalty, Tokenizer, TokenizerBuilder, load_dictionary

text = "関西国際空港限定トートバッグ"
penalty = Penalty(kanji_penalty_length_penalty=0, other_penalty_length_penalty=0)

builder = TokenizerBuilder().set_dictionary("embedded://ipadic")
tokenizer = builder.set_mode(Mode.decompose()).build()
print([token.surface for token in tokenizer.tokenize(text)])
# ['関西', '国際', '空港', '限定', 'トートバッグ']

tokenizer = builder.set_mode(Mode.decompose(penalty)).build()
print([token.surface for token in tokenizer.tokenize(text)])
# ['関西国際空港', '限定', 'トートバッグ']

tokenizer = Tokenizer(load_dictionary("embedded://ipadic"), mode=Mode.decompose(penalty))
print([token.surface for token in tokenizer.tokenize(text)])
# ['関西国際空港', '限定', 'トートバッグ']

Normal mode has no penalty: Mode.normal() takes no arguments.

Token

Token represents a single morphological token.

Properties

Token Methods

get_detail(index)

Returns the detail string at the specified index, or None if the index is out of range.

token = tokenizer.tokenize("東京")[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: str or None

to_dict()

Returns the token as a plain dict, keeping each field's natural Python type, so it serializes without a custom encoder.

import json

data = tokenizer.tokenize("東京")[0].to_dict()
# {'surface': '東京', 'byte_start': 0, 'byte_end': 6, 'position': 0,
#  'word_id': 12345, 'is_unknown': False, 'details': [...]}

json.dumps([token.to_dict() for token in tokens])

Returns: dict with the keys surface, byte_start, byte_end, position, word_id, is_unknown, and details.

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Error Handling

Lindera Python functions raise standard Python exceptions rather than a custom exception type:

• IOError (an alias of OSError) -- for I/O-related failures, such as a missing or unreadable file
• ValueError -- for everything else, such as invalid configuration, parse errors, or tokenization failures

from lindera import load_dictionary

try:
    dictionary = load_dictionary("/path/that/does/not/exist")
except ValueError as e:
    print(f"Failed to load dictionary: {e}")

A LinderaError class is also registered as lindera.LinderaError, but no function in this crate currently raises it -- it can only be constructed and raised manually. Catch IOError/ValueError (or the general Exception) when handling errors from this library, not LinderaError.

Dictionary Management

Lindera Python provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

from lindera import load_dictionary

dictionary = load_dictionary("/path/to/ipadic")

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = load_dictionary("embedded://ipadic")

A loaded Dictionary also exposes its own metadata:

print(dictionary.metadata_name())      # e.g. "ipadic"
print(dictionary.metadata_encoding())  # e.g. "UTF-8"
metadata = dictionary.metadata()       # the full Metadata object

A user dictionary must share the metadata (schema, encoding, etc.) of the system dictionary it augments. load_user_dictionary_for takes the system dictionary and uses its metadata, as in lindera-python/examples/tokenize_with_userdict.py (see User Dictionaries).

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary. Load one for the system dictionary it is used with:

from lindera import load_dictionary, load_user_dictionary_for

dictionary = load_dictionary("/path/to/ipadic")
user_dict = load_user_dictionary_for("/path/to/user_dictionary.csv", dictionary)

A .csv is built with the system dictionary's metadata and checked against its connection matrix as it is built: a row whose context ID lies outside the matrix raises a ValueError that names its CSV line and surface (see Context IDs). A .bin is loaded as it was built.

load_user_dictionary(uri, metadata) loads a user dictionary with metadata only. A .csv loaded this way is checked when it is combined with the system dictionary, and the error names the entry by its word id:

from lindera import load_user_dictionary, Metadata

metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary.csv", metadata)

Pass the user dictionary when building a tokenizer:

from lindera import Mode, Tokenizer, load_dictionary, load_user_dictionary_for

dictionary = load_dictionary("/path/to/ipadic")
user_dict = load_user_dictionary_for("/path/to/user_dictionary.csv", dictionary)

tokenizer = Tokenizer(dictionary, mode=Mode.normal(), user_dictionary=user_dict)

Or via the builder:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("/path/to/ipadic")
    .set_user_dictionary("/path/to/user_dictionary.csv")
    .build()
)

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

from lindera import build_dictionary, Metadata

metadata = Metadata(name="custom", encoding="UTF-8")
build_dictionary("/path/to/input_dir", "/path/to/output_dir", metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

Metadata loaded with Metadata.from_json_file keeps the space_penalty rules of the metadata.json, and the built dictionary ships them. For ko-dic this means the left-space penalty stays on by default in the dictionary you build; turn it off per tokenizer with TokenizerBuilder.set_space_penalty(False) (see Tokenizer API). A Metadata created with the constructor carries no rules.

User Dictionary

Build a user dictionary from a CSV file for the system dictionary it is used with:

from lindera import build_user_dictionary_for, load_dictionary, load_user_dictionary_for

dictionary = load_dictionary("/path/to/ipadic")
build_user_dictionary_for("user_words.csv", "/path/to/user_dictionary.bin", dictionary)
user_dict = load_user_dictionary_for("/path/to/user_dictionary.bin", dictionary)

The CSV is built with the system dictionary's metadata and checked against its connection matrix: a row whose context ID lies outside the matrix raises a ValueError that names its CSV line and surface, and no file is written (see Context IDs). The second argument is the path of the file to write. End it with .bin, since the loaders choose how to load a user dictionary by its extension.

build_user_dictionary(kind, input_file, output_dir, metadata) builds a user dictionary with metadata only. Despite its name, output_dir is the path of the file to write too, not a directory. The rows are checked when the user dictionary is combined with the system dictionary, and the error names the entry by its word id:

from lindera import build_user_dictionary, Metadata

metadata = Metadata()
build_user_dictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin", metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

build_user_dictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin")

Note: the first argument ("ipadic" above) is currently unused and reserved for future use -- it does not select or configure the build in any way. The build behavior is controlled entirely by metadata (in particular metadata.user_dictionary_schema). Any string may be passed today. To build for a system dictionary, use build_user_dictionary_for.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

from lindera import Metadata

# Default metadata
metadata = Metadata()

# Custom metadata
metadata = Metadata(
    name="my_dictionary",
    encoding="UTF-8",
    default_word_cost=-10000,
)

Loading from JSON

metadata = Metadata.from_json_file("metadata.json")

Properties

All properties support both getting and setting:

metadata = Metadata()
metadata.name = "custom_dict"
metadata.encoding = "EUC-JP"
print(metadata.name)  # "custom_dict"

to_dict()

Returns the metadata as a dict of strings: the properties above, with dictionary_schema_fields and user_dictionary_schema_fields (comma-separated field names) in place of the two schemas.

metadata = Metadata(name="test")
print(metadata.to_dict())

When the metadata carries left-space penalty rules, as ko-dic's does, the dict also has a space_penalty entry holding them as a JSON string; the key is absent otherwise:

metadata = Metadata.from_json_file("/path/to/ko-dic/metadata.json")
metadata.to_dict()["space_penalty"]
# '{"rules":[{"pos":["EC","EF",...],"cost":3000},{"pos":["JC","JKB",...],"cost":6000}]}'

Schema

Schema, FieldDefinition, and FieldType describe the field layout of a dictionary entry. A schema is used by Metadata.dictionary_schema and Metadata.user_dictionary_schema (see the table above).

FieldType

FieldType enumerates the category of a single field:

• FieldType.Surface -- surface form (word text)
• FieldType.LeftContextId -- left context ID
• FieldType.RightContextId -- right context ID
• FieldType.Cost -- word cost
• FieldType.Custom -- any other, dictionary-specific field

FieldDefinition

FieldDefinition describes a single field within a schema.

FieldDefinition(index, name, field_type, description=None)

from lindera import FieldDefinition, FieldType

field = FieldDefinition(0, "surface", FieldType.Surface, "Surface form")

Properties (read-only):

Creating a Schema

Schema holds an ordered list of field names and provides lookups between field name and index.

Schema(fields)

Creates a schema from a list of field names.

from lindera import Schema

schema = Schema([
    "surface",
    "left_context_id",
    "right_context_id",
    "cost",
    "major_pos",
    "reading",
])

Schema.create_default()

A static method that returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Schema.create_default()

Schema Methods and Properties

schema = Schema.create_default()

schema.field_count()               # 13
schema.get_field_index("cost")     # 3
schema.get_field_name(0)           # "surface"
schema.get_custom_fields()         # ["major_pos", "pos_detail_1", ..., "pronunciation"]
len(schema)                        # 13

field = schema.get_field_by_name("surface")
print(field.index, field.name, field.field_type)  # 0 surface FieldType.Surface

schema.validate_record([
    "東京", "1288", "1288", "100",
    "名詞", "固有名詞", "地域", "一般", "*", "*",
    "東京", "トウキョウ", "トーキョー",
])

Text Processing Pipeline

Lindera Python supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .build()
)

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("mapping", {
        "mapping": {
            "\u30fc": "-",
            "\uff5e": "~",
        }
    })
    .build()
)

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    .build()
)

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("lowercase", {})
    .build()
)

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_base_form", {})
    .build()
)

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞"],
    })
    .build()
)

Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*).

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed. The following keeps only general nouns:

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_keep_tags", {
        "tags": ["名詞,一般"],
    })
    .build()
)

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

from lindera import Mode, TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode(Mode.normal())
    .set_dictionary("embedded://ipadic")
    # Preprocessing
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    # Postprocessing
    .append_token_filter("japanese_base_form", {})
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞", "記号,句点", "記号,読点"],
    })
    .append_token_filter("lowercase", {})
    .build()
)

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and punctuation
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Python supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-python with the train feature enabled (enabled by default):

maturin develop --features train

Training a Model

Use lindera.train() to train a CRF model from a seed lexicon and annotated corpus:

import lindera

lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=1.0,
    max_iter=100,
    max_threads=4,
)

Training Parameters

The settings are checked before any file is read: an unknown regularization, or a lambda_, elastic_net_l1_ratio or eta outside its range, raises a ValueError. To train with v6's default settings (the model still differs from v6's, since the first line search changed in #1290), pass regularization="l1" and lambda_=0.01, and eta=0 to stop only at max_iter.

Exporting a Trained Model

After training, export the model to dictionary source files using lindera.export():

import lindera

lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

Export Parameters

The export creates the following files in the output directory, the same files that lindera export creates:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions (copied from the trained model)
• feature.def -- Feature template definitions (copied from the trained model)
• rewrite.def -- Feature rewrite rules (copied from the trained model)
• left-id.def -- Left context ID to feature string mapping
• right-id.def -- Right context ID to feature string mapping
• metadata.json -- Updated metadata (when metadata parameter is provided)

The model file must come from this version's training. A model file written by an earlier version, which includes every model file written before 7.0.0, is rejected: lindera.export() fails with an error that asks you to re-run training. Train such a model again with lindera.train().

Complete Workflow

The full workflow for training and using a custom dictionary:

import lindera

# Step 1: Train the CRF model
lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=1.0,
    max_iter=100,
)

# Step 2: Export to dictionary source files
lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

# Step 3: Build the dictionary from exported source files
metadata = lindera.Metadata.from_json_file("/tmp/dictionary_source/metadata.json")
lindera.build_dictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata)

# Step 4: Use the trained dictionary
tokenizer = (
    lindera.TokenizerBuilder()
    .set_dictionary("/tmp/dictionary")
    .set_mode(lindera.Mode.normal())
    .build()
)

tokens = tokenizer.tokenize("形態素解析のテスト")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Lindera Node.js

Lindera Node.js provides Node.js bindings for the Lindera morphological analysis engine, built with NAPI-RS. It brings Lindera's high-performance tokenization capabilities to the Node.js ecosystem, on the Node.js versions ^20.17.0 || ^22.13.0 || >= 23.5.0.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary
• TypeScript support: Full type definitions included out of the box

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from npm

Pre-built packages will be available on npm:

npm install lindera

[!NOTE] The npm package does not include dictionaries. See Obtaining Dictionaries below. For browser/WASM usage, see lindera-wasm.

Building from Source

Prerequisites

• Node.js ^20.17.0 || ^22.13.0 || >= 23.5.0 -- The engines range of lindera-nodejs/package.json (LTS versions recommended)
• Rust toolchain -- Install via rustup

The NAPI-RS CLI (@napi-rs/cli), which builds the native addon, needs no global install: it is a devDependency of lindera-nodejs, which npm install installs (see Development Build).

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build lindera-nodejs in release mode. npm install also installs the NAPI-RS CLI (@napi-rs/cli) at the version package.json pins, which npm run build runs:

cd lindera-nodejs
npm install
npm run build

For a debug build, run npm run build:debug instead of npm run build.

Or use the project Makefile, which makes the same release build:

make build-lindera-nodejs

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

npm run build -- --features train

Feature Flags

Multiple features can be combined:

npm run build -- --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

const dictionary = loadDictionary("embedded://ipadic");

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Node.js:

const lindera = require("lindera");

console.log(lindera.version());

[!NOTE] The lindera npm package's exports map declares types, import, and require conditions, so the package loads from CommonJS (const { TokenizerBuilder } = require("lindera")) and ES modules (import { TokenizerBuilder } from "lindera") alike:

import { TokenizerBuilder } from "lindera";

console.log(typeof TokenizerBuilder); // "function"

Quick Start

This guide shows how to tokenize text using lindera-nodejs.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

const { Mode, TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setMode(Mode.normal());
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Method Chaining

Every setter returns the builder itself (this), so the configuration can be chained:

const { Mode, TokenizerBuilder } = require("lindera");

const tokenizer = new TokenizerBuilder()
  .setMode(Mode.normal())
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("すもももももももものうち");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details[0]}`);
}

Calling the setters one statement at a time works just as well — both styles configure the same builder.

Accessing Token Properties

Each token exposes the following properties:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("東京タワー");
for (const token of tokens) {
  console.log(`Surface: ${token.surface}`);
  console.log(`Byte range: ${token.byteStart}..${token.byteEnd}`);
  console.log(`Position: ${token.position}`);
  console.log(`Word ID: ${token.wordId}`);
  console.log(`Unknown: ${token.isUnknown}`);
  console.log(`Details: ${token.details}`);
  console.log();
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  const surfaces = tokens.map((t) => t.surface);
  console.log(`Cost ${cost}: ${surfaces.join(" / ")}`);
}

TypeScript

Lindera Node.js includes TypeScript type definitions, so all classes and functions are fully typed. The package's exports map serves both require and import, and the following sample compiles under moduleResolution: node16 with strict enabled:

import type { Token } from "lindera";
import { Mode, TokenizerBuilder } from "lindera";

const builder = new TokenizerBuilder();
builder.setMode(Mode.normal());
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens: Token[] = tokenizer.tokenize("形態素解析");
for (const token of tokens) {
  console.log(`${token.surface}: ${token.details.join(",")}`);
}

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return the builder itself (this), so they can be chained or called one statement at a time.

setMode(mode)

Sets the tokenization mode, a Mode:

• Mode.normal() -- Standard tokenization (default)
• Mode.decompose(penalty?) -- Decomposes compound words into smaller units, with the Penalty the mode holds

A value that is not a Mode, such as the mode name "normal", throws an InvalidArg error. A mode name read at run time, such as a setting, becomes a Mode with Mode.fromStr().

const { Mode } = require("lindera");

builder.setMode(Mode.normal());
builder.setMode(Mode.decompose({ kanjiPenaltyLengthPenalty: 0 }));
builder.setMode(Mode.fromStr("decompose"));

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary.csv");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output. Kept whitespace also stays in the lattice, so the other tokens can differ from those of the default output, which skips whitespace in the lattice as MeCab does. To keep whitespace in the lattice without outputting it, use setSkipWhitespace(false) instead.

builder.setKeepWhitespace(true);

setSkipWhitespace(skip)

Sets whether whitespace is skipped in the lattice. The value means the same as segmenter.skip_whitespace in a configuration file.

A value set here overrides the dictionary's default and a skip_whitespace loaded with fromFile. It has no effect while whitespace tokens are kept (setKeepWhitespace(true)), because kept whitespace always stays in the lattice. The argument is required; any other value throws an Error.

With IPADIC, for example, Google が 新しい reads が as a particle (助詞) by default and as a conjunction (接続詞) with setSkipWhitespace(false). See Segmenter for details.

// Keep whitespace in the lattice, as before v7
builder.setSkipWhitespace(false);

// Back to the dictionary's default
builder.setSkipWhitespace(null);

setSpacePenalty(value)

Sets the left-space penalty for Korean (mecab-ko's left-space-penalty-factor): a candidate that starts right after whitespace and whose first part-of-speech tag is listed in a rule gets that rule's cost added. The value means the same as segmenter.space_penalty in a configuration file.

Any other value throws an Error.

// Turn the penalty off
builder.setSpacePenalty(false);

// Explicit rules (these are the ones ko-dic ships)
builder.setSpacePenalty({
  rules: [
    { pos: ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], cost: 3000 },
    { pos: ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], cost: 6000 },
  ],
});

// Back to the dictionary's default
builder.setSpacePenalty(null);

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, because its metadata.json ships mecab-ko-dic's rules. This changes the Korean output (for example, 이 in 검색 이 잘 된다 is the interjection IC with the penalty and the subject particle JKS without it). Call setSpacePenalty(false) to turn it off. For a Tokenizer created directly, pass false as the fourth argument: new Tokenizer(dictionary, Mode.normal(), null, false). A ko-dic downloaded from the v6.0.0 release ships no rules, so with it the penalty stays off and setSpacePenalty(true) makes build() throw. See Segmenter for details.

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

The arguments of setSpacePenalty, appendCharacterFilter and appendTokenFilter may nest arrays and objects at most 128 levels deep. A deeper value, or one that contains itself, throws an Error.

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?, spacePenalty?)

Creates a tokenizer directly from a loaded dictionary.

mode takes a Mode, as setMode does, with the same errors. null or undefined (the default) is normal mode. For decompose mode with a custom penalty, pass Mode.decompose(penalty).

spacePenalty takes the same values as setSpacePenalty. null or undefined (the default) uses the left-space penalty rules the dictionary ships. An invalid setting throws, as setSpacePenalty and build() do.

const { Mode, Tokenizer, loadDictionary } = require("lindera");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, Mode.normal());

// ko-dic without the left-space penalty
const koTokenizer = new Tokenizer(loadDictionary("embedded://ko-dic"), Mode.normal(), null, false);

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of plain token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

Tokens are plain JavaScript objects, not class instances, so they are reclaimed by ordinary garbage collection and pass through JSON.stringify, structuredClone, and worker transfer without conversion. See Token.

tokenizeSurfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no token objects are built and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map((t) => t.surface).

const surfaces = tokenizer.tokenizeSurfaces("形態素解析");
// ["形態素", "解析"]

Parameters:

Returns: string[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: NbestResult[], where each NbestResult is { tokens: Token[], cost: number }

Token

Token is a plain object describing a single morphological token. It is a TypeScript interface, not a class: it has no methods and no prototype, and every field is read directly.

Properties

Reading Details

Index details directly. Out-of-range indexes yield undefined.

const token = tokenizer.tokenize("東京")[0];
const pos = token.details[0];      // e.g., "名詞"
const subpos = token.details[1];   // e.g., "固有名詞"
const reading = token.details[7];  // e.g., "トウキョウ"

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

TokenizerBuilder.setMode() and the Tokenizer constructor take the mode as a Mode, which holds the penalty of decompose mode as a Penalty.

Mode

The tokenization mode: normal, or decompose with a Penalty. It is the counterpart of the Rust lindera::mode::Mode (see Segmenter), and setMode() and the Tokenizer constructor take it:

• Normal mode -- standard dictionary-cost-based tokenization
• Decompose mode -- penalty-based decomposition of compound words

Mode has no constructor, so new Mode() throws; the static factories below create a mode. A mode cannot be changed.

Mode.normal()

Creates normal mode.

const { Mode } = require("lindera");

const mode = Mode.normal();

Mode.decompose(penalty?)

Creates decompose mode with penalty, a Penalty object. null or undefined (the default) uses the default penalty, and a field left out keeps its default. A value that is not an object, or a field that is not an integer in its range, throws an InvalidArg error.

const decompose = Mode.decompose(); // the default penalty
const custom = Mode.decompose({ kanjiPenaltyLengthPenalty: 0 });

Mode.fromStr(name)

Creates a mode from its name, as the Rust Mode parses one: "normal" is normal mode, and "decompose" is decompose mode with the default penalty. The name is case sensitive; any other name, including "Normal" and "Decompose", throws an Error whose message contains Invalid mode. Use it for a mode name read at run time, such as a setting:

const mode = Mode.fromStr("decompose");
console.log(mode.equals(Mode.decompose()));
// true

Properties and Methods

=== compares objects, not modes: each call of Mode.normal() returns a new object, so compare modes with equals.

const { Mode } = require("lindera");

const mode = Mode.decompose({ kanjiPenaltyLengthPenalty: 0 });
console.log(mode.name, mode.isNormal(), mode.isDecompose());
// decompose false true

// A copy with every field set: the field left out has its default
const penalty = mode.penalty;
console.log(penalty.kanjiPenaltyLengthPenalty, penalty.otherPenaltyLengthPenalty);
// 0 1700

// Changing the copy does not change the mode
penalty.otherPenaltyLengthPenalty = 0;
console.log(mode.penalty.otherPenaltyLengthPenalty);
// 1700

console.log(Mode.normal().penalty);
// null

console.log(Mode.decompose().equals(Mode.fromStr("decompose")));
// true
console.log(mode.equals(Mode.decompose()));
// false
console.log(`${Mode.normal()}`);
// normal

Penalty

A plain object (the TypeScript type Penalty) with the penalty parameters of decompose mode. Every field is optional, and a field left out keeps its default:

Pass it to Mode.decompose(), and the mode to setMode() or to the Tokenizer constructor. Without length penalties, decompose mode keeps long words whole, as normal mode does:

const { Mode, Tokenizer, TokenizerBuilder, loadDictionary } = require("lindera");

const text = "関西国際空港限定トートバッグ";
const penalty = { kanjiPenaltyLengthPenalty: 0, otherPenaltyLengthPenalty: 0 };
const surfaces = (tokenizer) => tokenizer.tokenize(text).map((token) => token.surface);

const builder = new TokenizerBuilder().setDictionary("embedded://ipadic");
console.log(surfaces(builder.setMode(Mode.normal()).build()));
// [ '関西国際空港', '限定', 'トートバッグ' ]

console.log(surfaces(builder.setMode(Mode.decompose()).build()));
// [ '関西', '国際', '空港', '限定', 'トートバッグ' ]

console.log(surfaces(builder.setMode(Mode.decompose(penalty)).build()));
// [ '関西国際空港', '限定', 'トートバッグ' ]

const tokenizer = new Tokenizer(loadDictionary("embedded://ipadic"), Mode.decompose(penalty));
console.log(surfaces(tokenizer));
// [ '関西国際空港', '限定', 'トートバッグ' ]

Dictionary Management

Lindera Node.js provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use loadDictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

const { loadDictionary } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

const dictionary = loadDictionary("embedded://ipadic");

Dictionary exposes a few read-only accessors for inspecting the loaded dictionary's metadata:

console.log(dictionary.metadataName());     // e.g. "ipadic"
console.log(dictionary.metadataEncoding()); // e.g. "UTF-8"

const metadata = dictionary.metadata(); // full Metadata object
console.log(metadata.defaultWordCost);

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary. Load one for the system dictionary it is used with:

const { loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.csv", dictionary);

A .csv is built with the system dictionary's metadata and checked against its connection matrix as it is built: a row whose context ID lies outside the matrix throws an error that names its CSV line and surface (see Context IDs). A .bin is loaded as it was built.

loadUserDictionary(uri, metadata) loads a user dictionary with metadata only. A .csv loaded this way is checked when it is combined with the system dictionary, and the error names the entry by its word id:

const { loadUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary.csv", metadata);

Pass the user dictionary when building a tokenizer:

const { Mode, Tokenizer, loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.csv", dictionary);

const tokenizer = new Tokenizer(dictionary, Mode.normal(), userDict);

Or via the builder:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
builder.setUserDictionary("/path/to/user_dictionary.csv");
const tokenizer = builder.build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

const { buildDictionary, Metadata } = require("lindera");

const metadata = new Metadata({ name: "custom", encoding: "UTF-8" });
buildDictionary("/path/to/input_dir", "/path/to/output_dir", metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

To build a dictionary with the settings its metadata.json ships, load the file with Metadata.fromJsonFile() and pass the result. The file's space_penalty rules are kept, so a ko-dic built this way applies the left-space penalty by default (see setSpacePenalty). A Metadata created with new Metadata(...) carries no rules.

const metadata = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
buildDictionary("/path/to/mecab-ko-dic", "/path/to/output_dir", metadata);

User Dictionary

Build a user dictionary from a CSV file for the system dictionary it is used with:

const { buildUserDictionaryFor, loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
buildUserDictionaryFor("user_words.csv", "/path/to/user_dictionary.bin", dictionary);
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.bin", dictionary);

The CSV is built with the system dictionary's metadata and checked against its connection matrix: a row whose context ID lies outside the matrix throws an error that names its CSV line and surface, and no file is written (see Context IDs). The second argument is the path of the file to write. End it with .bin, since the loaders choose how to load a user dictionary by its extension.

buildUserDictionary(kind, inputFile, outputDir, metadata) builds a user dictionary with metadata only. Despite its name, outputDir is the path of the file to write too, not a directory. The rows are checked when the user dictionary is combined with the system dictionary, and the error names the entry by its word id:

const { buildUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
buildUserDictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin", metadata);

The metadata parameter is optional. When omitted, default metadata values are used:

buildUserDictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin");

[!NOTE] The first argument (kind) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today. To build for a system dictionary, use buildUserDictionaryFor.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

const { Metadata } = require("lindera");

// Default metadata
const metadata = new Metadata();

// Custom metadata
const metadata = new Metadata({
  name: "my_dictionary",
  encoding: "UTF-8",
  defaultWordCost: -10000,
});

Loading from JSON

const metadata = Metadata.fromJsonFile("metadata.json");

Properties

[!NOTE] Schema information (dictionary/user-dictionary field layout) is not exposed on this binding's Metadata object -- there are no dictionarySchema / userDictionarySchema properties. Use the standalone Schema class instead.

All properties support both getting and setting:

const metadata = new Metadata();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

toObject()

Returns a plain object representation of the metadata, with every value as a string:

const metadata = new Metadata({ name: "test" });
console.log(metadata.toObject());

When the metadata carries left-space penalty rules (for example, when it was loaded from ko-dic's metadata.json), the object also has a spacePenalty entry holding the rules as a JSON string. Otherwise the key is absent.

const koDic = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
const { rules } = JSON.parse(koDic.toObject().spacePenalty);
console.log(rules[1].pos); // ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"]

Schema

The Schema class defines the field structure of dictionary entries.

Creating a Schema

const { Schema } = require("lindera");

// Default IPADIC-compatible schema
const schema = Schema.createDefault();

// Custom schema
const custom = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Methods

const schema = Schema.createDefault();

console.log(schema.fieldCount());           // 13 (IPADIC format)
console.log(schema.getFieldIndex("pos1"));  // e.g., 4
console.log(schema.getAllFields());          // ["surface", "left_id", ...]
console.log(schema.getCustomFields());      // Fields from index 4 onward

FieldDefinition

FieldType

Text Processing Pipeline

Lindera Node.js supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. See Filters for the full, authoritative catalogue of every character and token filter that lindera-analysis ships, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
const tokenizer = builder.build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("mapping", {
  mapping: {
    "\u30fc": "-",
    "\uff5e": "~",
  },
});
const tokenizer = builder.build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendCharacterFilter("japanese_iteration_mark", {
  normalize_kanji: true,
  normalize_kana: true,
});
const tokenizer = builder.build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("lowercase", {});
const tokenizer = builder.build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_base_form", {});
const tokenizer = builder.build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_stop_tags", {
  tags: ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞"],
});
const tokenizer = builder.build();

[!NOTE] Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*).

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed. The following keeps only general nouns:

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.appendTokenFilter("japanese_keep_tags", {
  tags: ["名詞,一般"],
});
const tokenizer = builder.build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

const { Mode, TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setMode(Mode.normal());
builder.setDictionary("embedded://ipadic");
// Preprocessing
builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
builder.appendCharacterFilter("japanese_iteration_mark", {
  normalize_kanji: true,
  normalize_kana: true,
});
// Postprocessing
builder.appendTokenFilter("japanese_base_form", {});
builder.appendTokenFilter("japanese_stop_tags", {
  tags: ["助詞,格助詞,一般", "助詞,係助詞", "助詞,連体化", "助動詞", "記号,句点", "記号,読点"],
});
builder.appendTokenFilter("lowercase", {});
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and punctuation
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Node.js supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-nodejs with the train feature enabled (enabled by default):

npm run build -- --features train

Training a Model

[!NOTE] The file paths below (resources/training/*.csv, *.def) are illustrative placeholders — this repository does not ship those exact files. For a complete, runnable example that generates a seed lexicon, corpus, and definition files on the fly and then trains/exports/builds a real dictionary, see lindera-nodejs/examples/train_and_export.js.

Use train() to train a CRF model from a seed lexicon and annotated corpus:

const { train } = require("lindera");

train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 1.0,
  maxIter: 100,
  maxThreads: 4,
});

Training Parameters

The settings are checked before any file is read: an unknown regularization, or a lambda, elasticNetL1Ratio or eta outside its range, throws an InvalidArg error. To train with v6's default settings (the model still differs from v6's, since the first line search changed in #1290), pass regularization: "l1" and lambda: 0.01, and eta: 0 to stop only at maxIter.

Exporting a Trained Model

After training, export the model to dictionary source files using exportModel():

const { exportModel } = require("lindera");

exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

Export Parameters

The export creates the following files in the output directory, the same files that lindera export creates:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions (copied from the trained model)
• feature.def -- Feature template definitions (copied from the trained model)
• rewrite.def -- Feature rewrite rules (copied from the trained model)
• left-id.def -- Left context ID to feature string mapping
• right-id.def -- Right context ID to feature string mapping
• metadata.json -- Updated metadata (when metadata parameter is provided)

The model file must come from this version's training. A model file written by an earlier version, which includes every model file written before 7.0.0, is rejected: exportModel() fails with an error that asks you to re-run training. Train such a model again with train().

Complete Workflow

The full workflow for training and using a custom dictionary:

const {
  train,
  exportModel,
  buildDictionary,
  Metadata,
  Mode,
  TokenizerBuilder,
} = require("lindera");

// Step 1: Train the CRF model
train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 1.0,
  maxIter: 100,
});

// Step 2: Export to dictionary source files
exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

// Step 3: Build the dictionary from exported source files
const metadata = Metadata.fromJsonFile("/tmp/dictionary_source/metadata.json");
buildDictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata);

// Step 4: Use the trained dictionary
const builder = new TokenizerBuilder();
builder.setDictionary("/tmp/dictionary");
builder.setMode(Mode.normal());
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("形態素解析のテスト");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Lindera Ruby

Lindera Ruby provides Ruby bindings for the Lindera morphological analysis engine, built with Magnus and rb-sys. It brings Lindera's high-performance tokenization capabilities to the Ruby ecosystem with support for Ruby 3.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Integer arguments

An integer parameter takes only the values its setting can hold. default_word_cost of Lindera::Metadata.new takes -32768 to 32767, the context IDs 0 to 65535, and the penalties of Lindera::Penalty.new a 32-bit integer. A count or an index takes 0 or more: the thresholds of Lindera::Penalty.new, index of Schema#get_field_name and Token#get_detail, and n of Tokenizer#tokenize_nbest. max_iter and max_threads of Lindera.train take 1 or more. cost_threshold of Tokenizer#tokenize_nbest takes a 64-bit integer. A value outside the range raises a RangeError that names the parameter and the range; an index past the end still returns nil:

require 'lindera'

begin
  Lindera::Metadata.new(nil, nil, 100_000, nil, nil, nil, nil, nil)
rescue RangeError => e
  puts "#{e.class}: #{e.message}"
  # RangeError: default_word_cost must be an integer between -32768 and 32767, got 100000
end

tokenizer = Lindera::Tokenizer.new(Lindera.load_dictionary('embedded://ipadic'))

begin
  tokenizer.tokenize_nbest('東京都', -1, nil, nil)
rescue RangeError => e
  puts "#{e.class}: #{e.message}"
  # RangeError: n must be an integer between 0 and 18446744073709551615, got -1
end

The float settings of Lindera.train are checked the same way, before any file is read: lambda and eta: take a finite number of 0 or more, and elastic_net_l1_ratio: a number between 0 and 1 (see Training).

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

From v6.1.0, lindera-ruby installs from RubyGems as lindera. It is a source gem: gem install compiles the native extension from the bundled Rust source, which takes a few minutes.

Prerequisites

• Ruby 3.1 or later, with the tools needed to build native gems (a C compiler and make)
• Rust toolchain 1.89 or later -- Install via rustup
• libclang -- libclang-dev on Debian/Ubuntu; the Xcode command line tools on macOS
• Bundler -- only to build from source (gem install bundler)

Install from RubyGems

gem install lindera

Or add it to your Gemfile:

gem "lindera"

[!NOTE] Use lindera v6.1.0 or later. The lindera gems published before v6.1.0 fail to compile during installation; for those versions, build from source as described below.

The gem compiles against the crates.io releases of lindera and lindera-binding of the same version, which it pins exactly. It ships no Cargo.lock, so their dependencies, including the other lindera-* crates, resolve to the latest compatible releases at install time. It builds with the default features (train) and no embedded dictionary; see Obtaining Dictionaries. To enable other feature flags, set LINDERA_FEATURES when installing:

LINDERA_FEATURES="embed-ipadic" gem install lindera

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build the native extension of lindera-ruby in release mode. Rake puts it in lindera-ruby/lib/lindera/ and does not install the gem; Ruby run with bundle exec from lindera-ruby/ loads it from there:

cd lindera-ruby
bundle install
bundle exec rake compile

For a debug build, run bundle exec rake compile:dev instead of bundle exec rake compile.

Or use the project Makefile, which makes the same release build with the embed-ipadic and train features:

make build-lindera-ruby

Run make test-lindera-ruby to run both the Rust unit tests and the Ruby minitest suite.

Building the Gem

bundle exec rake build (or make package-lindera-ruby) writes the source gem to lindera-ruby/pkg/. It packages the crate as cargo package normalizes it, so the gem compiles outside this repository, against the crates.io releases of lindera and lindera-binding of the same version. Do not run gem build directly: the crate's own Cargo.toml only resolves inside the Cargo workspace.

make test-lindera-ruby-gem (requires Docker) builds the gem, installs it with gem install in a clean ruby container and tokenizes text with it. The test compiles the lindera crates from the checkout, so it also works for versions not yet released on crates.io.

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

LINDERA_FEATURES="train" bundle exec rake compile

Feature Flags

Features are specified through the LINDERA_FEATURES environment variable as a comma-separated list, both for bundle exec rake compile and for gem install.