Lindera

A morphological analysis library in Rust. Lindera is forked from kuromoji-rs and aims to provide easy installation and concise APIs for tokenizing text in multiple languages.

Key Features

Tokenization Flow

graph LR
    subgraph Your Application
        T["Text"]
    end
    subgraph Lindera
        CF["Character Filters"]
        SEG["Segmenter\n(Dictionary + Viterbi)"]
        TF["Token Filters"]
    end
    T --> CF --> SEG --> TF --> R["Tokens"]

Document Map

Quick Example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = tokenizer.tokenize(text)?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Run the example:

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

Output:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

License

Lindera is released under the MIT License.

Architecture

Lindera is organized as a Cargo workspace comprising multiple crates. Each crate has a focused responsibility, from low-level CRF computation to high-level CLI and language bindings.

Crate Dependency Graph

graph TB
    CRF["lindera-crf\n(CRF Engine)"]
    DICT["lindera-dictionary\n(Dictionary Base)"]
    TRAINER["lindera-trainer\n(CRF Training)"]
    IPADIC["lindera-ipadic"]
    UNIDIC["lindera-unidic"]
    KODIC["lindera-ko-dic"]
    CCCEDICT["lindera-cc-cedict"]
    JIEBA["lindera-jieba"]
    NEOLOGD["lindera-ipadic-neologd"]
    LIB["lindera\n(Segmenter)"]
    ANALYSIS["lindera-analysis\n(Analysis Chain)"]
    CLI["lindera-cli\n(CLI)"]
    BINDINGCORE["lindera-binding-core"]
    PY["lindera-python"]
    NODEJS["lindera-nodejs"]
    RUBY["lindera-ruby"]
    PHP["lindera-php"]
    WASM["lindera-wasm"]

    CRF --> TRAINER
    DICT --> TRAINER
    TRAINER -.->|"train feature"| LIB
    DICT --> IPADIC
    DICT --> UNIDIC
    DICT --> KODIC
    DICT --> CCCEDICT
    DICT --> JIEBA
    DICT --> NEOLOGD
    DICT --> LIB
    DICT --> ANALYSIS
    DICT --> WASM
    IPADIC --> LIB
    UNIDIC --> LIB
    KODIC --> LIB
    CCCEDICT --> LIB
    JIEBA --> LIB
    NEOLOGD --> LIB
    LIB --> ANALYSIS
    LIB --> CLI
    ANALYSIS --> CLI
    LIB --> BINDINGCORE
    ANALYSIS --> BINDINGCORE
    BINDINGCORE --> PY
    BINDINGCORE --> NODEJS
    BINDINGCORE --> RUBY
    BINDINGCORE --> PHP
    BINDINGCORE --> WASM

Crate Overview

Tokenization Pipeline

Lindera processes text through a multi-stage pipeline:

Input Text
  |
  v
Character Filters    -- Normalize characters (e.g., Unicode normalization, mapping)
  |
  v
Segmenter            -- Segment text into tokens using a dictionary and the Viterbi algorithm
  |
  v
Token Filters        -- Transform tokens (e.g., POS filtering, stop words, stemming)
  |
  v
Output Tokens

The Segmenter is the core component. It builds a lattice of candidate tokens from the dictionary, then applies the Viterbi algorithm to find the lowest-cost path, producing the most likely segmentation.

Feature Flags

Learn More

• Getting Started -- Installation and first steps
• Core Concepts -- Dictionaries, tokenization, and filters
• Lindera Library -- Segmenter and API
• Lindera Analysis -- Character filters, token filters, and the Tokenizer
• Lindera Dictionary -- Dictionary loading and building
• Lindera Trainer -- CRF-based dictionary training
• Lindera CRF -- The CRF engine
• Lindera CLI -- Command-line interface
• Development Guide -- Build, test, and contribute

Getting Started

This section will guide you through installing Lindera and running your first morphological analysis.

• Installation -- Add Lindera to your project and configure environment variables
• Quick Start -- Tokenize your first text in just a few lines of code
• Examples -- Explore example programs for common use cases

Installation

Put the following in Cargo.toml:

[dependencies]
lindera = "5.0"

[!NOTE] v5.0.0 is the next planned release and has not been published to crates.io yet; the current published version is 4.0.1. This guide describes the v5.0.0 API, which already exists on the main branch. See Migration v4 to v5 for details.

Dictionary Setup

Lindera requires a pre-built dictionary at runtime. Download a dictionary from GitHub Releases and specify its path when loading:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

#![allow(unused)]
fn main() {
// Cargo.toml: lindera = { version = "5.0", features = ["embed-ipadic"] }
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

Environment Variables

LINDERA_BUILD_DICTIONARY_CACHE_DIR

The LINDERA_BUILD_DICTIONARY_CACHE_DIR environment variable designates a build-time cache directory for the embedded-dictionary build pipeline. It is read only by the dictionary crates' build scripts and has no effect at runtime.

When set, each build stores two kinds of files under $LINDERA_BUILD_DICTIONARY_CACHE_DIR/<version>/ (where <version> is the dictionary crate version):

• the downloaded distribution archive (validated with MD5; invalid files are re-downloaded)
• the built binary dictionary that gets embedded into the crate

This enables:

• Offline builds: once cached, subsequent builds need no network access
• Faster builds: download and dictionary build are skipped when valid cached files exist
• Reproducible builds: consistent dictionary versions across builds

Usage:

export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache
cargo build --features=embed-ipadic

Notes:

• The directory is managed automatically and is safe to delete; contents are re-downloaded and rebuilt as needed
• Version subdirectories accumulate across upgrades and are not garbage-collected; old ones can be removed freely
• Setting this variable causes dictionary crates to download and build their dictionaries even when no embed-* feature is enabled (useful for pre-populating the cache)

Deprecated: the previous name LINDERA_DICTIONARIES_PATH still works as a fallback (the new name wins when both are set) and will be removed in v6.0.0.

LINDERA_CONFIG_PATH

The LINDERA_CONFIG_PATH environment variable specifies the path to a YAML configuration file for the tokenizer. This allows you to configure tokenizer behavior without modifying Rust code.

export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

See the Configuration section for details on the configuration format.

DOCS_RS

The DOCS_RS environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.

This is primarily used internally by docs.rs and typically doesn't need to be set by users.

LINDERA_WORKDIR

The LINDERA_WORKDIR environment variable is automatically set during the build process by the lindera-dictionary crate. It points to the directory containing the built dictionary data files and is used internally by dictionary crates to locate their data files.

This variable is set automatically and should not be modified by users.

Quick Start

This example covers the basic usage of Lindera.

It will:

• Create a segmenter in normal mode
• Segment the input text
• Output the tokens

This example uses the embed-ipadic feature, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

use std::borrow::Cow;

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = segmenter.segment(Cow::Borrowed(text))?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

The above example can be run as follows:

% cargo run --features embed-ipadic --example=segment

[!TIP] If you prefer not to embed the dictionary into the binary, download a pre-built IPADIC dictionary from GitHub Releases, extract it to a local directory (e.g., /path/to/ipadic), and call load_dictionary("/path/to/ipadic") instead — no embed-ipadic feature needed in that case. See Feature Flags for details.

You can see the result as follows:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

[!NOTE] Character filters, token filters, and the Tokenizer API are provided by the companion lindera-analysis crate (as of v5.0). Add lindera-analysis = "5.0" to your dependencies if you need the analysis chain.

Examples

Lindera includes several example programs that demonstrate common use cases. The source code is available in the examples directory on GitHub.

The tokenize* examples use the Tokenizer and filter APIs, which are provided by the lindera-analysis crate (as of v5.0).

All examples below are run with the embed-ipadic feature enabled, which downloads the IPADIC dictionary and embeds it into the binary automatically at build time — no manual dictionary download is required.

Available Examples

segment

Basic morphological segmentation with the Segmenter API — the lindera crate alone is enough.

cargo run -p lindera --features=embed-ipadic --example=segment

tokenize

Basic tokenization using an external IPADIC dictionary. Segments input text and prints each token with its part-of-speech details.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize

tokenize_with_user_dict

Tokenization with a user dictionary. Shows how to supplement the dictionary with custom entries for domain-specific terms.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_user_dict

tokenize_with_filters

Tokenization with character filters and token filters. Demonstrates the text processing pipeline, including Unicode normalization, part-of-speech filtering, and other transformations.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_filters

tokenize_with_config

Tokenization using a YAML configuration file. Shows how to configure the tokenizer declaratively instead of programmatically.

cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_config

Core Concepts

This section explains the fundamental concepts behind Lindera's morphological analysis system.

• Morphological Analysis - How Lindera segments and analyzes text.
• Dictionaries - Dictionary formats supported by Lindera.
• Tokenization - Tokenization modes and N-Best analysis.
• User Dictionary - Adding custom words with user dictionaries.
• Filters - Pre-processing text and post-processing tokens with character and token filters.

Morphological Analysis

What is morphological analysis?

Morphological analysis is the process of breaking down text into its smallest meaningful units (morphemes) and identifying their grammatical properties. For languages like Japanese, Chinese, and Korean -- where words are not separated by spaces -- morphological analysis is an essential first step for natural language processing tasks such as search indexing, text classification, and machine translation.

How Lindera works

Lindera is a dictionary-based morphological analyzer. It uses a pre-compiled system dictionary containing known words along with their costs, and applies the Viterbi algorithm to find the optimal segmentation of input text.

The analysis process works as follows:

1. Lattice construction: Lindera scans the input text and looks up all possible words in the dictionary at every position, building a directed acyclic graph (lattice) of candidate segmentations.
2. Cost assignment: Each candidate word has an associated word cost (from the dictionary), and each pair of adjacent words has a connection cost (from the connection cost matrix).
3. Optimal path search: The Viterbi algorithm finds the path through the lattice with the minimum total cost, producing the best segmentation.

Key terminology

Cost-based segmentation

The Viterbi algorithm selects the segmentation path with the minimum total cost. The total cost of a path is the sum of:

• Word costs: Each word in the dictionary has an associated cost. Lower cost means the word is more likely to appear. Common words tend to have lower costs, while rare words have higher costs.
• Connection costs: The cost of connecting two adjacent words, determined by the right context ID of the left word and the left context ID of the right word.

The algorithm computes:

Total cost = sum of word costs + sum of connection costs

By minimizing this total cost, Lindera finds the most natural segmentation of the input text.

Connection cost matrix

The connection cost matrix stores the cost of transitioning from one word to another. It is a two-dimensional matrix indexed by:

• The right context ID of the preceding word
• The left context ID of the following word

These context IDs encode grammatical information about word boundaries. For example, the connection cost between a noun and a particle is typically low (natural sequence), while the connection cost between two verbs in base form might be high (unnatural sequence).

The connection cost matrix is compiled into binary format as part of the dictionary build process and is loaded at runtime for efficient lookup.

Dictionaries

Lindera supports various dictionaries for Japanese, Korean, and Chinese morphological analysis. Each dictionary is provided as a separate crate.

Obtaining Dictionaries

Pre-built dictionaries are available for download from GitHub Releases. Download the dictionary archive for your target language and extract it to a local directory.

#![allow(unused)]
fn main() {
// Load an external dictionary from a local path
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you need a self-contained binary without external dictionary files, you can embed dictionaries using the embed-* feature flags and load them using the embedded:// scheme:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

See each dictionary crate's documentation for format details, build instructions, and usage examples.

Tokenization

Lindera provides multiple tokenization modes and supports N-Best analysis for enumerating alternative segmentation candidates.

Tokenization modes

Normal mode

Normal mode performs standard tokenization based on dictionary entries. Compound words that exist as single entries in the dictionary are kept as-is.

Example -- tokenizing "関西国際空港限定トートバッグ" in Normal mode:

関西国際空港 | 限定 | トートバッグ

The compound noun "関西国際空港" (Kansai International Airport) is preserved as a single token because it exists as one entry in the dictionary.

Decompose mode

Decompose mode further breaks down compound nouns into their constituent parts, even when the compound exists as a dictionary entry.

Example -- tokenizing "関西国際空港限定トートバッグ" in Decompose mode:

関西 | 国際 | 空港 | 限定 | トートバッグ

The compound "関西国際空港" is decomposed into "関西", "国際", and "空港".

Selecting a mode

In Rust, specify the mode when creating a Segmenter:

#![allow(unused)]
fn main() {
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;

// Normal mode
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

// Decompose mode
let segmenter = Segmenter::new(Mode::Decompose(Default::default()), dictionary, None);
}

With the CLI, use the --mode flag:

echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode normal
echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode decompose

[!NOTE] The embedded:// scheme requires lindera-cli to be built with the corresponding embed-* feature (e.g. cargo install lindera-cli --features=embed-ipadic). A default build of lindera-cli does not enable any embed-* feature, and passing --dict embedded://ipadic without it fails with Invalid dictionary scheme: embedded.

N-Best tokenization

N-Best tokenization enumerates the top N tokenization candidates ordered by total path cost (lower cost = better segmentation). This is useful when the best result is ambiguous, or when you want to explore alternative interpretations of the input text.

Algorithm

N-Best tokenization is based on the Forward-DP Backward-A* algorithm, which is compatible with MeCab's N-Best implementation. The forward pass computes optimal costs using dynamic programming, and the backward pass uses A* search to enumerate paths in order of increasing total cost.

Parameters

The tokenize_nbest method accepts the following parameters:

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "すもももももももものうち";

    // Get top 3 tokenization results
    let results = tokenizer.tokenize_nbest(text, 3, false, None)?;

    for (rank, (tokens, cost)) in results.iter().enumerate() {
        println!("--- NBEST {} (cost={}) ---", rank + 1, cost);
        for token in tokens {
            let details = token.details().join(",");
            println!("{}\t{}", token.surface.as_ref(), details);
        }
    }

    Ok(())
}

Output:

--- NBEST 1 (cost=21245) ---
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
--- NBEST 2 (cost=24541) ---
...

CLI example

echo "すもももももももものうち" | lindera tokenize --dict embedded://ipadic -N 3

Lattice reuse

For repeated tokenization, you can reuse a Lattice to reduce memory allocations:

#![allow(unused)]
fn main() {
use lindera_dictionary::viterbi::Lattice;

let mut lattice = Lattice::default();
let results = tokenizer.tokenize_nbest_with_lattice(text, &mut lattice, 3, false, None)?;
}

User Dictionary

A user dictionary is a supplementary dictionary that allows you to register custom words alongside the system dictionary. This is useful for domain-specific terms, brand names, proper nouns, or any words that are not in the default system dictionary.

CSV format

The simplest user dictionary format is a CSV file with three columns:

<surface>,<part_of_speech>,<reading>

Example CSV content

東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ

Each dictionary type (IPADIC, UniDic, ko-dic, etc.) also supports a detailed CSV format with full control over context IDs, costs, and all feature fields. See the Dictionaries section for the detailed format of each dictionary type.

Rust API example

use std::fs::File;
use std::path::PathBuf;

use lindera::dictionary::{Metadata, load_dictionary, load_user_dictionary};
use lindera::error::LinderaErrorKind;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let user_dict_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("user_dict")
        .join("ipadic_simple_userdic.csv");

    let metadata_file = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../lindera-ipadic")
        .join("metadata.json");
    let metadata: Metadata = serde_json::from_reader(
        File::open(metadata_file)
            .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
            .unwrap(),
    )
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
    .unwrap();

    let dictionary = load_dictionary("embedded://ipadic")?;
    let user_dictionary = load_user_dictionary(user_dict_path.to_str().unwrap(), &metadata)?;
    let segmenter = Segmenter::new(
        Mode::Normal,
        dictionary,
        Some(user_dictionary), // Using the loaded user dictionary
    );

    // Create a tokenizer.
    let tokenizer = Tokenizer::new(segmenter);

    // Tokenize a text.
    let text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です";
    let mut tokens = tokenizer.tokenize(text)?;

    // Print the text and tokens.
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Output:

text:   東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です
token:  東京スカイツリー        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリー,*
token:  の      助詞,連体化,*,*,*,*,の,ノ,ノ
token:  最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
token:  は      助詞,係助詞,*,*,*,*,は,ハ,ワ
token:  とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリーエキ,*
token:  です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス

Note that the base_form field (the 7th CSV/detail field) is * rather than the surface form. The simple three-column user-dictionary schema only supplies surface, part_of_speech, and reading — every other field, including base_form, is filled with the dictionary's metadata.default_field_value (* for IPADIC).

Building a user dictionary with CLI

You can build a user dictionary from CSV to binary format using the CLI:

lindera build --src <source_dir> --dest <dest_dir> --metadata <metadata.json> --user

Binary vs CSV user dictionary

• CSV format: Loaded and parsed at runtime. Convenient for development and small dictionaries.
• Binary format: Pre-compiled for faster loading. Recommended for production use with large user dictionaries.

Both formats can be specified when creating a Segmenter. The binary format skips the CSV parsing step, resulting in faster startup times.

Filters

Lindera's analysis pipeline (the Tokenizer provided by the lindera-analysis crate) has two extension points around the segmenter: character filters, which transform the raw input text before tokenization, and token filters, which transform the list of tokens after tokenization.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character filters

Character filters preprocess the input text before it reaches the segmenter. They're typically used to normalize text so that tokenization is more consistent -- for example, converting full-width characters to half-width, canonicalizing Unicode representations, or expanding Japanese iteration marks into their repeated form. Because a character filter can change the length of the text, Lindera tracks every transformation it makes and corrects each token's byte offsets back to positions in the original, unfiltered text.

Token filters

Token filters postprocess the list of tokens produced by the segmenter. They're typically used to normalize or reduce the token list for search and analysis -- for example, replacing a token with its base (dictionary) form, converting a token's surface text between hiragana and katakana, removing tokens by part-of-speech tag, or removing stop words.

Configuring filters

Every filter -- character or token -- is identified by a kind string and configured with a JSON args object of filter-specific parameters. Filters run in the order they're added, and there are two ways to add them:

• YAML configuration file: list filters under the character_filters and token_filters keys.

  character_filters:
    - kind: unicode_normalize
      args:
        kind: nfkc
  
  token_filters:
    - kind: japanese_base_form
  

• Rust API: append filters to a Tokenizer, in order, with append_character_filter and append_token_filter.

  #![allow(unused)]
  fn main() {
  // Character filters run first, transforming the raw input text;
  // token filters run last, transforming the resulting token list.
  tokenizer
      .append_character_filter(BoxCharacterFilter::from(unicode_normalize_char_filter))
      .append_token_filter(BoxTokenFilter::from(japanese_base_form_filter));
  }

Available filters

Lindera ships 4 character filters and 18 token filters, covering Japanese, Korean, and general-purpose text normalization:

See the Filters reference for what each filter does, its parameters, and runnable YAML and Rust API examples.

Lindera CRF

Lindera CRF is a pure Rust implementation of Conditional Random Fields (CRFs), forked from rucrf. It provides a trainer and an estimator for CRFs with support for lattice structures.

Key Features

• Lattices with variable length edges
• L1, L2, and Elastic Net regularization
• Multi-threaded training
• Zero-copy deserialization with rkyv
• no_std support via --no-default-features --features alloc

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Changes from rucrf

• Serialization backend: Switched from bincode to rkyv for zero-copy deserialization
• Elastic Net regularization: Added Regularization::ElasticNet combining L1 and L2 penalties
• Rust 2024 edition: Updated to Rust 2024 edition
• Dependency updates: Updated argmin, argmin-math, hashbrown, etc.

Architecture

Module Structure

lindera-crf/src/
├── lib.rs                # Public API re-exports
├── feature.rs            # FeatureSet, FeatureProvider
├── lattice.rs            # Edge, Node, Lattice
├── model.rs              # RawModel, MergedModel, Model trait
├── trainer.rs            # Trainer, Regularization enum
├── errors.rs             # Error types
├── forward_backward.rs   # Forward-backward algorithm
├── math.rs               # Mathematical utilities (logsumexp)
├── optimizers.rs         # Declares the optimizers module
├── optimizers/
│   └── lbfgs.rs          # L-BFGS optimization
└── utils.rs              # Utility traits

Key Components

FeatureProvider / FeatureSet

Manage per-label feature sets. Each FeatureSet holds unigram features and left/right bigram features for a given label (feature IDs only, no weights). FeatureProvider maps label IDs to FeatureSet instances. Weights are held separately, on RawModel (weights, unigram_weight_indices, bigram_weight_indices).

Lattice / Edge / Node

Lattice structure with variable-length edges for sequence labeling. Edge represents a candidate span with a label, while Node aggregates edges at a given position. The Lattice is constructed from input data and used by the model to find the best path.

Trainer

Trains a CRF model using L-BFGS optimization with configurable regularization. The trainer accepts labeled lattice examples, computes gradients via the forward-backward algorithm, and iteratively updates model weights.

Regularization

Configurable regularization strategies:

• L1: Sparse models via L1 penalty
• L2: Smooth models via L2 penalty
• ElasticNet: Combines L1 and L2 with a configurable l1_ratio

Model (trait)

Interface for searching the best path through a lattice. Two implementations are provided:

• RawModel: Stores weights in a flat vector indexed by feature ID
• MergedModel: Optimized for inference; merges feature weights into a compact representation serializable with rkyv, using MergedFeatureSet as its per-label element type

MergedFeatureSet holds the pre-summed unigram weight for a label, along with its left_id/right_id bigram connection IDs into MergedModel's bigram weight matrix.

Forward-backward Algorithm

Computes alpha (forward) and beta (backward) values over the lattice. Used during training to calculate expected feature counts and gradients.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-crf

Lindera Dictionary

Lindera Dictionary is the base library for morphological analysis dictionaries. It provides dictionary loading, building, and Viterbi-based segmentation. CRF-based dictionary training lives in the separate lindera-trainer crate.

Key Features

• Dictionary loading from filesystem or embedded data
• Dictionary building from MeCab-format CSV source files
• Viterbi algorithm for optimal segmentation
• N-best path generation (Forward-DP Backward-A*)
• Memory-mapped file support

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-dictionary/src/
├── lib.rs               # Public API
├── dictionary.rs        # Dictionary, UserDictionary
├── builder.rs           # DictionaryBuilder
├── loader.rs            # DictionaryLoader trait, FSDictionaryLoader
├── viterbi.rs           # Lattice, Edge, Viterbi segmentation
├── nbest.rs             # NBestGenerator (Forward-DP Backward-A*)
├── mode.rs              # Mode (Normal/Decompose), Penalty
├── error.rs             # LinderaError, LinderaErrorKind
├── assets.rs            # Download and file management
├── macros.rs            # embedded_dictionary! macro shared by dictionary crates
├── dictionary/
│   ├── character_definition.rs    # Character type definitions
│   ├── connection_cost_matrix.rs  # Connection cost matrix
│   ├── context_id_map.rs          # ContextIdMap: connection-cost context-ID remap
│   ├── prefix_dictionary.rs       # Double-array trie dictionary
│   ├── unknown_dictionary.rs      # Unknown word handling
│   ├── metadata.rs                # Dictionary metadata
│   └── schema.rs                  # Schema definitions

Key Components

Dictionary / UserDictionary

Main data structures holding the compiled dictionary data. A Dictionary contains the character definitions, connection cost matrix, prefix dictionary (double-array trie), and unknown word dictionary. UserDictionary allows users to add custom vocabulary on top of the system dictionary.

DictionaryBuilder

Fluent API for building dictionaries from source CSV files. It compiles MeCab-format dictionary sources into the binary format used at runtime. Its four build stages (metadata, unknown dictionary, prefix dictionary, connection cost matrix) run concurrently on scoped threads on non-wasm targets, falling back to a sequential build on wasm (which has no OS threads); the concurrent path has higher peak memory, since all four stages' working sets are held at once.

DictionaryLoader / FSDictionaryLoader

DictionaryLoader is a trait for loading compiled dictionaries. FSDictionaryLoader is the filesystem-based implementation that reads dictionary files from a directory, with optional memory-mapped file support.

Embedded Dictionary Macro

The embedded_dictionary! macro (lindera-dictionary/src/macros.rs, #[macro_export]) generates the boilerplate each dictionary crate needs to bake its compiled dictionary into the binary: a load() function that reads the dictionary components via include_bytes! and a loader struct implementing DictionaryLoader. Every dictionary crate's embedded.rs (lindera-ipadic, lindera-ipadic-neologd, lindera-unidic, lindera-ko-dic, lindera-cc-cedict, lindera-jieba) invokes this macro instead of duplicating the loading logic.

Viterbi (Lattice, Edge)

Builds a lattice of candidate tokens from the input text and finds the optimal segmentation path using the Viterbi algorithm. Each Edge in the lattice represents a candidate token with associated costs (word cost + connection cost).

NBestGenerator

Generates N-best segmentation paths using the Forward-DP Backward-A* algorithm. This enables applications to consider alternative segmentations beyond the single best path.

Mode

Controls tokenization behavior:

• Normal: Standard tokenization using the optimal Viterbi path
• Decompose: Further splits compound nouns based on configurable Penalty thresholds

Context ID Remapping

Dictionary metadata (lindera-dictionary/src/dictionary/metadata.rs) carries a connection_id_mapping: bool flag and an optional context_id_map: Option<ContextIdMap>. When a dictionary crate (e.g. lindera-unidic) enables connection_id_mapping, DictionaryBuilder relabels the connection matrix's left/right context IDs by access frequency at build time, so that frequently-used connection-cost cells cluster together for better cache locality; DictionaryBuilder::with_context_id_freq attaches an optional bundled frequency histogram used to rank the IDs, and the remap itself is computed in lindera-dictionary/src/builder/context_id_remap.rs. ContextIdMap (lindera-dictionary/src/dictionary/context_id_map.rs) holds the resulting left/right permutation and is persisted into the built metadata.json so the same mapping can be reapplied later. Because user dictionaries are always compiled in the original, un-remapped ID space, UserDictionary::remap_context_ids (lindera-dictionary/src/dictionary.rs) uses the persisted ContextIdMap to relabel a user dictionary's context IDs into the same space as the system dictionary it is attached to. The remap is a bijective relabeling, so it does not change tokenization output, only lookup locality.

Training

The CRF-based dictionary training pipeline lives in the separate lindera-trainer crate, which builds on this crate's runtime types. See the training pipeline documentation for details.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-dictionary

Lindera Trainer

Lindera Trainer implements the CRF-based dictionary training pipeline (lindera train), turning an annotated corpus and a seed lexicon into a trained model that can be exported to MeCab-format dictionary source files and compiled into a binary dictionary. It builds on the runtime types of lindera-dictionary and the CRF core of lindera-crf. The crate can be used directly, or through the lindera facade's train feature, which re-exports it as lindera::dictionary::trainer.

Key Features

• CRF-based weight learning via lindera-crf, with L1, L2, and Elastic Net regularization
• MeCab-compatible feature template parsing (feature.def: %F[n], %L[n], %R[n], %w, %u, %l, %r, and their optional ? variants)
• MeCab-compatible 3-section feature rewriting (rewrite.def: unigram / left / right rewrite rules)
• Dictionary-format agnostic: works with any lexicon whose columns follow surface,left_id,right_id,cost,feature... (IPADIC, UniDic, ko-dic, CC-CEDICT, etc.)
• Automatic unknown-word categorization driven by char.def character categories
• Connection cost matrix and MeCab-compatible cost conversion (tocost) derived from learned CRF weights
• Zero-copy rkyv binary model serialization, with legacy JSON fallback on read
• Direct export of Lindera/MeCab dictionary source files (lex.csv, matrix.def, unk.def, char.def, feature.def, rewrite.def, left-id.def, right-id.def)

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-trainer/src/
├── lib.rs                # Trainer: CRF training orchestration; public API re-exports
├── config.rs             # TrainerConfig: parses seed lexicon, char.def, feature.def, rewrite.def
├── corpus.rs              # Corpus, Example, Word: training data representation
├── feature_extractor.rs  # FeatureExtractor: feature template parsing and feature ID management
├── feature_rewriter.rs   # DictionaryRewriter: MeCab-compatible 3-section rewrite
└── model.rs               # Model, SerializableModel: trained model storage, serialization, dictionary output

Key Components

TrainerConfig

Parses the five training input files -- seed lexicon (lex.csv), character definition (char.def), unknown-word definition (unk.def), feature template (feature.def), and rewrite rules (rewrite.def) -- into the configuration consumed by Trainer. TrainerConfig::from_readers (or from_paths, its file-path convenience wrapper) extracts the surface/feature vocabulary from the seed lexicon, builds a FeatureExtractor from the parsed feature templates, builds a DictionaryRewriter from the rewrite rules, and assembles a minimal in-memory lindera_dictionary::dictionary::Dictionary (character definition, unknown-word categories, and a system lexicon) from the same input. It also exposes surfaces(), surface_features(), get_features(), a user lexicon accessible via user_lexicon() / add_user_lexicon_entry() / load_user_lexicon_from_content(), and metadata().

Corpus / Example / Word

Represent annotated training data. Word pairs a surface form with its (comma-joined) feature string. Example is one training sentence, built from a Vec<Word> whose surfaces are concatenated to reconstruct the original sentence text. Corpus is a collection of Examples; Corpus::from_reader parses tab-separated surface<TAB>features lines, treating blank lines or literal EOS lines as sentence boundaries.

FeatureExtractor

Parses MeCab-compatible feature templates and manages the mapping from generated feature strings to interned, NonZeroU32 feature IDs. Supported template placeholders are %F[n] / %F?[n] (feature field at index n, the ? form skipped when the value is *), %t (character category), %w (surface form, unigram only), %u / %l / %r (full rewritten unigram/left/right feature string), and %L[n] / %L?[n] / %R[n] / %R?[n] for bigram left/right context fields. extract_unigram_feature_ids[_with_ctx], extract_left_feature_ids[_with_ctx], and extract_right_feature_ids[_with_ctx] apply the parsed templates to a feature vector (plus, for bigram extraction, an optional TemplateContext carrying surface/ufeature/lfeature/rfeature) and return the resulting feature IDs, creating new IDs on first use.

DictionaryRewriter

Implements MeCab's 3-section rewrite.def format: [unigram rewrite], [left rewrite], and [right rewrite] sections, each holding an ordered list of pattern<TAB>replacement rules built into a FeatureRewriter prefix trie (via the internal FeatureRewriterBuilder). DictionaryRewriter::from_reader parses all three sections (falling back to treating an unsectioned file as legacy right-rewrite-only content, for backward compatibility with older Lindera rewrite.def files). rewrite() applies all three rewriters to a feature string and returns (ufeature, lfeature, rfeature), passing a section through unchanged when no rule matches; rewrite_cached() memoizes results per input feature string (MeCab's rewrite2 equivalent).

Model / SerializableModel

Model is the trained model produced by Trainer::train. It owns the trained lindera_crf::RawModel, the TrainerConfig used to train it, the extracted feature weights and labels, and any user lexicon entries added afterward via read_user_lexicon. It converts learned CRF weights into MeCab-compatible integer costs (tocost(weight, cost_factor) = clamp(-weight * cost_factor, -32767, 32767)), with the cost factor computed to make full use of the i16 range (calculate_cost_factor). Model can serialize itself (write_model) and export dictionary source files directly.

SerializableModel is the plain-data, rkyv-serializable counterpart returned by Model::read_model when loading a previously trained model back from a reader (as done by the lindera export CLI command). It carries the same trained information as Model -- feature weights, labels, POS info, connection cost matrix, unknown-word categories, preserved char.def/feature.def/rewrite.def content, cost factor, and left/right context ID maps -- as owned data, without a TrainerConfig, and exposes its own writer methods for producing the dictionary export files.

Model public methods

SerializableModel public methods

Available on the value returned by Model::read_model, for re-exporting dictionary source files from a previously trained and serialized model (used by the lindera export CLI command):

ModelMetadata and FeatureSetInfo

ModelMetadata records summary information about a training run: version, regularization (the cost coefficient used), iterations (max iterations configured), feature_count, and label_count. It is embedded in SerializableModel::metadata.

FeatureSetInfo holds the per-label connection information extracted from the merged CRF model after training: left_id, right_id, and weight. SerializableModel::feature_sets stores one entry per label, in the same order as labels.

API Reference

The API reference is available. Please see following URL:

• lindera-trainer

Lindera Library

The lindera crate is a pure morphological segmenter: it integrates the dictionary crates and provides the Segmenter API. It does not depend on lindera-analysis, lindera-crf, or lindera-trainer by default. This section covers segmentation, error handling, and API reference.

If you need the Tokenizer, character filters, or token filters (a Lucene-style analysis chain built on top of Segmenter), see the separate Lindera Analysis crate, including its Configuration and Filters pages.

• Segmenter - Core segmentation component using the Viterbi algorithm
• Error Handling - Error types and handling patterns
• API Reference - Links to generated API documentation

Segmenter

The Segmenter is the core component that performs morphological analysis. It uses the Viterbi algorithm to find the optimal segmentation of input text based on a dictionary and cost model.

Creating a Segmenter

A Segmenter requires three components:

• Mode - the tokenization strategy (Normal or Decompose)
• Dictionary - a system dictionary for morphological analysis
• UserDictionary (optional) - a supplementary dictionary for custom words

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
}

Tokenization Modes

Mode::Normal

Standard tokenization based on the dictionary entries. Words are segmented faithfully according to what is registered in the dictionary.

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Normal;
}

Mode::Decompose

Decomposes compound nouns into their constituent parts. This mode applies a configurable penalty to long compound words, encouraging the segmenter to split them into shorter components.

For example, with Mode::Normal, the compound word "関西国際空港" in the sentence "関西国際空港限定トートバッグ" remains part of a single token, while with Mode::Decompose, it is split into "関西", "国際", and "空港" (the surrounding context affects whether a compound is split; the same string in isolation may not split the same way).

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Decompose(Default::default());
}

Dictionary Loading

Lindera provides the load_dictionary function to load dictionaries from various sources.

Embedded Dictionaries

When built with the appropriate feature flag (e.g., embed-ipadic), dictionaries can be loaded directly from the binary:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;
}

Available embedded dictionary URIs:

• embedded://ipadic - IPADIC (Japanese)
• embedded://ipadic-neologd - IPADIC NEologd (Japanese)
• embedded://unidic - UniDic (Japanese)
• embedded://ko-dic - ko-dic (Korean)
• embedded://cc-cedict - CC-CEDICT (Chinese)
• embedded://jieba - Jieba (Chinese)

External Dictionaries

Pre-built dictionary directories can be loaded from the filesystem:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("/path/to/dictionary")?;
}

Using with Tokenizer

The Segmenter is typically used through the Tokenizer, which adds support for character filters and token filters:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let tokens = tokenizer.tokenize(text)?;

    for mut token in tokens {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Note the mut binding on token: Token::details takes &mut self, so iterating with a plain for token in tokens fails to compile (E0596: cannot borrow as mutable).

Building from Config

Segmenter::from_config builds a Segmenter from a SegmenterConfig (a serde_json::Value), the same configuration format used by Tokenizer/TokenizerBuilder (see Configuration) but scoped to just the segmenter: section:

#![allow(unused)]
fn main() {
use serde_json::json;
use lindera::segmenter::{Segmenter, SegmenterConfig};

let config: SegmenterConfig = json!({
    "mode": "normal",
    "dictionary": "embedded://ipadic",
    "keep_whitespace": false,
    "use_mmap": false
});
let segmenter = Segmenter::from_config(&config)?;
}

Memory-Mapped Loading

For a filesystem-based (not embedded://) dictionary, set use_mmap to true to route the connection-cost matrix and prefix dictionary through memory-mapped reads instead of a plain file read. Only the dictionary's largest word-list files stay lazily paged this way; the connection-cost matrix and the double-array trie are always fully materialized into owned memory regardless. use_mmap is silently ignored for embedded:// dictionaries, since their data is already a static, zero-copy byte slice. Requires the mmap cargo feature (enabled by default).

Whitespace Handling

By default, whitespace-only tokens are dropped from the output for MeCab compatibility. Call keep_whitespace(true) on a Segmenter to keep them:

#![allow(unused)]
fn main() {
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).keep_whitespace(true);
}

N-Best Segmentation

segment_nbest returns the top-n segmentations ordered by total path cost, each paired with its cost. Set unique to deduplicate results that share the same word boundaries but differ only in POS tags, and cost_threshold to discard paths whose cost exceeds best_cost + threshold:

#![allow(unused)]
fn main() {
let results = segmenter.segment_nbest(Cow::Borrowed("すもももももももものうち"), 3, false, None)?;
for (tokens, cost) in results {
    println!("cost={cost}");
    for token in tokens {
        println!("  {}", token.surface.as_ref());
    }
}
}

segment_nbest_with_lattice is the same operation but lets you pass in a reusable Lattice buffer to avoid reallocating one per call.

Error Handling

Lindera uses a structured error system based on anyhow and thiserror for ergonomic error handling throughout the library.

LinderaResult

The LinderaResult<T> type alias is the standard return type for fallible operations in Lindera:

#![allow(unused)]
fn main() {
pub type LinderaResult<T> = Result<T, LinderaError>;
}

LinderaError

LinderaError is the main error type, containing an error kind and a source error with full context:

#![allow(unused)]
fn main() {
pub struct LinderaError {
    pub kind: LinderaErrorKind,
    source: anyhow::Error,
}
}

The add_context method allows attaching additional context to an error:

#![allow(unused)]
fn main() {
let error = error.add_context("failed to load dictionary from /path/to/dict");
}

LinderaErrorKind

LinderaErrorKind is an enum that categorizes errors:

Creating Errors

Use LinderaErrorKind::with_error to create an error from a kind and a source:

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let error = LinderaErrorKind::Io.with_error(anyhow::anyhow!("file not found: config.yml"));
}

Using the ? Operator

Since Lindera functions return LinderaResult, the ? operator can propagate errors naturally:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn analyze(text: &str) -> LinderaResult<Vec<String>> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let tokens = tokenizer.tokenize(text)?;
    Ok(tokens.iter().map(|t| t.surface.as_ref().to_string()).collect())
}
}

Error Handling Patterns

Matching on Error Kind

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::error::LinderaErrorKind;

match load_dictionary("/path/to/dictionary") {
    Ok(dict) => { /* use dictionary */ }
    Err(e) if e.kind() == LinderaErrorKind::NotFound => {
        eprintln!("Dictionary not found: {}", e);
    }
    Err(e) if e.kind() == LinderaErrorKind::Io => {
        eprintln!("I/O error loading dictionary: {}", e);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}
}

Converting from External Errors

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let content = std::fs::read_to_string("config.yml")
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))?;
}

API Reference

The API reference is available. Please see following URL:

• lindera
• lindera-analysis

Lindera Analysis

Lindera Analysis layers a Lucene-style text analysis chain on top of the pure morphological segmenter provided by the lindera crate. It composes character filters, a Segmenter, and token filters into a single Tokenizer pipeline that can be built programmatically or loaded entirely from a YAML configuration file.

Key Features

• Character filters that transform input text before segmentation, with automatic byte-offset correction back to the original text
• Token filters that transform, merge, filter, or reorder the tokens produced by the segmenter
• Tokenizer / TokenizerBuilder to assemble a full analysis pipeline, either in Rust code or from a YAML file (LINDERA_CONFIG_PATH)
• Built-in filters covering Japanese, Korean, and general-purpose text normalization

Contents

• Configuration -- YAML configuration file format for the Tokenizer
• Filters -- Reference for all character filters and token filters
• Architecture -- Internal structure and key components
• API Reference -- API documentation

Configuration

Lindera is able to read YAML format configuration files. Specify the path to the following file in the environment variable LINDERA_CONFIG_PATH. You can use it easily without having to code the behavior of the tokenizer in Rust code.

segmenter:
  mode: "normal"
  dictionary: "embedded://ipadic"
  # user_dictionary: "./resources/user_dict/ipadic_simple_userdic.csv"
  # keep_whitespace: false
  # use_mmap: false # only meaningful for filesystem (non-embedded://) dictionaries

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true
  - kind: mapping
    args:
       mapping:
         リンデラ: Lindera

token_filters:
  - kind: "japanese_compound_word"
    args:
      tags:
        - "名詞,数"
        - "名詞,接尾,助数詞"
      new_tag: "名詞,数"
  - kind: "japanese_number"
    args:
      tags:
        - "名詞,数"
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "接続詞"
        - "助詞"
        - "助詞,格助詞"
        - "助詞,格助詞,一般"
        - "助詞,格助詞,引用"
        - "助詞,格助詞,連語"
        - "助詞,係助詞"
        - "助詞,副助詞"
        - "助詞,間投助詞"
        - "助詞,並立助詞"
        - "助詞,終助詞"
        - "助詞,副助詞／並立助詞／終助詞"
        - "助詞,連体化"
        - "助詞,副詞化"
        - "助詞,特殊"
        - "助動詞"
        - "記号"
        - "記号,一般"
        - "記号,読点"
        - "記号,句点"
        - "記号,空白"
        - "記号,括弧閉"
        - "その他,間投"
        - "フィラー"
        - "非言語音"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "remove_diacritical_mark"
    args:
      japanese: false

% export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

use std::path::PathBuf;

use lindera_analysis::tokenizer::TokenizerBuilder;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    // Load tokenizer configuration from file
    let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("config")
        .join("lindera.yml");

    let builder = TokenizerBuilder::from_file(&path)?;

    let tokenizer = builder.build()?;

    let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。".to_string();
    println!("text: {text}");

    let tokens = tokenizer.tokenize(&text)?;

    for token in tokens {
        println!(
            "token: {:?}, start: {:?}, end: {:?}, details: {:?}",
            token.surface, token.byte_start, token.byte_end, token.details
        );
    }

    Ok(())
}

Filters

Character filters and token filters are the two pre/post-processing stages of the lindera-analysis Tokenizer pipeline.

• Character filters transform the input text before segmentation. Byte offsets are corrected automatically so that the resulting tokens still report positions relative to the original, unfiltered text.
• Token filters transform the list of tokens produced by the segmenter after segmentation.

Both kinds of filters are configured the same way: a kind string (also used with the CLI's --character-filter / --token-filter flags in the form kind:{"json": "args"}) and a JSON args object with filter-specific parameters.

Character Filters

Character filters are configured under the character_filters key of the YAML configuration file. Each entry is applied in order, and each filter's output feeds into the next.

unicode_normalize

Normalizes the input text using one of the four standard Unicode normalization forms.

Parameters:

Example:

{
  "kind": "unicode_normalize",
  "args": {
    "kind": "nfkc"
  }
}

japanese_iteration_mark

Normalizes Japanese iteration marks (々, ゝ, ゞ, ヽ, ヾ) by replacing each mark with the character it repeats, adding or removing the voiced sound mark (dakuten) as needed for the hiragana/katakana variants.

Parameters:

Example:

{
  "kind": "japanese_iteration_mark",
  "args": {
    "normalize_kanji": true,
    "normalize_kana": true
  }
}

mapping (character filter)

Replaces occurrences of the keys in mapping with their corresponding values, using longest-match search over the input text (built with an Aho-Corasick automaton).

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "リンデラ": "Lindera"
    }
  }
}

regex

Replaces every match of a regular expression in the input text with a literal replacement string. Capture groups are not interpolated into the replacement.

Parameters:

Example:

{
  "kind": "regex",
  "args": {
    "pattern": "\\s{2,}",
    "replacement": " "
  }
}

Token Filters

Token filters are configured under the token_filters key of the YAML configuration file. Each filter is applied in order to the token list produced by the segmenter.

japanese_base_form

Replaces the token's surface text with its base (dictionary) form, as registered in the base_form or orthographic_base_form field of the dictionary. Acts as a lemmatizer for verbs and adjectives. Tokens whose first detail is UNK (unknown words) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_base_form"
}

japanese_compound_word

Merges consecutive tokens whose part-of-speech tag matches one of tags into a single compound token.

Parameters:

Example:

{
  "kind": "japanese_compound_word",
  "args": {
    "tags": [
      "名詞,数",
      "名詞,接尾,助数詞"
    ],
    "new_tag": "名詞,数"
  }
}

japanese_kana

Converts token text between hiragana and katakana.

Parameters:

Example:

{
  "kind": "japanese_kana",
  "args": {
    "kind": "hiragana"
  }
}

japanese_katakana_stem

Removes a trailing prolonged sound mark (ー, U+30FC) from katakana tokens, but only when the token is longer than min characters.

Parameters:

Example:

{
  "kind": "japanese_katakana_stem",
  "args": {
    "min": 3
  }
}

japanese_keep_tags

Keeps only tokens whose part-of-speech tag matches one of tags, removing all others.

Parameters:

Example:

{
  "kind": "japanese_keep_tags",
  "args": {
    "tags": [
      "名詞",
      "名詞,一般",
      "名詞,固有名詞"
    ]
  }
}

japanese_number

Converts Japanese numeral representations (kanji numerals, formal/legal kanji numerals, and fullwidth digits) in the token's surface text to Arabic numerals.

Parameters:

Example:

{
  "kind": "japanese_number",
  "args": {
    "tags": [
      "名詞,数"
    ]
  }
}

japanese_reading_form

Replaces the token's surface text with its reading, in katakana, as registered in the dictionary's reading field. Tokens whose first detail is UNK (unknown words) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_reading_form"
}

japanese_stop_tags

Removes tokens whose part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "japanese_stop_tags",
  "args": {
    "tags": [
      "助詞",
      "助動詞",
      "記号"
    ]
  }
}

keep_words

Keeps only tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "keep_words",
  "args": {
    "words": [
      "すもも",
      "もも"
    ]
  }
}

korean_keep_tags

Keeps only Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_keep_tags",
  "args": {
    "tags": [
      "NNG"
    ]
  }
}

korean_reading_form

Replaces the token's surface text with its reading, as registered in the dictionary's reading field. Tokens whose first detail is UNK (unknown words) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "korean_reading_form"
}

korean_stop_tags

Removes Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_stop_tags",
  "args": {
    "tags": [
      "EP",
      "EF",
      "JKG"
    ]
  }
}

length

Keeps only tokens whose surface text length (in characters) falls within [min, max].

Parameters:

Example:

{
  "kind": "length",
  "args": {
    "min": 2,
    "max": 3
  }
}

lowercase

Converts token surface text to lowercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "lowercase"
}

mapping (token filter)

Replaces occurrences of the keys in mapping with their corresponding values in each token's surface text, using longest-match search (built with an Aho-Corasick automaton). This is the token-level counterpart of the mapping character filter.

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "籠": "篭"
    }
  }
}

remove_diacritical_mark

Removes diacritical (combining) marks from token surface text, re-applying the text's original Unicode normalization form afterward.

Parameters:

Example:

{
  "kind": "remove_diacritical_mark",
  "args": {
    "japanese": false
  }
}

stop_words

Removes tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "stop_words",
  "args": {
    "words": [
      "も",
      "の"
    ]
  }
}

uppercase

Converts token surface text to uppercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "uppercase"
}

YAML Configuration

Character filters and token filters are configured together with the segmenter in a single YAML file. See Configuration for the full file format; the relevant excerpt looks like this:

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true

token_filters:
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "助詞"
        - "助動詞"
        - "記号"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "lowercase"
  - kind: "length"
    args:
      min: 2

Rust API

Character filters and token filters can also be created and applied programmatically:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::character_filter::BoxCharacterFilter;
use lindera_analysis::character_filter::unicode_normalize::{
    UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera_analysis::token_filter::BoxTokenFilter;
use lindera_analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera_analysis::token_filter::japanese_katakana_stem::JapaneseKatakanaStemTokenFilter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let mut tokenizer = Tokenizer::new(segmenter);

    // Add a character filter
    let normalize_filter = UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);
    tokenizer.append_character_filter(BoxCharacterFilter::from(normalize_filter));

    // Add token filters
    let stop_tags_filter = JapaneseStopTagsTokenFilter::new(
        vec![
            "助詞".to_string(),
            "助動詞".to_string(),
            "記号".to_string(),
        ]
        .into_iter()
        .collect(),
    );
    tokenizer.append_token_filter(BoxTokenFilter::from(stop_tags_filter));

    let katakana_stem_filter =
        JapaneseKatakanaStemTokenFilter::new(std::num::NonZeroUsize::new(3).unwrap());
    tokenizer.append_token_filter(BoxTokenFilter::from(katakana_stem_filter));

    // Tokenize with filters applied
    let tokens = tokenizer.tokenize("Linderaは形態素解析エンジンです。")?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

The append_character_filter and append_token_filter methods add filters in order. Character filters are applied sequentially to the text before segmentation; token filters are applied sequentially to the token list after segmentation.

Architecture

Module Structure

lindera-analysis/src/
├── lib.rs                          # Public API re-exports, CLI flag parsing helper
├── character_filter.rs             # CharacterFilter trait, OffsetMapping, CharacterFilterLoader
├── character_filter/
│   ├── unicode_normalize.rs        # Unicode normalization (NFC/NFD/NFKC/NFKD)
│   ├── japanese_iteration_mark.rs  # Japanese iteration mark normalization
│   ├── mapping.rs                  # Mapping-based text replacement
│   └── regex.rs                    # Regex-based text replacement
├── token_filter.rs                 # TokenFilter trait, TokenFilterLoader
├── token_filter/
│   ├── japanese_base_form.rs
│   ├── japanese_compound_word.rs
│   ├── japanese_kana.rs
│   ├── japanese_katakana_stem.rs
│   ├── japanese_keep_tags.rs
│   ├── japanese_number.rs
│   ├── japanese_reading_form.rs
│   ├── japanese_stop_tags.rs
│   ├── keep_words.rs
│   ├── korean_keep_tags.rs
│   ├── korean_reading_form.rs
│   ├── korean_stop_tags.rs
│   ├── length.rs
│   ├── lowercase.rs
│   ├── mapping.rs
│   ├── remove_diacritical_mark.rs
│   ├── stop_words.rs
│   ├── tags.rs                     # Shared keep/stop-tag filtering helpers (private)
│   └── uppercase.rs
└── tokenizer.rs                     # Tokenizer, TokenizerBuilder

Key Components

CharacterFilter

A trait for filters that preprocess text before segmentation. Each implementation provides a name() and an apply(&self, text: &mut String) -> LinderaResult<OffsetMapping> method that rewrites text in place and returns an OffsetMapping describing every transformation it performed.

The OffsetMapping (built from a list of Transformation records) lets the Tokenizer translate token byte offsets computed against the filtered text back to byte offsets in the original input, even after multiple filters have run in sequence. BoxCharacterFilter wraps any CharacterFilter implementation as a boxed, cloneable trait object, and CharacterFilterLoader builds one from a kind string plus a serde_json::Value of arguments (used both by YAML configuration loading and by CLI flag parsing).

TokenFilter

A trait for filters that post-process the tokens produced by the segmenter. Each implementation provides a name() and an apply(&self, tokens: &mut Vec<Token<'_>>) -> LinderaResult<()> method that modifies, merges, reorders, or removes tokens in place. BoxTokenFilter wraps any TokenFilter implementation as a boxed, cloneable trait object, and TokenFilterLoader builds one from a kind string plus a serde_json::Value of arguments, mirroring CharacterFilterLoader.

Tokenizer / TokenizerBuilder

Tokenizer composes character filters, a lindera::segmenter::Segmenter, and token filters into a single analysis pipeline. Calling tokenize runs the character filters over the input text, segments the filtered text, applies the token filters to the resulting tokens, and finally corrects each token's byte offsets back to the original text via the recorded OffsetMappings.

TokenizerBuilder assembles a Tokenizer from a TokenizerConfig (a serde_json::Value), which can be constructed programmatically, loaded from a YAML file (via TokenizerBuilder::from_file, or automatically from the LINDERA_CONFIG_PATH environment variable via TokenizerBuilder::new), or built up incrementally with set_segmenter_mode, set_segmenter_dictionary, append_character_filter, and append_token_filter. See Configuration for the YAML file format and Filters for the full filter reference.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-analysis

Lindera CLI

A morphological analysis command-line interface for Lindera.

• Installation - Install or build the CLI
• Commands - Command reference for list, tokenize, build, train, and export
• Tutorial - Step-by-step guide to get started

Installation

Install via Cargo

You can install the binary via cargo:

% cargo install lindera-cli

Download from GitHub Releases

Alternatively, you can download a pre-built binary from the release page:

• https://github.com/lindera/lindera/releases

Obtaining Dictionaries

Lindera does not bundle dictionaries with the binary. You need to download a pre-built dictionary separately from the GitHub Releases page:

# Example: download and extract the IPADIC dictionary
% curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
% unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Then specify the dictionary path when using the CLI:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict /path/to/ipadic

Build from Source

Build without dictionaries (default)

Build a binary containing only the tokenizer and trainer without embedded dictionaries:

% cargo build --release

Build with all features

% cargo build --release --all-features

Build with Embedded Dictionaries (Advanced)

For advanced users who want to embed dictionaries directly into the binary, use the embed-* feature flags. This eliminates the need for external dictionary files at runtime but increases the binary size.

IPADIC (Japanese dictionary)

% cargo build --release --features=embed-ipadic

IPADIC NEologd (Japanese dictionary)

% cargo build --release --features=embed-ipadic-neologd

UniDic (Japanese dictionary)

% cargo build --release --features=embed-unidic

ko-dic (Korean dictionary)

% cargo build --release --features=embed-ko-dic

CC-CEDICT (Chinese dictionary)

% cargo build --release --features=embed-cc-cedict

Jieba (Chinese dictionary)

% cargo build --release --features=embed-jieba

[!TIP] After building with an embed-* feature flag, use the embedded:// scheme to load the embedded dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic

See Feature Flags for details.

Commands

The Lindera CLI provides five main commands:

• list - List the morphological analysis dictionaries embedded in the binary
• tokenize - Perform morphological analysis on text
• build - Build a dictionary from source CSV files
• train - Train a CRF model from annotated corpus data
• export - Export a trained model to dictionary format

list

List the morphological analysis dictionaries that were embedded in the binary at build time (via the embed-* feature flags).

List parameters

This command takes no arguments.

List usage

% lindera list

ipadic

The output contains one dictionary name per line, limited to whichever embed-* features were enabled when the binary was built (e.g. --features=embed-ipadic). If no embed-* feature was enabled, the command produces no output.

tokenize

Perform morphological analysis (tokenization) on Japanese, Chinese, or Korean text using various dictionaries.

Parameters

• --dict / -d: Dictionary path or URI (required)
  • File path: /path/to/dictionary
  • Embedded: embedded://ipadic, embedded://unidic, etc.
• --output / -o: Output format (default: mecab)
  • mecab: MeCab-compatible format with part-of-speech info
  • wakati: Space-separated tokens only
  • json: Detailed JSON format with all token information
• --user-dict / -u: User dictionary path (optional)
• --mode / -m: Tokenization mode (default: normal)
  • normal: Standard tokenization
  • decompose: Decompose compound words
• --char-filter / -c: Character filter configuration (JSON)
• --token-filter / -t: Token filter configuration (JSON)
• --keep-whitespace: Keep whitespace tokens in the output (by default, whitespace is dropped for MeCab compatibility)
• --mmap: Use memory-mapped file loading for the dictionary directory's word list. Ignored for embedded:// dictionaries and when the mmap feature is disabled. Rebuilding or truncating the dictionary directory while a process holds it mapped can cause a SIGBUS on the next lookup.
• --nbest / -N: Number of N-best results to return (default: 1). When set to 2 or more, N-best output is enabled.
• --nbest-unique: Deduplicate N-best results by removing paths that produce the same segmentation.
• --nbest-cost-threshold: Maximum cost difference from the best path. Only paths with cost within best_cost + threshold are returned.
• Input file: Optional file path (default: stdin)

Basic usage

# Tokenize text using a dictionary directory
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /path/to/dictionary

# Tokenize text using embedded dictionary
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

# Tokenize with different output format
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

# Tokenize text from file
lindera tokenize \
  --dict /path/to/dictionary \
  --output wakati \
  input.txt

Examples with external dictionaries

Tokenize with external IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external IPADIC Neologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-neologd-0.0.7-20200820

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-unidic-2.1.2

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

Tokenize with external ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict /tmp/lindera-ko-dic-2.1.1-20180720

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

Tokenize with external CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-cc-cedict-0.1.0-20200409

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

Tokenize with external Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-jieba-0.1.1

Examples with embedded dictionaries

Lindera can include dictionaries directly in the binary when built with specific feature flags. This allows tokenization without external dictionary files.

Tokenize with embedded IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC dictionary in the binary, you must build with the --features=embed-ipadic option.

Tokenize with embedded UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://unidic

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-unidic option.

Tokenize with embedded IPADIC NEologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic-neologd

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-ipadic-neologd option.

Tokenize with embedded ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict embedded://ko-dic

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

NOTE: To include ko-dic dictionary in the binary, you must build with the --features=embed-ko-dic option.

Tokenize with embedded CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://cc-cedict

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

NOTE: To include CC-CEDICT dictionary in the binary, you must build with the --features=embed-cc-cedict option.

Tokenize with embedded Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://jieba

NOTE: To include Jieba dictionary in the binary, you must build with the --features=embed-jieba option.

User dictionary examples

Lindera supports user dictionaries to add custom words alongside system dictionaries. User dictionaries can be in CSV or binary format.

Use user dictionary (CSV format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Use user dictionary (Binary format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920 \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.bin

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Tokenization modes

Lindera provides two tokenization modes: normal and decompose.

Normal mode (default)

Tokenizes faithfully based on words registered in the dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode normal

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Decompose mode

Tokenizes compound noun words additionally:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Output formats

Lindera provides three output formats: mecab, wakati and json.

MeCab format (default)

Outputs results in MeCab-compatible format with part-of-speech information:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output mecab

お待ち  名詞,サ変接続,*,*,*,*,お待ち,オマチ,オマチ
し  動詞,自立,*,*,サ変・スル,連用形,する,シ,シ
て  助詞,接続助詞,*,*,*,*,て,テ,テ
おり  動詞,非自立,*,*,五段・ラ行,連用形,おる,オリ,オリ
ます  助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。  記号,句点,*,*,*,*,。,。,。
EOS

Wakati format

Outputs only the token text separated by spaces:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

お待ち し て おり ます 。

JSON format

Outputs detailed token information in JSON format:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

[
  {
    "base_form": "お待ち",
    "byte_end": 9,
    "byte_start": 0,
    "conjugation_form": "*",
    "conjugation_type": "*",
    "part_of_speech": "名詞",
    "part_of_speech_subcategory_1": "サ変接続",
    "part_of_speech_subcategory_2": "*",
    "part_of_speech_subcategory_3": "*",
    "pronunciation": "オマチ",
    "reading": "オマチ",
    "surface": "お待ち",
    "word_id": 14698
  },
  ...
]

N-Best tokenization

Lindera supports N-Best tokenization, which returns the top N tokenization candidates ordered by cost (lower cost = better). This is based on the Forward-DP Backward-A* algorithm, compatible with MeCab's N-Best implementation.

Basic N-Best example

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  -N 3

NBEST 1 (cost=7546)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 2 (cost=7914)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 3 (cost=10060)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
も      助詞,係助詞,*,*,*,*,も,モ,モ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

N-Best with unique results

When the same segmentation appears in multiple paths (differing only in internal Viterbi states), use --nbest-unique to deduplicate:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 5 --nbest-unique -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です
NBEST 4 (cost=19320)
営業 部長 谷川 で す
NBEST 5 (cost=20814)
営業 部 長谷 川 です

N-Best with cost threshold

Use --nbest-cost-threshold to limit results to paths within a certain cost range of the best path:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 10 --nbest-unique --nbest-cost-threshold 5000 -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です

Only 3 results are returned because the remaining candidates exceed 15760 + 5000 = 20760.

Advanced tokenization with filters

Lindera provides an analytical framework that combines character filters, tokenizers, and token filters for advanced text processing. Filters are configured using JSON.

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般"]}'

すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
EOS

build

Build (compile) a morphological analysis dictionary from source CSV files for use with Lindera.

Build parameters

• --src / -s: Source directory containing dictionary CSV files (or single CSV file for user dictionary)
• --dest / -d: Destination directory for compiled dictionary output
• --metadata / -m: Metadata configuration file (metadata.json) that defines dictionary structure
• --user / -u: Build user dictionary instead of system dictionary (optional flag)
• --context-id-freq / -f: Context-ID access-frequency file used to reorder connection-cost IDs (optional; only meaningful when the dictionary's metadata.json sets connection_id_mapping: true)

Dictionary types

System dictionary

A full morphological analysis dictionary containing:

• Lexicon entries (word definitions)
• Connection cost matrix
• Unknown word handling rules
• Character type definitions

User dictionary

A supplementary dictionary for custom words that works alongside a system dictionary.

Examples

Build IPADIC (Japanese dictionary)

# Download and extract IPADIC source files
% curl -L -o /tmp/mecab-ipadic-2.7.0-20250920.tar.gz "https://Lindera.dev/mecab-ipadic-2.7.0-20250920.tar.gz"
% tar zxvf /tmp/mecab-ipadic-2.7.0-20250920.tar.gz -C /tmp

# Build the dictionary
% lindera build \
  --src /tmp/mecab-ipadic-2.7.0-20250920 \
  --dest /tmp/lindera-ipadic-2.7.0-20250920 \
  --metadata ./lindera-ipadic/metadata.json

Build IPADIC NEologd (Japanese dictionary)

% curl -L -o /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz "https://lindera.dev/mecab-ipadic-neologd-0.0.7-20200820.tar.gz"
% tar zxvf /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ipadic-neologd-0.0.7-20200820 \
  --dest /tmp/lindera-ipadic-neologd-0.0.7-20200820 \
  --metadata ./lindera-ipadic-neologd/metadata.json

Build UniDic (Japanese dictionary)

% curl -L -o /tmp/unidic-mecab-2.1.2.tar.gz "https://Lindera.dev/unidic-mecab-2.1.2.tar.gz"
% tar zxvf /tmp/unidic-mecab-2.1.2.tar.gz -C /tmp

% lindera build \
  --src /tmp/unidic-mecab-2.1.2 \
  --dest /tmp/lindera-unidic-2.1.2 \
  --metadata ./lindera-unidic/metadata.json

Build CC-CEDICT (Chinese dictionary)

% curl -L -o /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz "https://lindera.dev/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz"
% tar zxvf /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz -C /tmp

% lindera build \
  --src /tmp/CC-CEDICT-MeCab-0.1.0-20200409 \
  --dest /tmp/lindera-cc-cedict-0.1.0-20200409 \
  --metadata ./lindera-cc-cedict/metadata.json

Build Jieba (Chinese dictionary)

% curl -L -o /tmp/mecab-jieba-0.1.1.tar.gz "https://lindera.dev/mecab-jieba-0.1.1.tar.gz"
% tar zxvf /tmp/mecab-jieba-0.1.1.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-jieba-0.1.1/dict-src \
  --dest /tmp/lindera-jieba-0.1.1 \
  --metadata ./lindera-jieba/metadata.json

Build ko-dic (Korean dictionary)

% curl -L -o /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz "https://Lindera.dev/mecab-ko-dic-2.1.1-20180720.tar.gz"
% tar zxvf /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ko-dic-2.1.1-20180720 \
  --dest /tmp/lindera-ko-dic-2.1.1-20180720 \
  --metadata ./lindera-ko-dic/metadata.json

Build user dictionaries

Build IPADIC user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera IPADIC Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ipadic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ipadic/metadata.json \
  --user

Build UniDic user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera UniDic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/unidic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-unidic/metadata.json \
  --user

Build CC-CEDICT user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera CC-CEDICT Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/cc-cedict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-cc-cedict/metadata.json \
  --user

Build Jieba user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera Jieba Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/jieba_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-jieba/metadata.json \
  --user

Build ko-dic user dictionary (Korean)

For more details about user dictionary format please refer to the following URL:

• Lindera ko-dic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ko-dic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ko-dic/metadata.json \
  --user

train

Train a new morphological analysis model from annotated corpus data. To use this feature, you must build with the train feature flag enabled. (The train feature flag is enabled by default.)

Train parameters

• --seed / -s: Seed lexicon file (CSV format) to be weighted
• --corpus / -c: Training corpus (annotated text)
• --char-def / -C: Character definition file (char.def)
• --unk-def / -u: Unknown word definition file (unk.def) to be weighted
• --feature-def / -f: Feature definition file (feature.def)
• --rewrite-def / -r: Rewrite rule definition file (rewrite.def)
• --output / -o: Output model file
• --lambda / -l: Regularization coefficient (0.0-1.0) (default: 0.01)
• --regularization / -R: Regularization type: l1, l2, or elasticnet (default: l1)
• --elastic-net-l1-ratio: L1 ratio for Elastic Net regularization (0.0-1.0), only used with --regularization elasticnet (default: 0.5)
• --max-iterations / -i: Maximum number of iterations for training (default: 100)
• --max-threads / -t: Maximum number of threads (defaults to CPU core count)

Basic workflow

1. Prepare training files

Seed lexicon file (seed.csv):

The seed lexicon file contains initial dictionary entries used for training the CRF model. Each line represents a word entry with comma-separated fields:

• Surface
• Left context ID
• Right context ID
• Word cost
• Part-of-speech tags (multiple fields)
• Base form
• Reading (katakana)
• Pronunciation

Note: The exact field definitions differ between dictionary formats (IPADIC, UniDic, ko-dic, CC-CEDICT). Please refer to each dictionary's format specification for details.

外国,0,0,0,名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人,0,0,0,名詞,接尾,一般,*,*,*,人,ジン,ジン

Training corpus (corpus.txt):

The training corpus file contains annotated text data used to train the CRF model. Each line consists of:

• A surface form (word) followed by a tab character
• Comma-separated morphological features (part-of-speech tags, base form, reading, pronunciation)
• Sentences are separated by "EOS" (End Of Sentence) markers

外国	名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人	名詞,接尾,一般,*,*,*,人,ジン,ジン
参政	名詞,サ変接続,*,*,*,*,参政,サンセイ,サンセイ
権	名詞,接尾,一般,*,*,*,権,ケン,ケン
EOS

For detailed information about file formats and advanced features, see Training Pipeline.

2. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

3. Training results

The trained model will contain:

• Existing words: All seed dictionary records with newly learned weights
• New words: Words from the corpus not in the seed dictionary, added with appropriate weights

export

Export a trained model file to Lindera dictionary format files. This feature requires building with the train feature flag enabled.

Export parameters

• --model / -m: Path to the trained model file (.dat format)
• --output / -o: Directory to output the dictionary files
• --metadata: Optional metadata.json file to update with trained model information
• --cost-factor: Override cost factor for weight-to-cost conversion (default: value from trained model, typically 700)

Output files

The export command creates the following dictionary files in the output directory:

• lex.csv: Lexicon file with learned weights (MeCab-compatible cost via tocost())
• matrix.def: Dense connection cost matrix covering all (right_id, left_id) pairs
• unk.def: Unknown word definitions
• char.def: Character type definitions
• feature.def: Feature template definitions (copied from trained model)
• rewrite.def: Feature rewrite rules (copied from trained model)
• left-id.def: Left context ID to feature string mapping
• right-id.def: Right context ID to feature string mapping
• metadata.json: Updated metadata file (if --metadata option is provided)

Complete workflow example

1. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

2. Export to dictionary format

lindera export \
  --model /tmp/lindera/training/model.dat \
  --metadata ./resources/training/metadata.json \
  --output /tmp/lindera/training/dictionary

3. Build dictionary

lindera build \
  --src /tmp/lindera/training/dictionary \
  --dest /tmp/lindera/training/compiled_dictionary \
  --metadata /tmp/lindera/training/dictionary/metadata.json

4. Use trained dictionary

echo "これは外国人参政権です。" | lindera tokenize \
  -d /tmp/lindera/training/compiled_dictionary

Metadata update feature

When the --metadata option is provided, the export command will:

1. Read the base metadata.json file to preserve existing configuration
2. Update specific fields with values from the trained model:
   • default_left_context_id: Maximum left context ID from trained model
   • default_right_context_id: Maximum right context ID from trained model
   • default_word_cost: Calculated from feature weight median
   • model_info: Training statistics including feature count, label count, matrix size, iterations, regularization, version, and timestamp
3. Preserve existing settings such as dictionary name, character encoding, schema definitions, and other user-defined configuration

Tutorial

This tutorial walks you through the basic usage of the Lindera CLI, from installation to advanced text processing.

1. Install the CLI

Install Lindera CLI with the embedded IPADIC dictionary:

% cargo install lindera-cli --features=embed-ipadic

Verify the installation:

% lindera --help

2. Basic tokenization with embedded dictionary

Tokenize Japanese text using the embedded IPADIC dictionary:

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic

Expected output:

東京    名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
日本    名詞,固有名詞,地域,国,*,*,日本,ニホン,ニホン
の      助詞,連体化,*,*,*,*,の,ノ,ノ
首都    名詞,一般,*,*,*,*,首都,シュト,シュト
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。      記号,句点,*,*,*,*,。,。,。
EOS

3. Try different output formats

Wakati format (word segmentation only)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

Expected output:

東京 は 日本 の 首都 です 。

JSON format (detailed information)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

This produces a JSON array with detailed token information including byte offsets, part-of-speech tags, readings, and more.

4. Use decompose mode

Decompose mode splits compound nouns into their constituent parts:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

Expected output:

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Compare with normal mode, where "関西国際空港" remains as a single token.

5. Apply character and token filters

Use Unicode normalization and keep only common nouns:

% echo "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般","名詞,固有名詞,組織"]}'

Expected output:

Lindera 名詞,固有名詞,組織,*,*,*,*,*,*
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
エンジン        名詞,一般,*,*,*,*,エンジン,エンジン,エンジン
EOS

The Unicode normalization converts full-width characters to half-width, and the token filter keeps only tokens matching the specified part-of-speech tags.

You can also combine multiple filters:

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --token-filter 'japanese_stop_tags:{"tags":["助詞","助詞,係助詞","助詞,連体化"]}'

6. Use user dictionary

Create a CSV file with custom word entries (e.g., my_dict.csv):

東京スカイツリー,カスタム名詞,トウキョウスカイツリー

Tokenize with the user dictionary:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./my_dict.csv

Without the user dictionary, "東京スカイツリー" would be split into multiple tokens. With the user dictionary, it is recognized as a single token.

For pre-built user dictionary examples, see:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

Expected output:

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Lindera Python

Lindera Python provides Python bindings for the Lindera morphological analysis engine, built with PyO3. It brings Lindera's high-performance tokenization capabilities to the Python ecosystem with support for Python 3.10 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from PyPI

Pre-built wheels are available on PyPI:

pip install lindera-python

[!NOTE] The PyPI package does not include dictionaries. See Obtaining Dictionaries below.

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Building from Source

If you need to build from source (e.g., to enable specific feature flags), the following prerequisites are required:

• Python 3.10 or later (up to 3.14)
• Rust toolchain -- Install via rustup
• maturin -- Python package for building Rust-based Python extensions

Install maturin with pip:

pip install maturin

Development Build

Build and install lindera-python in development mode:

cd lindera-python
maturin develop

Or use the project Makefile:

make python-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

maturin develop --features train

Feature Flags

Multiple features can be combined:

maturin develop --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Python:

import lindera

print(lindera.version())

Quick Start

This guide shows how to tokenize text using lindera-python.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

from lindera import TokenizerBuilder

builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("/path/to/ipadic")
tokenizer = builder.build()

tokens = tokenizer.tokenize("関西国際空港限定トートバッグ")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("/path/to/ipadic")
    .build()
)

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    print(f"{token.surface}\t{token.get_detail(0)}")

Accessing Token Properties

Each token exposes the following properties:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
tokens = tokenizer.tokenize("東京タワー")

for token in tokens:
    print(f"Surface: {token.surface}")
    print(f"Byte range: {token.byte_start}..{token.byte_end}")
    print(f"Position: {token.position}")
    print(f"Word ID: {token.word_id}")
    print(f"Unknown: {token.is_unknown}")
    print(f"Details: {token.details}")
    print()

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)

for tokens, cost in results:
    surfaces = [t.surface for t in tokens]
    print(f"Cost {cost}: {' / '.join(surfaces)}")

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

TokenizerBuilder()

Creates a new builder with default configuration.

from lindera import TokenizerBuilder

builder = TokenizerBuilder()

TokenizerBuilder().from_file(file_path)

Loads configuration from a YAML file and returns a new builder. See lindera-python/resources/lindera.yml for a complete example covering segmenter, character_filters, and token_filters.

builder = TokenizerBuilder().from_file("lindera.yml")

Configuration Methods

All setter methods return self for method chaining.

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode("normal")

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary("embedded://ipadic")

# Use an external dictionary
builder.set_dictionary("/path/to/dictionary")

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary("/path/to/user_dictionary")

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(True)

append_character_filter(kind, args=None)

Appends a character filter to the preprocessing pipeline.

builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

append_token_filter(kind, args=None)

Appends a token filter to the postprocessing pipeline.

builder.append_token_filter("lowercase", {})

Build

build()

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build()

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Tokenizer(dictionary, mode="normal", user_dictionary=None)

Creates a tokenizer directly from a loaded dictionary.

from lindera import Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode="normal")

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns a list of Token objects.

tokens = tokenizer.tokenize("形態素解析")

Parameters:

Returns: list[Token]

tokenize_nbest(text, n, unique=False, cost_threshold=None)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)
for tokens, cost in results:
    print(cost, [t.surface for t in tokens])

Parameters:

Returns: list[tuple[list[Token], int]]

Mode

Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder.set_mode() and the Tokenizer constructor currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Mode(mode_str=None)

Creates a Mode. Accepts "normal" / "Normal" (the default when omitted) or "decompose" / "Decompose"; any other value raises ValueError.

from lindera import Mode

mode = Mode("normal")
mode = Mode("decompose")
mode = Mode()  # defaults to "normal"

Methods

mode = Mode("decompose")
str(mode)            # "decompose"
repr(mode)           # "Mode.Decompose"
mode.is_normal()      # False
mode.is_decompose()   # True

Penalty

Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Penalty(kanji_penalty_length_threshold=2, kanji_penalty_length_penalty=3000, other_penalty_length_threshold=7, other_penalty_length_penalty=1700)

All arguments are optional and default to the values shown above.

from lindera import Penalty

penalty = Penalty(
    kanji_penalty_length_threshold=2,
    kanji_penalty_length_penalty=3000,
    other_penalty_length_threshold=7,
    other_penalty_length_penalty=1700,
)

Penalty Properties

All four fields support both getting and setting:

penalty.kanji_penalty_length_threshold = 3
print(penalty.kanji_penalty_length_threshold)  # 3

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode() and the Tokenizer constructor only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Token Methods

get_detail(index)

Returns the detail string at the specified index, or None if the index is out of range.

token = tokenizer.tokenize("東京")[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: str or None

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Error Handling

Lindera Python functions raise standard Python exceptions rather than a custom exception type:

• IOError (an alias of OSError) -- for I/O-related failures, such as a missing or unreadable file
• ValueError -- for everything else, such as invalid configuration, parse errors, or tokenization failures

from lindera import load_dictionary

try:
    dictionary = load_dictionary("/path/that/does/not/exist")
except ValueError as e:
    print(f"Failed to load dictionary: {e}")

A LinderaError class is also registered as lindera.LinderaError, but no function in this crate currently raises it -- it can only be constructed and raised manually. Catch IOError/ValueError (or the general Exception) when handling errors from this library, not LinderaError.

Dictionary Management

Lindera Python provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

from lindera import load_dictionary

dictionary = load_dictionary("/path/to/ipadic")

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = load_dictionary("embedded://ipadic")

A loaded Dictionary also exposes its own metadata:

print(dictionary.metadata_name())      # e.g. "ipadic"
print(dictionary.metadata_encoding())  # e.g. "UTF-8"
metadata = dictionary.metadata()       # the full Metadata object

This is useful for loading a user dictionary that must share the same metadata (schema, encoding, etc.) as the system dictionary it augments, as in lindera-python/examples/tokenize_with_userdict.py:

from lindera import Tokenizer, load_dictionary, load_user_dictionary

dictionary = load_dictionary("embedded://ipadic")
metadata = dictionary.metadata()
user_dictionary = load_user_dictionary("/path/to/user_dictionary.csv", metadata)

tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dictionary)

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

from lindera import load_user_dictionary, Metadata

metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

Pass the user dictionary when building a tokenizer:

from lindera import Tokenizer, load_dictionary, load_user_dictionary, Metadata

dictionary = load_dictionary("/path/to/ipadic")
metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dict)

Or via the builder:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("/path/to/ipadic")
    .set_user_dictionary("/path/to/user_dictionary")
    .build()
)

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

from lindera import build_dictionary, Metadata

metadata = Metadata(name="custom", encoding="UTF-8")
build_dictionary("/path/to/input_dir", "/path/to/output_dir", metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

from lindera import build_user_dictionary, Metadata

metadata = Metadata()
build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir")

Note: the first argument ("ipadic" above) is currently unused and reserved for future use -- it does not select or configure the build in any way. The build behavior is controlled entirely by metadata (in particular metadata.user_dictionary_schema). Any string may be passed today.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

from lindera import Metadata

# Default metadata
metadata = Metadata()

# Custom metadata
metadata = Metadata(
    name="my_dictionary",
    encoding="UTF-8",
    default_word_cost=-10000,
)

Loading from JSON

metadata = Metadata.from_json_file("metadata.json")

Properties

All properties support both getting and setting:

metadata = Metadata()
metadata.name = "custom_dict"
metadata.encoding = "EUC-JP"
print(metadata.name)  # "custom_dict"

to_dict()

Returns a dictionary representation of the metadata:

metadata = Metadata(name="test")
print(metadata.to_dict())

Schema

Schema, FieldDefinition, and FieldType describe the field layout of a dictionary entry. A schema is used by Metadata.dictionary_schema and Metadata.user_dictionary_schema (see the table above).

FieldType

FieldType enumerates the category of a single field:

• FieldType.Surface -- surface form (word text)
• FieldType.LeftContextId -- left context ID
• FieldType.RightContextId -- right context ID
• FieldType.Cost -- word cost
• FieldType.Custom -- any other, dictionary-specific field

FieldDefinition

FieldDefinition describes a single field within a schema.

FieldDefinition(index, name, field_type, description=None)

from lindera import FieldDefinition, FieldType

field = FieldDefinition(0, "surface", FieldType.Surface, "Surface form")

Properties (read-only):

Creating a Schema

Schema holds an ordered list of field names and provides lookups between field name and index.

Schema(fields)

Creates a schema from a list of field names.

from lindera import Schema

schema = Schema([
    "surface",
    "left_context_id",
    "right_context_id",
    "cost",
    "major_pos",
    "reading",
])

Schema.create_default()

A static method that returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Schema.create_default()

Schema Methods and Properties

schema = Schema.create_default()

schema.field_count()               # 13
schema.get_field_index("cost")     # 3
schema.get_field_name(0)           # "surface"
schema.get_custom_fields()         # ["major_pos", "pos_detail_1", ..., "pronunciation"]
len(schema)                        # 13

field = schema.get_field_by_name("surface")
print(field.index, field.name, field.field_type)  # 0 surface FieldType.Surface

schema.validate_record([
    "東京", "1288", "1288", "100",
    "名詞", "固有名詞", "地域", "一般", "*", "*",
    "東京", "トウキョウ", "トーキョー",
])

Text Processing Pipeline

Lindera Python supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .build()
)

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("mapping", {
        "mapping": {
            "\u30fc": "-",
            "\uff5e": "~",
        }
    })
    .build()
)

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    .build()
)

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("lowercase", {})
    .build()
)

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_base_form", {})
    .build()
)

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞", "助動詞"],
    })
    .build()
)

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_keep_tags", {
        "tags": ["名詞"],
    })
    .build()
)

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("embedded://ipadic")
    # Preprocessing
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    # Postprocessing
    .append_token_filter("japanese_base_form", {})
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞", "助動詞", "記号"],
    })
    .append_token_filter("lowercase", {})
    .build()
)

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Python supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-python with the train feature enabled (enabled by default):

maturin develop --features train

Training a Model

Use lindera.train() to train a CRF model from a seed lexicon and annotated corpus:

import lindera

lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
    max_threads=4,
)

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using lindera.export():

import lindera

lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

import lindera

# Step 1: Train the CRF model
lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
)

# Step 2: Export to dictionary source files
lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

# Step 3: Build the dictionary from exported source files
metadata = lindera.Metadata.from_json_file("/tmp/dictionary_source/metadata.json")
lindera.build_dictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata)

# Step 4: Use the trained dictionary
tokenizer = (
    lindera.TokenizerBuilder()
    .set_dictionary("/tmp/dictionary")
    .set_mode("normal")
    .build()
)

tokens = tokenizer.tokenize("形態素解析のテスト")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Lindera Node.js

Lindera Node.js provides Node.js bindings for the Lindera morphological analysis engine, built with NAPI-RS. It brings Lindera's high-performance tokenization capabilities to the Node.js ecosystem with support for Node.js 18 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary
• TypeScript support: Full type definitions included out of the box

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from npm

Pre-built packages will be available on npm:

npm install lindera-nodejs

[!NOTE] The npm package does not include dictionaries. See Obtaining Dictionaries below. For browser/WASM usage, see lindera-wasm.

Building from Source

Prerequisites

• Node.js 18 or later (LTS versions recommended)
• Rust toolchain -- Install via rustup
• NAPI-RS CLI -- CLI tool for building native Node.js addons in Rust

Install the NAPI-RS CLI globally:

npm install -g @napi-rs/cli

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build lindera-nodejs in development mode:

cd lindera-nodejs
npm install
npm run build

Or use the project Makefile:

make nodejs-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

npm run build -- --features train

Feature Flags

Multiple features can be combined:

npm run build -- --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

const dictionary = loadDictionary("embedded://ipadic");

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Node.js:

const lindera = require("lindera-nodejs");

console.log(lindera.version());

[!NOTE] lindera-nodejs's package.json currently declares only a require condition in its exports map (no import condition), so import { version } from "lindera-nodejs" fails with ERR_PACKAGE_PATH_NOT_EXPORTED. From an ES module, load it with Node's createRequire instead:

import { createRequire } from "node:module";

const require = createRequire(import.meta.url);
const lindera = require("lindera-nodejs");

console.log(lindera.version());

Quick Start

This guide shows how to tokenize text using lindera-nodejs.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

const { TokenizerBuilder } = require("lindera-nodejs");

const builder = new TokenizerBuilder();
builder.setMode("normal");
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("すもももももももものうち");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.getDetail(0)}`);
}

Accessing Token Properties

Each token exposes the following properties:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("東京タワー");
for (const token of tokens) {
  console.log(`Surface: ${token.surface}`);
  console.log(`Byte range: ${token.byteStart}..${token.byteEnd}`);
  console.log(`Position: ${token.position}`);
  console.log(`Word ID: ${token.wordId}`);
  console.log(`Unknown: ${token.isUnknown}`);
  console.log(`Details: ${token.details}`);
  console.log();
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .build();

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  const surfaces = tokens.map((t) => t.surface);
  console.log(`Cost ${cost}: ${surfaces.join(" / ")}`);
}

TypeScript

Lindera Node.js includes TypeScript type definitions. All classes and functions are fully typed:

import type { Token } from "lindera-nodejs";
import { createRequire } from "node:module";

// `lindera-nodejs` only exposes a CommonJS `require` entry point (see Installation),
// so ESM projects load the runtime values through `createRequire` while still getting
// full type information via `import type`.
const require = createRequire(import.meta.url);
const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("/path/to/ipadic")
  .build();

const tokens: Token[] = tokenizer.tokenize("形態素解析");
for (const token of tokens) {
  console.log(`${token.surface}: ${token.details.join(",")}`);
}

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera-nodejs");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return this for method chaining.

setMode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.setMode("normal");

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.setKeepWhitespace(true);

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?)

Creates a tokenizer directly from a loaded dictionary.

const { Tokenizer, loadDictionary } = require("lindera-nodejs");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: Array<{ tokens: Token[], cost: number }>

Token

Token represents a single morphological token.

Properties

Token Methods

getDetail(index)

Returns the detail string at the specified index, or null if the index is out of range.

const token = tokenizer.tokenize("東京")[0];
const pos = token.getDetail(0);      // e.g., "名詞"
const subpos = token.getDetail(1);   // e.g., "固有名詞"
const reading = token.getDetail(7);  // e.g., "トウキョウ"

Parameters:

Returns: string | null

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

Mode and Penalty are exported from lindera-nodejs, but they are not currently wired into any public API: TokenizerBuilder.setMode() / Tokenizer's constructor accept only a plain mode string ("normal" or "decompose"), and decompose mode always uses the default penalty configuration internally. These types are documented here for completeness; they cannot yet be used to customize penalty behavior from JavaScript.

Mode

A string enum with two values:

• Mode.Normal -- standard dictionary-cost-based tokenization
• Mode.Decompose -- penalty-based decomposition of compound words

Penalty

An object shape describing the penalty parameters used by decompose mode:

Dictionary Management

Lindera Node.js provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use loadDictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

const { loadDictionary } = require("lindera-nodejs");

const dictionary = loadDictionary("/path/to/ipadic");

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

const dictionary = loadDictionary("embedded://ipadic");

Dictionary exposes a few read-only accessors for inspecting the loaded dictionary's metadata:

console.log(dictionary.metadataName());     // e.g. "ipadic"
console.log(dictionary.metadataEncoding()); // e.g. "UTF-8"

const metadata = dictionary.metadata(); // full Metadata object
console.log(metadata.defaultWordCost);

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

const { loadUserDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

Pass the user dictionary when building a tokenizer:

const { Tokenizer, loadDictionary, loadUserDictionary, Metadata } = require("lindera-nodejs");

const dictionary = loadDictionary("/path/to/ipadic");
const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

const tokenizer = new Tokenizer(dictionary, "normal", userDict);

Or via the builder:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .setUserDictionary("/path/to/user_dictionary")
  .build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

const { buildDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata({ name: "custom", encoding: "UTF-8" });
buildDictionary("/path/to/input_dir", "/path/to/output_dir", metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

const { buildUserDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata();
buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata);

The metadata parameter is optional. When omitted, default metadata values are used:

buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir");

[!NOTE] The first argument (kind) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

const { Metadata } = require("lindera-nodejs");

// Default metadata
const metadata = new Metadata();

// Custom metadata
const metadata = new Metadata({
  name: "my_dictionary",
  encoding: "UTF-8",
  defaultWordCost: -10000,
});

Loading from JSON

const metadata = Metadata.fromJsonFile("metadata.json");

Properties

[!NOTE] Schema information (dictionary/user-dictionary field layout) is not exposed on this binding's Metadata object -- there are no dictionarySchema / userDictionarySchema properties. Use the standalone Schema class instead.

All properties support both getting and setting:

const metadata = new Metadata();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

toObject()

Returns a plain object representation of the metadata:

const metadata = new Metadata({ name: "test" });
console.log(metadata.toObject());

Schema

The Schema class defines the field structure of dictionary entries.

Creating a Schema

const { Schema } = require("lindera-nodejs");

// Default IPADIC-compatible schema
const schema = Schema.createDefault();

// Custom schema
const custom = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Methods

const schema = Schema.createDefault();

console.log(schema.fieldCount());           // 13 (IPADIC format)
console.log(schema.getFieldIndex("pos1"));  // e.g., 4
console.log(schema.getAllFields());          // ["surface", "left_id", ...]
console.log(schema.getCustomFields());      // Fields from index 4 onward

FieldDefinition

FieldType

Text Processing Pipeline

Lindera Node.js supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. lindera-analysis ships 4 character filters and 18 token filters in total. See Filters for the full, authoritative catalogue of every character and token filter, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("unicode_normalize", { kind: "nfkc" })
  .build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("mapping", {
    mapping: {
      "\u30fc": "-",
      "\uff5e": "~",
    },
  })
  .build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("japanese_iteration_mark", {
    normalize_kanji: true,
    normalize_kana: true,
  })
  .build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("lowercase", {})
  .build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_base_form", {})
  .build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_stop_tags", {
    tags: ["助詞", "助動詞"],
  })
  .build();

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_keep_tags", {
    tags: ["名詞"],
  })
  .build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("embedded://ipadic")
  // Preprocessing
  .appendCharacterFilter("unicode_normalize", { kind: "nfkc" })
  .appendCharacterFilter("japanese_iteration_mark", {
    normalize_kanji: true,
    normalize_kana: true,
  })
  // Postprocessing
  .appendTokenFilter("japanese_base_form", {})
  .appendTokenFilter("japanese_stop_tags", {
    tags: ["助詞", "助動詞", "記号"],
  })
  .appendTokenFilter("lowercase", {})
  .build();

const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Node.js supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-nodejs with the train feature enabled (enabled by default):

npm run build -- --features train

Training a Model

[!NOTE] The file paths below (resources/training/*.csv, *.def) are illustrative placeholders — this repository does not ship those exact files. For a complete, runnable example that generates a seed lexicon, corpus, and definition files on the fly and then trains/exports/builds a real dictionary, see lindera-nodejs/examples/train_and_export.js.

Use train() to train a CRF model from a seed lexicon and annotated corpus:

const { train } = require("lindera-nodejs");

train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
  maxThreads: 4,
});

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using exportModel():

const { exportModel } = require("lindera-nodejs");

exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

const {
  train,
  exportModel,
  buildDictionary,
  Metadata,
  TokenizerBuilder,
} = require("lindera-nodejs");

// Step 1: Train the CRF model
train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
});

// Step 2: Export to dictionary source files
exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

// Step 3: Build the dictionary from exported source files
const metadata = Metadata.fromJsonFile("/tmp/dictionary_source/metadata.json");
buildDictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata);

// Step 4: Use the trained dictionary
const tokenizer = new TokenizerBuilder()
  .setDictionary("/tmp/dictionary")
  .setMode("normal")
  .build();

const tokens = tokenizer.tokenize("形態素解析のテスト");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Lindera Ruby

Lindera Ruby provides Ruby bindings for the Lindera morphological analysis engine, built with Magnus and rb-sys. It brings Lindera's high-performance tokenization capabilities to the Ruby ecosystem with support for Ruby 3.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

[!NOTE] lindera-ruby is not yet published to RubyGems. You need to build from source.

Prerequisites

• Ruby 3.1 or later
• Rust toolchain -- Install via rustup
• Bundler -- Ruby dependency manager (gem install bundler)

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build and install lindera-ruby in development mode:

cd lindera-ruby
bundle install
bundle exec rake compile

Or use the project Makefile:

make build-lindera-ruby

Run make test-lindera-ruby to run both the Rust unit tests and the Ruby minitest suite.

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

LINDERA_FEATURES="train" bundle exec rake compile

Feature Flags

Features are specified through the LINDERA_FEATURES environment variable as a comma-separated list.

Multiple features can be combined:

LINDERA_FEATURES="train,embed-ipadic,embed-ko-dic" bundle exec rake compile

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = Lindera.load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Ruby:

require 'lindera'

puts Lindera.version

Quick Start

This guide shows how to tokenize text using lindera-ruby.

Basic Tokenization

The recommended way to create a tokenizer is through Lindera::TokenizerBuilder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('関西国際空港限定トートバッグ')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Sequential Configuration

TokenizerBuilder is configured through sequential method calls:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('すもももももももものうち')
tokens.each do |token|
  puts "#{token.surface}\t#{token.get_detail(0)}"
end

Accessing Token Properties

Each token exposes the following properties:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('東京タワー')

tokens.each do |token|
  puts "Surface: #{token.surface}"
  puts "Byte range: #{token.byte_start}..#{token.byte_end}"
  puts "Position: #{token.position}"
  puts "Word ID: #{token.word_id}"
  puts "Unknown: #{token.unknown?}"
  puts "Details: #{token.details}"
  puts
end

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)

results.each do |tokens, cost|
  surfaces = tokens.map(&:surface)
  puts "Cost #{cost}: #{surfaces.join(' / ')}"
end

Tokenizer API

TokenizerBuilder

Lindera::TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

Lindera::TokenizerBuilder.new

Creates a new builder with default configuration.

require 'lindera'

builder = Lindera::TokenizerBuilder.new

Lindera::TokenizerBuilder.from_file(file_path)

Loads configuration from a JSON file and returns a new builder. This is a class method, not chained off an existing instance.

builder = Lindera::TokenizerBuilder.from_file('config.json')

Configuration Methods

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode('normal')

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary('embedded://ipadic')

# Use an external dictionary
builder.set_dictionary('/path/to/dictionary')

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary('/path/to/user_dictionary')

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(true)

append_character_filter(kind, args)

Appends a character filter to the preprocessing pipeline. The args parameter is a hash with string keys.

builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })

append_token_filter(kind, args)

Appends a token filter to the postprocessing pipeline. The args parameter is a hash with string keys, or nil if the filter requires no arguments.

builder.append_token_filter('lowercase', nil)

Build

build

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build

Tokenizer

Lindera::Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Lindera::Tokenizer.new(dictionary, mode, user_dictionary)

Creates a tokenizer directly from a loaded dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('embedded://ipadic')
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', nil)

With a user dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

tokens = tokenizer.tokenize('形態素解析')

Parameters:

Returns: Array<Token>

tokenize_nbest(text, n, unique, cost_threshold)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)
results.each do |tokens, cost|
  puts "#{cost}: #{tokens.map(&:surface).inspect}"
end

Parameters:

Returns: Array<Array(Array<Token>, Integer)>

Mode

Lindera::Mode represents a tokenization mode. It is provided as a standalone helper for inspecting or comparing modes; TokenizerBuilder#set_mode and Tokenizer.new currently accept only a plain mode string ("normal" or "decompose"), not a Mode instance (see the limitation noted under Penalty below).

Creating a Mode

Lindera::Mode.new(mode_str)

Creates a Mode. The argument is required, but may be nil. Accepts "normal" / "Normal" (used when mode_str is nil) or "decompose" / "Decompose"; any other value raises ArgumentError.

require 'lindera'

mode = Lindera::Mode.new('normal')
mode = Lindera::Mode.new('decompose')
mode = Lindera::Mode.new(nil)  # defaults to "normal"

Mode Methods

mode = Lindera::Mode.new('decompose')
mode.to_s        # "decompose"
mode.normal?      # false
mode.decompose?   # true

Penalty

Lindera::Penalty configures the length-based penalty thresholds used by "decompose" mode segmentation.

Creating a Penalty

Lindera::Penalty.new(kanji_penalty_length_threshold, kanji_penalty_length_penalty, other_penalty_length_threshold, other_penalty_length_penalty)

All four positional arguments are required, but each may be nil to fall back to its default (shown below).

require 'lindera'

penalty = Lindera::Penalty.new(2, 3000, 7, 1700)
penalty = Lindera::Penalty.new(nil, nil, nil, nil)  # uses all defaults

Penalty Properties

All properties are read-only (there are no setter methods):

penalty = Lindera::Penalty.new(nil, nil, nil, nil)
penalty.kanji_penalty_length_threshold  # 2

Current limitation: there is currently no way to pass a Penalty into a Tokenizer or TokenizerBuilder. set_mode and Tokenizer.new only accept a plain mode string, and internally "decompose" mode always uses Penalty's default values -- constructing a custom Penalty instance has no effect on tokenization yet.

Token

Token represents a single morphological token.

Properties

Additionally, the predicate method unknown? returns true if the word is not in the dictionary:

token.unknown?  # => false

Token Methods

get_detail(index)

Returns the detail string at the specified index, or nil if the index is out of range.

token = tokenizer.tokenize('東京')[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: String or nil

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Schema

Lindera::Schema holds an ordered list of field names and provides lookups between field name and index. It is used by Metadata#dictionary_schema and Metadata#user_dictionary_schema (see Dictionary Management).

Creating a Schema

Lindera::Schema.new(fields)

Creates a schema from an array of field names.

require 'lindera'

schema = Lindera::Schema.new(%w[
  surface
  left_context_id
  right_context_id
  cost
  major_pos
  reading
])

Lindera::Schema.create_default

Returns the built-in default schema: 13 fields matching the IPADIC-style layout (surface, left_context_id, right_context_id, cost, major_pos, pos_detail_1, pos_detail_2, pos_detail_3, conjugation_type, conjugation_form, base_form, reading, pronunciation).

schema = Lindera::Schema.create_default

Schema Methods

schema = Lindera::Schema.create_default

schema.field_count                  # 13
schema.get_field_index('cost')      # 3
schema.get_field_name(0)            # "surface"
schema.get_custom_fields            # ["major_pos", "pos_detail_1", ..., "pronunciation"]

field = schema.get_field_by_name('surface')
puts "#{field.index} #{field.name} #{field.field_type}"  # 0 surface surface

schema.validate_record([
  '東京', '1288', '1288', '100',
  '名詞', '固有名詞', '地域', '一般', '*', '*',
  '東京', 'トウキョウ', 'トーキョー'
])

FieldDefinition

Lindera::FieldDefinition describes a single field within a Schema. Instances are only obtained from Schema#get_field_by_name -- there is no public constructor (Lindera::FieldDefinition.new raises TypeError).

FieldDefinition Properties

schema = Lindera::Schema.create_default
field = schema.get_field_by_name('surface')

field.index         # 0
field.name          # "surface"
field.field_type    # #<Lindera::FieldType: surface>
field.description    # nil (the default schema does not set descriptions)

FieldType

Lindera::FieldType enumerates the category of a single field. Like FieldDefinition, instances are only obtained from a Schema (via FieldDefinition#field_type) -- there is no public constructor.

to_s (and inspect) return one of:

• "surface" -- surface form (word text)
• "left_context_id" -- left context ID
• "right_context_id" -- right context ID
• "cost" -- word cost
• "custom" -- any other, dictionary-specific field

field = Lindera::Schema.create_default.get_field_by_name('surface')
field.field_type.to_s  # "surface"

Dictionary Management

Lindera Ruby provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera.load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

Pass the user dictionary when building a tokenizer:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Or via the builder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
builder.set_user_dictionary('/path/to/user_dictionary')
tokenizer = builder.build

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_dictionary('/path/to/input_dir', '/path/to/output_dir', metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', nil)

[!NOTE] The first argument (kind, 'ipadic' above) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today.

Metadata

The Lindera::Metadata class configures dictionary parameters.

Creating Metadata

require 'lindera'

# Create default metadata with standard settings
metadata = Lindera::Metadata.create_default

Lindera::Metadata.new takes all nine properties as required positional arguments (each may be nil to fall back to its default) -- use it only when you need to override specific values:

metadata = Lindera::Metadata.new(
  'my_dict', # name
  'UTF-8',   # encoding
  -10_000,   # default_word_cost
  1288,      # default_left_context_id
  1288,      # default_right_context_id
  '*',       # default_field_value
  false,     # flexible_csv
  false,     # skip_invalid_cost_or_id
  false      # normalize_details
)

Loading from JSON

metadata = Lindera::Metadata.from_json_file('metadata.json')

Getting Metadata from a Dictionary

A loaded dictionary's metadata can be retrieved directly:

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata

Properties

Text Processing Pipeline

Lindera Ruby supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. lindera-analysis ships 4 character filters and 18 token filters in total. See Filters for the full, authoritative catalogue of every character and token filter, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
tokenizer = builder.build

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('mapping', {
  'mapping' => {
    "\u30fc" => '-',
    "\uff5e" => '~'
  }
})
tokenizer = builder.build

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => true,
  'normalize_kana' => true
})
tokenizer = builder.build

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('lowercase', nil)
tokenizer = builder.build

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_base_form', nil)
tokenizer = builder.build

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞', '助動詞']
})
tokenizer = builder.build

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_keep_tags', {
  'tags' => ['名詞']
})
tokenizer = builder.build

japanese_katakana_stem

Removes trailing prolonged sound marks from katakana tokens that exceed a minimum length.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_katakana_stem', { 'min' => 3 })
tokenizer = builder.build

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('embedded://ipadic')

# Preprocessing
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => true,
  'normalize_kana' => true
})

# Postprocessing
builder.append_token_filter('japanese_base_form', nil)
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞', '助動詞', '記号']
})
builder.append_token_filter('lowercase', nil)

tokenizer = builder.build

tokens = tokenizer.tokenize('Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Ruby supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-ruby with the train feature enabled:

LINDERA_FEATURES="embed-ipadic,train" bundle exec rake compile

Training a Model

Use Lindera.train to train a CRF model from a seed lexicon and annotated corpus:

require 'lindera'

Lindera.train(
  'resources/training/seed.csv',
  'resources/training/corpus.txt',
  'resources/training/char.def',
  'resources/training/unk.def',
  'resources/training/feature.def',
  'resources/training/rewrite.def',
  '/tmp/model.dat',
  0.01,  # lambda (L1 regularization)
  100,   # max_iter
  nil    # max_threads (nil = auto-detect CPU cores)
)

Training Parameters

Parameters are passed as positional arguments in the following order:

Exporting a Trained Model

After training, export the model to dictionary source files using Lindera.export:

require 'lindera'

Lindera.export(
  '/tmp/model.dat',
  '/tmp/dictionary_source',
  'resources/training/metadata.json'
)

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

require 'lindera'

# Step 1: Train the CRF model
Lindera.train(
  'resources/training/seed.csv',
  'resources/training/corpus.txt',
  'resources/training/char.def',
  'resources/training/unk.def',
  'resources/training/feature.def',
  'resources/training/rewrite.def',
  '/tmp/model.dat',
  0.01,  # lambda
  100,   # max_iter
  nil    # max_threads
)

# Step 2: Export to dictionary source files
Lindera.export(
  '/tmp/model.dat',
  '/tmp/dictionary_source',
  'resources/training/metadata.json'
)

# Step 3: Build the dictionary from exported source files
metadata = Lindera::Metadata.from_json_file('/tmp/dictionary_source/metadata.json')
Lindera.build_dictionary('/tmp/dictionary_source', '/tmp/dictionary', metadata)

# Step 4: Use the trained dictionary
builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/tmp/dictionary')
builder.set_mode('normal')
tokenizer = builder.build

tokens = tokenizer.tokenize('形態素解析のテスト')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

Lindera PHP

Lindera PHP provides PHP bindings for the Lindera morphological analysis engine, built with ext-php-rs. It brings Lindera's high-performance tokenization capabilities to the PHP ecosystem with support for PHP 8.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

[!NOTE] lindera-php is not yet published to Packagist. You need to build from source.

Prerequisites

• PHP 8.1 or later
• Rust toolchain -- Install via rustup
• Composer -- PHP dependency manager (optional, for running tests)

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build the lindera-php extension from the project root:

cargo build -p lindera-php

Or use the project Makefile:

make build-lindera-php

Build with Training Support

The train feature enables CRF-based dictionary training functionality:

cargo build -p lindera-php --features train

Feature Flags

Multiple features can be combined:

cargo build -p lindera-php --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

$dictionary = Lindera\Dictionary::load('embedded://ipadic');

See Feature Flags for details.

Loading the Extension

Load the compiled shared library when running PHP:

php -d extension=target/debug/liblindera_php.so script.php

For release builds:

cargo build -p lindera-php --release
php -d extension=target/release/liblindera_php.so script.php

Alternatively, add the extension to your php.ini:

extension=/absolute/path/to/liblindera_php.so

Verifying the Installation

After building, verify that lindera is available in PHP:

php -d extension=target/debug/liblindera_php.so -r "echo Lindera\Dictionary::version() . PHP_EOL;"

Quick Start

This guide shows how to tokenize text using lindera-php.

Basic Tokenization

Load a dictionary, create a tokenizer, and tokenize text:

<?php

// Load the dictionary
$dictionary = Lindera\Dictionary::load('/path/to/ipadic');

// Create a tokenizer
$tokenizer = new Lindera\Tokenizer($dictionary, 'normal');

// Tokenize the text
$tokens = $tokenizer->tokenize('関西国際空港限定トートバッグ');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Using TokenizerBuilder

TokenizerBuilder gives you more flexible configuration options:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setMode('normal');
$builder->setDictionary('/path/to/ipadic');
$tokenizer = $builder->build();

$tokens = $tokenizer->tokenize('すもももももももものうち');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . $token->getDetail(0) . "\n";
}

Accessing Token Properties

Each token exposes the following properties:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('/path/to/ipadic');
$tokenizer = $builder->build();
$tokens = $tokenizer->tokenize('東京タワー');

foreach ($tokens as $token) {
    echo "Surface: {$token->surface}\n";
    echo "Byte range: {$token->byte_start}..{$token->byte_end}\n";
    echo "Position: {$token->position}\n";
    echo "Word ID: {$token->word_id}\n";
    echo "Unknown: " . ($token->is_unknown ? 'true' : 'false') . "\n";
    echo "Details: " . implode(',', $token->details) . "\n";
    echo "\n";
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('/path/to/ipadic');
$tokenizer = $builder->build();
$results = $tokenizer->tokenizeNbest('すもももももももものうち', 3);

foreach ($results as $result) {
    $surfaces = array_map(fn($t) => $t->surface, $result->tokens);
    echo "Cost {$result->cost}: " . implode(' / ', $surfaces) . "\n";
}

Tokenizer API

TokenizerBuilder

Lindera\TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new Lindera\TokenizerBuilder()

Creates a new builder with default configuration.

<?php

$builder = new Lindera\TokenizerBuilder();

$builder->fromFile($filePath)

Loads configuration from a JSON file.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->fromFile('config.json');

Configuration Methods

setMode($mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

<?php

$builder->setMode('normal');

setDictionary($path)

Sets the system dictionary path or URI.

<?php

// Use an embedded dictionary
$builder->setDictionary('embedded://ipadic');

// Use an external dictionary
$builder->setDictionary('/path/to/dictionary');

setUserDictionary($uri)

Sets the user dictionary URI.

<?php

$builder->setUserDictionary('/path/to/user_dictionary.csv');

setKeepWhitespace($keep)

Controls whether whitespace tokens appear in the output.

<?php

$builder->setKeepWhitespace(true);

appendCharacterFilter($kind, $args)

Appends a character filter to the preprocessing pipeline.

<?php

$builder->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc']);

appendTokenFilter($kind, $args)

Appends a token filter to the postprocessing pipeline.

<?php

$builder->appendTokenFilter('lowercase');

Build

build()

Builds and returns a Tokenizer with the configured settings.

<?php

$tokenizer = $builder->build();

Tokenizer

Lindera\Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Lindera\Tokenizer($dictionary, $mode, $userDictionary)

Creates a tokenizer directly from a loaded dictionary.

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');
$tokenizer = new Lindera\Tokenizer($dictionary, 'normal');

With a user dictionary:

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary.csv', $metadata);
$tokenizer = new Lindera\Tokenizer($dictionary, 'normal', $userDict);

Tokenizer Methods

tokenize($text)

Tokenizes the input text and returns an array of Token objects.

<?php

$tokens = $tokenizer->tokenize('形態素解析');

Parameters:

Returns: array<Token>

tokenizeNbest($text, $n, $unique, $costThreshold)

Returns the N-best tokenization results as an array of NbestResult objects.

<?php

$results = $tokenizer->tokenizeNbest('すもももももももものうち', 3);
foreach ($results as $result) {
    echo "Cost: {$result->cost}\n";
    foreach ($result->tokens as $token) {
        echo "  {$token->surface}\n";
    }
}

Parameters:

Returns: array<NbestResult>

NbestResult

Lindera\NbestResult represents a single N-best tokenization result.

NbestResult Properties

Token

Lindera\Token represents a single morphological token.

Token Properties

Token Methods

getDetail($index)

Returns the detail string at the specified index, or null if the index is out of range.

<?php

$token = $tokenizer->tokenize('東京')[0];
$pos = $token->getDetail(0);        // e.g., "名詞"
$subpos = $token->getDetail(1);     // e.g., "固有名詞"
$reading = $token->getDetail(7);    // e.g., "トウキョウ"

Parameters:

Returns: string|null

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode

Lindera\Mode represents the tokenization mode.

Creating a Mode

<?php

$mode = new Lindera\Mode('normal');
$mode = new Lindera\Mode('decompose');
$mode = new Lindera\Mode();  // default: 'normal'

Mode Properties

Mode Methods

Penalty

Lindera\Penalty configures how aggressively decompose mode splits compound words, based on character type and length thresholds.

Note: Penalty is not currently wired into TokenizerBuilder or the Tokenizer constructor -- there is no setter that accepts it, so constructing one has no effect on tokenization yet. decompose mode always uses the default penalty values shown below.

<?php

// All parameters are optional and default to the values used by decompose mode
$penalty = new Lindera\Penalty(
    kanji_penalty_length_threshold: 2,
    kanji_penalty_length_penalty: 3000,
    other_penalty_length_threshold: 7,
    other_penalty_length_penalty: 1700,
);

Penalty Properties

Dictionary Management

Lindera PHP provides static methods on the Lindera\Dictionary class for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera\Dictionary::load($uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary.csv', $metadata);

Pass the user dictionary when creating a tokenizer directly:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary.csv', $metadata);

$tokenizer = new Lindera\Tokenizer($dictionary, 'normal', $userDict);

Or via the builder:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('/path/to/ipadic');
$builder->setUserDictionary('/path/to/user_dictionary.csv');
$tokenizer = $builder->build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

<?php

$metadata = Lindera\Metadata::fromJsonFile('/path/to/metadata.json');
Lindera\Dictionary::build('/path/to/input_dir', '/path/to/output_dir', $metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

Here is an example that downloads and builds the IPADIC dictionary:

<?php

$url = 'https://lindera.dev/mecab-ipadic-2.7.0-20070801.tar.gz';
$filename = '/tmp/mecab-ipadic-2.7.0-20070801.tar.gz';

// Download and extract dictionary source
file_put_contents($filename, file_get_contents($url));
$phar = new PharData($filename);
$phar->extractTo('/tmp/', null, true);

// Load metadata and build
$metadata = Lindera\Metadata::fromJsonFile('resources/ipadic_metadata.json');
Lindera\Dictionary::build(
    '/tmp/mecab-ipadic-2.7.0-20070801',
    '/tmp/lindera-ipadic',
    $metadata
);

User Dictionary

Build a user dictionary from a CSV file:

<?php

$metadata = new Lindera\Metadata();
Lindera\Dictionary::buildUser('ipadic', 'user_words.csv', '/path/to/output_dir', $metadata);

Metadata

The Lindera\Metadata class configures dictionary parameters.

Creating Metadata

<?php

// Default metadata
$metadata = new Lindera\Metadata();

// Custom metadata
$metadata = new Lindera\Metadata(
    name: 'my_dictionary',
    encoding: 'UTF-8',
    default_word_cost: -10000,
);

// Create with all defaults explicitly
$metadata = Lindera\Metadata::createDefault();

Loading from JSON

<?php

$metadata = Lindera\Metadata::fromJsonFile('metadata.json');

Properties

All properties are read-only via getter methods:

<?php

$metadata = new Lindera\Metadata(name: 'custom_dict', encoding: 'EUC-JP');
echo $metadata->name;      // "custom_dict"
echo $metadata->encoding;  // "EUC-JP"

toArray()

Returns an associative array representation of the metadata:

<?php

$metadata = new Lindera\Metadata(name: 'test');
print_r($metadata->toArray());

Dictionary Info

The Lindera\Dictionary object provides metadata accessors:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
echo $dictionary->metadataName();      // Dictionary name
echo $dictionary->metadataEncoding();  // Dictionary encoding
$metadata = $dictionary->metadata();   // Full Metadata object

Version

Retrieve the Lindera library version:

<?php

echo Lindera\Dictionary::version();

Schema

The Lindera\Schema class defines the field layout of a dictionary.

Creating a Schema

<?php

// Default schema (IPADIC-compatible)
$schema = Lindera\Schema::createDefault();

// Custom schema
$schema = new Lindera\Schema(['surface', 'pos']);

Schema Methods

Schema Properties

Text Processing Pipeline

Lindera PHP supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

[!NOTE] This page shows a few commonly used filters as examples -- it is not the complete list. lindera-analysis ships 4 character filters and 18 token filters in total. See Filters for the full, authoritative catalogue of every character and token filter, including parameters and examples.

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc']);
$tokenizer = $builder->build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendCharacterFilter('mapping', [
    'mapping' => [
        'リンデラ' => 'lindera',
    ],
]);
$tokenizer = $builder->build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendCharacterFilter('japanese_iteration_mark', [
    'normalize_kanji' => 'true',
    'normalize_kana' => 'true',
]);
$tokenizer = $builder->build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendTokenFilter('lowercase');
$tokenizer = $builder->build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendTokenFilter('japanese_base_form', []);
$tokenizer = $builder->build();

japanese_katakana_stem

Removes the trailing long sound mark from katakana words to normalize spelling variants, stemming only words at least min characters long.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendTokenFilter('japanese_katakana_stem', ['min' => 3]);
$tokenizer = $builder->build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendTokenFilter('japanese_stop_tags', [
    'tags' => ['助詞', '助動詞'],
]);
$tokenizer = $builder->build();

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('embedded://ipadic');
$builder->appendTokenFilter('japanese_keep_tags', [
    'tags' => ['名詞'],
]);
$tokenizer = $builder->build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

<?php

$builder = new Lindera\TokenizerBuilder();

// Set mode and dictionary
$builder->setMode('normal');
$builder->setDictionary('embedded://ipadic');

// Preprocessing
$builder->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc']);
$builder->appendCharacterFilter(
    'japanese_iteration_mark',
    ['normalize_kanji' => 'true', 'normalize_kana' => 'true']
);
$builder->appendCharacterFilter('mapping', ['mapping' => ['リンデラ' => 'lindera']]);

// Postprocessing
$builder->appendTokenFilter('japanese_katakana_stem', ['min' => 3]);
$builder->appendTokenFilter('japanese_stop_tags', [
    'tags' => [
        '接続詞',
        '助詞',
        '助詞,格助詞',
        '助詞,格助詞,一般',
        '助詞,係助詞',
        '助詞,副助詞',
        '助詞,終助詞',
        '助詞,連体化',
        '助動詞',
        '記号',
        '記号,一般',
        '記号,読点',
        '記号,句点',
        '記号,空白',
    ],
]);
$builder->appendTokenFilter('lowercase');

// Build the tokenizer
$tokenizer = $builder->build();

// Tokenize
$text = 'Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。';
$tokens = $tokenizer->tokenize($text);

foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. mapping replaces the specified strings
4. japanese_katakana_stem stems katakana words
5. japanese_stop_tags removes particles, auxiliary verbs, and symbols
6. lowercase normalizes alphabetic characters to lowercase

Training

Lindera PHP supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-php with the train feature enabled (train is already enabled by default in lindera-php):

cargo build -p lindera-php --features train,embed-ipadic

Training a Model

Use Lindera\Trainer::train() to train a CRF model from a seed lexicon and annotated corpus:

<?php

Lindera\Trainer::train(
    seed: 'resources/training/seed.csv',
    corpus: 'resources/training/corpus.txt',
    char_def: 'resources/training/char.def',
    unk_def: 'resources/training/unk.def',
    feature_def: 'resources/training/feature.def',
    rewrite_def: 'resources/training/rewrite.def',
    output: '/tmp/model.dat',
    lambda: 0.01,
    max_iter: 100,
    max_threads: null,
);

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using Lindera\Trainer::export():

<?php

Lindera\Trainer::export(
    model: '/tmp/model.dat',
    output: '/tmp/dictionary_source',
    metadata: 'resources/training/metadata.json',
);

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when $metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

<?php

// Step 1: Train the CRF model
Lindera\Trainer::train(
    seed: 'resources/training/seed.csv',
    corpus: 'resources/training/corpus.txt',
    char_def: 'resources/training/char.def',
    unk_def: 'resources/training/unk.def',
    feature_def: 'resources/training/feature.def',
    rewrite_def: 'resources/training/rewrite.def',
    output: '/tmp/model.dat',
    lambda: 0.01,
    max_iter: 100,
);

// Step 2: Export to dictionary source files
Lindera\Trainer::export(
    model: '/tmp/model.dat',
    output: '/tmp/dictionary_source',
    metadata: 'resources/training/metadata.json',
);

// Step 3: Build the dictionary from exported source files
$metadata = Lindera\Metadata::fromJsonFile('/tmp/dictionary_source/metadata.json');
Lindera\Dictionary::build('/tmp/dictionary_source', '/tmp/dictionary', $metadata);

// Step 4: Use the trained dictionary
$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('/tmp/dictionary');
$builder->setMode('normal');
$tokenizer = $builder->build();

$tokens = $tokenizer->tokenize('形態素解析のテスト');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

Lindera WASM

Lindera WASM provides WebAssembly bindings for Lindera's morphological analysis engine, built with wasm-bindgen. It enables Japanese, Korean, and Chinese text tokenization directly in web browsers, Node.js, and bundler environments.

Distribution Formats

Lindera WASM supports multiple distribution formats via wasm-pack:

Dictionary Packages

Each package embeds a specific dictionary for offline use:

Sections

• Installation -- Building and installing lindera-wasm packages
• Quick Start -- Minimal working example
• Tokenizer API -- Full API reference for JavaScript/TypeScript
• Dictionary Management -- Loading and building dictionaries
• Browser Usage -- Integration with web applications
• OPFS Dictionary Storage -- Persistent dictionary caching with OPFS

Installation

Prerequisites

• Rust (stable toolchain)
• wasm-pack (v0.10+)

Obtaining Dictionaries

Lindera WASM does not bundle dictionaries by default. The recommended approach for browser environments is to download dictionaries at runtime using the OPFS (Origin Private File System) API.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. In browser environments, use the OPFS helpers to download and cache dictionaries:

import { downloadDictionary, hasDictionary } from 'lindera-wasm-web/opfs';

if (!await hasDictionary("ipadic")) {
    await downloadDictionary(
        "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
        "ipadic",
    );
}

See OPFS Dictionary Storage for the full workflow.

Building with wasm-pack

Build the WASM package for your target environment:

Web (ES Modules for browsers)

wasm-pack build --target web

Bundler (Webpack, Vite, Rollup)

wasm-pack build --target bundler

The output is written to the pkg/ directory inside the lindera-wasm crate.

Available Feature Flags (Advanced)

For advanced users who want to embed dictionaries directly into the WASM binary, the following feature flags are available. This increases the binary size significantly but eliminates the need to download dictionaries at runtime.

You can combine multiple dictionaries by enabling multiple feature flags:

wasm-pack build --target web --features embed-ipadic,embed-ko-dic

NPM Package Naming Convention

When publishing to npm, the recommended naming convention is:

lindera-wasm-{target}
lindera-wasm-{target}-{dict}

Examples:

• lindera-wasm-web
• lindera-wasm-web-ipadic
• lindera-wasm-bundler-unidic
• lindera-wasm-web-cjk

To set the package name before publishing, edit the name field in the generated pkg/package.json.

[!NOTE] This project's own release workflow (.github/workflows/release.yml) only builds and publishes two packages to npm -- lindera-wasm-web and lindera-wasm-bundler -- built without any embed-* feature. Dictionary-suffixed names such as lindera-wasm-web-ipadic are not published anywhere; they only illustrate what a local build with an embed-* feature (see Available Feature Flags above) would produce after you rename the package yourself.

Installing from npm

Pre-built packages are available on npm:

npm install lindera-wasm-web

Or with yarn:

yarn add lindera-wasm-web

[!NOTE] The npm package does not include dictionaries. Use the OPFS helpers to download dictionaries at runtime. See OPFS Dictionary Storage.

Quick Start

Web (Browser) -- OPFS Dictionary Loading

The recommended approach is to download dictionaries at runtime using the OPFS helpers:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { downloadDictionary, loadDictionaryFiles, hasDictionary } from 'lindera-wasm-web/opfs';

async function main() {
    await __wbg_init();

    // Download dictionary if not cached
    if (!await hasDictionary("ipadic")) {
        await downloadDictionary(
            "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
            "ipadic",
        );
    }

    // Load dictionary from OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    // Build tokenizer
    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

Note: Download a pre-built dictionary from GitHub Releases. See OPFS Dictionary Storage for the full workflow.

Expected output:

関西国際空港    名詞,固有名詞,一般,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Using Embedded Dictionaries (Advanced)

If you built with an embed-* feature flag, you can use embedded dictionaries:

[!NOTE] lindera-wasm-web-ipadic is an illustrative package name here, not something published to npm. Only lindera-wasm-web and lindera-wasm-bundler are actually published; see NPM Package Naming Convention for how to build and name a package like this yourself.

import __wbg_init, { TokenizerBuilder } from 'lindera-wasm-web-ipadic';

async function main() {
    await __wbg_init();

    const builder = new TokenizerBuilder();
    builder.setDictionary("embedded://ipadic");
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

Using Filters

You can add character filters and token filters to the tokenization pipeline:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

async function main() {
    await __wbg_init();

    // Assume dictionary is already cached in OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");

    // Add Unicode NFKC normalization
    builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

    // Add a stop-tags filter to remove particles and auxiliary verbs
    builder.appendTokenFilter("japanese_stop_tags", {
        tags: ["助詞", "助動詞"]
    });

    const tokenizer = builder.build();
    const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析エンジンです");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

N-Best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
results.forEach((result, rank) => {
    console.log(`--- NBEST ${rank + 1} (cost=${result.cost}) ---`);
    result.tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
});

Tokenizer API

This page documents the JavaScript/TypeScript API exposed by lindera-wasm.

TokenizerBuilder

Builder class for creating a configured Tokenizer instance.

Constructor

const builder = new TokenizerBuilder();

Creates a new builder with default settings.

Methods

setMode(mode)

Sets the tokenization mode.

• Parameters: mode (string) -- "normal" or "decompose"
• Returns: void

builder.setMode("normal");

setDictionary(uri)

Sets the dictionary to use for tokenization.

• Parameters: uri (string) -- Dictionary URI (e.g., "embedded://ipadic")
• Returns: void

builder.setDictionary("embedded://ipadic");

setDictionaryInstance(dictionary)

Sets a pre-loaded dictionary instance for tokenization. Use this when the dictionary has been loaded from bytes (e.g., via loadDictionaryFromBytes()) instead of from a URI.

• Parameters: dictionary (Dictionary) -- A loaded dictionary object
• Returns: void

import { loadDictionaryFromBytes } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

const files = await loadDictionaryFiles("ipadic");
const dictionary = loadDictionaryFromBytes(
    files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
    files.dictWords, files.matrixMtx, files.charDef, files.unk,
);

builder.setDictionaryInstance(dictionary);

setUserDictionary(uri)

Sets a user-defined dictionary by URI.

• Parameters: uri (string) -- Path or URI to the user dictionary
• Returns: void

builder.setUserDictionary("file:///path/to/user_dict.csv");

setUserDictionaryInstance(userDictionary)

Sets a pre-loaded user dictionary instance. Use this when the user dictionary has been loaded from bytes instead of from a URI.

• Parameters: userDictionary (UserDictionary) -- A loaded user dictionary object
• Returns: void

setKeepWhitespace(keep)