Lindera

A morphological analysis library in Rust. Lindera is forked from kuromoji-rs and aims to provide easy installation and concise APIs for tokenizing text in multiple languages.

Key Features

Tokenization Flow

graph LR
    subgraph Your Application
        T["Text"]
    end
    subgraph Lindera
        CF["Character Filters"]
        SEG["Segmenter\n(Dictionary + Viterbi)"]
        TF["Token Filters"]
    end
    T --> CF --> SEG --> TF --> R["Tokens"]

Document Map

Quick Example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = tokenizer.tokenize(text)?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Run the example:

cargo run --features=embed-ipadic --example=tokenize

Output:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

License

Lindera is released under the MIT License.

Architecture

Lindera is organized as a Cargo workspace comprising multiple crates. Each crate has a focused responsibility, from low-level CRF computation to high-level CLI and language bindings.

Crate Dependency Graph

graph TB
    CRF["lindera-crf\n(CRF Engine)"]
    DICT["lindera-dictionary\n(Dictionary Base)"]
    IPADIC["lindera-ipadic"]
    UNIDIC["lindera-unidic"]
    KODIC["lindera-ko-dic"]
    CCCEDICT["lindera-cc-cedict"]
    JIEBA["lindera-jieba"]
    NEOLOGD["lindera-ipadic-neologd"]
    LIB["lindera\n(Core Library)"]
    CLI["lindera-cli\n(CLI)"]
    PY["lindera-python\n(Python)"]
    WASM["lindera-wasm\n(WebAssembly)"]

    CRF --> DICT
    DICT --> IPADIC
    DICT --> UNIDIC
    DICT --> KODIC
    DICT --> CCCEDICT
    DICT --> JIEBA
    DICT --> NEOLOGD
    DICT --> LIB
    IPADIC --> LIB
    UNIDIC --> LIB
    KODIC --> LIB
    CCCEDICT --> LIB
    JIEBA --> LIB
    NEOLOGD --> LIB
    LIB --> CLI
    LIB --> PY
    LIB --> WASM

Crate Overview

Tokenization Pipeline

Lindera processes text through a multi-stage pipeline:

Input Text
  |
  v
Character Filters    -- Normalize characters (e.g., Unicode normalization, mapping)
  |
  v
Segmenter            -- Segment text into tokens using a dictionary and the Viterbi algorithm
  |
  v
Token Filters        -- Transform tokens (e.g., POS filtering, stop words, stemming)
  |
  v
Output Tokens

The Segmenter is the core component. It builds a lattice of candidate tokens from the dictionary, then applies the Viterbi algorithm to find the lowest-cost path, producing the most likely segmentation.

Feature Flags

Learn More

• Getting Started -- Installation and first steps
• Core Concepts -- Dictionaries, tokenization, and filters
• Lindera Library -- Configuration, segmenter, and API
• Lindera CLI -- Command-line interface
• Development Guide -- Build, test, and contribute

Getting Started

This section will guide you through installing Lindera and running your first morphological analysis.

• Installation -- Add Lindera to your project and configure environment variables
• Quick Start -- Tokenize your first text in just a few lines of code
• Examples -- Explore example programs for common use cases

Installation

Put the following in Cargo.toml:

[dependencies]
lindera = "3.0.0"

Dictionary Setup

Lindera requires a pre-built dictionary at runtime. Download a dictionary from GitHub Releases and specify its path when loading:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

#![allow(unused)]
fn main() {
// Cargo.toml: lindera = { version = "3.0.0", features = ["embed-ipadic"] }
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

Environment Variables

LINDERA_DICTIONARIES_PATH

The LINDERA_DICTIONARIES_PATH environment variable specifies a directory for caching dictionary source files. This enables:

• Offline builds: Once downloaded, dictionary source files are preserved for future builds
• Faster builds: Subsequent builds skip downloading if valid cached files exist
• Reproducible builds: Ensures consistent dictionary versions across builds

Usage:

export LINDERA_DICTIONARIES_PATH=/path/to/dicts
cargo build --features=ipadic

When set, dictionary source files are stored in $LINDERA_DICTIONARIES_PATH/<version>/ where <version> is the lindera-dictionary crate version. The cache validates files using MD5 checksums - invalid files are automatically re-downloaded.

[!NOTE] LINDERA_CACHE is deprecated but still supported for backward compatibility. It will be used if LINDERA_DICTIONARIES_PATH is not set.

LINDERA_CONFIG_PATH

The LINDERA_CONFIG_PATH environment variable specifies the path to a YAML configuration file for the tokenizer. This allows you to configure tokenizer behavior without modifying Rust code.

export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

See the Configuration section for details on the configuration format.

DOCS_RS

The DOCS_RS environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.

This is primarily used internally by docs.rs and typically doesn't need to be set by users.

LINDERA_WORKDIR

The LINDERA_WORKDIR environment variable is automatically set during the build process by the lindera-dictionary crate. It points to the directory containing the built dictionary data files and is used internally by dictionary crates to locate their data files.

This variable is set automatically and should not be modified by users.

Quick Start

This example covers the basic usage of Lindera.

It will:

• Create a tokenizer in normal mode
• Tokenize the input text
• Output the tokens

First, download a pre-built IPADIC dictionary from GitHub Releases and extract it to a local directory (e.g., /path/to/ipadic).

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("/path/to/ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "関西国際空港限定トートバッグ";
    let mut tokens = tokenizer.tokenize(text)?;
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

The above example can be run as follows:

% cargo run --example=tokenize

[!TIP] If you embed the dictionary into the binary using the embed-ipadic feature (advanced usage), you can use load_dictionary("embedded://ipadic") instead of specifying a file path. See Feature Flags for details.

You can see the result as follows:

text:   関西国際空港限定トートバッグ
token:  関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token:  限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token:  トートバッグ    名詞,一般,*,*,*,*,*,*,*

Examples

Lindera includes several example programs that demonstrate common use cases. The source code is available in the examples directory on GitHub.

Before running the examples, download a pre-built IPADIC dictionary from GitHub Releases and extract it to a local directory.

Available Examples

tokenize

Basic tokenization using an external IPADIC dictionary. Segments input text and prints each token with its part-of-speech details.

cargo run --example=tokenize

tokenize_with_user_dict

Tokenization with a user dictionary. Shows how to supplement the dictionary with custom entries for domain-specific terms.

cargo run --example=tokenize_with_user_dict

tokenize_with_filters

Tokenization with character filters and token filters. Demonstrates the text processing pipeline, including Unicode normalization, part-of-speech filtering, and other transformations.

cargo run --example=tokenize_with_filters

tokenize_with_config

Tokenization using a YAML configuration file. Shows how to configure the tokenizer declaratively instead of programmatically.

cargo run --example=tokenize_with_config

Core Concepts

This section explains the fundamental concepts behind Lindera's morphological analysis system.

• Morphological Analysis - How Lindera segments and analyzes text.
• Dictionaries - Dictionary formats supported by Lindera.
• Tokenization - Tokenization modes and N-Best analysis.
• User Dictionary - Adding custom words with user dictionaries.
• Character Filters - Pre-processing text before tokenization.

Morphological Analysis

What is morphological analysis?

Morphological analysis is the process of breaking down text into its smallest meaningful units (morphemes) and identifying their grammatical properties. For languages like Japanese, Chinese, and Korean -- where words are not separated by spaces -- morphological analysis is an essential first step for natural language processing tasks such as search indexing, text classification, and machine translation.

How Lindera works

Lindera is a dictionary-based morphological analyzer. It uses a pre-compiled system dictionary containing known words along with their costs, and applies the Viterbi algorithm to find the optimal segmentation of input text.

The analysis process works as follows:

1. Lattice construction: Lindera scans the input text and looks up all possible words in the dictionary at every position, building a directed acyclic graph (lattice) of candidate segmentations.
2. Cost assignment: Each candidate word has an associated word cost (from the dictionary), and each pair of adjacent words has a connection cost (from the connection cost matrix).
3. Optimal path search: The Viterbi algorithm finds the path through the lattice with the minimum total cost, producing the best segmentation.

Key terminology

Cost-based segmentation

The Viterbi algorithm selects the segmentation path with the minimum total cost. The total cost of a path is the sum of:

• Word costs: Each word in the dictionary has an associated cost. Lower cost means the word is more likely to appear. Common words tend to have lower costs, while rare words have higher costs.
• Connection costs: The cost of connecting two adjacent words, determined by the right context ID of the left word and the left context ID of the right word.

The algorithm computes:

Total cost = sum of word costs + sum of connection costs

By minimizing this total cost, Lindera finds the most natural segmentation of the input text.

Connection cost matrix

The connection cost matrix stores the cost of transitioning from one word to another. It is a two-dimensional matrix indexed by:

• The right context ID of the preceding word
• The left context ID of the following word

These context IDs encode grammatical information about word boundaries. For example, the connection cost between a noun and a particle is typically low (natural sequence), while the connection cost between two verbs in base form might be high (unnatural sequence).

The connection cost matrix is compiled into binary format as part of the dictionary build process and is loaded at runtime for efficient lookup.

Dictionaries

Lindera supports various dictionaries for Japanese, Korean, and Chinese morphological analysis. Each dictionary is provided as a separate crate.

Obtaining Dictionaries

Pre-built dictionaries are available for download from GitHub Releases. Download the dictionary archive for your target language and extract it to a local directory.

#![allow(unused)]
fn main() {
// Load an external dictionary from a local path
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you need a self-contained binary without external dictionary files, you can embed dictionaries using the embed-* feature flags and load them using the embedded:// scheme:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

See each dictionary crate's documentation for format details, build instructions, and usage examples.

Tokenization

Lindera provides multiple tokenization modes and supports N-Best analysis for enumerating alternative segmentation candidates.

Tokenization modes

Normal mode

Normal mode performs standard tokenization based on dictionary entries. Compound words that exist as single entries in the dictionary are kept as-is.

Example -- tokenizing "関西国際空港限定トートバッグ" in Normal mode:

関西国際空港 | 限定 | トートバッグ

The compound noun "関西国際空港" (Kansai International Airport) is preserved as a single token because it exists as one entry in the dictionary.

Decompose mode

Decompose mode further breaks down compound nouns into their constituent parts, even when the compound exists as a dictionary entry.

Example -- tokenizing "関西国際空港限定トートバッグ" in Decompose mode:

関西 | 国際 | 空港 | 限定 | トートバッグ

The compound "関西国際空港" is decomposed into "関西", "国際", and "空港".

Selecting a mode

In Rust, specify the mode when creating a Segmenter:

#![allow(unused)]
fn main() {
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;

// Normal mode
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

// Decompose mode
let segmenter = Segmenter::new(Mode::Decompose(Default::default()), dictionary, None);
}

With the CLI, use the --mode flag:

echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode normal
echo "関西国際空港限定トートバッグ" | lindera tokenize --dict embedded://ipadic --mode decompose

N-Best tokenization

N-Best tokenization enumerates the top N tokenization candidates ordered by total path cost (lower cost = better segmentation). This is useful when the best result is ambiguous, or when you want to explore alternative interpretations of the input text.

Algorithm

N-Best tokenization is based on the Forward-DP Backward-A* algorithm, which is compatible with MeCab's N-Best implementation. The forward pass computes optimal costs using dynamic programming, and the backward pass uses A* search to enumerate paths in order of increasing total cost.

Parameters

The tokenize_nbest method accepts the following parameters:

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "すもももももももものうち";

    // Get top 3 tokenization results
    let results = tokenizer.tokenize_nbest(text, 3, false, None)?;

    for (rank, (tokens, cost)) in results.iter().enumerate() {
        println!("--- NBEST {} (cost={}) ---", rank + 1, cost);
        for token in tokens {
            let details = token.details().join(",");
            println!("{}\t{}", token.surface.as_ref(), details);
        }
    }

    Ok(())
}

Output:

--- NBEST 1 (cost=7546) ---
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
--- NBEST 2 (cost=7914) ---
...

CLI example

echo "すもももももももものうち" | lindera tokenize --dict embedded://ipadic -N 3

Lattice reuse

For repeated tokenization, you can reuse a Lattice to reduce memory allocations:

#![allow(unused)]
fn main() {
use lindera_dictionary::viterbi::Lattice;

let mut lattice = Lattice::default();
let results = tokenizer.tokenize_nbest_with_lattice(text, &mut lattice, 3, false, None)?;
}

User Dictionary

A user dictionary is a supplementary dictionary that allows you to register custom words alongside the system dictionary. This is useful for domain-specific terms, brand names, proper nouns, or any words that are not in the default system dictionary.

CSV format

The simplest user dictionary format is a CSV file with three columns:

<surface>,<part_of_speech>,<reading>

Example CSV content

東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ

Each dictionary type (IPADIC, UniDic, ko-dic, etc.) also supports a detailed CSV format with full control over context IDs, costs, and all feature fields. See the Dictionaries section for the detailed format of each dictionary type.

Rust API example

use std::fs::File;
use std::path::PathBuf;

use lindera::dictionary::{Metadata, load_dictionary, load_user_dictionary};
use lindera::error::LinderaErrorKind;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let user_dict_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("user_dict")
        .join("ipadic_simple_userdic.csv");

    let metadata_file = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../lindera-ipadic")
        .join("metadata.json");
    let metadata: Metadata = serde_json::from_reader(
        File::open(metadata_file)
            .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
            .unwrap(),
    )
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
    .unwrap();

    let dictionary = load_dictionary("embedded://ipadic")?;
    let user_dictionary = load_user_dictionary(user_dict_path.to_str().unwrap(), &metadata)?;
    let segmenter = Segmenter::new(
        Mode::Normal,
        dictionary,
        Some(user_dictionary), // Using the loaded user dictionary
    );

    // Create a tokenizer.
    let tokenizer = Tokenizer::new(segmenter);

    // Tokenize a text.
    let text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です";
    let mut tokens = tokenizer.tokenize(text)?;

    // Print the text and tokens.
    println!("text:\t{}", text);
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("token:\t{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Output:

text:   東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です
token:  東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
token:  の      助詞,連体化,*,*,*,*,の,ノ,ノ
token:  最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
token:  は      助詞,係助詞,*,*,*,*,は,ハ,ワ
token:  とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
token:  です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス

Building a user dictionary with CLI

You can build a user dictionary from CSV to binary format using the CLI:

lindera build --src <source_dir> --dest <dest_dir> --metadata <metadata.json> --user

Binary vs CSV user dictionary

• CSV format: Loaded and parsed at runtime. Convenient for development and small dictionaries.
• Binary format: Pre-compiled for faster loading. Recommended for production use with large user dictionaries.

Both formats can be specified when creating a Segmenter. The binary format skips the CSV parsing step, resulting in faster startup times.

Character Filters

Character filters are pre-processing steps applied to the input text before tokenization. They normalize or transform characters to improve tokenization quality and consistency.

Available character filters

unicode_normalize

Applies Unicode normalization to the input text. This is useful for normalizing full-width characters to half-width, or for canonicalizing equivalent Unicode representations.

Supported normalization forms:

japanese_iteration_mark

Normalizes Japanese iteration marks into their expanded forms. Iteration marks are special characters that indicate the repetition of the preceding character.

The filter accepts two boolean parameters: whether to normalize Hiragana iteration marks and whether to normalize Katakana iteration marks.

mapping

Performs character-level string replacement based on a user-defined mapping table. This can be used for custom normalization rules.

For example, mapping "リンデラ" to "Lindera".

YAML configuration example

When using Lindera with a YAML configuration file, character filters can be specified in the character_filters section:

segmenter:
  mode: normal
  dictionary: "embedded://ipadic"

character_filters:
  - kind: unicode_normalize
    args:
      kind: nfkc
  - kind: japanese_iteration_mark
    args:
      normalize_kanji: true
      normalize_kana: true
  - kind: mapping
    args:
      mapping:
        リンデラ: Lindera

Rust API example

Character filters can be created and appended to a Tokenizer programmatically:

use lindera::character_filter::BoxCharacterFilter;
use lindera::character_filter::unicode_normalize::{
    UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera::character_filter::japanese_iteration_mark::JapaneseIterationMarkCharacterFilter;
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    // Create character filters.
    let unicode_normalize_char_filter =
        UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);

    let japanese_iteration_mark_char_filter =
        JapaneseIterationMarkCharacterFilter::new(true, true);

    // Create a tokenizer and append character filters.
    let mut tokenizer = Tokenizer::new(segmenter);

    tokenizer
        .append_character_filter(BoxCharacterFilter::from(unicode_normalize_char_filter))
        .append_character_filter(BoxCharacterFilter::from(
            japanese_iteration_mark_char_filter,
        ));

    // Tokenize text -- full-width "Ｌｉｎｄｅｒａ" will be normalized to "Lindera".
    let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。";
    let tokens = tokenizer.tokenize(text)?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

Output (with NFKC normalization applied):

token: "Lindera", details: Some(["名詞", "固有名詞", "組織", "*", "*", "*", "*", "*", "*"])
token: "は", details: Some(["助詞", "係助詞", "*", "*", "*", "*", "は", "ハ", "ワ"])
token: "形態素", details: Some(["名詞", "一般", "*", "*", "*", "*", "形態素", "ケイタイソ", "ケイタイソ"])
token: "解析", details: Some(["名詞", "サ変接続", "*", "*", "*", "*", "解析", "カイセキ", "カイセキ"])
token: "エンジン", details: Some(["名詞", "一般", "*", "*", "*", "*", "エンジン", "エンジン", "エンジン"])
token: "です", details: Some(["助動詞", "*", "*", "*", "特殊・デス", "基本形", "です", "デス", "デス"])
token: "。", details: Some(["記号", "句点", "*", "*", "*", "*", "。", "。", "。"])

Lindera CRF

Lindera CRF is a pure Rust implementation of Conditional Random Fields (CRFs), forked from rucrf. It provides a trainer and an estimator for CRFs with support for lattice structures.

Key Features

• Lattices with variable length edges
• L1, L2, and Elastic Net regularization
• Multi-threaded training
• Zero-copy deserialization with rkyv
• no_std support (without train feature)

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Changes from rucrf

• Serialization backend: Switched from bincode to rkyv for zero-copy deserialization
• Elastic Net regularization: Added Regularization::ElasticNet combining L1 and L2 penalties
• Rust 2024 edition: Updated to Rust 2024 edition
• Dependency updates: Updated argmin, argmin-math, hashbrown, etc.

Architecture

Module Structure

lindera-crf/src/
├── lib.rs                # Public API re-exports
├── feature.rs            # FeatureSet, FeatureProvider
├── lattice.rs            # Edge, Node, Lattice
├── model.rs              # RawModel, MergedModel, Model trait
├── trainer.rs            # Trainer, Regularization enum
├── errors.rs             # Error types
├── forward_backward.rs   # Forward-backward algorithm
├── math.rs               # Mathematical utilities (logsumexp)
├── optimizers/
│   └── lbfgs.rs          # L-BFGS optimization
└── utils.rs              # Utility traits

Key Components

FeatureProvider / FeatureSet

Manage per-label feature sets. Each FeatureSet holds unigram features and left/right bigram features for a given label. FeatureProvider aggregates FeatureSet instances and maps feature IDs to weights.

Lattice / Edge / Node

Lattice structure with variable-length edges for sequence labeling. Edge represents a candidate span with a label, while Node aggregates edges at a given position. The Lattice is constructed from input data and used by the model to find the best path.

Trainer

Trains a CRF model using L-BFGS optimization with configurable regularization. The trainer accepts labeled lattice examples, computes gradients via the forward-backward algorithm, and iteratively updates model weights.

Regularization

Configurable regularization strategies:

• L1: Sparse models via L1 penalty
• L2: Smooth models via L2 penalty
• ElasticNet: Combines L1 and L2 with a configurable l1_ratio

Model (trait)

Interface for searching the best path through a lattice. Two implementations are provided:

• RawModel: Stores weights in a flat vector indexed by feature ID
• MergedModel: Optimized for inference; merges feature weights into a compact representation serializable with rkyv

Forward-backward Algorithm

Computes alpha (forward) and beta (backward) values over the lattice. Used during training to calculate expected feature counts and gradients.

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-crf

Lindera Dictionary

Lindera Dictionary is the base library for morphological analysis dictionaries. It provides dictionary loading, building, Viterbi-based segmentation, and CRF-based training functionality.

Key Features

• Dictionary loading from filesystem or embedded data
• Dictionary building from MeCab-format CSV source files
• Viterbi algorithm for optimal segmentation
• N-best path generation (Forward-DP Backward-A*)
• Memory-mapped file support
• CRF-based dictionary training (with train feature)

Contents

• Architecture -- Internal structure and key components
• API Reference -- API documentation

Architecture

Module Structure

lindera-dictionary/src/
├── lib.rs               # Public API
├── dictionary.rs        # Dictionary, UserDictionary
├── builder.rs           # DictionaryBuilder
├── loader.rs            # DictionaryLoader trait, FSDictionaryLoader
├── viterbi.rs           # Lattice, Edge, Viterbi segmentation
├── nbest.rs             # NBestGenerator (Forward-DP Backward-A*)
├── mode.rs              # Mode (Normal/Decompose), Penalty
├── error.rs             # LinderaError, LinderaErrorKind
├── assets.rs            # Download and file management
├── dictionary/
│   ├── character_definition.rs    # Character type definitions
│   ├── connection_cost_matrix.rs  # Connection cost matrix
│   ├── prefix_dictionary.rs       # Double-array trie dictionary
│   ├── unknown_dictionary.rs      # Unknown word handling
│   ├── metadata.rs                # Dictionary metadata
│   └── schema.rs                  # Schema definitions
└── trainer/             # (train feature)
    ├── config.rs        # TrainerConfig
    ├── corpus.rs        # Corpus, Example, Word
    ├── feature_extractor.rs  # Feature template parsing
    ├── feature_rewriter.rs   # MeCab-compatible rewrite rules
    └── model.rs         # Trained model, tocost()

Key Components

Dictionary / UserDictionary

Main data structures holding the compiled dictionary data. A Dictionary contains the character definitions, connection cost matrix, prefix dictionary (double-array trie), and unknown word dictionary. UserDictionary allows users to add custom vocabulary on top of the system dictionary.

DictionaryBuilder

Fluent API for building dictionaries from source CSV files. It compiles MeCab-format dictionary sources into the binary format used at runtime.

DictionaryLoader / FSDictionaryLoader

DictionaryLoader is a trait for loading compiled dictionaries. FSDictionaryLoader is the filesystem-based implementation that reads dictionary files from a directory, with optional memory-mapped file support.

Viterbi (Lattice, Edge)

Builds a lattice of candidate tokens from the input text and finds the optimal segmentation path using the Viterbi algorithm. Each Edge in the lattice represents a candidate token with associated costs (word cost + connection cost).

NBestGenerator

Generates N-best segmentation paths using the Forward-DP Backward-A* algorithm. This enables applications to consider alternative segmentations beyond the single best path.

Mode

Controls tokenization behavior:

• Normal: Standard tokenization using the optimal Viterbi path
• Decompose: Further splits compound nouns based on configurable Penalty thresholds

Trainer (train feature)

CRF-based dictionary training pipeline using lindera-crf. The training workflow includes:

1. TrainerConfig: Parses seed dictionary, char.def, feature.def, and rewrite.def
2. Corpus: Manages training data as labeled examples
3. FeatureExtractor: Parses feature templates and assigns feature IDs
4. DictionaryRewriter: Applies MeCab-compatible 3-section rewrite rules
5. Model: Holds training results and exports dictionary files with cost conversion via tocost(weight, cost_factor)

Feature Flags

API Reference

The API reference is available. Please see following URL:

• lindera-dictionary

Lindera Library

The lindera crate is the core morphological analysis library. This section covers configuration, segmentation, token filters, error handling, and API reference.

• Configuration - YAML-based tokenizer configuration
• Segmenter - Core segmentation component using the Viterbi algorithm
• Token Filters - Post-processing filters for tokens
• Error Handling - Error types and handling patterns
• API Reference - Links to generated API documentation

Configuration

Lindera is able to read YAML format configuration files. Specify the path to the following file in the environment variable LINDERA_CONFIG_PATH. You can use it easily without having to code the behavior of the tokenizer in Rust code.

segmenter:
  mode: "normal"
  dictionary: "embedded://ipadic"
  # user_dictionary: "./resources/user_dict/ipadic_simple_userdic.csv"

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true
  - kind: mapping
    args:
       mapping:
         リンデラ: Lindera

token_filters:
  - kind: "japanese_compound_word"
    args:
      tags:
        - "名詞,数"
        - "名詞,接尾,助数詞"
      new_tag: "名詞,数"
  - kind: "japanese_number"
    args:
      tags:
        - "名詞,数"
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "接続詞"
        - "助詞"
        - "助詞,格助詞"
        - "助詞,格助詞,一般"
        - "助詞,格助詞,引用"
        - "助詞,格助詞,連語"
        - "助詞,係助詞"
        - "助詞,副助詞"
        - "助詞,間投助詞"
        - "助詞,並立助詞"
        - "助詞,終助詞"
        - "助詞,副助詞／並立助詞／終助詞"
        - "助詞,連体化"
        - "助詞,副詞化"
        - "助詞,特殊"
        - "助動詞"
        - "記号"
        - "記号,一般"
        - "記号,読点"
        - "記号,句点"
        - "記号,空白"
        - "記号,括弧閉"
        - "その他,間投"
        - "フィラー"
        - "非言語音"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "remove_diacritical_mark"
    args:
      japanese: false

% export LINDERA_CONFIG_PATH=./resources/config/lindera.yml

use std::path::PathBuf;

use lindera::tokenizer::TokenizerBuilder;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    // Load tokenizer configuration from file
    let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
        .join("../resources")
        .join("config")
        .join("lindera.yml");

    let builder = TokenizerBuilder::from_file(&path)?;

    let tokenizer = builder.build()?;

    let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。".to_string();
    println!("text: {text}");

    let tokens = tokenizer.tokenize(&text)?;

    for token in tokens {
        println!(
            "token: {:?}, start: {:?}, end: {:?}, details: {:?}",
            token.surface, token.byte_start, token.byte_end, token.details
        );
    }

    Ok(())
}

Segmenter

The Segmenter is the core component that performs morphological analysis. It uses the Viterbi algorithm to find the optimal segmentation of input text based on a dictionary and cost model.

Creating a Segmenter

A Segmenter requires three components:

• Mode - the tokenization strategy (Normal or Decompose)
• Dictionary - a system dictionary for morphological analysis
• UserDictionary (optional) - a supplementary dictionary for custom words

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
}

Tokenization Modes

Mode::Normal

Standard tokenization based on the dictionary entries. Words are segmented faithfully according to what is registered in the dictionary.

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Normal;
}

Mode::Decompose

Decomposes compound nouns into their constituent parts. This mode applies a configurable penalty to long compound words, encouraging the segmenter to split them into shorter components.

For example, with Mode::Normal, the compound word "関西国際空港" remains as a single token, while with Mode::Decompose, it is split into "関西", "国際", and "空港".

#![allow(unused)]
fn main() {
use lindera::mode::Mode;

let mode = Mode::Decompose(Default::default());
}

Dictionary Loading

Lindera provides the load_dictionary function to load dictionaries from various sources.

Embedded Dictionaries

When built with the appropriate feature flag (e.g., embed-ipadic), dictionaries can be loaded directly from the binary:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("embedded://ipadic")?;
}

Available embedded dictionary URIs:

• embedded://ipadic - IPADIC (Japanese)
• embedded://ipadic-neologd - IPADIC NEologd (Japanese)
• embedded://unidic - UniDic (Japanese)
• embedded://ko-dic - ko-dic (Korean)
• embedded://cc-cedict - CC-CEDICT (Chinese)
• embedded://jieba - Jieba (Chinese)

External Dictionaries

Pre-built dictionary directories can be loaded from the filesystem:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;

let dictionary = load_dictionary("/path/to/dictionary")?;
}

Using with Tokenizer

The Segmenter is typically used through the Tokenizer, which adds support for character filters and token filters:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let tokens = tokenizer.tokenize(text)?;

    for token in tokens {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }

    Ok(())
}

Token Filters

Token filters are post-processing components applied to tokens after segmentation. They can modify, remove, or transform tokens to suit specific use cases such as search indexing, text normalization, or linguistic analysis.

Available Token Filters

Japanese

Korean

General

YAML Configuration

Token filters can be configured in the YAML configuration file under the token_filters key:

token_filters:
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "助詞"
        - "助動詞"
        - "記号"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "lowercase"
  - kind: "length"
    args:
      min: 2

Rust API

Token filters can also be created and applied programmatically:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::token_filter::BoxTokenFilter;
use lindera::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera::token_filter::japanese_katakana_stem::JapaneseKatakanaStemTokenFilter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let mut tokenizer = Tokenizer::new(segmenter);

    // Add token filters
    let stop_tags_filter = JapaneseStopTagsTokenFilter::new(
        vec![
            "助詞".to_string(),
            "助動詞".to_string(),
            "記号".to_string(),
        ]
        .into_iter()
        .collect(),
    );
    tokenizer.append_token_filter(BoxTokenFilter::from(stop_tags_filter));

    let katakana_stem_filter = JapaneseKatakanaStemTokenFilter::new(3);
    tokenizer.append_token_filter(BoxTokenFilter::from(katakana_stem_filter));

    // Tokenize with filters applied
    let tokens = tokenizer.tokenize("Linderaは形態素解析エンジンです。")?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

The append_token_filter method adds filters in order. Filters are applied sequentially to the token list after segmentation.

Error Handling

Lindera uses a structured error system based on anyhow and thiserror for ergonomic error handling throughout the library.

LinderaResult

The LinderaResult<T> type alias is the standard return type for fallible operations in Lindera:

#![allow(unused)]
fn main() {
pub type LinderaResult<T> = Result<T, LinderaError>;
}

LinderaError

LinderaError is the main error type, containing an error kind and a source error with full context:

#![allow(unused)]
fn main() {
pub struct LinderaError {
    pub kind: LinderaErrorKind,
    source: anyhow::Error,
}
}

The add_context method allows attaching additional context to an error:

#![allow(unused)]
fn main() {
let error = error.add_context("failed to load dictionary from /path/to/dict");
}

LinderaErrorKind

LinderaErrorKind is an enum that categorizes errors:

Creating Errors

Use LinderaErrorKind::with_error to create an error from a kind and a source:

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let error = LinderaErrorKind::Io.with_error(anyhow::anyhow!("file not found: config.yml"));
}

Using the ? Operator

Since Lindera functions return LinderaResult, the ? operator can propagate errors naturally:

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn analyze(text: &str) -> LinderaResult<Vec<String>> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let tokens = tokenizer.tokenize(text)?;
    Ok(tokens.iter().map(|t| t.surface.as_ref().to_string()).collect())
}
}

Error Handling Patterns

Matching on Error Kind

#![allow(unused)]
fn main() {
use lindera::dictionary::load_dictionary;
use lindera::error::LinderaErrorKind;

match load_dictionary("/path/to/dictionary") {
    Ok(dict) => { /* use dictionary */ }
    Err(e) if e.kind() == LinderaErrorKind::NotFound => {
        eprintln!("Dictionary not found: {}", e);
    }
    Err(e) if e.kind() == LinderaErrorKind::Io => {
        eprintln!("I/O error loading dictionary: {}", e);
    }
    Err(e) => {
        eprintln!("Unexpected error: {}", e);
    }
}
}

Converting from External Errors

#![allow(unused)]
fn main() {
use lindera::error::LinderaErrorKind;

let content = std::fs::read_to_string("config.yml")
    .map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))?;
}

API Reference

The API reference is available. Please see following URL:

• lindera

Lindera CLI

A morphological analysis command-line interface for Lindera.

• Installation - Install or build the CLI
• Commands - Command reference for tokenize, build, train, and export
• Tutorial - Step-by-step guide to get started

Installation

Install via Cargo

You can install the binary via cargo:

% cargo install lindera-cli

Download from GitHub Releases

Alternatively, you can download a pre-built binary from the release page:

• https://github.com/lindera/lindera/releases

Obtaining Dictionaries

Lindera does not bundle dictionaries with the binary. You need to download a pre-built dictionary separately from the GitHub Releases page:

# Example: download and extract the IPADIC dictionary
% curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
% unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Then specify the dictionary path when using the CLI:

% lindera tokenize --dictionary /path/to/ipadic "関西国際空港限定トートバッグ"

Build from Source

Build without dictionaries (default)

Build a binary containing only the tokenizer and trainer without embedded dictionaries:

% cargo build --release

Build with all features

% cargo build --release --all-features

Build with Embedded Dictionaries (Advanced)

For advanced users who want to embed dictionaries directly into the binary, use the embed-* feature flags. This eliminates the need for external dictionary files at runtime but increases the binary size.

IPADIC (Japanese dictionary)

% cargo build --release --features=embed-ipadic

IPADIC NEologd (Japanese dictionary)

% cargo build --release --features=embed-ipadic-neologd

UniDic (Japanese dictionary)

% cargo build --release --features=embed-unidic

ko-dic (Korean dictionary)

% cargo build --release --features=embed-ko-dic

CC-CEDICT (Chinese dictionary)

% cargo build --release --features=embed-cc-cedict

Jieba (Chinese dictionary)

% cargo build --release --features=embed-jieba

[!TIP] After building with an embed-* feature flag, use the embedded:// scheme to load the embedded dictionary:

% lindera tokenize --dictionary embedded://ipadic "関西国際空港限定トートバッグ"

See Feature Flags for details.

Commands

The Lindera CLI provides four main commands:

• tokenize - Perform morphological analysis on text
• build - Build a dictionary from source CSV files
• train - Train a CRF model from annotated corpus data
• export - Export a trained model to dictionary format

tokenize

Perform morphological analysis (tokenization) on Japanese, Chinese, or Korean text using various dictionaries.

Parameters

• --dict / -d: Dictionary path or URI (required)
  • File path: /path/to/dictionary
  • Embedded: embedded://ipadic, embedded://unidic, etc.
• --output / -o: Output format (default: mecab)
  • mecab: MeCab-compatible format with part-of-speech info
  • wakati: Space-separated tokens only
  • json: Detailed JSON format with all token information
• --user-dict / -u: User dictionary path (optional)
• --mode / -m: Tokenization mode (default: normal)
  • normal: Standard tokenization
  • decompose: Decompose compound words
• --char-filter / -c: Character filter configuration (JSON)
• --token-filter / -t: Token filter configuration (JSON)
• --nbest / -N: Number of N-best results to return (default: 1). When set to 2 or more, N-best output is enabled.
• --nbest-unique: Deduplicate N-best results by removing paths that produce the same segmentation.
• --nbest-cost-threshold: Maximum cost difference from the best path. Only paths with cost within best_cost + threshold are returned.
• Input file: Optional file path (default: stdin)

Basic usage

# Tokenize text using a dictionary directory
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /path/to/dictionary

# Tokenize text using embedded dictionary
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

# Tokenize with different output format
echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

# Tokenize text from file
lindera tokenize \
  --dict /path/to/dictionary \
  --output wakati \
  input.txt

Examples with external dictionaries

Tokenize with external IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external IPADIC Neologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-neologd-0.0.7-20200820

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with external UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-unidic-2.1.2

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

Tokenize with external ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict /tmp/lindera-ko-dic-2.1.1-20180720

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

Tokenize with external CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-cc-cedict-0.1.0-20200409

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

Tokenize with external Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict /tmp/lindera-jieba-0.1.1

Examples with embedded dictionaries

Lindera can include dictionaries directly in the binary when built with specific feature flags. This allows tokenization without external dictionary files.

Tokenize with embedded IPADIC (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC dictionary in the binary, you must build with the --features=embed-ipadic option.

Tokenize with embedded UniDic (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://unidic

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-unidic option.

Tokenize with embedded IPADIC NEologd (Japanese dictionary)

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic-neologd

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-ipadic-neologd option.

Tokenize with embedded ko-dic (Korean dictionary)

% echo "한국어의형태해석을실시할수있습니다." | lindera tokenize \
  --dict embedded://ko-dic

한국어  NNG,*,F,한국어,Compound,*,*,한국/NNG/*+어/NNG/*
의      JKG,*,F,의,*,*,*,*
형태    NNG,*,F,형태,*,*,*,*
해석    NNG,행위,T,해석,*,*,*,*
을      JKO,*,T,을,*,*,*,*
실시    NNG,행위,F,실시,*,*,*,*
할      XSV+ETM,*,T,할,Inflect,XSV,ETM,하/XSV/*+ᆯ/ETM/*
수      NNB,*,F,수,*,*,*,*
있      VV,*,T,있,*,*,*,*
습니다  EF,*,F,습니다,*,*,*,*
.       SF,*,*,*,*,*,*,*
EOS

NOTE: To include ko-dic dictionary in the binary, you must build with the --features=embed-ko-dic option.

Tokenize with embedded CC-CEDICT (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://cc-cedict

可以    *,*,*,*,ke3 yi3,可以,可以,can/may/possible/able to/not bad/pretty good/
进行    *,*,*,*,jin4 xing2,進行,进行,to advance/to conduct/underway/in progress/to do/to carry out/to carry on/to execute/
中文    *,*,*,*,Zhong1 wen2,中文,中文,Chinese language/
形态学  *,*,*,*,xing2 tai4 xue2,形態學,形态学,morphology (in biology or linguistics)/
分析    *,*,*,*,fen1 xi1,分析,分析,to analyze/analysis/CL:個|个[ge4]/
。      *,*,*,*,*,*,*,*
EOS

NOTE: To include CC-CEDICT dictionary in the binary, you must build with the --features=embed-cc-cedict option.

Tokenize with embedded Jieba (Chinese dictionary)

% echo "可以进行中文形态学分析。" | lindera tokenize \
  --dict embedded://jieba

NOTE: To include Jieba dictionary in the binary, you must build with the --features=embed-jieba option.

User dictionary examples

Lindera supports user dictionaries to add custom words alongside system dictionaries. User dictionaries can be in CSV or binary format.

Use user dictionary (CSV format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Use user dictionary (Binary format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920 \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.bin

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Tokenization modes

Lindera provides two tokenization modes: normal and decompose.

Normal mode (default)

Tokenizes faithfully based on words registered in the dictionary:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode normal

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Decompose mode

Tokenizes compound noun words additionally:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Output formats

Lindera provides three output formats: mecab, wakati and json.

MeCab format (default)

Outputs results in MeCab-compatible format with part-of-speech information:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output mecab

お待ち  名詞,サ変接続,*,*,*,*,お待ち,オマチ,オマチ
し  動詞,自立,*,*,サ変・スル,連用形,する,シ,シ
て  助詞,接続助詞,*,*,*,*,て,テ,テ
おり  動詞,非自立,*,*,五段・ラ行,連用形,おる,オリ,オリ
ます  助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。  記号,句点,*,*,*,*,。,。,。
EOS

Wakati format

Outputs only the token text separated by spaces:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

お待ち し て おり ます 。

JSON format

Outputs detailed token information in JSON format:

% echo "お待ちしております。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

[
  {
    "base_form": "お待ち",
    "byte_end": 9,
    "byte_start": 0,
    "conjugation_form": "*",
    "conjugation_type": "*",
    "part_of_speech": "名詞",
    "part_of_speech_subcategory_1": "サ変接続",
    "part_of_speech_subcategory_2": "*",
    "part_of_speech_subcategory_3": "*",
    "pronunciation": "オマチ",
    "reading": "オマチ",
    "surface": "お待ち",
    "word_id": 14698
  },
  ...
]

N-Best tokenization

Lindera supports N-Best tokenization, which returns the top N tokenization candidates ordered by cost (lower cost = better). This is based on the Forward-DP Backward-A* algorithm, compatible with MeCab's N-Best implementation.

Basic N-Best example

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  -N 3

NBEST 1 (cost=7546)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 2 (cost=7914)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
NBEST 3 (cost=10060)
すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
も      助詞,係助詞,*,*,*,*,も,モ,モ
も      助詞,係助詞,*,*,*,*,も,モ,モ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
うち    名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

N-Best with unique results

When the same segmentation appears in multiple paths (differing only in internal Viterbi states), use --nbest-unique to deduplicate:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 5 --nbest-unique -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です
NBEST 4 (cost=19320)
営業 部長 谷川 で す
NBEST 5 (cost=20814)
営業 部 長谷 川 です

N-Best with cost threshold

Use --nbest-cost-threshold to limit results to paths within a certain cost range of the best path:

% echo "営業部長谷川です" | lindera tokenize \
  --dict embedded://ipadic \
  -N 10 --nbest-unique --nbest-cost-threshold 5000 -o wakati

NBEST 1 (cost=15760)
営業 部長 谷川 です
NBEST 2 (cost=17758)
営業 部長 谷 川 です
NBEST 3 (cost=18816)
営業 部 長谷川 です

Only 3 results are returned because the remaining candidates exceed 15760 + 5000 = 20760.

Advanced tokenization with filters

Lindera provides an analytical framework that combines character filters, tokenizers, and token filters for advanced text processing. Filters are configured using JSON.

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般"]}'

すもも  名詞,一般,*,*,*,*,すもも,スモモ,スモモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
もも    名詞,一般,*,*,*,*,もも,モモ,モモ
EOS

build

Build (compile) a morphological analysis dictionary from source CSV files for use with Lindera.

Build parameters

• --src / -s: Source directory containing dictionary CSV files (or single CSV file for user dictionary)
• --dest / -d: Destination directory for compiled dictionary output
• --metadata / -m: Metadata configuration file (metadata.json) that defines dictionary structure
• --user / -u: Build user dictionary instead of system dictionary (optional flag)

Dictionary types

System dictionary

A full morphological analysis dictionary containing:

• Lexicon entries (word definitions)
• Connection cost matrix
• Unknown word handling rules
• Character type definitions

User dictionary

A supplementary dictionary for custom words that works alongside a system dictionary.

Examples

Build IPADIC (Japanese dictionary)

# Download and extract IPADIC source files
% curl -L -o /tmp/mecab-ipadic-2.7.0-20250920.tar.gz "https://Lindera.dev/mecab-ipadic-2.7.0-20250920.tar.gz"
% tar zxvf /tmp/mecab-ipadic-2.7.0-20250920.tar.gz -C /tmp

# Build the dictionary
% lindera build \
  --src /tmp/mecab-ipadic-2.7.0-20250920 \
  --dest /tmp/lindera-ipadic-2.7.0-20250920 \
  --metadata ./lindera-ipadic/metadata.json

Build IPADIC NEologd (Japanese dictionary)

% curl -L -o /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz "https://lindera.dev/mecab-ipadic-neologd-0.0.7-20200820.tar.gz"
% tar zxvf /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ipadic-neologd-0.0.7-20200820 \
  --dest /tmp/lindera-ipadic-neologd-0.0.7-20200820 \
  --metadata ./lindera-ipadic-neologd/metadata.json

Build UniDic (Japanese dictionary)

% curl -L -o /tmp/unidic-mecab-2.1.2.tar.gz "https://Lindera.dev/unidic-mecab-2.1.2.tar.gz"
% tar zxvf /tmp/unidic-mecab-2.1.2.tar.gz -C /tmp

% lindera build \
  --src /tmp/unidic-mecab-2.1.2 \
  --dest /tmp/lindera-unidic-2.1.2 \
  --metadata ./lindera-unidic/metadata.json

Build CC-CEDICT (Chinese dictionary)

% curl -L -o /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz "https://lindera.dev/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz"
% tar zxvf /tmp/CC-CEDICT-MeCab-0.1.0-20200409.tar.gz -C /tmp

% lindera build \
  --src /tmp/CC-CEDICT-MeCab-0.1.0-20200409 \
  --dest /tmp/lindera-cc-cedict-0.1.0-20200409 \
  --metadata ./lindera-cc-cedict/metadata.json

Build Jieba (Chinese dictionary)

% curl -L -o /tmp/mecab-jieba-0.1.1.tar.gz "https://lindera.dev/mecab-jieba-0.1.1.tar.gz"
% tar zxvf /tmp/mecab-jieba-0.1.1.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-jieba-0.1.1/dict-src \
  --dest /tmp/lindera-jieba-0.1.1 \
  --metadata ./lindera-jieba/metadata.json

Build ko-dic (Korean dictionary)

% curl -L -o /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz "https://Lindera.dev/mecab-ko-dic-2.1.1-20180720.tar.gz"
% tar zxvf /tmp/mecab-ko-dic-2.1.1-20180720.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ko-dic-2.1.1-20180720 \
  --dest /tmp/lindera-ko-dic-2.1.1-20180720 \
  --metadata ./lindera-ko-dic/metadata.json

Build user dictionaries

Build IPADIC user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera IPADIC Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ipadic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ipadic/metadata.json \
  --user

Build UniDic user dictionary (Japanese)

For more details about user dictionary format please refer to the following URL:

• Lindera UniDic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/unidic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-unidic/metadata.json \
  --user

Build CC-CEDICT user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera CC-CEDICT Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/cc-cedict_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-cc-cedict/metadata.json \
  --user

Build Jieba user dictionary (Chinese)

For more details about user dictionary format please refer to the following URL:

• Lindera Jieba Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/jieba_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-jieba/metadata.json \
  --user

Build ko-dic user dictionary (Korean)

For more details about user dictionary format please refer to the following URL:

• Lindera ko-dic Builder/User Dictionary Format

% lindera build \
  --src ./resources/user_dict/ko-dic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ko-dic/metadata.json \
  --user

train

Train a new morphological analysis model from annotated corpus data. To use this feature, you must build with the train feature flag enabled. (The train feature flag is enabled by default.)

Train parameters

• --seed / -s: Seed lexicon file (CSV format) to be weighted
• --corpus / -c: Training corpus (annotated text)
• --char-def / -C: Character definition file (char.def)
• --unk-def / -u: Unknown word definition file (unk.def) to be weighted
• --feature-def / -f: Feature definition file (feature.def)
• --rewrite-def / -r: Rewrite rule definition file (rewrite.def)
• --output / -o: Output model file
• --lambda / -l: L1 regularization (0.0-1.0) (default: 0.01)
• --max-iterations / -i: Maximum number of iterations for training (default: 100)
• --max-threads / -t: Maximum number of threads (defaults to CPU core count, auto-adjusted based on dataset size)

Basic workflow

1. Prepare training files

Seed lexicon file (seed.csv):

The seed lexicon file contains initial dictionary entries used for training the CRF model. Each line represents a word entry with comma-separated fields:

• Surface
• Left context ID
• Right context ID
• Word cost
• Part-of-speech tags (multiple fields)
• Base form
• Reading (katakana)
• Pronunciation

Note: The exact field definitions differ between dictionary formats (IPADIC, UniDic, ko-dic, CC-CEDICT). Please refer to each dictionary's format specification for details.

外国,0,0,0,名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人,0,0,0,名詞,接尾,一般,*,*,*,人,ジン,ジン

Training corpus (corpus.txt):

The training corpus file contains annotated text data used to train the CRF model. Each line consists of:

• A surface form (word) followed by a tab character
• Comma-separated morphological features (part-of-speech tags, base form, reading, pronunciation)
• Sentences are separated by "EOS" (End Of Sentence) markers

外国	名詞,一般,*,*,*,*,外国,ガイコク,ガイコク
人	名詞,接尾,一般,*,*,*,人,ジン,ジン
参政	名詞,サ変接続,*,*,*,*,参政,サンセイ,サンセイ
権	名詞,接尾,一般,*,*,*,権,ケン,ケン
EOS

For detailed information about file formats and advanced features, see TRAINER_README.md.

2. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

3. Training results

The trained model will contain:

• Existing words: All seed dictionary records with newly learned weights
• New words: Words from the corpus not in the seed dictionary, added with appropriate weights

export

Export a trained model file to Lindera dictionary format files. This feature requires building with the train feature flag enabled.

Export parameters

• --model / -m: Path to the trained model file (.dat format)
• --output / -o: Directory to output the dictionary files
• --metadata: Optional metadata.json file to update with trained model information
• --cost-factor: Override cost factor for weight-to-cost conversion (default: value from trained model, typically 700)

Output files

The export command creates the following dictionary files in the output directory:

• lex.csv: Lexicon file with learned weights (MeCab-compatible cost via tocost())
• matrix.def: Dense connection cost matrix covering all (right_id, left_id) pairs
• unk.def: Unknown word definitions
• char.def: Character type definitions
• feature.def: Feature template definitions (copied from trained model)
• rewrite.def: Feature rewrite rules (copied from trained model)
• left-id.def: Left context ID to feature string mapping
• right-id.def: Right context ID to feature string mapping
• metadata.json: Updated metadata file (if --metadata option is provided)

Complete workflow example

1. Train model

lindera train \
  --seed ./resources/training/seed.csv \
  --corpus ./resources/training/corpus.txt \
  --unk-def ./resources/training/unk.def \
  --char-def ./resources/training/char.def \
  --feature-def ./resources/training/feature.def \
  --rewrite-def ./resources/training/rewrite.def \
  --output /tmp/lindera/training/model.dat \
  --lambda 0.01 \
  --max-iterations 100

2. Export to dictionary format

lindera export \
  --model /tmp/lindera/training/model.dat \
  --metadata ./resources/training/metadata.json \
  --output /tmp/lindera/training/dictionary

3. Build dictionary

lindera build \
  --src /tmp/lindera/training/dictionary \
  --dest /tmp/lindera/training/compiled_dictionary \
  --metadata /tmp/lindera/training/dictionary/metadata.json

4. Use trained dictionary

echo "これは外国人参政権です。" | lindera tokenize \
  -d /tmp/lindera/training/compiled_dictionary

Metadata update feature

When the --metadata option is provided, the export command will:

1. Read the base metadata.json file to preserve existing configuration
2. Update specific fields with values from the trained model:
   • default_left_context_id: Maximum left context ID from trained model
   • default_right_context_id: Maximum right context ID from trained model
   • default_word_cost: Calculated from feature weight median
   • model_info: Training statistics including feature count, label count, matrix size, iterations, regularization, version, and timestamp
3. Preserve existing settings such as dictionary name, character encoding, schema definitions, and other user-defined configuration

Tutorial

This tutorial walks you through the basic usage of the Lindera CLI, from installation to advanced text processing.

1. Install the CLI

Install Lindera CLI with the embedded IPADIC dictionary:

% cargo install lindera-cli --features=embed-ipadic

Verify the installation:

% lindera --help

2. Basic tokenization with embedded dictionary

Tokenize Japanese text using the embedded IPADIC dictionary:

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic

Expected output:

東京    名詞,固有名詞,地域,一般,*,*,東京,トウキョウ,トーキョー
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
日本    名詞,固有名詞,地域,国,*,*,日本,ニホン,ニホン
の      助詞,連体化,*,*,*,*,の,ノ,ノ
首都    名詞,一般,*,*,*,*,首都,シュト,シュト
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
。      記号,句点,*,*,*,*,。,。,。
EOS

3. Try different output formats

Wakati format (word segmentation only)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic \
  --output wakati

Expected output:

東京 は 日本 の 首都 です 。

JSON format (detailed information)

% echo "東京は日本の首都です。" | lindera tokenize \
  --dict embedded://ipadic \
  --output json

This produces a JSON array with detailed token information including byte offsets, part-of-speech tags, readings, and more.

4. Use decompose mode

Decompose mode splits compound nouns into their constituent parts:

% echo "関西国際空港限定トートバッグ" | lindera tokenize \
  --dict embedded://ipadic \
  --mode decompose

Expected output:

関西    名詞,固有名詞,地域,一般,*,*,関西,カンサイ,カンサイ
国際    名詞,一般,*,*,*,*,国際,コクサイ,コクサイ
空港    名詞,一般,*,*,*,*,空港,クウコウ,クーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*
EOS

Compare with normal mode, where "関西国際空港" remains as a single token.

5. Apply character and token filters

Use Unicode normalization and keep only common nouns:

% echo "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。" | lindera tokenize \
  --dict embedded://ipadic \
  --char-filter 'unicode_normalize:{"kind":"nfkc"}' \
  --token-filter 'japanese_keep_tags:{"tags":["名詞,一般","名詞,固有名詞,組織"]}'

Expected output:

Lindera 名詞,固有名詞,組織,*,*,*,*,*,*
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
エンジン        名詞,一般,*,*,*,*,エンジン,エンジン,エンジン
EOS

The Unicode normalization converts full-width characters to half-width, and the token filter keeps only tokens matching the specified part-of-speech tags.

You can also combine multiple filters:

% echo "すもももももももものうち" | lindera tokenize \
  --dict embedded://ipadic \
  --token-filter 'japanese_stop_tags:{"tags":["助詞","助詞,係助詞","助詞,連体化"]}'

6. Use user dictionary

Create a CSV file with custom word entries (e.g., my_dict.csv):

東京スカイツリー,カスタム名詞,トウキョウスカイツリー

Tokenize with the user dictionary:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./my_dict.csv

Without the user dictionary, "東京スカイツリー" would be split into multiple tokens. With the user dictionary, it is recognized as a single token.

For pre-built user dictionary examples, see:

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

Expected output:

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Lindera Python

Lindera Python provides Python bindings for the Lindera morphological analysis engine, built with PyO3. It brings Lindera's high-performance tokenization capabilities to the Python ecosystem with support for Python 3.10 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from PyPI

Pre-built wheels are available on PyPI:

pip install lindera-python

[!NOTE] The PyPI package does not include dictionaries. See Obtaining Dictionaries below.

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Building from Source

If you need to build from source (e.g., to enable specific feature flags), the following prerequisites are required:

• Python 3.10 or later (up to 3.14)
• Rust toolchain -- Install via rustup
• maturin -- Python package for building Rust-based Python extensions

Install maturin with pip:

pip install maturin

Development Build

Build and install lindera-python in development mode:

cd lindera-python
maturin develop

Or use the project Makefile:

make python-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

maturin develop --features train

Feature Flags

Multiple features can be combined:

maturin develop --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Python:

import lindera

print(lindera.version())

Quick Start

This guide shows how to tokenize text using lindera-python.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

from lindera import TokenizerBuilder

builder = TokenizerBuilder()
builder.set_mode("normal")
builder.set_dictionary("/path/to/ipadic")
tokenizer = builder.build()

tokens = tokenizer.tokenize("関西国際空港限定トートバッグ")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("/path/to/ipadic")
    .build()
)

tokens = tokenizer.tokenize("すもももももももものうち")
for token in tokens:
    print(f"{token.surface}\t{token.get_detail(0)}")

Accessing Token Properties

Each token exposes the following properties:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
tokens = tokenizer.tokenize("東京タワー")

for token in tokens:
    print(f"Surface: {token.surface}")
    print(f"Byte range: {token.byte_start}..{token.byte_end}")
    print(f"Position: {token.position}")
    print(f"Word ID: {token.word_id}")
    print(f"Unknown: {token.is_unknown}")
    print(f"Details: {token.details}")
    print()

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

from lindera import TokenizerBuilder

tokenizer = TokenizerBuilder().set_dictionary("/path/to/ipadic").build()
results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)

for tokens, cost in results:
    surfaces = [t.surface for t in tokens]
    print(f"Cost {cost}: {' / '.join(surfaces)}")

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

TokenizerBuilder()

Creates a new builder with default configuration.

from lindera import TokenizerBuilder

builder = TokenizerBuilder()

TokenizerBuilder().from_file(file_path)

Loads configuration from a JSON file and returns a new builder.

builder = TokenizerBuilder().from_file("config.json")

Configuration Methods

All setter methods return self for method chaining.

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode("normal")

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary("embedded://ipadic")

# Use an external dictionary
builder.set_dictionary("/path/to/dictionary")

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary("/path/to/user_dictionary")

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(True)

append_character_filter(kind, args=None)

Appends a character filter to the preprocessing pipeline.

builder.append_character_filter("unicode_normalize", {"kind": "nfkc"})

append_token_filter(kind, args=None)

Appends a token filter to the postprocessing pipeline.

builder.append_token_filter("lowercase", {})

Build

build()

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build()

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Tokenizer(dictionary, mode="normal", user_dictionary=None)

Creates a tokenizer directly from a loaded dictionary.

from lindera import Tokenizer, load_dictionary

dictionary = load_dictionary("embedded://ipadic")
tokenizer = Tokenizer(dictionary, mode="normal")

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns a list of Token objects.

tokens = tokenizer.tokenize("形態素解析")

Parameters:

Returns: list[Token]

tokenize_nbest(text, n, unique=False, cost_threshold=None)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest("すもももももももものうち", n=3)
for tokens, cost in results:
    print(cost, [t.surface for t in tokens])

Parameters:

Returns: list[tuple[list[Token], int]]

Token

Token represents a single morphological token.

Properties

Token Methods

get_detail(index)

Returns the detail string at the specified index, or None if the index is out of range.

token = tokenizer.tokenize("東京")[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: str or None

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Dictionary Management

Lindera Python provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

from lindera import load_dictionary

dictionary = load_dictionary("/path/to/ipadic")

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = load_dictionary("embedded://ipadic")

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

from lindera import load_user_dictionary, Metadata

metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

Pass the user dictionary when building a tokenizer:

from lindera import Tokenizer, load_dictionary, load_user_dictionary, Metadata

dictionary = load_dictionary("/path/to/ipadic")
metadata = Metadata()
user_dict = load_user_dictionary("/path/to/user_dictionary", metadata)

tokenizer = Tokenizer(dictionary, mode="normal", user_dictionary=user_dict)

Or via the builder:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("/path/to/ipadic")
    .set_user_dictionary("/path/to/user_dictionary")
    .build()
)

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

from lindera import build_dictionary, Metadata

metadata = Metadata(name="custom", encoding="UTF-8")
build_dictionary("/path/to/input_dir", "/path/to/output_dir", metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

from lindera import build_user_dictionary, Metadata

metadata = Metadata()
build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

build_user_dictionary("ipadic", "user_words.csv", "/path/to/output_dir")

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

from lindera import Metadata

# Default metadata
metadata = Metadata()

# Custom metadata
metadata = Metadata(
    name="my_dictionary",
    encoding="UTF-8",
    default_word_cost=-10000,
)

Loading from JSON

metadata = Metadata.from_json_file("metadata.json")

Properties

All properties support both getting and setting:

metadata = Metadata()
metadata.name = "custom_dict"
metadata.encoding = "EUC-JP"
print(metadata.name)  # "custom_dict"

to_dict()

Returns a dictionary representation of the metadata:

metadata = Metadata(name="test")
print(metadata.to_dict())

Text Processing Pipeline

Lindera Python supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .build()
)

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("mapping", {
        "mapping": {
            "\u30fc": "-",
            "\uff5e": "~",
        }
    })
    .build()
)

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    .build()
)

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("lowercase", {})
    .build()
)

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_base_form", {})
    .build()
)

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞", "助動詞"],
    })
    .build()
)

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

tokenizer = (
    TokenizerBuilder()
    .set_dictionary("embedded://ipadic")
    .append_token_filter("japanese_keep_tags", {
        "tags": ["名詞"],
    })
    .build()
)

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

from lindera import TokenizerBuilder

tokenizer = (
    TokenizerBuilder()
    .set_mode("normal")
    .set_dictionary("embedded://ipadic")
    # Preprocessing
    .append_character_filter("unicode_normalize", {"kind": "nfkc"})
    .append_character_filter("japanese_iteration_mark", {
        "normalize_kanji": True,
        "normalize_kana": True,
    })
    # Postprocessing
    .append_token_filter("japanese_base_form", {})
    .append_token_filter("japanese_stop_tags", {
        "tags": ["助詞", "助動詞", "記号"],
    })
    .append_token_filter("lowercase", {})
    .build()
)

tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Python supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-python with the train feature enabled (enabled by default):

maturin develop --features train

Training a Model

Use lindera.train() to train a CRF model from a seed lexicon and annotated corpus:

import lindera

lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
    max_threads=4,
)

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using lindera.export():

import lindera

lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

import lindera

# Step 1: Train the CRF model
lindera.train(
    seed="resources/training/seed.csv",
    corpus="resources/training/corpus.txt",
    char_def="resources/training/char.def",
    unk_def="resources/training/unk.def",
    feature_def="resources/training/feature.def",
    rewrite_def="resources/training/rewrite.def",
    output="/tmp/model.dat",
    lambda_=0.01,
    max_iter=100,
)

# Step 2: Export to dictionary source files
lindera.export(
    model="/tmp/model.dat",
    output="/tmp/dictionary_source",
    metadata="resources/training/metadata.json",
)

# Step 3: Build the dictionary from exported source files
metadata = lindera.Metadata.from_json_file("/tmp/dictionary_source/metadata.json")
lindera.build_dictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata)

# Step 4: Use the trained dictionary
tokenizer = (
    lindera.TokenizerBuilder()
    .set_dictionary("/tmp/dictionary")
    .set_mode("normal")
    .build()
)

tokens = tokenizer.tokenize("形態素解析のテスト")
for token in tokens:
    print(f"{token.surface}\t{','.join(token.details)}")

Lindera Node.js

Lindera Node.js provides Node.js bindings for the Lindera morphological analysis engine, built with NAPI-RS. It brings Lindera's high-performance tokenization capabilities to the Node.js ecosystem with support for Node.js 18 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary
• TypeScript support: Full type definitions included out of the box

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

Installing from npm

Pre-built packages will be available on npm:

npm install lindera-nodejs

[!NOTE] The npm package does not include dictionaries. See Obtaining Dictionaries below. For browser/WASM usage, see lindera-wasm.

Building from Source

Prerequisites

• Node.js 18 or later (LTS versions recommended)
• Rust toolchain -- Install via rustup
• NAPI-RS CLI -- CLI tool for building native Node.js addons in Rust

Install the NAPI-RS CLI globally:

npm install -g @napi-rs/cli

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build lindera-nodejs in development mode:

cd lindera-nodejs
npm install
npm run build

Or use the project Makefile:

make nodejs-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality. It is enabled by default:

npm run build -- --features train

Feature Flags

Multiple features can be combined:

npm run build -- --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

const dictionary = loadDictionary("embedded://ipadic");

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Node.js:

const lindera = require("lindera-nodejs");

console.log(lindera.version());

Or with ES modules:

import { version } from "lindera-nodejs";

console.log(version());

Quick Start

This guide shows how to tokenize text using lindera-nodejs.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

const { TokenizerBuilder } = require("lindera-nodejs");

const builder = new TokenizerBuilder();
builder.setMode("normal");
builder.setDictionary("/path/to/ipadic");
const tokenizer = builder.build();

const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("すもももももももものうち");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.getDetail(0)}`);
}

Accessing Token Properties

Each token exposes the following properties:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .build();

const tokens = tokenizer.tokenize("東京タワー");
for (const token of tokens) {
  console.log(`Surface: ${token.surface}`);
  console.log(`Byte range: ${token.byteStart}..${token.byteEnd}`);
  console.log(`Position: ${token.position}`);
  console.log(`Word ID: ${token.wordId}`);
  console.log(`Unknown: ${token.isUnknown}`);
  console.log(`Details: ${token.details}`);
  console.log();
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .build();

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  const surfaces = tokens.map((t) => t.surface);
  console.log(`Cost ${cost}: ${surfaces.join(" / ")}`);
}

TypeScript

Lindera Node.js includes TypeScript type definitions. All classes and functions are fully typed:

import { TokenizerBuilder, Token } from "lindera-nodejs";

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("/path/to/ipadic")
  .build();

const tokens: Token[] = tokenizer.tokenize("形態素解析");
for (const token of tokens) {
  console.log(`${token.surface}: ${token.details?.join(",")}`);
}

Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera-nodejs");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a JSON file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("config.json");

Configuration Methods

All setter methods return this for method chaining.

setMode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.setMode("normal");

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.setKeepWhitespace(true);

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?)

Creates a tokenizer directly from a loaded dictionary.

const { Tokenizer, loadDictionary } = require("lindera-nodejs");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: Array<{ tokens: Token[], cost: number }>

Token

Token represents a single morphological token.

Properties

Token Methods

getDetail(index)

Returns the detail string at the specified index, or null if the index is out of range.

const token = tokenizer.tokenize("東京")[0];
const pos = token.getDetail(0);      // e.g., "名詞"
const subpos = token.getDetail(1);   // e.g., "固有名詞"
const reading = token.getDetail(7);  // e.g., "トウキョウ"

Parameters:

Returns: string | null

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Dictionary Management

Lindera Node.js provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use loadDictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

const { loadDictionary } = require("lindera-nodejs");

const dictionary = loadDictionary("/path/to/ipadic");

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

const dictionary = loadDictionary("embedded://ipadic");

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

const { loadUserDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

Pass the user dictionary when building a tokenizer:

const { Tokenizer, loadDictionary, loadUserDictionary, Metadata } = require("lindera-nodejs");

const dictionary = loadDictionary("/path/to/ipadic");
const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary", metadata);

const tokenizer = new Tokenizer(dictionary, "normal", userDict);

Or via the builder:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("/path/to/ipadic")
  .setUserDictionary("/path/to/user_dictionary")
  .build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

const { buildDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata({ name: "custom", encoding: "UTF-8" });
buildDictionary("/path/to/input_dir", "/path/to/output_dir", metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

const { buildUserDictionary, Metadata } = require("lindera-nodejs");

const metadata = new Metadata();
buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir", metadata);

The metadata parameter is optional. When omitted, default metadata values are used:

buildUserDictionary("ipadic", "user_words.csv", "/path/to/output_dir");

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

const { Metadata } = require("lindera-nodejs");

// Default metadata
const metadata = new Metadata();

// Custom metadata
const metadata = new Metadata({
  name: "my_dictionary",
  encoding: "UTF-8",
  defaultWordCost: -10000,
});

Loading from JSON

const metadata = Metadata.fromJsonFile("metadata.json");

Properties

All properties support both getting and setting:

const metadata = new Metadata();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

toObject()

Returns a plain object representation of the metadata:

const metadata = new Metadata({ name: "test" });
console.log(metadata.toObject());

Schema

The Schema class defines the field structure of dictionary entries.

Creating a Schema

const { Schema } = require("lindera-nodejs");

// Default IPADIC-compatible schema
const schema = Schema.createDefault();

// Custom schema
const custom = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Methods

const schema = Schema.createDefault();

console.log(schema.fieldCount());           // 13 (IPADIC format)
console.log(schema.getFieldIndex("pos1"));  // e.g., 4
console.log(schema.getAllFields());          // ["surface", "left_id", ...]
console.log(schema.getCustomFields());      // Fields after index 4

FieldDefinition

FieldType

Text Processing Pipeline

Lindera Node.js supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("unicode_normalize", { kind: "nfkc" })
  .build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("mapping", {
    mapping: {
      "\u30fc": "-",
      "\uff5e": "~",
    },
  })
  .build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendCharacterFilter("japanese_iteration_mark", {
    normalize_kanji: true,
    normalize_kana: true,
  })
  .build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("lowercase", {})
  .build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_base_form", {})
  .build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_stop_tags", {
    tags: ["助詞", "助動詞"],
  })
  .build();

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

const tokenizer = new TokenizerBuilder()
  .setDictionary("embedded://ipadic")
  .appendTokenFilter("japanese_keep_tags", {
    tags: ["名詞"],
  })
  .build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

const { TokenizerBuilder } = require("lindera-nodejs");

const tokenizer = new TokenizerBuilder()
  .setMode("normal")
  .setDictionary("embedded://ipadic")
  // Preprocessing
  .appendCharacterFilter("unicode_normalize", { kind: "nfkc" })
  .appendCharacterFilter("japanese_iteration_mark", {
    normalize_kanji: true,
    normalize_kana: true,
  })
  // Postprocessing
  .appendTokenFilter("japanese_base_form", {})
  .appendTokenFilter("japanese_stop_tags", {
    tags: ["助詞", "助動詞", "記号"],
  })
  .appendTokenFilter("lowercase", {})
  .build();

const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Node.js supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-nodejs with the train feature enabled (enabled by default):

npm run build -- --features train

Training a Model

Use train() to train a CRF model from a seed lexicon and annotated corpus:

const { train } = require("lindera-nodejs");

train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
  maxThreads: 4,
});

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using exportModel():

const { exportModel } = require("lindera-nodejs");

exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

const {
  train,
  exportModel,
  buildDictionary,
  Metadata,
  TokenizerBuilder,
} = require("lindera-nodejs");

// Step 1: Train the CRF model
train({
  seed: "resources/training/seed.csv",
  corpus: "resources/training/corpus.txt",
  charDef: "resources/training/char.def",
  unkDef: "resources/training/unk.def",
  featureDef: "resources/training/feature.def",
  rewriteDef: "resources/training/rewrite.def",
  output: "/tmp/model.dat",
  lambda: 0.01,
  maxIter: 100,
});

// Step 2: Export to dictionary source files
exportModel({
  model: "/tmp/model.dat",
  output: "/tmp/dictionary_source",
  metadata: "resources/training/metadata.json",
});

// Step 3: Build the dictionary from exported source files
const metadata = Metadata.fromJsonFile("/tmp/dictionary_source/metadata.json");
buildDictionary("/tmp/dictionary_source", "/tmp/dictionary", metadata);

// Step 4: Use the trained dictionary
const tokenizer = new TokenizerBuilder()
  .setDictionary("/tmp/dictionary")
  .setMode("normal")
  .build();

const tokens = tokenizer.tokenize("形態素解析のテスト");
for (const token of tokens) {
  console.log(`${token.surface}\t${token.details.join(",")}`);
}

Lindera Ruby

Lindera Ruby provides Ruby bindings for the Lindera morphological analysis engine, built with Magnus and rb-sys. It brings Lindera's high-performance tokenization capabilities to the Ruby ecosystem with support for Ruby 3.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

[!NOTE] lindera-ruby is not yet published to RubyGems. You need to build from source.

Prerequisites

• Ruby 3.1 or later
• Rust toolchain -- Install via rustup
• Bundler -- Ruby dependency manager (gem install bundler)

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build and install lindera-ruby in development mode:

cd lindera-ruby
bundle install
bundle exec rake compile

Or use the project Makefile:

make ruby-develop

Build with Training Support

The train feature enables CRF-based dictionary training functionality:

LINDERA_FEATURES="train" bundle exec rake compile

Feature Flags

Features are specified through the LINDERA_FEATURES environment variable as a comma-separated list.

Multiple features can be combined:

LINDERA_FEATURES="train,embed-ipadic,embed-ko-dic" bundle exec rake compile

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

dictionary = Lindera.load_dictionary("embedded://ipadic")

See Feature Flags for details.

Verifying the Installation

After installation, verify that lindera is available in Ruby:

require 'lindera'

puts Lindera.version

Quick Start

This guide shows how to tokenize text using lindera-ruby.

Basic Tokenization

The recommended way to create a tokenizer is through Lindera::TokenizerBuilder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('関西国際空港限定トートバッグ')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Sequential Configuration

TokenizerBuilder is configured through sequential method calls:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('すもももももももものうち')
tokens.each do |token|
  puts "#{token.surface}\t#{token.get_detail(0)}"
end

Accessing Token Properties

Each token exposes the following properties:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

tokens = tokenizer.tokenize('東京タワー')

tokens.each do |token|
  puts "Surface: #{token.surface}"
  puts "Byte range: #{token.byte_start}..#{token.byte_end}"
  puts "Position: #{token.position}"
  puts "Word ID: #{token.word_id}"
  puts "Unknown: #{token.is_unknown}"
  puts "Details: #{token.details}"
  puts
end

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
tokenizer = builder.build

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)

results.each do |tokens, cost|
  surfaces = tokens.map(&:surface)
  puts "Cost #{cost}: #{surfaces.join(' / ')}"
end

Tokenizer API

TokenizerBuilder

Lindera::TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

Lindera::TokenizerBuilder.new

Creates a new builder with default configuration.

require 'lindera'

builder = Lindera::TokenizerBuilder.new

Lindera::TokenizerBuilder.new.from_file(file_path)

Loads configuration from a JSON file and returns a new builder.

builder = Lindera::TokenizerBuilder.new.from_file('config.json')

Configuration Methods

set_mode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.set_mode('normal')

set_dictionary(path)

Sets the system dictionary path or URI.

# Use an embedded dictionary
builder.set_dictionary('embedded://ipadic')

# Use an external dictionary
builder.set_dictionary('/path/to/dictionary')

set_user_dictionary(uri)

Sets the user dictionary URI.

builder.set_user_dictionary('/path/to/user_dictionary')

set_keep_whitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.set_keep_whitespace(true)

append_character_filter(kind, args)

Appends a character filter to the preprocessing pipeline. The args parameter is a hash with string keys.

builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })

append_token_filter(kind, args)

Appends a token filter to the postprocessing pipeline. The args parameter is a hash with string keys, or nil if the filter requires no arguments.

builder.append_token_filter('lowercase', nil)

Build

build

Builds and returns a Tokenizer with the configured settings.

tokenizer = builder.build

Tokenizer

Lindera::Tokenizer performs morphological analysis on text.

Creating a Tokenizer

Lindera::Tokenizer.new(dictionary, mode, user_dictionary)

Creates a tokenizer directly from a loaded dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('embedded://ipadic')
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', nil)

With a user dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)
tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of Token objects.

tokens = tokenizer.tokenize('形態素解析')

Parameters:

Returns: Array<Token>

tokenize_nbest(text, n, unique, cost_threshold)

Returns the N-best tokenization results, each paired with its total path cost.

results = tokenizer.tokenize_nbest('すもももももももものうち', 3, false, nil)
results.each do |tokens, cost|
  puts "#{cost}: #{tokens.map(&:surface).inspect}"
end

Parameters:

Returns: Array<Array(Array<Token>, Integer)>

Token

Token represents a single morphological token.

Properties

Token Methods

get_detail(index)

Returns the detail string at the specified index, or nil if the index is out of range.

token = tokenizer.tokenize('東京')[0]
pos = token.get_detail(0)        # e.g., "名詞"
subpos = token.get_detail(1)     # e.g., "固有名詞"
reading = token.get_detail(7)    # e.g., "トウキョウ"

Parameters:

Returns: String or nil

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Dictionary Management

Lindera Ruby provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera.load_dictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

dictionary = Lindera.load_dictionary('embedded://ipadic')

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

Pass the user dictionary when building a tokenizer:

require 'lindera'

dictionary = Lindera.load_dictionary('/path/to/ipadic')
metadata = dictionary.metadata
user_dict = Lindera.load_user_dictionary('/path/to/user_dictionary', metadata)

tokenizer = Lindera::Tokenizer.new(dictionary, 'normal', user_dict)

Or via the builder:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/path/to/ipadic')
builder.set_user_dictionary('/path/to/user_dictionary')
tokenizer = builder.build

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_dictionary('/path/to/input_dir', '/path/to/output_dir', metadata)

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

require 'lindera'

metadata = Lindera::Metadata.from_json_file('metadata.json')
Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', metadata)

The metadata parameter is optional. When omitted, default metadata values are used:

Lindera.build_user_dictionary('ipadic', 'user_words.csv', '/path/to/output_dir', nil)

Metadata

The Lindera::Metadata class configures dictionary parameters.

Creating Metadata

require 'lindera'

# Default metadata
metadata = Lindera::Metadata.new

# Create default metadata with standard settings
metadata = Lindera::Metadata.create_default

Loading from JSON

metadata = Lindera::Metadata.from_json_file('metadata.json')

Properties

Text Processing Pipeline

Lindera Ruby supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
tokenizer = builder.build

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('mapping', {
  'mapping' => {
    "\u30fc" => '-',
    "\uff5e" => '~'
  }
})
tokenizer = builder.build

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => 'true',
  'normalize_kana' => 'true'
})
tokenizer = builder.build

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('lowercase', nil)
tokenizer = builder.build

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_base_form', nil)
tokenizer = builder.build

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞', '助動詞']
})
tokenizer = builder.build

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_keep_tags', {
  'tags' => ['名詞']
})
tokenizer = builder.build

japanese_katakana_stem

Removes trailing prolonged sound marks from katakana tokens that exceed a minimum length.

builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('embedded://ipadic')
builder.append_token_filter('japanese_katakana_stem', { 'min' => 3 })
tokenizer = builder.build

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

require 'lindera'

builder = Lindera::TokenizerBuilder.new
builder.set_mode('normal')
builder.set_dictionary('embedded://ipadic')

# Preprocessing
builder.append_character_filter('unicode_normalize', { 'kind' => 'nfkc' })
builder.append_character_filter('japanese_iteration_mark', {
  'normalize_kanji' => 'true',
  'normalize_kana' => 'true'
})

# Postprocessing
builder.append_token_filter('japanese_base_form', nil)
builder.append_token_filter('japanese_stop_tags', {
  'tags' => ['助詞', '助動詞', '記号']
})
builder.append_token_filter('lowercase', nil)

tokenizer = builder.build

tokens = tokenizer.tokenize('Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera Ruby supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-ruby with the train feature enabled:

LINDERA_FEATURES="embed-ipadic,train" bundle exec rake compile

Training a Model

Use Lindera::Trainer.train to train a CRF model from a seed lexicon and annotated corpus:

require 'lindera'

Lindera::Trainer.train(
  'resources/training/seed.csv',
  'resources/training/corpus.txt',
  'resources/training/char.def',
  'resources/training/unk.def',
  'resources/training/feature.def',
  'resources/training/rewrite.def',
  '/tmp/model.dat',
  0.01,  # lambda (L1 regularization)
  100,   # max_iter
  nil    # max_threads (nil = auto-detect CPU cores)
)

Training Parameters

Parameters are passed as positional arguments in the following order:

Exporting a Trained Model

After training, export the model to dictionary source files using Lindera::Trainer.export:

require 'lindera'

Lindera::Trainer.export(
  '/tmp/model.dat',
  '/tmp/dictionary_source',
  'resources/training/metadata.json'
)

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

require 'lindera'

# Step 1: Train the CRF model
Lindera::Trainer.train(
  'resources/training/seed.csv',
  'resources/training/corpus.txt',
  'resources/training/char.def',
  'resources/training/unk.def',
  'resources/training/feature.def',
  'resources/training/rewrite.def',
  '/tmp/model.dat',
  0.01,  # lambda
  100,   # max_iter
  nil    # max_threads
)

# Step 2: Export to dictionary source files
Lindera::Trainer.export(
  '/tmp/model.dat',
  '/tmp/dictionary_source',
  'resources/training/metadata.json'
)

# Step 3: Build the dictionary from exported source files
metadata = Lindera::Metadata.from_json_file('/tmp/dictionary_source/metadata.json')
Lindera.build_dictionary('/tmp/dictionary_source', '/tmp/dictionary', metadata)

# Step 4: Use the trained dictionary
builder = Lindera::TokenizerBuilder.new
builder.set_dictionary('/tmp/dictionary')
builder.set_mode('normal')
tokenizer = builder.build

tokens = tokenizer.tokenize('形態素解析のテスト')
tokens.each do |token|
  puts "#{token.surface}\t#{token.details.join(',')}"
end

Lindera PHP

Lindera PHP provides PHP bindings for the Lindera morphological analysis engine, built with ext-php-rs. It brings Lindera's high-performance tokenization capabilities to the PHP ecosystem with support for PHP 8.1 and later.

Features

• Multi-language support: Tokenize Japanese (IPADIC, IPADIC NEologd, UniDic), Korean (ko-dic), and Chinese (CC-CEDICT, Jieba) text
• Text processing pipeline: Compose character filters and token filters for flexible preprocessing and postprocessing
• CRF-based dictionary training: Train custom morphological analysis models from annotated corpora (requires train feature)
• Multiple tokenization modes: Normal and decompose modes for different analysis granularity
• N-best tokenization: Retrieve multiple tokenization candidates ranked by cost
• User dictionaries: Extend system dictionaries with custom vocabulary

Documentation

• Installation -- Prerequisites, build instructions, and feature flags
• Quick Start -- A minimal example to get started
• Tokenizer API -- TokenizerBuilder, Tokenizer, and Token class reference
• Dictionary Management -- Loading, building, and managing dictionaries
• Text Processing Pipeline -- Character filters and token filters
• Training -- Training custom CRF models and exporting dictionaries

Installation

[!NOTE] lindera-php is not yet published to Packagist. You need to build from source.

Prerequisites

• PHP 8.1 or later
• Rust toolchain -- Install via rustup
• Composer -- PHP dependency manager (optional, for running tests)

Obtaining Dictionaries

Lindera does not bundle dictionaries with the package. You need to obtain a pre-built dictionary separately.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. Download and extract the dictionary archive to a local directory:

# Example: download and extract the IPADIC dictionary
curl -LO https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip
unzip lindera-ipadic-<version>.zip -d /path/to/ipadic

Development Build

Build the lindera-php extension from the project root:

cargo build -p lindera-php

Or use the project Makefile:

make php-build

Build with Training Support

The train feature enables CRF-based dictionary training functionality:

cargo build -p lindera-php --features train

Feature Flags

Multiple features can be combined:

cargo build -p lindera-php --features "train,embed-ipadic,embed-ko-dic"

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

$dictionary = Lindera\Dictionary::load('embedded://ipadic');

See Feature Flags for details.

Loading the Extension

Load the compiled shared library when running PHP:

php -d extension=target/debug/liblindera_php.so script.php

For release builds:

cargo build -p lindera-php --release
php -d extension=target/release/liblindera_php.so script.php

Alternatively, add the extension to your php.ini:

extension=/absolute/path/to/liblindera_php.so

Verifying the Installation

After building, verify that lindera is available in PHP:

php -d extension=target/debug/liblindera_php.so -r "echo Lindera\Dictionary::version() . PHP_EOL;"

Quick Start

This guide shows how to tokenize text using lindera-php.

Basic Tokenization

The recommended way to create a tokenizer is through TokenizerBuilder:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setMode('normal');
$builder->setDictionary('/path/to/ipadic');
$tokenizer = $builder->build();

$tokens = $tokenizer->tokenize('関西国際空港限定トートバッグ');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

Note: Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory.

Expected output:

関西国際空港    名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    UNK

Method Chaining

TokenizerBuilder supports method chaining for concise configuration:

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setMode('normal')
    ->setDictionary('/path/to/ipadic')
    ->build();

$tokens = $tokenizer->tokenize('すもももももももものうち');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . $token->getDetail(0) . "\n";
}

Accessing Token Properties

Each token exposes the following properties:

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder->setDictionary('/path/to/ipadic')->build();
$tokens = $tokenizer->tokenize('東京タワー');

foreach ($tokens as $token) {
    echo "Surface: {$token->surface}\n";
    echo "Byte range: {$token->byte_start}..{$token->byte_end}\n";
    echo "Position: {$token->position}\n";
    echo "Word ID: {$token->word_id}\n";
    echo "Unknown: " . ($token->is_unknown ? 'true' : 'false') . "\n";
    echo "Details: " . implode(',', $token->details) . "\n";
    echo "\n";
}

N-best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder->setDictionary('/path/to/ipadic')->build();
$results = $tokenizer->tokenizeNbest('すもももももももものうち', 3);

foreach ($results as $result) {
    $surfaces = array_map(fn($t) => $t->surface, $result->tokens);
    echo "Cost {$result->cost}: " . implode(' / ', $surfaces) . "\n";
}

Tokenizer API

TokenizerBuilder

Lindera\TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new Lindera\TokenizerBuilder()

Creates a new builder with default configuration.

<?php

$builder = new Lindera\TokenizerBuilder();

$builder->fromFile($filePath)

Loads configuration from a JSON file.

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->fromFile('config.json');

Configuration Methods

All setter methods return $this for method chaining.

setMode($mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

<?php

$builder->setMode('normal');

setDictionary($path)

Sets the system dictionary path or URI.

<?php

// Use an embedded dictionary
$builder->setDictionary('embedded://ipadic');

// Use an external dictionary
$builder->setDictionary('/path/to/dictionary');

setUserDictionary($uri)

Sets the user dictionary URI.

<?php

$builder->setUserDictionary('/path/to/user_dictionary');

setKeepWhitespace($keep)

Controls whether whitespace tokens appear in the output.

<?php

$builder->setKeepWhitespace(true);

appendCharacterFilter($kind, $args)

Appends a character filter to the preprocessing pipeline.

<?php

$builder->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc']);

appendTokenFilter($kind, $args)

Appends a token filter to the postprocessing pipeline.

<?php

$builder->appendTokenFilter('lowercase');

Build

build()

Builds and returns a Tokenizer with the configured settings.

<?php

$tokenizer = $builder->build();

Tokenizer

Lindera\Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Lindera\Tokenizer($dictionary, $mode, $userDictionary)

Creates a tokenizer directly from a loaded dictionary.

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');
$tokenizer = new Lindera\Tokenizer($dictionary, 'normal');

With a user dictionary:

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary', $metadata);
$tokenizer = new Lindera\Tokenizer($dictionary, 'normal', $userDict);

Tokenizer Methods

tokenize($text)

Tokenizes the input text and returns an array of Token objects.

<?php

$tokens = $tokenizer->tokenize('形態素解析');

Parameters:

Returns: array<Token>

tokenizeNbest($text, $n, $unique, $costThreshold)

Returns the N-best tokenization results as an array of NbestResult objects.

<?php

$results = $tokenizer->tokenizeNbest('すもももももももものうち', 3);
foreach ($results as $result) {
    echo "Cost: {$result->cost}\n";
    foreach ($result->tokens as $token) {
        echo "  {$token->surface}\n";
    }
}

Parameters:

Returns: array<NbestResult>

NbestResult

Lindera\NbestResult represents a single N-best tokenization result.

NbestResult Properties

Token

Lindera\Token represents a single morphological token.

Token Properties

Token Methods

getDetail($index)

Returns the detail string at the specified index, or null if the index is out of range.

<?php

$token = $tokenizer->tokenize('東京')[0];
$pos = $token->getDetail(0);        // e.g., "名詞"
$subpos = $token->getDetail(1);     // e.g., "固有名詞"
$reading = $token->getDetail(7);    // e.g., "トウキョウ"

Parameters:

Returns: string|null

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Dictionary Management

Lindera PHP provides static methods on the Lindera\Dictionary class for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera\Dictionary::load($uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary.

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary', $metadata);

Pass the user dictionary when creating a tokenizer directly:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary', $metadata);

$tokenizer = new Lindera\Tokenizer($dictionary, 'normal', $userDict);

Or via the builder:

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('/path/to/ipadic')
    ->setUserDictionary('/path/to/user_dictionary')
    ->build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

<?php

$metadata = Lindera\Metadata::fromJsonFile('/path/to/metadata.json');
Lindera\Dictionary::build('/path/to/input_dir', '/path/to/output_dir', $metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

User Dictionary

Build a user dictionary from a CSV file:

<?php

$metadata = new Lindera\Metadata();
Lindera\Dictionary::buildUser('ipadic', 'user_words.csv', '/path/to/output_dir', $metadata);

Metadata

The Lindera\Metadata class configures dictionary parameters.

Creating Metadata

<?php

// Default metadata
$metadata = new Lindera\Metadata();

// Custom metadata
$metadata = new Lindera\Metadata(
    name: 'my_dictionary',
    encoding: 'UTF-8',
    default_word_cost: -10000,
);

// Create with all defaults explicitly
$metadata = Lindera\Metadata::createDefault();

Loading from JSON

<?php

$metadata = Lindera\Metadata::fromJsonFile('metadata.json');

Properties

All properties are read-only via getter methods:

<?php

$metadata = new Lindera\Metadata(name: 'custom_dict', encoding: 'EUC-JP');
echo $metadata->name;      // "custom_dict"
echo $metadata->encoding;  // "EUC-JP"

toArray()

Returns an associative array representation of the metadata:

<?php

$metadata = new Lindera\Metadata(name: 'test');
print_r($metadata->toArray());

Dictionary Info

The Lindera\Dictionary object provides metadata accessors:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
echo $dictionary->metadataName();      // Dictionary name
echo $dictionary->metadataEncoding();  // Dictionary encoding
$metadata = $dictionary->metadata();   // Full Metadata object

Version

Retrieve the Lindera library version:

<?php

echo Lindera\Dictionary::version();

Text Processing Pipeline

Lindera PHP supports a composable text processing pipeline that applies character filters before tokenization and token filters after tokenization. Filters are added to the TokenizerBuilder and executed in the order they are appended.

Input Text
  --> Character Filters (preprocessing)
  --> Tokenization
  --> Token Filters (postprocessing)
  --> Output Tokens

Character Filters

Character filters transform the input text before tokenization.

unicode_normalize

Applies Unicode normalization to the input text.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc'])
    ->build();

Supported normalization forms: "nfc", "nfkc", "nfd", "nfkd".

mapping

Replaces characters or strings according to a mapping table.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendCharacterFilter('mapping', [
        'mapping' => [
            "\u{30FC}" => '-',
            "\u{FF5E}" => '~',
        ],
    ])
    ->build();

japanese_iteration_mark

Resolves Japanese iteration marks (odoriji) into their full forms.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendCharacterFilter('japanese_iteration_mark', [
        'normalize_kanji' => 'true',
        'normalize_kana' => 'true',
    ])
    ->build();

Token Filters

Token filters transform or remove tokens after tokenization.

lowercase

Converts token surface forms to lowercase.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendTokenFilter('lowercase')
    ->build();

japanese_base_form

Replaces inflected forms with their base (dictionary) form using the morphological details from the dictionary.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendTokenFilter('japanese_base_form', [])
    ->build();

japanese_stop_tags

Removes tokens whose part-of-speech matches any of the specified tags.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendTokenFilter('japanese_stop_tags', [
        'tags' => ['助詞', '助動詞'],
    ])
    ->build();

japanese_keep_tags

Keeps only tokens whose part-of-speech matches one of the specified tags. All other tokens are removed.

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('embedded://ipadic')
    ->appendTokenFilter('japanese_keep_tags', [
        'tags' => ['名詞'],
    ])
    ->build();

Complete Pipeline Example

The following example combines multiple character filters and token filters into a single pipeline:

<?php

$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setMode('normal')
    ->setDictionary('embedded://ipadic')
    // Preprocessing
    ->appendCharacterFilter('unicode_normalize', ['kind' => 'nfkc'])
    ->appendCharacterFilter('japanese_iteration_mark', [
        'normalize_kanji' => 'true',
        'normalize_kana' => 'true',
    ])
    // Postprocessing
    ->appendTokenFilter('japanese_base_form', [])
    ->appendTokenFilter('japanese_stop_tags', [
        'tags' => ['助詞', '助動詞', '記号'],
    ])
    ->appendTokenFilter('lowercase')
    ->build();

$tokens = $tokenizer->tokenize('Ｌｉｎｄｅｒａは形態素解析を行うライブラリです。');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

In this pipeline:

1. unicode_normalize converts full-width characters to half-width (NFKC normalization)
2. japanese_iteration_mark resolves iteration marks
3. japanese_base_form converts inflected tokens to base form
4. japanese_stop_tags removes particles, auxiliary verbs, and symbols
5. lowercase normalizes alphabetic characters to lowercase

Training

Lindera PHP supports training custom CRF-based morphological analysis models from annotated corpora. This functionality requires the train feature.

Prerequisites

Build lindera-php with the train feature enabled:

cargo build -p lindera-php --features train,embed-ipadic

Training a Model

Use Lindera\Trainer::train() to train a CRF model from a seed lexicon and annotated corpus:

<?php

Lindera\Trainer::train(
    seed: 'resources/training/seed.csv',
    corpus: 'resources/training/corpus.txt',
    char_def: 'resources/training/char.def',
    unk_def: 'resources/training/unk.def',
    feature_def: 'resources/training/feature.def',
    rewrite_def: 'resources/training/rewrite.def',
    output: '/tmp/model.dat',
    lambda: 0.01,
    max_iter: 100,
    max_threads: null,
);

Training Parameters

Exporting a Trained Model

After training, export the model to dictionary source files using Lindera\Trainer::export():

<?php

Lindera\Trainer::export(
    model: '/tmp/model.dat',
    output: '/tmp/dictionary_source',
    metadata: 'resources/training/metadata.json',
);

Export Parameters

The export creates the following files in the output directory:

• lex.csv -- Lexicon entries with trained costs
• matrix.def -- Connection cost matrix
• unk.def -- Unknown word definitions
• char.def -- Character category definitions
• metadata.json -- Updated metadata (when $metadata parameter is provided)

Complete Workflow

The full workflow for training and using a custom dictionary:

<?php

// Step 1: Train the CRF model
Lindera\Trainer::train(
    seed: 'resources/training/seed.csv',
    corpus: 'resources/training/corpus.txt',
    char_def: 'resources/training/char.def',
    unk_def: 'resources/training/unk.def',
    feature_def: 'resources/training/feature.def',
    rewrite_def: 'resources/training/rewrite.def',
    output: '/tmp/model.dat',
    lambda: 0.01,
    max_iter: 100,
);

// Step 2: Export to dictionary source files
Lindera\Trainer::export(
    model: '/tmp/model.dat',
    output: '/tmp/dictionary_source',
    metadata: 'resources/training/metadata.json',
);

// Step 3: Build the dictionary from exported source files
$metadata = Lindera\Metadata::fromJsonFile('/tmp/dictionary_source/metadata.json');
Lindera\Dictionary::build('/tmp/dictionary_source', '/tmp/dictionary', $metadata);

// Step 4: Use the trained dictionary
$builder = new Lindera\TokenizerBuilder();
$tokenizer = $builder
    ->setDictionary('/tmp/dictionary')
    ->setMode('normal')
    ->build();

$tokens = $tokenizer->tokenize('形態素解析のテスト');
foreach ($tokens as $token) {
    echo $token->surface . "\t" . implode(',', $token->details) . "\n";
}

Lindera WASM

Lindera WASM provides WebAssembly bindings for Lindera's morphological analysis engine, built with wasm-bindgen. It enables Japanese, Korean, and Chinese text tokenization directly in web browsers, Node.js, and bundler environments.

Distribution Formats

Lindera WASM supports multiple distribution formats via wasm-pack:

Dictionary Packages

Each package embeds a specific dictionary for offline use:

Sections

• Installation -- Building and installing lindera-wasm packages
• Quick Start -- Minimal working example
• Tokenizer API -- Full API reference for JavaScript/TypeScript
• Dictionary Management -- Loading and building dictionaries
• Browser Usage -- Integration with web applications
• OPFS Dictionary Storage -- Persistent dictionary caching with OPFS

Installation

Prerequisites

• Rust (stable toolchain)
• wasm-pack (v0.10+)

Obtaining Dictionaries

Lindera WASM does not bundle dictionaries by default. The recommended approach for browser environments is to download dictionaries at runtime using the OPFS (Origin Private File System) API.

Download from GitHub Releases

Pre-built dictionaries are available on the GitHub Releases page. In browser environments, use the OPFS helpers to download and cache dictionaries:

import { downloadDictionary, hasDictionary } from 'lindera-wasm-web/opfs';

if (!await hasDictionary("ipadic")) {
    await downloadDictionary(
        "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
        "ipadic",
    );
}

See OPFS Dictionary Storage for the full workflow.

Building with wasm-pack

Build the WASM package for your target environment:

Web (ES Modules for browsers)

wasm-pack build --target web

Bundler (Webpack, Vite, Rollup)

wasm-pack build --target bundler

The output is written to the pkg/ directory inside the lindera-wasm crate.

Available Feature Flags (Advanced)

For advanced users who want to embed dictionaries directly into the WASM binary, the following feature flags are available. This increases the binary size significantly but eliminates the need to download dictionaries at runtime.

You can combine multiple dictionaries by enabling multiple feature flags:

wasm-pack build --target web --features embed-ipadic,embed-ko-dic

NPM Package Naming Convention

When publishing to npm, the recommended naming convention is:

lindera-wasm-{target}
lindera-wasm-{target}-{dict}

Examples:

• lindera-wasm-web
• lindera-wasm-web-ipadic
• lindera-wasm-bundler-unidic
• lindera-wasm-web-cjk

To set the package name before publishing, edit the name field in the generated pkg/package.json.

Installing from npm

Pre-built packages are available on npm:

npm install lindera-wasm-web

Or with yarn:

yarn add lindera-wasm-web

[!NOTE] The npm package does not include dictionaries. Use the OPFS helpers to download dictionaries at runtime. See OPFS Dictionary Storage.

Quick Start

Web (Browser) -- OPFS Dictionary Loading

The recommended approach is to download dictionaries at runtime using the OPFS helpers:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { downloadDictionary, loadDictionaryFiles, hasDictionary } from 'lindera-wasm-web/opfs';

async function main() {
    await __wbg_init();

    // Download dictionary if not cached
    if (!await hasDictionary("ipadic")) {
        await downloadDictionary(
            "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
            "ipadic",
        );
    }

    // Load dictionary from OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    // Build tokenizer
    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

Note: Download a pre-built dictionary from GitHub Releases. See OPFS Dictionary Storage for the full workflow.

Expected output:

関西国際空港    名詞,固有名詞,一般,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
限定    名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
トートバッグ    名詞,一般,*,*,*,*,*,*,*

Using Embedded Dictionaries (Advanced)

If you built with an embed-* feature flag, you can use embedded dictionaries:

import __wbg_init, { TokenizerBuilder } from 'lindera-wasm-web-ipadic';

async function main() {
    await __wbg_init();

    const builder = new TokenizerBuilder();
    builder.setDictionary("embedded://ipadic");
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

Using Filters

You can add character filters and token filters to the tokenization pipeline:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

async function main() {
    await __wbg_init();

    // Assume dictionary is already cached in OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");

    // Add Unicode NFKC normalization
    builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

    // Add a stop-tags filter to remove particles and auxiliary verbs
    builder.appendTokenFilter("japanese_stop_tags", {
        tags: ["助詞", "助動詞"]
    });

    const tokenizer = builder.build();
    const tokens = tokenizer.tokenize("Ｌｉｎｄｅｒａは形態素解析エンジンです");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

N-Best Tokenization

Retrieve multiple tokenization candidates ranked by cost:

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
results.forEach((result, rank) => {
    console.log(`--- NBEST ${rank + 1} (cost=${result.cost}) ---`);
    result.tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
});

Tokenizer API

This page documents the JavaScript/TypeScript API exposed by lindera-wasm.

TokenizerBuilder

Builder class for creating a configured Tokenizer instance.

Constructor

const builder = new TokenizerBuilder();

Creates a new builder with default settings.

Methods

setMode(mode)

Sets the tokenization mode.

• Parameters: mode (string) -- "normal" or "decompose"
• Returns: void

builder.setMode("normal");

setDictionary(uri)

Sets the dictionary to use for tokenization.

• Parameters: uri (string) -- Dictionary URI (e.g., "embedded://ipadic")
• Returns: void

builder.setDictionary("embedded://ipadic");

setDictionaryInstance(dictionary)

Sets a pre-loaded dictionary instance for tokenization. Use this when the dictionary has been loaded from bytes (e.g., via loadDictionaryFromBytes()) instead of from a URI.

• Parameters: dictionary (Dictionary) -- A loaded dictionary object
• Returns: void

import { loadDictionaryFromBytes } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

const files = await loadDictionaryFiles("ipadic");
const dictionary = loadDictionaryFromBytes(
    files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
    files.dictWords, files.matrixMtx, files.charDef, files.unk,
);

builder.setDictionaryInstance(dictionary);

setUserDictionary(uri)

Sets a user-defined dictionary by URI.

• Parameters: uri (string) -- Path or URI to the user dictionary
• Returns: void

builder.setUserDictionary("file:///path/to/user_dict.csv");

setUserDictionaryInstance(userDictionary)

Sets a pre-loaded user dictionary instance. Use this when the user dictionary has been loaded from bytes instead of from a URI.

• Parameters: userDictionary (UserDictionary) -- A loaded user dictionary object
• Returns: void

setKeepWhitespace(keep)

Sets whether whitespace tokens are preserved in the output.

• Parameters: keep (boolean) -- true to keep whitespace tokens
• Returns: void

builder.setKeepWhitespace(true);

appendCharacterFilter(name, args)

Appends a character filter to the preprocessing pipeline.

• Parameters:
  • name (string) -- Filter name (e.g., "unicode_normalize", "japanese_iteration_mark")
  • args (object, optional) -- Filter configuration
• Returns: void

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(name, args)

Appends a token filter to the postprocessing pipeline.

• Parameters:
  • name (string) -- Filter name (e.g., "japanese_stop_tags", "lowercase")
  • args (object, optional) -- Filter configuration
• Returns: void

builder.appendTokenFilter("japanese_stop_tags", {
    tags: ["助詞", "助動詞", "記号"]
});

build()

Builds and returns a configured Tokenizer instance. Consumes the builder.

• Returns: Tokenizer

const tokenizer = builder.build();

Tokenizer

The main tokenizer class. Can be created via TokenizerBuilder.build() or directly via the constructor.

Tokenizer Constructor

const tokenizer = new Tokenizer(dictionary, mode, userDictionary);

• Parameters:
  • dictionary (Dictionary) -- A loaded dictionary object
  • mode (string, optional) -- Tokenization mode ("normal" or "decompose", defaults to "normal")
  • userDictionary (UserDictionary, optional) -- A loaded user dictionary

Tokenizer Methods

tokenize(text)

Tokenizes the input text.

• Parameters: text (string) -- Text to tokenize
• Returns: Token[] -- Array of token objects

const tokens = tokenizer.tokenize("関西国際空港");

tokenizeNbest(text, n, unique?, costThreshold?)

Returns N-best tokenization results ordered by total path cost.

• Parameters:
  • text (string) -- Text to tokenize
  • n (number) -- Number of results to return
  • unique (boolean, optional) -- Deduplicate results with identical segmentation (default: false)
  • costThreshold (number, optional) -- Only return paths within bestCost + threshold
• Returns: Array of { tokens: object[], cost: number }

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);

Token

Represents a single token produced by the tokenizer.

Properties

Token Methods

getDetail(index)

Returns the detail string at the specified index.

• Parameters: index (number) -- Zero-based index into the details array
• Returns: string | undefined

const pos = token.getDetail(0);   // e.g., "名詞"
const reading = token.getDetail(7); // e.g., "トウキョウ"

toJSON()

Returns a plain JavaScript object representation of the token.

• Returns: object with keys: surface, byteStart, byteEnd, position, wordId, isUnknown, details

console.log(JSON.stringify(token.toJSON(), null, 2));

Helper Functions

loadDictionary(uri)

Loads a dictionary from the specified URI.

• Parameters: uri (string) -- Dictionary URI (e.g., "embedded://ipadic")
• Returns: Dictionary

import { loadDictionary } from 'lindera-wasm-web-ipadic';

const dict = loadDictionary("embedded://ipadic");

loadUserDictionary(uri, metadata)

Loads a user dictionary from the specified URI.

• Parameters:
  • uri (string) -- Path or URI to the user dictionary file
  • metadata (Metadata) -- Dictionary metadata object
• Returns: UserDictionary

buildDictionary(inputDir, outputDir, metadata)

Builds a compiled dictionary from source files.

• Parameters:
  • inputDir (string) -- Path to the directory containing source dictionary files
  • outputDir (string) -- Path to the output directory
  • metadata (Metadata) -- Dictionary metadata object
• Returns: void

buildUserDictionary(inputFile, outputDir, metadata?)

Builds a compiled user dictionary from a CSV file.

• Parameters:
  • inputFile (string) -- Path to the user dictionary CSV file
  • outputDir (string) -- Path to the output directory
  • metadata (Metadata, optional) -- Dictionary metadata object
• Returns: void

version() / getVersion()

Returns the version string of the lindera-wasm package.

• Returns: string

import { version } from 'lindera-wasm-web-ipadic';

console.log(version()); // e.g., "2.1.1"

Enums and Utility Classes

Mode

Tokenization mode enum.

Penalty

Configuration for decompose mode. Controls how aggressively compound words are decomposed.

const penalty = new Penalty(
    kanjiThreshold?,     // Kanji length threshold (default: 2)
    kanjiPenalty?,       // Kanji length penalty (default: 3000)
    otherThreshold?,     // Other character length threshold (default: 7)
    otherPenalty?,       // Other character length penalty (default: 1700)
);

LinderaError

Error type for Lindera operations.

const error = new LinderaError("message");
console.log(error.message);    // "message"
console.log(error.toString()); // "message"

Snake-Case Aliases

For consistency with the Python API, all methods are also available in snake_case form:

Dictionary Management

Loading Dictionaries from OPFS

The recommended way to use dictionaries in WASM is to download them from GitHub Releases and load them via OPFS. This avoids embedding large dictionaries in the WASM binary.

Loading from Bytes

Use loadDictionaryFromBytes() to construct a Dictionary from raw byte arrays stored in OPFS or other browser storage.

loadDictionaryFromBytes(metadata, dictDa, dictVals, dictWordsIdx, dictWords, matrixMtx, charDef, unk)

• Parameters:
  • metadata (Uint8Array) -- Contents of metadata.json
  • dictDa (Uint8Array) -- Contents of dict.da (Double-Array Trie)
  • dictVals (Uint8Array) -- Contents of dict.vals (word value data)
  • dictWordsIdx (Uint8Array) -- Contents of dict.wordsidx (word details index)
  • dictWords (Uint8Array) -- Contents of dict.words (word details)
  • matrixMtx (Uint8Array) -- Contents of matrix.mtx (connection cost matrix)
  • charDef (Uint8Array) -- Contents of char_def.bin (character definitions)
  • unk (Uint8Array) -- Contents of unk.bin (unknown word dictionary)
• Returns: Dictionary

import { loadDictionaryFromBytes, TokenizerBuilder } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

// Load dictionary files from OPFS
const files = await loadDictionaryFiles("ipadic");

// Create a Dictionary from bytes
const dictionary = loadDictionaryFromBytes(
    files.metadata,
    files.dictDa,
    files.dictVals,
    files.dictWordsIdx,
    files.dictWords,
    files.matrixMtx,
    files.charDef,
    files.unk,
);

// Use with TokenizerBuilder
const builder = new TokenizerBuilder();
builder.setDictionaryInstance(dictionary);
builder.setMode("normal");
const tokenizer = builder.build();

See OPFS Dictionary Storage for the full OPFS workflow including downloading and caching.

Embedded Dictionaries (Advanced)

If you built with an embed-* feature flag, you can load embedded dictionaries via the embedded:// URI scheme. This increases the WASM binary size significantly.

Loading an Embedded Dictionary

import { loadDictionary } from 'lindera-wasm-web-ipadic';

const dictionary = loadDictionary("embedded://ipadic");

Available embedded dictionary URIs (depending on which features were enabled at build time):

Using with TokenizerBuilder

const builder = new TokenizerBuilder();
builder.setDictionary("embedded://ipadic");
builder.setMode("normal");
const tokenizer = builder.build();

Using with Tokenizer Constructor

import { loadDictionary, Tokenizer } from 'lindera-wasm-web-ipadic';

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

Dictionary Class

The Dictionary class represents a loaded morphological analysis dictionary.

Properties

console.log(dictionary.name);     // "ipadic"
console.log(dictionary.encoding); // "utf-8"

User Dictionaries

User dictionaries allow you to add custom words that are not in the system dictionary.

Loading a User Dictionary

import { loadUserDictionary } from 'lindera-wasm-web';

const metadata = dictionary.metadata;
const userDict = loadUserDictionary("/path/to/user_dict.csv", metadata);

Using a User Dictionary with Tokenizer

import { loadDictionaryFromBytes, loadUserDictionary, Tokenizer } from 'lindera-wasm-web';
import { loadDictionaryFiles } from 'lindera-wasm-web/opfs';

const files = await loadDictionaryFiles("ipadic");
const dictionary = loadDictionaryFromBytes(
    files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
    files.dictWords, files.matrixMtx, files.charDef, files.unk,
);
const userDict = loadUserDictionary("/path/to/user_dict.csv", dictionary.metadata);
const tokenizer = new Tokenizer(dictionary, "normal", userDict);

User Dictionary CSV Format

The user dictionary CSV follows the same format as the Lindera user dictionary:

東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン

Each line contains: surface,part_of_speech,reading

Building Dictionaries

You can build compiled dictionaries from source files using the JavaScript API.

Building a System Dictionary

import { buildDictionary } from 'lindera-wasm-web';

const metadata = {
    name: "custom-dict",
    encoding: "utf-8",
    // ... other metadata fields
};

buildDictionary("/path/to/source/dir", "/path/to/output/dir", metadata);

Building a User Dictionary

import { buildUserDictionary } from 'lindera-wasm-web';

buildUserDictionary("/path/to/user_dict.csv", "/path/to/output/dir");

The metadata parameter is optional for buildUserDictionary. If omitted, default metadata is used.

Metadata

The Metadata class configures dictionary parameters.

Constructor

const metadata = new Metadata(name?, encoding?);

• Parameters:
  • name (string, optional) -- Dictionary name (default: "default")
  • encoding (string, optional) -- Character encoding (default: "UTF-8")

Static Methods

Metadata.createDefault()

Creates a Metadata instance with default values.

const metadata = Metadata.createDefault();

Metadata Properties

All properties support both getting and setting:

const metadata = Metadata.createDefault();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

You can also access the metadata from a loaded dictionary via dictionary.metadata.

Schema

The Schema class defines the field structure of dictionary entries.

Schema Constructor

const schema = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Static Methods

• Schema.create_default() -- Creates a default IPADIC-like schema

Schema Methods

FieldDefinition

FieldType

Browser Usage

ES Module Import

In browser environments, you must initialize the WASM module before using any Lindera functions. The default export __wbg_init handles this initialization.

The recommended approach is to load dictionaries from OPFS rather than embedding them in the WASM binary:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { downloadDictionary, loadDictionaryFiles, hasDictionary } from 'lindera-wasm-web/opfs';

async function main() {
    // Initialize the WASM module (must be called once before using any API)
    await __wbg_init();

    // Download dictionary if not cached
    if (!await hasDictionary("ipadic")) {
        await downloadDictionary(
            "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
            "ipadic",
        );
    }

    // Load dictionary from OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("形態素解析を行います");
    tokens.forEach(token => {
        console.log(`${token.surface}: ${token.details.join(',')}`);
    });
}

main();

Using Embedded Dictionaries (Advanced)

If you built with an embed-* feature flag, you can use embedded dictionaries instead of OPFS:

import __wbg_init, { TokenizerBuilder } from 'lindera-wasm-web-ipadic';

async function main() {
    await __wbg_init();

    const builder = new TokenizerBuilder();
    builder.setDictionary("embedded://ipadic");
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("形態素解析を行います");
    tokens.forEach(token => {
        console.log(`${token.surface}: ${token.details.join(',')}`);
    });
}

main();

HTML Example

A minimal HTML page using lindera-wasm with OPFS dictionary loading:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Lindera WASM Demo</title>
</head>
<body>
    <textarea id="input" rows="4" cols="50">関西国際空港限定トートバッグ</textarea>
    <br>
    <button id="tokenize" disabled>Tokenize</button>
    <pre id="output">Loading dictionary...</pre>

    <script type="module">
        import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from './pkg/lindera_wasm.js';
        import { downloadDictionary, loadDictionaryFiles, hasDictionary } from './pkg/opfs.js';

        let tokenizer;

        async function init() {
            await __wbg_init();

            // Download dictionary if not cached
            if (!await hasDictionary("ipadic")) {
                document.getElementById('output').textContent = 'Downloading dictionary...';
                await downloadDictionary(
                    "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip",
                    "ipadic",
                );
            }

            // Load dictionary from OPFS
            const files = await loadDictionaryFiles("ipadic");
            const dictionary = loadDictionaryFromBytes(
                files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
                files.dictWords, files.matrixMtx, files.charDef, files.unk,
            );

            const builder = new TokenizerBuilder();
            builder.setDictionaryInstance(dictionary);
            builder.setMode("normal");
            tokenizer = builder.build();

            document.getElementById('tokenize').disabled = false;
            document.getElementById('output').textContent = 'Ready!';
        }

        document.getElementById('tokenize').addEventListener('click', () => {
            const text = document.getElementById('input').value;
            const tokens = tokenizer.tokenize(text);
            const output = tokens.map(t =>
                `${t.surface}\t${t.details.join(',')}`
            ).join('\n');
            document.getElementById('output').textContent = output;
        });

        init();
    </script>
</body>
</html>

Webpack Configuration

When using Webpack 5, enable the asyncWebAssembly experiment:

// webpack.config.js
module.exports = {
    experiments: {
        asyncWebAssembly: true,
    },
    module: {
        rules: [
            {
                test: /\.wasm$/,
                type: "webassembly/async",
            },
        ],
    },
};

Then import using the bundler target build:

import { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-bundler';
import { loadDictionaryFiles } from 'lindera-wasm-bundler/opfs';

// Load dictionary from OPFS (see OPFS Dictionary Storage for setup)
const files = await loadDictionaryFiles("ipadic");
const dictionary = loadDictionaryFromBytes(
    files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
    files.dictWords, files.matrixMtx, files.charDef, files.unk,
);

const builder = new TokenizerBuilder();
builder.setDictionaryInstance(dictionary);
builder.setMode("normal");
const tokenizer = builder.build();

With the bundler target, __wbg_init() is called automatically by the bundler.

Vite / Rollup Setup

Vite supports WASM out of the box with the web target. Place the built pkg/ directory in your project and import directly:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from './pkg/lindera_wasm.js';
import { loadDictionaryFiles } from './pkg/opfs.js';

await __wbg_init();
// Load dictionary from OPFS and use TokenizerBuilder as shown above

For the bundler target with Vite, you may need the vite-plugin-wasm plugin:

// vite.config.js
import wasm from 'vite-plugin-wasm';

export default {
    plugins: [wasm()],
};

Chrome Extension Considerations

Chrome extensions using Manifest V3 restrict WebAssembly.compile and WebAssembly.instantiate by default. To use lindera-wasm in an extension, you need to add wasm-unsafe-eval to your Content Security Policy:

{
    "content_security_policy": {
        "extension_pages": "script-src 'self' 'wasm-unsafe-eval'; object-src 'self'"
    }
}

Note that wasm-unsafe-eval only allows WebAssembly execution and does not permit arbitrary JavaScript eval().

Performance Tips

• Initialize once: Call __wbg_init() once at application startup, not on every tokenization request.
• Reuse the tokenizer: Create the Tokenizer instance once and reuse it for multiple calls to tokenize().
• Web Workers: For heavy tokenization workloads, consider running Lindera in a Web Worker to avoid blocking the main thread.

OPFS Dictionary Storage

Lindera WASM provides OPFS (Origin Private File System) helper utilities for persistent dictionary caching in web browsers. This allows you to download dictionaries once and reuse them across sessions without embedding them in the WASM binary.

Overview

The OPFS helpers are distributed as a separate JavaScript module (opfs.js) alongside the WASM package. They provide functions to download, store, load, and manage dictionaries using the browser's Origin Private File System.

Dictionaries are stored under the OPFS path lindera/dictionaries/<name>/.

Import

import { downloadDictionary, loadDictionaryFiles, removeDictionary,
         listDictionaries, hasDictionary } from 'lindera-wasm-web/opfs';

Functions

downloadDictionary(url, name, options?)

Downloads a dictionary zip archive, extracts it, and stores the files in OPFS.

The archive should be a zip file containing the 8 required dictionary files, optionally nested in a subdirectory.

• Parameters:
  • url (string) -- URL of the dictionary zip archive
  • name (string) -- Name to store the dictionary under (e.g., "ipadic")
  • options (object, optional):
    • onProgress (function) -- Progress callback
• Returns: Promise<void>

await downloadDictionary(
    "https://example.com/ipadic.zip",
    "ipadic",
    {
        onProgress: (progress) => {
            switch (progress.phase) {
                case "downloading":
                    console.log(`Downloading: ${progress.loaded}/${progress.total} bytes`);
                    break;
                case "extracting":
                    console.log("Extracting archive...");
                    break;
                case "storing":
                    console.log("Storing in OPFS...");
                    break;
                case "complete":
                    console.log("Done!");
                    break;
            }
        },
    },
);

Progress Callback

The onProgress callback receives an object with the following shape:

loadDictionaryFiles(name)

Loads dictionary files from OPFS as an object of Uint8Array values.

The returned object can be passed directly to loadDictionaryFromBytes().

• Parameters: name (string) -- The dictionary name (e.g., "ipadic")
• Returns: Promise<DictionaryFiles>

const files = await loadDictionaryFiles("ipadic");

DictionaryFiles

removeDictionary(name)

Removes a dictionary from OPFS.

• Parameters: name (string) -- The dictionary name to remove
• Returns: Promise<void>

await removeDictionary("ipadic");

listDictionaries()

Lists all dictionaries stored in OPFS.

• Returns: Promise<string[]> -- Array of dictionary names

const names = await listDictionaries();
console.log(names); // e.g., ["ipadic", "unidic"]

hasDictionary(name)

Checks if a dictionary exists in OPFS.

• Parameters: name (string) -- The dictionary name to check
• Returns: Promise<boolean>

if (await hasDictionary("ipadic")) {
    console.log("Dictionary is cached");
}

Complete Workflow

A typical workflow for using OPFS-based dictionaries:

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm-web';
import { downloadDictionary, loadDictionaryFiles, hasDictionary } from 'lindera-wasm-web/opfs';

async function main() {
    await __wbg_init();

    const DICT_NAME = "ipadic";
    const DICT_URL = "https://github.com/lindera/lindera/releases/download/<version>/lindera-ipadic-<version>.zip";

    // Download dictionary if not already cached
    if (!await hasDictionary(DICT_NAME)) {
        await downloadDictionary(DICT_URL, DICT_NAME, {
            onProgress: ({ phase, loaded, total }) => {
                if (phase === "downloading" && total) {
                    console.log(`${(loaded / total * 100).toFixed(1)}%`);
                }
            },
        });
    }

    // Load dictionary from OPFS
    const files = await loadDictionaryFiles(DICT_NAME);
    const dictionary = loadDictionaryFromBytes(
        files.metadata, files.dictDa, files.dictVals, files.dictWordsIdx,
        files.dictWords, files.matrixMtx, files.charDef, files.unk,
    );

    // Build tokenizer
    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dictionary);
    builder.setMode("normal");
    const tokenizer = builder.build();

    // Tokenize
    const tokens = tokenizer.tokenize("形態素解析を行います");
    tokens.forEach(token => {
        console.log(`${token.surface}\t${token.details.join(',')}`);
    });
}

main();

Required Dictionary Files

A valid dictionary archive must contain these 8 files:

Browser Compatibility

OPFS requires a secure context (HTTPS or localhost) and is supported in:

• Chrome 86+
• Edge 86+
• Firefox 111+
• Safari 15.2+

The zip extraction uses the DecompressionStream API, which requires:

• Chrome 80+
• Edge 80+
• Firefox 113+
• Safari 16.4+

Lindera IPADIC

Lindera IPADIC is a Japanese dictionary crate based on IPADIC. IPADIC is the most common dictionary for Japanese morphological analysis.

Contents

• Dictionary Format -- Field definitions for system and user dictionaries
• Build -- How to build the dictionary from source
• Examples -- Tokenization examples

API Reference

• lindera-ipadic

Lindera IPADIC

Dictionary version

This repository contains mecab-ipadic.

Dictionary format

Refer to the manual for details on the IPADIC dictionary format and part-of-speech tags.

User dictionary format (CSV)

Simple version

Detailed version

API reference

The API reference is available. Please see following URL:

• lindera-ipadic

Build

This page describes how to build the IPADIC dictionary from source files.

Build system dictionary

Download the IPADIC source files and build the dictionary:

# Download and extract IPADIC source files
% curl -L -o /tmp/mecab-ipadic-2.7.0-20250920.tar.gz "https://Lindera.dev/mecab-ipadic-2.7.0-20250920.tar.gz"
% tar zxvf /tmp/mecab-ipadic-2.7.0-20250920.tar.gz -C /tmp

# Build the dictionary
% lindera build \
  --src /tmp/mecab-ipadic-2.7.0-20250920 \
  --dest /tmp/lindera-ipadic-2.7.0-20250920 \
  --metadata ./lindera-ipadic/metadata.json

Build user dictionary

Build a user dictionary from a CSV file:

% lindera build \
  --src ./resources/user_dict/ipadic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-ipadic/metadata.json \
  --user

For more details about user dictionary format, see Dictionary Format.

Embedding in binary

To embed the IPADIC dictionary directly into the binary:

cargo build --features=embed-ipadic

This allows using embedded://ipadic as the dictionary path without external dictionary files.

Examples

This page shows tokenization examples using the IPADIC dictionary.

Tokenize with external IPADIC

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Tokenize with embedded IPADIC

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素  名詞,一般,*,*,*,*,形態素,ケイタイソ,ケイタイソ
解析    名詞,サ変接続,*,*,*,*,解析,カイセキ,カイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC dictionary in the binary, you must build with the --features=embed-ipadic option.

Tokenize with user dictionary (CSV format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict embedded://ipadic \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.csv

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Tokenize with user dictionary (binary format)

% echo "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です" | lindera tokenize \
  --dict /tmp/lindera-ipadic-2.7.0-20250920 \
  --user-dict ./resources/user_dict/ipadic_simple_userdic.bin

東京スカイツリー        カスタム名詞,*,*,*,*,*,東京スカイツリー,トウキョウスカイツリー,*
の      助詞,連体化,*,*,*,*,の,ノ,ノ
最寄り駅        名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
は      助詞,係助詞,*,*,*,*,は,ハ,ワ
とうきょうスカイツリー駅        カスタム名詞,*,*,*,*,*,とうきょうスカイツリー駅,トウキョウスカイツリーエキ,*
です    助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let mut tokens = tokenizer.tokenize(text)?;
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }
    Ok(())
}

Lindera IPADIC NEologd

Lindera IPADIC NEologd is a Japanese dictionary crate based on IPADIC NEologd, which includes neologisms (new words). It extends the standard IPADIC dictionary with additional vocabulary covering recent terms and proper nouns.

Contents

• Dictionary Format -- Field definitions for system and user dictionaries
• Build -- How to build the dictionary from source
• Examples -- Tokenization examples

API Reference

• lindera-ipadic-neologd

Lindera IPADIC NEologd

Dictionary version

This repository contains mecab-ipadic-neologd.

Dictionary format

Refer to the manual for details on the IPADIC dictionary format and part-of-speech tags.

User dictionary format (CSV)

Simple version

Detailed version

API reference

The API reference is available. Please see following URL:

• lindera-ipadic-neologd

Build

This page describes how to build the IPADIC NEologd dictionary from source files.

Build system dictionary

Download the IPADIC NEologd source files and build the dictionary:

% curl -L -o /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz "https://lindera.dev/mecab-ipadic-neologd-0.0.7-20200820.tar.gz"
% tar zxvf /tmp/mecab-ipadic-neologd-0.0.7-20200820.tar.gz -C /tmp

% lindera build \
  --src /tmp/mecab-ipadic-neologd-0.0.7-20200820 \
  --dest /tmp/lindera-ipadic-neologd-0.0.7-20200820 \
  --metadata ./lindera-ipadic-neologd/metadata.json

Embedding in binary

To embed the IPADIC NEologd dictionary directly into the binary:

cargo build --features=embed-ipadic-neologd

This allows using embedded://ipadic-neologd as the dictionary path without external dictionary files.

Examples

This page shows tokenization examples using the IPADIC NEologd dictionary.

Tokenize with external IPADIC NEologd

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-ipadic-neologd-0.0.7-20200820

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

Notice that NEologd treats "形態素解析" (morphological analysis) as a single compound noun, whereas standard IPADIC splits it into "形態素" and "解析".

Tokenize with embedded IPADIC NEologd

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://ipadic-neologd

日本語  名詞,一般,*,*,*,*,日本語,ニホンゴ,ニホンゴ
の      助詞,連体化,*,*,*,*,の,ノ,ノ
形態素解析      名詞,固有名詞,一般,*,*,*,形態素解析,ケイタイソカイセキ,ケイタイソカイセキ
を      助詞,格助詞,一般,*,*,*,を,ヲ,ヲ
行う    動詞,自立,*,*,五段・ワ行促音便,基本形,行う,オコナウ,オコナウ
こと    名詞,非自立,一般,*,*,*,こと,コト,コト
が      助詞,格助詞,一般,*,*,*,が,ガ,ガ
でき    動詞,自立,*,*,一段,連用形,できる,デキ,デキ
ます    助動詞,*,*,*,特殊・マス,基本形,ます,マス,マス
。      記号,句点,*,*,*,*,。,。,。
EOS

NOTE: To include IPADIC NEologd dictionary in the binary, you must build with the --features=embed-ipadic-neologd option.

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic-neologd")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let mut tokens = tokenizer.tokenize(text)?;
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }
    Ok(())
}

Lindera UniDic

Lindera UniDic is a Japanese dictionary crate based on UniDic, which uses uniform word unit definitions. UniDic provides more detailed morphological information than IPADIC, with 21 fields per entry.

Contents

• Dictionary Format -- Field definitions for system and user dictionaries
• Build -- How to build the dictionary from source
• Examples -- Tokenization examples

API Reference

• lindera-unidic

Lindera UniDic

Dictionary version

This repository contains unidic-mecab.

Dictionary format

Refer to the manual for details on the unidic-mecab dictionary format and part-of-speech tags.

User dictionary format (CSV)

Simple version

Detailed version

API reference

The API reference is available. Please see following URL:

• lindera-unidic

Build

This page describes how to build the UniDic dictionary from source files.

Build system dictionary

Download the UniDic source files and build the dictionary:

% curl -L -o /tmp/unidic-mecab-2.1.2.tar.gz "https://Lindera.dev/unidic-mecab-2.1.2.tar.gz"
% tar zxvf /tmp/unidic-mecab-2.1.2.tar.gz -C /tmp

% lindera build \
  --src /tmp/unidic-mecab-2.1.2 \
  --dest /tmp/lindera-unidic-2.1.2 \
  --metadata ./lindera-unidic/metadata.json

Build user dictionary

Build a user dictionary from a CSV file:

% lindera build \
  --src ./resources/user_dict/unidic_simple_userdic.csv \
  --dest ./resources/user_dict \
  --metadata ./lindera-unidic/metadata.json \
  --user

For more details about user dictionary format, see Dictionary Format.

Embedding in binary

To embed the UniDic dictionary directly into the binary:

cargo build --features=embed-unidic

This allows using embedded://unidic as the dictionary path without external dictionary files.

Examples

This page shows tokenization examples using the UniDic dictionary.

Tokenize with external UniDic

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict /tmp/lindera-unidic-2.1.2

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

Notice that UniDic splits "日本語" into "日本" and "語", and "形態素" into "形態" and "素", reflecting its uniform word unit definitions.

Tokenize with embedded UniDic

% echo "日本語の形態素解析を行うことができます。" | lindera tokenize \
  --dict embedded://unidic

日本    名詞,固有名詞,地名,国,*,*,ニッポン,日本,日本,ニッポン,日本,ニッポン,固,*,*,*,*
語      名詞,普通名詞,一般,*,*,*,ゴ,語,語,ゴ,語,ゴ,漢,*,*,*,*
の      助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*
形態    名詞,普通名詞,一般,*,*,*,ケイタイ,形態,形態,ケータイ,形態,ケータイ,漢,*,*,*,*
素      接尾辞,名詞的,一般,*,*,*,ソ,素,素,ソ,素,ソ,漢,*,*,*,*
解析    名詞,普通名詞,サ変可能,*,*,*,カイセキ,解析,解析,カイセキ,解析,カイセキ,漢,*,*,*,*
を      助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*
行う    動詞,一般,*,*,五段-ワア行,連体形-一般,オコナウ,行う,行う,オコナウ,行う,オコナウ,和,*,*,*,*
こと    名詞,普通名詞,一般,*,*,*,コト,事,こと,コト,こと,コト,和,コ濁,基本形,*,*
が      助詞,格助詞,*,*,*,*,ガ,が,が,ガ,が,ガ,和,*,*,*,*
でき    動詞,非自立可能,*,*,上一段-カ行,連用形-一般,デキル,出来る,でき,デキ,できる,デキル,和,*,*,*,*
ます    助動詞,*,*,*,助動詞-マス,終止形-一般,マス,ます,ます,マス,ます,マス,和,*,*,*,*
。      補助記号,句点,*,*,*,*,,。,。,,。,,記号,*,*,*,*
EOS

NOTE: To include UniDic dictionary in the binary, you must build with the --features=embed-unidic option.

Rust API example

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://unidic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
    let tokenizer = Tokenizer::new(segmenter);

    let text = "日本語の形態素解析を行うことができます。";
    let mut tokens = tokenizer.tokenize(text)?;
    for token in tokens.iter_mut() {
        let details = token.details().join(",");
        println!("{}\t{}", token.surface.as_ref(), details);
    }
    Ok(())
}

Lindera ko-dic

Lindera ko-dic is a Korean dictionary crate based on mecab-ko-dic.

Contents

• Dictionary Format -- Field definitions for system and user dictionaries
• Build -- How to build the dictionary from source
• Examples -- Tokenization examples

API Reference

• lindera-ko-dic

Lindera ko-dic

Dictionary version

This repository contains mecab-ko-dic.

Dictionary format

Information about the dictionary format and part-of-speech tags used by mecab-ko-dic id documented in this Google Spreadsheet, linked to from mecab-ko-dic's repository readme.

Note how ko-dic has one less feature column than NAIST JDIC, and has an altogether different set of information (e.g. doesn't provide the "original form" of the word).

The tags are a slight modification of those specified by 세종 (Sejong), whatever that is. The mappings from Sejong to mecab-ko-dic's tag names are given in tab 태그 v2.0 on the above-linked spreadsheet.

The dictionary format is specified fully (in Korean) in tab 사전 형식 v2.0 of the spreadsheet. Any blank values default to *.

User dictionary format (CSV)

Simple version

Detailed version