Feature Flags

Lindera uses Cargo feature flags to control optional functionality and dictionary embedding.

Core Features

• mmap is enabled by default in the main lindera crate.
• The analysis chain (character filters, token filters, and the Tokenizer) is not a feature of this crate: as of v5.0 it lives in the companion lindera-analysis crate. The lindera crate itself is a pure segmenter around the Segmenter API.
• train is enabled by default in lindera-cli, lindera-python, lindera-nodejs, lindera-ruby, and lindera-php. It is disabled by default in the core lindera library crate -- enable it explicitly with features = ["train"] for library usage. It is not available in lindera-wasm.

Using External Dictionaries (Recommended)

The recommended approach is to use pre-built dictionaries as external files. Download a dictionary from GitHub Releases and specify its path at runtime:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

No additional feature flags are required for this usage.

Dictionary Embedding Features (Advanced)

These features embed pre-built dictionaries directly into the binary, eliminating the need for external dictionary files at runtime. This is intended for advanced users who need self-contained binaries.

None of these are enabled by default. Enable them as needed:

[dependencies]
lindera = { version = "5", features = ["embed-ipadic"] }

When embedding is enabled, you can load the dictionary with:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("embedded://ipadic")?;
}

[!NOTE] Dictionary data carries its own license, separate from Lindera's. When you distribute a binary with an embedded dictionary (or redistribute a built dictionary), reproduce the attribution from the corresponding dictionary crate's NOTICE.txt (e.g. lindera-unidic/NOTICE.txt) in the documentation or other materials provided with your distribution.

Combination Features

These meta-features enable multiple dictionaries at once for multilingual applications.

Combining Feature Flags

Multiple feature flags can be combined. For example, to embed both Japanese and Korean dictionaries:

[dependencies]
lindera = { version = "5", features = ["embed-ipadic", "embed-ko-dic"] }

Or from the command line:

cargo build --features embed-ipadic,embed-ko-dic

Notes

• Embedding dictionaries increases binary size significantly. Only embed dictionaries you actually need.
• The train feature adds a dependency on lindera-crf and increases compile time. It is not needed for tokenization-only use cases.
• The mmap feature enables memory-mapped dictionary loading for filesystem-based dictionaries, requested via --mmap (CLI) or the use_mmap segmenter config key. Every large component stays lazily paged and costs no anonymous memory: the word-list files (dict.vals/dict.wordsidx/dict.words), the connection-cost matrix (matrix.mtx), and the prefix-dictionary trie (dict.trie), which is walked in place over its serialized bytes. Nothing is fully materialized at load. It has no effect on embedded dictionaries, whose data is read in place from the binary regardless.