Installation

Requirements

Rust 1.89 or newer. The workspace declares this as rust-version, so an older toolchain reports it as a cargo error rather than failing to compile.

Adding the Dependency

Put the following in Cargo.toml:

[dependencies]
lindera = "7"

[!NOTE] This single dependency also provides the analysis chain (character filters, token filters, and the Tokenizer) as lindera::analysis; use default-features = false if you only need the segmenter. Upgrading from v6? See Migration v6 to v7 for details.

Dictionary Setup

Lindera requires a pre-built dictionary at runtime. Download a dictionary from GitHub Releases and specify its path when loading:

#![allow(unused)]
fn main() {
let dictionary = load_dictionary("/path/to/ipadic")?;
}

[!TIP] If you want to embed a dictionary directly into the binary (advanced usage), enable the corresponding embed-* feature flag and load it using the embedded:// scheme:

#![allow(unused)]
fn main() {
// Cargo.toml: lindera = { version = "7", features = ["embed-ipadic"] }
let dictionary = load_dictionary("embedded://ipadic")?;
}

See Feature Flags for details.

Environment Variables

LINDERA_BUILD_DICTIONARY_CACHE_DIR

The LINDERA_BUILD_DICTIONARY_CACHE_DIR environment variable designates a build-time cache directory for the embedded-dictionary build pipeline. It is read only by the dictionary crates' build scripts and has no effect at runtime.

When set, each build stores two kinds of files under $LINDERA_BUILD_DICTIONARY_CACHE_DIR/<version>-fmt<format>/ (where <version> is the dictionary crate version and <format> is the dictionary format version):

  • the downloaded distribution archive (validated with MD5; invalid files are re-downloaded)
  • the built binary dictionary that gets embedded into the crate, with a build_inputs.json that records what it was built from

The format version is part of the path so that a cache written by a build with a different on-disk dictionary layout is a miss rather than a stale hit. A cached dictionary is reused only when all of the following match the current build, and is rebuilt otherwise:

  • the dictionary format version and the contents of the crate's metadata.json
  • the lindera-dictionary crate that builds it: its version and a digest of its source, so that any change to the dictionary builder, an uncommitted one included, rebuilds the dictionary
  • the source archive (file name and MD5) and the directory the build reads in it
  • the context-id frequency histogram the build reads (context_id_freq.txt, or the file named by LINDERA_CTX_FREQ_FILE) when connection_id_mapping is enabled

A dictionary left behind by an interrupted build, or cached by a release from before build_inputs.json, is rebuilt. The versions of the builder's dependencies are not compared: after upgrading a dependency of lindera-dictionary without a format version change, clear the cache if the built dictionaries may differ. Directories from older format versions are not deleted automatically; remove them by hand if the cache grows.

cargo build -vv shows, for each dictionary crate, whether the cached dictionary was used and why or why not, on lines that start with lindera dictionary cache:.

This enables:

  • Offline builds: once cached, subsequent builds need no network access
  • Faster builds: download and dictionary build are skipped when valid cached files exist
  • Reproducible builds: consistent dictionary versions across builds

Usage:

export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache
cargo build --features=embed-ipadic

Notes:

  • The directory is managed automatically and is safe to delete; contents are re-downloaded and rebuilt as needed
  • Version subdirectories accumulate across upgrades and are not garbage-collected; old ones can be removed freely
  • Setting this variable causes dictionary crates to download and build their dictionaries even when no embed-* feature is enabled (useful for pre-populating the cache). Such a build embeds nothing, so it keeps a dictionary already in the cache without checking it; a build with the embed-* feature checks it and rebuilds it if needed
  • Use one cache directory per checkout. Two checkouts with different builder code that share a directory rebuild each other's dictionaries, and a crate whose build script does not rerun can embed the dictionary the other checkout built. In this repository, a relative path resolves under the workspace root, which gives each checkout its own cache
  • Builds that share the directory, such as cargo build alongside rust-analyzer with its own target directory, write each dictionary one at a time: a build that finds another build writing the same dictionary waits for it, and then uses the dictionary it left (cargo build -vv shows lindera dictionary cache: waiting for another build). The lock is an empty <dictionary>.lock file next to the dictionary, held while a build writes it. It works between builds on one machine, and on a network file system only as far as the file system supports file locks. Where the file cannot be locked, the build prints a warning and goes on without the lock, as builds before 7.0.0 did, so two such builds can still remove each other's files
  • Builds with DOCS_RS set ignore this variable (see DOCS_RS below)

LINDERA_CONFIG_FILE

The LINDERA_CONFIG_FILE environment variable names a YAML configuration file for the tokenizer, which TokenizerBuilder::new() reads. This allows you to configure tokenizer behavior without modifying Rust code.

export LINDERA_CONFIG_FILE=./resources/config/lindera.yml

LINDERA_CONFIG_PATH, the name of this variable up to v6, still works until v8.0.0, with a deprecation warning on standard error. When both are set, LINDERA_CONFIG_FILE is used.

See the Configuration section for details on the configuration format.

DOCS_RS

The DOCS_RS environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.

Such a build ignores LINDERA_BUILD_DICTIONARY_CACHE_DIR: the dummy dictionaries are built in the build script's output directory, never in the cache, so a later build without DOCS_RS does not embed them.

This is primarily used internally by docs.rs and typically doesn't need to be set by users.