Migrating from v4 to v5

Lindera v5.0.0 restructures the workspace around a lean core: the default build of the lindera crate is now a pure morphological segmenter, and the dictionary-training pipeline lives in its own crate. This guide lists every breaking change and the one-line fixes for each.

[!NOTE] v5.0.0 is the next planned release and has not been published to crates.io yet; the current published version is 4.0.1. The changes described below already exist on the main branch, ahead of the version bump.

Overview

The language bindings (Python, Node.js, Ruby, PHP, WASM) and the CLI are unaffected: they enable the required features themselves, and their APIs and output are unchanged. Tokenization output is also unchanged — v5 produces byte-for-byte identical tokens to v4 for the same input and dictionary.

The analysis chain moved to lindera-analysis

In v5.0 the lindera crate is a pure morphological segmenter: the character_filter, token_filter, and tokenizer modules moved to the new lindera-analysis crate (mirroring Lucene's split between a tokenizer core and analyzer modules).

If you use Tokenizer or any filter, depend on the new crate and update the import paths — the APIs themselves are unchanged:

# v4
[dependencies]
lindera = "4.0"

# v5
[dependencies]
lindera = "5.0"
lindera-analysis = "5.0"

#![allow(unused)]
fn main() {
// v4
use lindera::tokenizer::Tokenizer;
use lindera::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;

// v5
use lindera_analysis::tokenizer::Tokenizer;
use lindera_analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
}

If you only segment text, nothing changes in your code — and your dependency tree shrinks (kanaria, unicode-normalization, unicode-segmentation, unicode-blocks, and serde_yaml_ng are no longer built):

#![allow(unused)]
fn main() {
use std::borrow::Cow;

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("/path/to/ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
let tokens = segmenter.segment(Cow::Borrowed("関西国際空港限定トートバッグ"))?;
}

Training moved to the lindera-trainer crate

The CRF training pipeline (TrainerConfig, Trainer, Corpus, Model, SerializableModel) moved from lindera-dictionary's train-gated trainer module into the new lindera-trainer crate. As a result, lindera-dictionary no longer depends on lindera-crf or regex.

Through the lindera facade nothing changes — the train feature now pulls lindera-trainer and re-exports it under the same path:

#![allow(unused)]
fn main() {
// Works in both v4 and v5 (with the `train` feature):
use lindera::dictionary::trainer::{Corpus, Trainer, TrainerConfig};
}

Only direct users of lindera-dictionary --features train need to switch:

# v4
[dependencies]
lindera-dictionary = { version = "4.0", features = ["train"] }

# v5
[dependencies]
lindera-dictionary = "5.0"
lindera-trainer = "5.0"

#![allow(unused)]
fn main() {
// v4
use lindera_dictionary::trainer::{Corpus, Trainer, TrainerConfig};

// v5
use lindera_trainer::{Corpus, Trainer, TrainerConfig};
}

The lindera train → lindera export → lindera build CLI workflow is unchanged.

Build cache environment variable renamed

The build-time dictionary cache variable LINDERA_DICTIONARIES_PATH is renamed to LINDERA_BUILD_DICTIONARY_CACHE_DIR to make its contract explicit: it is read only at build time by the dictionary crates' build scripts, and it designates an auto-managed cache holding the downloaded dictionary archives and the built binary dictionaries.

The old name keeps working through v5.x as a deprecated fallback (the new name wins when both are set) and will be removed in v6.0.0.

# v4
export LINDERA_DICTIONARIES_PATH=/path/to/cache

# v5
export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache