Migrating from v4 to v5

Lindera v5.0.0 restructures the workspace around a lean core: the default build of the lindera crate is now a pure morphological segmenter, and the dictionary-training pipeline lives in its own crate. This guide lists every breaking change and the one-line fixes for each.

Overview

The language bindings (Python, Node.js, Ruby, PHP, WASM) and the CLI are unaffected: they enable the required features themselves, and their APIs and output are unchanged. Tokenization output is also unchanged — v5 produces byte-for-byte identical tokens to v4 for the same input and dictionary.

The analysis chain moved to lindera-analysis

In v5.0 the lindera crate is a pure morphological segmenter: the character_filter, token_filter, and tokenizer modules moved to the new lindera-analysis crate (mirroring Lucene's split between a tokenizer core and analyzer modules).

If you use Tokenizer or any filter, depend on the new crate and update the import paths — the APIs themselves are unchanged:

# v4
[dependencies]
lindera = "4.0"

# v5
[dependencies]
lindera = "5.0"
lindera-analysis = "5.0"

#![allow(unused)]
fn main() {
// v4
use lindera::tokenizer::Tokenizer;
use lindera::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;

// v5
use lindera_analysis::tokenizer::Tokenizer;
use lindera_analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
}

If you only segment text, nothing changes in your code — and your dependency tree shrinks (kanaria, unicode-normalization, unicode-segmentation, unicode-blocks, and serde_yaml_ng are no longer built):

#![allow(unused)]
fn main() {
use std::borrow::Cow;

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;

let dictionary = load_dictionary("/path/to/ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
let tokens = segmenter.segment(Cow::Borrowed("関西国際空港限定トートバッグ"))?;
}

Training moved to the lindera-trainer crate

The CRF training pipeline (TrainerConfig, Trainer, Corpus, Model, SerializableModel) moved from lindera-dictionary's train-gated trainer module into the new lindera-trainer crate. As a result, lindera-dictionary no longer depends on lindera-crf or regex.

Through the lindera facade nothing changes — the train feature now pulls lindera-trainer and re-exports it under the same path:

#![allow(unused)]
fn main() {
// Works in both v4 and v5 (with the `train` feature):
use lindera::dictionary::trainer::{Corpus, Trainer, TrainerConfig};
}

Only direct users of lindera-dictionary --features train need to switch:

# v4
[dependencies]
lindera-dictionary = { version = "4.0", features = ["train"] }

# v5
[dependencies]
lindera-dictionary = "5.0"
lindera-trainer = "5.0"

#![allow(unused)]
fn main() {
// v4
use lindera_dictionary::trainer::{Corpus, Trainer, TrainerConfig};

// v5
use lindera_trainer::{Corpus, Trainer, TrainerConfig};
}

The lindera train → lindera export → lindera build CLI workflow is unchanged.

Build cache environment variable renamed

The build-time dictionary cache variable LINDERA_DICTIONARIES_PATH is renamed to LINDERA_BUILD_DICTIONARY_CACHE_DIR to make its contract explicit: it is read only at build time by the dictionary crates' build scripts, and it designates an auto-managed cache holding the downloaded dictionary archives and the built binary dictionaries.

The old name keeps working through v5.x as a deprecated fallback (the new name wins when both are set) and will be removed in v6.0.0.

# v4
export LINDERA_DICTIONARIES_PATH=/path/to/cache

# v5
export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache