Migrating from v5 to v6

Lindera v6.0.0 rebuilds prebuilt user dictionaries, changes tokenization output in two targeted ways (a Decompose-mode accuracy fix and a new default-on unknown-word feature), reshapes the JavaScript bindings' token type, renames the published binding packages on npm and PyPI to bare names, and renames several lindera-dictionary Rust APIs. Most users only need to rebuild their dictionaries; direct users of lindera_dictionary::viterbi/mode types, JavaScript users, and anyone relying on exact legacy output for out-of-vocabulary text have more to check.

If you are upgrading from v5.2 or earlier, the system-dictionary format change described below applies to you as well. It shipped in v5.3.0, so upgrading from v5.3.0 does not require rebuilding system dictionaries — but it does require rebuilding user dictionaries, for an unrelated reason.

Overview

Package renames on npm, PyPI, and crates.io

Starting with v6.0.0, the language bindings are published under bare names — the registry already scopes the package to its ecosystem, so the -nodejs/-python suffixes carried no information:

The RubyGems gem name (lindera) and the core Rust crates on crates.io are unchanged.

Node.js

Update the dependency and the module specifier; the API itself is unaffected:

// v5
const { TokenizerBuilder } = require("lindera-nodejs");

// v6
const { TokenizerBuilder } = require("lindera");

The platform packages (lindera-darwin-arm64, lindera-linux-x64-gnu, …) are optionalDependencies of the main package, so npm selects the right one automatically — you never install them by name.

The old lindera-nodejs and lindera-nodejs-* packages are npm-deprecated: they remain installable at their last v5 versions but receive no further releases.

Python

Only the install name changes — the import name was always lindera and stays that way:

# v5
pip install lindera-python

# v6
pip install lindera

A final lindera-python release is published as a transition stub that depends on lindera, so an unchanged pip install lindera-python still ends up with the real package during the transition. Update your requirements to lindera anyway — the stub will not be maintained.

WASM

The two v5 packages are consolidated into a single lindera-wasm package, built with wasm-pack --target web. A web-target build requires calling the default-exported async init function once before using any API:

import __wbg_init, { TokenizerBuilder } from "lindera-wasm";

await __wbg_init();

• Coming from lindera-wasm-web: only the package name changes — your code already calls the init function.
• Coming from lindera-wasm-bundler: install lindera-wasm and add the init call — the bundler-target build initialized implicitly, the web-target build does not. Modern bundlers consume the web-target build directly (Vite out of the box; Webpack 5 with the asyncWebAssembly experiment). See Browser Usage for bundler configuration.

The old lindera-wasm-web and lindera-wasm-bundler packages are npm-deprecated at their last v5 versions.

crates.io

The binding crates (lindera-python, lindera-nodejs, lindera-ruby, lindera-wasm) are no longer published to crates.io — they were never usable as Rust dependencies, only as sources for the registry packages above. The existing 5.3.0 releases remain available there. The core crates (lindera, lindera-dictionary, the dictionary crates, and so on) continue to be published as before.

Prebuilt user dictionaries must be rebuilt

Lindera v6 upgrades daachorse from 4.x to 5.0, which changed the serialized form of the Aho-Corasick automaton that a user dictionary embeds. A .bin file built by any v5 release therefore fails to load:

LinderaError(kind=Deserialize, source=InvalidAutomatonError: invalid serialized automaton)

Unlike system dictionaries, user-dictionary .bin files carry no format_version, so there is no version check to produce a friendlier message — the failure surfaces as the deserialize error above.

Rebuild from the CSV source with v6:

lindera build --user \
  --src ./user_dict.csv \
  --dest ./build \
  --metadata ./lindera-ipadic/metadata.json

If you load a user dictionary from a .csv file rather than a prebuilt .bin, it is compiled at load time and no action is needed.

Trained model files must be regenerated

lindera train now writes a byte-reproducible model.dat: training the same inputs twice with the same flags produces identical bytes, so the artifact can be checksummed, cached and diffed.

Getting there required storing the connection matrix and the unknown-word categories as ordered maps rather than hash maps — rkyv lays an archived HashMap out in source iteration order, and that order is seeded per instance. This changes the on-disk layout, so a model.dat written by an earlier build fails to load:

failed to deserialize model: ... re-run `lindera train` to regenerate it.

model.dat is an intermediate between lindera train and lindera export, not a distributed artifact — regenerate it by re-running lindera train. Dictionaries already exported from an old model are unaffected, and the trained weights are identical either way: only the serialization order changed.

Two related notes:

• SerializableModel::connection_matrix and SerializableModel::unk_categories are now BTreeMap rather than HashMap. This matters only if you read those public fields directly; the writer methods are unchanged.
• Reproducibility holds for any --max-threads value: the gradient and the loss are summed over a fixed partition of the training data in a fixed order, so the thread count affects only speed, never the trained model.
• lindera export output is byte-reproducible too: metadata.json no longer carries an updated_at timestamp, and every export writer emits entries in a deterministic order. Accordingly, the ModelInfo.updated_at field was removed from the Rust API (lindera-dictionary); old metadata.json files that still contain the key keep parsing.
• DictionaryRewriter::rewrite_cached and clear_cache were removed: the cache was keyed on the full feature string, which is unique per row in a real lexicon, so it never hit and only retained memory. Call rewrite instead — it no longer requires &mut.
• The features parameter of the six FeatureExtractor::extract_* methods changed from &[String] to &[&str], so callers no longer have to allocate one String per feature field. A caller holding a &[String] can pass &v.iter().map(|s| s.as_str()).collect::<Vec<_>>().
• lindera-wasm removed the filesystem-based exports loadUserDictionary, buildDictionary, buildUserDictionary, and TokenizerBuilder.setUserDictionary(uri) (plus their snake_case aliases): there is no filesystem on wasm32-unknown-unknown, so they always failed at runtime. Load system dictionaries with loadDictionaryFromBytes() (or embedded://), and user dictionaries with the new loadUserDictionaryFromBytes() / loadUserDictionaryBinFromBytes() + setUserDictionaryInstance(). loadDictionary() now rejects file URIs and bare paths up front with an error naming the bytes API.
• The fixed partition changed the summation order, so weights trained before this change are not byte-identical to weights trained after it — at any thread count, including --max-threads 1. Re-run lindera train if you need artifacts that compare equal from now on.

Dictionary format version 2 (shipped in v5.3.0)

This change shipped in v5.3.0, not in v6.0.0. If you are upgrading from v5.3.0 your system dictionaries already use format version 2 and load unchanged. Follow this section only when upgrading from v5.2 or earlier.

The system prefix dictionary is no longer stored as a serialized daachorse Aho-Corasick automaton (dict.da). The builder now constructs a char-wise double-array trie with crawdad at build time and writes it as dict.trie, together with a u32 prefix-sum index into the word values (dict.valsidx). At runtime the trie is walked in place over its serialized bytes: loading it is an O(1) header check instead of a deserialization pass, and under mmap it stays lazily paged like every other large component.

A built dictionary directory now contains these 9 files:

All files other than dict.trie and dict.valsidx are unchanged from v5; dict.da no longer exists.

Loading an old dictionary now fails with a clear error

metadata.json in a built dictionary records format_version: 2, and the loader checks it. Loading a dictionary built by v5.2 or earlier fails with an actionable error instead of misreading the headerless binary files:

Dictionary 'ipadic' has format version 1, but this build of Lindera reads format version 2. To fix this, rebuild it with `lindera build`, or download a matching prebuilt dictionary with `lindera download`.

Tokenization output changes

Unlike the format change above, these two changes can alter the tokens Lindera produces for the same input and dictionary.

Decompose-mode penalty accuracy fix

Mode::Decompose's length penalty used to approximate a span's character count as (stop_byte - start_byte) / 3, which is exact only for text made entirely of 3-byte UTF-8 characters (common Japanese kanji/kana). Lindera's Viterbi lattice is now character-indexed internally, so the penalty uses the real character count. This only changes Decompose output for spans containing 1-byte (ASCII), 2-byte, or 4-byte (e.g. emoji, some CJK extension characters) UTF-8 characters; pure-Japanese Decompose output is unaffected. There is no option to restore the old (inexact) behavior, since it was a bug fix rather than a behavior choice.

Unknown-word length ladder (new, on by default)

For a run of out-of-vocabulary characters, MeCab and Vibrato generate a "ladder" of candidate unknown-word lengths (1..=LENGTH, where LENGTH is a per-category field in the dictionary's char.def) and let the Viterbi search pick whichever length yields the lowest-cost path. Lindera parsed LENGTH from char.def but never used it at runtime, so it only ever generated a single 1-character candidate or (for grouping categories) one candidate spanning the whole run — never anything in between.

Starting in v6.0.0, Lindera generates this candidate ladder by default. For IPADIC this affects the KANJI, HIRAGANA, and KATAKANA categories (LENGTH=2); most other categories declare LENGTH=0 and are unaffected. The practical effect is that consecutive out-of-vocabulary kanji or kana characters can now be grouped into multi-character unknown words when that scores lower than splitting them one character at a time — for example, an unrecognized two-kanji compound is now tokenized as one word instead of two.

This is additive to (and independent of) max_grouping_len, the new option described below for capping whole-run grouping.

To reproduce v5-identical output on text containing out-of-vocabulary kanji/kana runs, disable the ladder:

let segmenter = Segmenter::new(mode, dictionary, None)
    .unknown_word_ladder(false);

lindera tokenize -d ipadic --disable-unknown-word-ladder input.txt

AnalysisWorker/SegmentWorker expose the same toggle via set_unknown_word_ladder(bool), and TokenizerBuilder via set_segmenter_unknown_word_ladder(bool).

New option: max_grouping_len (default unchanged)

v6 adds MeCab's max-grouping-size semantics: when a groupable run of out-of-vocabulary characters extends more than max_grouping_len characters beyond the first, the grouped candidate is not emitted and single-character unknown words are used instead. The default is unbounded, which is exactly what v5 did, so this changes nothing unless you opt in:

let segmenter = Segmenter::new(mode, dictionary, None)
    .max_grouping_len(Some(24)); // MeCab's own default

lindera tokenize -d ipadic --max-grouping-len 24 input.txt

Also available as set_max_grouping_len(Option<usize>) on SegmentWorker/AnalysisWorker, set_segmenter_max_grouping_len(usize) on TokenizerBuilder, and the max_grouping_len key in a segmenter config.

Rust API changes in lindera_dictionary

These affect only code that uses lindera_dictionary::viterbi::{Lattice, Edge} or lindera_dictionary::mode::{Mode, Penalty} directly (for example, custom lattice inspection or alternative tokenizer front-ends). The lindera crate's Segmenter, Tokenizer, SegmentWorker, and AnalysisWorker APIs are unaffected — they gained the unknown_word_ladder/max_grouping_len options above but no signatures changed.

set_text and set_text_nbest also gained a documented panic: they assert that the sentence is shorter than u16::MAX characters, because an edge now stores its start position as a u16. Every path through Segmenter is safe because it splits sentences at MAX_SENTENCE_BYTES; only code that drives a Lattice directly needs to split its own input.

JavaScript bindings: tokens are plain objects

Both JS bindings previously returned Token class instances. Those instances own memory outside the JavaScript heap, released only when the host runs a finalizer — which napi defers to the event loop, and which wasm-bindgen leaves to the caller. A synchronous loop that tokenizes a large input without yielding therefore accumulated memory even under forced GC. In v6 both bindings return plain objects instead, which the JavaScript GC owns outright.

Practical benefits: memory stays flat in batch loops, and results work directly with JSON.stringify, structuredClone, and worker transfer.

getDetail(i) is removed (Node.js and WASM)

A plain object has no methods, so read the array instead:

// v5
const pos = token.getDetail(0);

// v6
const pos = token.details[0];

Out-of-range reads now yield undefined rather than null.

Node.js: tokenizeObjects is removed

tokenizeObjects existed only as a v5-era workaround for the memory problem above. tokenize now returns the same plain objects, so drop the call:

// v5
const tokens = tokenizer.tokenizeObjects(text);

// v6
const tokens = tokenizer.tokenize(text);

tokenizeNbest returns plain objects too, so it no longer has the accumulation behavior documented as a known limitation in v5. NbestResult stopped being a class along with Token: it is now a plain object with the same tokens and cost properties.

Because neither is a class any more, they are no longer exported from the package. Token, JsToken, NbestResult, and JsNbestResult are all gone from index.js, so require("lindera").Token is undefined and instanceof checks against them no longer compile or run:

// v5
const { Token } = require("lindera-nodejs");
if (tokens[0] instanceof Token) { /* ... */ }

// v6 — the values are plain objects; check a property instead
if (typeof tokens[0].surface === "string") { /* ... */ }

TypeScript users: the interface describing a token is now called Token (it was JsTokenData while Token was taken by the class).

WASM: field names are now camelCase

The removed Token class exposed snake_case fields while its own toJSON() already emitted camelCase. v6 settles on camelCase, matching the Node.js binding:

// v5
token.byte_start, token.byte_end, token.word_id, token.is_unknown

// v6
token.byteStart, token.byteEnd, token.wordId, token.isUnknown

surface, position, and details are unchanged. Code that already called toJSON() and read the result was using these names, and keeps working — except that toJSON() itself is gone, since the token is already a plain object:

// v5
const data = token.toJSON();

// v6
const data = token; // already plain

Token is also no longer exported from the module, so import { Token } from 'lindera-wasm' fails. There is nothing to import in its place — tokenize hands back plain objects directly.

tokenizeNbest in WASM already returned plain objects in v5 and is unchanged.

Who does not need to act

• Python, Ruby, and PHP bindings: unaffected by the JavaScript changes above. They keep their Token classes, because their finalization is deterministic and never had the memory problem that motivated the JS rework. They each gained a plain-data conversion method — to_dict(), to_h (also to_hash), and toArray() respectively — but nothing was removed or renamed.
• User dictionaries loaded from .csv: compiled at load time, so they pick up the new format automatically. (Prebuilt .bin files do need rebuilding — see the first section.)
• lindera crate's public API: Segmenter, Tokenizer, SegmentWorker, and AnalysisWorker signatures are unchanged (aside from the additive unknown_word_ladder/max_grouping_len options); code that loads a dictionary from a path or URI compiles and runs unchanged once the dictionary itself is rebuilt or re-downloaded, and embed-* dictionaries are compiled by each dictionary crate's build script, so they always match the running version.

Upgrade checklist

Everyone:

• Rebuild prebuilt user-dictionary .bin files from their CSV source with lindera build --user. This is required regardless of which v5 you are coming from.
• If you pin exact tokenization output for out-of-vocabulary or non-Japanese text, re-check it — the Decompose penalty fix and the unknown-word ladder (default on) can both change it. Set unknown_word_ladder(false) if you need v5-identical unknown-word output.

Package names (see Package renames on npm, PyPI, and crates.io):

• npm: depend on lindera instead of lindera-nodejs, and on lindera-wasm instead of lindera-wasm-web / lindera-wasm-bundler; update require/import specifiers to match.
• PyPI: pip install lindera instead of lindera-python; import lindera is unchanged.
• Former lindera-wasm-bundler users: add await __wbg_init() before the first API call — the consolidated package is a web-target build and does not initialize implicitly.

JavaScript (Node.js and WASM):

• Replace token.getDetail(i) with token.details[i].
• Node.js: replace tokenizeObjects(text) with tokenize(text); stop importing Token/NbestResult (they are no longer exported); TypeScript users rename the JsTokenData type to Token.
• WASM: switch token field reads to camelCase (byteStart, byteEnd, wordId, isUnknown), drop toJSON() calls, and stop importing Token.

Direct lindera_dictionary users:

• Update call sites per the Rust API table above, including the two new set_text/set_text_nbest arguments.

Upgrading from v5.2 or earlier (these shipped in v5.3.0):

• Rebuild self-built system dictionaries with lindera build, or re-download prebuilt ones with lindera download.
• WASM: update loadDictionaryFromBytes() calls to the 9-argument signature (dictTrie and dictValsIdx in place of dictDa).
• WASM: remove v5-era dictionaries from OPFS and download v6 archives.