Migrating from v6 to v7

Lindera v7.0.0 turns the lindera crate into a facade: it re-exports the morphological segmenter, now published as lindera-segmenter, and — through the new analysis feature, on by default — the analysis chain of lindera-analysis as lindera::analysis. One dependency line gives you the Segmenter, the Tokenizer, and every filter. The release also renames lindera-binding-core to lindera-binding and removes the deprecated LINDERA_DICTIONARIES_PATH fallback, as announced in v5.0.0, corrects the IPADIC schema, which had the conjugation_type and conjugation_form names swapped, keeps dictionary entries whose surface is whitespace or starts or ends with it, skips whitespace in the lattice as MeCab does (SudachiDict excepted, as Sudachi keeps it), resolves overlapping char.def lines as MeCab does, carries the context across 、 and 。 within a line as MeCab does, returns N-best results that each cover the whole input, lets unknown words start at every position as MeCab does, creates them from each character's default category and groups the characters that share a category, as MeCab does, keeps the dashes and tildes of dictionary entries as written instead of rewriting them, removing the normalize_details setting, outputs the first CSV row among tied dictionary entries as MeCab does, chooses the segmentation whose last word starts later when two cost the same, as MeCab does, and has the lattice backtraces of lindera-dictionary return each token's end offset. The bundled CC-CEDICT dictionary gets new costs, with which unknown words no longer cost less than dictionary words, so that runs of Latin letters, kana and digits stay one word. lindera train builds its lattices as MeCab's learner does, with the seed words and the unknown words that the runtime creates, and keeps every unk.def entry, so models trained with earlier versions must be trained again, and lindera build rejects a dictionary whose context IDs fall outside its connection matrix, as building a tokenizer does for such a user dictionary. Dictionary directories built or downloaded with v6.2.0 or earlier no longer load and must be rebuilt or downloaded again (dictionary format version 3). Building Lindera now takes Rust 1.89 or newer. This guide lists every breaking change and the one-line fixes for each.

Overview

The language bindings (Python, Node.js, Ruby, PHP, WASM) and the CLI keep their package names, and their APIs apart from the mode and one setting: a binding takes the mode as a Mode, which holds the penalty of decompose mode as the Rust Mode does, instead of a mode name (see Bindings: a mode is a Mode), and the Metadata classes of the Python, Node.js, Ruby and PHP bindings no longer take or expose normalize_details (see The normalize_details setting is removed). The PHP binding also rejects a value of the wrong type for an optional argument, which v6 read as null (see PHP: optional arguments reject a value of the wrong type), and the PHP, Node.js and WASM bindings reject an integer argument outside the range of its setting, which v6 wrapped (see Bindings: integer arguments outside their range are rejected). Their version number moves to 7.0.0. Like the Rust crates, they no longer load a dictionary directory built or downloaded with v6.2.0 or earlier (see Dictionary directories must be rebuilt). A model file written by their training functions, or by lindera train, with v6.2.0 or earlier has to be trained again (see lindera train builds its lattices as MeCab's learner does). There are fourteen output differences, all described below; the last two are in the CLI only and in the WASM binding only. With IPADIC or IPADIC-NEologd, the values reported under the names conjugation_type and conjugation_form trade places. The dictionaries now contain the entries whose surface is or starts or ends with whitespace, which changes the segmentation of text that contains such whitespace. Text that contains whitespace is segmented as MeCab segments it, except with SudachiDict, which keeps whitespace in the lattice as Sudachi does. Ð, 々 and 〇 get the character categories that MeCab gives them, which changes the segmentation of text that contains them. Within a line, the words after 、 and 。 are read in the context of the words before them, which changes the segmentation of some such text and the N-best costs of every such line. For input with more than one sentence, the N-best results from the second one on are the cheapest segmentations of the whole input, and an empty input gets one N-best result without tokens. And in normal mode, an unknown word can start inside a run of characters of one kind, as in MeCab, which splits some runs of katakana, symbols and Latin letters. Unknown words come from each character's default category, and a grouped unknown word goes on while each character shares a category with the one before it, as in MeCab, which makes a ko-dic hanja numeral and the hanja after it one word and 〇 a symbol with ko-dic. With IPADIC and IPADIC-NEologd, the entries spelled with ― or ～ are found under that spelling, as in MeCab, which changes the segmentation of text that contains ―, —, ～ or 〜. Among dictionary entries that share the surface, the context IDs and the cost, the first CSV row is output instead of the last, as in MeCab, which changes the reading, the base form or other details of such words, and the first N-best result is always the 1-best. When two segmentations cost exactly the same, the one whose last word starts later is chosen, as in MeCab, which changes the segmentation of a few lines with the Japanese and Korean dictionaries and Jieba and of about a quarter of the sentences with CC-CEDICT. With CC-CEDICT, unknown words no longer cost less than dictionary words, which makes entries such as T恤 one word and gives 〇 and entries such as PK their pinyin and definition. Jieba's costs are trained again on UD Chinese GSD, which segments Chinese text as that treebank does and raises segmentation F1 on its test split from 77.3 to 87.7, and 〇 is a numeral entry, one word with its pinyin instead of part of an unknown word. And lindera tokenize removes only the line terminator from each input line instead of all the whitespace at its ends, which keeps U+3000 at the start or end of a line and makes the offsets index the line, and it writes a result for every line, also in the wakati format and with -N. And the WASM binding's TokenizerBuilder applies all its settings and the user dictionary whichever way the dictionary was set, where v6 ignored some of them. Otherwise, for the same input and dictionary, v7.0.0 produces the same tokens with the same positional details as v6.2.0.

The lindera crate is now a facade

Up to v6 the Rust API was split between two crates that you had to depend on separately: lindera (the segmenter and the dictionary loading API) and lindera-analysis (character filters, token filters, and the Tokenizer). In v7 the lindera crate is a thin facade over both:

lindera ──┬─▶ lindera-segmenter ──┬─▶ lindera-dictionary
          │                       ├─▶ dictionary crates   [features: embed-*]
          │                       └─▶ lindera-trainer     [feature: train]
          └─▶ lindera-analysis    [feature: analysis, default on] ──▶ lindera-segmenter

Its public tree keeps every lindera::… path of v6 and adds lindera::analysis:

lindera
├── LinderaResult, get_version()
├── dictionary, error, mode, segmenter, space_penalty, token, worker   (from lindera-segmenter, as in v6)
└── analysis   [feature: analysis, default on]   (lindera-analysis: character_filter, token_filter, tokenizer, worker)

One dependency instead of two

# v6
[dependencies]
lindera = "6"
lindera-analysis = "6"

# v7
[dependencies]
lindera = "7"

#![allow(unused)]
fn main() {
// v6
use lindera_analysis::tokenizer::Tokenizer;
use lindera_analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;

// v7
use lindera::analysis::tokenizer::Tokenizer;
use lindera::analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
}

Keeping a direct lindera-analysis = "7" dependency also works: lindera::analysis is that same crate, so lindera_analysis::tokenizer::Tokenizer and lindera::analysis::tokenizer::Tokenizer are one type and existing use paths need not change.

Update lindera and lindera-analysis together

If you keep both dependencies, bump them in the same step. lindera-analysis 6.x depends on the lindera 6.x segmenter, so a manifest with the lindera line moved to "7" while lindera-analysis stays at "6" pulls two versions of the segmenter into the build:

# Two Segmenter types: lindera 7 (via lindera-segmenter) and lindera 6 (via lindera-analysis 6)
[dependencies]
lindera = "7"
lindera-analysis = "6"

lindera::segmenter::Segmenter is then the v7 type while lindera_analysis::tokenizer::Tokenizer::new expects the v6 one, and the call fails to compile with a type mismatch. Move both lines to "7" at once, or drop lindera-analysis as shown above.

default-features = false now means segmenter only

The facade's default features are mmap and analysis. With default-features = false you get the pure segmenter, and both are off: lindera::analysis does not exist, and memory-mapped loading is no longer the default for filesystem dictionaries (as in v6, where default-features = false also turned mmap off). Re-enable what you need:

# Segmenter with memory-mapped loading (the v6 default build)
lindera = { version = "7", default-features = false, features = ["mmap"] }

# Segmenter and analysis chain, without memory-mapped loading as the default
lindera = { version = "7", default-features = false, features = ["analysis"] }

memmap2 itself stays linked either way through lindera-dictionary's default features; the facade's mmap feature only selects the default for use_mmap in load_dictionary and friends, and load_dictionary_with_options(uri, true) works without it.

The segmenter is published as lindera-segmenter

The lindera crate of v6 and earlier continues as lindera-segmenter 7.0.0, with the same modules (dictionary, error, mode, segmenter, space_penalty, token, worker) and the same API. The facade re-exports every one of them at its root, so every lindera::… path that compiled in v6 compiles in v7. The crate exists so that lindera-analysis has a segmenter to build on without depending on the facade that re-exports it; nobody needs to depend on lindera-segmenter directly, and application code should keep depending on lindera.

lindera::dictionary exposes the raw dictionary crate

Code that reached into the building blocks of lindera-dictionary — the dictionary data structures, the dictionary builder, the Viterbi lattice — used to need a direct lindera-dictionary dependency. In v7 these modules are re-exported inside lindera::dictionary, next to the high-level loading API:

lindera::dictionary
├── Dictionary, Metadata, Schema, load_dictionary, load_user_dictionary, DictionaryKind, …   (unchanged)
├── core          lindera_dictionary::dictionary (prefix_dictionary, character_definition, connection_cost_matrix, unknown_dictionary, …)
├── builder, error, loader, mode, nbest, space_penalty, util, viterbi
├── embedded_dictionary!, include_bytes_aligned!
└── trainer       lindera_trainer, with the `train` feature (unchanged)

All existing lindera::dictionary::* paths stay valid, and the re-exports are the same items — lindera::dictionary::Dictionary and lindera::dictionary::core::Dictionary are one type. A direct lindera-dictionary dependency keeps working; it is simply no longer required:

#![allow(unused)]
fn main() {
// v6: needs lindera-dictionary in Cargo.toml
use lindera_dictionary::dictionary::prefix_dictionary::PrefixDictionary;
use lindera_dictionary::viterbi::Lattice;

// v7: reachable through the facade
use lindera::dictionary::core::prefix_dictionary::PrefixDictionary;
use lindera::dictionary::viterbi::Lattice;
}

[!NOTE] lindera::dictionary::core shares its name with Rust's core crate. A glob import such as use lindera::dictionary::*; brings that module into scope and shadows the core crate for the rest of the module, so paths like core::fmt::Debug stop resolving to the standard library there. Import the items you need by name instead.

Implicit dictionary features removed

In v6 the optional dictionary dependencies of lindera doubled as Cargo features: features = ["lindera-ipadic"] was accepted, and pulled in the crate without embedding a dictionary. The facade has no dictionary dependencies of its own, so these seven implicit features no longer exist: lindera-ipadic, lindera-ipadic-neologd, lindera-unidic, lindera-sudachidict, lindera-ko-dic, lindera-cc-cedict, and lindera-jieba. Cargo rejects a manifest that names one of them. Use the embed-* features, which have always been the documented way to embed a dictionary:

# v6
[dependencies]
lindera = { version = "6", features = ["lindera-ipadic"] }

# v7
[dependencies]
lindera = { version = "7", features = ["embed-ipadic"] }

lindera-binding-core renamed to lindera-binding

This only matters if you build your own language binding on the shared helper crate. lindera-binding 7.0.0 depends on the lindera facade (with the analysis feature) instead of on lindera plus lindera-analysis; its own API — CoreTokenizerBuilder, CoreTokenizer, TokenView, and the argument, metadata, and schema helpers — is unchanged, except that CoreMetadata no longer has normalize_details (see The normalize_details setting is removed). The published lindera-binding-core releases (4.0.0 through 6.2.0) stay on crates.io as they are and are not yanked, but receive no further versions.

# v6
[dependencies]
lindera-binding-core = "6"

# v7
[dependencies]
lindera-binding = "7"

#![allow(unused)]
fn main() {
// v6
use lindera_binding_core::tokenizer::CoreTokenizer;

// v7
use lindera_binding::tokenizer::CoreTokenizer;
}

LINDERA_DICTIONARIES_PATH removed

v5.0.0 renamed the build-time dictionary cache variable to LINDERA_BUILD_DICTIONARY_CACHE_DIR and kept the old name, LINDERA_DICTIONARIES_PATH, as a deprecated fallback with a build warning (see Migration v4 to v5). v7.0.0 removes the fallback. A build that sets only the old name no longer uses the cache — and no longer warns about it: the dictionary crates download and build their dictionary as if no cache were configured. Rename the variable wherever it is set (shell profiles, CI configuration, container images):

# v6
export LINDERA_DICTIONARIES_PATH=/path/to/cache

# v7
export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache

Rust 1.89 or newer is required

v7.0.0 declares rust-version = "1.89"; v6.2.0 declared 1.88. The build scripts of the dictionary crates now lock the entry of LINDERA_BUILD_DICTIONARY_CACHE_DIR they write with File::lock, which Rust 1.89 stabilized, so that builds sharing the cache no longer remove each other's files. With an older toolchain, cargo stops with a message that names the required version. Update the toolchain:

rustup update stable

Dictionary directories must be rebuilt

v7.0.0 writes dictionaries in format version 3 and rejects a dictionary directory built or downloaded with v6.2.0 or earlier, which is at version 2, when it loads it:

Dictionary 'ipadic' has format version 2, but this build of Lindera reads format version 3. To fix this, rebuild it with `lindera build`, or download a matching prebuilt dictionary with `lindera download`.

Version 3 lists each code point's default category first in char_def.bin, because unknown words now come from that category only (see Unknown words come from the default category, as in MeCab). A version 2 file has the same layout, so without the check it would load and silently give some characters the wrong default category. The version also covers the other changes in what the dictionary builder writes, which a v6.2.0 dictionary does not have: the IPADIC conjugation names, the entries with whitespace, the overlapping char.def lines and the dash and tilde spellings.

IPADIC conjugation field names corrected

IPADIC stores 活用型, the conjugation type (for example 五段・カ行イ音便), in column 8 and 活用形, the conjugation form (for example 連用タ接続), in column 9. From v1.0.0 through v6.2.0, the lindera-ipadic and lindera-ipadic-neologd schemas named the two columns the other way round, so every lookup by name returned the other value. v7.0.0 corrects the names. UniDic and SudachiDict were already correct.

For the token 書い in 書いた:

Only access by name changes: Token::get("conjugation_type") and Token::get("conjugation_form"), the JSON from Token::as_value() and lindera tokenize -o json, and the field names in the dictionary schema that the bindings expose. Positional access — details, and the MeCab and wakati output formats — is unchanged, because the values were always in the right columns. If your code reads these fields by name, or swapped them back as a workaround, update it.

The names come from the metadata.json stored in each built dictionary, not from the library. The embedded dictionaries (embed-ipadic, embed-ipadic-neologd) and the dictionaries that the 7.0.0 CLI fetches with lindera download carry the corrected names. A dictionary directory built or downloaded with v6.2.0 or earlier no longer loads (see Dictionary directories must be rebuilt); rebuild it with the v7.0.0 lindera-ipadic/metadata.json (or lindera-ipadic-neologd/metadata.json), or download the 7.0.0 release asset.

If you build dictionaries with your own copy of the IPADIC metadata.json, swap "conjugation_form" and "conjugation_type" in its dictionary_schema.fields list.

Dictionary entries with whitespace are kept

Up to v6.2.0, the dictionary builder trimmed the surface of every lexicon entry. An entry whose surface is whitespace was dropped, and an entry that starts or ends with whitespace was stored under the trimmed surface, where a cheaper whitespace variant could take over the entry without whitespace. v7.0.0 reads the surface exactly as written, as MeCab and Sudachi do:

Entries that end with whitespace (55 in IPADIC-NEologd, 4 in ko-dic and 4 in SudachiDict) now match only where the text has that whitespace, as in MeCab: with IPADIC-NEologd, the entry GeForce GTX Titan X followed by a space matches GeForce GTX Titan X です but no longer GeForce GTX Titan Xです. The token ends with that whitespace, as in MeCab, whether whitespace stays in the lattice (keep_whitespace(true), or skip_whitespace(false), SudachiDict's default) or is skipped (the default for the other dictionaries, see the next section). Skipping leaves out only the whitespace after the entry: in GeForce GTX Titan X です, with two spaces, the token ends with the first space. 50 of the 55 IPADIC-NEologd entries end with a space, as do all those of ko-dic and SudachiDict; the other 5 end with U+3000 or U+00A0, which are not SPACE characters and are never skipped. The Decompose length penalty counts the entry's own whitespace, as it does when whitespace stays in the lattice. With unique, N-best results that differ only in that whitespace are no longer folded into one, such as ko-dic's 에듀 followed by the space (NNG) and 에듀 (NNP) in 에듀 센터. A word right after such an entry pays no left-space penalty (ko-dic), as in mecab-ko, whose penalty applies only after whitespace that the word before does not end with: 에듀 이 reads as 에듀 followed by the space and 이/VCP.

U+3000 is not in the SPACE character category, so it is a token even with keep_whitespace(false), as in MeCab. To remove it, drop 記号,空白 (IPADIC) or 空白 (UniDic) with the japanese_stop_tags token filter, or turn it into U+0020 with the unicode_normalize character filter (NFKC).

The change is in the dictionary builder. The embedded dictionaries and the dictionaries that the 7.0.0 CLI fetches with lindera download include these entries; a dictionary directory built or downloaded with v6.2.0 or earlier no longer loads (see Dictionary directories must be rebuilt).

Whitespace is skipped in the lattice

With keep_whitespace false (the default), v7.0.0 skips whitespace in the Viterbi lattice, as MeCab does: the word after a space connects directly to the word before it. v6 dropped whitespace from the output but kept it in the lattice as the SPACE unknown word, so the words around a space connected to that node instead of to each other. How much that changed depends on the dictionary's connection costs for the node: in ko-dic they are all zero, so every space cut the context between the words around it; in IPADIC and UniDic they are not zero, and the node's costs stood in for the connection between the words.

The output changes only for sentences that contain whitespace. That includes a sentence that ends with the \n or \t it was split at, such as a line that does not end with 。: its last word now connects to the end of the sentence directly.

The default comes from the dictionary's metadata.json; a dictionary that does not set skip_whitespace skips. A SudachiDict built by Lindera 6.x has no such setting, but it no longer loads anyway; the rebuilt one has it.

Token surfaces and offsets never include the skipped whitespace; whitespace that belongs to a dictionary entry, inside it or at its end, stays in that entry's token, as in MeCab (see the previous section). Skipping does not change the segmentation of text without whitespace. keep_whitespace(true) keeps whitespace in the lattice, so its output is the same as in v6; beyond the whitespace tokens, it can differ from the default v7 output, which skips whitespace.

To get the v6 handling of whitespace back, turn skipping off; whitespace is still dropped from the output. This does not undo the other output changes in this guide, such as Unknown words start at every position, as in MeCab. Leaving skip_whitespace out of the configuration, or setting it to null, keeps the dictionary's default:

#![allow(unused)]
fn main() {
// Segmenter
let segmenter = Segmenter::new(Mode::Normal, dictionary, None).skip_whitespace(false);
// SegmentWorker
worker.set_skip_whitespace(false);
}

# Configuration file / TokenizerBuilder::set_segmenter_skip_whitespace(false)
segmenter:
  skip_whitespace: false

lindera tokenize --disable-skip-whitespace

The language bindings set it on TokenizerBuilder: set_skip_whitespace(False) in Python, set_skip_whitespace(false) in Ruby, and setSkipWhitespace(false) in Node.js, PHP and WASM. null (None in Python, nil in Ruby) goes back to the dictionary's default.

Overlapping char.def lines are resolved as in MeCab

A dictionary's char.def can list the same code point on several lines. Up to v6.2.0, the dictionary builder gave such a code point the categories of every line that covers it. v7.0.0 uses only the last line, as MeCab, vibrato and Kuromoji do. In the bundled dictionaries, three characters change (ko-dic calls the kanji categories HANJA and HANJANUMERIC, and CC-CEDICT and Jieba call them CHINESE and CHINESENUMERIC):

The kanji numerals 一 to 九, 十, 百, 千, 万, 億 and 兆 are also listed on two lines, but keep their categories.

lindera train now reads char.def with the dictionary builder. It used to keep only the first category of each line, skip lines that name a single code point (such as 0x0020 SPACE), and resolve overlaps by start position. The %t feature template now gives the first category of the last line that covers the character, which is MeCab's default type. The bundled training files have neither kind of line, so the models trained from them do not change.

A range line without a category is now an error in lindera build and lindera train, as in MeCab; it would otherwise reset the earlier lines to DEFAULT.

The change is in the dictionary builder. The embedded dictionaries and the dictionaries that the 7.0.0 CLI fetches with lindera download have the new categories; a dictionary directory built or downloaded with v6.2.0 or earlier no longer loads (see Dictionary directories must be rebuilt).

The context is carried across 、 and 。

The segmenter cuts the input into sentences at \n, \t, 。 and 、, and after 32 KiB without any of them, to keep each lattice small. Up to v6, every sentence started from the beginning-of-sentence context (BOS) and ended with the connection to the end of the sentence (EOS), so the word after 、 or 。 was read as if it began a text. MeCab does not cut inside a line. v7.0.0 carries the context across the cuts at 、 and 。 and across the forced cuts: the next sentence starts from the words that end the previous one, and EOS is paid only at a \n, a \t or the end of the input. The sentences up to that point form a segment, and segments stay independent. The best path of a segment, and its cost, are those of one lattice over the whole segment, which is what MeCab gives for a line:

On the sentences of UD Japanese GSD that contain 、, the change alters 50 lines with UniDic: 27 come closer to the gold segmentation and part-of-speech tags, 7 move away (sign test p = 0.0008), and 16 are ties. Of 505 paragraphs of Botchan, the number that Lindera segments and tags exactly as MeCab does went from 352 to 438 with IPADIC and from 428 to 503 with UniDic. Text without 、, 。 and forced cuts, such as text cut only at \n and \t, is not affected. Carrying the context costs the 1-best segmentation about 1% more instructions.

The result can still differ from one lattice over the segment:

• No word spans a cut. A dictionary entry that contains 、 or 。 never matches: MeCab reads UniDic's 一、二塁 as one noun, Lindera as 一, 、 and 二塁. An unknown word is not grouped across a cut, and a word that would span a forced cut is split there.
• The left-space penalty (ko-dic) does not apply to a word that starts right at a forced cut, even when whitespace precedes the cut.

The N-best results follow the same model. Within a segment, they are exactly the N best paths of one lattice over the segment: unique compares the word boundaries of the whole segment, and the cost threshold is measured from the first result. Across segments, the results are the cheapest combinations of one path per segment (see the next section). The first result is the 1-best segmentation. The costs of a line with 、 or 。 change, the first result's included, because they now include the connections across the cuts and the EOS connection only once.

There is no setting to restore the v6 behavior.

N-best results cover the whole input

The N-best methods (Segmenter::segment_nbest, Tokenizer::tokenize_nbest, their _with_lattice and worker forms, lindera tokenize -N and the bindings' N-best methods) cut the input into sentences as the 1-best segmentation does. Up to v6, they searched each sentence on its own, and the k-th result joined the k-th path of every sentence. From the second result on, that changed every sentence at once, so the results were not the cheapest segmentations of the input and their costs could go down, and a sentence with fewer than k paths was missing from the k-th result. v7.0.0 searches the input segment by segment (see the previous section) and returns the N cheapest combinations of one path per segment, in ascending order of total cost, and every result covers the whole input:

Each of these lines is one segment, so its v7 results are the three best paths of one lattice over the line, the ones MeCab returns. The first result is the 1-best segmentation, and the output for input with a single sentence does not change. The cost threshold (cost_threshold, --nbest-cost-threshold) is now measured over the whole input: a result is kept when its total cost is within the threshold of the first result's. v6 applied the threshold to each sentence separately. With unique, the results still have distinct word boundaries.

An empty input now gets one result without tokens, with the connection cost from BOS to EOS, as in MeCab and as an input of only whitespace already did; v6 gave no result, because an empty input has no sentence. This applies to segment_nbest, tokenize_nbest, their worker forms, the bindings' N-best methods and lindera tokenize -N, which now writes a result for an empty line.

Unknown words start at every position, as in MeCab

A word that is not in the dictionary is read as an unknown word, built from the character categories of the dictionary's char.def (katakana, Latin letters, digits, symbols and so on). Most categories are set to group: a run of characters that share a category, such as a run of katakana, becomes one candidate, a grouped unknown word. Up to v6, normal mode created no unknown words at the positions inside the last grouped unknown word, a shortcut inherited from Kuromoji that MeCab does not have. After a dictionary word that ends inside such a run, no unknown word could continue the path, so the grouped unknown word that swallowed the dictionary word was often the only way through. v7.0.0 creates unknown-word candidates at every position that a path reaches, in every mode, under MeCab's condition: the category that creates them is set to always create them (INVOKE in char.def), or no dictionary word starts at that position. Decompose mode already did this, and its output does not change.

• Japanese: IPADIC, IPADIC-NEologd and UniDic put ・ in the katakana range of their char.def. After a dictionary word and ・, an unknown katakana word could not start, so ジョン・レノン became one unknown word. Now the unknown word starts after them, as in MeCab. A run that starts with an unknown word, such as レノン・ジョン, stays one unknown word, in MeCab too. In the same way, a dictionary symbol is no longer merged into the unknown symbols after it.
• Korean: a sentence-final .", ?", !" or .' was one symbol (SY). Now ., ? and ! are sentence-final punctuation (SF), as in MeCab, and the ending before them can change with it, as in "좋아.".
• Chinese: Jieba was not affected with the mecab-jieba-0.1.2 dictionary (not measured with 0.1.3, see Jieba: costs retrained on UD Chinese GSD), and CC-CEDICT keeps Latin words and runs of kana in one word, as in v6. With the costs of the CC-CEDICT dictionary of v6 (CC-CEDICT-MeCab-0.1.0-20200409), the change would split them, as MeCab does with that dictionary: every unknown word costs −3,200 whatever its length, so a path with more words costs less, and T (an entry, −400) followed by the unknown word he (−3,200) beats the unknown word The. The shortcut hid this. v7.0.0 builds CC-CEDICT with new costs (see CC-CEDICT: unknown words no longer cost less than dictionary words).
• SudachiDict output can change in the same kinds of runs.
• N-best: the first result changes as the 1-best segmentation does, and the other results can now include splits such as G and oogle for Google (UniDic).

With dictionaries built from the same sources, the number of lines that Lindera segments and tags exactly as MeCab does rises, on top of the other changes in this guide, from 589 to 596 of 600 Japanese sentences and from 438 to 459 of 505 paragraphs of Botchan with IPADIC, from 590 to 596 and from 441 to 459 with IPADIC-NEologd, and from 877 to 1,498 of 1,500 Korean sentences with ko-dic (with the left-space penalty off, as MeCab has none). No line that matched before stops matching, and UniDic is unchanged on these texts. On UD Japanese GSD with UniDic, 64 sentences change: 53 come closer to the gold segmentation and part-of-speech tags, 10 move away (sign test p = 3.4 × 10⁻⁸), and 1 is a tie. The 10 all match MeCab's output; GSD keeps some names joined by ・ and some numbers as one word.

Text with many runs of katakana or Latin letters costs more, because every position inside a run now gets candidates, as in MeCab. Against the same release without this change, counting the speedups made with it, the instruction count rises by 13% on UD Japanese GSD (Wikipedia text; 14% to 17% for N-best), and falls by 0.8% on Botchan and on Korean and by 14% to 16% in Decompose mode. A line of 10,240 katakana without a delimiter costs 3 to 5 times as much in 1-best and 3 to 15 times in N-best, and the peak memory of lindera tokenize -N 3 on it grows from 12 to 56 MiB with IPADIC and from 18 to 127 MiB with UniDic.

There is no setting to restore the v6 behavior. unknown_word_ladder(false) (--disable-unknown-word-ladder) still turns off the shorter unknown-word candidates, but no longer reproduces the output of Lindera before v6.

Unknown words come from the default category, as in MeCab

A character can have more than one category in char.def: the kanji numerals are KANJINUMERIC KANJI in IPADIC and UniDic, HANJANUMERIC HANJA in ko-dic and CHINESENUMERIC CHINESE in CC-CEDICT and Jieba, and 〇 (U+3007) is SYMBOL plus the numeral category in all of them but Jieba (see Jieba: 〇 is a numeral entry). The first category of the line that decides a character's categories is its default category. Up to v6.2.0, Lindera created unknown-word candidates for every category of a character, and went on with a grouped unknown word only while the next character had the same category at the same position of its category list. v7.0.0 follows MeCab:

• Unknown words come from the default category only, with its INVOKE, GROUP and LENGTH settings and its unk.def entries.
• A grouped unknown word goes on while each character shares a category with the one before it.
• The shorter candidates of the length ladder go on while each character shares a category with the first one.

• Korean: many ko-dic numerals, such as 三 and 十, are not dictionary words, and their default category HANJANUMERIC groups, so a numeral and the hanja after it become one SH word, as in MeCab.
• 〇: its default category is SYMBOL, which creates unknown words even where a dictionary word starts, so it is a symbol with ko-dic (SY). CC-CEDICT has 〇 as an entry, which beats such a group with the costs of the v7.0.0 dictionary, so 二〇二六年 stays 二, 〇, 二, 六 and 年 (see CC-CEDICT: unknown words no longer cost less than dictionary words). With Jieba, whose group ran on through the numerals and the Chinese characters after them, 〇 is now a numeral entry (see Jieba: 〇 is a numeral entry).
• Japanese: IPADIC, IPADIC-NEologd and UniDic have the kanji numerals and 〇 as dictionary words, and no line of the texts compared changes. SudachiDict was not compared.

With dictionaries built from the same sources, the number of lines that Lindera segments and tags exactly as MeCab does rises, on top of the other changes in this guide, from 3,913 to 3,917 of the 3,919 non-empty lines of Sangnoksu with ko-dic (with the left-space penalty off, as MeCab has none); every generated sentence that combines the characters with several categories with other characters now matches MeCab with IPADIC, IPADIC-NEologd, UniDic, ko-dic, CC-CEDICT and Jieba. No line that matched before stops matching. The instruction count falls by 0.3% to 4.8%.

CharacterDefinition::lookup_categories (in lindera_dictionary::dictionary::character_definition) returns the default category first, then the other categories in category id order. The 7.0.0 char_def.bin stores them in this order, which is why the dictionary format version changes (see Dictionary directories must be rebuilt).

There is no setting to restore the v6 behavior.

CC-CEDICT: unknown words no longer cost less than dictionary words

v7.0.0 builds the CC-CEDICT dictionary from CC-CEDICT-MeCab-0.1.0-20261021 instead of CC-CEDICT-MeCab-0.1.0-20200409 (#1128). The CC-CEDICT data (2020-04-09), char.def and the other entries are the same, and two kinds of cost change:

• Every unknown word costs 400 instead of −3,200.
• The 20 entries made only of Latin letters (such as T, PK, TA, cos and Tony) and the 2 made only of digits (88 and 996) cost 399, one less than an unknown word, instead of −400 to −3,200.

CC-CEDICT has no connection costs, so a path costs the sum of the costs of its words, and with the old costs a path with more unknown words cost less. Unknown words start at every position would then split Latin words and runs of kana: T (−400) and the unknown word he (−3,200) beat the unknown word The (−3,200). With the new costs, more unknown words cost more, and an entry made only of Latin letters or of digits wins only for exactly its own characters (399 against 400), never as the start of a longer run (399 + 400 against 400). Latin words, runs of kana and numbers stay one word, as in v6, and MeCab segments them the same way with the new dictionary.

What changes from v6 is that dictionary words now beat the unknown words that cover the same characters:

• An entry that mixes Latin letters or digits with Chinese characters, such as T恤, B超, AA制, 卡拉OK, 阿Q, 3C, U盘 or X光, used to lose to unknown words over its Latin letters or digits.
• 〇 and the entries made only of Latin letters or of digits come with their pinyin and definition, where an unknown word over the same characters used to win.

Against v7.0.0 with the 0.1.0-20200409 dictionary, 167 of the 4,997 sentences of UD Chinese GSD change: 163 come closer to the gold segmentation and 4 move away (sign test p = 3.4 × 10⁻⁴³). The 4 are CC-CEDICT words that GSD writes as two words (X光, T恤 and K書) or splits at · (喬治·華盛頓). The simplified UD Chinese GSDSimp gives the same counts. Lindera and MeCab agree on the same 4,951 GSD sentences with either dictionary (the other 46 differ where Lindera cuts the sentence after 。 or 、), and on every one of 8,064 generated sentences around the characters with several categories with the new one. The instruction count on GSD falls by 0.1%.

The costs of paths change with the costs of unknown words, so N-best costs and the cost threshold of N-best results are on a different scale. Entries of a user dictionary keep their own costs (−10,000 for the simple format, the default_word_cost of the metadata), so they still cost less than an unknown word.

A CC-CEDICT dictionary directory built from the 0.1.0-20200409 archive keeps the old costs (and has to be rebuilt anyway, see Dictionary directories must be rebuilt): build it from the 0.1.0-20261021 archive, as lindera-cc-cedict and the 7.0.0 release assets do. There is no setting to restore the old costs.

Jieba: 〇 is a numeral entry

v7.0.0 builds the Jieba dictionary from mecab-jieba-0.1.3 instead of mecab-jieba-0.1.1. Since 0.1.2 (#1162), it reads 〇 (U+3007 IDEOGRAPHIC NUMBER ZERO) as a numeral, like 一 to 兆; 0.1.3 trains the costs again (see Jieba: costs retrained on UD Chinese GSD):

• In char.def, 〇 is CHINESENUMERIC CHINESE, the categories of the other numerals, instead of SYMBOL CHINESENUMERIC.
• 〇 is an entry like 零: part of speech m, pinyin ling2 and definition zero.

With the 0.1.1 dictionary, 〇 was an unknown word. Since unknown words come from the default category, its SYMBOL group went on through the numerals, which are CHINESE too, and the Chinese characters after them, as in MeCab: 二〇〇八年北京奧運會 was 二 and 〇〇八年北京奧運會 (w). An entry alone would not stop the group: SYMBOL makes unknown words even where an entry starts, and with the connection costs of 0.1.1 one unknown word over 〇〇八年北京奧運會 costs less than the path through 北京奧運 and 會, because entering or leaving their context ID costs 32,767. With the numeral categories, which make no unknown word where an entry starts, and the entry, no unknown word starts at 〇:

With 0.1.2, only text with 〇 changed: none of the 4,997 sentences of UD Chinese GSD has 〇. Lindera and MeCab segment the same way with the 0.1.3 dictionary: 28 hand-written lines, 2,160 generated lines around the numerals and 〇, the 8,064 generated sentences around the characters with several categories, and 4,951 of the 4,997 GSD sentences, as many as with the older dictionaries. As with the other numerals, a character that starts no entry and the 〇 after it can make one unknown word, as in MeCab: 両〇頻 is 両〇 and 頻.

A Jieba dictionary directory built from the 0.1.1 archive keeps 〇 as a symbol (and has to be rebuilt anyway, see Dictionary directories must be rebuilt): build it from the 0.1.3 archive, as lindera-jieba and the 7.0.0 release assets do.

Jieba: costs retrained on UD Chinese GSD

v7.0.0 builds the Jieba dictionary from mecab-jieba-0.1.3, whose word and connection costs are trained again (#1250). Up to 0.1.2, every connection into or out of context ID 2, which 52 of the 55 parts of speech shared (362,085 of the 584,430 entries, such as every v, nr and ns), cost 32,767. Three defects added up: the training corpus never made a gold word a dictionary word, the trainer of lindera 2.3.3 put no dictionary words in its lattices, and its export wrote 32,767 for a connection without a trained weight. The best path was the one with the fewest such connections, close to longest match, and the word costs had never been trained against each other. 0.1.3 is trained with the trainer of v7.0.0 on the training splits of UD Chinese GSD and of the same sentences in simplified script (GSDSimp), with each gold word that is in the dictionary given the features of a dictionary entry. Every part of speech has its own context ID (a 56 × 56 matrix), and no connection costs 32,767.

Jieba now segments text as UD Chinese GSD does, which is finer than jieba and than 0.1.2. Both columns are v7.0.0; MeCab segments every line the same way with the same dictionary:

GSD's gold segmentation has 中華 / 人民 / 共和 / 國 (54 times) and 這 / 個 (69 times, never 這個). Segmentation F1 over word spans, with models trained on the training splits only:

MeCab 0.996 with the same dictionaries gives the same F1 within 0.09, and segments 4,951 of the 4,997 GSD sentences as Lindera does, with either dictionary. The parts of speech are jieba's, as before.

A simple user dictionary row (a surface, a part of speech and a pinyin) costs the default_word_cost of metadata.json. The trained word costs go down to −25,149, so lindera-jieba sets it to −32768 instead of 1625: with 1625, or with the −10000 of the other dictionaries, the user word 羽田机场 lost to 羽田 and 机场. Of user words made of two adjacent GSD words that are not in the dictionary, one per sentence of the test splits, −32768 keeps 99.4% as one word, −10000 53% to 58% and 1625 4%. A detailed row gives its own context IDs and cost: the IDs of 0.1.2 (0 to 6) name other parts of speech in 0.1.3, so take them from the left-id.def and right-id.def of the 0.1.3 archive.

A Jieba dictionary directory built from an older archive has to be rebuilt from mecab-jieba-0.1.3, as lindera-jieba and the 7.0.0 release assets are. There is no setting to restore the old costs.

Dash and tilde spellings are kept as written

Japanese text spells the dash and the tilde in two ways each: the dash as ― (U+2015, HORIZONTAL BAR) or — (U+2014, EM DASH), the tilde as ～ (U+FF5E, FULLWIDTH TILDE) or 〜 (U+301C, WAVE DASH). Up to v6.2.0, the dictionary builder rewrote ― to — and ～ to 〜 in every entry of IPADIC and IPADIC-NEologd, whose metadata.json turned the rewrite on with normalize_details. It rewrote the surface, the spelling under which the segmenter finds the entry in the text, and the detail fields, such as the base form and the reading. The text to segment was not rewritten, so an entry that the dictionary source spells with ― or ～ matched only text that spelled it with — or 〜. The rewrite was a workaround for an old EUC-JP conversion. The dictionary sources are UTF-8 now, and ― is the dash that the common Shift_JIS and EUC-JP converters produce and that Aozora Bunko texts use. v7.0.0 keeps every entry as written in the dictionary's CSV files, as MeCab does:

The detail fields keep the CSV spelling as well: the entry ＣＤ―ＲＯＭ gives the base form ＣＤ―ＲＯＭ, where v6 gave ＣＤ—ＲＯＭ.

With dictionaries built from the same sources, the number of lines that Lindera segments and tags exactly as MeCab does rises, on top of the other changes in this guide, from 459 to 503 of 505 paragraphs of Botchan (from 2,690 to 2,744 of its 2,746 sentences) and from 3,377 to 3,581 of 3,587 paragraphs of Kokoro and I Am a Cat, two other novels from Aozora Bunko. The figures are the same with IPADIC and IPADIC-NEologd, and no line that matched before stops matching.

Text that writes these characters as — or 〜 no longer finds the entries that it found only through the rewrite: 12 in IPADIC, all spelled with ―, and 1,085 in IPADIC-NEologd, most of them spelled with ～. In the table above, ＣＤ—ＲＯＭ is split and —— is an unknown noun, as in MeCab. IPADIC-NEologd has most of its words with a tilde in both spellings, so text with either tilde still finds them.

To have both spellings find the entries, map the text to the dictionary's spelling with the mapping character filter. IPADIC spells all its entries with a dash with ― and all those with a tilde with 〜, so map — to ― and ～ to 〜:

lindera tokenize --dict ipadic --char-filter 'mapping:{"mapping":{"—":"―","～":"〜"}}'

# Configuration file
character_filters:
  - kind: mapping
    args:
      mapping:
        "—": "―"
        "～": "〜"

With this filter, ＣＤ—ＲＯＭ is one word and —— is 記号,一般 again, and text with ～ also finds IPADIC's entries with 〜, which it did not in v6 either. The token's surface is the mapped text, such as ＣＤ―ＲＯＭ; its byte offsets point into the original text. Mapping the tilde the other way, 〜 to ～, would hide IPADIC's entries with 〜. With IPADIC-NEologd, a mapping trades words: some of its words exist in only one spelling, 129 only with ― and 85 only with —, 957 only with ～ and 1,813 only with 〜, so mapping one spelling to the other makes text find the words of one and miss those of the other. NFKC normalization (the unicode_normalize character filter) does not fold these characters: it turns ～ into ~ and leaves ―, — and 〜 as they are.

User dictionaries were never rewritten and do not change. The change is in the dictionary builder. The embedded dictionaries and the dictionaries that the 7.0.0 CLI fetches with lindera download keep the spellings; a dictionary directory built or downloaded with v6.2.0 or earlier no longer loads (see Dictionary directories must be rebuilt). UniDic, ko-dic, CC-CEDICT, Jieba and SudachiDict never turned the rewrite on.

The normalize_details setting is removed

normalize_details turned this rewrite on and did nothing else, so v7.0.0 removes it from the bundled metadata.json files and from the APIs that create or expose dictionary metadata:

A metadata.json that still contains normalize_details, such as one written by v6, loads as before, whether it comes with a built dictionary, is passed to lindera build or is read by a binding; the key is ignored and is not written back.

Tied entries resolve to the first CSV row, as in MeCab

A dictionary can have several entries with the same surface, the same left and right context IDs and the same cost, which differ only in their details, such as the reading or the base form. Such entries are tied: a path costs the same with any of them, so the order in which the segmenter considers them decides which one it outputs. Up to v6.2.0, Lindera added the dictionary entries that start at a position in reverse order, and the search keeps the earlier of two candidates with equal cost, so the last CSV row of a tied group won. The reverse order came in with v2.1.0, apparently by accident. v7.0.0 outputs the first row, as MeCab does:

IPADIC's Adj.csv has the two rows in this order:

狡い,19,19,4682,形容詞,自立,*,*,形容詞・アウオ段,基本形,狡い,ズルイ,ズルイ
狡い,19,19,4682,形容詞,自立,*,*,形容詞・アウオ段,基本形,狡い,コスイ,コスイ

Tied entries exist for 5,637 surfaces in IPADIC, 90,991 in IPADIC-NEologd, 3,324 in UniDic and 175 in ko-dic. In the Japanese dictionaries, the rows of a tied group never differ in the part of speech, only in fields such as the reading and the base form; in ko-dic, 5 groups also differ in the part of speech. With each tied surface on its own line, the number of surfaces whose full details equal those of MeCab 0.996 rises from 1,607 to 5,637 of 5,637 with IPADIC, from 524 to 3,324 of 3,324 with UniDic, and from 8,205 to 86,319 of 86,464 with IPADIC-NEologd (surfaces with whitespace or control characters left out). The other 145 IPADIC-NEologd surfaces contain 、, which no word spans (see The context is carried across 、 and 。), or belong to groups whose rows MeCab reads in another order (see below). With ko-dic, all 175 agree when MeCab's dictionary is built from the CSV files in name order.

On running text, the number of lines whose full details equal MeCab's rises, on top of the other changes in this guide, from 7,989 to 8,096 of the 8,100 sentences of UD Japanese GSD and from 3,278 to 3,583 of the 3,587 paragraphs of Kokoro and I Am a Cat with IPADIC, and from 3,394 to 3,571 of the same 3,587 paragraphs with UniDic. No line is segmented differently. In all the texts compared, 15 tokens change their part of speech, all to MeCab's. Most of them are entries of the same surface with different context IDs and costs whose paths happen to cost the same in that sentence, and the same order decides between them: UniDic, for example, now reads 御前 as a pronoun (代名詞) instead of a common noun (名詞,普通名詞,一般), as MeCab does.

User dictionaries follow the same rule: among tied user entries, the first row is now chosen instead of the last. A user entry that ties with a system entry is still chosen over it, as in v6. This is the one intended difference from MeCab, which chooses the system entry. Such a tie needs a user entry in the detailed format that copies the context IDs and the cost of a system entry, for example to give a word another reading. An entry in the simple three-column format gets the cost and the context IDs of the dictionary's metadata.json (default_word_cost, default_left_context_id and default_right_context_id: -10000, 0 and 0 for IPADIC), so it practically never ties with a system entry.

N-best search uses the same order. In addition, the first N-best result is now always the 1-best segmentation, the output of segment; up to v6 it could be another path when several paths had exactly the same cost. The costs of the N-best results do not change: only results with the same cost can change places, which can also change which of them is cut off at the n-th result. N-best search takes about 3% to 4% more instructions (-N 3); the speed of the 1-best segmentation does not change (within 0.2%).

With unique (--nbest-unique), every result is the cheapest path of its word boundaries, and of tied entries it now holds the first CSV row in every result, as the 1-best segmentation does. Up to v6, unique N-best went through the paths in order of cost and kept the first path of each segmentation, which could hold either row of a tied pair, and it went through every path of a segmentation before it reached the next one: a short line whose words have many entries could take tens of seconds and gigabytes. The search now goes over word boundaries, so it does not depend on the number of paths. Again, only results with the same cost can change.

MeCab's mecab-dict-index reads the CSV files in the order the file system lists them (readdir order), so for tied groups whose rows are in different files (in IPADIC-NEologd and ko-dic), MeCab's choice depends on the file system, while Lindera reads the files in name order.

The change is in the segmenter, not in the dictionary files. There is no setting to restore the v6 behavior.

Equal-cost segmentations keep the later-starting word, as in MeCab

Two segmentations can reach the same position with exactly the same cost while their last words start at different positions. MeCab keeps the one whose last word starts later: it puts every word at the head of the list of words that end where it ends, so the words that start later come first in that list, and of words of equal cost it keeps the first. Up to v6.2.0, Lindera kept the one whose last word starts earlier. v7.0.0 chooses as MeCab does:

Both segmentations of 腸窒扶斯 cost 28,382. With the Japanese and Korean dictionaries, such ties are rare. The texts compared are Botchan (as one file, as paragraphs and as sentences), the 3,587 paragraphs of Kokoro and I Am a Cat and the 8,100 sentences of UD Japanese GSD, each with IPADIC, IPADIC-NEologd and UniDic, and the 7,759 lines of Sangnoksu with ko-dic: about 54,000 lines in all. 9 of them change, all to MeCab's segmentation, and none moves away from it. In Decompose mode, which MeCab does not have, 0 to 2 lines per text change in the same way.

With CC-CEDICT, ties are common, because its costs take few values. CC-CEDICT has no connection costs, and an entry of len characters costs max(-36000, -400 * len ** 1.5) (except the 22 entries made only of Latin letters or only of digits, see CC-CEDICT: unknown words no longer cost less than dictionary words), so two segmentations into entries of the same lengths cost the same, such as 電視 / 台 / 上班 and 電視 / 台上 / 班. Jieba's costs are trained (see Jieba: costs retrained on UD Chinese GSD), so ties are rare with it; with the mecab-jieba-0.1.2 dictionary, whose 584,430 entries had 12 distinct costs and 4 context IDs, 1,186 sentences changed. Compared with v7.0.0 changed to keep the earlier-starting word on ties, about a quarter of the 4,997 sentences of UD Chinese GSD change with CC-CEDICT:

The other sentences that change come neither closer to the gold segmentation nor farther from it, and segmentation F1 rises from 77.94 to 78.30 with CC-CEDICT and stays 88.79 with Jieba. Every sentence that changes costs the same either way.

When whitespace is skipped (see Whitespace is skipped in the lattice), the word that ends later wins first, as in MeCab, which looks the next word up from every position a word ends at: an entry that ends with the space wins over a word that ends before it. One difference from MeCab remains in theory: after skipped whitespace, MeCab keeps a node for every position the previous word can end at, while Lindera keeps one, so two words that start right after the same whitespace and tie through previous words that end at different positions can be chosen differently. This needs an entry that ends with whitespace and an exact tie, and it does not occur in the texts compared.

Among words that start at the same position, the order of Tied entries resolve to the first CSV row, as in MeCab still applies, and the first N-best result is still the 1-best segmentation. When the context is carried across 、 or 。, a tie between the words that end the previous sentence resolves the same way, as in one lattice over the line: the word that starts later wins (see The context is carried across 、 and 。).

The 1-best segmentation takes 0.6% to 0.9% more instructions with IPADIC, IPADIC-NEologd and UniDic, with no measurable change in time, and about 2% more instructions and 2% to 5% more time with ko-dic. N-best search (-N 3) takes about 1% more instructions.

The change is in the segmenter, not in the dictionary files. There is no setting to restore the v6 behavior.

lindera tokenize removes only the line terminator and writes every line

lindera tokenize reads its input line by line. Up to v6.2.0, it removed all the whitespace at both ends of each line before tokenizing it: spaces, tabs, U+3000, U+00A0 and the other Unicode whitespace (str::trim). The library, the language bindings and MeCab do not. v7.0.0 removes only the line terminator (\n, or \r\n) and tokenizes the rest of the line as the library does:

• Spaces and tabs are SPACE characters, which the segmenter skips and drops from the output, so they still give no tokens; only the byte offsets of the tokens after them change, and now index the input line, as the library's offsets do.
• U+3000 at the start or end of a line is a token, as in the middle of a line, with IPADIC, IPADIC-NEologd (記号,空白) and UniDic (空白) (see Dictionary entries with whitespace are kept). In the 538 lines of Botchan as one file, where 202 lines start with an indent of U+3000, 206 such tokens appear and no other token changes. Text without whitespace at the ends of its lines, such as the 8,100 sentences of UD Japanese GSD and the 3,587 paragraphs of Kokoro and I Am a Cat, gives the same output as before.
• With --keep-whitespace, the spaces at the start and end of a line are tokens too: two spaces, 東京 and a space give three tokens.
• With -N, a line of spaces or tabs and an empty line give one result without tokens (NBEST 1 and EOS), as in MeCab, instead of no result (see N-best results cover the whole input for the empty input). The costs of a line with U+3000 at an end include that token.
• The wakati format writes a line for every input line, or for every result with -N: a line without tokens, such as an empty line or a line of spaces, gives an empty line, as in MeCab. v6 wrote nothing for it, so the output had fewer lines than the input (Botchan as one file: 505 output lines for its 538 input lines, 33 of them empty).
• The \r of a line that ends with \r\n is removed, and only that one \r. MeCab removes only the \n and outputs the \r as an unknown word; Lindera removes it so that text with Windows line endings does not end every line with a \r token.

The library and the language bindings never trimmed their input, so they do not change, except for the N-best result of an empty input. The change is in the CLI, not in the dictionary files. No option restores the v6 behavior; to get the v6 offsets and tokens, remove the whitespace at the ends of each line before passing the text to lindera tokenize.

WASM: the builder settings apply whichever way the dictionary is set

Up to v6.2.0, TokenizerBuilder.build() in the WASM binding ignored part of the builder's configuration, depending on how the dictionary was set:

v7.0.0 applies every setting and the user dictionary either way, as the Tokenizer API page describes. For example, with IPADIC loaded from bytes, setKeepWhitespace(true) and the unicode_normalize (NFKC) and lowercase filters turn Ｌｉｎｄｅｒａ 東京 into lindera, a space and 東京; v6 gave Ｌｉｎｄｅｒａ and 東京.

With a dictionary instance, build() now also checks what it checks for a dictionary set by URI: unless whitespace is kept, the dictionary's char.def must define the SPACE category. Every bundled dictionary does; a custom dictionary without it now needs setKeepWhitespace(true).

The other bindings, the CLI and the library do not change, and neither do the dictionary files. No option restores the v6 behavior; to get it, leave out the settings that v6 ignored.

PHP: optional arguments reject a value of the wrong type

Up to v6.2.0, the PHP binding read a value of the wrong type for an optional parameter as null, so the call silently used the default, with or without strict_types. For example, $tokenizer->tokenizeNbest('東京都', 2, 'yes') returned results that were not deduplicated, and Lindera\Trainer::train(..., max_iter: $argv[1]) trained for the default 100 iterations whatever the command line said. A required parameter already rejected such a value.

v7.0.0 rejects it for the optional parameters too: those of the Penalty, Metadata, FieldDefinition and Tokenizer constructors, of Mode::decompose(), Tokenizer::tokenizeNbest(), Trainer::train(), Trainer::export() and Dictionary::buildUser(). The call throws a TypeError, Invalid value given for argument followed by the parameter's name. A required parameter now throws the same TypeError for such a value; v6 threw an Exception with the same message. null and leaving the argument out still select the default. $lambda of Trainer::train() now takes an int as well as a float, as a PHP float parameter does; v6 read an int there as null too.

Pass each argument its declared type, and cast values that arrive as strings, such as (int) $argv[1] (see Argument types). Code that catches such a value with catch (Exception $e) must catch TypeError instead: a TypeError is an Error, not an Exception. The parameters declared mixed, the other bindings, the CLI and the library do not change. No option restores the v6 behavior.

Bindings: integer arguments outside their range are rejected

Up to v6.2.0, the PHP, Node.js and WASM bindings converted an integer argument to the type of its setting by wrapping, so a value outside the range silently became another value. The PHP binding took every int as a 64-bit integer and cast it to the narrower type. In Node.js and WASM, the conversion from a JS number to a 32-bit integer (ECMAScript's ToUint32 or ToInt32) wrapped it before the binding saw it, and a WASM BigInt was taken modulo 2^64.

v7.0.0 rejects such a value. The PHP binding throws a ValueError, the Node.js binding an Error with the code InvalidArg, and the WASM binding a string, as for its other errors. The message names the parameter and the range, such as default_word_cost must be an integer between -32768 and 32767, got 100000. The parameters are the integer settings of Metadata (the default word cost and context IDs) and Penalty, the index of FieldDefinition, getFieldName and PHP's Token::getDetail, the number of N-best results and, in Node.js and WASM, the N-best cost threshold, and the iteration and thread counts of the training functions. The checks apply to the constructors and to the property setters (Node.js Metadata, WASM Penalty and FieldDefinition).

With them, some values that v6 accepted now throw:

• A negative index, for which v6 returned null (PHP, Node.js) or undefined (WASM). An index past the end still returns null or undefined.
• In Node.js and WASM, a number that is not an integer, such as 1.5, NaN or Infinity, which v6 truncated or turned into 0.
• In WASM, a BigInt cost threshold outside the 64-bit range, and a cost threshold that is not a BigInt.

Pass an integer in the range. The TypeScript declarations of the Node.js and WASM bindings do not change. The Python and Ruby bindings already rejected these values, with OverflowError and RangeError, and still do. The RangeError of the Ruby binding now has the message above, which names the parameter and the range, where v6 named only the Rust type (fixnum too big to convert into `i16` ); this covers the N-best cost threshold too, for an Integer outside the 64-bit range. The OverflowError of a Python property setter now names the property in its note (while processing 'default_word_cost' instead of 'cost'). No option restores the v6 behavior.

Bindings: a mode is a Mode

Up to v6.2.0, every binding exported a Mode, but no function or constructor took one: set_mode / setMode and the Tokenizer constructors took a mode name, and decompose mode always used the default penalty. Node.js exported a string enum JsMode, whose values 'Normal' and 'Decompose' setMode rejected, and WASM a numeric enum. A custom penalty reached the segmenter only through a configuration file loaded with from_file / fromFile. In Rust, TokenizerBuilder::set_segmenter_mode lost the penalty of a Mode::Decompose the same way, because it wrote only the name of the mode into the configuration.

In v7.0.0, the Mode of each binding is a copy of the Rust lindera::mode::Mode: normal mode, or decompose mode with a Penalty. set_mode / setMode and the Tokenizer constructors take a Mode, as set_segmenter_mode and Segmenter::new do in Rust, and no longer take a mode name. Leaving the mode of a constructor out, or passing null / None / nil, still means normal mode, and set_segmenter_mode keeps the penalty:

The Tokenizer constructors take the same Mode as their second argument, for example Tokenizer(dictionary, Mode.normal()) in Python. The penalty is optional: Mode.decompose() uses the default one. With IPADIC and both length penalties 0, decompose mode segments 関西国際空港限定トートバッグ as 関西国際空港 / 限定 / トートバッグ instead of 関西 / 国際 / 空港 / 限定 / トートバッグ.

A mode name read at run time, such as a setting, becomes a Mode with Mode.from_str(name) (Mode.fromStr in PHP, Node.js and WASM). It parses the name as the Rust Mode does: "normal" or "decompose", case sensitive, the latter with the default penalty; another name throws the error that set_mode threw for it in v6. A mode name, or any other value that is not a Mode, passed to set_mode / setMode or a constructor now throws a type error: TypeError in Python, Ruby and PHP, an InvalidArg error in Node.js and expected instance of Mode in WASM.

The other changes:

• The Mode classes: a Mode has no constructor; Mode.normal(), Mode.decompose() and Mode.from_str() create one. It has a name, a penalty (a copy, or none for normal mode), the normal and decompose checks, and equality of the name and the penalty (==, or equals() in Node.js and WASM). Python's Mode(mode_str) and its attributes Mode.Normal / Mode.Decompose, Ruby's Lindera::Mode.new, PHP's new Lindera\Mode(), Node.js's JsMode and WASM's Mode.Normal / Mode.Decompose are gone. The Python and Ruby constructors accepted "Normal" and "Decompose", and PHP's any case; from_str does not. A Python Mode is not hashable, as the Rust Mode is not Hash.
• Node.js: the TypeScript type JsPenalty is renamed Penalty, and its fields become optional; a field left out keeps its default. TypeScript code that names JsPenalty uses Penalty. The package exports no value for the type, so JavaScript code does not change.
• Rust: TokenizerBuilder::set_segmenter_mode with a Mode::Decompose whose penalty is not the default now applies that penalty, so the output changes. A mode parsed from "decompose", such as the one the CLI's --mode gives, carries the default penalty and is not affected.
• Binding authors: CoreTokenizerBuilder::set_mode, CoreTokenizer::from_segmenter and CoreTokenizer::from_segmenter_with_space_penalty take a lindera::mode::Mode instead of the &str that v6.2.0's lindera-binding-core took. set_mode no longer returns a Result. lindera_binding::parse_mode parses a mode name, as the bindings' from_str do.

lindera train builds its lattices as MeCab's learner does

Up to v6.2.0, lindera train built training lattices that differ from those of MeCab's learner:

• The seed lexicon was not in the lattice, so a gold token competed only with unknown words, never with the other seed words.
• The unknown-word candidates ignored char.def. A fixed classifier (ASCII digit, ASCII letter, kanji, katakana, hiragana, other) gave the class of the first character, every candidate was one to three characters long, and the class number served as the unk.def entry, so only an unk.def with one entry per class, in the order DEFAULT, HIRAGANA, KATAKANA, KANJI, ALPHA, NUMERIC, worked.
• Each char.def category got one label, with the feature string of its last unk.def line, and lindera export wrote one unk.def line per category. mecab-ipadic's unk.def has 40 lines for 11 categories; they became 11 lines, and the unknown ALPHA, HIRAGANA and KATAKANA words all became 感動詞, the feature of the last line of each of these categories.
• A gold token that was not a seed word got an unknown-word label through a check written for IPADIC features (名詞,一般 and the like), or a label without features. An unk.def line with fewer than five fields was silently skipped.
• When regularization left no bigram weight for some context IDs, lindera export wrote a matrix.def smaller than the IDs of the entries (0 0 when no weight was left at all), and the built dictionary panicked at tokenization. model_info.connection_matrix_size in metadata.json was a square of the larger ID.
• lindera export wrote 32767 (i16::MAX) to matrix.def for a connection without a trained weight, whose bigram features never met in training or whose weights regularization zeroed. Training gives such a connection the cost 0, and so does MeCab's mecab-dict-gen, so the built dictionary avoided transitions that the model had learned to be neutral. In a five-fold cross-validation with mecab-ipadic's files and the default L1 regularization, writing 0 raised the boundary F1 from 0.9552 to 0.9572 in every fold (#1168).
• model_info.training_iterations in metadata.json was the --max-iterations value, not the number of iterations that ran.
• The export functions of the Python, Node.js, Ruby and PHP bindings wrote lex.csv, matrix.def, unk.def, metadata.json and a fixed char.def with the categories DEFAULT, HIRAGANA, KATAKANA, KANJI, ALPHA and NUMERIC, not the char.def the model was trained with, and no feature.def, rewrite.def, left-id.def or right-id.def.
• The seed lexicon was read line by line and split on ,, and a line that starts with # was skipped as a comment. mecab-ko-dic's quoted comma surface "," became the surface ", and lindera export wrote it unquoted, so lindera build read the rest of lex.csv as one field and dropped every row after it without an error. The # rows of UniDic, SudachiDict and IPADIC NEologd were not trained.
• A feature was split on , for the rewrite rules, the feature templates, unknown-word matching and the context IDs of user lexicon entries. A quoted column with a comma in it became several columns and shifted the columns after it: in 19,955 rows of mecab-jieba's training seed, whose quoted English definition is followed by four more columns, %F[6] to %F[9] were pieces of the definition.
• A %F[n], %L[n] or %R[n] without ? that read past the last column of a feature read * there, so the template still gave a feature, such as U09:*, which every feature with fewer columns shared. MeCab's learner skips the template. The unk.def features of mecab-jieba have five columns, and its feature.def reads %F[6] to %F[9], so its 11 unknown-word labels had four features, U06:* to U09:*, that no seed row had.
• Model::write_dictionary gave an entry of the user lexicon that Model::read_user_lexicon reads whose context IDs and cost are 0,0,0 the context IDs of another user lexicon entry: the one at the index of the seed row whose feature is the most similar to the entry's, or else the first one. An entry whose feature had no column in common with any seed row got the index of its first character's category in char.def as both IDs, which is not a context ID.
• Model::write_dictionary gave every such entry with 0,0,0 the same cost, tocost(0.8, cost_factor), whatever its feature. Model::read_user_lexicon read an empty context ID or cost, and a value that is not a number, as 0, and a negative context ID wrapped around to 16 bits (-1 became 65535); the parameters were filled in only when all three were 0. So an entry with an empty cost and given context IDs got the cost 0, and one with empty context IDs and a given cost got the BOS/EOS IDs 0,0.

v7.0.0 trains as MeCab's learner does (see How training works):

• The lattice has the seed words that start at each position, and the unknown words that the runtime creates there, with the same code: those of the character's default category, with its INVOKE, GROUP and LENGTH settings, for every unk.def entry of the category.
• Every unk.def line is an entry and a CRF label of its own, with the ID of its category as %t, and lindera export writes every line, in file order. Comment lines and empty lines are skipped, and a line with fewer than five fields is an error.
• A gold token gets the label of the first seed row with the same span and the same whole feature string; else that of the first unk.def entry of an unknown-word candidate with the same span whose feature columns other than * all equal the gold token's; else a virtual label, which is trained and not exported, as MeCab's learner adds a virtual node. lindera train prints the counts in a Lattice statistics: line.
• matrix.def covers every context ID, plus 0 for BOS/EOS, and model_info.connection_matrix_size gives its size, rows by columns.
• A connection without a trained weight costs 0 in matrix.def, as in training and in MeCab.
• model_info.training_iterations is the number of L-BFGS iterations that ran: --max-iterations when that stopped training, fewer when it converged earlier.
• The export functions of the Python, Node.js, Ruby and PHP bindings write the files lindera export writes, with the same code: the char.def, feature.def and rewrite.def the model was trained with, and left-id.def and right-id.def too.
• model.dat has the format version 2.0.0. lindera export and the bindings' export functions reject a model.dat of any other version, including every model.dat written by v6.2.0 or earlier, with an error that ends in "re-run lindera train to regenerate it".
• The seed lexicon is read as CSV, as lindera build reads lex.csv (see Seed Lexicon): a field may be quoted, a line that starts with # is a row, and the feature columns are kept as written. A quoted field that spans lines is an error. lindera export quotes a surface that contains ,, " or a line break. The user lexicon that Model::read_user_lexicon reads is read the same way.
• A feature is split into columns as MeCab's learner splits it (see Seed Lexicon): a quoted column is one column, spaces and tabs before a column are skipped, and the text from a closing quote to the next , is dropped. The rewrite rules join the columns they rewrite with ,, quoting a column that contains , or ", as MeCab does.
• A template that reads past the last column of a feature gives no feature for it, with or without ?, as in MeCab's learner (see Feature Template). In the training files of mecab-ipadic, mecab-ipadic-neologd, unidic-mecab, mecab-ko-dic, mecab-jieba and resources/training/, only the unk.def lines of mecab-jieba have features that a template reads past the last column of, so only a model trained on mecab-jieba's files changes.
• Model::write_dictionary fills in the parameters of a user lexicon entry separately, as mecab-dict-index does: an empty cost is computed, and when either context ID is empty or negative, both are inferred. 0,0,0 still fills in all three. A context ID or cost that is neither empty nor a number in its range is an error that names the entry.
• A computed cost is the one MeCab's calcCost computes: the sum of the trained weights of the entry's unigram features, converted as the costs of lex.csv are, so an entry whose feature and first character's category are those of a seed row gets that row's cost.
• Inferred context IDs are those that lex.csv gives the seed row whose feature is the most similar to the entry's. An entry whose feature has no column in common with any seed row is an error, and nothing is written, as mecab-dict-index fails for a user dictionary entry whose feature has no context ID.

Training takes longer, because every lattice now has the seed words. It is slower than MeCab's learner, and it can stop at --max-iterations before it converges (#1167). The files in resources/training/ now define a SPACE category in char.def and a SPACE line in unk.def, so a dictionary trained from them tokenizes without --keep-whitespace, which it needed in v6 (#1166).

What you do: train every model again with the 7.0.0 lindera train or the bindings' training functions, then export it and build the dictionary again. Give every unk.def line at least five fields. An unk.def with several entries per category, such as mecab-ipadic's, is now trained and exported as it is. Remove the comment lines from your seed lexicons: a line that starts with # is now a row. Give the context IDs of a user lexicon entry whose feature has no column in common with any seed row. Write each context ID and cost of a user lexicon entry as a number, or leave it empty to have it filled in. There is no setting to restore the v6 behavior.

lindera build rejects context IDs outside the connection matrix

Up to v6.2.0, lindera build accepted a lex.csv row or an unk.def line whose context IDs fall outside matrix.def, and the built dictionary panicked when such a word was looked up during tokenization. A matrix.def row whose ID exceeded its axis was accepted too, and its cost landed in another cell. v7.0.0 rejects all of these, as MeCab's mecab-dict-index does: in every lex.csv row and unk.def line, the right context ID must be below the first value of the matrix.def header, and the left context ID below the second. The error names the file, the line, the ID and the matrix size:

Error: LinderaError { kind: Content, source: ./dict-source/lex.csv: line 2: right context id 2 is out of range for the connection matrix (matrix.def size 2 2; right context ids must be below 2) }

• A dictionary whose metadata.json sets skip_invalid_cost_or_id drops such a lex.csv row instead, with a warning, which the CLI writes to standard error (see The CLI writes library warnings to standard error). unk.def lines are always checked.
• An unk.def line with fewer than four fields, such as a comment or an empty line, is an error instead of a panic, also in DictionaryBuilder::build_unknown_dictionary and parse_unk of lindera_dictionary. The error names the line, and so does the error for a context ID or a cost that is not a number, which did not name it before.
• A user dictionary is checked when it is combined with the system dictionary: Segmenter::try_new, the configuration path, lindera tokenize --user-dict, and the Tokenizer constructors and builders of the language bindings return an error that names the first entry outside the matrix (#1171; see Context IDs). Segmenter::new does not check, as before. lindera build --user checks the rows already when the new --system-dict option names the system dictionary (DictionaryBuilder::with_system_matrix in the Rust API), and names the first row outside the matrix by its CSV line and surface (#1182). A .csv user dictionary that the configuration path, lindera tokenize --user-dict or the bindings' TokenizerBuilder loads by path, or that the new load_user_dictionary_for of the Rust API loads, is checked the same way as it is built for the system dictionary, and its error names the CSV line and surface too (#1187). The language bindings have such a loader for a loaded system dictionary too: load_user_dictionary_for in Python and Ruby, loadUserDictionaryFor in Node.js, Lindera\Dictionary::loadUserFor in PHP and loadUserDictionaryFromBytesFor in WebAssembly (#1193). The loaders that take metadata are unchanged. The user dictionary builders of these bindings have such a variant as well, which checks the rows against a loaded system dictionary as it builds the CSV: build_user_dictionary_for in Python and Ruby, buildUserDictionaryFor in Node.js and Lindera\Dictionary::buildUserFor in PHP (#1188). The builders that take metadata are unchanged too: their output_dir is, as before, the path of the file they write, not a directory.
• The errors for a user dictionary row whose cost or context ID is not a number, from lindera build --user or from loading a .csv user dictionary, name the row's CSV line (at line 1). Before, they counted the rows in the order of their surfaces (at row 2 for a row on line 1 whose surface sorts second).

Dictionaries that mecab-dict-index builds pass, since it checks the same bounds. A dictionary exported by lindera export of v6.2.0 or earlier can fail, since its matrix.def could be smaller than its context IDs; train and export it again with 7.0.0 (see the previous section). For another dictionary that fails, fix the IDs in lex.csv or unk.def or the size in matrix.def.

lindera build and lindera train check char.def against unk.def

Up to v6.2.0, lindera build and lindera train accepted a char.def and an unk.def that MeCab's mecab-dict-index rejects (#1195):

• A range line of char.def that names a category no line defines. The built dictionary panicked on a character of the category, and so did training when the corpus had one.
• A range line before the definition of its category, or a category defined twice. The categories defined after it got the INVOKE, GROUP and LENGTH settings of another category, which changed the unknown words.
• A char.def category without an unk.def line. A character of it that no word covers got no unknown word, so tokenization stopped its output before it, without an error, and a training lattice had no path through it.
• An unk.def line whose category char.def does not define. The line was never used, and lindera export wrote it back.

v7.0.0 rejects all of these, as mecab-dict-index does. The error names the file, the category and, except for a category without an unk.def line, the line. For example, with 0x3005 KANJI_ITERATION on line 8 of char.def, whose category no line defines:

Error: LinderaError { kind: Content, source: ./dict-source/char.def: line 8: category KANJI_ITERATION is not defined; a range line can only name a category defined on an earlier line }

lindera train and the bindings' training functions check their input in the same way before they train, and number the unk.def lines as in the file, comment and empty lines included.

Every bundled dictionary passes, and so does a dictionary that mecab-dict-index builds. For another dictionary that fails, define each category once, before the range lines that name it, give every category at least one unk.def line, and remove the unk.def lines of categories that char.def does not define.

lindera build gives a cell missing from matrix.def the cost 0

Up to v6.2.0, lindera build gave a cell that matrix.def does not list the cost 32767 (i16::MAX), which made the connection very expensive. MeCab's mecab-dict-index gives such a cell the cost 0, and v7.0.0 does the same (#1211; see Cells missing from matrix.def).

The matrix.def of every bundled dictionary lists every cell, and so does the matrix.def that lindera export writes, so neither changes. A dictionary built from a matrix.def that leaves cells out can segment differently, since a connection through such a cell now costs 0. To keep a connection expensive, list its cell with a cost.

The CLI writes library warnings to standard error

Up to v6.2.0, the lindera CLI installed no logger, so the warnings that the Lindera libraries report through the log crate were dropped (#1107). v7.0.0 writes them to standard error, warnings and errors by default. Standard output does not change. For example:

• lindera tokenize warns when it cuts a sentence because more than 32 KiB of text have no sentence delimiter.
• lindera build warns when it drops a lex.csv row because of skip_invalid_cost_or_id, and when the dictionary's metadata.json sets connection_id_mapping (ko-dic, UniDic) and --context-id-freq is not given. Pass the context_id_freq.txt of the dictionary crate, as the build examples do.

[WARN  lindera_segmenter::segmenter] no sentence delimiter (\n, \t, 。, 、) found within 32768 bytes from offset 0; forcing a sentence boundary to bound lattice size (see https://github.com/lindera/lindera/issues/871)

The RUST_LOG environment variable selects the messages: RUST_LOG=error hides the warnings, and RUST_LOG=debug adds debugging messages (see Warnings and log messages).

lindera build and lindera train no longer warn about an unk.def line whose left and right context IDs differ, which mecab-dict-index accepts: every line of mecab-ko-dic's unk.def is such a line. Rust programs that install a logger no longer get this warning either.

Rust API changes in lindera_dictionary

These affect only code that calls the lattice backtraces directly, that creates dictionary metadata with Metadata::new, that reads the character categories of a dictionary, that generates unknown words with UnknownDictionary, or that builds a dictionary from a char.def and an unk.def that mecab-dict-index rejects (see the previous section): lindera_dictionary::viterbi::Lattice and lindera_dictionary::nbest::NBestGenerator, also reachable as lindera::dictionary::viterbi::…, lindera::dictionary::Lattice and lindera::dictionary::nbest::…, and lindera_dictionary::dictionary::metadata::Metadata, also reachable as lindera::dictionary::Metadata, and lindera_dictionary::dictionary::character_definition::CharacterDefinition, and lindera_dictionary::dictionary::unknown_dictionary, also reachable as lindera::dictionary::core::unknown_dictionary, and lindera_dictionary::builder, also reachable as lindera::dictionary::builder. The Segmenter, Tokenizer, SegmentWorker and AnalysisWorker APIs and the CLI are unaffected; the language bindings change only in their Metadata class (see The normalize_details setting is removed).

Each token is now (start, end, word_id) instead of (start, word_id), with byte offsets within the sentence. With whitespace skipped (LatticeOptions::skip_whitespace, new in v7), a token does not always end where the next one starts: skipped whitespace can lie in between, and only the lattice knows how much of it belongs to the entry. The returned end leaves the skipped whitespace out and keeps the whitespace an entry ends with (see Dictionary entries with whitespace are kept). With whitespace kept in the lattice, as set_text and set_text_nbest do, the end is the next token's start, or the end of the sentence for the last token, as before. nbest_tokens_offset with unique now compares the (start, end) pairs of the paths instead of their starts.

#![allow(unused)]
fn main() {
use lindera::dictionary::viterbi::TokenOffset;

// v6: a token ends where the next one starts
let offsets = lattice.tokens_offset();
for (i, &(start, word_id)) in offsets.iter().enumerate() {
    let end = offsets.get(i + 1).map_or(sentence.len(), |&(next, _)| next);
    handle(&sentence[start..end], word_id);
}

// v7: the end is returned
let offsets: Vec<TokenOffset> = lattice.tokens_offset();
for &(start, end, word_id) in &offsets {
    handle(&sentence[start..end], word_id);
}
}

The paths of nbest_tokens_offset and NBestGenerator::next hold the same triples.

tokens_offset_into also returns the BOS index of the best path: the index in LatticeOptions::bos of the BOS edge it starts from, Some(0) with the default single BOS edge, and None when the lattice holds no complete path. A call written as a statement compiles unchanged; where a () value is expected, such as the body of a closure passed to for_each, add a ;.

The new items are what the segmenter uses to carry the context across 、 and 。 (see The context is carried across 、 and 。). They change nothing for existing code:

• BosContext and LatticeOptions::bos: a sentence can start from several BOS edges, each with a right context ID and a cost. The default, an empty slice, is the single BOS edge of v6.
• LatticeExit, Lattice::exits_into and Lattice::exit_tokens_offset_into: for each right context ID of the words that end the sentence, the best path to such a word without the EOS connection, and its tokens.
• NBestGenerator::from_exit and NBestGenerator::next_with_bos: the paths that end with the right context ID of an exit, without EOS, in ascending order of cost, and each path together with the BOS index it starts from. next returns the paths of next_with_bos without the BOS index.

UniqueNBestGenerator has the methods of NBestGenerator (new, from_exit, next and next_with_bos) and yields the cheapest path of each pair of word boundaries and BOS edge, in ascending order of cost; it is what nbest_tokens_offset with unique and the segmenter use.

nbest_tokens_offset is meant for the default single BOS edge: with several, it does not say which one a path starts from. Use NBestGenerator::next_with_bos there.

Metadata::new loses its ninth argument, normalize_details: delete it from the call. Metadata no longer has the field, and PrefixDictionaryBuilderOptions (in lindera_dictionary::builder::prefix_dictionary) no longer has the method, because the builder keeps every entry as written (see Dash and tilde spellings are kept as written). Metadata still reads a metadata.json that contains the key, and ignores it. For binding authors, lindera_binding::CoreMetadata loses the field the same way, and CoreMetadata::new takes 10 arguments instead of 11.

Metadata gains the public field skip_whitespace: Option<bool>, the dictionary's default for whitespace skipping (see Whitespace is skipped in the lattice), and so does lindera_binding::CoreMetadata. Neither struct is #[non_exhaustive], so a struct literal that lists every field without .. no longer compiles: add skip_whitespace: None, or end the literal with ..Default::default(). Metadata::new and CoreMetadata::new set it to None, which skips whitespace.

CharacterDefinition::lookup_categories keeps its signature. Its first category is the one that creates unknown words (see Unknown words come from the default category, as in MeCab); code that only checks whether a character has a category is unaffected. DICTIONARY_FORMAT_VERSION, the version that the builder writes and the loaders accept, is now 3 (see Dictionary directories must be rebuilt).

UnknownDictionary::gen_unk_words and UnknownDictionary::compatible_unk_index are removed, together with the UnkWord and WordIdx types they used. Only lindera train called them, and they ignored char.def (see lindera train builds its lattices as MeCab's learner does). To list the unknown words that the runtime creates, call Lattice::unknown_word_candidates(dict, char_definitions, unknown_dictionary, text, max_grouping_len, out). It fills out with an UnknownWordCandidate { start_char, end_char, word_id } for every candidate of every start position of text, with character positions and the unk.def entry ID, as set_text_with_options creates them with the default options and the given max_grouping_len, also at positions that no path reaches. The tokenization path does not change.

Rust API changes in lindera_trainer

These affect only Rust code that uses lindera-trainer directly or through lindera::dictionary::trainer (the train feature) (see lindera train builds its lattices as MeCab's learner does):

A model is now written with MODEL_FORMAT_VERSION as its metadata.version, and Model::read_model checks it, in both the rkyv and the legacy JSON format, so a model written by v6.2.0 or earlier has to be trained again. Code that reads unk_category_names or unk_categories reads unk_entries instead: one pair per unk.def line, several per category included, in the order of the exported unk.def. If you set Trainer::max_grouping_len, pass the cap that the dictionary will be used with (max_grouping_len of the segmenter, --max-grouping-len of the CLI).

Who does not need to act

• Users of the language bindings and the CLI: the Python, Node.js, Ruby, PHP, and WASM packages and lindera-cli keep their package names, and their APIs apart from the normalize_details setting of Metadata (see The normalize_details setting is removed) and, in PHP, optional arguments of the wrong type (see PHP: optional arguments reject a value of the wrong type), and, in PHP, Node.js and WASM, integer arguments outside their range (see Bindings: integer arguments outside their range are rejected), and the mode, which they take as a Mode instead of a name, and the Node.js TypeScript type JsPenalty, now Penalty (see Bindings: a mode is a Mode). The restructuring is internal to the Rust crates. A dictionary directory built or downloaded with v6.2.0 or earlier has to be rebuilt or downloaded again (see Dictionary directories must be rebuilt); the embedded dictionaries need nothing. A model file trained with v6.2.0 or earlier has to be trained again (see lindera train builds its lattices as MeCab's learner does). The output changes are the IPADIC conjugation fix, which matters only if you read those two fields by name, the whitespace entries, which matter only for text with U+3000 (IPADIC, IPADIC-NEologd, UniDic) or with spaces (SudachiDict), the segmentation of text that contains whitespace, the char.def fix, which matters only for text with Ð, 々 or 〇, the context carried across 、 and 。, which matters only for text with 、 or 。 inside a line, the N-best fix, which matters only for N-best results for input with more than one sentence or for an empty input, the unknown words that start at every position, which matter for runs of katakana, symbols or Latin letters in normal mode, the unknown words made from the default category, which matter for ko-dic hanja numerals followed by hanja and for 〇 with ko-dic, the new costs of CC-CEDICT, which matter for its entries that mix Latin letters or digits with Chinese characters and for its N-best costs, the 〇 entry of Jieba, which matters only for text with 〇, the retrained costs of Jieba, which change the segmentation of most Chinese text with Jieba and the cost of simple Jieba user dictionary words, the dash and tilde spellings, which matter only for text with ―, —, ～ or 〜 with IPADIC or IPADIC-NEologd, the choice among tied entries, which matters only if you read token details such as the reading or the base form, or the order of N-best results with the same cost, the choice between segmentations of equal cost, which changes the segmentation of a few lines with the Japanese and Korean dictionaries and Jieba and of about a quarter of the sentences with CC-CEDICT, in the CLI only, the whitespace at the ends of an input line and the output of lines without tokens, which matter only for lines that start or end with whitespace and for the wakati and N-best output of empty lines and lines of spaces, and the warnings on standard error, which matter only for scripts that read it, and, in the WASM binding only, the builder settings that v6 ignored for one way of setting the dictionary.
• Projects staying on lindera = "6": the 6.x releases of lindera and lindera-analysis remain on crates.io and keep working together. Nothing changes until you bump the major version.
• Users of the segmenter API only: code that uses Segmenter, load_dictionary, Mode, and the other lindera::… items of v6 compiles unchanged with lindera = "7", unless it calls the lattice backtraces of lindera::dictionary::Lattice or creates metadata with lindera::dictionary::Metadata::new (see Rust API changes in lindera_dictionary). Dictionary directories loaded by path still have to be rebuilt. The default build now also compiles lindera-analysis and its dependencies (kanaria, regex, serde_yaml_ng, unicode-blocks, unicode-normalization, unicode-segmentation); to keep the v6 dependency tree, set default-features = false, features = ["mmap"].

Upgrade checklist

Rust crate users:

• Bump lindera to "7" and, in the same commit, either bump lindera-analysis to "7" or remove it and import from lindera::analysis::….
• Optionally replace lindera_analysis:: with lindera::analysis:: in use paths — both name the same types.
• If you set default-features = false, add "analysis" (for the Tokenizer and filters) and "mmap" (for memory-mapped loading) to features as needed.
• Replace features = ["lindera-<dictionary>"] with features = ["embed-<dictionary>"].
• Optionally drop a direct lindera-dictionary dependency in favor of lindera::dictionary::core, ::builder, ::viterbi, and the other re-exported modules.
• If you call Lattice::tokens_offset, tokens_offset_into, nbest_tokens_offset or NBestGenerator::next, destructure (start, end, word_id) and use the returned end instead of the next token's start.
• If you use the () value of Lattice::tokens_offset_into, for example as the body of a closure, add a ;: it now returns the BOS index of the best path.
• If you call Metadata::new, delete its ninth argument, normalize_details, and stop using the Metadata::normalize_details field and PrefixDictionaryBuilderOptions::normalize_details.
• If you build Metadata with a struct literal that lists every field, add skip_whitespace: None or end it with ..Default::default().
• If you read CharacterDefinition::lookup_categories, expect the default category first and the others in category id order.
• If you call UnknownDictionary::gen_unk_words or UnknownDictionary::compatible_unk_index, or use UnkWord or WordIdx, list the unknown words with Lattice::unknown_word_candidates instead.
• If you read SerializableModel::unk_category_names or unk_categories, read unk_entries instead, and if you set Trainer::max_grouping_len, pass it the cap that the dictionary will be used with.
• If you pass TokenizerBuilder::set_segmenter_mode a Mode::Decompose whose penalty is not the default, expect that penalty to apply.

Binding authors:

• Depend on lindera-binding instead of lindera-binding-core, and replace lindera_binding_core:: with lindera_binding::.
• Delete the normalize_details argument of CoreMetadata::new, which now takes 10 arguments, and stop using the CoreMetadata::normalize_details field.
• If you build CoreMetadata with a struct literal that lists every field, add skip_whitespace: None or end it with ..Default::default().
• Pass a lindera::mode::Mode instead of a mode name to CoreTokenizerBuilder::set_mode, which no longer returns a Result, and to CoreTokenizer::from_segmenter and CoreTokenizer::from_segmenter_with_space_penalty; parse a name with lindera_binding::parse_mode.

Build environments:

• Use Rust 1.89 or newer.
• Rename LINDERA_DICTIONARIES_PATH to LINDERA_BUILD_DICTIONARY_CACHE_DIR in shell profiles, CI configuration, and container images.

Everyone who loads a dictionary from files:

• Rebuild with lindera build, or download again, every dictionary directory made with v6.2.0 or earlier, including those saved in the WASM binding's OPFS storage: v7.0.0 rejects them (format version 3). The embedded dictionaries and user dictionaries need nothing.

Users of IPADIC or IPADIC-NEologd:

• If you read conjugation_type or conjugation_form by name (including the CLI's JSON output), expect the two values to trade places.
• If you build dictionaries with your own copy of the IPADIC metadata.json, swap the two names in it.

Users of IPADIC, IPADIC-NEologd, UniDic or SudachiDict with text that contains U+3000 or spaces:

• Expect U+3000 to be 記号,空白 (IPADIC) or 空白 (UniDic) tokens, and SudachiDict to segment spaced text much closer to Sudachi. Drop U+3000 with japanese_stop_tags or NFKC normalization if you do not want it.

Everyone who segments text that contains whitespace:

• Expect MeCab's segmentation for such text (most visible with ko-dic; not with SudachiDict, which keeps whitespace in the lattice as Sudachi does). If you need the v6 handling of whitespace, set skip_whitespace(false), "skip_whitespace": false, or --disable-skip-whitespace; the other output changes in this guide still apply.
• Expect the token of a dictionary entry that ends with a space (in IPADIC-NEologd, ko-dic and SudachiDict) to include that space, as in MeCab.

Users of any bundled dictionary with text that contains Ð, 々 or 〇:

• Expect Ð to stay in the output, and 々 after an unknown kanji to be a token of its own.

Users of lindera train:

• Give every range line of your char.def at least one category.
• If your char.def has single-code-point or overlapping lines, retrain: the %t feature now follows those lines as MeCab does.
• Train every model again with the 7.0.0 lindera train (or the bindings' training functions), then export it and build the dictionary again: lindera export rejects a model.dat written by v6.2.0 or earlier, and the training lattices, the unknown-word labels and the exported unk.def and matrix.def have changed.
• Give every line of your unk.def at least five fields: the category, the two context IDs, the cost and at least one feature column.
• If your char.def and unk.def were copied from resources/training/ and have no SPACE category, add it as those files now do, so that the trained dictionary tokenizes without --keep-whitespace (#1166).
• If training reports a category of char.def or unk.def, define each category once, before the range lines that name it, give every category an unk.def line, and remove the unk.def lines of undefined categories (#1195).

Everyone who builds a system dictionary with lindera build:

• If the build reports a context ID out of range for the connection matrix, fix the IDs in lex.csv or unk.def or the size in matrix.def, or set skip_invalid_cost_or_id to drop such lex.csv rows. Such a dictionary used to build and then panic at tokenization. A dictionary exported by v6.2.0 or earlier is best trained and exported again.
• If the build reports a category of char.def or unk.def, define each category once, before the range lines that name it, give every category an unk.def line, and remove the unk.def lines of undefined categories (#1195).

Everyone who uses a user dictionary with detailed entries:

• If building the tokenizer reports a user dictionary entry whose context ID is out of range for the connection matrix, fix the IDs in the user dictionary CSV for the system dictionary you use, and build the .bin again if you use one. Such a user dictionary used to panic at tokenization when the entry matched the input.

Users of Japanese text with 、 or 。 inside a line:

• Expect some words after 、 or 。 to be read in the context of the words before them, mostly as MeCab reads them. No setting restores the v6 output.

Users of N-best results:

• Expect every result to cover the whole input, and the results from the second one on to change for input with more than one sentence.
• For a line with 、 or 。, expect every cost to change, the first result's included: the costs are those of one lattice over the line.
• If you set a cost threshold, it now applies to the total cost of the whole input, not to each sentence.
• Expect an empty input to give one result without tokens, as an input of only whitespace does, instead of no result.
• Expect the first result to be the 1-best segmentation also when several paths have exactly the same cost; results with the same cost can change places.

Users of normal mode with text that has runs of katakana, symbols or Latin letters:

• Expect such runs to be split where MeCab splits them: katakana joined by ・ (IPADIC, IPADIC-NEologd, UniDic), unknown symbols right after a dictionary symbol, and Korean sentence-final punctuation such as .". No setting restores the v6 output, and unknown_word_ladder(false) no longer gives the output of Lindera before v6.
• Expect katakana- and Latin-heavy text to take about 13% more instructions, and very long katakana runs in N-best to take much more time and memory.

Users of ko-dic with text that has hanja numerals or 〇:

• Expect a ko-dic hanja numeral and the hanja after it to be one unknown word (三國史記 is one SH word), and 〇 to be a symbol (SY), as in MeCab. No setting restores the v6 output.

Users of CC-CEDICT:

• Expect entries that mix Latin letters or digits with Chinese characters, such as T恤 and 卡拉OK, to be one word, and 〇 and entries such as PK to come with their pinyin and definition. Latin words, runs of kana and numbers stay one word, as in v6.
• Expect the costs of paths, and so N-best costs, to change: unknown words cost 400 instead of −3,200.
• Build a CC-CEDICT dictionary directory from the CC-CEDICT-MeCab-0.1.0-20261021 archive.

Users of Jieba with text that has 〇:

• Expect each 〇 to be one numeral word (m, pinyin ling2) instead of part of an unknown word, as MeCab segments it with the same dictionary.
• Build a Jieba dictionary directory from the mecab-jieba-0.1.3 archive.

Users of Jieba:

• Expect Chinese text to be segmented as UD Chinese GSD segments it, finer than jieba (中华 / 人民 / 共和 / 国, 这 / 个), and as MeCab segments it with the same dictionary.
• Expect simple user dictionary words to cost −32768. Take the context IDs of detailed user dictionary rows from the left-id.def and right-id.def of mecab-jieba-0.1.3.
• Build a Jieba dictionary directory from the mecab-jieba-0.1.3 archive.

Users of IPADIC or IPADIC-NEologd with text that contains ―, —, ～ or 〜:

• Expect entries spelled with ― or ～ to match text spelled the same way, as in MeCab, and text with — or 〜 to miss the 12 IPADIC and 1,085 IPADIC-NEologd entries that it found only through the old rewrite. With IPADIC, map — to ― and ～ to 〜 with the mapping character filter if both spellings should find the entries.

Users who read token details such as the reading or the base form:

• Expect words with tied entries to get the details of the first CSV row, as in MeCab: with IPADIC, 狡い reads ズルイ, not コスイ. No setting restores the v6 output.
• In a user dictionary, expect the first of tied rows to be chosen. A user entry still wins a tie with a system entry, unlike in MeCab.

Everyone who compares the segmentation with v6 or with MeCab:

• Expect the places where two segmentations cost exactly the same, few with the Japanese and Korean dictionaries and Jieba but about a quarter of the sentences with CC-CEDICT, to be segmented as MeCab segments them, with the last word that starts later (IPADIC 腸窒扶斯 is 腸 / 窒扶 / 斯, CC-CEDICT 電視台上班 is 電視 / 台上 / 班). No setting restores the v6 output.

Language bindings and CLI:

• Remove normalize_details where you create Metadata: the Python keyword argument, the ninth positional argument of Ruby's Metadata.new, the last argument of PHP's Metadata constructor, and the normalizeDetails option in Node.js. Stop reading the property and the key of the same name in to_dict(), to_h, toArray() and toObject().
• With lindera tokenize, expect the offsets of a line that starts with whitespace to index the input line, U+3000 at the start or end of a line to be a token (IPADIC, IPADIC-NEologd, UniDic), --keep-whitespace to output the spaces there, -N to give one result for an empty line or a line of spaces, and the wakati format to write an empty line for a line without tokens. To get the v6 offsets and tokens, remove the whitespace at the ends of each line first.
• With the WASM TokenizerBuilder, expect setKeepWhitespace() and the appended filters to apply to a dictionary set with setDictionaryInstance(), and a user dictionary set with setUserDictionaryInstance() to apply to a dictionary set with setDictionary().
• With PHP, pass each argument its declared type: a value of the wrong type for an optional parameter, which v6 read as null, now throws an Exception. Cast strings such as $argv values with (int) or (float).
• With PHP, Node.js and WASM, pass integer arguments in the range of their setting: a value outside it, which v6 wrapped, now throws, as do a negative index and, in Node.js and WASM, a number that is not an integer.
• Pass a Mode to set_mode / setMode and to the Tokenizer constructor instead of "normal" or "decompose": Mode.normal() or Mode.decompose() (Ruby Lindera::Mode.normal, PHP Lindera\Mode::normal()), with a Penalty to customize decompose mode, or Mode.from_str(name) (Mode.fromStr in PHP, Node.js and WASM) for a name read at run time. Replace uses of the old Mode types, such as Python's Mode("normal"), Node.js's JsMode and WASM's Mode.Normal, in the same way.
• With Node.js and TypeScript, rename the type JsPenalty to Penalty.
• With the CLI, expect warnings on standard error, for example for a sentence cut after 32 KiB without a delimiter; set RUST_LOG=error to hide them.
• Nothing else to do beyond taking the 7.0.0 release, apart from the dictionary rebuild, the dictionary, whitespace, 、 and 。, N-best, unknown-word, default-category, dash and tilde, tied-entry and equal-cost items above, the lindera train and lindera build items, and the lindera tokenize, WASM, PHP, integer-argument, Mode, JsPenalty and CLI warning items.