Architecture

Module Structure

lindera-trainer/src/
├── lib.rs                # Trainer: CRF training orchestration; public API re-exports
├── config.rs             # TrainerConfig: parses seed lexicon, char.def, unk.def, feature.def, rewrite.def
├── corpus.rs              # Corpus, Example, Word: training data representation
├── feature_extractor.rs  # FeatureExtractor: feature template parsing and feature ID management
├── feature_rewriter.rs   # DictionaryRewriter: MeCab-compatible 3-section rewrite
└── model.rs               # Model, SerializableModel: trained model storage, serialization, dictionary output

Parses the five training input files -- seed lexicon (lex.csv), character definition (char.def), unknown-word definition (unk.def), feature template (feature.def), and rewrite rules (rewrite.def) -- into the configuration consumed by Trainer. TrainerConfig::from_readers (or from_paths, its file-path convenience wrapper) extracts the surface/feature vocabulary from the seed lexicon, builds a FeatureExtractor from the parsed feature templates, builds a DictionaryRewriter from the rewrite rules, and assembles an in-memory lindera_dictionary::dictionary::Dictionary from the seed lexicon, the character definition and the unknown-word definition. char.def is parsed by the dictionary builder (CharacterDefinitionBuilder::build_from_str), so training sees the same character categories as the built dictionary; the default category of each character, which the %t feature uses, comes from the builder's build_default_category_table. The Dictionary (dict()) holds what the training lattices are built from: its prefix_dictionary (system_lexicon()) is the seed lexicon, a prefix dictionary over the seed surfaces whose word ID is the row index, and its unknown_dictionary (unk_handler()) holds the unk.def entries, read with the dictionary builder's parse_unk, one entry per line in file order. Their costs and context IDs are not used in training, and the Dictionary's connection matrix is a placeholder. TrainerConfig also keeps the seed feature strings and the unk.def entries as (category, feature string) pairs in file order, from which the labels and the exported unk.def are made. In unk.def, lines that start with # and empty lines are skipped, and a line with fewer than five fields, or whose context IDs or cost are not numbers, is an error that names the line. As mecab-dict-index and lindera build do, from_readers also rejects a char.def range line that names a category not defined on an earlier line, a category defined twice, an unk.def line whose category char.def does not define, and a char.def category without an unk.def line (lindera_dictionary::dictionary::unknown_dictionary::check_unknown_word_categories), so that such input fails before training rather than when the exported files are built. It also exposes surfaces() and features(), the surfaces and the feature strings of the seed rows in row order, get_features(), a user lexicon accessible via user_lexicon() / add_user_lexicon_entry() / load_user_lexicon_from_content(), and metadata().

Corpus / Example / Word

Represent annotated training data. Word pairs a surface form with its (comma-joined) feature string. Example is one training sentence, built from a Vec<Word> whose surfaces are concatenated to reconstruct the original sentence text. Corpus is a collection of Examples; Corpus::from_reader parses tab-separated surface<TAB>features lines, treating blank lines or literal EOS lines as sentence boundaries.

FeatureExtractor

Parses MeCab-compatible feature templates and manages the mapping from generated feature strings to interned, NonZeroU32 feature IDs. Supported template placeholders are %F[n] / %F?[n] (feature field at index n, the ? form skipped when the value is *), %t (character category), %w (surface form, unigram only), %u / %l / %r (full rewritten unigram/left/right feature string), and %L[n] / %L?[n] / %R[n] / %R?[n] for bigram left/right context fields. A template whose %F[n], %L[n] or %R[n] reads past the last column gives no feature, with or without ?, as in MeCab's learner. extract_unigram_feature_ids[_with_ctx], extract_left_feature_ids[_with_ctx], and extract_right_feature_ids[_with_ctx] apply the parsed templates to a feature vector (plus, for bigram extraction, an optional TemplateContext carrying surface/ufeature/lfeature/rfeature) and return the resulting feature IDs, creating new IDs on first use. The IDs live in three FeatureIdMaps, read through unigram_feature_ids(), left_feature_ids() and right_feature_ids(): the k-th string a map meets gets ID k, an ID is never reused (a feature that training removed and that comes again gets a new one), and the strings are kept back to back in one buffer with only the IDs in the hash table, since the unigram map of a full dictionary holds millions of strings (#1286).

The trainer takes the left and right feature IDs of a label from a cache keyed by the label's rewritten left and right features: the left and right templates read nothing else, so the pair decides the IDs. The first label with a pair makes them with extract_left_feature_ids_with_ctx and extract_right_feature_ids_with_ctx, so new IDs are minted in the order they would be without the cache, and the labels with the same pair share the lists (the 347,344 rows of mecab-ipadic have 1,329 pairs). Training forgets the cache once it removes the unused features. A template is expanded into a buffer that the extractor keeps, so a feature string is allocated only when it is new (#1263).

DictionaryRewriter

Implements MeCab's 3-section rewrite.def format: [unigram rewrite], [left rewrite], and [right rewrite] sections, each holding an ordered list of pattern<TAB>replacement rules built into a FeatureRewriter prefix trie (via the internal FeatureRewriterBuilder). DictionaryRewriter::from_reader parses all three sections (falling back to treating an unsectioned file as legacy right-rewrite-only content, for backward compatibility with older Lindera rewrite.def files). rewrite() applies all three rewriters to a feature string and returns (ufeature, lfeature, rfeature), passing a section through unchanged when no rule matches. As in MeCab, the feature and the pattern and replacement of each rule are split into columns with the rules of MeCab's tokenizeCSV (the crate-private feature_csv::split_feature), so a quoted column with , in it is one column, and rewrite() joins the rewritten columns with ,, quoting a column that contains , or ". The trainer splits the three results the same way for the feature templates, and has them written into buffers it reuses from one label to the next, so that the rewrite allocates only when a buffer grows (#1286). (An earlier rewrite_cached() memoized per full feature string, but real lexicon features are unique per row, so the cache never hit and was removed — #975.)

Trainer

Trainer::new creates the CRF labels: one per seed row, in row order (label row + 1, so that a match in the training lexicon, whose word ID is the row index, maps to its label), then one per unk.def entry, in file order, whose %t is the ID of the entry's category. Trainer::train builds one lindera_crf::Lattice per example, as MeCab's learner does:

The candidates at each position are the seed words that start there and the unknown words that the runtime creates there, listed by lindera_dictionary::viterbi::Lattice::unknown_word_candidates: those of the character's default category, with its INVOKE, GROUP and LENGTH settings, for every unk.def entry of the category.
Each gold token gets the label of the first seed row with the same span and the same whole feature string; else that of the first unk.def entry, in file order, of an unknown-word candidate with the same span whose feature columns other than * all equal the gold token's (both split as rewrite() splits a feature); else a virtual label built from the gold feature string (%t 0, no %w, one per distinct feature string), as MeCab's learner adds a virtual node. Virtual labels follow the other labels, are trained, and are never exported.
Every other candidate becomes a negative edge.

Once the lattices are built, it prints a Lattice statistics: line with the number of gold tokens labelled each way (gold_lexicon, gold_unknown, gold_virtual), the number of virtual labels (virtual_labels), and the negative edges from the seed lexicon and from the unknown-word candidates (negative_lexicon, negative_unknown). When there are gold tokens and none is a seed word, or the gold tokens whose surface has seed rows but none with their feature string outnumber the seed words, it then writes a warning on stderr with eprintln!, not through the log crate, for which the bindings install no logger. The warning gives the first such gold token next to the feature strings of the first five seed rows with its surface. Trainer::max_grouping_len caps unknown-word grouping with the runtime's meaning (characters beyond the first; None, the default, leaves it unbounded); the other builder methods set the regularization (L2 with DEFAULT_LAMBDA, 1.0, by default; use_l2(false) selects L1), the iteration cap, the tolerance of the convergence test (Trainer::eta, default DEFAULT_ETA, 5e-5; see the lindera-crf architecture) and the thread count. After training it prints why the optimizer stopped, with the number of iterations and of evaluations of the loss and its gradient. When an iteration left the objective unchanged or the line search failed far from the optimum, it then writes a warning on stderr, with eprintln! as for the warning about seed words.

Model / SerializableModel

Model is the trained model produced by Trainer::train. It owns the trained lindera_crf::RawModel, the TrainerConfig used to train it, the extracted feature weights and labels, the feature strings of the virtual labels, and any user lexicon entries added afterward via read_user_lexicon. It converts learned CRF weights into MeCab-compatible integer costs (tocost(weight, cost_factor) = clamp(round(-weight * cost_factor), i16::MIN, i16::MAX), i.e. clamped to [-32768, 32767]), with the cost factor computed to make full use of the i16 range (calculate_cost_factor). Model can serialize itself (write_model) and export dictionary source files directly.

SerializableModel is the plain-data, rkyv-serializable counterpart returned by Model::read_model when loading a previously trained model back from a reader (as done by the lindera export CLI command). It carries the same trained information as Model -- feature weights, labels, POS info, connection cost matrix, the unk.def entries (unk_entries, (category, feature string) pairs in file order), preserved char.def/feature.def/rewrite.def content, cost factor, and left/right context ID maps -- as owned data, without a TrainerConfig, and exposes its own writer methods for producing the dictionary export files.

`Model` public methods

Method	Description
`read_user_lexicon<R: Read>(&mut self, rdr: R) -> Result<()>`	Loads a user-defined lexicon (same `surface,left_id,right_id,cost,feature...` CSV format as the seed lexicon) into the model so that `write_dictionary` can fill in the parameters its entries do not give. As in MeCab's `mecab-dict-index`, the cost and the connection IDs are filled in separately: an empty cost is computed, and when either connection ID is empty or negative, both are inferred. An entry whose parameters are `0,0,0` gets all three, as in Vibrato's trainer (`mecab-dict-index` keeps them). A connection ID or cost that is neither empty nor a number in its range is an error that names the entry, and no entry is added then; `mecab-dict-index` reads such a value as a number with `atoi`, but `lindera build` rejects it. Must be called before writing the dictionary if a user lexicon is needed.
`write_model<W: Write>(&self, writer: &mut W) -> Result<()>`	Serializes the full trained model (feature weights, labels, POS info, connection matrix, `unk.def` entries, preserved definition file contents, cost factor, left/right ID maps) to `rkyv` binary format, with `metadata.version` set to `model::MODEL_FORMAT_VERSION` (`3.0.0`). The stored matrix size covers every context ID of the model, whether or not a weight survived for it. The output is byte-reproducible: the connection matrix is an ordered map and the `unk.def` entries a `Vec`, so the bytes depend only on the model's contents.
`read_model<R: Read>(reader: R) -> Result<SerializableModel>`	Associated function. Deserializes a `SerializableModel` from data written by `write_model`. It tries `rkyv` first, and falls back to legacy JSON only when the payload actually begins a JSON object; anything else is reported as a format mismatch with the instruction to re-train. A model, in either format, whose `metadata.version` is not `MODEL_FORMAT_VERSION` is rejected with the same instruction, so every model written before 7.0.0 must be retrained. Backfills `feature_sets` from `feature_weights` for older models that predate that field.
`write_dictionary<W1, W2, W3, W4>(&self, lexicon_wtr, connector_wtr, unk_handler_wtr, user_lexicon_wtr) -> Result<()>`	Convenience method that writes the lexicon, connection cost matrix, unknown-word dictionary, and user lexicon in one call. The user lexicon is written from the entries loaded by `read_user_lexicon`, in input order. An entry without connection IDs gets those that `lex.csv` gives the seed row whose feature is the most similar to its own. An entry without a cost gets the one MeCab's `calcCost` computes: the sum of the trained weights of its unigram features, converted with `tocost` and the cost factor of `lex.csv`. Its features are made as a seed row's are, from the rewritten feature, the default category of its first character (`%t`) and its surface (`%w`, which MeCab does not accept in a unigram template), so an entry whose feature and first character's category are those of a seed row gets the cost `lex.csv` gives that row; a feature the model has no weight for adds nothing. A parameter the entry gives is written as it is. The most similar row is the one with the highest score, where each column equal to the entry's at the same position scores more the earlier it is, and the first one on a tie. An entry without connection IDs whose feature has no column in common with any seed row is an error, as in MeCab's `mecab-dict-index`, and nothing is written then.
`write_lexicon<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `lex.csv`-style entries (`surface,left_id,right_id,cost,features...`) for every seed vocabulary entry, using the connection IDs and costs from the merged CRF model.
`write_connection_costs<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes a dense `matrix.def`-style connection cost matrix over every `(right_id, left_id)` pair of the merged model's context IDs (including 0 for BOS/EOS). A pair gets its trained weight converted with `tocost`; a pair without a trained weight, because its bigram features never met in training or regularization zeroed their weights, gets 0, the cost the transition has in training and in MeCab's `mecab-dict-gen`. It is the matrix that `SerializableModel::write_connection_costs` writes for the model that `write_model` writes.
`write_unknown_dictionary<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes one `unk.def` line for each `unk.def` entry the model was trained with, in file order, with the entry's category name and feature string and its learned connection IDs and cost.
`get_unknown_word_cost(&self, category: usize) -> i32`	Returns the input cost of a `char.def` category index, read from `unk.def`'s cost column rather than learned; for a category with several entries, the cost of its last line. Defaults to `2000` if the category has no configured cost.
`num_features(&self) -> usize`	Number of feature weights held by the trained model.
`num_labels(&self) -> usize`	Number of exported labels in the trained model: the seed rows plus the `unk.def` entries. The virtual labels are not counted.
`raw_model(&self) -> &lindera_crf::RawModel`	Access to the underlying `lindera-crf` raw model, for advanced or lower-level operations.
`write_bigram_details<L: Write, R: Write, C: Write>(&self, left_wtr, right_wtr, cost_wtr) -> Result<()>`	Writes three diagnostic files describing bigram features and costs (left feature list, right feature list, and one line per raw bigram feature pair with `BOS`/`EOS` for id 0 and names from the feature extractor). All three files are byte-reproducible for the same model.
`evaluate(&self, test_lattices: &[lindera_crf::Lattice]) -> f64`	Returns the mean absolute value of the raw model's feature weights as a simple evaluation score. The `test_lattices` argument is currently unused; it does not yet score the model against held-out data.
`write_dictionary_buffers(&self, lexicon, connector, unk_handler, user_lexicon: &mut Vec<u8>) -> Result<()>`	Serializes labels, feature weights, user-entry count, and surfaces into four raw `rkyv` byte buffers -- a lower-level alternative to the CSV-oriented `write_dictionary`.
`write_left_id_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `left-id.def`: `0 BOS/EOS`, then each left context ID of the merged model -- the IDs that `write_dictionary` writes to `lex.csv`, `unk.def` and `matrix.def` -- with the left feature of the first label with that ID. It is the file that `SerializableModel::write_left_id_def` writes for the model that `write_model` writes: both use one mapping.
`write_right_id_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `right-id.def`: `0 BOS/EOS`, then each right context ID of the merged model with the right feature of the first label with that ID, as `write_left_id_def` does for the left IDs. It is the file that `SerializableModel::write_right_id_def` writes for the model that `write_model` writes.

`SerializableModel` public methods

Available on the value returned by Model::read_model, for re-exporting dictionary source files from a previously trained and serialized model (used by the lindera export CLI command and the export functions of the Python, Node.js, Ruby and PHP bindings):

Method	Description
`export(&self, output_dir: &Path, base_metadata: Option<&Path>) -> Result<Vec<PathBuf>>`	Writes the dictionary source files to `output_dir`, which it creates when it does not exist, with the methods below: `lex.csv`, `matrix.def`, `unk.def`, `char.def`, `feature.def`, `rewrite.def`, `left-id.def` and `right-id.def`, and `metadata.json` (`update_metadata_json`) when `base_metadata` is given. Returns the paths written, in that order. `lindera export` and the bindings' export functions call it, so they write the same files. `metadata.json` is made first, so a base `metadata.json` that cannot be read fails with no file written; an error names the file.
`write_lexicon<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `lex.csv`-style entries from the stored `feature_sets`, `labels`, and `pos_info`, skipping the `unk.def` entry labels that follow the seed rows.
`write_connection_costs<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes the dense `matrix.def`-style connection cost matrix from the stored `connection_matrix`, `max_left_id`, and `max_right_id`: `max_right_id + 1` rows by `max_left_id + 1` columns, which cover every context ID of the model even when no bigram weight survived training. A cell missing from `connection_matrix`, a connection without a trained weight, gets 0, as in `Model::write_connection_costs`.
`write_unknown_dictionary<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes one `unk.def` line for each stored entry of `unk_entries`, in file order, with its category name, learned connection IDs and cost, and feature string.
`write_char_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes the `char.def` content preserved verbatim from the original training input.
`write_feature_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes the `feature.def` content preserved verbatim from the original training input.
`write_rewrite_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes the `rewrite.def` content preserved verbatim from the original training input.
`write_left_id_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `left-id.def` from the stored `left_id_map`, with `0 BOS/EOS` as the first line.
`write_right_id_def<W: Write>(&self, writer: &mut W) -> Result<()>`	Writes `right-id.def` from the stored `right_id_map`, with `0 BOS/EOS` as the first line.
`update_metadata_json<W: Write>(&self, base_metadata_path: &Path, writer: &mut W) -> Result<()>`	Reads a base `metadata.json`, updates it with training-derived values (`default_word_cost` estimated from the median feature weight, plus a `model_info` section with feature/label counts, context ID ranges, the `matrix.def` size as `connection_matrix_size` (`{rows}x{columns}`), and training metadata), and writes the result. The output carries no timestamp and is byte-reproducible for the same model.

`ModelMetadata` and `FeatureSetInfo`

ModelMetadata records summary information about a training run: version (the model format version, MODEL_FORMAT_VERSION), regularization (the cost coefficient used), regularization_type (the RegularizationType used, exported as model_info.regularization_type), elastic_net_l1_ratio (the L1 ratio, None unless the type is Elastic Net; exported for Elastic Net only), iterations (the number of L-BFGS iterations that ran, at most the maximum set with Trainer::max_iter; exported as model_info.training_iterations), feature_count, and label_count. It is embedded in SerializableModel::metadata.

FeatureSetInfo holds the per-label connection information extracted from the merged CRF model after training: left_id, right_id, and weight. SerializableModel::feature_sets stores one entry per CRF label: one for each entry of labels, in the same order, then one for each virtual label, which is not exported.

Lindera Documentation