Dictionary Management

Lindera Node.js provides functions for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use loadDictionary(uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

const { loadDictionary } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

const dictionary = loadDictionary("embedded://ipadic");

Dictionary exposes a few read-only accessors for inspecting the loaded dictionary's metadata:

console.log(dictionary.metadataName());     // e.g. "ipadic"
console.log(dictionary.metadataEncoding()); // e.g. "UTF-8"

const metadata = dictionary.metadata(); // full Metadata object
console.log(metadata.defaultWordCost);

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary. Load one for the system dictionary it is used with:

const { loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.csv", dictionary);

A .csv is built with the system dictionary's metadata and checked against its connection matrix as it is built: a row whose context ID lies outside the matrix throws an error that names its CSV line and surface (see Context IDs). A .bin is loaded as it was built.

loadUserDictionary(uri, metadata) loads a user dictionary with metadata only. A .csv loaded this way is checked when it is combined with the system dictionary, and the error names the entry by its word id:

const { loadUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
const userDict = loadUserDictionary("/path/to/user_dictionary.csv", metadata);

Pass the user dictionary when building a tokenizer:

const { Mode, Tokenizer, loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.csv", dictionary);

const tokenizer = new Tokenizer(dictionary, Mode.normal(), userDict);

Or via the builder:

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();
builder.setDictionary("/path/to/ipadic");
builder.setUserDictionary("/path/to/user_dictionary.csv");
const tokenizer = builder.build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

const { buildDictionary, Metadata } = require("lindera");

const metadata = new Metadata({ name: "custom", encoding: "UTF-8" });
buildDictionary("/path/to/input_dir", "/path/to/output_dir", metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

To build a dictionary with the settings its metadata.json ships, load the file with Metadata.fromJsonFile() and pass the result. The file's space_penalty rules are kept, so a ko-dic built this way applies the left-space penalty by default (see setSpacePenalty). A Metadata created with new Metadata(...) carries no rules.

const metadata = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
buildDictionary("/path/to/mecab-ko-dic", "/path/to/output_dir", metadata);

User Dictionary

Build a user dictionary from a CSV file for the system dictionary it is used with:

const { buildUserDictionaryFor, loadDictionary, loadUserDictionaryFor } = require("lindera");

const dictionary = loadDictionary("/path/to/ipadic");
buildUserDictionaryFor("user_words.csv", "/path/to/user_dictionary.bin", dictionary);
const userDict = loadUserDictionaryFor("/path/to/user_dictionary.bin", dictionary);

The CSV is built with the system dictionary's metadata and checked against its connection matrix: a row whose context ID lies outside the matrix throws an error that names its CSV line and surface, and no file is written (see Context IDs). The second argument is the path of the file to write. End it with .bin, since the loaders choose how to load a user dictionary by its extension.

buildUserDictionary(kind, inputFile, outputDir, metadata) builds a user dictionary with metadata only. Despite its name, outputDir is the path of the file to write too, not a directory. The rows are checked when the user dictionary is combined with the system dictionary, and the error names the entry by its word id:

const { buildUserDictionary, Metadata } = require("lindera");

const metadata = new Metadata();
buildUserDictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin", metadata);

The metadata parameter is optional. When omitted, default metadata values are used:

buildUserDictionary("ipadic", "user_words.csv", "/path/to/user_dictionary.bin");

[!NOTE] The first argument (kind) is currently unused -- it is reserved for future use and has no effect on the build. Any string value can be passed for it today. To build for a system dictionary, use buildUserDictionaryFor.

Metadata

The Metadata class configures dictionary parameters.

Creating Metadata

const { Metadata } = require("lindera");

// Default metadata
const metadata = new Metadata();

// Custom metadata
const metadata = new Metadata({
  name: "my_dictionary",
  encoding: "UTF-8",
  defaultWordCost: -10000,
});

Loading from JSON

const metadata = Metadata.fromJsonFile("metadata.json");

Properties

[!NOTE] Schema information (dictionary/user-dictionary field layout) is not exposed on this binding's Metadata object -- there are no dictionarySchema / userDictionarySchema properties. Use the standalone Schema class instead.

All properties support both getting and setting:

const metadata = new Metadata();
metadata.name = "custom_dict";
metadata.encoding = "EUC-JP";
console.log(metadata.name); // "custom_dict"

toObject()

Returns a plain object representation of the metadata, with every value as a string:

const metadata = new Metadata({ name: "test" });
console.log(metadata.toObject());

When the metadata carries left-space penalty rules (for example, when it was loaded from ko-dic's metadata.json), the object also has a spacePenalty entry holding the rules as a JSON string. Otherwise the key is absent.

const koDic = Metadata.fromJsonFile("/path/to/lindera-ko-dic/metadata.json");
const { rules } = JSON.parse(koDic.toObject().spacePenalty);
console.log(rules[1].pos); // ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"]

Schema

The Schema class defines the field structure of dictionary entries.

Creating a Schema

const { Schema } = require("lindera");

// Default IPADIC-compatible schema
const schema = Schema.createDefault();

// Custom schema
const custom = new Schema(["surface", "left_id", "right_id", "cost", "pos", "reading"]);

Schema Methods

const schema = Schema.createDefault();

console.log(schema.fieldCount());           // 13 (IPADIC format)
console.log(schema.getFieldIndex("pos1"));  // e.g., 4
console.log(schema.getAllFields());          // ["surface", "left_id", ...]
console.log(schema.getCustomFields());      // Fields from index 4 onward

FieldDefinition

FieldType