Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return the builder itself (this), so they can be chained or called one statement at a time.

setMode(mode)

Sets the tokenization mode, a Mode:

• Mode.normal() -- Standard tokenization (default)
• Mode.decompose(penalty?) -- Decomposes compound words into smaller units, with the Penalty the mode holds

A value that is not a Mode, such as the mode name "normal", throws an InvalidArg error. A mode name read at run time, such as a setting, becomes a Mode with Mode.fromStr().

const { Mode } = require("lindera");

builder.setMode(Mode.normal());
builder.setMode(Mode.decompose({ kanjiPenaltyLengthPenalty: 0 }));
builder.setMode(Mode.fromStr("decompose"));

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary.csv");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output. Kept whitespace also stays in the lattice, so the other tokens can differ from those of the default output, which skips whitespace in the lattice as MeCab does. To keep whitespace in the lattice without outputting it, use setSkipWhitespace(false) instead.

builder.setKeepWhitespace(true);

setSkipWhitespace(skip)

Sets whether whitespace is skipped in the lattice. The value means the same as segmenter.skip_whitespace in a configuration file.

A value set here overrides the dictionary's default and a skip_whitespace loaded with fromFile. It has no effect while whitespace tokens are kept (setKeepWhitespace(true)), because kept whitespace always stays in the lattice. The argument is required; any other value throws an Error.

With IPADIC, for example, Google が 新しい reads が as a particle (助詞) by default and as a conjunction (接続詞) with setSkipWhitespace(false). See Segmenter for details.

// Keep whitespace in the lattice, as before v7
builder.setSkipWhitespace(false);

// Back to the dictionary's default
builder.setSkipWhitespace(null);

setSpacePenalty(value)

Sets the left-space penalty for Korean (mecab-ko's left-space-penalty-factor): a candidate that starts right after whitespace and whose first part-of-speech tag is listed in a rule gets that rule's cost added. The value means the same as segmenter.space_penalty in a configuration file.

Any other value throws an Error.

// Turn the penalty off
builder.setSpacePenalty(false);

// Explicit rules (these are the ones ko-dic ships)
builder.setSpacePenalty({
  rules: [
    { pos: ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], cost: 3000 },
    { pos: ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], cost: 6000 },
  ],
});

// Back to the dictionary's default
builder.setSpacePenalty(null);

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, because its metadata.json ships mecab-ko-dic's rules. This changes the Korean output (for example, 이 in 검색 이 잘 된다 is the interjection IC with the penalty and the subject particle JKS without it). Call setSpacePenalty(false) to turn it off. For a Tokenizer created directly, pass false as the fourth argument: new Tokenizer(dictionary, Mode.normal(), null, false). A ko-dic downloaded from the v6.0.0 release ships no rules, so with it the penalty stays off and setSpacePenalty(true) makes build() throw. See Segmenter for details.

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

The arguments of setSpacePenalty, appendCharacterFilter and appendTokenFilter may nest arrays and objects at most 128 levels deep. A deeper value, or one that contains itself, throws an Error.

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?, spacePenalty?)

Creates a tokenizer directly from a loaded dictionary.

mode takes a Mode, as setMode does, with the same errors. null or undefined (the default) is normal mode. For decompose mode with a custom penalty, pass Mode.decompose(penalty).

spacePenalty takes the same values as setSpacePenalty. null or undefined (the default) uses the left-space penalty rules the dictionary ships. An invalid setting throws, as setSpacePenalty and build() do.

const { Mode, Tokenizer, loadDictionary } = require("lindera");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, Mode.normal());

// ko-dic without the left-space penalty
const koTokenizer = new Tokenizer(loadDictionary("embedded://ko-dic"), Mode.normal(), null, false);

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of plain token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

Tokens are plain JavaScript objects, not class instances, so they are reclaimed by ordinary garbage collection and pass through JSON.stringify, structuredClone, and worker transfer without conversion. See Token.

tokenizeSurfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no token objects are built and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map((t) => t.surface).

const surfaces = tokenizer.tokenizeSurfaces("形態素解析");
// ["形態素", "解析"]

Parameters:

Returns: string[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: NbestResult[], where each NbestResult is { tokens: Token[], cost: number }

Token

Token is a plain object describing a single morphological token. It is a TypeScript interface, not a class: it has no methods and no prototype, and every field is read directly.

Properties

Reading Details

Index details directly. Out-of-range indexes yield undefined.

const token = tokenizer.tokenize("東京")[0];
const pos = token.details[0];      // e.g., "名詞"
const subpos = token.details[1];   // e.g., "固有名詞"
const reading = token.details[7];  // e.g., "トウキョウ"

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

TokenizerBuilder.setMode() and the Tokenizer constructor take the mode as a Mode, which holds the penalty of decompose mode as a Penalty.

Mode

The tokenization mode: normal, or decompose with a Penalty. It is the counterpart of the Rust lindera::mode::Mode (see Segmenter), and setMode() and the Tokenizer constructor take it:

• Normal mode -- standard dictionary-cost-based tokenization
• Decompose mode -- penalty-based decomposition of compound words

Mode has no constructor, so new Mode() throws; the static factories below create a mode. A mode cannot be changed.

Mode.normal()

Creates normal mode.

const { Mode } = require("lindera");

const mode = Mode.normal();

Mode.decompose(penalty?)

Creates decompose mode with penalty, a Penalty object. null or undefined (the default) uses the default penalty, and a field left out keeps its default. A value that is not an object, or a field that is not an integer in its range, throws an InvalidArg error.

const decompose = Mode.decompose(); // the default penalty
const custom = Mode.decompose({ kanjiPenaltyLengthPenalty: 0 });

Mode.fromStr(name)

Creates a mode from its name, as the Rust Mode parses one: "normal" is normal mode, and "decompose" is decompose mode with the default penalty. The name is case sensitive; any other name, including "Normal" and "Decompose", throws an Error whose message contains Invalid mode. Use it for a mode name read at run time, such as a setting:

const mode = Mode.fromStr("decompose");
console.log(mode.equals(Mode.decompose()));
// true

Properties and Methods

=== compares objects, not modes: each call of Mode.normal() returns a new object, so compare modes with equals.

const { Mode } = require("lindera");

const mode = Mode.decompose({ kanjiPenaltyLengthPenalty: 0 });
console.log(mode.name, mode.isNormal(), mode.isDecompose());
// decompose false true

// A copy with every field set: the field left out has its default
const penalty = mode.penalty;
console.log(penalty.kanjiPenaltyLengthPenalty, penalty.otherPenaltyLengthPenalty);
// 0 1700

// Changing the copy does not change the mode
penalty.otherPenaltyLengthPenalty = 0;
console.log(mode.penalty.otherPenaltyLengthPenalty);
// 1700

console.log(Mode.normal().penalty);
// null

console.log(Mode.decompose().equals(Mode.fromStr("decompose")));
// true
console.log(mode.equals(Mode.decompose()));
// false
console.log(`${Mode.normal()}`);
// normal

Penalty

A plain object (the TypeScript type Penalty) with the penalty parameters of decompose mode. Every field is optional, and a field left out keeps its default:

Pass it to Mode.decompose(), and the mode to setMode() or to the Tokenizer constructor. Without length penalties, decompose mode keeps long words whole, as normal mode does:

const { Mode, Tokenizer, TokenizerBuilder, loadDictionary } = require("lindera");

const text = "関西国際空港限定トートバッグ";
const penalty = { kanjiPenaltyLengthPenalty: 0, otherPenaltyLengthPenalty: 0 };
const surfaces = (tokenizer) => tokenizer.tokenize(text).map((token) => token.surface);

const builder = new TokenizerBuilder().setDictionary("embedded://ipadic");
console.log(surfaces(builder.setMode(Mode.normal()).build()));
// [ '関西国際空港', '限定', 'トートバッグ' ]

console.log(surfaces(builder.setMode(Mode.decompose()).build()));
// [ '関西', '国際', '空港', '限定', 'トートバッグ' ]

console.log(surfaces(builder.setMode(Mode.decompose(penalty)).build()));
// [ '関西国際空港', '限定', 'トートバッグ' ]

const tokenizer = new Tokenizer(loadDictionary("embedded://ipadic"), Mode.decompose(penalty));
console.log(surfaces(tokenizer));
// [ '関西国際空港', '限定', 'トートバッグ' ]