Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return the builder itself (this), so they can be chained or called one statement at a time.

setMode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.setMode("normal");

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.setKeepWhitespace(true);

setSpacePenalty(value)

Sets the left-space penalty for Korean (mecab-ko's left-space-penalty-factor): a candidate that starts right after whitespace and whose first part-of-speech tag is listed in a rule gets that rule's cost added. The value means the same as segmenter.space_penalty in a configuration file.

Any other value throws an Error.

// Turn the penalty off
builder.setSpacePenalty(false);

// Explicit rules (these are the ones ko-dic ships)
builder.setSpacePenalty({
  rules: [
    { pos: ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], cost: 3000 },
    { pos: ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], cost: 6000 },
  ],
});

// Back to the dictionary's default
builder.setSpacePenalty(null);

[!NOTE] Since v6.1.0, ko-dic applies the left-space penalty by default, because its metadata.json ships mecab-ko-dic's rules. This changes the Korean output compared with v6.0 (for example, 시 in 서울 시 에서 becomes the noun NNG instead of the ending EP). Call setSpacePenalty(false) to turn it off. For a Tokenizer created directly, pass false as the fourth argument: new Tokenizer(dictionary, "normal", null, false). A ko-dic downloaded from the v6.0.0 release ships no rules, so with it the penalty stays off and setSpacePenalty(true) makes build() throw. See Segmenter for details.

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

The arguments of setSpacePenalty, appendCharacterFilter and appendTokenFilter may nest arrays and objects at most 128 levels deep. A deeper value, or one that contains itself, throws an Error.

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?, spacePenalty?)

Creates a tokenizer directly from a loaded dictionary.

spacePenalty takes the same values as setSpacePenalty. null or undefined (the default) uses the left-space penalty rules the dictionary ships. An invalid setting throws, as setSpacePenalty and build() do.

const { Tokenizer, loadDictionary } = require("lindera");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

// ko-dic without the left-space penalty
const koTokenizer = new Tokenizer(loadDictionary("embedded://ko-dic"), "normal", null, false);

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of plain token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

Tokens are plain JavaScript objects, not class instances, so they are reclaimed by ordinary garbage collection and pass through JSON.stringify, structuredClone, and worker transfer without conversion. See Token.

tokenizeSurfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no token objects are built and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map((t) => t.surface).

const surfaces = tokenizer.tokenizeSurfaces("形態素解析");
// ["形態素", "解析"]

Parameters:

Returns: string[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: NbestResult[], where each NbestResult is { tokens: Token[], cost: number }

Token

Token is a plain object describing a single morphological token. It is a TypeScript interface, not a class: it has no methods and no prototype, and every field is read directly.

Properties

Reading Details

Index details directly. Out-of-range indexes yield undefined.

const token = tokenizer.tokenize("東京")[0];
const pos = token.details[0];      // e.g., "名詞"
const subpos = token.details[1];   // e.g., "固有名詞"
const reading = token.details[7];  // e.g., "トウキョウ"

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

Mode and Penalty are exported from the lindera npm package, but they are not currently wired into any public API: TokenizerBuilder.setMode() / Tokenizer's constructor accept only a plain mode string ("normal" or "decompose"), and decompose mode always uses the default penalty configuration internally. These types are documented here for completeness; they cannot yet be used to customize penalty behavior from JavaScript.

Mode

A string enum with two values:

• Mode.Normal -- standard dictionary-cost-based tokenization
• Mode.Decompose -- penalty-based decomposition of compound words

Penalty

An object shape describing the penalty parameters used by decompose mode: