Tokenizer API

TokenizerBuilder

TokenizerBuilder configures and constructs a Tokenizer instance using the builder pattern.

Constructors

new TokenizerBuilder()

Creates a new builder with default configuration.

const { TokenizerBuilder } = require("lindera");

const builder = new TokenizerBuilder();

new TokenizerBuilder().fromFile(filePath)

Loads configuration from a YAML file and returns a new builder.

const builder = new TokenizerBuilder().fromFile("lindera.yml");

See lindera-nodejs/resources/lindera.yml for a complete example configuration file, including dictionary, character filter, and token filter settings.

Configuration Methods

All setter methods return the builder itself (this), so they can be chained or called one statement at a time.

setMode(mode)

Sets the tokenization mode.

• "normal" -- Standard tokenization (default)
• "decompose" -- Decomposes compound words into smaller units

builder.setMode("normal");

setDictionary(path)

Sets the system dictionary path or URI.

// Use an embedded dictionary
builder.setDictionary("embedded://ipadic");

// Use an external dictionary
builder.setDictionary("/path/to/dictionary");

setUserDictionary(uri)

Sets the user dictionary URI.

builder.setUserDictionary("/path/to/user_dictionary");

setKeepWhitespace(keep)

Controls whether whitespace tokens appear in the output.

builder.setKeepWhitespace(true);

appendCharacterFilter(kind, args?)

Appends a character filter to the preprocessing pipeline.

builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });

appendTokenFilter(kind, args?)

Appends a token filter to the postprocessing pipeline.

builder.appendTokenFilter("lowercase", {});

Build

build()

Builds and returns a Tokenizer with the configured settings.

const tokenizer = builder.build();

Tokenizer

Tokenizer performs morphological analysis on text.

Creating a Tokenizer

new Tokenizer(dictionary, mode?, userDictionary?)

Creates a tokenizer directly from a loaded dictionary.

const { Tokenizer, loadDictionary } = require("lindera");

const dictionary = loadDictionary("embedded://ipadic");
const tokenizer = new Tokenizer(dictionary, "normal");

Tokenizer Methods

tokenize(text)

Tokenizes the input text and returns an array of plain token objects.

const tokens = tokenizer.tokenize("形態素解析");

Parameters:

Returns: Token[]

Tokens are plain JavaScript objects, not class instances, so they are reclaimed by ordinary garbage collection and pass through JSON.stringify, structuredClone, and worker transfer without conversion. See Token.

tokenizeSurfaces(text)

Tokenizes the input text and returns only the token surfaces, as an array of strings. This is the fast path for wakati-style use: no token objects are built and no morphological details are loaded, so it is significantly faster than tokenize when only the surface strings are needed. The result equals tokenizer.tokenize(text).map((t) => t.surface).

const surfaces = tokenizer.tokenizeSurfaces("形態素解析");
// ["形態素", "解析"]

Parameters:

Returns: string[]

tokenizeNbest(text, n, unique?, costThreshold?)

Returns the N-best tokenization results, each containing tokens and total path cost.

const results = tokenizer.tokenizeNbest("すもももももももものうち", 3);
for (const { tokens, cost } of results) {
  console.log(cost, tokens.map((t) => t.surface));
}

Parameters:

Returns: NbestResult[], where each NbestResult is { tokens: Token[], cost: number }

Token

Token is a plain object describing a single morphological token. It is a TypeScript interface, not a class: it has no methods and no prototype, and every field is read directly.

Properties

Reading Details

Index details directly. Out-of-range indexes yield undefined.

const token = tokenizer.tokenize("東京")[0];
const pos = token.details[0];      // e.g., "名詞"
const subpos = token.details[1];   // e.g., "固有名詞"
const reading = token.details[7];  // e.g., "トウキョウ"

The structure of details depends on the dictionary:

• IPADIC: [品詞, 品詞細分類1, 品詞細分類2, 品詞細分類3, 活用型, 活用形, 原形, 読み, 発音]
• UniDic: Detailed morphological features following the UniDic specification
• ko-dic / CC-CEDICT / Jieba: Dictionary-specific detail formats

Mode and Penalty

Mode and Penalty are exported from the lindera npm package, but they are not currently wired into any public API: TokenizerBuilder.setMode() / Tokenizer's constructor accept only a plain mode string ("normal" or "decompose"), and decompose mode always uses the default penalty configuration internally. These types are documented here for completeness; they cannot yet be used to customize penalty behavior from JavaScript.

Mode

A string enum with two values:

• Mode.Normal -- standard dictionary-cost-based tokenization
• Mode.Decompose -- penalty-based decomposition of compound words

Penalty

An object shape describing the penalty parameters used by decompose mode: