Filters

Character filters and token filters are the two pre/post-processing stages of the lindera-analysis Tokenizer pipeline.

• Character filters transform the input text before segmentation. Byte offsets are corrected automatically so that the resulting tokens still report positions relative to the original, unfiltered text.
• Token filters transform the list of tokens produced by the segmenter after segmentation.

Both kinds of filters are configured the same way: a kind string (also used with the CLI's --character-filter / --token-filter flags in the form kind:{"json": "args"}) and a JSON args object with filter-specific parameters.

Character Filters

Character filters are configured under the character_filters key of the YAML configuration file. Each entry is applied in order, and each filter's output feeds into the next.

unicode_normalize

Normalizes the input text using one of the four standard Unicode normalization forms.

Parameters:

Example:

{
  "kind": "unicode_normalize",
  "args": {
    "kind": "nfkc"
  }
}

japanese_iteration_mark

Normalizes Japanese iteration marks (々, ゝ, ゞ, ヽ, ヾ) by replacing each mark with the character it repeats, adding or removing the voiced sound mark (dakuten) as needed for the hiragana/katakana variants.

Parameters:

Example:

{
  "kind": "japanese_iteration_mark",
  "args": {
    "normalize_kanji": true,
    "normalize_kana": true
  }
}

mapping (character filter)

Replaces occurrences of the keys in mapping with their corresponding values, using longest-match search over the input text (built with an Aho-Corasick automaton).

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "リンデラ": "Lindera"
    }
  }
}

regex

Replaces every match of a regular expression in the input text with a literal replacement string. Capture groups are not interpolated into the replacement.

Parameters:

Example:

{
  "kind": "regex",
  "args": {
    "pattern": "\\s{2,}",
    "replacement": " "
  }
}

Token Filters

Token filters are configured under the token_filters key of the YAML configuration file. Each filter is applied in order to the token list produced by the segmenter.

japanese_base_form

Replaces the token's surface text with its base (dictionary) form, as registered in the base_form or orthographic_base_form field of the dictionary. Acts as a lemmatizer for verbs and adjectives. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_base_form"
}

japanese_compound_word

Merges consecutive tokens whose part-of-speech tag matches one of tags into a single compound token.

The merged token's details carry new_tag at the part-of-speech position resolved from the dictionary schema; any leading non-part-of-speech field (SudachiDict's display_surface) and every remaining field are set to *.

Parameters:

Example:

{
  "kind": "japanese_compound_word",
  "args": {
    "tags": [
      "名詞,数",
      "名詞,接尾,助数詞"
    ],
    "new_tag": "名詞,数"
  }
}

japanese_kana

Converts token text between hiragana and katakana.

Parameters:

Example:

{
  "kind": "japanese_kana",
  "args": {
    "kind": "hiragana"
  }
}

japanese_katakana_stem

Removes a trailing prolonged sound mark (ー, U+30FC) from katakana tokens, but only when the token is longer than min characters.

Parameters:

Example:

{
  "kind": "japanese_katakana_stem",
  "args": {
    "min": 3
  }
}

japanese_keep_tags

Keeps only tokens whose part-of-speech tag matches one of tags, removing all others.

Parameters:

Example:

{
  "kind": "japanese_keep_tags",
  "args": {
    "tags": [
      "名詞,一般",
      "名詞,固有名詞,一般"
    ]
  }
}

japanese_number

Converts Japanese numeral representations (kanji numerals, formal/legal kanji numerals, and fullwidth digits) in the token's surface text to Arabic numerals.

A token is converted only when every one of its characters takes part in a numeral, the rule Lucene's JapaneseNumberFilter applies. An ordinary word that merely begins with a numeral character is therefore returned unchanged: 一部 stays 一部 rather than becoming 1部.

When this filter follows japanese_compound_word, merge only the 名詞,数 tokens. A counter (名詞,接尾,助数詞) merged into the number token makes the merged token something other than a numeral, so 一万円 is left alone instead of being converted.

Parameters:

Example:

{
  "kind": "japanese_number",
  "args": {
    "tags": [
      "名詞,数"
    ]
  }
}

japanese_reading_form

Replaces the token's surface text with its reading, in katakana, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "japanese_reading_form"
}

japanese_stop_tags

Removes tokens whose part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "japanese_stop_tags",
  "args": {
    "tags": [
      "助詞,格助詞,一般",
      "助詞,係助詞",
      "助詞,連体化",
      "助動詞"
    ]
  }
}

Tags are normalized to exactly four comma-separated levels (missing levels are padded with *) and compared for exact equality against the four part-of-speech details of each token, starting at the field named part_of_speech in the dictionary schema (or at the first detail when the schema has no such field). A bare 助詞 therefore never matches IPADIC particle tokens — they always carry a subcategory such as 助詞,係助詞 — while a bare 助動詞 does match, because auxiliary verbs have no subcategory (助動詞,*,*,*). The same matching rule applies to japanese_keep_tags, japanese_compound_word, and the tags option of japanese_number. Because the position is resolved through the schema, the rule also holds for dictionaries whose schema has leading non-part-of-speech columns, such as SudachiDict's display_surface.

keep_words

Keeps only tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "keep_words",
  "args": {
    "words": [
      "すもも",
      "もも"
    ]
  }
}

korean_compound_word

Merges consecutive Korean tokens whose first part-of-speech tag matches one of tags into a single compound token. The merged token keeps the first token's byte_start and position, takes the last token's byte_end, and its position_length is the number of tokens merged. A matching token whose neighbours do not match is left as it is.

ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so this filter is what turns 이/NR 천/NR 이/NR 십/NR 육/NR back into one token before korean_number converts it:

Parameters:

Example:

{
  "kind": "korean_compound_word",
  "args": {
    "tags": [
      "SN",
      "NR"
    ],
    "new_tag": "NR"
  }
}

만 right after a space is NR under the default left-space penalty, but at the start of the text or right after another numeral ko-dic tags it JX (만 원, 십만 원), which stays outside ["SN", "NR"].

korean_decompound

Splits ko-dic compound, inflected and pre-analyzed tokens into the morphemes their expression field lists, reproducing Lucene nori's DecompoundMode. This is the inverse of korean_compound_word, which merges neighbouring tokens. Lindera's Mode::Decompose applies a length penalty tuned for Japanese kanji and does not read these fields, so 무궁화 stays whole in every mode without this filter.

Fragment offsets follow nori: only Compound fragments get their own byte range, each covering the text it spells. Inflect and Preanalysis fragments share the whole token's span, because an inflected form's fragments need not be substrings of it (갔 decomposes to 가 + 았).

A Compound token is carved only when its fragments spell its surface exactly; otherwise it shares its span too. That covers the 12 bundled ko-dic entries whose fragments are not the surface's own text (안지랑역 = 안지 + 역, 고춧값 = 고추 + 값), where nori's offsets point some fragments at text they do not spell, as well as a user-dictionary entry such as 프린터3D = 프린터 + 쓰리디 and a surface an earlier filter has rewritten. For every other bundled entry the offsets are nori's, and every offset is safe to slice the text with.

In discard the fragments take consecutive positions starting at the compound's own; in mixed the compound keeps that position with a position_length spanning its fragments, which are then emitted from the same position onward. Tokens after the expansion shift by the positions it added, and only by those, so a gap left by an earlier filter such as korean_stop_tags survives.

Only tokens spanning a single position are expanded, so the compound a mixed pass keeps is not split again. Put the filter in a chain once all the same: a single-fragment entry kept by mixed still spans one position. When combining it with korean_compound_word, place the merge first; merging never joins tokens that overlap in byte range, so the reverse order does not corrupt anything, but merging first lets it see the segmenter's own tokens.

A fragment takes its own part-of-speech tag and semantic class and blanks the fields that describe the compound (type, expression, first_part_of_speech, last_part_of_speech, reading, presence_absence). Blanking the reading matches nori, whose DecompoundToken.getReading() returns null; korean_reading_form skips *, so a fragment keeps its own surface.

Applied to a dictionary whose schema has no type/expression fields, the filter warns once per configured filter and passes every token through unchanged.

Parameters:

Example:

{
  "kind": "korean_decompound",
  "args": {
    "mode": "discard",
    "types": [
      "Compound",
      "Inflect",
      "Preanalysis"
    ]
  }
}

korean_keep_tags

Keeps only Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_keep_tags",
  "args": {
    "tags": [
      "NNG"
    ]
  }
}

korean_number

Converts Sino-Korean numeral representations (Hangul numerals, Hanja numerals, and fullwidth digits) in the token's surface text to Arabic numerals. Native Korean numerals (하나, 둘, 열, 스물, ...) have no positional structure and are left unchanged.

Each token is converted on its own. ko-dic tokenizes a Sino-Korean numeral into one token per morpheme, so the tokens of a single number do not add up to one Arabic numeral:

Put korean_compound_word in front of this filter to merge those tokens first, the way japanese_compound_word does for IPADIC. Its new_tag must be one of this filter's tags for the merged token to be converted:

[
  {"kind": "korean_compound_word", "args": {"tags": ["SN", "NR"], "new_tag": "NR"}},
  {"kind": "korean_number", "args": {"tags": ["SN", "NR"]}}
]

With that chain 이천이십육년 becomes 2026 년, 10만 becomes 100000 and 2천26 becomes 2026. A single token that holds the whole numeral, such as the Hanja 二千二十六/SH, is converted in full by this filter alone.

Parameters:

A token is converted only when every one of its characters is a numeral, the rule Lucene's KoreanNumberFilter applies. That is what keeps native Korean numerals intact: 일곱 ("seven") starts with the Sino-Korean 일, and a character-by-character conversion would give 1곱.

Hangul numerals are also homographs of very common morphemes (이 is a subject particle, 만 an auxiliary particle), and those are single characters that the all-numeral rule cannot separate from a numeral. That is why the default is restricted to the numeral tags.

Add SH to convert Hanja numerals as well. The coverage is partial in two ways: ko-dic tags whole Hanja words as NNG (參加, 萬歲 are untouched), while a single Hanja character it cannot attach to a word is SH on its own, so 十字架 becomes 10 字 架 and 百貨店 becomes 100 貨 店.

Example:

{
  "kind": "korean_number",
  "args": {
    "tags": [
      "SN",
      "NR"
    ]
  }
}

korean_reading_form

Replaces the token's surface text with its reading, as registered in the dictionary's reading field. Tokens produced by unknown-word processing (token.word_id.is_unknown()) are left unchanged.

This filter takes no configuration parameters.

Example:

{
  "kind": "korean_reading_form"
}

korean_stop_tags

Removes Korean tokens whose first part-of-speech tag matches one of tags.

Parameters:

Example:

{
  "kind": "korean_stop_tags",
  "args": {
    "tags": [
      "EP",
      "EF",
      "JKG"
    ]
  }
}

length

Keeps only tokens whose surface text length (in characters) falls within [min, max].

Parameters:

Example:

{
  "kind": "length",
  "args": {
    "min": 2,
    "max": 3
  }
}

lowercase

Converts token surface text to lowercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "lowercase"
}

mapping (token filter)

Replaces occurrences of the keys in mapping with their corresponding values in each token's surface text, using longest-match search (built with an Aho-Corasick automaton). This is the token-level counterpart of the mapping character filter.

Parameters:

Example:

{
  "kind": "mapping",
  "args": {
    "mapping": {
      "籠": "篭"
    }
  }
}

remove_diacritical_mark

Removes diacritical (combining) marks from token surface text, re-applying the text's original Unicode normalization form afterward.

Parameters:

Example:

{
  "kind": "remove_diacritical_mark",
  "args": {
    "japanese": false
  }
}

stop_words

Removes tokens whose surface text exactly matches one of words.

Parameters:

Example:

{
  "kind": "stop_words",
  "args": {
    "words": [
      "も",
      "の"
    ]
  }
}

uppercase

Converts token surface text to uppercase.

This filter takes no configuration parameters.

Example:

{
  "kind": "uppercase"
}

YAML Configuration

Character filters and token filters are configured together with the segmenter in a single YAML file. See Configuration for the full file format; the relevant excerpt looks like this:

character_filters:
  - kind: "unicode_normalize"
    args:
      kind: "nfkc"
  - kind: "japanese_iteration_mark"
    args:
      normalize_kanji: true
      normalize_kana: true

token_filters:
  - kind: "japanese_stop_tags"
    args:
      tags:
        - "助詞,格助詞,一般"
        - "助詞,係助詞"
        - "助詞,連体化"
        - "助動詞"
  - kind: "japanese_katakana_stem"
    args:
      min: 3
  - kind: "lowercase"
  - kind: "length"
    args:
      min: 2

Rust API

Character filters and token filters can also be created and applied programmatically:

use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::analysis::character_filter::BoxCharacterFilter;
use lindera::analysis::character_filter::unicode_normalize::{
    UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera::analysis::token_filter::BoxTokenFilter;
use lindera::analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera::analysis::token_filter::japanese_katakana_stem::JapaneseKatakanaStemTokenFilter;
use lindera::analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;

fn main() -> LinderaResult<()> {
    let dictionary = load_dictionary("embedded://ipadic")?;
    let segmenter = Segmenter::new(Mode::Normal, dictionary, None);

    let mut tokenizer = Tokenizer::new(segmenter);

    // Add a character filter
    let normalize_filter = UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);
    tokenizer.append_character_filter(BoxCharacterFilter::from(normalize_filter));

    // Add token filters
    let stop_tags_filter = JapaneseStopTagsTokenFilter::new(
        vec![
            "助詞,格助詞,一般".to_string(),
            "助詞,係助詞".to_string(),
            "助詞,連体化".to_string(),
            "助動詞".to_string(),
        ]
        .into_iter()
        .collect(),
    );
    tokenizer.append_token_filter(BoxTokenFilter::from(stop_tags_filter));

    let katakana_stem_filter =
        JapaneseKatakanaStemTokenFilter::new(std::num::NonZeroUsize::new(3).unwrap());
    tokenizer.append_token_filter(BoxTokenFilter::from(katakana_stem_filter));

    // Tokenize with filters applied
    let tokens = tokenizer.tokenize("Linderaは形態素解析エンジンです。")?;

    for token in tokens {
        println!(
            "token: {:?}, details: {:?}",
            token.surface, token.details
        );
    }

    Ok(())
}

The append_character_filter and append_token_filter methods add filters in order. Character filters are applied sequentially to the text before segmentation; token filters are applied sequentially to the token list after segmentation.