Lindera ko-dic

Dictionary version

This repository contains mecab-ko-dic.

Dictionary format

Information about the dictionary format and part-of-speech tags used by mecab-ko-dic id documented in this Google Spreadsheet, linked to from mecab-ko-dic's repository readme.

Note how ko-dic has one less feature column than NAIST JDIC, and has an altogether different set of information (e.g. doesn't provide the "original form" of the word).

The tags are a slight modification of those specified by 세종 (Sejong), whatever that is. The mappings from Sejong to mecab-ko-dic's tag names are given in tab 태그 v2.0 on the above-linked spreadsheet.

The dictionary format is specified fully (in Korean) in tab 사전 형식 v2.0 of the spreadsheet. Any blank values default to *.

Left-space penalty

mecab-ko (the MeCab fork mecab-ko-dic is built for) raises the connection cost of a word whose left side is preceded by whitespace when its pos-id.def id is listed in dicrc:

left-space-penalty-factor = 100,3000,120,6000,172,3000,183,3000,184,3000,185,3000,200,3000,210,6000,220,3000,221,3000,222,3000,230,3000

Lindera does not store pos-id.def ids, but every listed id corresponds to the entry's first part-of-speech tag (for Inflect rows, the First part-of-speech column), so the same rules are written by tag. ko-dic ships them in its metadata.json (space_penalty), and Segmenter::new applies them by default. Opt out with Segmenter::space_penalty(None), "space_penalty": false in the segmenter config, or lindera tokenize --disable-space-penalty; explicit rules go through Segmenter::space_penalty, a space_penalty object, or --space-penalty-rules. The shipped rules are present only in a ko-dic built by Lindera 6.1.0 or later; with a ko-dic from the v6.0.0 release the penalty silently stays off until the dictionary is rebuilt, while --space-penalty-rules works regardless.

{
  "rules": [
    { "pos": ["EC", "EF", "EP", "ETM", "ETN", "VCP", "XSA", "XSN", "XSV"], "cost": 3000 },
    { "pos": ["JC", "JKB", "JKC", "JKG", "JKO", "JKQ", "JKS", "JKV", "JX"], "cost": 6000 }
  ]
}

echo "검색 이 잘 된다" | lindera tokenize --dict embedded://ko-dic

이 after the space is read as the interjection IC (as mecab-ko does) instead of the subject particle JKS; with --disable-space-penalty it reads as JKS again. See the Segmenter documentation for the semantics.

Whitespace

mecab-ko skips whitespace when it builds the lattice, so a word after a space connects to the word before it. The SPACE unknown-word entry (unk.def: SPACE,1798,3563,996) is therefore never connected to anything during training, and its row and column of matrix.def are all zero. Lindera skips whitespace the same way by default. With --disable-skip-whitespace (Segmenter::skip_whitespace(false), "skip_whitespace": false) whitespace stays in the lattice as that entry, which resets the context at every space: 2년 전 대회 then reads 전 as 저/NP + ㄴ/JX instead of the noun NNG, and 하고 있다 reads 있 as a main verb instead of the auxiliary VX.

User dictionary format (CSV)

Simple version

Detailed version

API reference

The API reference is available. Please see following URL:

• lindera-ko-dic