Examples
This page shows tokenization examples using the UniDic CSJ dictionary.
Tokenize with external UniDic CSJ
Download the dictionary with lindera download unidic-csj, or build it, and pass its directory:
% echo "そんなら君の指を切ってみろ。" | lindera tokenize \
--dict /tmp/lindera-unidic-csj-202512
そんな 形状詞,一般,*,*,*,*,ソンナ,そんな,そんな,ソンナ,そんな,ソンナ,和,*,*,*,*,*,*,相,ソンナ,ソンナ,ソンナ,ソンナ,0,C2,*,5912632401994240,21510
ら 接尾辞,名詞的,一般,*,*,*,ラ,等,ら,ラ,ら,ラ,和,*,*,*,*,*,*,接尾体,ラ,ラ,ラ,ラ,*,C3,*,10877477157216768,39572
君 接尾辞,名詞的,一般,*,*,*,クン,君,君,クン,君,クン,漢,*,*,*,*,*,*,接尾体,クン,クン,クン,クン,*,C4,*,2928557877764608,10654
の 助詞,格助詞,*,*,*,*,ノ,の,の,ノ,の,ノ,和,*,*,*,*,*,*,格助,ノ,ノ,ノ,ノ,*,名詞%F1,*,7968444268028416,28989
指 名詞,普通名詞,一般,*,*,*,ユビ,指,指,ユビ,指,ユビ,和,*,*,*,*,*,*,体,ユビ,ユビ,ユビ,ユビ,2,C3,*,10688361157239296,38884
を 助詞,格助詞,*,*,*,*,ヲ,を,を,オ,を,オ,和,*,*,*,*,*,*,格助,ヲ,ヲ,ヲ,ヲ,*,動詞%F2@0,名詞%F1,形容詞%F2@-1,*,11381878116459008,41407
切っ 動詞,非自立可能,*,*,五段-ラ行,連用形-促音便,キル,切る,切っ,キッ,切る,キル,和,キ濁,基本形,*,*,*,*,用,キッ,キル,キッ,キル,1,C1,*,2626192180126340,9554
て 助詞,接続助詞,*,*,*,*,テ,て,て,テ,て,テ,和,*,*,*,*,*,*,接助,テ,テ,テ,テ,*,動詞%F1,形容詞%F2@-1,*,6837321680953856,24874
みろ 動詞,非自立可能,*,*,上一段-マ行,命令形,ミル,見る,みろ,ミロ,みる,ミル,和,*,*,*,*,*,*,用,ミロ,ミル,ミロ,ミル,1,C1,*,10148500948001573,36920
。 補助記号,句点,*,*,*,*,*,。,。,*,。,*,記号,*,*,*,*,*,*,補助,*,*,*,*,*,*,*,6880571302400,25
EOS
The fields are those of UniDic CWJ, but the costs are not, so the two can split a text differently: UniDic CSJ splits "そんなら" as "そんな" and "ら", where UniDic CWJ splits it as "そん" and "なら".
Tokenize with embedded UniDic CSJ
% echo "そんなら君の指を切ってみろ。" | lindera tokenize \
--dict embedded://unidic-csj
The embedded dictionary is built from the same files, so the output is the same as above.
NOTE: To include the UniDic CSJ dictionary in the binary, build with the --features=embed-unidic-csj option. The binary then carries the 1.2 GB dictionary.
Rust API example
use lindera::dictionary::load_dictionary; use lindera::mode::Mode; use lindera::segmenter::Segmenter; use lindera::analysis::tokenizer::Tokenizer; use lindera::LinderaResult; fn main() -> LinderaResult<()> { let dictionary = load_dictionary("embedded://unidic-csj")?; let segmenter = Segmenter::new(Mode::Normal, dictionary, None); let tokenizer = Tokenizer::new(segmenter); let text = "そんなら君の指を切ってみろ。"; let mut tokens = tokenizer.tokenize(text)?; for token in tokens.iter_mut() { let details = token.details().join(","); println!("{}\t{}", token.surface.as_ref(), details); } Ok(()) }