Dictionary Management

Lindera PHP provides static methods on the Lindera\Dictionary class for loading, building, and managing dictionaries used in morphological analysis.

Loading Dictionaries

System Dictionaries

Use Lindera\Dictionary::load($uri) to load a system dictionary. Download a pre-built dictionary from GitHub Releases and specify the path to the extracted directory:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');

Embedded dictionaries (advanced) -- if you built with an embed-* feature flag, you can load an embedded dictionary:

<?php

$dictionary = Lindera\Dictionary::load('embedded://ipadic');

User Dictionaries

User dictionaries add custom vocabulary on top of a system dictionary. Load one for the system dictionary it is used with:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$userDict = Lindera\Dictionary::loadUserFor('/path/to/user_dictionary.csv', $dictionary);

A .csv is built with the system dictionary's metadata and checked against its connection matrix as it is built: a row whose context ID lies outside the matrix throws a ValueError that names its CSV line and surface (see Context IDs). A .bin is loaded as it was built.

Lindera\Dictionary::loadUser($uri, $metadata) loads a user dictionary with metadata only. A .csv loaded this way is checked when it is combined with the system dictionary, and the error names the entry by its word id:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$metadata = $dictionary->metadata();
$userDict = Lindera\Dictionary::loadUser('/path/to/user_dictionary.csv', $metadata);

Pass the user dictionary when creating a tokenizer directly:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
$userDict = Lindera\Dictionary::loadUserFor('/path/to/user_dictionary.csv', $dictionary);

$tokenizer = new Lindera\Tokenizer($dictionary, Lindera\Mode::normal(), $userDict);

Or via the builder:

<?php

$builder = new Lindera\TokenizerBuilder();
$builder->setDictionary('/path/to/ipadic');
$builder->setUserDictionary('/path/to/user_dictionary.csv');
$tokenizer = $builder->build();

Building Dictionaries

System Dictionary

Build a system dictionary from source files:

<?php

$metadata = Lindera\Metadata::fromJsonFile('/path/to/metadata.json');
Lindera\Dictionary::build('/path/to/input_dir', '/path/to/output_dir', $metadata);

The input directory should contain the dictionary source files (CSV lexicon, matrix.def, etc.).

Metadata loaded with Lindera\Metadata::fromJsonFile keeps the space_penalty rules of the metadata.json, and the built dictionary ships them. For ko-dic this means the left-space penalty stays on by default in the dictionary you build; turn it off per tokenizer with TokenizerBuilder::setSpacePenalty(false) (see Tokenizer API). Metadata created with the constructor carries no rules.

Here is an example that downloads and builds the IPADIC dictionary:

<?php

$url = 'https://lindera.dev/mecab-ipadic-2.7.0-20070801.tar.gz';
$filename = '/tmp/mecab-ipadic-2.7.0-20070801.tar.gz';

// Download and extract dictionary source
file_put_contents($filename, file_get_contents($url));
$phar = new PharData($filename);
$phar->extractTo('/tmp/', null, true);

// Load metadata and build
$metadata = Lindera\Metadata::fromJsonFile('resources/ipadic_metadata.json');
Lindera\Dictionary::build(
    '/tmp/mecab-ipadic-2.7.0-20070801',
    '/tmp/lindera-ipadic',
    $metadata
);

User Dictionary

Build a user dictionary from a CSV file for the system dictionary it is used with:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
Lindera\Dictionary::buildUserFor('user_words.csv', '/path/to/user_dictionary.bin', $dictionary);
$userDict = Lindera\Dictionary::loadUserFor('/path/to/user_dictionary.bin', $dictionary);

The CSV is built with the system dictionary's metadata and checked against its connection matrix: a row whose context ID lies outside the matrix throws a ValueError that names its CSV line and surface, and no file is written (see Context IDs). The second argument is the path of the file to write. End it with .bin, since the loaders choose how to load a user dictionary by its extension.

Lindera\Dictionary::buildUser($kind, $input_file, $output_dir, $metadata) builds a user dictionary with metadata only. Despite its name, $output_dir is the path of the file to write too, not a directory. The rows are checked when the user dictionary is combined with the system dictionary, and the error names the entry by its word id:

<?php

$metadata = new Lindera\Metadata();
Lindera\Dictionary::buildUser('ipadic', 'user_words.csv', '/path/to/user_dictionary.bin', $metadata);

The first argument ($kind, 'ipadic' above) is currently unused and reserved for future use. To build for a system dictionary, use Lindera\Dictionary::buildUserFor.

Metadata

The Lindera\Metadata class configures dictionary parameters.

Creating Metadata

<?php

// Default metadata
$metadata = new Lindera\Metadata();

// Custom metadata
$metadata = new Lindera\Metadata(
    name: 'my_dictionary',
    encoding: 'UTF-8',
    default_word_cost: -10000,
);

// Create with all defaults explicitly
$metadata = Lindera\Metadata::createDefault();

Loading from JSON

<?php

$metadata = Lindera\Metadata::fromJsonFile('metadata.json');

Properties

All properties are read-only via getter methods:

<?php

$metadata = new Lindera\Metadata(name: 'custom_dict', encoding: 'EUC-JP');
echo $metadata->name;      // "custom_dict"
echo $metadata->encoding;  // "EUC-JP"

toArray()

Returns an associative array representation of the metadata:

<?php

$metadata = new Lindera\Metadata(name: 'test');
print_r($metadata->toArray());

When the metadata carries left-space penalty rules (for example, loaded from ko-dic's metadata.json), the array also has a space_penalty entry holding them as a JSON string; the key is absent otherwise:

<?php

$metadata = Lindera\Metadata::fromJsonFile('/path/to/ko-dic/metadata.json');
$rules = json_decode($metadata->toArray()['space_penalty'], true);
// ['rules' => [['pos' => ['EC', 'EF', ...], 'cost' => 3000], ...]]

Dictionary Info

The Lindera\Dictionary object provides metadata accessors:

<?php

$dictionary = Lindera\Dictionary::load('/path/to/ipadic');
echo $dictionary->metadataName();      // Dictionary name
echo $dictionary->metadataEncoding();  // Dictionary encoding
$metadata = $dictionary->metadata();   // Full Metadata object

Version

Retrieve the Lindera library version:

<?php

echo Lindera\Dictionary::version();

Schema

The Lindera\Schema class defines the field layout of a dictionary.

Creating a Schema

<?php

// Default schema (IPADIC-compatible)
$schema = Lindera\Schema::createDefault();

// Custom schema
$schema = new Lindera\Schema(['surface', 'pos']);

Schema Methods

Schema Properties