Japanese Word Tokenizer

A lightweight Japanese word tokenizer library for Dart and Flutter based on wakachigaki.

Features

• Tokenizes Japanese text into individual words
• Lightweight and fast
• Easy to use and integrate into Dart and Flutter projects
• Based on the wakachigaki tokenizer algorithm by Yushak Inoue

Installation

Add the following dependency to your pubspec.yaml file:

dependencies:
  japanese_word_tokenizer: ^0.0.3

Sample usage

Import the package in your Dart code:

import 'package:japanese_word_tokenizer/japanese_word_tokenizer.dart';

In your main.dart

void main() {
  String text = 'ここでテキストを分かち書きします';
  List<dynamic> tokens = tokenize(text);
  print(tokens);
}

Sample output

[ここで, テキスト, を, 分か, ち, 書き, します]