Python library for fast fuzzy search over a big file written in Rust

Table of Contents

• Table of Contents
• About The Project
  • Built With
  • Performance
  • Installation
• Usage
• License
• Contact

About The Project

Fastzy is a library written in Rust that can search through a file looking for text based on its distance (Levenshtein). For measuring the Levenshtein distance, the library uses mbleven's algorithm. In situations where the requested distance exceeds 3, where mbleven is slower, Wagner-Fischer is used instead of mbleven. This library loads the whole file into memory, and creates a lightweight index based on the length of the lines. The result is that only potential lines are looked up, opposed to a large number of lines.

Built With

• mbleven
• Pyo3

Performance

Library	Function	Time
polyleven	polyleven.levenshtein('text')	8.48s
fastzy	fastzy.search('text)	0.003s

Installation

pip3 install fastzy

Usage

import fastzy

# open a file and index it in memory
searcher = fastzy.Searcher(
    file_path='input_text_file.txt',
    separator='',
)

# search for the input text 'text' with the distance of 1
searcher.search(
    pattern='text',
    max_distance=1,
)
['test', 'texts', 'next']

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Gal Ben David - [email protected]

Project Link: https://github.com/Intsights/fastzy