Skip to content

@hplt-project HPLT - High Performance Language Technologies

Change the repository type filter

All

    Repositories list

    21 repositories

    • warc2text-runner

      Public
      Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.
      HTML
      •0•3•5•0•Updated Feb 7, 2025Feb 7, 2025

    • monotextor-slurm

      Public
      Set of scripts to run monotextor-like pipeline under slurm HPCs
      Rust
      •
      GNU General Public License v3.0
      •0•2•0•0•Updated Feb 7, 2025Feb 7, 2025

    • data-analytics-tool

      Public
      Data Analytics Tool
      JavaScript
      •
      GNU General Public License v3.0
      •1•13•0•1•Updated Feb 6, 2025Feb 6, 2025

    • HPLT-WP4

      Public
      Information and pipelines on WP4: language models training
      Python
      •
      Creative Commons Zero v1.0 Universal
      •3•2•0•0•Updated Feb 6, 2025Feb 6, 2025

    • release2_inspection

      Public
      Jupyter Notebook
      •7•1•1•0•Updated Feb 1, 2025Feb 1, 2025

    • ia-download

      Public
      Internet archive downloader
      Jupyter Notebook
      •0•2•2•0•Updated Jan 26, 2025Jan 26, 2025

    • cc-download

      Public
      Shell
      •0•0•0•0•Updated Jan 26, 2025Jan 26, 2025

    • bitextor-slurm

      Public
      Scripts for running bitextor jobs
      Shell
      •1•0•1•0•Updated Jan 20, 2025Jan 20, 2025

    • OpusPocus

      Public
      Marian machine translation training pipeline for thousands of models
      Python
      •0•2•19•1•Updated Jan 17, 2025Jan 17, 2025

    • OpusCleaner

      Public
      OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.
      machine-translationdata-cleaning
      Python
      •14•49•56•1•Updated Jan 16, 2025Jan 16, 2025

    • OpusTrainer

      Public
      Curriculum training
      machine-translationtraining-pipeline
      Python
      •
      MIT License
      •6•16•19•0•Updated Jan 13, 2025Jan 13, 2025

    • bitextor-mt-models

      Public
      Shell
      •0•1•3•0•Updated Dec 19, 2024Dec 19, 2024

    • monolingual-multilingual-instruction-tuning

      Public
      Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
      large-language-modelsinstruction-tuninginstruction-data
      Python
      •1•9•0•0•Updated Nov 2, 2024Nov 2, 2024

    • sacremoses

      Public
      Python port of Moses tokenizer, truecaser and normalizer
      nlptokenizermachine-translation
      Python
      •
      MIT License
      •59•490•27•4•Updated May 26, 2024May 26, 2024

    • document-aligner

      Public
      tf/idf-based document aligner from Bitextor
      C++
      •
      Apache License 2.0
      •0•0•0•1•Updated Mar 19, 2024Mar 19, 2024

    • OPUS-MT-dashboard

      Public
      PHP
      •
      MIT License
      •1•0•0•0•Updated Mar 9, 2024Mar 9, 2024

    • HPLT-MT-Models

      Public
      This contains the configuration and scripts for HPLT MT model releases.
      Python
      •0•4•1•0•Updated Mar 6, 2024Mar 6, 2024

    • OpusFilter

      Public
      OpusFilter - Parallel corpus processing toolkit
      Python
      •
      MIT License
      •20•0•0•0•Updated Jan 3, 2024Jan 3, 2024

    • clianer

      Public
      A lightweight command-line frontend to OpusCleaner
      Python
      •
      MIT License
      •1•0•0•0•Updated Nov 27, 2023Nov 27, 2023

    • paracrawl-dashboard

      Public
      Make-shift interface for managing Paracrawl processing and exploring its outputs
      HTML
      •1•0•0•0•Updated Oct 10, 2023Oct 10, 2023

    • MT-winterschool-2023

      Public
      0•1•0•0•Updated Feb 7, 2023Feb 7, 2023