v2024.3.0 MinKNOW 6.0 compatibilty, pass reads and bug fixes

What's new

The highlights:

• Updated to use MinKNOW >= 6.0 🥳 There were some minknow_api endpoint changes, around the basecaller version. Big one is the switch from using read_id to read_number, and new strand classifications.

Warning

Breaking for any version of MinKNOW <= 6.0.0
As of readfish >=2024.3.0 we no longer support guppy.

• Uses py_basecall_client.pass_reads to send reads to the base caller for improved performance.
• Bug fixes, most are for the flongle, due to the final column of pores being 6 channels not 10, like the preceding columns.
• Chores - updated to use Node 20 actions which was surprisingly easy over all

What's Changed

• Feature/minknow-6-compatibilyt by @Adoni5 in #368
• Feature/pass reads by @Adoni5 in #361
• Delete dc918797-33da-4414-9999-eb9132d22971_readfish.tsv by @Adoni5 in #380
• Catch some "placeholder" channel positions being 0 on flongle by @Adoni5 in #384
• Fix issue340 - live TOML handling by @Adoni5 in #383
• bump all node 16 actions to node 20 by @Adoni5 in #377
• Minor change to readme to better reflect update to 6.0 and deprecatio… by @mattloose in https://github.com/LooseLab/readfish/pull/

Full Changelog: 2024.2.0...v2024.3.0

🙏🏼 for Rory's thesis

v2024.2.0 - Dorado >= 7.3.9 support

What's new

Dorado 7.3.9 support

As discussed in #347, due to the move to ont_pybasecall_client_lib by ONT with the release of Dorado 7.3.9, readfish was unable to connect to the basecall socket, as it was using ont_pyguppy_client_lib.

This release supports both connections to dorado 7.3.9 and upwards with the dorado plugin, (which also handles the new parameter sample_rate on package_reads smartly) and dorado < 7.3.9 with the guppy plugin.

To use the new plugin, simply switch the caller_settings table in your toml from .guppy to .dorado, leaving other parameters the same, like so:

[caller_settings.dorado]

MinKNOW version checking

Readfish also now lists a tested MinKNOW version range, and will raise a warning if the MinKNOW version being connected to is outside the range for the given version of readfish.
However, readfish will still attempt to run adaptive sampling, and will most likely succeed!
As recommended in the logged warning, if a newer version of readfish doesn't exist, we recommend opening an issue on the github.

Warning

The Guppy plugin will likely be deprecated in the next major release of readfish.

What's Changed

• Dorado plugin by @mattloose and @Adoni5 in #344
• Minknow version checking by @mattloose and @Adoni5 #351

Full Changelog: 2024.1.0...2024.2.0

v2024.1.0 - Duplex implementation and bug fixes

What's new

• Implements some basic Experimental Duplex based adaptive sampling.
  Currently there are two modes, "Duplex simple" and "duplex", which are accessed under the --chemistry flag under the targets sub command. They both take targets in the same way that the normal target commands do. We currently do not support mixing simplex and duplex targets.
  duplex_simple accepts a read if the previous read seen on the channel was stop receiving and the new read isn't, duplex checks that the previous reads alignment was on the same contig and opposite strand.
  NB. Only duplex_simple has been tested.
• This pull request refactors the code that the check for reads to be over-ridden, and also adds some new reasons for read decisions:

    #: Potential second half of a duplex read
    duplex_override = "duplex_override"
    #: Read sequenced as translocated portion was of unknown length at start of readfish
    first_read_override = "first_read_override"

Bug fixes

• Don't throw an error if there are multiple suffixes on the provided .mmi file
• --wait-on-ready actually works, and is an int, so User can specify to wait for the Sequencing state for as long as they want (default now 120s)
• Fixes links in documentation to bulk files and adds a promethion 4KhZ R10 bulk file
• --unblock-duration type to float, and manually specifying it was crashing argparsing

What's Changed - Associated PRs

• Issue309 by @mattloose in #310 - adds documentation and promethion 4KhZ R10 bulk file
• Closes issue #312 by @mattloose in #313 - unblock duration type to float
• Fix changing directory after 60 seconds if no run folder by @Adoni5 in #323
• Change type on --wait-for-ready to int by @Adoni5 in #327
• Adding additional .mmi extensions to all possible fasta file format l… by @mattloose in #330
• Feature/duplex by @Adoni5 in #324

Full Changelog: 2023.1.1...2024.1.0

2023.1.1 Bug fix

A small release, with a bug fix.

For major new features introduced in refactored readfish, please see https://github.com/LooseLab/readfish/releases/tag/2023.1.0.

Bug fixes

• Fixes a bug caught by @jamesemery in #289, where regions were required to be present in a TOML file, even when the intent was to use barcoding only.

Other changes

• Adds a Changelog, found in the docs and at the bottom of the README.
• Minor documentation fixes to broken links and problems installing grpcio on Apple Silicon.

2023.1.0 Stable refactored readfish for all platforms

Major Changes

1. This release is a complete overhaul of the inner workings of readfish. Fundamentally nothing has changed in the way readfish works, but we have dramatically improved the compatibility with nanopore platforms, as well as supporting more compute systems including Apple Silicon.

2. This release is compatible with all flavours of PromethION from the lonely p2solo to the gregarious P48. To do this we include a new rust based wrapper for minimap2 which enables multi-threaded mapping. The release also supports GridION and MinION. It is untested on mk1c and we don't recommend using it.

3. This release is compatible with both guppy and dorado via the ont-pyguppy-client-lib and so introduces native support for apple silicon.

4. We also introduce a new readfish stats module which will summarise an experiment given the TOML configuration for readfish and the path to the FASTQs from the run.

5. The code includes all new documentation - see https://looselab.github.io/readfish - and will prevent users from making many common mistakes.

Note

Important: The TOML file specification has been modified for this version and users should update their TOML files accordingly.

This includes a description of the new plugin framework, developed by @alexomics , which will enable users to develop their own custom mappers, squiggle analysis packages and other tools and then integrate them into a common readfish environment.

In principle readfish can be used as a library by other packages.

Full information on new features can be found in the readme and associated documentation. All issues should be reported by the GitHub issue tracker and not by email to the authors.

What's Changed

• Add unblock all script by @alexomics
• Add github worklflow action to close stale issues by @Adoni5
• Move logger initialisation into check that we have arguments by @Adoni5
• Add pre commit and make all files compliant by @Adoni5
• validation script by @alexomics
• Switching behaviour of validate by @mattloose
• Add readfish identifier to appear in guppy server logs by @mattloose
• Feature/describe plugin by @Adoni5
• Remove --chunk-log option by @alexomics
• Implement runsafe and additional logging. by @mattloose
• Implementing check on basecalling model in guppy plugin. This will va… by @mattloose
• Fixes to describe by @Adoni5
• Update _config.py to check barcodes if regions and barcodes are present. by @Adoni5
• Enabling changes to the log for the readuntil client by tools which u… by @mattloose
• Add readfish[mappy,guppy] as default install target by @Adoni5
• Split mappy plugins by @alexomics
• Feature/summarise-entry-point by @Adoni5
• Fix-stats-macos-aarch-test by @Adoni5
• Adding a check for the current break_reads_seconds value and testing … by @mattloose
• Feature/logging-stats-padding by @Adoni5

No take backsies (for real)

BossRuns/V0.0.2

This version of ReadFish contains the updated BossRuns code in response to reviewer comments. ReadFish BossRuns should now be compatible with minknow-core <=5.1.4 and >= 5.0.0 and Guppy >= 6.0.0. The command to run readfish bossruns remains the same, however the guppy address and port are now most likely ipc:///tmp/guppy and 5555.

BossRuns Publication

This release ran with MinKNOW 21.11.7 and MinKNOW Core 4.5.4 and ONT-Guppy 5.1.13.

Nature Biotech paper code archive

This release archives the code used in the Nature Biotech readfish paper.

Important

The README contains instructions on running this set of code, however many of the tools at the given versions, such as ont-pyguppy-client-lib will no longer work with modern MinKNOW. We recommend you use the most up to date readfish, found here https://github.com/LooseLab/readfish/.