Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use testing backends with functions notebooks #2240

Closed
3 tasks
frankharkins opened this issue Nov 4, 2024 · 0 comments · Fixed by #2463
Closed
3 tasks

Use testing backends with functions notebooks #2240

frankharkins opened this issue Nov 4, 2024 · 0 comments · Fixed by #2463
Assignees
@frankharkins
Labels
infra 🏗️

Comments

@frankharkins
Copy link
Member

We can test functions that don't post-process results using the IBM test backends. This will submit jobs that should run very quickly and return nonsense results. We'll need to add some more patching code, but it'll hopefully mean we can test these notebooks on PR, which would be great.

Todo:

  • Identify candidate notebooks (that is, any notebooks that don't post-process results)
  • Add feature to patch these notebooks and prevent writing results
  • Add candidate notebooks to PR checks
@frankharkins frankharkins mentioned this issue Nov 4, 2024
Closed
@frankharkins frankharkins self-assigned this Nov 4, 2024
@frankharkins frankharkins linked a pull request Dec 9, 2024 that will close this issue
Test notebooks with test-eagle #2447
Closed
@frankharkins frankharkins mentioned this issue Dec 11, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 16, 2024
@frankharkins
Test notebooks with test-eagle (#2463)
6ba4574 
There are three changes in this PR

1. (2d2fb9c) **Make notebooks runnable
with nb-tester**

This commit modifies some of the notebooks to make them executable from
start to finish. Most common change is defining `instance` and
`backend_name` in a hidden cell and giving example values through a
comment, rather than having `backend_name = '<your-backend>'`.

2. (945b8d7) **Add extended strategy
and reclassify notebooks**

Adds a new "extended" test that runs the notebooks on IBM Quantum's
`test-eagle` device.

This device accepts jobs and returns random bit strings in a very short
time period, which is great for testing notebooks that submit jobs too
large for a local simulator. This also allows us to test functions
notebooks, which take a backend name (string) of an IBM Quantum device,
rather than the backend object itself.

3. (9f3df9e) **Add extended notebook
tests action**

I originally hoped we could test notebooks with test-eagle in CI, but
the jobs can be flaky and take just a little too long to complete.
Running these jobs on every commit is too expensive and could block
writers.

Instead, I've added a new "extended" test that we can trigger manually
on branches when we need it. Examples could be dependabot PRs, or PRs to
functions notebooks; when the PR is ready, we'll trigger the workflow on
the PR branch to check the notebooks run correctly. To trigger the
workflow, go to Actions > Test notebook (extended) > Run workflow, then
select the PR branch and click "Run workflow".


***

Fixes #2240, fixes #2239, and fixes #2198.
@javabster javabster removed this from Docs Planning Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frankharkins frankharkins

Labels
infra 🏗️
Projects
None yet
Milestone
No milestone
3 participants
@Eric-Arellano @javabster @frankharkins