Revert removal of onefuzz repro get_files... command (#3541)

* Release 8.7.1 (hotfix) (#3459)

* Remove the retention policy setting (#3452)

---------

Co-authored-by: Cheick Keita <[email protected]>

* Revert "Release 8.7.1 (hotfix) (#3459)" (#3468)

This reverts commit c69deed.

* Redo 8.7.1 (#3469)

* Redo-8.7.1-hotfix

---------

Co-authored-by: Cheick Keita <[email protected]>

* Support custom ado fields that mark work items as duplicate (#3467)

* Add field to ado config for checking duplicate work items

* Make duplicate fields nullable and add it to python models

* Update broken tests

* Update docs to include new ado_duplicate_fields property

* Update readme with archive message (#3408)

Co-authored-by: Adam <[email protected]>

* Bump tokio from 1.30.0 to 1.32.0 in /src/proxy-manager (#3425)

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tokio from 1.30.0 to 1.32.0 in /src/agent (#3424)

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Remove unnecessary method argument (#3473)

* Bump elsa from 1.8.1 to 1.9.0 in /src/agent (#3411)

Bumps [elsa](https://github.com/manishearth/elsa) from 1.8.1 to 1.9.0.
- [Commits](Manishearth/elsa@v1.8.1...v1.9.0)

---
updated-dependencies:
- dependency-name: elsa
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tempfile from 3.7.1 to 3.8.0 in /src/agent (#3437)

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](Stebalien/tempfile@v3.7.1...v3.8.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump tempfile from 3.7.1 to 3.8.0 in /src/proxy-manager (#3436)

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](Stebalien/tempfile@v3.7.1...v3.8.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updating requirements.txt to accept >= onefuzztypes. (#3477)

* Updating requirements.txt to accept >= onefuzztypes.

* Trying to loosen restriction.

* Bump notify from 6.0.1 to 6.1.1 in /src/agent (#3435)

Bumps [notify](https://github.com/notify-rs/notify) from 6.0.1 to 6.1.1.
- [Release notes](https://github.com/notify-rs/notify/releases)
- [Changelog](https://github.com/notify-rs/notify/blob/main/CHANGELOG.md)
- [Commits](notify-rs/notify@notify-6.0.1...notify-6.1.1)

---
updated-dependencies:
- dependency-name: notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump azure_* crates (#3478)

* Release 8.8.0 (#3466)

* Release 8.8.0

* Bump clap from 4.3.21 to 4.4.2 in /src/agent (#3484)

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@v4.3.21...v4.4.2)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump gimli from 0.27.3 to 0.28.0 in /src/agent (#3414)

Bumps [gimli](https://github.com/gimli-rs/gimli) from 0.27.3 to 0.28.0.
- [Changelog](https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md)
- [Commits](gimli-rs/gimli@0.27.3...0.28.0)

---
updated-dependencies:
- dependency-name: gimli
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump clap from 4.3.21 to 4.4.2 in /src/proxy-manager (#3474)

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@v4.3.21...v4.4.2)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump winreg from 0.50.0 to 0.51.0 in /src/agent (#3434)

Bumps [winreg](https://github.com/gentoo90/winreg-rs) from 0.50.0 to 0.51.0.
- [Release notes](https://github.com/gentoo90/winreg-rs/releases)
- [Changelog](https://github.com/gentoo90/winreg-rs/blob/master/CHANGELOG.md)
- [Commits](gentoo90/winreg-rs@v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: winreg
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adam <[email protected]>

* Starting integration tests (#3438)

* Starting integration tests

* Ready to test the test

* Parametrize test

* checkpoint

* Test works

* Run integration tests in pipeline

* fmt

* .

* -p

* Install clang

* quotes not required in yaml?

* Hopefully fixed windows?

* Try without killondrop

* lint

* small test

* another test

* Reuse core name

* Wrong step

* bump tokio?

* Try with rust

* make build happy

* Bump pete and small clean up

* Clean up and make the test pass regularly

* fix broken ci

* Lower the poll timeout

* Set the timeout in a nicer way

* fix windows

* fmt

* Include and copy pdbs

* Ignore if pdb is missing on linux

* It takes too long for coverage to be generated

* lint

* Only warn on missing coverage since it's flaky

* Fix windows build

* Small clean up

* Try lowering the poll delay

* fix coverage

* PR comments

* .

* Apparently make is missing?

* Remove aggressive step skipping in CI

* Fix sed checks for CLI versioning (#3486)

* Fix sed checks for CLI versioning

* Fix.

* Fix.

* Changing build_cli

* Trying greater than

* Tring once more.

* Trying major minor

* trying to replace major minor

* Using major minor

* Bump bytes from 1.4.0 to 1.5.0 in /src/agent (#3488)

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](tokio-rs/bytes@v1.4.0...v1.5.0)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Improve area/iteration path validation (#3489)

* Add more comprehensive checks and better error messages to area/iteration path validation

* Join invalid chars with space instead of comma

* Make tree path validation more testable

* Add error code for invalid ADO project in config

* Write unit tests for tree path validation

* Format tree path unit tests

* Merge escape character and control character checks and clarify error message

* Improve handling of unexpected breakpoints (#3493)

* Improve handling of unexpected breakpoints

* fmt

* Update azure_* crates (#3503)

* Fuzz coverage recording (#3322)

* Fuzz coverage recording

* Update cargo.toml

* Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs

Co-authored-by: George Pollard <[email protected]>

* Fix fuzz

---------

Co-authored-by: George Pollard <[email protected]>

* Reporting coverage on task start up (#3502)

* Reporting coverage on task start up

* Moving metric up.

* Remove feature flag from heartbeat metrics. (#3505)

* Update archive notice. (#3507)

* Add onefuzz service version to job created events (#3504)

* Tevoinea/add version checking in local tasks (#3517)

* Compare task version to service version

* Swallow output when looking for appropriate name

* Create directories if they don't exist in the template (#3522)

* Create directories if they don't exist in the template

* fmt

* Support for retention policies on containers (#3501)

- [x] ability to specify a retention period on a container, which applies to newly-created blobs
- [x] specify default retention periods in templates from CLI side 

There's a small breaking change to the Python JobHelper class.

* Bump rayon from 1.7.0 to 1.8.0 in /src/agent (#3520)

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.7.0 to 1.8.0.
- [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md)
- [Commits](rayon-rs/rayon@rayon-core-v1.7.0...rayon-core-v1.8.0)

---
updated-dependencies:
- dependency-name: rayon
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump insta from 1.31.0 to 1.32.0 in /src/agent (#3521)

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.31.0 to 1.32.0.
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md)
- [Commits](mitsuhiko/insta@1.31.0...1.32.0)

---
updated-dependencies:
- dependency-name: insta
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Disable `repro` and `debug` VM CLI commands. (#3494)

* Disable  and  VM CLI commands.

* Formatting.

* More formatting.

* More formatting.

* Removing Repro check.

* Make modules case insenstive on windows (#3527)

* Make modules and coverage allowlist case insensitive on Windows

* Tests and fmt

* PR comments

* fmt

* Debugging missing file coverage

* fmt

* Broken linux test

* Add a case insensitive transformer for better perf

* cargo fix

* Update windows interceptor list (#3528)

* Template creation command (#3531)

* Tasks are selectable

* Almost there

* It works

* fmt

* remove dead code

* Remove unnecessary comments

* Improve instructions

* fix bug

* Add some dummy values for paths

* Terminate process on timeout in windows for the coverage task (#3529)

* Terminate process on timeout in windows for the coverage task

* set the timeout before we start the debugger

* split the target launch from the debugger initialization
wait for the process to finish on a separate thread

* fix build

* move comments

* Ignore regression update when the work item is in some states (#3532)

* Ignore regression update when the work item is in some states

* format

* formatting

* don't hide messages in the poison queue

* fix typo

* update regression logic
update test_template to support regression

* build fix

* mypy fix

* build fix

* move regression ignore state under ADODuplicateTemplate

* replace extend with append

* update set_tcp_keepalive

* mke mypy happy

* copy ADODuplicateTemplate.OnDuplicate.RegressionIgnoreStates

* Revert removal of  command.

* Another update.

* adding back reference.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Adam <[email protected]>
Co-authored-by: Cheick Keita <[email protected]>
Co-authored-by: Kanan B <[email protected]>
Co-authored-by: Marc Greisen <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: George Pollard <[email protected]>
Co-authored-by: Teo Voinea <[email protected]>
Co-authored-by: George Pollard <[email protected]>

src/cli/onefuzz/api.py

@@ -528,6 +528,81 @@ def _download_tasks(
             azcopy_sync(to_download[name], outdir)
 
 
+class Repro(Endpoint):
+    """Interact with repro files"""
+
+    endpoint = "repro_vms"
+
+    def get_files(
+        self,
+        report_container: primitives.Container,
+        report_name: str,
+        include_setup: bool = False,
+        output_dir: primitives.Directory = primitives.Directory("."),
+    ) -> None:
+        """downloads the files necessary to locally repro the crash from given report"""
+        report_bytes = self.onefuzz.containers.files.get(report_container, report_name)
+        report = json.loads(report_bytes)
+
+        crash_info = {
+            "input_blob_container": primitives.Container(""),
+            "input_blob_name": "",
+            "job_id": "",
+        }
+        if "input_blob" in report:
+            crash_info["input_blob_container"] = report["input_blob"]["container"]
+            crash_info["input_blob_name"] = report["input_blob"]["name"]
+            crash_info["job_id"] = report["job_id"]
+        elif "crash_test_result" in report and "original_crash_test_result" in report:
+            if report["original_crash_test_result"]["crash_report"] is None:
+                self.logger.error(
+                    "No crash report found in the original crash test result, repro files cannot be retrieved"
+                )
+                return
+            elif report["crash_test_result"]["crash_report"] is None:
+                self.logger.info(
+                    "No crash report found in the new crash test result, falling back on the original crash test result for job_id"
+                    "Note: if using --include_setup, the downloaded fuzzer binaries may be out-of-date"
+                )
+
+            original_report = report["original_crash_test_result"]["crash_report"]
+            new_report = (
+                report["crash_test_result"]["crash_report"] or original_report
+            )  # fallback on original_report
+
+            crash_info["input_blob_container"] = original_report["input_blob"][
+                "container"
+            ]
+            crash_info["input_blob_name"] = original_report["input_blob"]["name"]
+            crash_info["job_id"] = new_report["job_id"]
+        else:
+            self.logger.error(
+                "Encountered an unhandled report format, repro files cannot be retrieved"
+            )
+            return
+
+        self.logger.info(
+            "downloading files necessary to locally repro crash %s",
+            crash_info["input_blob_name"],
+        )
+        self.onefuzz.containers.files.download(
+            primitives.Container(crash_info["input_blob_container"]),
+            crash_info["input_blob_name"],
+            os.path.join(output_dir, crash_info["input_blob_name"]),
+        )
+
+        if include_setup:
+            setup_container = list(
+                self.onefuzz.jobs.containers.list(
+                    crash_info["job_id"], enums.ContainerType.setup
+                )
+            )[0]
+
+            self.onefuzz.containers.files.download_dir(
+                primitives.Container(setup_container), output_dir
+            )
+
+
 class Notifications(Endpoint):
     """Interact with models.Notifications"""
 
@@ -1588,6 +1663,7 @@ def __init__(
             client_secret=client_secret,
         )
         self.containers = Containers(self)
+        self.repro = Repro(self)
         self.notifications = Notifications(self)
         self.tasks = Tasks(self)
         self.jobs = Jobs(self)