Folder Structure in this repo?

[wayliums]

This is the repo we are using to add built-in assets. First thing to discuss is the folder structure:

Here are a few options:

• Option 1 - fixed folder structure.
  • teams/<team_name>/components/<asset_name>
  • teams/<team_name>/environments/<asset_name>
• Option 2 - publishConfig file,
  • teams/<team_name>/anyFolderStructure
  • teams/<team_name>/.publishConfig
• Option 3 - restrict on the build output
  • teams/<team_name>/anyFolderStructure
  • during the build process, use python tools to generate an output structure like Option 1

Of course, we could do a combination too, by default suggest Option 1 as best practice, but accept Option 3 too.

Option 1 is easy to understand and normally by default preferred. But it makes component code structure a bit weird, especially if multiple components are sharing code. SDK 1.5 have additional_includes. But DPV2 doesn't have the support, and intellisense doesn't work either.

src/components/name/spec.yaml
src/components/name/spec.additional_includes

Some repo for reference

• https://github.com/microsoft/lightgbm-benchmark
• https://github.com/Azure/RAI-vNext-Preview

[dkmiller]

Some initial thoughts:

• Team names change, how about we use "product" or "area" instead? E.g., I'd rather see lightgbm-benchmarks than AML DS.
• Ideally we'll have a central build + release, because that's where security and other governance checks are enforced.
• Shared code should be shared the "DPv2 way" (Teams thread), i.e. in a shared environment.

My vote: something like this:

.azuredevops/
    azure-pipelines.yml
    ...

docs/
    rfc/
        put_your_nice_design_doc_here.md

src/
    environments/
        lightgbm-benchmark/
            cool_shared_environment/
                environment.yaml
                Dockerfile
                requirements.txt

        {area_name}/
            {environment_name}/
                ...

    components/
        aml-build-configuration.yml

        lightgbm-benchmark/
            cool_component/
                component.yaml
        {area_name}/
            {component_name}/
                ...

[wayliums]

@dkmiller I don't quite understand the conclusion of the team thread. Is it to embed the code into the docker image? Coz that won't work in this case since the images are updated to MCR.

[wayliums]

@dkmiller at this point, we don't allow that, because we need to release to Airgap cloud, which doesn't have Pypi and have restricted network access. Getting from MCR is the easiest approach. And Airgap is required for GA.

Later on if they figure out pypi, then we could see if we can open up.

[dkmiller]

Hmm... sounds like we'll need to implement an "additional includes-style" functionality :/ . It could be a good feature request for Shrike (additional includes for DPv2) cc @fuhuifang

[wayliums]

@dkmiller how does this repo use shrike though? I thought shrike is purely internal but this repo is gonna be public? And solution architects will also look at this repo for best practices on code structure etc. Registry is going to be critical for cross workspace MLOps scenario.

Another thought is to extract just some the part of Shrike, put it in this repo as a tooling, and of course could publish from here too.

[fuhuifang]

Hmm... sounds like we'll need to implement an "additional includes-style" functionality :/ . It could be a good feature request for Shrike (additional includes for DPv2) cc @fuhuifang

Opened a new shrike issue for tracking: https://github.com/Azure/shrike/issues/339

[wayliums]

Cool. @fuhuifang what would be a rough timeline?

[wayliums]

To finalize the discussion, here's what we are gonna do.
- teams/area/src/components/xxx

Long Term - Shrike+additional_includes

• Folder Structure
  • /area/src/common/xxx
  • /area/src/components/component1/component_spec.yaml
  • /area/src/components/component1/run.py
  • /area/src/components/component1/component_spec.additional_includes
  • Exmple: https://github.com/Azure/shrike/tree/main/examples/components/count_rows_and_log
• To share common lib code, have a way to integrate with Shrike. Tracked here: https://github.com/Azure/shrike/issues/339
• Shrike could cover both local testing, unit test and final registration

Short Term - before shrike

• If there's no common code needed
  • /area/src/components/component1/component_spec.yaml
  • /area/src/components/component1/run.py
• If there's common code
  • /area/src/common/xxx
  • /area/src/common/component1_run.py
  • /area/src/components/component1/component_spec.yaml <- snapshotting the base

[dkmiller]

@wayliums , sounds good!

[jfomhover]

In that new schema that I like, what would be the relationship between components and their assets?

would we have /area/environments/ as well?

[jfomhover]

should each area take care of their own unit tests?

[dkmiller]

My vote would be: each area take care of their own unit tests.

[wayliums]

yes, each area has its own GHA workflow, both for unit test and e2e test too. GHA workflow triggered by its corresponding folders. We don't need a generic big workflow.

@jfomhover, yes, /area/environments/ too, but initially there would be /general/environments/ that EMS team maintain

[jfomhover]

Is there a place in this repository for pipelines that would consume from these components? I see two kind of pipelines that could relevant here:

• canary pipelines for testing the components before deployment
• benchmark pipelines (maybe same as canary) for testing their performance on a couple key scenarios.

I can see pros/cons of adding them in the assets repo:

• pros:
  • the pipelines using the assets are close to the assets themselves, and can evolve when them whenever version is upgraded
  • before pr merge, we can run a validation of the components in our github workflows / devops builds
• cons:
  • it will add more and more content to the repo
  • having pipelines will require us to have datasets as well?

[wayliums]

I do want these pipelines to be reused both for local testing, e2e test cases and production synthesized runners. We need to figure out a way to swap ARMID for different scenarios.

Here's the flow for dev phase
Development Phase: Dev workspace and Dev registry (Internal Only)

• local testing, submit job run to validation workspace.
• (Internal Only) They could also publish to dev registry for testing.
• Write e2e test cases
• Send PR, e2e test cases automatically trigger
• Recommend all of the above automation are using anon workspace asset, no registry in place yet.