Evaluate: "Governable" deployment patterns w/ Waltz for GenAI - Cross-Project Collaboration Hypothesis

[karlmoll]

Description

Feature Request

Description of Problem:

I am looking to evaluate the potential for Waltz to data observability and control capabilities addressing threats, risks, & controls described by Governance documents (e.g. AI Readiness Governance Framework ), in conjunction with other FINOS/Open Source projects.

Feedback we have received: "everyone knows what the threats, risks, controls - but not how to implement".

Ideally would augment existing work done by CCC/CFI to map services described in AIR GF Reference Architecture
https://github.com/finos/common-cloud-controls/milestone/6
finos/common-cloud-controls#314

There is also work needed by the AIR SIG to make GF (and regulations) machine readable. There is a general move to describe every part of the deployment as X-as-Code, so in the case of Waltz might fit into a Data-as-Code description for auditability, observability, and automated change deployment.

A similar hypothesis has been addressed to CALM as a feature request to evaluate officially, and has been discussed unofficially with other projects described below:
finos/architecture-as-code#544

Potential Solutions:

This is a rough draft of the vision so far:

• Make every part of the governance framework machine readable (eventually, also do the same to regulations - similar to LCR)
• Map services/architecture of the AIR GF to controls and compliant infrastructure (partially in progress, AIR needs to provide machine readable architecture, etc)
• CALM takes the reference architecture and builds controls, deployment patterns, and metrics/logging/data observability
• Morphir manages governance/regulation/common business logic "as code" so that applications across the tech stack can be updated on changes
  • I believe morphir can also be used to manage to deployment of controls, architecture of code, etc
  • Morphir provides the ability to monitor/visualize business logic rules being executed in production which provides powerful observability capabilities
• Waltz provides data observability for RAG queries, training data, user queries, etc
• Models fine tuning using proposed open source benchmarks from LLM Exploration
  • Fine tuning efforts based on regulatory interpretation
• There might be additional projects that can be looped in which I don't have as much exposure (certainly OpenRegTech, probably Backstage)
• The next step would be adding new tools for Model Governance

Benefits:

• Solves "we know what the threats/risks/controls are but we don't know how to implement" problem heard during workshop
• Allows seamless, updatable, auditable deployment compliant with governance and regulations (even as they change)
• Give industry a way to demonstrate use cases to regulators and other supervisors/governance concerns where they can showcase use cases on certified architecture/infrastructure/business logic so they can get sign-off before deploying in production

Resourcing

We would like to collaborate on this feature