Merge pull request #493 from gatici/Deploy-applications-with-custom-r…

…esources

#493

## Description

This PR provides to use custom (OCI image) charm resources with applications. The application resource could be a charm revision or a custom OCI image which needs to be pulled from a repository. The following "juju_application" resource definition becomes valid within this PR. This PR both works with all the Juju versions which are supported by Juju Terraform provider. (Tested with Juju 2.9.x and 3.4.x versions)

```
resource "juju_application" "ausf" {
 name = "ausf"
 model = var.model_name

 charm {
 name = "sdcore-ausf-k8s"
 channel = var.channel
 }
 resources = {
 ausf-image = "gatici/sdcore-ausf:1.4.0"
 // ausf-image = "2"
 }
 units = 1
 trust = true
}
```

Design Spec: https://docs.google.com/document/d/1i236ntmw-qXyqifmEtk_rcszWXcXSUhXkeJ6ByKXIeE/edit

Fixes: #460

## Type of change

- Change existing resource
- Change in the tests (one or several tests have been changed)
- Requires a documentation update

## Environment

The following steps are performed to prepare the environment.

Install Microk8s:

```
sudo snap install microk8s --channel=1.29-strict/stable
sudo usermod -a -G snap_microk8s jenkins
newgrp snap_microk8s
sudo microk8s enable hostpath-storage
```

Install Juju:

```
sudo snap install juju --channel=3.4/stable
juju bootstrap microk8s
```

Install Terraform:

```
sudo snap install --classic terraform
```

Create `terraformrc` file with following contents. Write the path of `terraform.d/plugins` directory if it is in a different path.

```
touch .terraformrc 
```

```
provider_installation {
 filesystem_mirror {
 path = "~/.terraform.d/plugins"
 include = ["juju/juju"]
 }
 direct {
 exclude = ["juju/juju"]
 }
}
```
 

## QA steps

Manual QA steps should be done to test this PR.

Create a folder to store the Terraform files for testing.

mkdir testing
cd testing

Create a `main.tf` file with following contents:

```
resource "juju_application" "ausf" {
 name = "ausf"
 model = var.model_name

 charm {
 name = "sdcore-ausf-k8s"
 channel = var.channel
 }
 resources = {
 ausf-image = "gatici/sdcore-ausf:1.4.0"
 }
 units = 1
 trust = true
}
```

Create a `terraform.tf ` file with following contents:

```
terraform {
 required_providers {
 juju = {
 source = "juju/juju"
 version = "0.12.0"
 }
 }
}
```

Create a `terraform.tfvars ` file with following contents:

```
model_name ="test40"
```

Create a `variables.tf ` file with following contents:

```
variable "model_name" {
 description = "Name of Juju model to deploy application to."
 type = string
 default = ""
}

variable "channel" {
 description = "The channel to use when deploying a charm."
 type = string
 default = "1.4/edge"
}

variable "db_application_name" {
 description = "The name of the application providing the `database` endpoint."
 type = string
 default = "mongodb-k8s"
}

variable "certs_application_name" {
 description = "Name of the application providing the `certificates` integration endpoint."
 type = string
 default = "self-signed-certificates"
}

variable "nrf_application_name" {
 description = "The name of the application providing the `fiveg_nrf` endpoint."
 type = string
 default = "nrf"
}
```

Fetch the PR and run following commads to install it.
```
make go-install
make install
```

Create a juju model:

```
juju add-model test40
```

Initialise the provider:

```
terraform init
```


Run Terraform plan by providing a var-file:

```
terraform plan -var-file="terraform.tfvars" 
```

Deploy the application:
```
terraform apply -auto-approve 
```

Check the pod to see the attached image.
```
microk8s.kubectl describe pod ausf-0 -n test40
```

## Additional notes

Test with changing the resources part with different image versions. 

Using a revision as string:
```
resources = {
 ausf-image = "30"
}
```

docs/resources/application.md

@@ -25,6 +25,10 @@ resource "juju_application" "this" {
     series   = "trusty"
   }
 
+  resources = {
+    gosherve-image = "gatici/gosherve:1.0"
+  }
+
   units = 3
 
   placement = "0,1,2"
@@ -55,17 +59,14 @@ resource "juju_application" "this" {
 - `expose` (Block List) Makes an application publicly available over the network (see [below for nested schema](#nestedblock--expose))
 - `name` (String) A custom name for the application deployment. If empty, uses the charm's name.
 - `placement` (String) Specify the target location for the application's units
-- `resources` (Map of Number) Charm resource revisions. Must evaluate to an integer.
-
-	There are a few scenarios that need to be considered:
-	* If the plan does not specify resource revision and resources are added to the plan,
-	resources with specified revisions will be attached to the application (equivalent
-	to juju attach-resource).
-	* If the plan does specify resource revisions and:
-		* If the charm revision or channel is updated, then resources get updated to the 
-		  latest revision.
-	    * If the charm revision or channel are not updated, then no changes will take 
-		  place (juju does not have an "un-attach" command for resources).
+- `resources` (Map of String) Charm resources. Must evaluate to a string. A resource could be a resource revision number from CharmHub or a custom OCI image resource.
+Specify a resource other than the default for a charm. Note that not all charms have resources.
+
+Notes:
+* A resource can be specified by a revision number or by URL to a OCI image repository. Resources of type 'file' can only be specified by revision number. Resources of type 'oci-image' can be specified by revision number or URL.
+* A resource can be added or changed at any time. If the charm has resources and None is specified in the plan, Juju will use the resource defined in the charm's specified channel.
+* If a charm is refreshed, by changing the charm revision or channel and if the resource is specified by a revision in the plan, Juju will use the resource defined in the plan.
+* Resources specified by URL to an OCI image repository will never be refreshed (upgraded) by juju during a charm refresh unless explicitly changed in the plan.
 - `storage` (Attributes Set) Storage used by the application. (see [below for nested schema](#nestedatt--storage))
 - `storage_directives` (Map of String) Storage directives (constraints) for the juju application. The map key is the label of the storage defined by the charm, the map value is the storage directive in the form <pool>,<count>,<size>. Changing an existing key/value pair will cause the application to be replaced. Adding a new key/value pair will add storage to the application on upgrade.
 - `trust` (Boolean) Set the trust for the application.

examples/resources/juju_application/resource.tf

@@ -10,6 +10,10 @@ resource "juju_application" "this" {
     series   = "trusty"
   }
 
+  resources = {
+    gosherve-image = "gatici/gosherve:1.0"
+  }
+
   units = 3
 
   placement = "0,1,2"

internal/juju/applications.go

@@ -34,6 +34,7 @@ import (
 	apispaces "github.com/juju/juju/api/client/spaces"
 	apicommoncharm "github.com/juju/juju/api/common/charm"
 	"github.com/juju/juju/cmd/juju/application/utils"
+	resourcecmd "github.com/juju/juju/cmd/juju/resource"
 	corebase "github.com/juju/juju/core/base"
 	"github.com/juju/juju/core/constraints"
 	"github.com/juju/juju/core/instance"
@@ -110,7 +111,7 @@ func newApplicationClient(sc SharedClient) *applicationsClient {
 	}
 }
 
-// ConfigEntry is an auxiliar struct to keep information about
+// ConfigEntry is an auxiliary struct to keep information about
 // juju application config entries. Specially, we want to know
 // if they have the default value.
 type ConfigEntry struct {
@@ -162,14 +163,14 @@ type CreateApplicationInput struct {
 	Placement          string
 	Constraints        constraints.Value
 	EndpointBindings   map[string]string
-	Resources          map[string]int
+	Resources          map[string]string
 	StorageConstraints map[string]jujustorage.Constraints
 }
 
 // validateAndTransform returns transformedCreateApplicationInput which
 // validated and in the proper format for both the new and legacy deployment
 // methods. Select input is not transformed due to differences in the
-// 2 deployement methods, such as config.
+// 2 deployment methods, such as config.
 func (input CreateApplicationInput) validateAndTransform(conn api.Connection) (parsed transformedCreateApplicationInput, err error) {
 	parsed.charmChannel = input.CharmChannel
 	parsed.charmName = input.CharmName
@@ -265,7 +266,7 @@ type transformedCreateApplicationInput struct {
 	units            int
 	trust            bool
 	endpointBindings map[string]string
-	resources        map[string]int
+	resources        map[string]string
 	storage          map[string]jujustorage.Constraints
 }
 
@@ -293,7 +294,7 @@ type ReadApplicationResponse struct {
 	Placement        string
 	EndpointBindings map[string]string
 	Storage          map[string]jujustorage.Constraints
-	Resources        map[string]int
+	Resources        map[string]string
 }
 
 type UpdateApplicationInput struct {
@@ -313,7 +314,7 @@ type UpdateApplicationInput struct {
 	Constraints        *constraints.Value
 	EndpointBindings   map[string]string
 	StorageConstraints map[string]jujustorage.Constraints
-	Resources          map[string]int
+	Resources          map[string]string
 }
 
 type DestroyApplicationInput struct {
@@ -347,8 +348,15 @@ func (c applicationsClient) CreateApplication(ctx context.Context, input *Create
 	}
 
 	applicationAPIClient := apiapplication.NewClient(conn)
+	resourceAPIClient, err := apiresources.NewClient(conn)
+	if err != nil {
+		return nil, err
+	}
 	if applicationAPIClient.BestAPIVersion() >= 19 {
-		err = c.deployFromRepository(applicationAPIClient, transformedInput)
+		err := c.deployFromRepository(applicationAPIClient, resourceAPIClient, transformedInput)
+		if err != nil {
+			return nil, err
+		}
 	} else {
 		err = c.legacyDeploy(ctx, conn, applicationAPIClient, transformedInput)
 		err = jujuerrors.Annotate(err, "legacy deploy method")
@@ -366,20 +374,14 @@ func (c applicationsClient) CreateApplication(ctx context.Context, input *Create
 	}, err
 }
 
-func (c applicationsClient) deployFromRepository(applicationAPIClient *apiapplication.Client, transformedInput transformedCreateApplicationInput) error {
+func (c applicationsClient) deployFromRepository(applicationAPIClient ApplicationAPIClient, resourceAPIClient ResourceAPIClient, transformedInput transformedCreateApplicationInput) error {
 	settingsForYaml := map[interface{}]interface{}{transformedInput.applicationName: transformedInput.config}
 	configYaml, err := goyaml.Marshal(settingsForYaml)
 	if err != nil {
 		return jujuerrors.Trace(err)
 	}
-
-	resources := make(map[string]string)
-	for k, v := range transformedInput.resources {
-		resources[k] = strconv.Itoa(v)
-	}
-
 	c.Tracef("Calling DeployFromRepository")
-	_, _, errs := applicationAPIClient.DeployFromRepository(apiapplication.DeployFromRepositoryArg{
+	deployInfo, localPendingResources, errs := applicationAPIClient.DeployFromRepository(apiapplication.DeployFromRepositoryArg{
 		CharmName:        transformedInput.charmName,
 		ApplicationName:  transformedInput.applicationName,
 		Base:             &transformedInput.charmBase,
@@ -391,14 +393,26 @@ func (c applicationsClient) deployFromRepository(applicationAPIClient *apiapplic
 		Placement:        transformedInput.placement,
 		Revision:         &transformedInput.charmRevision,
 		Trust:            transformedInput.trust,
-		Resources:        resources,
+		Resources:        transformedInput.resources,
 		Storage:          transformedInput.storage,
 	})
-	return errors.Join(errs...)
+
+	if len(errs) != 0 {
+		return errors.Join(errs...)
+	}
+
+	fileSystem := osFilesystem{}
+	// Upload the provided local resources to Juju
+	uploadErr := uploadExistingPendingResources(deployInfo.Name, localPendingResources, fileSystem, resourceAPIClient)
+
+	if uploadErr != nil {
+		return uploadErr
+	}
+	return nil
 }
 
 // TODO (hml) 23-Feb-2024
-// Remove the funcationality associated with legacyDeploy
+// Remove the functionality associated with legacyDeploy
 // once the provider no longer supports a version of juju
 // before 3.3.
 func (c applicationsClient) legacyDeploy(ctx context.Context, conn api.Connection, applicationAPIClient *apiapplication.Client, transformedInput transformedCreateApplicationInput) error {
@@ -669,9 +683,9 @@ func (c applicationsClient) baseToUse(modelconfigAPIClient *apimodelconfig.Clien
 	return supportedBases[0], nil
 }
 
-// processExpose is a local function that executes an expose request.
+// processExpose is a local function that executes an exposed request.
 // If the exposeConfig argument is nil it simply exits. If not,
-// an expose request is done populating the request arguments with
+// an exposed request is done populating the request arguments with
 // the endpoints, spaces, and cidrs contained in the exposeConfig
 // map.
 func (c applicationsClient) processExpose(applicationAPIClient ApplicationAPIClient, applicationName string, expose map[string]interface{}) error {
@@ -732,14 +746,14 @@ func splitCommaDelimitedList(list string) []string {
 
 // processResources is a helper function to process the charm
 // metadata and request the download of any additional resource.
-func (c applicationsClient) processResources(charmsAPIClient *apicharms.Client, conn api.Connection, charmID apiapplication.CharmID, appName string, resources map[string]int) (map[string]string, error) {
+func (c applicationsClient) processResources(charmsAPIClient *apicharms.Client, conn api.Connection, charmID apiapplication.CharmID, appName string, resourcesToUse map[string]string) (map[string]string, error) {
 	charmInfo, err := charmsAPIClient.CharmInfo(charmID.URL)
 	if err != nil {
 		return nil, typedError(err)
 	}
 
 	// check if we have resources to request
-	if len(charmInfo.Meta.Resources) == 0 && len(resources) == 0 {
+	if len(charmInfo.Meta.Resources) == 0 && len(resourcesToUse) == 0 {
 		return nil, nil
 	}
 
@@ -748,7 +762,7 @@ func (c applicationsClient) processResources(charmsAPIClient *apicharms.Client,
 		return nil, err
 	}
 
-	return addPendingResources(appName, charmInfo.Meta.Resources, resources, charmID, resourcesAPIClient)
+	return addPendingResources(appName, charmInfo.Meta.Resources, resourcesToUse, charmID, resourcesAPIClient)
 }
 
 // ReadApplicationWithRetryOnNotFound calls ReadApplication until
@@ -1072,10 +1086,10 @@ func (c applicationsClient) ReadApplication(input *ReadApplicationInput) (*ReadA
 	if err != nil {
 		return nil, jujuerrors.Annotate(err, "failed to list application resources")
 	}
-	resourceRevisions := make(map[string]int)
+	usedResources := make(map[string]string)
 	for _, iResources := range resources {
 		for _, resource := range iResources.Resources {
-			resourceRevisions[resource.Name] = resource.Revision
+			usedResources[resource.Name] = strconv.Itoa(resource.Revision)
 		}
 	}
 
@@ -1094,7 +1108,7 @@ func (c applicationsClient) ReadApplication(input *ReadApplicationInput) (*ReadA
 		Placement:        placement,
 		EndpointBindings: endpointBindings,
 		Storage:          storages,
-		Resources:        resourceRevisions,
+		Resources:        usedResources,
 	}
 
 	return response, nil
@@ -1402,7 +1416,7 @@ func (c applicationsClient) computeSetCharmConfig(
 }
 
 func resolveCharm(charmsAPIClient *apicharms.Client, curl *charm.URL, origin apicommoncharm.Origin) (*charm.URL, apicommoncharm.Origin, []corebase.Base, error) {
-	// Charm or bundle has been supplied as a URL so we resolve and
+	// Charm or bundle has been supplied as a URL, so we resolve and
 	// deploy using the store but pass in the origin command line
 	// argument so users can target a specific origin.
 	resolved, err := charmsAPIClient.ResolveCharms([]apicharms.CharmToResolve{{URL: curl, Origin: origin}})
@@ -1420,27 +1434,18 @@ func strPtr(in string) *string {
 	return &in
 }
 
-func (c applicationsClient) updateResources(appName string, resources map[string]int, charmsAPIClient *apicharms.Client,
+func (c applicationsClient) updateResources(appName string, resources map[string]string, charmsAPIClient *apicharms.Client,
 	charmID apiapplication.CharmID, resourcesAPIClient ResourceAPIClient) (map[string]string, error) {
 	meta, err := utils.GetMetaResources(charmID.URL, charmsAPIClient)
 	if err != nil {
 		return nil, err
 	}
-
-	resourceRevisions := make(map[string]string)
-	for k, v := range resources {
-		resourceRevisions[k] = strconv.Itoa(v)
-	}
-
-	// TODO (cderici): Provided resources for GetUpgradeResources are user inputs.
-	// It's a map[string]string that should come from the plan itself. We currently
-	// don't have a resources block in the charm.
 	filtered, err := utils.GetUpgradeResources(
 		charmID,
 		charmsAPIClient,
 		resourcesAPIClient,
 		appName,
-		resourceRevisions, // nil
+		resources,
 		meta,
 	)
 	if err != nil {
@@ -1453,45 +1458,86 @@ func (c applicationsClient) updateResources(appName string, resources map[string
 	return addPendingResources(appName, filtered, resources, charmID, resourcesAPIClient)
 }
 
-func addPendingResources(appName string, resourcesToBeAdded map[string]charmresources.Meta, resourceRevisions map[string]int,
-	charmID apiapplication.CharmID, resourcesAPIClient ResourceAPIClient) (map[string]string, error) {
-	pendingResources := []charmresources.Resource{}
-	for _, v := range resourcesToBeAdded {
-		aux := charmresources.Resource{
-			Meta:     v,
-			Origin:   charmresources.OriginStore,
-			Revision: -1,
+func addPendingResources(appName string, charmResourcesToAdd map[string]charmresources.Meta, resourcesToUse map[string]string,
+	charmID apiapplication.CharmID, resourceAPIClient ResourceAPIClient) (map[string]string, error) {
+	pendingResourcesforAdd := []charmresources.Resource{}
+	resourceIDs := map[string]string{}
+
+	for _, resourceMeta := range charmResourcesToAdd {
+		if resourcesToUse == nil {
+			// If there are no resource revisions, the Charm is deployed with
+			// default resources according to channel.
+			resourceFromCharmhub := charmresources.Resource{
+				Meta:     resourceMeta,
+				Origin:   charmresources.OriginStore,
+				Revision: -1,
+			}
+			pendingResourcesforAdd = append(pendingResourcesforAdd, resourceFromCharmhub)
+			continue
 		}
-		if resourceRevisions != nil {
-			if revision, ok := resourceRevisions[v.Name]; ok {
-				aux.Revision = revision
+
+		deployValue, ok := resourcesToUse[resourceMeta.Name]
+		if !ok {
+			continue
+		}
+		if providedRev, err := strconv.Atoi(deployValue); err == nil {
+			// A resource revision is provided
+			resourceFromCharmhub := charmresources.Resource{
+				Meta:   resourceMeta,
+				Origin: charmresources.OriginStore,
+				// If the resource is removed, providedRev is -1. Then, Charm
+				// is deployed with default resources according to channel.
+				// Otherwise, Charm is deployed with the provided revision.
+				Revision: providedRev,
 			}
+			pendingResourcesforAdd = append(pendingResourcesforAdd, resourceFromCharmhub)
+			continue
 		}
 
-		pendingResources = append(pendingResources, aux)
+		// A new resource to be uploaded by the ResourceApi client.
+		localResource := charmresources.Resource{
+			Meta:   resourceMeta,
+			Origin: charmresources.OriginUpload,
+		}
+		fileSystem := osFilesystem{}
+		t, typeParseErr := charmresources.ParseType(resourceMeta.Type.String())
+		if typeParseErr != nil {
+			return nil, typedError(typeParseErr)
+		}
+		r, openResErr := resourcecmd.OpenResource(deployValue, t, fileSystem.Open)
+		if openResErr != nil {
+			return nil, typedError(openResErr)
+		}
+		toRequestUpload, err := resourceAPIClient.UploadPendingResource(appName, localResource, deployValue, r)
+		if err != nil {
+			return nil, typedError(err)
+		}
+		// Add the resource name and the corresponding UUID to the resources map.
+		resourceIDs[resourceMeta.Name] = toRequestUpload
 	}
 
-	resourcesReq := apiresources.AddPendingResourcesArgs{
+	if len(pendingResourcesforAdd) == 0 {
+		return resourceIDs, nil
+	}
+
+	resourcesReqforAdd := apiresources.AddPendingResourcesArgs{
 		ApplicationID: appName,
 		CharmID: apiresources.CharmID{
 			URL:    charmID.URL,
 			Origin: charmID.Origin,
 		},
-		Resources: pendingResources,
+		Resources: pendingResourcesforAdd,
 	}
-
-	toRequest, err := resourcesAPIClient.AddPendingResources(resourcesReq)
+	toRequestAdd, err := resourceAPIClient.AddPendingResources(resourcesReqforAdd)
 	if err != nil {
 		return nil, typedError(err)
 	}
-
-	// now build a map with the resource name and the corresponding UUID
-	toReturn := map[string]string{}
-	for i, argsResource := range pendingResources {
-		toReturn[argsResource.Meta.Name] = toRequest[i]
+	// Add the resource name and the corresponding UUID to the resources map
+	for i, argsResource := range pendingResourcesforAdd {
+		resourceIDs[argsResource.Meta.Name] = toRequestAdd[i]
 	}
 
-	return toReturn, nil
+	return resourceIDs, nil
 }
 
 func computeUpdatedBindings(modelDefaultSpace string, currentBindings map[string]string, inputBindings map[string]string, appName string) (params.ApplicationMergeBindingsArgs, error) {