introduce stage dimension to spec variables

[motatoes]

I just noticed this part of the code:

digger/backend/services/spec.go

Line 104 in 55068d3

justNames := lo.Map(variablesSpec, func(item spec.VariableSpec, i int) string {

	variablesSpec := make([]spec.VariableSpec, 0)
	stateVariables := getVariablesSpecFromEnvMap(jobSpec.StateEnvVars)
	commandVariables := getVariablesSpecFromEnvMap(jobSpec.CommandEnvVars)
	runVariables := getVariablesSpecFromEnvMap(jobSpec.RunEnvVars)
	variablesSpec = append(variablesSpec, stateVariables...)
	variablesSpec = append(variablesSpec, commandVariables...)
	variablesSpec = append(variablesSpec, runVariables...)

	// check for duplicates in list of variablesSpec
	justNames := lo.Map(variablesSpec, func(item spec.VariableSpec, i int) string {
		return item.Name
	})
	hasDuplicates := len(justNames) != len(lo.Uniq(justNames))
	if hasDuplicates {
		return nil, fmt.Errorf("could not load variables due to duplicates")
	}

and it seems that we combine all of state, commands and run variables into a single list and then send them to a cli. Furthermore we are checking for duplicates accross all of the variables which is incorrect. For context digger.yml has variables defined like this:

workflows:
  staging:
    env_vars:
      state:
        - name: TF_LOG
          value: trace
      commands:
workflows:
  staging:
    env_vars:
      state:
        - name: TF_LOG
          value: trace
      commands:
        - name: AWS_ACCESS_KEY_ID
          value_from: STAGING_TF_ACCESS_KEY_ID 
      run:
        - name: MY_VAR
          value_from: VAL_ENV

What we are doing in that code block is combining all three types into one list and checking for duplicates which is incorrect. What we are also doing on the cli is we are exposing each variable in all three stages. What we should do instead:

• check for duplication only in state individually, command individually, run individually
• add an atribute to spec variable "stage" which can be state, commands, run
• ensure that in the cli the variables are parsed to their correct stages

In the future we might have a key called "all" for variables that are exposed to all stages

[octocamocoder47]

@motatoes I am working on this issue. could you elaborate me how currently it is working? I mean could you provide me example for which it works perfectly when there is no staging in yml. It will help me to work better on this issue.

[motatoes]

Hi @octocamocoder47 so for example if we had a workflow like this:

workflows:
  staging:
    env_vars:
      state:
        - name: TF_LOG
          value: trace
      commands:
workflows:
  staging:
    env_vars:
      state:
        - name: AWS_ACCESS_KEY_ID
          value: STAGING_STATE_ACCESS_KEY_ID
      commands:
        - name: AWS_ACCESS_KEY_ID
          value_from: STAGING_TF_ACCESS_KEY_ID 

It would fail saying that there are duplicate AWS_ACCESS_KEY_ID variable defined. What we want is that in this case is that it should not fail with this error and instead it should check them correctly. We need to also ensure that "state" is exposed only during the terraform init phase and the "commands" is only exposed during the "terraform plan" or "terraform apply" phase. The "run" stage should be exposed for any custom commands