Using environment variables in .scuba.yml or passing them

[mpdude]

Is it already possible – or do you consider it a valuable addition – to use environment variables in the .scuba.yml file?

For one, these could be already present in the shell where scuba is run. Additionally, I think some variables added by scuba might be helpful. (Is there a list of already used variables available somewhere, like the directory where scuba was started, path and directory to .scubafile.yml etc.)?

The goal would be do define an alias like build: ./do-build.sh ${SCUBA_HOME} ${SOME_ENV}.

With regard to passing environment variables into the container, what about a new configuration setting, either global or per-alias, that lists env vars to be passed?

image: ...
aliases:
    build: 
        command: ./do-build.sh ${TARGET}
        env:
            - THIS_VAR
            - THAT_VAR
        
env:
    - TARGET_ARCH
    - COMPILE_MODE

Note that we'd probably need to change the alias syntax somewhat or allow another structure to place the env key.

This would add -e THIS_VAR -e THAT_VAR to the docker run command when using the build alias, and add -e TARGET_ARCH -e COMPILE_MODE in any case.

Does that make sense?

[JonathonReinhart]

Hi @mpdude. Thanks for the well thought-out report! You're right, currently Scuba doesn't do much with environment variables, but I think it would certainly be a valuable addition. I started to bump into this issue when I noted #85.

Scuba does support passing environment variables to Docker, but it's not yet exposed to the user. Currently add_env() is only used for passing SCUBAINIT_ variables to scubainit.

---

To reply to a couple of your points:

For one, these could be already present in the shell where scuba is run.

AFAIK, Docker doesn't pass anything from the shell into a container; you get a pretty "clean" environment, and I'm sure this is by design. There isn't any way for software to automatically "know" which env vars to pass through and which ones to filter out. That's not to say that scuba couldn't be told which environment variables are to be set to a specific value, and which ones are to be "passed-through" from the shell. (More on this below.)

---

Additionally, I think some variables added by scuba might be helpful. (Is there a list of already used variables available somewhere, like the directory where scuba was started, path and directory to .scubafile.yml etc.)?

There is no documented list, but perhaps that should change. scubainit (which is run behind the scenes at startup) does two things with env vars:

• process_envvars() processes the SCUBAINIT_ vars passed from scuba. It will only unset variables.
• change_user() will set USER, LOGNAME and HOME.

---

Note that we'd probably need to change the alias syntax somewhat or allow another structure to place the env key.

Actually, I don't think it would have to change. I've realized there is a gap in the aliases documentation. See, aliases (as well as hooks) also use the common script schema, but it's not mentioned in the documentation. Oops!

In the examples directory, there's alias_multiline/.scuba.yml which shows a so-called "multi-line alias". So I think env or variables can simply be a sibling to script, like this:

image: debian:8
aliases:
  simple: echo 'a simple alias'

  complex:
    script:
      - echo "This is a complex script"
      - echo "We can use variables like \$FOO=$FOO"
    env:
      FOO: 42
      BAR: $BAR    # pull from env. during scuba invocation

I think what I've shown here will also work for pulling environment variables during scuba invocation. $FOO is automatically expanded by the shell in the script, but we'll have to expand $BAR ourselves when loading the .scuba.yml config file. I think this is clean and straightforward.

[JonathonReinhart]

Also related is #80.

[xanarin]

@jreinhart have you thought about how you'd like the expansion of variables to work during scuba invocation? In your example above you use BAR: $BAR. Do you only want to support that syntax of expansion, or also support the additional types of parameter expansion supported by bash? It looks like other Python projects have run into this issue and have decided on varying solutions:

• Proper Variable Expansion anishathalye/dotbot#56
• Interpolate environment variables in docker-compose.yml docker/compose#1377
• https://stackoverflow.com/q/30734967

The main solutions seem to be:

1. Use os.path.expandvars, which works on both POSIX and Windows, but leaves unset environmental variables instead of replacing them with "".
   • Users will not expect this atypical behavior, so implementation of solution 2 might also be required
2. Use a regular expression to parse the expansion and then replace captured values by querying os.environ.
   • Regexes of this complexity are fragile and not friendly to future developers
3. Write a custom parser that supports a subset of bash-supported variable expansion features.
   • Requires more development time than all other solutions
4. Punt the expansion to bash with subprocess.check_output and echo.
   • Allows users to run arbitrary commands in their scubafile by using the $(...) syntax

Solution 4 seems like the most robust solution to me, as long as we can get the escaping correct for the echo invocation.

[xanarin]

In an offline discussion with @JonathonReinhart, it was decided that full parameter expansion wasn't necessary. Instead, it would be sufficient to set environmental variables to static values, or to expand their value from the host environment. The following syntax would then be acceptable in any Scubafile:

env:
  - BAR        # This would be set to it's value in the host environment, or set to the empty string
  - FOO =      # $FOO='' in the container
  - SNAP=abc   # $SNAP='abc' in the container
  - ZOOP=$SNAP # $ZOOP='$SNAP' in the container. Scuba won't expand any variables in the variable definitions

Values that weren't mapped would be resolved from the host's environment, falling back to the empty string if the variable was not defined in the host environment.

The env block could appear at the top level of a Scubafile, or inside of a specific alias. Alias-level definitions would be combined with global definitions, with alias-level definitions overriding in cases of conflict.

[JonathonReinhart]

I think it is easiest to simply follow the method of docker-compose:
https://docs.docker.com/compose/compose-file/#environment

Add environment variables. You can use either an array or a dictionary. Any boolean values; true, false, yes no, need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser.

Environment variables with only a key are resolved to their values on the machine Compose is running on, which can be helpful for secret or host-specific values.

environment:
  RACK_ENV: development
  SHOW: 'true'
  SESSION_SECRET:

environment:
  - RACK_ENV=development
  - SHOW=true
  - SESSION_SECRET