amazon-cloudwatch-agent: init at 1.300049.1

[commiterate]

Description of changes

Initialize amazon-cloudwatch-agent at 1.300049.1.

This is commonly deployed alongside applications run on Amazon Web Services compute (e.g. EC2, Fargate).

Fixes #6848.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[commiterate]

Tried writing the service module and it seems like this might be a little complicated. The following doesn't work:

{ lib, pkgs, config, ... }:
let
  cfg = config.services.amazon-cloudwatch-agent;
in
{
  options.services.amazon-cloudwatch-agent = {
    enable = lib.mkEnableOption "amazon-cloudwatch-agent";
    package = lib.mkPackageOption pkgs "amazon-cloudwatch-agent" { };
    settings = lib.mkOption {
      type = types.submodule {
        freeformType = (pkgs.formats.json { }).type;
      };
      default = { };
      description = ''
        Amazon CloudWatch Agent configuration. See
        <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html>
        for supported values.
      '';
    };
  };

  # See https://github.com/aws/amazon-cloudwatch-agent/blob/v1.300045.0/packaging/dependencies/amazon-cloudwatch-agent.service.
  config = lib.mkIf cfg.enable {
    systemd.services.amazon-cloudwatch-agent = {
      description = "Amazon CloudWatch Agent";
      after = [ "network.target" ];
      wantedBy = [ "multi-user.target" ];
      serviceConfig = {
        Type = "simple";
        # TODO: "start-amazon-cloudwatch-agent" assumes the agent is installed at "/opt/aws/amazon-cloudwatch-agent".
        #
        # It's inconvenient to use "amazon-cloudwatch-agent" directly since "start-amazon-cloudwatch-agent" calls
        # "config-translator" to translate the JSON configuration into separate TOML (for CloudWatch Logs + Metrics)
        # and YAML (for X-Ray traces) configuration files. This doesn't seem to be a 1:1 transformation.
        ExecStart = "${cfg.package}/bin/start-amazon-cloudwatch-agent";
        KillMode = "process";
        Restart = "on-failure";
        RestartSec = "60s";
      };
    };
  };

  environment.etc."amazon-cloudwatch-agent/amazon-cloudwatch-agent.json".text = toJSON cfg.settings;
}

The problem is start-amazon-cloudwatch-agent which hardcodes the expected package installation path as /opt/aws/amazon-cloudwatch-agent. This is used to derive other binary paths and generated output file paths.

https://github.com/aws/amazon-cloudwatch-agent/blob/v1.300045.0/tool/paths/paths_unix.go#L12

start-amazon-cloudwatch-agent is unfortunately not just a simple wrapper program that calls amazon-cloudwatch-agent with pre-defined arguments. It runs config-translator to translate /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json into:

• /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
  • For CloudWatch Logs + Metrics.
• /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml
  • For X-Ray traces.

These don't seem to be 1:1 translations either (there may be some key renames).

We'll probably need to ask upstream to provide the following options:

• Installation directory.
  • Where the package is installed.
  • Maybe they can switch to relative paths for executing config-translator and amazon-cloudwatch-agent instead or get the current directory? Then this won't be needed at all.
• Input JSON configuration file path.
  • Input amazon-cloudwatch-agent.json.
  • This is something NixOS users would control with services.amazon-cloudwatch-agent.settings and we'd write to /etc/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json.
• Runtime output directory.
  • Where to write amazon-cloudwatch-agent.toml, amazon-cloudwatch-agent.yaml, amazon-cloudwatch-agent.env, amazon-cloudwatch-agent.pid, etc.
  • This has to be a mutable directory in NixOS.
    • Probably /run/amazon-cloudwatch-agent. /etc can be immutable if system.etc.overlay.mutable is set to false (docs).

Alternatively, they might want to have amazon-cloudwatch-agent accept the unified JSON configuration directly and work with it in memory instead of writing out intermediary results to other files (though they still want to write out their optional PID file somewhere). I'm not sure why these derived configuration files need to exist.

---

Going to swap this PR to a draft in the meanwhile.

[commiterate]

Opened an issue upstream about start-amazon-cloudwatch-agent.

aws/amazon-cloudwatch-agent#1319

We might be able to get away with using ExecStartPre to call config-translator:

${cfg.package}/bin/config-translator \
  -input /etc/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json \
  -input-dir /etc/amazon-cloudwatch-agent/amazon-cloudwatch-agent.d \
  -mode auto \
  -output /run/amazon-cloudwatch-agent/amazon-cloudwatch-agent.toml

Then we'll make ExecStart do:

${cfg.package}/bin/amazon-cloudwatch-agent \
  -config /run/amazon-cloudwatch-agent/amazon-cloudwatch-agent.toml \
  -envconfig /run/amazon-cloudwatch-agent/env-config.json \
  -otelconfig /run/amazon-cloudwatch-agent/amazon-cloudwatch-agent.yaml \
  -pidfile /run/amazon-cloudwatch-agent/amazon-cloudwatch-agent.pid

Reference: https://github.com/aws/amazon-cloudwatch-agent/blob/v1.300045.0/cmd/start-amazon-cloudwatch-agent/path.go#L67-L79

[commiterate]

Added the system service. Need to figure out how to test this in a VM now.

[commiterate]

We'll stick with the current version in the MR (1.300047.0) then given that 1.300048.1 is mostly feature additions with a minor bug fix.

As for soliciting a review, the only place I know of is the review request mega-thread in Discourse. If there's another avenue for doing so like a Matrix room, please share this PR there as well.

If the reviewer with write permissions asks for changes, I'll also bump the version as well since we'll need to wait for another full OfBorg eval anyways.

[commiterate]

Posted to the Nixpkgs Review Requests channel in Matrix.

[commiterate]

Still no bites. Need to rebase anyways to fix a merge conflict with the release notes. Will bump the version as well.

[commiterate]

Looks like we're stuck waiting for an aarch64-darwin OfBorg runner again.

Going to bump to 1.300049.1 since it has some important enhancements and bug fixes.

Given that there's no aarch64-darwin tests and I've already tested build on an M2 Mac, we should be good to merge once all of the Linux tests pass.

[commiterate]

OfBorg Linux checks passed. Good to merge.