Configuration Alignment Request for chrony across AWS and GCP images

[SergeiMalkov]

Abstract

Please consider moving this issue to private repo if it's required!

Our line of work heavily relies on machine images provisioned by this repo across AWS and GCP.
There's a configuration drift that we have to align manually, due to inheritance from Scylla Machine Images.
To be more specific, there's an inconsistency in how chrony is being configured by the image across different cloud providers.
This has an impact on Scylla Cloud's configuration drift (we are consuming VMs images) and this issue's scope are the most recent images with Scylla Enterprise 2024.2.4-0.20250124.2bc4ec25a8db

What Do We Need?

Enforce same approach for chrony's configuration across AWS and GCP machine images.
Sure thing, time sources would be different, but that difference should be handled at the templating layer.

So, either, timesources should be specified directly in /etc/chrony/chrony.conf or under /etc/chrony/sources.d/${uniform_name.sources} and that's what we asking to enforce at image creation level. Any approach will work for us, but we need it to be uniform for all Cloud Providers we use.

Adjusting things in runtime isn't feasible for us; because scylla-server.service requires time to be in sync.
If we want to reconfgiure chrony for Scylla nodes running their workflow, we would have to:

• Wait for approved maintenance interval
• Drain and stop ScyllaDB
• Reconfigure and restart chrony on the server
• Start Scylla and give it some time for caches warm-up
• Repeat for several next nodes

However, during scale-outs or machine replacement there would be new instances created with non-uniform configuration.
With that said, we need to ensure that spinning up new machines won't cause further drift and after that we would be able to enforce uniform configuration across our server fleet.

Chrony Deployment Overview

AWS Deployment

AMI Details - ami-0b2d43d06d0e507d3

ubuntu@maintenance-prod:~$ ansible node -i 31543.json -a 'curl -s http://169.254.169.254/latest/meta-data/ami-id' 
ami-0b2d43d06d0e507d3

Here's an example of chrony's configuration layout on AWS and the inline contents of chrony.conf.
TL;DR - main config /etc/chrony/chrony.conf is just a default config shipped with chrony by OS Vendor (Ubuntu);
sources for time sync are being shipped through /etc/chrony/sources.d/ntp-pool.sources.

ubuntu@maintenance-prod:~$ ansible node -i 31543.json -a 'tree /etc/chrony' -b

/etc/chrony
|-- chrony.conf
|-- chrony.keys
|-- conf.d
|   |-- 00-cpc.conf
|   `-- README
`-- sources.d
    |-- README
    `-- ntp-pool.sources

ubuntu@maintenance-prod:~$ ansible node -i 31543.json -a 'cat /etc/chrony/chrony.conf' -b

# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usable directives.

# Include configuration files found in /etc/chrony/conf.d.
confdir /etc/chrony/conf.d

# This will use (up to):
# - 4 sources from ntp.ubuntu.com which some are ipv6 enabled
# - 2 sources from 2.ubuntu.pool.ntp.org which is ipv6 enabled as well
# - 1 source from [01].ubuntu.pool.ntp.org each (ipv4 only atm)
# This means by default, up to 6 dual-stack and up to 2 additional IPv4-only
# sources will be used.
# At the same time it retains some protection against one of the entries being
# down (compare to just using one of the lines). See (LP: #1754358) for the
# discussion.
#
# About using servers from the NTP Pool Project in general see (LP: #104525).
# Approved by Ubuntu Technical Board on 2011-02-08.
# See http://www.pool.ntp.org/join.html for more information.
# pool ntp.ubuntu.com        iburst maxsources 4
# pool 0.ubuntu.pool.ntp.org iburst maxsources 1
# pool 1.ubuntu.pool.ntp.org iburst maxsources 1
# pool 2.ubuntu.pool.ntp.org iburst maxsources 2

# Use time sources from DHCP.
sourcedir /run/chrony-dhcp

# Use NTP sources found in /etc/chrony/sources.d.
sourcedir /etc/chrony/sources.d

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Uncomment the following line to turn logging on.
#log tracking measurements statistics

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1 3

# Get TAI-UTC offset and leap seconds from the system tz database.
# This directive must be commented out when using time sources serving
# leap-smeared time.
leapsectz right/UTC    

GCP Deployment

GCP Deployment is different, used timesoureces are consumed directly from /etc/chrony/chrony.conf instead and include files are empty.

ubuntu@maintenance-prod:~$ ansible node -i 31675.json -a 'tree /etc/chrony' -b
server-152213 | CHANGED | rc=0 >>
/etc/chrony
|-- chrony.conf
|-- chrony.conf.dist
|-- chrony.keys
|-- conf.d
|   `-- README
`-- sources.d
    `-- README

ubuntu@maintenance-prod:~$ ansible node -i 31675.json -a 'cat /etc/chrony/chrony.conf' -b
server-152213 | CHANGED | rc=0 >>
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usable directives.

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
# servers
server metadata.google.internal iburst

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Uncomment the following line to turn logging on.
#log tracking measurements statistics

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1 3

[SergeiMalkov]

@yaronkaikov , @fruch , @syuu1228 , this issue has to do with Scylla Cloud, please consider transferring it to some private repo if that's required.

cc: @d-helios

[mykaul]

It's unclear to me what the diff is. It should be minimal and related to the source, I reckon.
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configure-ec2-ntp.html :

server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

(with IPv4. With IPv6 it's fd00:ec2::123 )

GCP (from https://cloud.google.com/compute/docs/instances/configure-ntp#linux-chrony ):
Use metadata.google.internal

OK - what else is needed?

(BTW, need to add Azure as well!)

[d-helios]

I believe main point here is that source should be configured either in /etc/chrony/chrony.conf or in /etc/chrony/sources.d/ntp-pool.sources
right now aws and gcp configured in a different way.

I vote for /etc/chrony/sources.d/ntp-pool.sources + adjust configuration in /etc/chrony/chrony.conf as it is done in aws

of course sources should be different.