Speed issue with Ansible configured to use AWS SSM instead SSH

[erandu]

Summary

I am trying to configure Ansible to use AWS SSM to connect to my EC2 instances instead SSH.
I have a playbook with a duration of approximatly 12 min when using SSH. With AWS SSM setup, this one takes 24 minutes,
so it takes twice as long which is a problem for me.
The playbook consists of classic tasks: install packages, setup permissions, configure elasticsearch...
I have investigate the root cause of this, it seems that setup SSM connection takes approximatly 3 seconds for each task, whereas it's instantaneous with SSH.

Below an exemple, with an Ansible task, in which setup the connection takes 3 seconds:

I didn't find a lot of documentation on this issue (here: #1853 and here: https://forum.ansible.com/t/how-to-re-use-connection-across-tasks/38171) .

So I am creating this issue to track this (maybe it could also be a feature) and check if it's possible to improve speed or find othe solutions.

Issue Type

Bug Report

Component Name

community.aws.aws_ssm

Ansible Version

# ansible --version
ansible [core 2.16.14]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Aug 26 2024, 07:20:54) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Collection Versions

# ansible-galaxy collection list

# /root/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
StephenSorriaux.ansible_kafka_admin      0.20.0
community.docker                         3.10.2
community.library_inventory_filtering_v1 1.0.2

# /usr/local/lib/python3.11/dist-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               7.0.0
ansible.netcommon                        5.3.0
ansible.posix                            1.5.4
ansible.utils                            2.11.0
ansible.windows                          2.1.0
arista.eos                               6.2.1
awx.awx                                  23.3.1
azure.azcollection                       1.19.0
check_point.mgmt                         5.1.1
chocolatey.chocolatey                    1.5.1
cisco.aci                                2.8.0
cisco.asa                                4.0.3
cisco.dnac                               6.7.6
cisco.intersight                         2.0.3
cisco.ios                                5.2.0
cisco.iosxr                              6.1.0
cisco.ise                                2.5.16
cisco.meraki                             2.16.14
cisco.mso                                2.5.0
cisco.nxos                               5.2.1
cisco.ucs                                1.10.0
cloud.common                             2.1.4
cloudscale_ch.cloud                      2.3.1
community.aws                            7.0.0
community.azure                          2.0.0
community.ciscosmb                       1.0.7
community.crypto                         2.16.0
community.digitalocean                   1.24.0
community.dns                            2.6.3
community.docker                         3.4.11
community.general                        8.0.2
community.grafana                        1.6.1
community.hashi_vault                    6.0.0
community.hrobot                         1.8.2
community.libvirt                        1.3.0
community.mongodb                        1.6.3
community.mysql                          3.8.0
community.network                        5.0.2
community.okd                            2.3.0
community.postgresql                     3.2.0
community.proxysql                       1.5.1
community.rabbitmq                       1.2.3
community.routeros                       2.10.0
community.sap                            2.0.0
community.sap_libs                       1.4.1
community.sops                           1.6.7
community.vmware                         4.0.0
community.windows                        2.0.0
community.zabbix                         2.1.0
containers.podman                        1.11.0
cyberark.conjur                          1.2.2
cyberark.pas                             1.0.23
dellemc.enterprise_sonic                 2.2.0
dellemc.openmanage                       8.4.0
dellemc.powerflex                        2.0.1
dellemc.unity                            1.7.1
f5networks.f5_modules                    1.27.0
fortinet.fortimanager                    2.3.0
fortinet.fortios                         2.3.4
frr.frr                                  2.0.2
gluster.gluster                          1.0.2
google.cloud                             1.2.0
grafana.grafana                          2.2.3
hetzner.hcloud                           2.3.0
hpe.nimble                               1.1.4
ibm.qradar                               2.1.0
ibm.spectrum_virtualize                  2.0.0
ibm.storage_virtualize                   2.1.0
infinidat.infinibox                      1.3.12
infoblox.nios_modules                    1.5.0
inspur.ispim                             2.1.0
inspur.sm                                2.3.0
junipernetworks.junos                    5.3.0
kubernetes.core                          2.4.0
lowlydba.sqlserver                       2.2.2
microsoft.ad                             1.3.0
netapp.aws                               21.7.1
netapp.azure                             21.10.1
netapp.cloudmanager                      21.22.1
netapp.elementsw                         21.7.0
netapp.ontap                             22.8.2
netapp.storagegrid                       21.11.1
netapp.um_info                           21.8.1
netapp_eseries.santricity                1.4.0
netbox.netbox                            3.15.0
ngine_io.cloudstack                      2.3.0
ngine_io.exoscale                        1.1.0
openstack.cloud                          2.1.0
openvswitch.openvswitch                  2.1.1
ovirt.ovirt                              3.2.0
purestorage.flasharray                   1.22.0
purestorage.flashblade                   1.14.0
purestorage.fusion                       1.6.0
sensu.sensu_go                           1.14.0
splunk.es                                2.1.0
t_systems_mms.icinga_director            2.0.1
telekom_mms.icinga_director              1.34.1
theforeman.foreman                       3.14.0
vmware.vmware_rest                       2.3.1
vultr.cloud                              1.10.0
vyos.vyos                                4.1.0
wti.remote                               1.0.5

AWS SDK versions

# pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.11/dist-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.35.76
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.11/dist-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.35.76
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.11/dist-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

# ansible-config dump --only-changed
CONFIG_FILE() = None

OS / Environment

Debian 12, in a Docker container

Steps to Reproduce

Launch a playbook with AWS SSM connection configured.

Expected Results

Same duration of the playbook compared to using SSH connection.

Actual Results

Duration is twice as long in my case compared to using SSH connection.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[markuman]

I've never used it. But per design, using SSM connection feature will be always slower as using ssh. Because it's using S3 with multiple HTTP requests per TASK.
AFAIU, this is an expected behaviour.

[tremble]

I've never used it. But per design, using SSM connection feature will be always slower as using ssh. Because it's using S3 with multiple HTTP requests per TASK. AFAIU, this is an expected behaviour.

Sort of. The setup for both SSH and SSM connections can be painfully slow, however with SSH the tool itself has a mechanism ("ControlMaster") for setting up persistent connections which most people use as a way to boost SSH performance.

If you're a Red Hat Ansible customer, it's worth pushing for this module to get moved over to amazon.aws with official support added, in doing so it would be possible for the Cloud team to try and borrow some expertise from folks who've written the networking connection plugins which use Ansible's persistence support.

[ksubileau]

I think it might be interesting to see how Packer handles this and take inspiration from it. Instead of entirely relying on SSM, it open an SSH tunnel through SSM.
It has the advantage of not requiring an external S3 bucket for small data transfers, as it's still possible to do it via SSH, while still retaining the benefits of SSM : IAM authentication without using SSH keys, no inbound port required, no bastion host, ...

[shadycuz]

The issue I am having is that sometimes it just hangs indefinitely. I have tried setting ansible_aws_ssm_timeout to a low number but that doesn't change its behavior.

It will usually run one or maybe two tasks. Then it will hang indefinitely. I am working remotely on a slow VPN... but I don't have these kinda issues with any other tool like Curl, SSH, and packer.