Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: ConnectionRefusedError: [Errno 111] Connection refused via docker-compose on Github Actions #558

Open
stv8 opened this issue Apr 28, 2024 · 5 comments
Open

Bug: ConnectionRefusedError: [Errno 111] Connection refused via docker-compose on Github Actions #558

stv8 opened this issue Apr 28, 2024 · 5 comments

Comments

@stv8
Copy link

stv8 commented Apr 28, 2024

Describe the bug

This seems potentially related to #517 and #475

TLDR; I am able to connect to the PG testcontainer from within docker running locally on a macbook, but running in github actions I get a connection error.

I'm experiencing a ConnectionRefusedError: [Errno 111] Connection refused error when attempting connecting to the postgres testcontainer when running my tests via docker-compose on a Github action. However I am able to run the tests within docker-compose locally on my macbook.

I followed the instructions here for running testcontainers from within docker

https://golang.testcontainers.org/system_requirements/ci/gitlab_ci/#example-using-docker-socket

This is the error I see in Github Actions

        for _ in range(50):
            try:
                Reaper._socket = socket()
>               Reaper._socket.connect((container_host, container_port))
E               ConnectionRefusedError: [Errno 111] Connection refused
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:228: ConnectionRefusedError

Below is the full stack trace

Github Actions Stack Trace 
________________________ ERROR at setup of test_search _________________________
    @pytest.fixture(scope="session")
    def db_url():
        print("YO")
        print(set([a[4][0] for a in socket.getaddrinfo(socket.gethostname(), None)]))
>       with PostgresContainer("postgres:13.1") as postgres:
tests/conftest.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:111: in __enter__
    return self.start()
/venv/lib/python3.11/site-packages/testcontainers/core/generic.py:70: in start
    super().start()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:87: in start
    Reaper.get_instance()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:188: in get_instance
    Reaper._instance = Reaper._create_instance()
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:242: in _create_instance
    raise last_connection_exception
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cls = <class 'testcontainers.core.container.Reaper'>
    @classmethod
    def _create_instance(cls) -> "Reaper":
        logger.debug(f"Creating new Reaper for session: {SESSION_ID}")
    
        Reaper._container = (
            DockerContainer(c.ryuk_image)
            .with_name(f"testcontainers-ryuk-{SESSION_ID}")
            .with_exposed_ports(8080)
            .with_volume_mapping(c.ryuk_docker_socket, "/var/run/docker.sock", "rw")
            .with_kwargs(privileged=c.ryuk_privileged, auto_remove=True)
            .with_env("RYUK_RECONNECTION_TIMEOUT", c.ryuk_reconnection_timeout)
            .start()
        )
        wait_for_logs(Reaper._container, r".* Started!")
    
        container_host = Reaper._container.get_container_host_ip()
        container_port = int(Reaper._container.get_exposed_port(8080))
    
        last_connection_exception: Optional[Exception] = None
        for _ in range(50):
            try:
                Reaper._socket = socket()
>               Reaper._socket.connect((container_host, container_port))
E               ConnectionRefusedError: [Errno 111] Connection refused
/venv/lib/python3.11/site-packages/testcontainers/core/container.py:228: ConnectionRefusedError

The following is my test setup:

conftest.py

@pytest.fixture(scope="session")
def db_url():
    with PostgresContainer("postgres:13.1") as postgres:
        url = postgres.get_connection_url()
        yield url

The following is the docker command I'm using to run the tests via make test. I've also tried multiple variations of the volume mapping to with no luck.

test:
	docker-compose run --rm \
	 -v ${PWD}:${PWD} -w ${PWD}/server \
	 -v /var/run/docker.sock:/var/run/docker.sock \
	 --entrypoint "" server poetry run pytest -vv

where the server image is in my docker-compose.yml is

  server:
    container_name: svs_server
    platform: linux/amd64
    build: ./server
    image: "svs_server:local"
    ports:
      - "8000:8000"
    volumes:
      - "./server:/app"
      - "./db:/app/db"
      - "./tmp:/tmp"
    env_file:
      - .env
      - .env.secret

I am able to successfully get the testcontainers/docker-compose setup running locally on my macbook (docker desktop) with the following, notice the only difference is the additional host override and lack of mapped volumes.

test-local:
	docker-compose run --rm \
	 -v /var/run/docker.sock:/var/run/docker.sock \
	 -e TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal \
	 --entrypoint "" server poetry run pytest -vv

Runtime environment

I also tried finding more info about the network via running the following within the action.

print(set([a[4][0] for a in socket.getaddrinfo(socket.gethostname(), None)]))

This returns an IP such as {'172.18.0.2'} but it is slightly different each time.

  • Running within Github Actions
  • Python 3.11 via the FROM python:3.11-slim-buster image
  • docker info
Client: Docker Engine - Community Version: 24.0.9 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.14.0 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.23.3 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 14 Server Version: 24.0.9 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: e377cd56a71523140ca6ae87e30244719194a521 runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.5.0-1018-azure Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 7.744GiB Name: fv-az822-516 ID: 584d62f5-11d9-4074-87e4-acacf60905a6 Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

  • docker-compose version
    docker-compose version 1.29.2, build 5becea4c

  • uname -a
    Linux fv-az822-516 6.5.0-1018-azure #19~22.04.2-Ubuntu SMP Thu Mar 21 16:45:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Let me know if there's any other additional debugging info I can provide that would help, thanks!!
@alexanderankin
Copy link
Member

alexanderankin commented Apr 28, 2024 via email

@stv8
Copy link
Author

stv8 commented Apr 29, 2024

@alexanderankin I just tried this and my build still failed with the same connection refused error

test:
	docker-compose run --rm \
	 -v ${PWD}:${PWD} -w ${PWD}/server \
	 -e TESTCONTAINERS_HOST_OVERRIDE=localhost \
	 -v /var/run/docker.sock:/var/run/docker.sock \
	 --entrypoint "" server poetry run pytest -vv

Is there any other debugging information on the github action runner that would be good to figure out?

@alexanderankin
Copy link
Member

I'm sorry, I skimmed your email and misunderstood it. Again, i dont really understand this one because i don't use dind. if you just replace this docker-compose command with "poetry run pytest", you too can live in ignorance/bliss.

this is the comment i was looking for before. #475 (comment) - it is how it used to work in v3 but then we shipped #388 and this broke dind and ever since im not really sure what to tell people ¯_(ツ)_/¯

@alexanderankin
Copy link
Member

alexanderankin commented Apr 29, 2024 •
edited
Loading

im not sure what other debugging information would help since im pretty sure some combination of what people have already left in comments does work, just have to figure out how to fix the library. contributions welcome, i at least try to make time to review/prioritize PRs

@stv8
Copy link
Author

stv8 commented Apr 30, 2024

just replace this docker-compose command with "poetry run pytest", you too can live in ignorance/bliss.

@alexanderankin this worked out spectacularly. As I'm no longer trying to do docker-in-docker in GH actions, feel free to do as you wish with this issue.

Thanks again for the feedback, this library is awesome by the way.

Cheers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stv8 @alexanderankin