Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pushing image docker-registry.default.svc:5000/... failed #2

Open
BrunoVernay opened this issue Sep 2, 2017 · 5 comments
Open

Pushing image docker-registry.default.svc:5000/... failed #2

BrunoVernay opened this issue Sep 2, 2017 · 5 comments

Comments

@BrunoVernay
Copy link
Owner

BrunoVernay commented Sep 2, 2017 •
edited
Loading

I have a clean install, systemctl show no apparent errors, but when trying to build my first app it fails:

Writing lock file
Generating optimized autoload files
Pushing image docker-registry.default.svc:5000/picture-uploader/cam-cli:latest ...
Warning: Push failed, retrying in 5s ...
...

Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: [email protected]
Registry server Password: <<non-empty>>
error: build error: Failed to push image: After retrying 6 times, Push image still failed

And I may have done something really wrong because

  oc logs dc/docker-registry 
  Error from server: deploymentconfigs.apps.openshift.io "docker-registry" not found
@BrunoVernay
Copy link
Owner Author

Some extracts from the ansible.log

TASK [openshift_hosted : Create OpenShift registry] ***************************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "results": {
        "results": [
            {
                "cmd": "/bin/oc create -f /tmp/deploymentconfigIjkLmy -n default", 
                "results": {}, 
                "returncode": 0
            }
        ], 
        "returncode": 0
    }, 
    "state": "present"
}

TASK [openshift_hosted : Ensure OpenShift registry correctly rolls out (best-effort today)] ***********************************************************************************************************************
changed: [192.168.56.10] => {
    "ansible_job_id": "477320030950.33739", 
    "changed": true, 
    "cmd": [
        "oc", 
        "rollout", 
        "status", 
        "deploymentconfig", 
        "docker-registry", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig"
    ], 
    "delta": "0:03:41.817930", 
    "end": "2017-09-02 21:19:24.829289", 
    "failed": false, 
    "failed_when_result": false, 
    "finished": 1, 
    "rc": 0, 
    "start": "2017-09-02 21:15:43.011359"
}

STDOUT:

Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 of 1 updated replicas are available...
Waiting for latest deployment config spec to be observed by the controller loop...
replication controller "docker-registry-1" successfully rolled out


TASK [openshift_hosted : Determine the latest version of the OpenShift registry deployment] ***********************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "cmd": [
        "oc", 
        "get", 
        "deploymentconfig", 
        "docker-registry", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig", 
        "-o", 
        "jsonpath={ .status.latestVersion }"
    ], 
    "delta": "0:00:00.256147", 
    "end": "2017-09-02 21:19:31.771675", 
    "rc": 0, 
    "start": "2017-09-02 21:19:31.515528"
}

STDOUT:

1


TASK [openshift_hosted : Sanity-check that the OpenShift registry rolled out correctly] ***************************************************************************************************************************
changed: [192.168.56.10] => {
    "attempts": 1, 
    "changed": true, 
    "cmd": [
        "oc", 
        "get", 
        "replicationcontroller", 
        "docker-registry-1", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig", 
        "-o", 
        "jsonpath={ .metadata.annotations.openshift\\.io/deployment\\.phase }"
    ], 
    "delta": "0:00:00.262986", 
    "end": "2017-09-02 21:19:32.262025", 
    "failed": false, 
    "failed_when_result": false, 
    "rc": 0, 
    "start": "2017-09-02 21:19:31.999039"
}

STDOUT:

Complete


TASK [openshift_hosted : Get registry DeploymentConfig] ***********************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}

TASK [openshift_hosted : Wait for registry pods] ******************************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}

TASK [openshift_hosted : Determine registry fsGroup] **************************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}


TASK [cockpit-ui : fetch the docker-registry route] ***************************************************************************************************************************************************************
ok: [192.168.56.10] => {
    "changed": false, 
    "results": [
        {
            "apiVersion": "v1", 
            "kind": "Route", 
            "metadata": {
                "creationTimestamp": "2017-09-02T19:15:34Z", 
                "name": "docker-registry", 
                "namespace": "default", 
                "resourceVersion": "1371", 
                "selfLink": "/oapi/v1/namespaces/default/routes/docker-registry", 
                "uid": "181d770c-9013-11e7-ad41-080027491194"
            }, 
            "spec": {
                "host": "docker-registry-default.ocp-2.192.168.56.12.nip.io", 
                "tls": {
                    "termination": "passthrough"
                }, 
                "to": {
                    "kind": "Service", 
                    "name": "docker-registry", 
                    "weight": 100
                }, 
                "wildcardPolicy": "None"
            }, 
            "status": {
                "ingress": [
                    {
                        "conditions": [
                            {
                                "lastTransitionTime": "2017-09-02T19:15:34Z", 
                                "status": "True", 
                                "type": "Admitted"
                            }
                        ], 
                        "host": "docker-registry-default.ocp-2.192.168.56.12.nip.io", 
                        "routerName": "router", 
                        "wildcardPolicy": "None"
                    }
                ]
            }
        }
    ], 
    "state": "list"
}

TASK [cockpit-ui : Create passthrough route for registry-console] *************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "results": {
        "cmd": "/bin/oc get route registry-console -o json -n default", 
        "results": [
            {
                "apiVersion": "v1", 
                "kind": "Route", 
                "metadata": {
                    "annotations": {
                        "openshift.io/host.generated": "true"
                    }, 
                    "creationTimestamp": "2017-09-02T19:19:36Z", 
                    "name": "registry-console", 
                    "namespace": "default", 
                    "resourceVersion": "1492", 
                    "selfLink": "/oapi/v1/namespaces/default/routes/registry-console", 
                    "uid": "a8607999-9013-11e7-ad41-080027491194"
                }, 
                "spec": {
                    "host": "registry-console-default.ocp-2.192.168.56.12.nip.io", 
                    "tls": {
                        "termination": "passthrough"
                    }, 
                    "to": {
                        "kind": "Service", 
                        "name": "registry-console", 
                        "weight": 100
                    }, 
                    "wildcardPolicy": "None"
                }, 
                "status": {
                    "ingress": [
                        {
                            "conditions": [
                                {
                                    "lastTransitionTime": "2017-09-02T19:19:36Z", 
                                    "status": "True", 
                                    "type": "Admitted"
                                }
                            ], 
                            "host": "registry-console-default.ocp-2.192.168.56.12.nip.io", 
                            "routerName": "router", 
                            "wildcardPolicy": "None"
                        }
                    ]
                }
            }
        ], 
        "returncode": 0
    }, 
    "state": "present"
}

....................................

TASK [cockpit-ui : Deploy registry-console] ***********************************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "cmd": [
        "oc", 
        "new-app", 
        "--template=registry-console", 
        "-p", 
        "OPENSHIFT_OAUTH_PROVIDER_URL=https://ocp-1.192.168.56.10.nip.io:8443", 
        "-p", 
        "REGISTRY_HOST=docker-registry-default.ocp-2.192.168.56.12.nip.io", 
        "-p", 
        "COCKPIT_KUBE_URL=https://registry-console-default.ocp-2.192.168.56.12.nip.io", 
        "--config=/tmp/openshift-ansible-f8Z6wJ/admin.kubeconfig", 
        "-n", 
        "default"
    ], 
    "delta": "0:00:00.365694", 
    "end": "2017-09-02 21:19:37.538520", 
    "failed": false, 
    "failed_when_result": false, 
    "rc": 0, 
    "start": "2017-09-02 21:19:37.172826"
}

STDOUT:

--> Deploying template "openshift/registry-console" to project default

     registry-console
     ---------
     Template for deploying registry web console. Requires cluster-admin.

     * With parameters:
        * IMAGE_NAME=cockpit/kubernetes
        * IMAGE_VERSION=latest
        * OPENSHIFT_OAUTH_PROVIDER_URL=https://ocp-1.192.168.56.10.nip.io:8443
        * COCKPIT_KUBE_URL=https://registry-console-default.ocp-2.192.168.56.12.nip.io
        * OPENSHIFT_OAUTH_CLIENT_SECRET=user6RlOqTUCr2e8rKyxmvJvVlj5Tn1L0i3hTHA1YxJb0ibCHNdM0b0NSJaJo6LUs3eB # generated
        * OPENSHIFT_OAUTH_CLIENT_ID=cockpit-oauth-client
        * REGISTRY_HOST=docker-registry-default.ocp-2.192.168.56.12.nip.io

--> Creating resources ...
    deploymentconfig "registry-console" created
    service "registry-console" created
    imagestream "registry-console" created
    oauthclient "cockpit-oauth-client" created
--> Success
    Run 'oc status' to view your app.

@BrunoVernay
Copy link
Owner Author

BrunoVernay commented Sep 2, 2017 •
edited
Loading

/tmp/Ansible.Log only contains very limited info ! I should have redirected the installation output to a file !!!
ansible.olg.txt

@BrunoVernay
Copy link
Owner Author

BrunoVernay commented Sep 3, 2017 •
edited
Loading

Looking at /etc/origin/master/admin.kubeconfig there is nothing about registry, nothing on port 5000.

/etc/containers/registries.d/default.yaml looks like a default file, never edited.

default-docker:
  sigstore-staging: file:///var/lib/atomic/sigstore

Also systemctl status docker-registry ... Unit docker-registry.service could not be found.

I will try to install 'Docker-Distribution' (it is the new name for docker-registry). The Ansible script features lots of docker-registry, but no 'docker-distribution' .

@BrunoVernay
Copy link
Owner Author

BrunoVernay commented Sep 3, 2017 •
edited
Loading

ansiop.log.txt
New test with docker-distribution installed, enabled and started, before installing openshift:
The install went fine (complete verbose log joined)
Building the app still fails, same message.

> grep -r -i registry /etc/origin/
/etc/origin/master/policy.json:                "name": "system:registry",
/etc/origin/master/policy.json:                "name": "registry-admin",
/etc/origin/master/policy.json:                "name": "registry-editor",
/etc/origin/master/policy.json:                "name": "registry-viewer",

The file /etc/sysconfig/origin-master contains OPENSHIFT_DEFAULT_REGISTRY=docker-registry.default.svc:5000

The URL is OK and accessible from the master (to itself):

curl -v http://docker-registry.default.svc:5000/
* About to connect() to docker-registry.default.svc port 5000 (#0)
*   Trying 192.168.56.10...
* Connected to docker-registry.default.svc (192.168.56.10) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc:5000
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Sun, 03 Sep 2017 18:53:17 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host docker-registry.default.svc left intact

But on the node, the connexion is refused:

curl -v http://docker-registry.default.svc:5000/
* About to connect() to docker-registry.default.svc port 5000 (#0)
*   Trying 192.168.56.10...
* Connected to docker-registry.default.svc (192.168.56.10) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc:5000
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Sun, 03 Sep 2017 18:53:17 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host docker-registry.default.svc left intact

@BrunoVernay
Copy link
Owner Author 

TASK [openshift_hosted : create the default registry service] *****************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true,
    "results": {
        "clusterip": "172.30.208.115",
        "cmd": "/bin/oc get service docker-registry -o json -n default",
        "results": [
            {
                "apiVersion": "v1",
                "kind": "Service",
                "metadata": {
                    "creationTimestamp": "2017-09-03T17:22:36Z",
                    "name": "docker-registry",
                    "namespace": "default",
                    "resourceVersion": "1289",
                    "selfLink": "/api/v1/namespaces/default/services/docker-registry",
                    "uid": "7aacc3d2-90cc-11e7-af4a-080027491194"
                },
                "spec": {
                    "clusterIP": "172.30.208.115",
                    "ports": [
                        {
                            "name": "5000-tcp",
                            "port": 5000,
                            "protocol": "TCP",
                            "targetPort": 5000
                        }
                    ],
                    "selector": {
                        "docker-registry": "default"
                    },
                    "sessionAffinity": "ClientIP",
                    "type": "ClusterIP"
                },
                "status": {
                    "loadBalancer": {}
                }
            }
        ],
        "returncode": 0
    },
    "state": "present"
}

TASK [openshift_hosted : include] *********************************************************************************************************************************************************************************
included: /usr/share/ansible/openshift-ansible/roles/openshift_hosted/tasks/registry/secure.yml for 192.168.56.10

TASK [openshift_hosted : Set fact docker_registry_route_hostname] *************************************************************************************************************************************************
ok: [192.168.56.10] => {
    "ansible_facts": {
        "docker_registry_route_hostname": "docker-registry-default.ocp-2.192.168.56.12.nip.io"
    },
    "changed": false

Seems the registry is created!
There are also certificates and routes created
Then

TASK [openshift_hosted : Sanity-check that the OpenShift registry rolled out correctly] ***************************************************************************************************************************
changed: [192.168.56.10] => {
    "attempts": 1,
    "changed": true,
    "cmd": [
        "oc",
        "get",
        "replicationcontroller",
        "docker-registry-1",
        "--namespace",
        "default",
        "--config",
        "/etc/origin/master/admin.kubeconfig",
        "-o",
        "jsonpath={ .metadata.annotations.openshift\\.io/deployment\\.phase }"
    ],
    "delta": "0:00:00.279206",
    "end": "2017-09-03 19:25:05.051730",
    "failed": false,
    "failed_when_result": false,
    "rc": 0,
    "start": "2017-09-03 19:25:04.772524"
}

STDOUT:

Complete

But it skip others

TASK [openshift_hosted : Get registry DeploymentConfig] ***********************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@BrunoVernay