Can't push to staging charmstore

[cmars]

I'm unable to push a charm to staging for testing:

$ charm push ./docker-registry cs:\~yellow/docker-registry
ERROR cannot post archive: cannot put archive blob: failed to PUT object 6c7efa2332159e63-ae6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 4641529 bytes
caused by: EOF

[wallyworld]

We don't think this is a charmstore issue. See go-goose/goose#66

[johnsca]

@wallyworld I don't know if that issue is the cause of the failures I see because it would imply that immediately calling charm push again would succeed but that's not what I see. When I do it in a while loop, it can take anywhere between 1 and 30-something attempts before it succeeds, but it always does eventually, and there's no consistency in how many attempts it takes. (NB: the failures I see are identical to what @cmars put above.)

[johnsca]

It's also worth noting that even for a brand new charm or bundle, when it fails the rev counter gets incremented even when nothing gets uploaded. An example:

[johnsca@murdoch:~/juju/envs/caas/bundles] $ while ! JUJU_CHARMSTORE=https://api.staging.jujucharms.com/charmstore charm push ./kubeflow cs:~johnsca/kubeflow; do sleep 1; done
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-a36bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-a26bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-dd6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-dc6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-df6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-de6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-d96bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-d86bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-db6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-da6bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-d56bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-d46bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
ERROR cannot post archive: cannot put archive blob: failed to PUT object 767bafc550926094-d76bf8e21af639c0 from container charmstore-blobs
caused by: failed reading the request data, read 0 of 756 bytes
caused by: EOF
url: cs:~johnsca/bundle/kubeflow-13
channel: unpublished

Then:

[johnsca@murdoch:~/juju/envs/caas/bundles] $ for i in {0..13}; do JUJU_CHARMSTORE=https://api.staging.jujucharms.com/charmstore charm show cs:~johnsca/kubeflow-$i; done
ERROR cannot get metadata from /~johnsca/kubeflow-0/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-0
ERROR cannot get metadata from /~johnsca/kubeflow-1/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-1
ERROR cannot get metadata from /~johnsca/kubeflow-2/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-2
ERROR cannot get metadata from /~johnsca/kubeflow-3/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-3
ERROR cannot get metadata from /~johnsca/kubeflow-4/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-4
ERROR cannot get metadata from /~johnsca/kubeflow-5/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-5
ERROR cannot get metadata from /~johnsca/kubeflow-6/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-6
ERROR cannot get metadata from /~johnsca/kubeflow-7/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-7
ERROR cannot get metadata from /~johnsca/kubeflow-8/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-8
ERROR cannot get metadata from /~johnsca/kubeflow-9/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-9
ERROR cannot get metadata from /~johnsca/kubeflow-10/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-10
ERROR cannot get metadata from /~johnsca/kubeflow-11/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-11
ERROR cannot get metadata from /~johnsca/kubeflow-12/meta/any?include=perm&include=charm-metadata&include=bundle-metadata&include=published&include=promulgated&include=owner&include=terms&include=id-name&include=id-revision&include=supported-series&include=common-info: no matching charm or bundle for cs:~johnsca/kubeflow-12
Name                    kubeflow
Owner                   johnsca
Revision                13
Promulgated             false
Home page
Bugs url
Read                    johnsca
Write                   johnsca
                         
CHANNEL                 CURRENT

[hmlanigan]

@wallyworld, it looks like the charmstore imports swift from github.com/ncw/swift, not the goose version? I tracked the error to the blob store, but I haven't found the link from the blob store to goose/swift yet.