Failed to download artifact : getter subprocess failed: exit status 1 #18189
Comments
|
Thanks for the report @spanner4715. Could you check if the Nomad client agent is running as root? And do you see any relevant log lines in the client agent logs? |
|
Hello @lgfa29 However, I still got the same error as title. Until I use |
|
@spanner4715 please post client logs, they should describe what went wrong. |
|
Hello @shoenig , thanks for your reply |
|
There should be an additional line in the client logs, one that contains the string Here's an example from a while back, from a similar bug report. Notice the second line contains the actual error. |
|
Hi @shoenig I am able to consistently reproduce this running recent Debian 12 + Nomad v1.6.3 + Docker v24.0.7. In my case, I have an artifact download in my job, and it errors out with this: Followed by this in the client logs: Note that my Happy to provide more logs or anything else if it helps. Thank you. |
|
@kmott can you show more of your job file and describe the system versions? The docker driver shouldn't have anything to do with it - artifact downloading happens before the task is started. I just tried to repro a simple job in a debian vm and it worked fine |
|
Sure @shoenig , here's my Nomad Job (edited with your example pulling from job "fabio" {
datacenters = ["kitchen"]
type = "system"
update {
max_parallel = 1
min_healthy_time = "10s"
healthy_deadline = "3m"
progress_deadline = "10m"
auto_revert = false
canary = 0
}
group "loadbalancer" {
count = 1
restart {
attempts = 20
interval = "3m"
delay = "5s"
mode = "delay"
}
ephemeral_disk {
size = 128
}
network {
mode = "bridge"
port "admin" {
static = 9168
to = 9168
}
port "frontend_http" {
static = 80
to = 8080
}
port "frontend_https" {
static = 443
to = 8443
}
}
task "fabio-loadbalancer" {
driver = "docker"
config {
image = "fabiolb/fabio:1.5.15-go1.15.5"
args = [
"-cfg", "/local/etc/fabio/fabio.properties",
"-registry.consul.addr", "< ... >",
"-insecure"
]
ports = ["admin","frontend_http","frontend_https"]
}
artifact {
source = "https://raw.githubusercontent.com/hashicorp/nomad/main/.go-version"
}
# artifact {
# source = "s3::https://s3-us-west-2.amazonaws.com/my-custom-bucket/nomad/kitchen/fabio"
# destination = "local/etc/fabio"
#
# options {
# aws_access_key_id = "<...>"
# aws_access_key_secret = "<...>"
# }
# }
resources {
cpu = 500 # 500 MHz
memory = 256 # 256MB
}
service {
name = "fabio"
tags = [
"loadbalancer",
"admin",
]
port = "admin"
check {
type = "tcp"
port = "admin"
interval = "10s"
timeout = "2s"
}
}
service {
name = "fabio-frontend-http"
tags = ["loadbalancer", "frontend", "http"]
port = "frontend_http"
check {
type = "tcp"
port = "frontend_http"
interval = "10s"
timeout = "2s"
}
}
service {
name = "fabio-frontend-https"
tags = ["loadbalancer", "frontend", "https"]
port = "frontend_https"
check {
type = "tcp"
port = "frontend_https"
interval = "10s"
timeout = "2s"
}
}
}
}
}
Here's some logs (attached for reference) from And here is system info and nomad info: FWIW, I also tried running Also, I am using CNI plugin v1.3.0--not sure if that matters. |
|
Thanks @kmott - just so I can keep trying to reproduce your environment exactly, how did you install docker? And can you show the output of |
|
Thank you for your patience @shoenig, I will work on getting a reliable repro using Vagrant. If I am not able to come up with something by early next week, I'll let you know and we can probably close this out. More info soon (hopefully!). |
|
After much digging, this was caused by an older version (2.29) of glibc linked in with |
github-project-automation
bot
moved this from Triaging
to Done
in Nomad - Community Issues Triage
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
1.6.1
Operating system and Environment details
ubuntu-22.04
Issue
Couldn't download artifact "Hadoop" from "archive.apache.org", also I noticed the template file that I wrote couldn't be written in allocation
(core-site.xml, hdfs-site.xml...)
Expected Result
Artifact being downloaded completely
Actual Result
Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
The text was updated successfully, but these errors were encountered: