GRPC IP address strange behavior. Changed in configuration file to 10.... lead to 127.0.0.1

[rmalenko]

GRPC IP address strange behavior. Changed in configuration file to 10.... lead to 127.0.0.1

If I set grpc_listen_address: 10.33.3.184 I can't connect by GRPC from Grafana.

And I have got the error:

Query error
rpc error: code = Internal desc = error querying ingesters in Querier.Search: failed to execute f() for 127.0.0.1:9095: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:9095: connect: connection refused"

server:
  log_level: info

  http_listen_address: 10.33.3.184
  http_listen_port: 3200

  grpc_listen_address: 10.33.3.184
  grpc_listen_port: 9095

  http_server_write_timeout: 30s
  http_server_idle_timeout: 120s

P.S. I can connect by HTTP. IP and ports are already checked.

[joe-elliott]

In single binary mode Tempo assumes it can connect to itself at 127.0.0.1:

https://github.com/grafana/tempo/blob/main/cmd/tempo/app/modules.go#L363-L366

If you would like to use the above configuration you need to configure the frontend worker address as well.

querier:
    frontend_worker:
        frontend_address: <here>

[rmalenko]

Thank you for your answer. Unfortunately, the error is still present
I have added this

querier:
  frontend_worker:
    frontend_address: 10.33.3.184:9095

if I use without port number frontend_address: 10.33.3.184 I have got this error
10.133.3.184:443 i.e. 443 was substituted

[joe-elliott]

Yup, it takes the string typed in literally and, i suppose, defaults to 443. This could be improved but may be a bit finicky given all of the various modes and configuration options in Tempo.

Is the full host/port specification 10.33.3.184:9095 working?

[rmalenko]

Unfortunately, 10.33.3.184:9095 doesn't work.
My main aim is to security. All services are working inside private VPC and protected by firewall additionally. And it isn't a Kubernetes :)

[joe-elliott]

I'm honestly not sure. I think you're going to need to do some network level debugging from both inside and outside the network namespace of the Tempo container.

Can you connect to 10.33.3.184:9095 from anywhere? I would use a tool like nc to debug this.

[rmalenko]

Yes, off course. I can connect.

Error is changing IP address into 127.0.0.1 in spite of 10.3.3.184 is already present in configuration file.