Store binary data into etcd using the SDK

[Tochemey]

After reading through the documentation I have only seen we can only store string value into etcd using the SDK. My requirement is to store protobuf. Therefore to go with that I decided to encode the protobuf message byte array into base64. However when reading the data back I kind of getting a decoding error from base64.

This is the persisted data log in etcd:

{"level":"debug","ts":"2023-05-20T20:32:22.671678Z","caller":"etcdserver/server.go:2231","msg":"applyEntryNormal","raftReq":"header:<ID:17612320554033594411 > put:<key:\"goaktcluster-9051ca50-0876-43a7-aea3-ca8f19268e4f/f575de67-940d-46ac-9aa5-fb4a95e0161b\" value:\"CiRmNTc1ZGU2Ny05NDBkLTQ2YWMtOWFhNS1mYjRhOTVlMDE2MWISWAoHMC4wLjAuMBCoRhokZjU3NWRlNjctOTQwZC00NmFjLTlhYTUtZmI0YTk1ZTAxNjFiIiRhZTYwOWFkOC05NTJjLTRiZTktOGNiZi0xZDg3MTE1ZDQ3MjAaEhH/gQMBAQVydHlwZQH/ggAAAA==\" > "}

and this is the outcome when I was trying to decode it via the Get call:

"level":"debug","ts":"2023-05-20T20:33:45.385788Z","caller":"etcdserver/server.go:2231","msg":"applyEntryNormal","raftReq":"header:<ID:17612320554033594417 > put:<key:\"goaktcluster-9051ca50-0876-43a7-aea3-ca8f19268e4f/a3db99fd-21c6-43ed-a1fd-1fdf5396707c\" value:\"CiRhM2RiOTlmZC0yMWM2LTQzZWQtYTFmZC0xZmRmNTM5NjcwN2MSWAoHMC4wLjAuMBCoRhokYTNkYjk5ZmQtMjFjNi00M2VkLWExZmQtMWZkZjUzOTY3MDdjIiRjMmUwNmRmMC0xYzQ2LTQ4YWUtODk1Mi0wOTE1NGQwNzIwZGMaEhH/gQMBAQVydHlwZQH/ggAAAA==\" > "}
{"level":"debug","ts":"2023-05-20T20:34:00.864452Z","caller":"v3rpc/interceptor.go:182","msg":"request stats","start time":"2023-05-20T20:34:00.862927Z","time spent":"1.48175ms","remote":"10.244.0.98:44890","response type":"/etcdserverpb.KV/Range","request count":0,"request size":88,"response count":1,"response size":330,"request content":"key:\"goaktcluster-9051ca50-0876-43a7-aea3-ca8f19268e4f/a3db99fd-21c6-43ed-a1fd-1fdf5396707c\" "}
{"level":"error","ts":1684614840.8657634,"msg":"failed to decode actor=a3db99fd-21c6-43ed-a1fd-1fdf5396707c record: illegal base64 data at input byte 3"}

This is how I am getting the value from the Get call response:

resp.Kvs[0].String()

I look forward to your help and guidance. Thanks in advance.

[ahrtr]

This is how I am getting the value from the Get call response:

resp.Kvs[0].String()

Note resp.Kvs[0].Key is the key, and resp.Kvs[0].Value is the value. So please try string(resp.Kvs[0].Value).

[Tochemey]

@ahrtr Giving it a try now. I will revert back. Thanks

[Tochemey]

@ahrtr is there a way to provide an endpoint to just accept byte array in the near term? I think that can be a great feature.

[ahrtr]

Do you mean etcd proactively push data to a client provided endpoint? No, etcd doesn't support that.

But etcd supports watch. You can watch a key, and etcd will send any new value to you via the watch channel. FYI.

• https://pkg.go.dev/github.com/coreos/etcd/clientv3#example-Watcher-Watch
• https://etcd.io/docs/v3.5/learning/api/#watch-api

[Tochemey]

Nice to know that.

[Tochemey]

@ahrtr closing this discussion. Thanks for the assistance.