Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for otlp endpoints #18

Open
kruskall opened this issue Aug 21, 2023 · 8 comments · May be fixed by #45
Open

feat: add support for otlp endpoints #18

kruskall opened this issue Aug 21, 2023 · 8 comments · May be fixed by #45

Comments

@kruskall
Copy link
Member

The soaktest runner sends events to the intake v2 endpoints and is able to generate load from the major agents however there's no option to send events to the otlp endpoint.

We should add support for that

@endorama
Copy link
Member

endorama commented Sep 4, 2023

Do we want a boolean switch (sending either Elastic APM or OTLP) or an enable switch (sending Elastic APM only or both Elastic APM and OTLP)?

In case is the first occurrence, would we prefer:

  • apmsoak run --protocol=apm|otlp
  • apmsoak elastic-apm and apmsoak otlp run

Does internal/soaktest support generating OTLP already? Also it seems apmsoak only sends payload through HTTP, which would leave out OTLP/gRPC.

@kruskall
Copy link
Member Author

kruskall commented Sep 4, 2023

Sorry the description was unclear.

apmsoak uses a scenarios file to send load to the intake api. Each scenario can be customized by tuning specific settings:

type ScenarioConfig struct {
ProjectID string `yaml:"project_id"`
ServerURL string `yaml:"server"`
APIKey string `yaml:"api_key"`
AgentName string `yaml:"agent_name"`
AgentsReplicas int `yaml:"agent_replicas"`
EventRate string `yaml:"event_rate"`
RewriteIDs bool `yaml:"rewrite_ids"`
RewriteTimestamps bool `yaml:"rewrite_timestamps"`
RewriteServiceNames bool `yaml:"rewrite_service_names"`
RewriteServiceNodeNames bool `yaml:"rewrite_service_node_names"`
RewriteServiceTargetNames bool `yaml:"rewrite_service_target_names"`
RewriteSpanNames bool `yaml:"rewrite_span_names"`
RewriteTransactionNames bool `yaml:"rewrite_transaction_names"`
RewriteTransactionTypes bool `yaml:"rewrite_transaction_types"`
Headers map[string]string `yaml:"headers"`
}

Among those there is agent_name which allows us to choose specific agents. What I had in mind was to add support for something like agent_name: otlp.

AFAIK the events are hardcoded in the repository in ndjson files so we would need to store the otlp payload but I wouldn't focus too much on that. A basic/simple example is enough. We can then iterate and use a more realistic payload. WDYT ?

@endorama
Copy link
Member

endorama commented Sep 5, 2023

It make sense, but were you thinking of supporting both OTLP via HTTP and gRPC? I briefly looked at the code but it seems it is HTTP oriented so supporting gRPC would require more adjustments.

The main reason I'm thinking we need gRPC too is that if the purpose of the test is to create constant load we can observe differences between HTTP and gRPC that we are probably interested in.

@kruskall
Copy link
Member Author

kruskall commented Sep 5, 2023

That's a good point!

I was initially thinking of otlp via HTTP but we could add gRPC too. I think HTTP should be enough/minimum for our purpose and then we can iterate and add gRPC in a followup issue to avoid growing the scope of this issue too much. WDYT ?

@simitt simitt added the good first issue Good for newcomers label Dec 10, 2023
@simitt
Copy link

simitt commented Jan 10, 2024

Agree, let's start with HTTP and scope this task to OTLP over HTTP.

@endorama
Copy link
Member

I started working on this again, following @kruskall suggestion.

@endorama endorama linked a pull request Jan 17, 2024 that will close this issue
@carsonip carsonip self-assigned this Feb 12, 2024
@carsonip
Copy link
Member

The initial effort in #45 using otlp/http in JSON protobuf encoding is blocked by elastic/apm-server#12636 . It may be better to evaluate alternative tools like https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen

@simitt
Copy link

simitt commented May 1, 2024

#58 addressed this partially. Support for apmsoak is still missing.

@simitt simitt removed the good first issue Good for newcomers label May 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants