Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to split CSV input file amongst all containers/tasks? #35

Closed
jayczech23 opened this issue Feb 9, 2021 · 12 comments
Closed

How to split CSV input file amongst all containers/tasks? #35

jayczech23 opened this issue Feb 9, 2021 · 12 comments
Labels
question Further information is requested

Comments

@jayczech23
Copy link

jayczech23 commented Feb 9, 2021

I am trying to use a large single input CSV file split amonst all of the containers / tasks I am executing load tests on. It appears that each container is using the same users (rows 1 - 100 for example) for each container, which results in only the first 100 users being used for authentication step in JMeter test, even if trying to test 1000 concurrent users.

How do you solve for this? If I have a large CSV input file and I want to split it between each container so that each has a set of unique users to use for testing?

@jayczech23 jayczech23 added the question Further information is requested label Feb 9, 2021
@beomseoklee
Copy link
Member

@jayczech23 Thanks for your feedback.

Unfortunately, we only support what JMeter script itself is doing, so I believe once Fargate containers are running, they only go through what JMeter script says. But I'll look for a solution if splitting a large file in pieces to each Fargate tasks is possible.

@jayczech23
Copy link
Author

@beomseoklee

Thank you very much for your prompt reply!

I was thinking of breaking the single large input file into multiple ones and distribute that amongst the tasks somehow.
Thank you for looking into this, please let me know if need any additional information for use case.

Will keep this thread updated with any findings on a potential workaround solution for this use case.

@madhu-gudivaka
Copy link

Request made in #54 if accommodated in the solution would eliminate this need as well.

@tpark-itops
Copy link

Any update to this?

@tpark-itops
Copy link

I found following pages on BlazeMeter:

@serdarbuyuran
Copy link

Any update to this?
I've tried. {__P(InstanceID)} or ${__env(TAURUS_SESSIONS_INDEX)} does not work on DLT.
is there a property in DLT that identifies instance ID?

@tpark-itops
Copy link

Yes, this didn't work for me either. To make it work, I had to manually tweak TaskRunner lambda function and pass in InstanceID as the environment variable. Here's the feature enhancement request that I submitted - #101

@serdarbuyuran
Copy link

Thanks tpark-itops

Hi beomseoklee, In jmeter performance tests, each computer should have a different parameter file(.csv) to use. Otherwise, if the same parameter file is sent to all computers, all computers will run the test with the same parameters.This is undesirable for many performance testing scenarios. Like other cloud performance testing solutions, you can create a solution for this.

Thank you for your support.

@G-Lenz
Copy link
Contributor

G-Lenz commented Nov 21, 2022

Hi,

#54 has been since accommodated, hopefully this provides a workaround.

We have also added the environment variable mentioned in #101 to our backlog in order to allow for multiple JMeter engines.

Another potential workaround is supply a different CSV dataset to different thread groups in JMeter. This does not have each container running a different set of parameters, but you will have multiple scenarios running, they will just be running on the same container. For example if you specified a thread group using CSV X and a thread group with CSV Y, each Fargate machine will be running separate requests, one with X and one with Y.

@phillee-cqe
Copy link

@jayczech23 Thanks for your feedback.

Unfortunately, we only support what JMeter script itself is doing, so I believe once Fargate containers are running, they only go through what JMeter script says. But I'll look for a solution if splitting a large file in pieces to each Fargate tasks is possible.

Hi @beomseoklee - I know this is an old thread but I am unable to locate any solution. Please could you kindly provide an update on this issue/ challenge. Has a solution been implemented that would split a given data file into different batches across task instances? If not, is there an accessible variable (via a JMeter Groovy script) that can identify the Task ID?

Thanks,
Phil.

@kamyarz-aws
Copy link
Member

Hi @phillee-cqe

Unfortunately we havent been able to prioritize this. I can look into workaround in more detail and see what I can find and get back to you. Is this any helpful #35 (comment)?

@phillee-cqe
Copy link

Hi @kamyarz-aws

Many thanks for responding.

The suggestions within [https://github.com//issues/35#issuecomment-1322420712?] in particular that suggested by @tpark-itops doesn't provide a clear understanding of its implementation. The available task URI variables don't allow you to distinguish a particular task (for example the ARN) in order to then split the shared CSV file accordingly (using in combination the threadNum, for example).

I, and I suspect the community using DLT, would very much appreciate the proposal of a workaround or indeed an update with the feature included. This particular capability is fundamental in being to run a representative performance test. Without it the tool is significantly limited in its application and alternatives, such as BlazeMeter, would need to be used.

Regards,
Phil.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

8 participants