add probe command to rama cli

[GlenDC]

In spirit of tools like https://github.com/projectdiscovery/httpx as a starting point.

[AlvinKuruvilla]

Hi @GlenDC, It's been a while, and I've been meaning to jump back in the horse and contribute again. It's just been challenging with work and school commitments 😅. The issue looks like it's right up my alley, though. I just want to clarify what you're looking for in a first cut of this feature.

What I imagined for a first cut was a command like
just rama probe domains.txt
where domains.txt would look like:

google.com
facebook.com

And then, similar to httpx, after visiting each domain in the file for the moment, we would print out the request status

[GlenDC]

Yeah the probe command is going to be a big beast, but we need to start somewhere.

I would say for now:

• do the https first and http if fails flow already.
• for now just report failure if no http request possible, and status code otherwise (we can provide options on what to match for sucess and what to print later)
• I think to keep things easy I would accept "hosts" in the following order:
  • if we detect that data is being piped in, then you can read the host one line at a time from stdin
  • if -l / --list is defined we will read it from a text file (one host per list) (e.g. rama probe -l domains.txt)
  • if hosts are defined as "VARGS" then use these (e.g. rama probe google.com facebook.com)
• support optional upstream proxy as well, similar to the rama http command which offers this option through cli options as first priority and fallback to PROXY env var otherwise

For now just spawn a task (using the GracefulShutdown api) per host that is queried without worrying about concurrency, configs for this can be added alter.

Take you time with this one though, no need to hurry up.

If you do want to go the extra mile then you already tackle one of all of:

• do feel free to already add an option to control the concurrency
• add -o/--output option to write result to file instead of stdout
• add matchers:
  • --match-code: accepts a single number status code (e.g. "200") but also a range (e.g. "200..300"), option can be defined multiple times to define variants
  • --match-string: allows to check if one of the given strings is found in the given response
  • --match-regex: allows to check if one of the given regexes matches the given response
• add -H, --host option to allow one to specify the given host to use (can be useful in case a list of ips is given to be probed)

In case there are questions do feel free to ask them here or on discord. Have fun.
Once you open a PR I can also give feedback there whenever you ask for it.

[AlvinKuruvilla]

Hey @GlenDC, I wanted to follow up on this issue and ask if you had a chance to look over the PR for the new probe command. Hope you're doing well :)

[GlenDC]

Hey @GlenDC, I wanted to follow up on this issue and ask if you had a chance to look over the PR for the new probe command. Hope you're doing well :)

I'm doing well, but very busy because of rama getting close to 0.2 (but still plenty of work) and at the same time starting to integrate rama in large scale production settings. So there's a lot of work being done in background and foreground.

So for now I'm afraid I don't really have the mental energy to look at your PR, for which sorry.

I'm going to first play with tools like httpx myself a bit to get a feeling of it, and then I'll start using your branch and providing feedback on how we can make it move forward. Thanks for the patience and hard work so far. Cannot promise an ETA, but will try to do it this month still. That should be possible.

[AlvinKuruvilla]

Glad to hear Rama is gaining traction @GlenDC kudos.

No rush on the PR, thank you for the quick reply. On my side I'll try httpx where I can and monitor the issues to see if I can pitch in anywhere 😀