Questions & problems with first time setup.

[Waffles-2]

So I have followed the docs on the guide and I have 3 terminals

• one that starts the two fdaemon, enrich and the front-end python manage.py runserver
• one with the back-end python manange.py run_thug
• one with the back-end python manange.py consumer

It seems the web GUI & back-end are set up and running. However, when I submit to the web GUI the back-end does not receive anything.

I don't know why this is.

A few questions:

• The installation guide says RabbitMQ is a requirement, I have installed this but I am a bit lost on what I should be doing with it? Like configuration and the such.
• In the front end config file there is the following line BE: IP addresses of none master back-ends (0.0.0.0, 1.1.1.1 ...). How do you determine what IP address should be here?

Thank you for your help in advance!

[Dennisparchkov]

Hi @Waffles-2,

I would suggest you to tun 5 terminals:

• one that starts fdaemon (so you can see the errors it gives when trying to submit a task)
• one for python manage.py runserver
• one for run_thug
• one for consumer
• and the last for sudo mongod (mongo database)

The enrich is optional used to add details to scans.

RabbitMQ is used as a queue messaging broker to send tasks to a backend, you should be doing anything with it apart from starting the service. The only setup that is required (that i forgot to put in the docs) is that you have to create a new admin user:

rabbitmqctl add_user admin admin
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"

We created the messaging in the hopes that multiple backends would be deployed so when you can submit a scan from the front end you could pick one of the backends (BE) listed from the config file or 'any' which currently submits it to localhost. Nothing needs to be added to this part of the config file.

Hope this helps, let me know if you are having issues.

[Waffles-2]

Hi @Dennisparchkov,

Thanks for you fast reply! I have now set up RabbitMQ. And have all the scripts running.

Like you said

fdaemon so we can see the errors it gives when trying to submit a task

So here is the error I am getting & also to note run_thug is not getting any tasks.

2017-03-29 13:52:15 UTC [interface.management.commands.fdaemon] DEBUG: Got 1 new tasks
2017-03-29 13:52:16 UTC [interface.management.commands.fdaemon] DEBUG: Posting task 5
2017-03-29 13:52:16 UTC [interface.management.commands.fdaemon] DEBUG: [5] Marking task as running
2017-03-29 13:52:16 UTC [interface.management.commands.fdaemon] DEBUG: Checking for complete tasks
2017-03-29 13:52:16 UTC [interface.management.commands.fdaemon] DEBUG: Sleeping for 6 seconds
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] DEBUG: Fetching new tasks to post to backend.
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] DEBUG: Got 0 new tasks
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] DEBUG: Checking for complete tasks
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] INFO: Task 5 Authentication Error
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] DEBUG: [5] Marking task as failed
2017-03-29 13:52:22 UTC [interface.management.commands.fdaemon] DEBUG: Sleeping for 6 seconds
2017-03-29 13:52:28 UTC [interface.management.commands.fdaemon] DEBUG: Fetching new tasks to post to backend.
2017-03-29 13:52:28 UTC [interface.management.commands.fdaemon] DEBUG: Got 0 new tasks
2017-03-29 13:52:28 UTC [interface.management.commands.fdaemon] DEBUG: Checking for complete tasks
2017-03-29 13:52:28 UTC [interface.management.commands.fdaemon] DEBUG: Sleeping for 6 seconds
2017-03-29 13:52:34 UTC [interface.management.commands.fdaemon] DEBUG: Fetching new tasks to post to backend.

Seems as if I run sudo python manage.py fdaemon rather than python manage.py fdaemon it doesn't get that error and the task gets through to run_thug

[Dennisparchkov]

I guess you have to run fdaemon with sudo, communicating via Rabbitmq needs root

[Waffles-2]

You were correct.

But now run_thug seems to hang here until timeout:
2017-03-29 14:01:21 UTC [main.management.commands.run_thug] DEBUG: [1] Will run command: unbuffer /usr/bin/sudo /usr/bin/docker run --rm -a stdin -a stdout -a stderr -it thugsrumal/thug_docker:latest thug -D 172.17.0.1:27017 www.test.org

When I run the above command unbuffer is trying to run I get the following output:

docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:37924->[::1]:53: read: connection refused.

I guess I must have missed something when installed/setting up docker.

[Waffles-2]

That was just a networking problem nevermind! I think I have it set up and working now. I will get back to you if it isn't!

Thanks for your help @Dennisparchkov

[Dennisparchkov]

cool, i though there was an issue with pulling the docker image

[Waffles-2]

It is downloading the docker image now so hopefully it should work!

[Waffles-2]

So now run_thug still hangs here until timeout:
2017-03-29 14:01:21 UTC [main.management.commands.run_thug] DEBUG: [1] Will run command: unbuffer /usr/bin/sudo /usr/bin/docker run --rm -a stdin -a stdout -a stderr -it thugsrumal/thug_docker:latest thug -D 172.17.0.1:27017 www.test.org

When I run the command unbuffer /usr/bin/sudo /usr/bin/docker run --rm -a stdin -a stdout -a stderr -it thugsrumal/thug_docker:latest thug -D 172.17.0.1:27017 www.test.org the terminal just hangs until it timesout.

But when I run the command /usr/bin/sudo docker run --rm -a stdin -a stdout -a stderr -it thugsrumal/thug_docker:latest thug -D 172.17.0.1:27017 www.test.org it works... this is odd. Is the unbuffer part necessary?

[Waffles-2]

So if I go into run_thug.py and remove unbuffer the command that gets run is the following: /usr/bin/sudo docker run --rm -a stdin -a stdout -a stderr -it thugsrumal/thug_docker:latest thug -D 172.17.0.1:27017 www.test.org

This results in the following lines:

2017-03-30 10:15:06 UTC [main.management.commands.run_thug] INFO: [120] Got ObjectID: 58dcdaa46b35c900015bb115
2017-03-30 10:15:06 UTC [main.management.commands.run_thug] INFO: Now making flat tree.
2017-03-30 10:15:06 UTC [main.management.commands.run_thug] DEBUG: [120] Marking task as completed
2017-03-30 10:15:06 UTC [main.management.commands.run_thug] INFO: Sleeping for 10 seconds waiting for new tasks

EDIT: Never mind the above code seems to be correct the report is not empty, you have to double click on the nodes to see more information.

[pdelsante]

Hello,

The fact that you actually got a valid ObjectID in the log means that Thug was run successfully and it did manage to connect to MongoDB; the unbuffer command was added to prevent data loss in the stdout of the command invoked by subprocess, when it terminates. Probably it's not strictly necessary, but I think it's a good idea to keep it; I'm sure @Dennisparchkov can comment better than me on this.

Glad to hear the report is not empty, by the way! :-)

[Waffles-2]

Hey @pdelsante thanks for they fast reply you are absolutely right after some more investigation I got it sorted, now I have everything set up and working except the location map doesn't have any points on it :(

I have a file called GeoPlugin-config.conf with the following contents:

[db_path]
city = /home/waffles/geoip/db/GeoLite2-city.mmdb
anonymous_ip = None
connection_type = None
isp = None
[dbs]
city = True
anonymous_ip = False
connection_type = False
isp = False

I also have the enrich daemon running. Is my city path wrong? Or is there something else I should be doing in the setup?

Sorry for all the questions @Dennisparchkov I'll try and make this my last one!

[pdelsante]

Did you install the geoip-database-contrib package (provided that you're running under Ubuntu or Debian)? Do you actually have anything under /home/waffles/geoip/db/GeoLite2-city.mmdb?

[Waffles-2]

I have just installed geoip-database-contrib.

But I don't understand what I should be putting under city =. Obviously it is a database but which database? GeoLite2-city.mmdb was just a database I downloaded from their website I have no idea if it is the correct thing to be pointing to.

[pdelsante]

To be honest, I have no idea what the mmdb format looks like. Maybe it's the correct format, but I cannot guarantee.

Rumal expects to find a .dat file such as the one you should find under /var/lib/geoip-database-contrib/GeoLiteCity.dat after installing the geoip-database-contrib package. You may want to try with this one.

[Dennisparchkov]

Hi,
Good too see that the scans work.
You can have a look the Docker Image of Rumal to help the the GeoPlugin:
https://github.com/thugs-rumal/rumal_docker/blob/master/frontend/start.sh#L34-L48

I'm not to sure what the issue is, it might be the name of the conf file, try calling it GeoPlugin.conf.

[Waffles-2]

Just changed the database file to /var/lib/geoip-database-contrib/GeoLiteCity.dat and changed the conf file name to GeoPlugin.conf but unforuntately the map still isn't showing any data.

[Dennisparchkov]

It seems there is some problem with using the geoip-database-contrib package, i cant get it to work either.

The Docker image of Rumal downloads the .mmdb file from maxmind.com (https://github.com/thugs-rumal/rumal_docker/blob/master/frontend/start.sh#L35-L36). You can try using this instead of geoip-database-contrib.