In 2020 NASA sent a Mars rover, Perseverance, and a robotic helicopter, Ingenuity, to explore Mars and its habitability. NASA funded this project to achieve four main objectives: determine if there was life on Mars at some point, characterize past and current climate, characterize the geology of Mars, and prepare for human exploration. It has been nearly three years and thus a lot of data has been collected.
Combining information from both Perseverance and Ingenuity's datasets yields a richer dataset. However, it is currently very difficult to work with two large and seperate datasets. Through this project we seek to close the gap by being able to emphasize areas where the datasets overlap. While we do not have access the data collected by the two robots, we hope that researchers can build upon this app in order to make important discoveries regarding Mars and its habitability. Furthermore, this project shows the potential that can result from the synchronization of data from various robots when studying an unknown environment.
The public data is maintained and provided by NASA. It provides real-time location of both Perseverance and Ingenuity. The website also provides other information regarding rock samples, waypoints, oaths, and more. For this project we are only using the rover path and helicopter flight path. The rover path data consist of sols, clock, and latitude/logitude coordinates. The helicopter path data consist of sols, flight, clock, and latitude/longitude/altitude coordinates.
This program uses the Python Flask library. Flask is a web framework used to develop generalized web applications. To install Flask, please enter the following command into your terminal:
$ pip3 install --user flask
This script uses the Redis, a NoSQL database, to store all app data to ensure that it is not lost when the Flask app stops running and allow for multiple processes to access the data at once. If the Redis Python library is not already installed on your machine and you plan on doing development with this repo, please install it using
pip3 install redis
otherwise, Docker will take care of the Redis image for this application.
The Imgur API is a RESTful API that is based on HTTP requests and it returns JSON responses. To allow the user to see the plots they produce the Flask application makes a requests to the url https://api.imgur.com/3/upload. The Flask app then return to the user the response from the Imgur API which includes a link to the image among other information
To ensure functionality across macnines, this app is containerized according to the included DockerFile and launched according to the included docker-copose.yml file. To run the app as an end user, please refer to the Pull the Docker Image and docker-compose sections below. To develop using the app in this repo, please refer to the DockerFile section below.
As an end user, running the app has four simple steps. First, change the Redis database client host in the app.py script from final-redis-service to redis-db. Then, pull the image from the Docker Hub and then build the image. To do so, please run the following command
$ docker pull lajoiekatelyn/perseverance_and_ingenuity_tracker:kube
and then, in the root of the repo,
$ docker build -t lajoiekatelyn/perseverance_and_ingenuity_tracker:kube .
Then the image should be good to go.
To launch the app alongside Redis, please run the following command in the root of the directory
$ docker-compose up -d
to run the app and Redis in the background. Then, to terminate the app and Redis,
$ docker-compose down
To build upon this repo, any new package used in the main script will need to be added to the DockerFile. Then, the docker image will need to be rebuilt. To do so,
$ docker build -t <docker_username>:perseverance_and_ingenuity_tracker:<version_number>
where <docker_username> is your Docker username and <version_number> is the verion of the image that you wish to build.
If you develop and push a new Docker image to Docker Hub, you will need to change the name of the Docker image in docker-compose.yml to the name of the image that you pushed from the build above.
NOTE: for the purpose of using docker-compose, the host declared for the Redis client in the get_redis_client() funciton in gene_api.py is set to pull from the Kubernetes environment using os. In order to develop using Flask, change the host to 127.0.0.1. Then, when it comes time to use docker-compose again, change it back to redis_ip, the variable which os pulls from the environment, for correct use with Kubernetes. Additionally, if the service used to reach the Redis database in Kubernetes is changed, update the name of the service from final-redis-service to the new name in both docker-compose.yml and final-flask-deployment.yml for use with Kubernetes.
To run this app on a Kubernetes cluster, please follow the instructions below.
Each yaml file in this repo, save for docker-compose.yml, is a file that needs to be applied to Kubernetes. To do so, enter the following commands in the console from which you have Kubernetes access:
$ kubectl apply -f final-redis-deployment.yml
$ kubectl apply -f final-pvc.yml
$ kubectl apply -f final-flask-deployment.yml
$ kubectl apply -f final-worker-deployment.yml
$ kubectl apply -f final-redis-service.yml
$ kubectl apply -f final-flask-service.yml
$ kubectl apply -f final-python-debug.yml
The console should output confirmation that you properly applied each deployment, persistent volume control, service, etc after each kube apply -f command and then you should be good to go using Kubernetes!
NOTE: if users wish to user their own Flask API in the kubernetes cluster, they must change the image being pulled in final-flask-deployment to their image on Docker Hub and then re-apply the kubernetes depolyment.
To use the cluster, first run the following comand
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
final-flask-deployment-755fbb484b-gpsnc 1/1 Running 0 44m
final-flask-deployment-755fbb484b-s54ml 1/1 Running 0 44m
final-redis-deployment-57c6d74c5b-v6hpc 1/1 Running 0 8h
final-worker-deployment-6988b44b44-tf4vv 1/1 Running 0 44m
py-debug-deployment-f484b4b99-dmgnz 1/1 Running 0 35h
Note the python debug deployment and use it to access the cluster:
$ kubectl exec -it py-debug-deployment-f484b4b99-r9vff -- /bin/bash
You will end up in a terminal something like:
root@py-debug-deployment-f484b4b99-r9vff:/#
where you can use any of the commands below, under Usage, replacing localhost with final-flask-service. An example of this can be seen in the usage section.
Here is a table of the routes:
| Route | Method | Description |
|---|---|---|
/data |
POST | Store data into redis |
| GET | Return all data from redis as a json-formatted list from both path datasets | |
| DELETE | Delete all data in redis | |
/rover |
GET | Return json-formatted list of the rover path data |
/rover/sols |
GET | Return json-formatted list of all the sols where the rover was operational |
/rover/sols/<string:sol> |
GET | Return dictionary with all the information in the rover dataset associated with that sol |
/helicopter/ |
GET | Return json-formatted list of the heli path data |
/helicopter/flights |
GET | Return json-formatted list of the heli flights |
/helicopter/flights/<string:flight> |
GET | Return dictionary with information related to a given flight |
/helicopter/sols |
GET | Return json-formatted list of all the sols where the heli was operational |
/helicopter/sols/<string:sol> |
GET | Return dictionary with all the information in the heli dataset associated with that sol |
/rover/sols/<string:sol>/helicopter |
GET | Return dictionary containing the shortest distance between the robots |
/both_deployed |
GET | Return list where both Perseverance and Ingenuity were deployed |
/map/<string:jid> |
GET | Return a dictionary containing link to the image. |
| DELETE | Delete an image from the image database | |
/jobs |
POST | Request an image be curated according to upper and lower sol bounds. |
/help |
GET | Return route usage. |
$ curl -X POST localhost:5000/data
Data loaded into db.
$ curl localhost:5000/data
{
"geometry": {
"coordinates": [
[
77.451388,
18.442665
],
[
77.449155,
18.441573
],
[
77.449181,
18.441336
],
[
77.449943,
18.44166
]
],
"type": "LineString"
},
"properties": {
"Flight": 6,
"Length_m": 202.363,
"SCLK_END": 675019237.6,
"SCLK_START": 675019097.7,
"Sol": 91
},
"type": "Feature"
}
]
$ curl -X DELETE localhost:5000/data
Data deleted.
$ curl localhost:5000/rover/sols
[
"sol:0014",
"sol:0015",
"sol:0016",
"sol:0020",
"sol:0023",
"sol:0029",
"sol:0031",
"sol:0032",
"sol:0033",
"sol:0034",
"sol:0043",
"sol:0044",
...
]
$ curl localhost:5000/rover/sols/sol:0014
{
"geometry": {
"coordinates": [
[
77.450886,
18.444627,
-7.8e-05
],
[
77.450897,
18.444617,
-7.8e-05
],
[
77.45091,
18.444606,
-7.8e-05
],
...
}
$ curl localhost:5000/helicopter/flights
[
"Flight:41",
"Flight:13",
"Flight:12",
"Flight:48",
"Flight:16",
"Flight:30",
"Flight:47",
"Flight:37",
...
]
$ curl localhost:5000/helicopter/flights/Flight:41
{
"geometry": {
"coordinates": [
[
77.407122,
18.458697
],
[
77.405825,
18.459777
],
[
77.407091,
18.459016
]
],
"type": "LineString"
},
"properties": {
"Flight": 41,
"Length_m": 181.349,
"SCLK_END": 728119180,
"SCLK_START": 728119071,
"Sol": 689
},
"type": "Feature"
}
$ curl localhost:5000/helicopter/sols
[
"sol:0058",
"sol:0061",
"sol:0064",
"sol:0069",
"sol:0076",
"sol:0091",
"sol:0107",
"sol:0120",
"sol:0133",
...
]
$ curl localhost:5000/helicopter/sols/sol:0714
{
"geometry": {
"coordinates": [
[
77.398408,
18.472231
],
[
77.398368,
18.474394
],
[
77.392373,
18.477141
]
],
"type": "LineString"
},
"properties": {
"Flight": 45,
"Length_m": 502.6,
"SCLK_END": 730342605.9,
"SCLK_START": 730342461.3,
"Sol": 714
},
"type": "Feature"
}
$ curl localhost:5000/both_deployed
[
"sol:0398",
"sol:0362",
"sol:0717",
"sol:0107",
"sol:0418",
"sol:0714",
"sol:0681",
"sol:0384",
"sol:0708",
"sol:0414",
"sol:0388",
"sol:0091"
]
$ curl localhost:5000/rover/sols/sol:0091/helicopter
{
"shortest_dist": 85.31574053466534
}
To create an image and have it generate an Imgur link:
$ curl localhost:5000/jobs -X POST -H "Content-Type:application/json" -d '{"upper": 500, "lower": 300}
Job submitted. Please wait a moment to check jid: 1720f3a2-00ea-442a-aa2f-3a5f617dcd4c for completion.
To view the image created:
$ curl localhost:5000/map/1720f3a2-00ea-442a-aa2f-3a5f617dcd4c
{
"data": {
"account_id": null,
"account_url": null,
"ad_type": null,
"ad_url": null,
"animated": false,
"bandwidth": 0,
"datetime": 1682566281,
"deletehash": "YJ3cR3AXvdBWdVl",
"description": null,
"favorite": false,
"has_sound": false,
"height": 480,
"hls": "",
"id": "MDd1NXp",
"in_gallery": false,
"in_most_viral": false,
"is_ad": false,
"link": "https://i.imgur.com/MDd1NXp.png",
"mp4": "",
"name": "",
"nsfw": null,
"section": null,
"size": 49028,
"tags": [],
"title": null,
"type": "image/png",
"views": 0,
"vote": null,
"width": 640
},
"status": 200,
"success": true
}
To delete the image from the database:
$ curl localhost:5000/map/1720f3a2-00ea-442a-aa2f-3a5f617dcd4c -X DELETE
Plot has been deleted from redis and imgur.
Within app.py edit the get_redis_client(db_num:int, decode:bool) funtion to so that we are that the host is '127.0.0.1'. The new get_redis_client should look like the example below:
def get_redis_client(db_num:int, decode:bool):
#redis_ip = os.environ.get('REDIS_IP')
#if not redis_ip:
# raise Exception()
return redis.Redis(host='127.0.0.1', port=6379, db=db_num, decode_responses=decode)
Within the jobs.py file we will also make modifications to the redis host. Edit the lines at the top of the jobs.py file to look like the example below:
#redis_ip = os.environ.get('REDIS_IP')
#if not redis_ip:
# raise Exception()
q = HotQueue("queue", host='127.0.0.1', port=6379, db=4)
rd_jobs = redis.Redis(host='127.0.0.1', port=6379, db=3)
Lastly, modify the redis host inside of the worker file get_redis_client() function. Your code should look like the example below:
def get_redis_client(db_num: int, decode: bool):
#redis_ip = os.environ.get('REDIS_IP')
#if not redis_ip:
# raise Exception()
return redis.Redis(host='127.0.0.1', port=6379, db=db_num, decode_responses=decode)