Task1 implment location table (#27)

* Task 1
#9
Task 1: implement the location table (table structure, operation functions, and testing functions) #9

* Adapted to the data model after 1103 discussion
1. Update the location model and its operation for the data model update
2. Add/Update test cases to support the updated data model
   update_location_bbox_by_id
   update_location_basic_by_id
   create_location

* Code Review Update
1. Comply  to the coding standard / numpydoc style guide
2. Set location table/year non-nullable, remove default value, and add type check

* Update location table
Compliant to the data model update on 11/17:
1. Move display information to answers
2. Add  get_locations for API

* Compliant to coding standard
1. Remove spaces around = for default value
2. Don't use "l" as variable name
3. Remove boolean comparison

* Add function for statistic API
implement get_location_is_done_count and its test function

* Fix gold_standard_size 0 bug and add get_location_count
1. Enhance exception string
2. fix the None list bug when gold_standard_size 0
3. Add get_location_count()

* Add test_get_location_count

* Update get_locations() and test function
Exclude user answered locations when picking locations.

* Update user model and answer model
Upload the code for answer and user table when implementing the location table.

* Update for the code review
1. Add check_answer_correctness for Answer API
2. Fix the parameter from client_id to user_id for API
3. Change get_user_done_location_count design according to the answer create process clarified  on 12/1

* Update answer and location_done algorithm
1. Change Answer. is_gold_standard to gold_standard_status
2. Add batch_process_answers and location done_at algorithm, and its test code
3. Modify get_locations to exclude done done locations.

* Enhance function name and add test case
1. Change check_answer_quality to exam_gold_standard
2. Change check_gold_candidate_status to is_answer_reliable
3. Add test case to is_answer_reliable

* Move batch_process_answer to answer_operation
Code clean up and move batch_process_answer and its test function to answer_operation

* Allow bbox*, zoom_level not required
In create_answer, make bounding box and zoom level parameters optional for gold answers.

* Update is_answer_reliable
1. Explain more clearly
2. Remove redundant code to make it simpler

* Make batch_process_answers more readable
Add gold_test_pass_status documents and default value

* Change answer_count to all answers
1. Change this function to get all answers (not excluding gold standards)
2. Add exception handlers for batch_process_answers

* Add sourc_url_root checking
1. Add sourc_url_root checking
2. Add test case for checking it

* Enhance the error messages
When missing parameters, give clear error message.

* Initial for utilities
Add utilities for importing gold standards and location table from CSV.

* Fix CORS and add answer tests

* Create export_answers.py
#28

* Add client_id and factory_id
Dump more "readable" data for testers.

* Add year_old, year_new to access photo

back-end/www/application.py

@@ -4,12 +4,19 @@
 from controllers import root
 from controllers import template_controller
 from controllers import user_controller
-
+from controllers import location_controller
+from controllers import status_controller
+from controllers import answer_controller
+from flask_cors import CORS
 
 # Register all routes to the blueprint
 app.register_blueprint(root.bp)
 app.register_blueprint(template_controller.bp, url_prefix="/template")
 app.register_blueprint(user_controller.bp, url_prefix="/user")
+app.register_blueprint(location_controller.bp, url_prefix="/location")
+app.register_blueprint(status_controller.bp, url_prefix="/status")
+app.register_blueprint(answer_controller.bp, url_prefix="/answer")
+CORS(app, resources={r"/.*": {"origins": ["https://disfactory-spotdiff.netlify.app"]}}) 
 
 # Set database migration
 migrate = Migrate(app, db)

back-end/www/controllers/answer_controller.py

@@ -0,0 +1,123 @@
+"""The controller for https://[PATH]/answer/"""
+
+from flask import Blueprint
+from flask import request
+from flask import jsonify
+from util.util import InvalidUsage
+from util.util import handle_invalid_usage
+from util.util import decode_user_token
+from config.config import config
+from models.model_operations.answer_operations import batch_process_answers
+
+bp = Blueprint("answer_controller", __name__)
+
+@bp.route("/", methods=["POST"])
+def answer():
+    """
+    The function for the front-end to submit answers if passing the gold-standard test.
+
+    Sample command to test:
+
+    $ curl -d '{"user_token":"xxxx","data":[{"location_id":1, "year_new":2017, "year_old":2010, 
+    "source_url_root":"www.test.org", "is_gold_standard":false, "bbox_left_top_lat":24.0962704615941, 
+    "bbox_left_top_lng":"120.462878886353","bbox_bottom_right_lat":24.0962704615941, 
+    "bbox_bottom_right_lng":120.462878886353, "land_usage":1, "expansion":1, "zoom_level":0}, 
+    {"location_id":8, "year_new":2017, "year_old":2010, "source_url_root":"www.test.org", "is_gold_standard":false, 
+    "bbox_left_top_lat":24.0962704615941, "bbox_left_top_lng":"120.462878886353",
+    "bbox_bottom_right_lat":24.0962704615941, "bbox_bottom_right_lng":120.462878886353, 
+    "land_usage":1, "expansion":1, "zoom_level":0}]}' -H "Content-Type: application/json"
+     -X POST http://localhost:5000/answer/
+
+    Parameters
+    ----------
+    user_token : str
+        The encoded user JWT, issued by the back-end.
+        (required)
+    data : list of dict
+        The answers, in the format [{"FIELD1:"VALUE1","FIELDS2":"VALUE2", ...}].
+        (required)
+        FIELDS:
+            year_old: int
+                year Marks which year the satellite photo was taken from our geo sources.
+                (required)
+            year_new: int
+                A newer photo to be compared with the one taken in year_old.
+                (required)
+            source_url_root : str
+                URL to store the location on the map.
+                (required)
+            land_usage : int
+                User's answer of judging a construction is built.
+                0 means unknown.
+                1 means building.
+                2 means farm.
+                (required)
+            expansion : int
+                User's answer of judging the construction is expanded.
+                0 means unknown.
+                1 means no new expansion.
+                2 means yes (there is expansion).
+                (required)
+            bbox_left_top_lat : float
+                The latitude of the top-left corner of the bounding box for displaying the focus.
+                (optional)
+            bbox_left_top_lng : float
+                The longitude of the top-left corner of the bounding box for displaying the focus.
+                (optional)
+            bbox_bottom_right_lat : float
+                The latitude of the bottom-right corner of the bounding box for displaying the focus.
+                (optional)
+            bbox_bottom_right_lng : float
+                The longitude of the bottom-right corner of the bounding box for displaying the focus.
+                (optional)
+            zoom_level : int
+                The zoom level for displaying the location.
+                (optional)
+            user_id : int
+                Foreign key to the user table.
+                (required)
+            location_id : int
+                Foreign key to the location table.
+                (required)
+
+    Returns
+    -------
+        {"Passed":True|False}
+    """    
+    if request.method == "POST":
+        rj = request.get_json()
+
+        if rj is None:
+            e = InvalidUsage("Please provide correct parameters.", status_code=400)
+            return handle_invalid_usage(e)
+
+        # Get user id from user_token.
+        error, user_json = decode_user_token(rj, config.JWT_PRIVATE_KEY, check_if_admin=False)
+        if error is not None: return error
+
+        user_id = user_json["user_id"]
+
+        if user_id is None:
+            e = InvalidUsage("Please provide correct user token.", status_code=400)
+            return handle_invalid_usage(e) 
+
+        if "data" not in rj:
+            e = InvalidUsage("Please provide data.", status_code=400)
+            return handle_invalid_usage(e) 
+
+        # Check all the answers from frontend to decide the next step.
+        try:
+            pass_status = batch_process_answers(user_id, rj["data"])
+        except Exception as errmsg:
+            e = InvalidUsage(repr(errmsg), status_code=400)
+            return handle_invalid_usage(e)
+
+        if pass_status:
+            return_status = {"Passed" : True}
+        else:            
+            return_status = {"Passed" : False}
+
+        return jsonify(return_status)
+
+
+

back-end/www/controllers/location_controller.py

@@ -0,0 +1,107 @@
+"""The controller for https://[PATH]/location/"""
+
+from flask import Blueprint
+from flask import request
+from flask import jsonify
+import jwt
+from util.util import decode_jwt
+from config.config import config
+from util.util import InvalidUsage
+from util.util import handle_invalid_usage
+from util.util import try_wrap_response
+from config.config import config
+from models.model_operations.location_operations import get_locations
+from models.schema import locations_schema
+
+bp = Blueprint("location_controller", __name__)
+
+@bp.route("", methods=["GET"])
+def location():
+    """
+    The function for the front-end to retrieve random location data by specifying amount.
+
+    Sample command to test:
+    $ curl -H "Content-Type: application/json" -X GET http://localhost:5000/location?size=5\&gold_standard_size=1\&user_token=xxxx
+    $ https://localhost:5000/location?&size=5&gold_standard_size=1?user_token=xxxxx
+    
+    Parameters
+    ----------
+    user_token : str
+        The encoded user JWT, issued by the back-end.
+        (required)
+    size : int
+        Total number of locations to be returned.
+        (required)
+    gold_standard_size : int
+        The number of locations that should include gold standard answers.
+        There should be ("size" - "gold_standard_size") locations that are not labeled yet.
+        (required)
+
+    Returns
+    -------
+        The encoded JWT that stores location information:
+        id : int
+            ID of the location.
+        factory_id : string
+            The uuid imported from disfactory factory table.
+    """    
+    size = request.args.get("size")
+    gold_standard_size = request.args.get("gold_standard_size")
+    user_token = request.args.get("user_token")
+    if size is None:
+        e = InvalidUsage("Please provide size, the number of locations you want to get.")
+        return handle_invalid_usage(e)
+
+    try:
+        i = int(size)
+    except ValueError as ex:
+        e = InvalidUsage("size must be an integer.")
+        return handle_invalid_usage(e)
+    except Exception as ex: 
+        e = InvalidUsage("size must be an integer.")       
+        return handle_invalid_usage(e)
+
+    if int(size) < 2:
+        e = InvalidUsage("The size must be greater or equal to 2.")
+        return handle_invalid_usage(e)
+
+    if gold_standard_size is None:                
+        e = InvalidUsage("Please provide gold_standard_size, the number of gold standards.")
+        return handle_invalid_usage(e)
+    try:
+        i = int(gold_standard_size)
+    except ValueError as ex:
+        e = InvalidUsage("gold_standard_size must be an integer.")
+        return handle_invalid_usage(e)
+    except Exception as ex: 
+        e = InvalidUsage("gold_standard_size must be an integer.")       
+        return handle_invalid_usage(e)
+
+    if user_token is None:
+        e = InvalidUsage("Please provide user_token.")
+        return handle_invalid_usage(e)
+
+    try:
+        user_json = decode_jwt(user_token, config.JWT_PRIVATE_KEY)
+    except jwt.InvalidSignatureError as ex:
+        e = InvalidUsage(ex.args[0], status_code=401)
+        return (handle_invalid_usage(e), None)
+    except Exception as ex:
+        e = InvalidUsage(ex.args[0], status_code=401)
+        return (handle_invalid_usage(e), None)
+
+    user_id = user_json["user_id"]
+    if user_id is None:
+        e = InvalidUsage("Cannot find user_id")
+        return handle_invalid_usage(e)
+
+    return try_get_locations(user_id, int(size), int(gold_standard_size))
+
+@try_wrap_response
+def try_get_locations(user_id, size, gold_standard_size):
+    try:
+        data = get_locations(user_id, size, gold_standard_size)
+    except Exception as errmsg:
+        e = InvalidUsage(repr(errmsg), status_code=400)
+        return handle_invalid_usage(e)
+    return jsonify({"data": locations_schema.dump(data)})

back-end/www/controllers/root.py

@@ -1,6 +1,9 @@
 """The controller for https://[PATH]/"""
 
 from models.model import User
+from models.model_operations import user_operations
+from models.model_operations import location_operations
+from models.model_operations import answer_operations
 from flask import Blueprint
 
 
@@ -9,5 +12,29 @@
 
 @bp.route("/")
 def hello_world():
-    print(User.query.all())
-    return "Hello, World!"
+    if False:
+        users = user_operations.get_all_users()
+        location_count = location_operations.get_location_count()
+        #gold_answer_count = answer_operations.get_gold_answer_count()
+
+        BBOX_LEFT_TOP_LAT = 0.1
+        BBOX_LEFT_TOP_LNG = 0.2
+        BBOX_BOTTOM_RIGHT_LAT = 0.3
+        BBOX_BOTTOM_RIGHT_LNG = 0.4
+
+
+        answer_operations.create_answer(1, 3, 2000, 2010, "", 0, 1, 0, BBOX_LEFT_TOP_LAT, BBOX_LEFT_TOP_LNG, BBOX_BOTTOM_RIGHT_LAT, BBOX_BOTTOM_RIGHT_LNG, 0)        
+        answer_operations.create_answer(1, 4, 2000, 2010, "", 0, 1, 0, BBOX_LEFT_TOP_LAT, BBOX_LEFT_TOP_LNG, BBOX_BOTTOM_RIGHT_LAT, BBOX_BOTTOM_RIGHT_LNG, 0)        
+        gold_answer_count = answer_operations.get_gold_answer_count()
+        answer_count = answer_operations.get_answer_count()
+        #location_count = 10
+        #print("Users: ", len(users))
+        #return_string = "Hello, world! users:{} ".format(len(users))
+        return_string = "Hello, world! users:{} locations:{} answers:{}".format(len(users), location_count, answer_count)
+        #return_string = "Hello, world! users:{} locations:".format(0, location_count)
+        #return "Hello, World!"
+        return return_string
+    else:
+        return "Hello, World!"
+
+

back-end/www/controllers/status_controller.py

@@ -0,0 +1,83 @@
+"""The controller for https://[PATH]/status/"""
+
+from flask import Blueprint
+from flask import request
+from flask import jsonify
+import jwt
+from util.util import InvalidUsage
+from util.util import handle_invalid_usage
+from util.util import decode_jwt
+from config.config import config
+from models.model_operations.user_operations import get_user_count
+from models.model_operations.location_operations import get_location_is_done_count
+from models.model_operations.answer_operations import get_answer_count
+
+bp = Blueprint("status_controller", __name__)
+@bp.route("", methods=["GET"])
+def status():
+    """
+    The function for the front-end to retrieve status data.
+
+    Sample command to test:
+    $ curl -H "Content-Type: application/json" -X GET http://localhost:5000/status?user_token=xxxx
+    $ https://localhost:5000/status?user_token=xxxxx
+
+    Parameters
+    ----------
+    user_token : str
+        The encoded user JWT, issued by the back-end.
+        (required)
+
+    Returns
+    -------
+        The encoded JWT that stores status information, including:
+        individual_done_count : Int
+            Number of locations identified by the user.
+        user_count : Int
+            Number of total users.
+        location_is_done_count : Int
+            The number of locations that have been labeled.
+    """    
+    if request.method == "GET":
+        user_token = request.args.get("user_token")
+        if user_token is None:
+            e = InvalidUsage("Please provide user_token.")
+            return handle_invalid_usage(e)
+
+    try:
+        user_json = decode_jwt(user_token, config.JWT_PRIVATE_KEY)
+    except jwt.InvalidSignatureError as ex:
+        e = InvalidUsage(ex.args[0], status_code=401)
+        return (handle_invalid_usage(e), None)
+    except Exception as ex:
+        e = InvalidUsage(ex.args[0], status_code=401)
+        return (handle_invalid_usage(e), None)
+
+    user_id = user_json["user_id"]
+
+    try:
+        user_done_count = get_answer_count(user_id)
+    except Exception as errmsg:
+        e = InvalidUsage(repr(errmsg), status_code=400)
+        return handle_invalid_usage(e)
+
+    try:
+        user_count = get_user_count()
+    except Exception as errmsg:
+        e = InvalidUsage(repr(errmsg), status_code=400)
+        return handle_invalid_usage(e)
+
+    try:
+        loc_done_count = get_location_is_done_count()
+    except Exception as errmsg:
+        e = InvalidUsage(repr(errmsg), status_code=400)
+        return handle_invalid_usage(e)
+
+    return_status = {"individual_done_count" : user_done_count, 
+                        "user_count" : user_count, 
+                        "answer_count" : get_answer_count(),
+                        "location_is_done_count" : loc_done_count}
+
+    return jsonify(return_status)
+
+