Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace encode/databases with psycopg & pydantic model validation #128

Merged
merged 74 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
da5de1a
build: strip out unecessary backend requirements
spwoodcock Aug 6, 2024
1216d1c
fix(backend): attach lifespan to fastapi for startup events
spwoodcock Aug 6, 2024
3ac8a5c
refactor: use psycopg connection pool
spwoodcock Aug 6, 2024
e6b0a3c
build: correctly use alembic binary directly (no pdm required)
spwoodcock Aug 6, 2024
ca53e51
build: bind port for local db, add initdb args for locale
spwoodcock Aug 6, 2024
cf150d4
build: update fastapi, pydantic, add email-validator as pydantic extras
spwoodcock Aug 6, 2024
65fdcc8
build: update drone-flight plan --> v0.2.3
spwoodcock Aug 6, 2024
dab56a8
fix: log level using text level instead of number levels
spwoodcock Aug 6, 2024
9412c7d
refactor: update geojson parsing logic, add slugify to utils
spwoodcock Aug 6, 2024
99c036c
feat: working project routes using psycopg + pydantic models
spwoodcock Aug 6, 2024
492a9d9
refactor: rename create_project route to POST on /projects
spwoodcock Aug 6, 2024
8ebec65
refactor: rename outline_geojson & outline_no_fly_zones
spwoodcock Aug 6, 2024
5aa3787
refactor: replace all refs to Database --> Connection (psycopg)
spwoodcock Aug 6, 2024
b5a4b77
style: update import sorting for project_schemas
spwoodcock Aug 7, 2024
c0d5ede
refractor: Refactor database interactions to use psycopg. Updated que…
Pradip-p Aug 9, 2024
fd803f8
refractor: refractor the user profile update
Pradip-p Aug 9, 2024
e91599f
refractor: refractor over user module
Pradip-p Aug 9, 2024
388b5cc
feat: DbDrone schema with crud functions
nrjadkry Aug 9, 2024
326da39
drone crud with psycopg and pydantic schemas
nrjadkry Aug 9, 2024
52c863e
remove create_drone function from drones
nrjadkry Aug 9, 2024
e402893
feat: drone deps to get one drone
nrjadkry Aug 9, 2024
04e9284
fix: drone schemas to delete drone
nrjadkry Aug 9, 2024
538a0f4
delete drone schemas
nrjadkry Aug 9, 2024
11f0a4e
update: drone cruds
nrjadkry Aug 9, 2024
62ff03b
get a single drone
nrjadkry Aug 9, 2024
81b6d7d
update: drone retrieve api output model
nrjadkry Aug 9, 2024
2685d9b
remove redundancy in drone schemas
nrjadkry Aug 9, 2024
fc700b0
Merge pull request #137 from hotosm/refactor-psycopg-drone-module
spwoodcock Aug 9, 2024
4ac534e
refractor: refactor UserProfile schemas by introducing a shared BaseU…
Pradip-p Aug 10, 2024
f82fc17
fix: added base class UserProfileIn & DbUserProfile
Pradip-p Aug 10, 2024
4b32fbe
refractor: refactor user update profile functions
Pradip-p Aug 10, 2024
5c2820d
refactor: Moved dependency functions to _deps.py and retained CRUD lo…
Pradip-p Aug 11, 2024
6272c05
refactor: Moved dependency functions to _crud.py & _deps.py
Pradip-p Aug 11, 2024
fb8e19c
refactor: user schams & remove fixme code
Pradip-p Aug 12, 2024
76f2120
fix: import error from user_crud * user_routes
Pradip-p Aug 12, 2024
3fbe4cd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 12, 2024
5148c49
Merge pull request #136 from hotosm/feat/psycopg-user-module
nrjadkry Aug 12, 2024
648ed14
refactor: refractor task module using pyscopg and pydantic
Pradip-p Aug 12, 2024
5fc9afd
feat: shift the _shemas to _crud.py
Pradip-p Aug 12, 2024
e7c3d8e
fix: changes update to update_task_state
Pradip-p Aug 12, 2024
6f68ee8
refractor: update request mapping functions
Pradip-p Aug 13, 2024
a6870d1
fix: rename functions name in pending tasks
Pradip-p Aug 13, 2024
99f12dc
Fix: Access project details and drone operator.
Pradip-p Aug 13, 2024
7a2280a
Merge pull request #141 from hotosm/feat/psycopg-task-module
nrjadkry Aug 13, 2024
a22c37c
fix: reslove merge conflict with main
Pradip-p Aug 13, 2024
df0df44
refractor: refractor the statistics & read_task with pyscopg
Pradip-p Aug 13, 2024
855539b
refractor: changes waypoints module with pyscopg
Pradip-p Aug 13, 2024
b38b58c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 13, 2024
1ac7ec8
fix: fixes the project creator issues
Pradip-p Aug 13, 2024
c3a4edb
Merge branch 'feat/psycopg-pydantic' of github.com:hotosm/drone-tm in…
Pradip-p Aug 13, 2024
f59dfcd
fix(frontend): fixes task boundary api fail
Pradip-p Aug 13, 2024
74f544e
fix: fixes the task listing issues
Pradip-p Aug 13, 2024
af7a555
refractor: changes task module with pyscopg
Pradip-p Aug 13, 2024
3887f43
fix: get task waypoints
Pradip-p Aug 14, 2024
6ffebf1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 14, 2024
8481dfe
Refactor: Rename to across modules
Pradip-p Aug 14, 2024
09c172c
fix: no fly zone cliping isuues
Pradip-p Aug 14, 2024
ea75e65
fix: upload project task boundaries
Pradip-p Aug 14, 2024
15462e2
remove: remove unnecessary code
Pradip-p Aug 14, 2024
ceec01c
fix: remove pending task api endpoint
Pradip-p Aug 14, 2024
ac27402
feat: update request function for auto approval
Pradip-p Aug 15, 2024
0774cd9
fix: task statistics endpoint
Pradip-p Aug 15, 2024
20eb00e
fix: resolve the merge conflict with main
Pradip-p Aug 20, 2024
c59b044
fix: issues slove in list tasks
Pradip-p Aug 20, 2024
4037b3a
fix: issues slove in get project details
Pradip-p Aug 20, 2024
cb1ced4
fix: refine the project get all
Pradip-p Aug 20, 2024
1f4b3ae
fix: remove xxx_crud.py from all module
Pradip-p Aug 20, 2024
1967805
fix: issues slove in get project endpoint
Pradip-p Aug 20, 2024
8b47c6d
fix: update user profile dashboard
Pradip-p Aug 20, 2024
47d1a27
fix: issues request_mapping & update state
Pradip-p Aug 20, 2024
53d277d
fix: issues on read task endpoint
Pradip-p Aug 20, 2024
696caa4
feat: Add no-fly-zones on project details
Pradip-p Aug 21, 2024
0b36027
fix(pdm.lock): slove merge conflict with main branch
Pradip-p Aug 21, 2024
588f1eb
fix: comment the unsed var
Pradip-p Aug 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: upload project task boundaries
  • Loading branch information
Pradip-p committed Aug 14, 2024
commit ea75e65021e2f9abc76f5d5cdfa5fabcb507a93f
2 changes: 1 addition & 1 deletion src/backend/app/projects/project_logic.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ async def create_tasks_from_geojson(
log.debug(
"COMPLETE: creating project boundary, based on task boundaries"
)
# return True
return True
except Exception as e:
log.exception(e)
raise HTTPException(e) from e
Expand Down
23 changes: 5 additions & 18 deletions src/backend/app/projects/project_routes.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import os
from typing import Annotated
import uuid
import geojson
from datetime import timedelta
from fastapi import APIRouter, HTTPException, Depends, UploadFile, File, Form
Expand All @@ -17,7 +16,6 @@
from app.config import settings
from app.users.user_deps import login_required
from app.users.user_schemas import AuthUser
from psycopg.rows import dict_row

router = APIRouter(
prefix=f"{settings.API_PREFIX}/projects",
Expand Down Expand Up @@ -75,10 +73,9 @@ async def create_project(

@router.post("/{project_id}/upload-task-boundaries", tags=["Projects"])
async def upload_project_task_boundaries(
# project: Annotated[
# project_schemas.DbProject, Depends(project_deps.get_project_by_id)
# ],
project_id: uuid.UUID,
project: Annotated[
project_schemas.DbProject, Depends(project_deps.get_project_by_id)
],
db: Annotated[Connection, Depends(database.get_db)],
user: Annotated[AuthUser, Depends(login_required)],
task_featcol: Annotated[FeatureCollection, Depends(project_deps.geojson_upload)],
Expand All @@ -92,18 +89,8 @@ async def upload_project_task_boundaries(
"""
log.debug("Creating tasks for each polygon in project")

async with db.cursor(row_factory=dict_row) as cur:
await cur.execute(
"""SELECT * FROM projects WHERE id = %(project_id)s LIMIT 1;""",
{"project_id": project_id},
)
project = await cur.fetchone()
if not project:
raise HTTPException(
status_code=HTTPStatus.BAD_REQUEST, detail="Project not found."
)
await project_logic.create_tasks_from_geojson(db, project["id"], task_featcol)
return {"message": "Project Boundary Uploaded", "project_id": f"{project['id']}"}
await project_logic.create_tasks_from_geojson(db, project.id, task_featcol)
return {"message": "Project Boundary Uploaded", "project_id": f"{project.id}"}


@router.post("/preview-split-by-square/", tags=["Projects"])
Expand Down
29 changes: 14 additions & 15 deletions src/backend/app/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -312,36 +312,35 @@ def get_address_from_lat_lon(latitude, longitude):
return address_str


def multipolygon_to_polygon(
featcol: geojson.FeatureCollection,
) -> geojson.FeatureCollection:
def multipolygon_to_polygon(features: Union[Feature, FeatCol, MultiPolygon, Polygon]):
"""Converts a GeoJSON FeatureCollection of MultiPolygons to Polygons.

Args:
featcol : A GeoJSON FeatureCollection containing MultiPolygons/Polygons.
features : A GeoJSON FeatureCollection containing MultiPolygons/Polygons.

Returns:
geojson.FeatureCollection: A GeoJSON FeatureCollection containing Polygons.
"""
final_features = []
geojson_feature = []
features = parse_featcol(features)

for feature in featcol.get("features", []):
properties = feature["properties"]
try:
geom = shape(feature["geometry"])
except ValueError:
log.warning(f"Geometry is not valid, so was skipped: {feature['geometry']}")
continue
# handles both collection or single feature
features = features.get("features", [features])

for feature in features:
properties = feature["properties"]
geom = shape(feature["geometry"])
if geom.geom_type == "Polygon":
final_features.append(geojson.Feature(geometry=geom, properties=properties))
geojson_feature.append(
geojson.Feature(geometry=geom, properties=properties)
)
elif geom.geom_type == "MultiPolygon":
final_features.extend(
geojson_feature.extend(
geojson.Feature(geometry=polygon_coords, properties=properties)
for polygon_coords in geom.geoms
)

return geojson.FeatureCollection(final_features)
return geojson.FeatureCollection(geojson_feature)


def normalise_featcol(featcol: geojson.FeatureCollection) -> geojson.FeatureCollection:
Expand Down