Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Production Release #453

Merged
merged 43 commits into from
Jan 29, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
4af55a9
feat(task-description): store rotation angle on redux store
suzit-10 Jan 24, 2025
5964617
feat(task-description): use rotation angle of redux state as a final …
suzit-10 Jan 24, 2025
ef97e98
feat(task-description): downlaod rotated plight plan .kmz file
suzit-10 Jan 24, 2025
7f4020f
feat: add `preserveDrawingBuffer: true` to preserving the drawing buffer
suzit-10 Jan 24, 2025
6f1c87d
feat: create export section page
suzit-10 Jan 24, 2025
38f911f
feat: create mapsection for export section
suzit-10 Jan 24, 2025
ff44ab8
fix: white space formating
suzit-10 Jan 24, 2025
a0164c8
feat: export project info
suzit-10 Jan 24, 2025
8b3432a
feat: add project and task on download file
suzit-10 Jan 27, 2025
e593ee0
feat: hide start procesing button if `image_processing_status` id `SU…
suzit-10 Jan 27, 2025
3c129ff
feat: add progress bar component
suzit-10 Jan 27, 2025
9fd341b
feat: add upload progress state on task details slice
suzit-10 Jan 27, 2025
8f82064
feat: save upload progress details on redux
suzit-10 Jan 27, 2025
3f44598
feat: show upload progress on task description section
suzit-10 Jan 27, 2025
d74b3ad
Merge pull request #449 from hotosm/feat/download-rotated-flight-plan
nrjadkry Jan 27, 2025
9159496
[pre-commit.ci] pre-commit autoupdate
pre-commit-ci[bot] Jan 27, 2025
38927e2
add orthophoto url and assets url to the projects endpoint
nrjadkry Jan 28, 2025
a2549cb
fix: url for assets and project orthophoto
nrjadkry Jan 28, 2025
f11b331
feat: prevent multiple event trigger on gcp editor component
suzit-10 Jan 27, 2025
f55b2b6
feat: add `ChooseProcessingParameter` modal compoennt to choose proce…
suzit-10 Jan 27, 2025
992d0d6
feat: update process imagery service
suzit-10 Jan 27, 2025
f0f0e94
feat: add toas message on processing started
suzit-10 Jan 27, 2025
9ad18e3
feat: update processing starter parameter on individual project
suzit-10 Jan 27, 2025
b6af60a
feat: render buttons as per project processing status
suzit-10 Jan 27, 2025
deef7dc
feat: popup toast message on processing started
suzit-10 Jan 27, 2025
1870c39
feat: add key on breadcrumb items
suzit-10 Jan 27, 2025
9577085
fix: cannot removed source the layer is using it
suzit-10 Jan 28, 2025
e169be5
feat(individual-project): store visible list of orthophoto sources on…
suzit-10 Jan 28, 2025
f4bc618
feat: toggle tasks orthophoto visibility from table row
suzit-10 Jan 28, 2025
8c55f41
feat(individual-project): visualize and toggle overall project orthop…
suzit-10 Jan 28, 2025
1712ea1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 28, 2025
3cca30f
Merge pull request #451 from hotosm/feat/overall-orthophoto
nrjadkry Jan 28, 2025
719f1f0
fix: image processing failed condtion
nrjadkry Jan 28, 2025
be532e2
feat(landing-page): add mobile app download section
suzit-10 Jan 28, 2025
c1ee5e3
feat(landing-page): remove unnecessary image
suzit-10 Jan 28, 2025
fe59742
fix: add type guard on user-profile image
suzit-10 Jan 28, 2025
392a3bc
fix: change password issue
suzit-10 Jan 28, 2025
734e914
feat: update dashboard card design
suzit-10 Jan 28, 2025
36d0eab
feat: show `no data available` message if there is no data
suzit-10 Jan 28, 2025
cd1723f
feat: zoom to orthophoto of overall project
suzit-10 Jan 28, 2025
c507985
fix: remove white space
suzit-10 Jan 28, 2025
885d03f
Merge pull request #450 from hotosm/pre-commit-ci-update-config
nrjadkry Jan 29, 2025
8bb9fe9
Merge pull request #452 from hotosm/feat/UI-fixes
nrjadkry Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -79,23 +79,23 @@ repos:

# Deps: ensure Python uv lockfile is up to date
- repo: https://github.com/astral-sh/uv-pre-commit
rev: 0.5.21
rev: 0.5.24
hooks:
- id: uv-lock
files: src/backend/pyproject.toml
args: [--project, src/backend]

# Versioning: Commit messages & changelog
- repo: https://github.com/commitizen-tools/commitizen
rev: v4.1.0
rev: v4.1.1
hooks:
- id: commitizen
stages: [commit-msg]

# Lint / autoformat: Python code
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: "v0.9.2"
rev: "v0.9.3"
hooks:
# Run the linter
- id: ruff
Expand Down
146 changes: 71 additions & 75 deletions src/backend/app/projects/image_processing.py
Original file line number Diff line number Diff line change
Expand Up @@ -404,88 +404,84 @@ async def process_assets_from_odm(
task = node.get_task(odm_task_id)
log.info(f"Downloading results for task {odm_task_id} to {output_file_path}")

assets_path = task.download_zip(output_file_path)
if not os.path.exists(assets_path):
log.error(f"Downloaded file not found: {assets_path}")
raise
log.info(f"Successfully downloaded ZIP to {assets_path}")

# Construct the S3 path dynamically to avoid empty segments
task_segment = f"{dtm_task_id}/" if dtm_task_id else ""
s3_path = f"dtm-data/projects/{dtm_project_id}/{task_segment}assets.zip"
log.info(f"Uploading {assets_path} to S3 path: {s3_path}")
add_file_to_bucket(settings.S3_BUCKET_NAME, assets_path, s3_path)

with zipfile.ZipFile(assets_path, "r") as zip_ref:
zip_ref.extractall(output_file_path)

orthophoto_path = os.path.join(
output_file_path, "odm_orthophoto", "odm_orthophoto.tif"
)
if not os.path.exists(orthophoto_path):
log.error(f"Orthophoto not found at {orthophoto_path}")
raise FileNotFoundError("Orthophoto file is missing")

reproject_to_web_mercator(orthophoto_path, orthophoto_path)
s3_ortho_path = f"dtm-data/projects/{dtm_project_id}/{task_segment}orthophoto/odm_orthophoto.tif"
log.info(f"Uploading reprojected orthophoto to S3 path: {s3_ortho_path}")
add_file_to_bucket(settings.S3_BUCKET_NAME, orthophoto_path, s3_ortho_path)

images_json_path = os.path.join(output_file_path, "images.json")
if os.path.exists(images_json_path):
s3_images_json_path = (
f"dtm-data/projects/{dtm_project_id}/{task_segment}images.json"
)
log.info(f"Uploading images.json to S3 path: {s3_images_json_path}")
add_file_to_bucket(
settings.S3_BUCKET_NAME, images_json_path, s3_images_json_path
if str(odm_status_code) == "30":
status = ImageProcessingStatus.FAILED
elif str(odm_status_code) == "40":
assets_path = task.download_zip(output_file_path)
if not os.path.exists(assets_path):
log.error(f"Downloaded file not found: {assets_path}")
raise
log.info(f"Successfully downloaded ZIP to {assets_path}")

# Construct the S3 path dynamically to avoid empty segments
task_segment = f"{dtm_task_id}/" if dtm_task_id else ""
s3_path = f"dtm-data/projects/{dtm_project_id}/{task_segment}assets.zip"
log.info(f"Uploading {assets_path} to S3 path: {s3_path}")
add_file_to_bucket(settings.S3_BUCKET_NAME, assets_path, s3_path)

with zipfile.ZipFile(assets_path, "r") as zip_ref:
zip_ref.extractall(output_file_path)

orthophoto_path = os.path.join(
output_file_path, "odm_orthophoto", "odm_orthophoto.tif"
)
else:
log.warning(f"images.json not found in {output_file_path}")

log.info(f"Processing complete for project {dtm_project_id}")

if state and dtm_task_id and dtm_user_id:
# NOTE: This function uses a separate database connection pool because it is called by an internal server
# and doesn't rely on FastAPI's request context. This allows independent database access outside FastAPI's lifecycle.
pool = await database.get_db_connection_pool()
async with pool as pool_instance:
async with pool_instance.connection() as conn:
await task_logic.update_task_state(
db=conn,
project_id=dtm_project_id,
task_id=dtm_task_id,
user_id=dtm_user_id,
comment=message,
initial_state=state,
final_state=State.IMAGE_PROCESSING_FINISHED,
updated_at=timestamp(),
)
log.info(
f"Task {dtm_task_id} state updated to IMAGE_PROCESSING_FINISHED in the database."
)

s3_path_url = (
f"dtm-data/projects/{dtm_project_id}/{dtm_task_id}/assets.zip"
)
# update the task table
await project_logic.update_task_field(
conn, dtm_project_id, dtm_task_id, "assets_url", s3_path_url
)

if not os.path.exists(orthophoto_path):
log.error(f"Orthophoto not found at {orthophoto_path}")
raise FileNotFoundError("Orthophoto file is missing")

reproject_to_web_mercator(orthophoto_path, orthophoto_path)
s3_ortho_path = f"dtm-data/projects/{dtm_project_id}/{task_segment}orthophoto/odm_orthophoto.tif"
log.info(f"Uploading reprojected orthophoto to S3 path: {s3_ortho_path}")
add_file_to_bucket(settings.S3_BUCKET_NAME, orthophoto_path, s3_ortho_path)

images_json_path = os.path.join(output_file_path, "images.json")
if os.path.exists(images_json_path):
s3_images_json_path = (
f"dtm-data/projects/{dtm_project_id}/{task_segment}images.json"
)
log.info(f"Uploading images.json to S3 path: {s3_images_json_path}")
add_file_to_bucket(
settings.S3_BUCKET_NAME, images_json_path, s3_images_json_path
)
else:
log.warning(f"images.json not found in {output_file_path}")

log.info(f"Processing complete for project {dtm_project_id}")

if state and dtm_task_id and dtm_user_id:
# NOTE: This function uses a separate database connection pool because it is called by an internal server
# and doesn't rely on FastAPI's request context. This allows independent database access outside FastAPI's lifecycle.
pool = await database.get_db_connection_pool()
async with pool as pool_instance:
async with pool_instance.connection() as conn:
await task_logic.update_task_state(
db=conn,
project_id=dtm_project_id,
task_id=dtm_task_id,
user_id=dtm_user_id,
comment=message,
initial_state=state,
final_state=State.IMAGE_PROCESSING_FINISHED,
updated_at=timestamp(),
)
log.info(
f"Task {dtm_task_id} state updated to IMAGE_PROCESSING_FINISHED in the database."
)

s3_path_url = f"dtm-data/projects/{dtm_project_id}/{dtm_task_id}/assets.zip"
# update the task table
await project_logic.update_task_field(
conn, dtm_project_id, dtm_task_id, "assets_url", s3_path_url
)

status = ImageProcessingStatus.SUCCESS
if not dtm_task_id:
# Update the image processing status
pool = await database.get_db_connection_pool()
async with pool as pool_instance:
async with pool_instance.connection() as conn:
await project_logic.update_processing_status(
conn,
dtm_project_id,
(
ImageProcessingStatus.SUCCESS
if odm_status_code == 40
else ImageProcessingStatus.FAILED
),
conn, dtm_project_id, status
)

except Exception as e:
Expand Down
35 changes: 34 additions & 1 deletion src/backend/app/projects/project_schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,12 @@
)
from psycopg.rows import dict_row
from app.config import settings
from app.s3 import generate_static_url, get_presigned_url
from app.s3 import (
generate_static_url,
get_presigned_url,
get_assets_url_for_project,
get_orthophoto_url_for_project,
)


class CentroidOut(BaseModel):
Expand Down Expand Up @@ -219,6 +224,8 @@ class DbProject(BaseModel):
regulator_emails: Optional[List[EmailStr]] = None
regulator_approval_status: Optional[str] = None
image_processing_status: Optional[str] = None
assets_url: Optional[str] = None
orthophoto_url: Optional[str] = None
regulator_comment: Optional[str] = None
commenting_regulator_id: Optional[str] = None
author_id: Optional[str] = None
Expand Down Expand Up @@ -594,6 +601,8 @@ class ProjectInfo(BaseModel):
regulator_emails: Optional[List[EmailStr]] = None
regulator_approval_status: Optional[str] = None
image_processing_status: Optional[str] = None
assets_url: Optional[str] = None
orthophoto_url: Optional[str] = None
regulator_comment: Optional[str] = None
commenting_regulator_id: Optional[str] = None
author_name: Optional[str] = None
Expand All @@ -616,6 +625,30 @@ def set_image_url(cls, values):
values.image_url = get_presigned_url(settings.S3_BUCKET_NAME, image_dir, 5)
return values

@model_validator(mode="after")
def set_assets_url(cls, values):
"""Set assets_url before rendering the model."""
project_id = values.id
if project_id:
values.assets_url = (
get_assets_url_for_project(project_id)
if values.image_processing_status == "SUCCESS"
else None
)
return values

@model_validator(mode="after")
def set_orthophoto_url(cls, values):
"""Set orthophoto_url before rendering the model."""
project_id = values.id
if project_id:
values.orthophoto_url = (
get_orthophoto_url_for_project(project_id)
if values.image_processing_status == "SUCCESS"
else None
)
return values

@model_validator(mode="after")
def calculate_status(cls, values):
"""Set the project status based on task counts."""
Expand Down
20 changes: 20 additions & 0 deletions src/backend/app/s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,23 @@ def generate_static_url(bucket_name: str, s3_path: str):
protocol = "https" if is_secure else "http"
base_url = f"{protocol}://{minio_url}/{bucket_name}/"
return urljoin(base_url, s3_path)


def get_assets_url_for_project(project_id: str):
"""Get the assets URL for a project."""
project_assets_path = f"dtm-data/projects/{project_id}/assets.zip"
s3_download_root = settings.S3_DOWNLOAD_ROOT
if s3_download_root:
return urljoin(s3_download_root, project_assets_path)
return get_presigned_url(settings.S3_BUCKET_NAME, project_assets_path, 3)


def get_orthophoto_url_for_project(project_id: str):
"""Get the orthophoto URL for a project."""
project_orthophoto_path = (
f"dtm-data/projects/{project_id}/orthophoto/odm_orthophoto.tif"
)
s3_download_root = settings.S3_DOWNLOAD_ROOT
if s3_download_root:
return urljoin(s3_download_root, project_orthophoto_path)
return get_presigned_url(settings.S3_BUCKET_NAME, project_orthophoto_path, 3)
9 changes: 9 additions & 0 deletions src/frontend/src/assets/images/LandingPage/MobileView.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 6 additions & 2 deletions src/frontend/src/components/Dashboard/DashboardCard/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,12 @@ const DashboardCard = ({ title, count, active }: IDashboardCardProps) => {
>
<Image src={graphImage} />
<FlexColumn>
<h2>{count}</h2>
<p>{title}</p>
<h2 className="naxatw-text-4xl naxatw-font-semibold naxatw-text-red">
{count}
</h2>
<p className="naxatw-text-xs naxatw-font-medium naxatw-text-gray-600">
{title}
</p>
</FlexColumn>
</FlexRow>
);
Expand Down
Loading
Loading