Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARPA Exporter - Add python business-logic for building zipfile and emailing user #3934

Merged
merged 101 commits into from
Feb 10, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
31056c7
Add arpa-exporter python project
TylerHendrickson Jan 17, 2025
829745f
Add Dockerfile for arpa-exporter workers
TylerHendrickson Jan 17, 2025
7c95035
Fix programming issues in worker
TylerHendrickson Jan 17, 2025
5738acc
Target py3.13 and add dev dependencies
TylerHendrickson Jan 17, 2025
0541adb
Multistage dockerfile for python exporter
TylerHendrickson Jan 17, 2025
cfe0ed2
Add python gitignore rules
TylerHendrickson Jan 17, 2025
963606e
Add pip ecosystem to Dependabot config
TylerHendrickson Jan 17, 2025
11c6258
Fix gitignore py rules
TylerHendrickson Jan 17, 2025
0f5153d
Rename dockerfile
TylerHendrickson Jan 17, 2025
cc8f118
Build arpa-exporter docker image in GHA
TylerHendrickson Jan 17, 2025
4d50a77
Update CI to include arpa-exporter QA
TylerHendrickson Jan 17, 2025
7290069
Add arpa-exporter image build to deploy steps
TylerHendrickson Jan 17, 2025
cb8fe32
Configure datadog logs injection
TylerHendrickson Jan 18, 2025
801bbb1
(formatting) Configure datadog logs injection
TylerHendrickson Jan 18, 2025
e03ddf3
Remove manual datadog log injection
TylerHendrickson Jan 18, 2025
2be3151
Merge branch 'main' into tsh/poc-python-sqs-worker
TylerHendrickson Jan 23, 2025
2a9c53f
Make postgres optional in sqs_consumer_task module
TylerHendrickson Jan 24, 2025
05cd392
Add arpa_exporter tf module instancce
TylerHendrickson Jan 24, 2025
f39a7b8
Add arpa_exporter_exporter_image_tag tf var
TylerHendrickson Jan 24, 2025
869d987
Add healthcheck definition for datadog agent ECS containers
TylerHendrickson Jan 25, 2025
5707fd4
Make SQS consumer task command optional (allow default to docker imag…
TylerHendrickson Jan 25, 2025
6a4504b
Bump poetry.lock (via `poetry update`)
TylerHendrickson Jan 25, 2025
5c21785
Make python log level configurable via env var
TylerHendrickson Jan 25, 2025
0fc909c
Add boto3-stubs to py dev dependencies
TylerHendrickson Jan 25, 2025
2293973
feat: add schema for messages and source-paths. add metadata-file to …
as1729 Jan 28, 2025
8b98a98
fix: revert naming to stay consistent
as1729 Jan 28, 2025
039c098
feat: add ability to queue python task from JS module
as1729 Jan 28, 2025
871a6f1
feat: add placeholder to send email
as1729 Jan 28, 2025
6919f37
Merge branch 'main' into tsh/poc-python-sqs-worker
TylerHendrickson Jan 29, 2025
403b3d7
Fix typo in workflow call param
TylerHendrickson Jan 29, 2025
aad019f
fine-tuning pytest configuration
TylerHendrickson Jan 29, 2025
f48a4f1
Refactor ShutdownHandler to new module
TylerHendrickson Jan 29, 2025
2e51b7a
Set dummy env var during tests
TylerHendrickson Jan 30, 2025
3f19a27
Add tests for src.lib.shutdown_handler
TylerHendrickson Jan 30, 2025
ecf5fad
Fix missing pytest coverage report
TylerHendrickson Jan 30, 2025
b901d53
Fix missing Python coverage reports in QA summaries
TylerHendrickson Jan 30, 2025
e710ae2
Ensure safe use of multiprocessing in tests
TylerHendrickson Jan 30, 2025
2cd91e7
Ignore all python coverage files
TylerHendrickson Jan 30, 2025
2b7de6c
Add recommended vscode extensions for python
TylerHendrickson Jan 30, 2025
697679c
Add vscode project settings for python
TylerHendrickson Jan 30, 2025
fdb7aad
Handle multiline markdown report output
TylerHendrickson Jan 30, 2025
022e7ba
Handle multiline markdown report output
TylerHendrickson Jan 30, 2025
df390f3
Remove unnecessary step output
TylerHendrickson Jan 30, 2025
76a78cf
Fix terraform validation failure
TylerHendrickson Jan 30, 2025
016258f
Bump LOG_LEVEL to INFO in arpa-exporter tasks
TylerHendrickson Jan 30, 2025
011dd02
Fix arpa-exporter image name
TylerHendrickson Jan 30, 2025
99a7268
Provide arpa_exporter_image_tag TF var in plans
TylerHendrickson Jan 30, 2025
1cebd51
Provide full image identifiers to terraform-plan workflow
TylerHendrickson Jan 31, 2025
c7c72dd
Include arpa-exporter in TF plans for staging
TylerHendrickson Jan 31, 2025
4655140
Specify docker repo for arpa-exporter consumer tasks
TylerHendrickson Jan 31, 2025
d616f97
Decouple arpa-exporter website feature flag from consumer task provis…
TylerHendrickson Jan 31, 2025
e94082e
chore: merge latest changes
as1729 Feb 3, 2025
39d645d
Merge branch 'main' into tsh/poc-python-sqs-worker
TylerHendrickson Feb 3, 2025
a0f7845
Merge branch 'tsh/poc-python-sqs-worker' into as/add-unit-tests-for-z…
as1729 Feb 3, 2025
710f910
feat: add ability to send emails. move metadata csv to s3.
as1729 Feb 4, 2025
7d48fa0
chore: add moto3 dependency to mock aws services in tests
as1729 Feb 4, 2025
4e498ce
chore: add pydantic for data validation and chevron for emails
as1729 Feb 4, 2025
e79fab5
feat: add test coverage for email functionality
as1729 Feb 4, 2025
6c0ed60
feat: tests for email module
as1729 Feb 4, 2025
c0092ee
fix: update method to load csv data correctly and add tests
as1729 Feb 4, 2025
6b8e665
chore: remove duplicate code
as1729 Feb 4, 2025
51c94b9
chore: add placeholder test for build_zip function
as1729 Feb 5, 2025
a7f1be3
chore: merge changes from main
as1729 Feb 5, 2025
74af336
fix: remove unnecessary file
as1729 Feb 6, 2025
ca03cce
fix: add eof empty line
as1729 Feb 6, 2025
b40d44d
fix: revert package versions to match main branch
as1729 Feb 6, 2025
bc0417f
fix: unused imports
as1729 Feb 6, 2025
94a6340
fix: remove explicit type for ses
as1729 Feb 6, 2025
8526664
fix: type issue. ensures function always receives a string for email_…
as1729 Feb 6, 2025
cbe602e
fix: remove unused import
as1729 Feb 6, 2025
ee7ef3d
fix: specify region
as1729 Feb 6, 2025
b7249a4
fix: add sample upload files for testing
as1729 Feb 6, 2025
1c415ec
chore: add meaningful assertions to build_zip
as1729 Feb 6, 2025
c611c23
chore: add additional assertions to build_zip
as1729 Feb 6, 2025
cb0a7ef
Ensures we're sending the user an email with the appropriate metadata…
as1729 Feb 6, 2025
53edb5b
makes the URL simple as there's no need to pass these variables
as1729 Feb 6, 2025
5454bb8
removing JS code from this PR and going to open separately
as1729 Feb 6, 2025
155cd62
fix: remove unused imports
as1729 Feb 6, 2025
29cafb8
Merge branch 'main' into as/add-unit-tests-for-zip-functionality
as1729 Feb 6, 2025
580a739
Run `ruff format`
TylerHendrickson Feb 6, 2025
6079b4a
Install boto3-stubs[essential] dev dependency
TylerHendrickson Feb 6, 2025
75f5c97
Organize worker module imports
TylerHendrickson Feb 6, 2025
24cbca7
Narrow try/except to preserve context in exception logging
TylerHendrickson Feb 6, 2025
f5d8033
Add boto3 type annotations to worker module signatures
TylerHendrickson Feb 6, 2025
b89fff2
Refactor to simplify call chain dependencies
TylerHendrickson Feb 6, 2025
8d88e2a
Remove log-based test assertions
TylerHendrickson Feb 6, 2025
be75b2d
Account for no modifications to zip
TylerHendrickson Feb 8, 2025
6eb1ae0
Improve separation of concerns and guard edge cases
TylerHendrickson Feb 8, 2025
744cb2e
Add moto test setup and fixtures
TylerHendrickson Feb 8, 2025
a4989ae
Expand test coverage for src.worker module
TylerHendrickson Feb 8, 2025
6aca48e
Reorganize fixture data files
TylerHendrickson Feb 8, 2025
7465cd8
Expand test coverage for src.lib.email module
TylerHendrickson Feb 8, 2025
85b95d4
Fixed/updated type annotations (mypy passing)
TylerHendrickson Feb 8, 2025
d626f63
Add docstrings to `src.worker` module
TylerHendrickson Feb 8, 2025
cb732ad
Fill out remaining docstrings
TylerHendrickson Feb 10, 2025
e49b628
Add follow-up TODO annotations
TylerHendrickson Feb 10, 2025
d2fd23f
Enable mypy checking on `worker.main()` function
TylerHendrickson Feb 10, 2025
efa4ca5
Add docstrings for `src.lib.email` module functions
TylerHendrickson Feb 10, 2025
539e093
Add type-checking dependency for chevron
TylerHendrickson Feb 10, 2025
1e10833
Merge remote-tracking branch 'origin' into as/add-unit-tests-for-zip-…
TylerHendrickson Feb 10, 2025
3327b2a
Regenerate `poetry.lock`
TylerHendrickson Feb 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,4 @@ __pycache__/
arpa-exporter/.coverage
arpa-exporter/.coverage.*
arpa-exporter/htmlcov/
arpa-exporter/src/data/*
898 changes: 878 additions & 20 deletions arpa-exporter/poetry.lock

Large diffs are not rendered by default.

7 changes: 5 additions & 2 deletions arpa-exporter/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,19 @@ python = "^3.13"
boto3 = "^1.36.16"
ddtrace = "^2.19.0"
structlog = "^25.1.0"

pydantic = "^2.10.3"
chevron="^0.14.0"

[tool.poetry.group.dev.dependencies]
pytest = "^8.3.4"
ruff = "^0.9.5"
ipython = "^8.32.0"
pytest-cov = "^6.0.0"
mypy = "^1.15.0"
boto3-stubs = "^1.36.16"
boto3-stubs = {extras = ["essential", "ses"], version = "^1.36.14"}
pytest-timeout = "^2.3.1"
moto = "^5.0.28"
types-chevron = "^0.14.2.20250103"

[build-system]
requires = ["poetry-core"]
Expand Down
120 changes: 120 additions & 0 deletions arpa-exporter/src/lib/email.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
"""
Email template structure:

```
<<Level 1: base.html>>
(Common parameterized elements go here)
<<Level 2: formatted_body.html>>
Title param value
<<Level 3: inner body text (or more HTML)>>
```

Each inner level is rendered and provided as a parameter ("partial") of its outer level.
Arguably, levels 1 & 2 should be combined in the arpa-exporter project, but we are
deferring to conventions established elsewhere.
"""

from __future__ import annotations

import os
import typing
import urllib

import chevron

if typing.TYPE_CHECKING: # pragma: nocover
from mypy_boto3_ses import SESClient

CHARSET = "UTF-8"
TEMPLATES_DIR = os.path.abspath("src/static/email_templates")
NOTIFICATIONS_EMAIL = os.environ["NOTIFICATIONS_EMAIL"]


def generate_email(
download_url: str,
) -> typing.Tuple[str, str, str]:
"""Generates content to send for a notification email, informing the recipient
that a zip file is ready to download.

Args:
download_url: The URL where the downloadable zip file is hosted.
This should generally be a presigned S3 object URL.

Returns:
A 3-tuple containing (email_html, email_plaintext, subject), where:
email_html: Generated HTML content for the email body
email_plaintext: Generated alternative plaintext content for email
clients that do not support HTML
subject: Subject line to use when sending the email
"""
# Level 3:
with open(os.path.join(TEMPLATES_DIR, "messages", "full_file_export.html")) as tpl:
message_html = chevron.render(tpl, {"url": urllib.parse.quote(download_url)})

# Level 2:
with open(os.path.join(TEMPLATES_DIR, "formatted_body.html")) as tpl:
formatted_body_html = chevron.render(
tpl, {"body_title": "Hello,", "body_detail": message_html}
)

# Level 1:
with open(os.path.join(TEMPLATES_DIR, "base.html")) as tpl:
email_html = chevron.render(
tpl,
{
"tool_name": "ARPA Reporter",
"title": "Full File Export",
"usdr_logo_url": "https://grants.usdigitalresponse.org/usdr_logo_transparent.png",
},
partials_dict={"email_body": formatted_body_html},
)

# Alternate plaintext content
with open(os.path.join(TEMPLATES_DIR, "messages", "full_file_export.txt")) as tpl:
email_plaintext = chevron.render(tpl, {"url": download_url})

subject = "USDR Full File Export"
return email_html, email_plaintext, subject


def send_email(
email_client: SESClient,
dest_email: str,
email_html: str,
email_text: str,
subject: str,
) -> str:
"""Sends an email to a single recipient via SES.

Args:
email_client: SES client for sending the email.
dest_email: Email address of the intended recipient (a ``TO`` destination address).
email_html: Rich HTML content generated for the email body.
email_text: Alternative plaintext content generated for the email body
(for compatibility with recipient email clients that do not support HTML).
subject: Subject line for the outgoing email.

Returns:
The SES message ID generated for the outgoing email.
"""
response = email_client.send_email(
Source=NOTIFICATIONS_EMAIL,
Destination={"ToAddresses": [dest_email]},
Message={
"Subject": {
"Charset": CHARSET,
"Data": subject,
},
"Body": {
"Html": {
"Charset": CHARSET,
"Data": email_html,
},
"Text": {
"Charset": CHARSET,
"Data": email_text,
},
},
},
)
return response["MessageId"]
2 changes: 1 addition & 1 deletion arpa-exporter/src/lib/logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import structlog


LOG_LEVEL = getattr(logging, os.environ.get("LOG_LEVEL", "INFO").upper())

shared_processors: List[Callable] = [
Expand Down Expand Up @@ -42,6 +41,7 @@


def get_logger(*args: str, **kwargs: str) -> structlog.stdlib.BoundLogger:
"""Convenience wrapper for ``structlog.get_logger()`` function."""
return structlog.get_logger(*args, **kwargs)


Expand Down
Loading
Loading