Load data into Vercel using GitHub Actions #161
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
.github/workflows/etl_to_neon.yml
Outdated
There was a problem hiding this comment.
While in development, the workflow is only triggered once a week - to save on data transfer costs. The free limit of data transfer is 5 GB/month.
There was a problem hiding this comment.
Is this a neon or Github Actions limit? Just curious.
There was a problem hiding this comment.
This is a neon limit
There was a problem hiding this comment.
Got a mypy error: Incompatible types in assignment (expression has type "dict[str, tuple[float, float] | None]", variable has type "dict[str, tuple[float, float]]"). Wasn't sure if I can safely change the type, so temporarily silenced the warning.
There was a problem hiding this comment.
The duplicate check relied on the auto increment PK, which prevented duplicates from being detected. Replaced the autoincrement key with a composite PK
There was a problem hiding this comment.
Added a UNIQUE constraint to the Property_address field, so that it can be used to prevent duplicates in the Soft_story_properties table.
| @@ -3,7 +3,7 @@ | |||
| from backend.api.config import settings | |||
|
|
|||
| # Set up the database engine using settings | |||
| engine = create_engine(settings.database_url_sqlalchemy, echo=True) | |||
| engine = create_engine(settings.neon_url, echo=True) | |||
There was a problem hiding this comment.
Are there consequences to changing this script to connect to the Neon DB as opposed to the local DB? As in, if I want to test my local DB, do I need to change this line back to connect to database_url_sqlalchemy?
There was a problem hiding this comment.
Yes, you'd have to change this line back to test locally. If this is a problem, I can add an if-clause that would switch the url based on the environment
There was a problem hiding this comment.
Ok, I think it’s fine for now. I would probably prefer putting more data in the test DB.
Description
ST_Simplifywith the tolerance level of 0.0001Closes #150, addresses but doesn't close #154
Type of changes
Testing
How to test
Manually trigger the
ETL to Neonworkflow.