Task: Convert GER Columns to Expected Types #298 #302
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
JulianForeman
commented
May 16, 2024
JulianForeman
commented
- Updating model & constants
tim738745
approved these changes
May 21, 2024
kuanfandevops
added a commit
that referenced
this pull request
Jun 6, 2024
* update to 0.2.0 * Superset changes (#74) * add sysdig (#76) * resolve log4j issue * Hydrogen Fleet Model, Migration & Loader (#75) * pipeline * pipeline * Initial commit * update pipeline * add s2i for frontend * add backend s2i * update s2i * update backend build * add minio * Changed migration and data to reflect zeva * update minio * update minio * update minio dc * update templates * update dev * Committing some missing files * Re-enabled metabase in docker-compose. Fixed the name for ICBC Table * Added more use of environment variables (#10) * Removed hardcoded variables for keycloak * Removed database url * Test Login (#12) * Removed hardcoded variables for keycloak * Removed database url * Temporary test for logging in * Add gunicorn and update backend template (#11) * Temp change to cthub-9 (#14) * Reorganizing so it loads our application (#16) * CTHUB-17: Set-up Login Page (#15) * CTHUB-17: Set-up Login Page * change hint to 'idir' * remove extra styles Co-authored-by: Richard <[email protected]> * ctub 37 - set up jest (#13) * -adds jest to package json under dev dependencies and scripts * -removes coverage files * -adds coverage to git ignore * Major Refactor for keycloak (#17) * update backend dc (#18) * Development Settings for Webpack (#19) * remove .s2i/bin (#21) * Fixed hot keyword in webpack (#22) * update ALLOWED_HOSTS (#23) * Changed how we refresh tokens (#25) * add gunicorn (#24) * update websocket port * update websocket port * Update all package with the latest versions and updated some deprecated functions. (#20) * Update all package with the latest versions and updated some depricated functions. * Added node version to fix docker issue * Auto-logout when refreshing the token doesn't work (#26) * Removed hardcoded variables for keycloak * Removed database url * Temporary test for logging in * Changed to cthub-9 * Reorganizing the code * Removed ReactKeycloakProvider * keycloak.json checks on host now * Added a check to toggle development settings for webpack * Moved hot into devServer * Removed home.js * Some cleanup * Changed how refreshing the token works * Minor comments * Fixed the automatic logout on failing to refresh the token * Fixed merge conflicts * Fixed merge conflicts * use artifactory (#27) * roll back (#29) * Metabase build (#28) * update frontend build (#30) * Allow environment variable for keycloak.json (#32) * CHUB-56 - model for ldv rebates (#31) * adds model for ldv rebates * -removes db comments * -adds migration for ldv rebates * -adds company (dealer) city to ldv rebates table * Cthub 59/60: Vin decoder function and saving decoded data to the DB table. (#33) * VIN Decoder Function * Created command file to run decoder function * CTHUB-60: VIN Decoder Function - Save Data * Fix migration conflict * Cthub 57 - ldv rebates service (#35) * -adds service for parsing ldv rebates * -creates service for parsing ldv rebates -migration to update fields so they fit everything from ldv rebates -adds pandas to requirements * -removes conflicting migration * -adds new migration for ldv rebates * Cthub 57 - linting (#36) * -adds service for parsing ldv rebates * -creates service for parsing ldv rebates -migration to update fields so they fit everything from ldv rebates -adds pandas to requirements * -removes conflicting migration * -adds new migration for ldv rebates * lints, updates comments in import_ldv_rebates file * Allowed multiple pages for parsing (#37) * Redirect fix (#38) * Fixed how redirect works * Removed test code * Removed unused code * Made router file so its a bit more organized * Added tracking of VIN for vin decoding information (#39) * Model and Import function for SUVI Data (#40) * Model and Import function for SUVI Data * Check if fleet or indivisual columns contain string then put fleet or individual into applicant type column, else leave it blank * Metabase 0.1.0 (#34) * Patroni 0.1.0 (#42) * update (#43) * update (#44) * Fix tags 0.1.0 (#45) * update config * create metabase service for backupcontainer * update metabase service name * CTHUB-81: Create models and migration - LCFS Credit Transfers (RLCF-017) (#41) * CTHUB-81: Create models and migration - LCFS Credit Transfers (RLCF-017) * Removed managers and other properties that are not needed for importing * Removed unnecessary declaration AUTH_USER_MODEL in settings Co-authored-by: Richard Tan <[email protected]> * cleanup (#46) * update clean up script * Cthub 92 - frontend for upload (#47) * -adds upload container * -adds file drop area files, -adds routes for upload, -adds dropzone to package.json -adds css files -adds file size utility * -lints -aligns everything properly including button and table -adds proptypes -fixes capitization -removes extra imports -removes unused props * -removes unused link syle * -removes unused svg style * CTHUB-107: Added Upload to Minio viewset (#48) * Added minio to docker * File cleanup. Installed minio for python * Added upload functionality * cthub 107 - backend upload (#49) * -adds material ui icons, updates to use icons, adds css * -adds backend viewset for uploading spreadsheet (grabs minio url and parses and inserts to database) -some linting * -passes dataset selected from frontend * Code cleanup (#50) * Added minio to docker * File cleanup. Installed minio for python * Added upload functionality * Code cleanup * Fixed merge conflicts * Further cleanup * Tekton 0.1.0 (#51) * CTHUB - Add dataset Table, Dropdown (#52) * -adds material ui icons, updates to use icons, adds css * -adds backend viewset for uploading spreadsheet (grabs minio url and parses and inserts to database) -some linting * -passes dataset selected from frontend * -adds model for datasets table to contain names of datasets that users will be able to import data into -adds fixtures for datasets table -adds serializer and viewset to retrieve a list of datasets -displays list of datasets in frontend on a dropdown * -fixes tabbing for "if done" * -adds suvi dataset * Setup prod (#54) * Cthub 113 - import suvi data (#53) * -imports function to parse suvi data * -adds alias for import ldv -initializes done variable * update hostname * Metabase resource 0.1.0 (#55) * update metabase * Update doc (#56) * add condition to build * Cthub 114 - replace or add to data (#58) * -adds ability to replace existing ldv file * -adds modal if records are being replaced * -makes replacedata proptypes string or bool because radio button is inconsistant * -renames alert file so it is less vague -adds title to alertdialog props * Charger Model and Migration (#59) * Charger model * The include * Migration running Documented process as the README was looking sad and lonely.😢 * Fixed duplicate column * Set condition from artifactory (#57) * Cthub 96/78 - import public charging data (#61) * -adds model for public charging * -adds migration for public charging * -adds public charging service -adds public charging fixture -adds public charging to upload viewset so service gets used if it is selected * -changes id for new fixture * Tekton4 0.1.0 (#63) * update assemble * CTHUB-31: Trial VIN Decoder from DataOne (#62) * Different vendor for the decode api * Created a copy of the old vin decoder * Added superset as part of docker compose (#64) * CTHUB-106 login redirect (#65) * -adds dashboard container and router -adds dashboard container to app.js that redirects to /upload -updates icbc router so it renders when user goes to /icbc * -removes extra empty line * moves dashboard switch to last place * -switches redirect to usehistory * Charger model * The include * Fixed duplicate column * No date fields * Removing migration that I'm re-writting * Fixed migration * Fixed migration again * Import is running I think * Working for real now * Documentation * Alternative method of accessing the spreadsheet. * Rolling back charger rebate migration * Charger Rebate table, model & migration (#67) * Charger model * The include * Fixed duplicate column * No date fields * Removing migration that I'm re-writting * Fixed migration * Fixed migration again * Import is running I think * Working for real now * Documentation * Alternative method of accessing the spreadsheet. * Rolling back charger rebate migration * use minio secret * CTHUB-95 - add option for uploading ev charging rebates (#66) * -adds charger rebates to dropdown -needs function still in the viewset (currently using import_ldv * -adds import charger rebates service to upload viewset * Fixed columns * Superset 0.1.0 (#68) * Fixed garbled columns (#69) * Charger model * The include * Fixed duplicate column * No date fields * Removing migration that I'm re-writting * Fixed migration * Fixed migration again * Import is running I think * Working for real now * Documentation * Alternative method of accessing the spreadsheet. * Rolling back charger rebate migration * Fixed columns * -adds migration for charger rebates model changes (#70) * Hydrogen model * Hydrogen loader function * Hydrogen loading script * Cthub 65/123 - hydrogen fueling model (#71) * -adds decoder to git ignore * Removing ignored files * -adds hydrogen fueling model * -adds migration for hydrogen fueling * -updates hydrogen fueling model to add boolean fields for bar 700 and 350 * -adds function to parse spreadsheet and insert to hydrogen fueling table -adds ability to call hydrogen fuelign and public charging parsers from command line -adds viewset for calling hydrogen fueling -fixture for adding hydrogen fueling to dropdown * Hydrogen migration and loading script Co-authored-by: Kuan Fan <[email protected]> Co-authored-by: Richard Tan <[email protected]> Co-authored-by: Kuan Fan <[email protected]> Co-authored-by: NavpreetGrewal <[email protected]> Co-authored-by: Emily <[email protected]> * update backup container (#77) * Update the value of minio endpoint (#78) * Dev resource 0.2.0 (#79) * Fixed the routes so they don't do multi-loads (#81) * Combined the superset containers into one (#82) * Data Fleets Table (#83) * Model for Data Fleets * Remaining columns for loader script * New fixtures * Data Fleets table migration and import * Arc Project Tracking model * ARC Project Tracking (#84) * Model for Data Fleets * Remaining columns for loader script * New fixtures * Data Fleets table migration and import * Arc Project Tracking model * ARC table migration/service/command * Data is loading * Lost this file for some reason * Scrap it model * Scrap it service * Data import working * The dropdown for all new data imports * Cthub 128 - whitelisted users (#85) * -adds model for whitelisted users * -adds migration for whitelisted users * filters for usernames matching current user and gives 403 if not found * -adds some frontend stuff for upload alert (unfinished) * -fixes issue where post error wasnt getting caught, is now getting caught and updating alert state so error message appears * -removes uneccessary line break * -adds alert if upload succeeds * -creates whitelisted users decorator, calls it for both minio and upload functions -updates frontend to match new response from whitelisted_users * -removes uneeded imports * -fixes migration conflict * Made secret key as an environment variable (#86) * Rebuild metabase on prod 0.2.0 (#88) * update for metabase * update metabase build contact * increate build resource * redeploy on prod * update for metabase * Update issue templates (#89) * Update issue templates (#90) Co-authored-by: Alex Zorkin <[email protected]> * update logoutHostName * release based pipeline (#91) * add repo name * remove dev pr based * update db secret name * add DB_HOST * Update README.md (#95) * Update build-release.yaml * chore: add logging to get new preferred_usernames (#98) * add instruction how to migrate to crunchydb (#116) * connect to crunchy (#126) * Update namespace name * fix: adds filllna to 0 for numeric columns (#131) * Fix: emily- cthub 122 get upload utility working (#133) * fix: typo in hydrogen fleets model and some mistakes in script * chore: adds axis keyword to all dataframe.drop to avoid future errors * Feat: Cthub 146 - upload feedback (#147) * -fix: removes keycloak, redis, and superset from dockerfile as we will be using the keycloak shared realm * chore: adds some info to readme, removes keycloak references * feat: adds error handling to upload and 3 scripts. Will show specific messages depending on error. Still in progress * fix: forgot to move this try outside of the iteration, need to double check with PO * -fix: adds keycloak.env file with shared realm url * increase memory for frontend build * feat: adds error handling for remainder of scripts (#153) * update approvers * add two new env vars for backend doc uploads * replace minio with object storeage * cthub-167 chore: replaces whitelisted users table with user, permissions, and user_permissions tables. updated code to reflect changes, only those with 'uploader' permission can upload datasets, using idir to identify users (#171) * Feat: Download Dataset Spreadsheet #150 (#176) * Adding endpoint and logic for downloading formatted data spreadhseet for uploader * Removing missed console.log * update minioBucketName * fix: 169 - get uploader working on test (#184) * update backend host name * fix: 150 - add xlsxwriter (#189) * Feat Cthub 168 PARTIAL admin page (#193) * feat: adds backend work for retrieving current user and user list * feat: adds (UNSTYLED) frontend work for listing users and permissions * chore: removes idir serializermethodField * feat: adds styling to user admin * CTHUB - Frontend styling for upload and user permissions page (#196) * feat: frontend changes for upload and user admin * chore: removes react Fragment and replaces with grid container, adds grey background to file upload grid * chore: adds responsiveness to permissions * fix: aligns names vertically in row (#202) * add new ci pipeline (#206) * update folder location * update SUFFIX * update trigger for frontend build * feat: 201 - Logout button + cleanup (#214) * feat: CTHUB 198 - save/update users (#218) * chore: -backend: updates users moved into new service -frontend: adds immer for setting state, uses original users array rather than creating new one * chore: moves transaction.atomic to service * feat CTHUB 212 : clear upload message (#219) * feat: removes upload message when new dataset is selected or file is dropped * chore: sets alert to false instead of empty string * feat: 197 - header and footer components (#216) * feat: 197 - header and footer components * remove header and footer from login page * feat: 210 - page refinements + cleanup (#223) * chore: fixes padding on app-body, disables adding empty string for new users, adds tooltip to disabled add user button, clears response message from adding uses/updating permissions if user box is typed into or checkbox clicked (#225) * update frontend build * feat: 210 - do not display "download spreadsheet" when no dataset selected (#227) * list the build foders * update to docker strategy * update Dockerfile-Openshift file name * udpate docker build * upgrade npm * add yes to insall webpack-cli * install webpack client globally * move webpack to dev dependencies * try to fin modules * install webpack as dev dependency * update NODE_PATH * set ENV NODE_PATH * install copy-webpack-plugin * install all modules * install all dev dependencies with -g * move NODE_ENV right before run build * copy the right folder * remove tekton * Backend build 0.2.0 (#251) * disable django migration * open the CD * feat: CTHUB 248 -Delete user (#253) * feat: turns x icon into buttton, adds empty handler function * feat: adds functionality for deleting users, reuses and slightly adjusts alert dialogue for confirming, adds checks on frontend and backend for ensuring user does not delete their own self from user table * fix: adds missed backend check * chore: makes minor changes to handle delelete user and delete viewset * refactor AlertDialog plus a few small changes --------- Co-authored-by: tim738745 <[email protected]> * update backend bind port 8080 * update to use node 16 * not to install npm latest * rebuild frontend using create-react-app (#259) * add --force for npm or it will fail * use node 20 * update oc-login * add noCache forcePull for frontend build * only build backend * updtae frontend docker build * update docker build * update build folder * open the backend build * Add Superset (#123) * chore: 164 - metabase models on git (#256) * update workflow (#262) * only apply on prod * update frontend build * tag after test approval * Feat: Uploader & Error Refactor #166 (#188) * Uploader refactor to just use 1 function for all services. WIP working state for arc_project_tracking * New service files * Moving constants to their own file, changing dictionary to enums, seperating import function into function steps, updating logic to support new data formats, added a couple helper functions * Cleaning up logic, adding working upload state for data fleets & charger rebates * Working state for all uploader services refactored to use general import function * Fixing field type mapping and adding type casts & error exceptions for field types * Individual cell error handling with response message formatting and code cleanup * Making uploaded data store the idir of who uploaded it & updating suvi model * Updating README with instructions for creating a user account locally. * Excluding a list of datasets so they can be re-implemented as we get updated models for them * Moving dataset config to constants, updating constants import to avoid namespace clashing, updated how alerts are displayed to use existing functionality * Removing dataset from dropdown to be added back later (#265) * test new pip * login to test * read pre-release number from backend * update dev namespace * open deployment on Test * correct tagging * login before deploying on test * task: 257 - add guidance regarding package updates (#260) * update workflow name * show PRE_RELEASE * add pre-release to issue title for apprival * add pre-release to issue title for apprival * add prod pipeline * Added more styling options to loading component, loading component now displays instead of the upload button on click until the upload is finished. Added state variable for page refreshes instead of using loading. (#264) * add image cleanup workflow * login first * update image cleanup messages * coeect var usage * add continue-on-error: true * echo message before delete tool images * comment tools env image cleanup * update cleanuo script * update cleanup script * update test * further clean up pods, builds and buildconfigs * except crunchy pods * ignore cleanup errors * feat: updates radio text and order (#274) * Clearing text box on button click & sorting list when adding users (#275) * chore: 217 - install formatters and do initial format (#277) * add formatters * format python code with black * format js code with prettier * Task: Backend Tests for Uploader #165 (#276) * Writing backend tests for the uploader service * Another test * Successful upload test * task: backend tests for user permissions (#279) * chore: adds tests for user permissions * removes extra space * cleanup knps: * Cleanup 0.2.0 (#286) * Adding test for changing a user's permissions (#290) * Task: Go Electric Rebates Uploader Update #283 (#287) * Updating SUVI model to new GER model. Updating logic for null values to avoid incorrect error messages * Adding in new model file and migration * Removing new dataset from incomplete dataset list * feat: 284 and 285 - vin decoding (#289) * feat: 284 and 285 - vin decoding * update migration * add db table comments * Updating model & constants (#302) * fix: CTHUB 297 Adds success/error message to icbc vin upload (#303) * fix: adds messages to icbc vins upload in import data function, adds basic error messaging to frontend in case errors don't have our standard format * feat: adds check for csv on backend fix: updates frontend to catch keycloak logout and display message to user when attempting upload * Task queue 0.2.0 (#309) \ * add taskq build * cleanup * update task queue image tag * feat: adds redirect to login page if user attempts any axios call after timing out (#308) * feat: disables save button on user permissions until any permission is changed (#306) * update cleanup cron * fix: 285 - update default chunksize for processing vin files (#313) * tag taskq image for test * add task queue to prod pipeline * update knps --------- Co-authored-by: Jamie Popkin <[email protected]> Co-authored-by: Richard Tan <[email protected]> Co-authored-by: NavpreetGrewal <[email protected]> Co-authored-by: Emily <[email protected]> Co-authored-by: kcabhar <[email protected]> Co-authored-by: Alex Zorkin <[email protected]> Co-authored-by: tim738745 <[email protected]> Co-authored-by: JulianForeman <[email protected]> Co-authored-by: tim738745 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.