Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advent day 2: Clean up #265

Open
4 of 7 tasks
ErnestaP opened this issue Dec 19, 2023 · 0 comments
Open
4 of 7 tasks

Advent day 2: Clean up #265

ErnestaP opened this issue Dec 19, 2023 · 0 comments

Comments

@ErnestaP
Copy link

ErnestaP commented Dec 19, 2023

Airflow workflows:
Please write your name next to the task, so that people would not overlap with the same issue:

  • 1. Not all links in documentation work properly
  • 2. Some tests have extensive "expected" values, it would be good to move them to another file
  • 3. Align harvesting DAGs schedules with production (aps.py, elsevier_pull_ftp.py, hindawi.py, dag_pull_ftp.py, oup_pull_ftp.py, dag_pull_ftp.py)
  • 4. Change the names of harvesting DAGs and the file names they are in, that would reflect their source and publisher correctly: for example, IOP DAG is dag_pull_ftp, should be renamed to iop_pull_sftp.
  • IOP, Springer, Elsevier - SFTP
  • OUP - FTP
  • Hindawi, APS - API
  • 5. Some value in parsers are just put in the arrays like this: extra_function=lambda x: [int(x)]. It makes more sense to do this in generic parsing because there we are forming the correct data structures. It will be quite a big task because changes have to be reflected to in tests as well
  • 6. Remove types: somewhere they are used, somewhere not
  • 7. Make code more Pythonic: we have Java-style code (interfaces). Rewrite them to abstract classes, maybe even with concrete implementation in order to avoid boilerplate code (for example the methods get_by_id, delete_all from IRepository have the same implementation in all publishers' Repository classes )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant