Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingestion service that fetches data from endpoint at regular intervals #12

Open
7 tasks done
alex855k opened this issue Nov 20, 2018 · 2 comments · May be fixed by CSProjectsEAL/GraBID-Microservices#1
Open
7 tasks done
Assignees
Labels
invalid This doesn't seem right, either standards are not being followed and it needs to be updated priority: high Priority for issues which MUST BE implemented. type: infrastructure Task related to improvement of infrastructure. type: story Feature to be implemented based on a user story description
Milestone

Comments

@alex855k
Copy link
Contributor

alex855k commented Nov 20, 2018

📋 User story

As a system I want a service that can ingest data from an endpoint at regular intervals from the latest cached data entry point, so I can update the data in a cache database with the latest logs entries.

Description

To divide some responsibilities onto smaller service, it's necessary to have a small scale-able service, that is able to request data on a daily interval(3AM CET +0100) from Mono Solutions endpoint, ingest the data. And send it to the correct queue in at RabbitMQ. This will be done using a docker image of .net core.

Prerequisites

  • Find an appropiate .net core docker image and configure it to run with our source code data.

Acceptance Criteria

  • Create LogSource class with logic for which endpoints to connect to, the payload; credentials and format of the data. Additionally it needs to have a list of queue names where the payload will be send.
  • Create a crontab that will use LogSource to run the job, every 24th hour starting at 3AM.
  • Create the class will send the envelope object to the appropriate queues.
  • Log errors and actions
  • When the user story is done host it in a docker hub repository and put the repository data in /CSProjectEAL/GraBID main github repository in a seperate folder.
  • Unit tests.

Additional Comments

In it's current state you cant run solution before refiner service has been fixed. So hold off on the review.

child #6

@alex855k alex855k added the type: story Feature to be implemented based on a user story description label Nov 20, 2018
@alex855k alex855k changed the title As a system I want a service that can ingest data from an endpoint at regular intervals Ingestion service from an endpoint at regular intervals Nov 21, 2018
@alex855k alex855k changed the title Ingestion service from an endpoint at regular intervals Ingestion service that fetches data from endpoint at regular intervals Nov 21, 2018
@IulianBogdan
Copy link

Testing

@alex855k
Copy link
Contributor Author

Reopened

@elit0451 elit0451 added invalid This doesn't seem right, either standards are not being followed and it needs to be updated and removed review labels Dec 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid This doesn't seem right, either standards are not being followed and it needs to be updated priority: high Priority for issues which MUST BE implemented. type: infrastructure Task related to improvement of infrastructure. type: story Feature to be implemented based on a user story description
Projects
None yet
3 participants