#50: Create or adjust endpoint for accepting and saving checkpoint data #74

TebaleloS · 2023-09-18T11:24:05Z

This PR implement a functionality of saving data into the postgreSQL database using slickpg.
Added Fa-db library for communication with the database.
impelemented a checkpoint controller as PrototypeController and injected DatabaseService for communicating with the db.
Changed checkpoint service to DatabaseService.
Defined postgres engine in the application.properties in the server package.
Added postgres configuration in the AtumConfig.
Created and instantiated runs class for the server to be able to communicate with the sql function that create a checkpoint in the database.
Added the package for access provider and implemented access providers.
Closes Create (adjust) endpoint for accepting and saving checkpoint data #50

github-actions · 2023-09-18T11:26:38Z

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

There is no coverage information present for the Files changed

Total Project Coverage	83.26%	🍏

github-actions · 2023-09-18T11:26:39Z

JaCoCo server module code coverage report - scala 2.12.12

File	Coverage [21.32%]	❌
AtumConfig.scala	100%	🍏
TestService.scala	100%	🍏
PostgresAccessProvider.scala	95.45%	🍏
CheckpointController.scala	42.86%	❌
DatabaseService.scala	42.11%	❌
Runs.scala	22.08%	❌
BaseApiService.scala	10.91%	❌
ControlMeasureService.scala	9.85%	❌
BaseApiController.scala	8.11%	❌
ControlMeasureController.scala	4.76%	❌
Checkpoint.scala	2.66%	❌

Total Project Coverage	18.31%	❌

…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data # Conflicts: # server/src/main/scala/za/co/absa/atum/web/model/Checkpoint.scala

…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data # Conflicts: # project/plugins.sbt

…accepting-and-saving-checkpoint-data

miroslavpojer

pull

run test (sbt clean test)

BaseApiController.scala - problem with imports - failing tests
Solution proposal (tested):

import jakarta.servlet.http.HttpServletRequest   ---- remove
import javax.servlet.http.HttpServletRequest      ---- add

ControlMeasureController.scala
Solution proposal (tested):

import jakarta.servlet.http.HttpServletRequest   ---- remove
import javax.servlet.http.HttpServletRequest      ---- add

failing test "TestServiceSpec"

An exception or error caused a run to abort: Failed to load ApplicationContext 
java.lang.IllegalStateException: Failed to load ApplicationContext

code coverage - visible very low code coverage of changed/touched scala files.

miroslavpojer · 2023-10-31T11:13:08Z

Is here plan to focus the low code coverage of changed/touched classes?

…ing as other projects

benedeki · 2023-11-01T08:09:39Z

server/src/main/scala/za/co/absa/atum/web/api/AtumConfig.scala

  private val config = ConfigFactory.load("application.properties")

+  override def dbConfig: Config = config.getConfig("postgres")


benedeki · 2023-11-01T08:12:41Z

server/src/main/scala/za/co/absa/atum/web/api/provider/PostgresAccessProvider.scala

+
+  private def databaseConfig: Config = {
+    val baseConfig = AtumConfig.dbConfig
+    Map.empty.foldLeft(baseConfig) { case (acc, (configPath, configVal)) =>


What's the point of this code?
It's a foldLeft on an empty Map, so no op.

I added a TODO comment as promised & created a new ticket for this

…accepting-and-saving-checkpoint-data

… problematic

…ts in Scala that goes to the DB

miroslavpojer

miroslavpojer · 2023-11-03T06:36:06Z

server/src/main/scala/za/co/absa/atum/web/model/Checkpoint.scala

+class CheckpointStatusTypeRef extends TypeReference[CheckpointStatus.type]
+
+
+// To check By Forgiveness


Forgotten comment?

miroslavpojer · 2023-11-03T06:49:18Z

I am missing unit tests for at least new code. Is their implementation planned?

… fnc

miroslavpojer · 2023-11-06T07:18:23Z

After last commit build is failing:

08:14:07.848 [pool-8-thread-3-ScalaTest-running-MeasureTest] DEBUG io.netty.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
[error] /Users/ab024ll/repos/absa/branches/atum-service_74/server/src/main/scala/za/co/absa/atum/web/api/service/DatabaseService.scala:36:46: missing parameter type
[error]     wroteCheckpointFuture.thenApply[Boolean](_ => TRUE)
[error]                                              ^
[error] one error found

server/src/main/scala/za/co/absa/atum/web/api/database/Runs.scala

benedeki

code reviewed
pulled
built
run

lsulak · 2023-11-06T16:30:38Z

@miroslavpojer tests will be part of DB test PR, it would be too big here; otherwise, no need to test model/ and server/ so far is actually not doing too much - no crazy data transformations etc, mostly just calling DB

TebaleloS added 4 commits September 14, 2023 10:06

Fixes #50

6b94980

Fixes #50

3c2ef9c

Fixes #50

6453d64

Fixes #50

2ddebaf

benedeki added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Sep 18, 2023

benedeki assigned TebaleloS Sep 18, 2023

Fixed #50 - Added liscense

7316188

lsulak changed the base branch from master to feature/37-dao-between-agent-server-adjustments September 21, 2023 10:44

lsulak and others added 5 commits September 21, 2023 12:46

Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…

6d6c71a

…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data # Conflicts: # server/src/main/scala/za/co/absa/atum/web/model/Checkpoint.scala

#50: post-merge fixes

55fe01f

Fixed #50 - added db connection class

2314e82

Merge remote-tracking branch 'origin/feature/37-dao-between-agent-ser…

ec9832a

…ver-adjustments' into feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data # Conflicts: # project/plugins.sbt

Fixed #50 - refactored the endpoint to use JDBC drivers

798e54f

TebaleloS marked this pull request as ready for review September 28, 2023 10:30

TebaleloS requested review from benedeki, lsulak, Zejnilovic and dk1844 as code owners September 28, 2023 10:30

Base automatically changed from feature/37-dao-between-agent-server-adjustments to master October 6, 2023 15:55

TebaleloS added 5 commits October 7, 2023 22:26

Fixed #50 - refactored

4aaba46

Fixed #50 - refactored

b5a8677

Fixed #50 - Loaded db configs

0ca557d

Fixes #50 - fixed conflicts

12b514f

Fixed #50 - merged master and fixed the conflicts

1238951

TebaleloS force-pushed the feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data branch from 32cf9e7 to 1238951 Compare October 10, 2023 12:27

TebaleloS added 3 commits October 13, 2023 14:23

Fixed #50 - merged master and fixed the conflict

c418ba2

Fixed #50 - Added slickpg dependancy

e19061b

Fixed #50 - Changed slickpg dependancy version.

f702170

Resolved comments

e710558

TebaleloS force-pushed the feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data branch from 3581485 to e710558 Compare October 31, 2023 06:43

TebaleloS and others added 2 commits October 31, 2023 08:45

removed out directory

f2fb62b

Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…

4147167

…accepting-and-saving-checkpoint-data

miroslavpojer reviewed Oct 31, 2023

View reviewed changes

lsulak added 4 commits October 31, 2023 16:11

#50: fixing the build, correcting dependencies and using the same Spr…

6eb5eb0

…ing as other projects

#50: adding handy util for options

248a0a5

#50: fixing the build and simplifying the server code

8be7ba6

#50: adding missing licence header

c5a269f

benedeki reviewed Nov 1, 2023

View reviewed changes

lsulak added 6 commits November 1, 2023 15:35

Merge branch 'master' into feature/#50-Create-or-adjust-endpoint-for-…

5e1dc3a

…accepting-and-saving-checkpoint-data

#50: todo comment

4845372

#50: some post-review tweaks

8680c6a

#50: fixing Run

d9e2a41

#50: temporarily commentint out TEXT[][] fields as they are currently…

70e6e02

… problematic

#50: hopefully fixing build - serialization of arrays and JSONB objec…

f8ca2e8

…ts in Scala that goes to the DB

miroslavpojer reviewed Nov 3, 2023

View reviewed changes

lsulak added 4 commits November 3, 2023 12:21

#50: adding possibility to serialize and deserialize java offset time

8484075

#50: tiny refactoring

1940f72

#50: proper serDe in server and actually calling the write checkpoint…

8905667

… fnc

#50: making the db func call callable now :)

77912e0

benedeki reviewed Nov 6, 2023

View reviewed changes

server/src/main/scala/za/co/absa/atum/web/api/database/Runs.scala Show resolved Hide resolved

#50: fixing the build and changing the endpoint slightly

98a8121

benedeki approved these changes Nov 6, 2023

View reviewed changes

lsulak merged commit c8ba0c6 into master Nov 6, 2023
5 of 6 checks passed

lsulak deleted the feature/#50-Create-or-adjust-endpoint-for-accepting-and-saving-checkpoint-data branch November 6, 2023 16:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#50: Create or adjust endpoint for accepting and saving checkpoint data #74

#50: Create or adjust endpoint for accepting and saving checkpoint data #74

TebaleloS commented Sep 18, 2023 •

edited

Loading

github-actions bot commented Sep 18, 2023 •

edited

Loading

github-actions bot commented Sep 18, 2023 •

edited

Loading

miroslavpojer left a comment •

edited

Loading

miroslavpojer commented Oct 31, 2023

benedeki Nov 1, 2023

benedeki Nov 1, 2023

lsulak Nov 1, 2023 •

edited

Loading

miroslavpojer left a comment •

edited

Loading

miroslavpojer Nov 3, 2023

miroslavpojer commented Nov 3, 2023

miroslavpojer commented Nov 6, 2023

benedeki left a comment

lsulak commented Nov 6, 2023

		private val config = ConfigFactory.load("application.properties")

		override def dbConfig: Config = config.getConfig("postgres")

		class CheckpointStatusTypeRef extends TypeReference[CheckpointStatus.type]


		// To check By Forgiveness

#50: Create or adjust endpoint for accepting and saving checkpoint data #74

#50: Create or adjust endpoint for accepting and saving checkpoint data #74

Conversation

TebaleloS commented Sep 18, 2023 • edited Loading

github-actions bot commented Sep 18, 2023 • edited Loading

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

github-actions bot commented Sep 18, 2023 • edited Loading

JaCoCo server module code coverage report - scala 2.12.12

miroslavpojer left a comment • edited Loading

Choose a reason for hiding this comment

miroslavpojer commented Oct 31, 2023

benedeki Nov 1, 2023

Choose a reason for hiding this comment

benedeki Nov 1, 2023

Choose a reason for hiding this comment

lsulak Nov 1, 2023 • edited Loading

Choose a reason for hiding this comment

miroslavpojer left a comment • edited Loading

Choose a reason for hiding this comment

miroslavpojer Nov 3, 2023

Choose a reason for hiding this comment

miroslavpojer commented Nov 3, 2023

miroslavpojer commented Nov 6, 2023

benedeki left a comment

Choose a reason for hiding this comment

lsulak commented Nov 6, 2023

TebaleloS commented Sep 18, 2023 •

edited

Loading

github-actions bot commented Sep 18, 2023 •

edited

Loading

github-actions bot commented Sep 18, 2023 •

edited

Loading

miroslavpojer left a comment •

edited

Loading

lsulak Nov 1, 2023 •

edited

Loading

miroslavpojer left a comment •

edited

Loading