Server endpoints v2 returning the checkpoins data of a partitioning: 190 #194

TebaleloS · 2024-04-30T12:08:36Z

Created a class to call the DB function to get checkpoint based on the provided partitioning.
Added a function to return the checkpoint from database in the repository.
Defined CheckpointQuery and CheckpointFromDB as input and output types for the partitioning checkpoint operation.
Created a function to get partitioning checkpoints from the database and convert it to checkpointDTO.
Added Unit and integration testing.
Modified some of the properties of a DB function that get partitioning checkpoints.
Added REST API, v2: get checkpoints based on the provided partitioning.

closes #190

…point

github-actions · 2024-04-30T12:10:33Z

JaCoCo server module code coverage report - scala 2.13.11

Overall Project	72.06% `-23.65%`	🍏
Files changed	73.64%	❌

File	Coverage
BaseRepository.scala	100%	🍏
PartitioningRepositoryImpl.scala	100%	🍏
PartitioningControllerImpl.scala	100%	🍏
BaseController.scala	100%	🍏
PartitioningServiceImpl.scala	100%	🍏
BaseService.scala	100%	🍏
CheckpointFromDB.scala	85.83% `-308.66%`	❌
PlayJsonImplicits.scala	54.84%	🍏
GetPartitioningCheckpoints.scala	54.55% `-56.36%`	❌

…-the-checkpoins-data-of-a-partitioning

…point.sql

…tioningCheckpoints.scala

…ss and its object

…oints class and its object

…tory

…related classes.

project/Dependencies.scala

github-actions · 2024-06-04T10:11:24Z

JaCoCo model module code coverage report - scala 2.13.11

Overall Project	60.76% `-38.65%`	🍏
Files changed	0%	❌

File	Coverage
CheckpointQueryDTO.scala	0% `-1385.71%`	❌

github-actions · 2024-06-04T10:11:25Z

JaCoCo agent module code coverage report - scala 2.13.11

Overall Project	86.3%	🍏

There is no coverage information present for the Files changed

database/src/main/postgres/flows/V1.9.1__get_flow_checkpoints.sql

database/src/main/postgres/runs/V1.8.3__get_partitioning_checkpoints.sql

…-the-checkpoins-data-of-a-partitioning

...rc/test/scala/za/co/absa/atum/database/runs/GetPartitioningCheckpointsIntegrationTests.scala

lsulak · 2024-06-07T13:29:21Z

Looks good to me, review finished, just 1 minor thing, kind of consistency reasons; no other remarks outside of that, happy to approve soon!

…ns-data-of-a-partitioning' of https://github.com/AbsaOSS/atum-service into feature/#190-Server-endpoints-v2-returning-the-checkpoins-data-of-a-partitioning

salamonpavel · 2024-06-11T10:38:56Z

server/src/main/scala/za/co/absa/atum/server/api/http/Endpoints.scala

@@ -54,6 +54,15 @@ trait Endpoints extends BaseEndpoints {
      .out(jsonBody[AdditionalDataSubmitDTO])
  }

+  protected val getPartitioningCheckpointsEndpoint


We will have endpoints of different versions alongside one another therefore I would suggest a suffix V2 getPartitioningCheckpointsEndpointV2.

salamonpavel · 2024-06-11T10:47:11Z

...est/scala/za/co/absa/atum/server/api/controller/PartitioningControllerIntegrationTests.scala

@@ -27,18 +27,30 @@ import zio.test._

 object PartitioningControllerIntegrationTests extends ZIOSpecDefault with TestData {


Please rename the class. It's not an integration test and therefore should be executed as a unit test.

salamonpavel · 2024-06-11T10:47:50Z

...est/scala/za/co/absa/atum/server/api/controller/PartitioningControllerIntegrationTests.scala

@@ -64,6 +76,24 @@ object PartitioningControllerIntegrationTests extends ZIOSpecDefault with TestDa
            failsWithA[InternalServerErrorResponse]
          )
        }
+      ),
+
+      suite("GetPartitioningMeasuresSuite")(


GetPartitioningCheckpointsSuite

salamonpavel · 2024-06-11T10:55:28Z

...bsa/atum/server/api/database/runs/functions/GetPartitioningCheckpointsIntegrationTests.scala

+          checkpointName = Some("checkpointName")
+        )
+        // Read[CheckpointFromDB] implicit validation
+        Read[CheckpointFromDB]


Is this needed here? Implicits are resolved on compile time. Your code wouldn't compile if the read instance couldn't be constructed.

I left it there because of the conversation that you and I had that we should keep it there for future references.

salamonpavel · 2024-06-11T10:58:38Z

.../src/test/scala/za/co/absa/atum/server/api/service/PartitioningServiceIntegrationTests.scala

@@ -56,6 +57,11 @@ class PartitioningServiceIntegrationTests extends ZIOSpecDefault with TestData {
  when(partitioningRepositoryMock.getPartitioningAdditionalData(partitioningDTO2))
    .thenReturn(ZIO.fail(DatabaseError("boom!")))

+  when(partitioningRepositoryMock.getPartitioningCheckpoints(checkpointQueryDTO1))


Rename the class as these are unit tests and therefore should be executed as such. Also change it to object and remove the runwith annotation.

…-the-checkpoins-data-of-a-partitioning

salamonpavel

There are inconsistent namings. I would personally use getPartitioningCheckpoints everywhere (sometimes you use getPartitioningMeasures instead).

lsulak · 2024-06-12T14:48:51Z

There are inconsistent namings. I would personally use getPartitioningCheckpoints everywhere (sometimes you use getPartitioningMeasures instead).

Absolutely, I support this idea. However, I think this PR (and few others) has been here for far too long, and we know that we already have some technical debt in endpoint/operation namings, plus we need to update the agent to support some of the missing operations (especially those for v2).

I'll approve now as I think that it's complete from my PoV.

TebaleloS added 11 commits April 19, 2024 12:04

Fixes #189 - Define and implemented a db function to return the check…

8733ffd

…point

Fixes #189 - Modifying a db function

d211bd8

Fixes #189 - Modifying a db function

4dd02f1

Fixes #189 - Adding test cases

eea41f4

Fixes #189 - Added two input fields to the db function

1baac16

Fixes #189 - Fixing type mismatch

e2c61d1

Testing #120 db integration

388492a

merging changes from master

c386aec

Fixes #189 - Adding test cases

b3fc2bb

Fixes #189 - Changing back the details of flows.flows

6edc40a

Fixes #189 - Implementing GitHub comments

6c9be0c

TebaleloS self-assigned this May 6, 2024

TebaleloS and others added 17 commits May 6, 2024 10:52

Merge branch 'master' into feature/#190-Server-endpoints-v2-returning…

97f7269

…-the-checkpoins-data-of-a-partitioning

Fixes #190 - defined the dto to submit

6c2c99d

Merge branch 'master' into feature/#190-Server-endpoints-v2-returning…

b315d4e

…-the-checkpoins-data-of-a-partitioning

Update V1.7.1__flows.alter.ddl

dba7d34

Delete database/src/main/postgres/runs/V1.8.3__get_partitioning_check…

7f976e0

…point.sql

Update GetPartitioningMeasuresTest.scala

c39dbe0

Delete database/src/test/scala/za/co/absa/atum/database/runs/GetParti…

814b0ce

…tioningCheckpoints.scala

Fixees #190 - Merging master and Added GetPartitioningCheckpoints cla…

f4bdfec

…ss and its object

Fixees #190 - Added checkpoint query resultDTO, GetPartitioningCheckp…

bf22aa0

…oints class and its object

Fixes #190 - implementing get checkpoints functionality in the reposi…

f301d3f

…tory

Fixes #190 - renamed CheckpointDTO to CheckpointSubmitDTO

59602ac

Fixes #190 - Implementing partitioningCheckpoint service

91099fa

Fixes #190 - Implementing partitioningCheckpoint in the partitioning …

98fe612

…related classes.

Fixes #190 - Implementing partitioningCheckpoint in the partitioning …

0335ab9

…related classes.

Fixes #190 - fixing controller

888f455

Fixes #190 - updating current branch

9788148

Fixes #190 - Adding layers

d7d891d

lsulak reviewed May 30, 2024

View reviewed changes

project/Dependencies.scala Outdated Show resolved Hide resolved

TebaleloS added 4 commits May 30, 2024 12:27

Fixing the build

46da906

Fixing V1.8.3__get_partitioning_checkpoints.sql fields

f5e3d4c

Fixing service test cases

6d8cbe3

Fixing measure_columns to measured_columns

29b9e31

Changing from object CheckpointFromDBObject to CheckpointFromDB

80f86af

lsulak reviewed Jun 5, 2024

View reviewed changes

database/src/main/postgres/flows/V1.9.1__get_flow_checkpoints.sql Outdated Show resolved Hide resolved

lsulak reviewed Jun 5, 2024

View reviewed changes

database/src/main/postgres/flows/V1.9.1__get_flow_checkpoints.sql Outdated Show resolved Hide resolved

lsulak reviewed Jun 5, 2024

View reviewed changes

database/src/main/postgres/runs/V1.8.3__get_partitioning_checkpoints.sql Outdated Show resolved Hide resolved

Changing from object CheckpointFromDBObject to CheckpointFromDB

b591397

TebaleloS force-pushed the feature/#190-Server-endpoints-v2-returning-the-checkpoins-data-of-a-partitioning branch from 708067c to b591397 Compare June 5, 2024 12:50

Merge branch 'master' into feature/#190-Server-endpoints-v2-returning…

a44a681

…-the-checkpoins-data-of-a-partitioning

lsulak reviewed Jun 7, 2024

View reviewed changes

...rc/test/scala/za/co/absa/atum/database/runs/GetPartitioningCheckpointsIntegrationTests.scala Outdated Show resolved Hide resolved

TebaleloS added 2 commits June 7, 2024 15:43

Defining partitioningJson in a common way

c16df3c

Merge branch 'feature/#190-Server-endpoints-v2-returning-the-checkpoi…

55b83bf

…ns-data-of-a-partitioning' of https://github.com/AbsaOSS/atum-service into feature/#190-Server-endpoints-v2-returning-the-checkpoins-data-of-a-partitioning

salamonpavel reviewed Jun 11, 2024

View reviewed changes

TebaleloS and others added 2 commits June 11, 2024 14:42

Implementing GitHub comments

b6b35b6

Merge branch 'master' into feature/#190-Server-endpoints-v2-returning…

ca79884

…-the-checkpoins-data-of-a-partitioning

salamonpavel approved these changes Jun 11, 2024

View reviewed changes

lsulak approved these changes Jun 12, 2024

View reviewed changes

TebaleloS merged commit 5ab72ea into master Jun 13, 2024
7 checks passed

TebaleloS deleted the feature/#190-Server-endpoints-v2-returning-the-checkpoins-data-of-a-partitioning branch June 13, 2024 06:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Server endpoints v2 returning the checkpoins data of a partitioning: 190 #194

Server endpoints v2 returning the checkpoins data of a partitioning: 190 #194

TebaleloS commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Jun 4, 2024 •

edited

Loading

github-actions bot commented Jun 4, 2024

lsulak commented Jun 7, 2024

salamonpavel Jun 11, 2024

salamonpavel Jun 11, 2024

salamonpavel Jun 11, 2024

salamonpavel Jun 11, 2024

TebaleloS Jun 11, 2024

salamonpavel Jun 11, 2024

salamonpavel left a comment

lsulak commented Jun 12, 2024

		@@ -27,18 +27,30 @@ import zio.test._

		object PartitioningControllerIntegrationTests extends ZIOSpecDefault with TestData {

Server endpoints v2 returning the checkpoins data of a partitioning: 190 #194

Server endpoints v2 returning the checkpoins data of a partitioning: 190 #194

Conversation

TebaleloS commented Apr 30, 2024 • edited Loading

github-actions bot commented Apr 30, 2024 • edited Loading

JaCoCo server module code coverage report - scala 2.13.11

github-actions bot commented Jun 4, 2024 • edited Loading

JaCoCo model module code coverage report - scala 2.13.11

github-actions bot commented Jun 4, 2024

JaCoCo agent module code coverage report - scala 2.13.11

lsulak commented Jun 7, 2024

salamonpavel Jun 11, 2024

Choose a reason for hiding this comment

salamonpavel Jun 11, 2024

Choose a reason for hiding this comment

salamonpavel Jun 11, 2024

Choose a reason for hiding this comment

salamonpavel Jun 11, 2024

Choose a reason for hiding this comment

TebaleloS Jun 11, 2024

Choose a reason for hiding this comment

salamonpavel Jun 11, 2024

Choose a reason for hiding this comment

salamonpavel left a comment

Choose a reason for hiding this comment

lsulak commented Jun 12, 2024

TebaleloS commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Apr 30, 2024 •

edited

Loading

github-actions bot commented Jun 4, 2024 •

edited

Loading