#2213 Explicitly disable control metrics at the end of the job. #2212
Conversation
This is needed when the main method can be invoked more than once externally.
…he-end-of-the-job
| @@ -60,6 +61,7 @@ object StandardizationAndConformanceJob extends StandardizationAndConformanceExe | |||
| // post processing deliberately rereads the output ... same as above | |||
| runPostProcessing(SourcePhase.Conformance, preparationResult, cmd) | |||
| } finally { | |||
| spark.disableControlMeasuresTracking() | |||
There was a problem hiding this comment.
I see this in StandardizationAndConformanceJob.scala. What about StandardizationJob.scala and DynamicConformanceJob.scala?
Maybe add implicit spark context to the finishJob function below?
There was a problem hiding this comment.
I'm fixing only Pramen use case that always runs Standardization+Conformance.
When running individually the logic is more complicated. There is already selectively disabling of Atum measurements. I decided not to touch it and go only with absolutely safe proof solution.
But you are welcome to refactor the logic as a separate PR.
There was a problem hiding this comment.
I get it. But there is no ticket connected with it, no tracking & IMHO, a partial solution. No approval from me, but happy if any other codeowners approve.
There was a problem hiding this comment.
The solution is not partial. Again, when running Standardization and Conformance separately, it works fine.
The purpose of this PR is not refactoring of the existing code. Just a critical fix to DB2 pipelines.
There was a problem hiding this comment.
This is philosophically complicated. 🤔
On one hand this project is to be sunset, so putting too much effort for the cleanest effort seems like a wasted time.
On the other hand there are questions:
- Pramen is the future, perhaps even for Unify. Will this be part of main Pramen or just a "special edition?
- So Pramen in integration actually calls Encelauds'
mainfunction? That's wild. - It feels weird that the control measurement is disable only on one case - I understand your reasoning, why bother with others, but still. Well at least a comment would be nice - why it's here
- It's also architecturally incorrect, that the Atum is enabled on one level (
StandardizationExecutionorConformanceExecutionbut disable on themainlevel only (furthermore selectively). But the counter argument is the same as in the previous poiint - Why is the disabling done in the
finallysection? Is it expected that the job can fail and still recover and do some Atum measurements? - If the answer to the previous one is true, why this switching-off can't be done in the "envelope" of Enceladus call in Pramen?
There was a problem hiding this comment.
- As part of 'pramen-extras' module there is 'Enceladus sink'. It is OSS since Enceladus is also OSS. But it will be sunset. This sink will go away if/when Enceladus is no longer used. Pramen won't be used in Unify, at least not in near future. This PR is just an Ecneladus patch for a transient solution and not part of something long-term.
- Yes. Also does it by reflection so it doesn't have compile time dependency on Enceladus.
- The reasoning is:
a) if I modify too much, I'd need to test all cases, and I prefer to avoid it,
b) this is a support branch, with minimal change to patch only what is needed I thing is okay, - I don't know what to say.
- Yes, failing jobs should not fail the full Pramen pipeline.
- Becaude 'pramen-extras' does not have compile time dependency on Enceladus, nor Atum. Enceladus builds are spark specific. Pramen builds are not.
yruslan
changed the title
...cala/za/co/absa/enceladus/standardization_conformance/StandardizationAndConformanceJob.scala
Outdated
Show resolved
Hide resolved
…_conformance/StandardizationAndConformanceJob.scala Co-authored-by: David Benedeki <[email protected]>
|
yruslan
deleted the
feature/disable-control-metrics-at-the-end-of-the-job
branch
This is needed when the main method can be invoked more than once externally.
The PR fixes running multiple Standardization+Conformance jobs from a single Spark session via Pramen. If this patch is not applied, the second and other pipelines will fail with "control metrics already initialized" exception.
This was tested end to end.
