BQ e2e updated tests #1273

priyabhatnagar25 · 2023-07-28T15:37:01Z

No description provided.

src/e2e-test/features/bigquery/source/BigQueryToBigQuery_withConnections.feature

src/e2e-test/java/io/cdap/plugin/bigquery/locators/BigQueryLocators.java

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidation.java

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQuery.java

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/TestSetupHooks.java

sahusanket · 2023-08-04T08:21:27Z

src/e2e-test/java/io/cdap/plugin/bigquery/locators/BigQueryLocators.java

+/**
+ * BigQuery Related Locators.
+ */
+public class BigQueryLocators {


All these locators are already present in the framework repo: https://github.com/cdapio/cdap-e2e-tests/blob/develop/src/main/java/io/cdap/e2e/pages/locators/CdfBigQueryPropertiesLocators.java

Why are we re-inventing the wheel here?

Reverting the changes

sahusanket · 2023-08-04T08:32:50Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/TestSetupHooks.java

+/**
+ * GCP test hooks.
+ */
+public class TestSetupHooks {


These test setup hooks are already present here: https://github.com/data-integrations/google-cloud/blob/develop/src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

Why are we adding them again here?

Reverting the changes

Reverted. @sahusanket

sahusanket · 2023-08-04T08:33:25Z

src/e2e-test/java/io/cdap/plugin/gcs/stepsdesign/GCSBase.java

@@ -310,33 +310,4 @@ private enum FilePart {

    String filePart;
  }
-
-  @Then("Validate the values of records transferred to GCS bucket is equal to the values from source BigQuery table")


Why are these removed?

Added back.

itsankit-google · 2023-08-08T08:22:21Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Enter input plugin property: "table" with value: "bqTargetTable"
+    And Select radio button plugin property: "operation" with value: "upsert"
+    Then Click plugin property: "updateTableSchema"
+    Then Enter the Table Key property for performing update and upsert operation with value: "TableKey"


Replace this step with a generic step definiton like: https://github.com/cdapio/cdap-e2e-tests/blob/4ec16f7af29ffcfb5e44cf549ec7a919cf44d2fa/src/main/java/stepsdesign/PipelineSteps.java#L457

Adding a generic step according to these fields (tableKey, deDupeBy and clusteringOrder) as a locator is different from the one which we used in Cdap e2e framework step.

I know it is different, my comment was to add a similar generic locator in cdap-e2e-framework and use it here.

itsankit-google · 2023-08-08T08:24:04Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Enter input plugin property: "dataset" with value: "dataset"
+    Then Enter input plugin property: "table" with value: "bqTargetTable"
+    Then Enter BigQuery sink property partition field "bqPartitionFieldTime"
+    Then Enter the Cluster property : "clusterValue"


similarly here, replace it with a generic step.

Replaced with a generic step.

itsankit-google · 2023-08-08T08:24:51Z

src/e2e-test/java/io/cdap/plugin/bigquery/actions/BigQueryActions.java

+/**
+ * BigQuery plugin step actions.
+ */
+public class BigQueryActions {


remove this class, similar class is present in cdap-e2e-tests repo, any changes should be maintained at one place.

itsankit-google · 2023-08-08T16:13:28Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryBase.java

@@ -1,5 +1,5 @@
 /*
- * Copyright © 2021 Cask Data, Inc.
+ * Copyright © 2023 Cask Data, Inc.


why this change? Looks like nothing is changed in this file.

An unintended step was initially removed and is now added. Aryan has also suggested changing the year to 2023 as part of the same change.

you said some step definition was added, but I still don't anything changed in this file, am I missing something?

I Apologize for the misunderstanding. The changes were made to the GCS Base, not the BigQueryBase. Reverted the year back to 2021. commit link

itsankit-google

Build is failing with some duplicate step definitons, please fix:

Duplicate step definitions in io.cdap.plugin.bigquery.stepsdesign.BigQuery.validateTheValuesOfRecordsTransferredToGcsBucketIsEqualToTheValuesFromSourceBigQueryTable() in file:/tmp/google-cloud/google-cloud/plugin/target/test-classes/ and io.cdap.plugin.gcs.stepsdesign.GCSBase.validateTheValuesOfRecordsTransferredToGcsBucketIsEqualToTheValuesFromSourceBigQueryTable() in file:/tmp/google-cloud/google-cloud/plugin/target/test-classes/

bharatgulati · 2023-08-09T05:24:49Z

Build is failing with some duplicate step definitons, please fix:

Duplicate step definitions in io.cdap.plugin.bigquery.stepsdesign.BigQuery.validateTheValuesOfRecordsTransferredToGcsBucketIsEqualToTheValuesFromSourceBigQueryTable() in file:/tmp/google-cloud/google-cloud/plugin/target/test-classes/ and io.cdap.plugin.gcs.stepsdesign.GCSBase.validateTheValuesOfRecordsTransferredToGcsBucketIsEqualToTheValuesFromSourceBigQueryTable() in file:/tmp/google-cloud/google-cloud/plugin/target/test-classes/

Resolved.

itsankit-google · 2023-08-09T06:15:16Z

Resolved.

Bigquery tests are still failing and does not look like related to existing chrome driver issue, ptal, thanks!

itsankit-google · 2023-08-09T10:35:37Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidation.java

+
+

remove extra empty lines.

itsankit-google · 2023-08-09T16:30:35Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidation.java

+    validateSourceBQToTargetBQRecord("E2E_SOURCE_32bee2c2_8e92_46ab_86af_9f9333371f58",
+                                     "E2E_TARGET_9905c77f_c04d_4958_accc_de3f02de1bba");


what are these hardcoded UUIDs??

The main function for debugging and fixing e2e build failures with source and sink tables was added to check on local. Now that they've passed. I've now removed it.

itsankit-google · 2023-08-09T18:23:54Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

@@ -255,7 +255,7 @@ public static void createTempSourceBQTable() throws IOException, InterruptedExce
      records.append(" (").append(index).append(", ").append((int) (Math.random() * 1000 + 1)).append(", '")
        .append(UUID.randomUUID()).append("'), ");
    }
-    BigQueryClient.getSoleQueryResult("create table `test_automation." + bqSourceTable + "` as " +
+    BigQueryClient.getSoleQueryResult("create table `bq_automation." + bqSourceTable + "` as " +


The value of dataset should come from plugin properties.

Using a dataset constant instead of a hard-coded value.

itsankit-google · 2023-08-09T18:24:15Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

@@ -285,11 +286,11 @@ public static void deleteTempSourceBQTable() throws IOException, InterruptedExce
  @Before(order = 1, value = "@BQ_PARTITIONED_SOURCE_TEST")
  public static void createTempPartitionedSourceBQTable() throws IOException, InterruptedException {
    bqSourceTable = "E2E_SOURCE_" + UUID.randomUUID().toString().replaceAll("-", "_");
-    BigQueryClient.getSoleQueryResult("create table `test_automation." + bqSourceTable + "` " +
+    BigQueryClient.getSoleQueryResult("create table `bq_automation." + bqSourceTable + "` " +


similar comment at all places using the hard-coded value.

itsankit-google

Plz squash commits before merge.

bharatgulati · 2023-08-10T14:59:33Z

All commits have been squashed. However, before you merge the PR. We will raise an another PR for the review of the BQ step in the cdap-e2e-framework because three steps have been skipped

itsankit-google · 2023-08-10T15:53:38Z

All commits have been squashed. However, before you merge the PR. We will raise an another PR for the review of the BQ step in the cdap-e2e-framework because three steps have been skipped

Sure, will wait for the framework PR to be merged.

bharatgulati · 2023-08-11T11:20:12Z

All commits have been squashed. However, before you merge the PR. We will raise an another PR for the review of the BQ step in the cdap-e2e-framework because three steps have been skipped

Sure, will wait for the framework PR to be merged.

Please review the step added for Big Query. commit Link

sau42shri requested review from Aryan-Verma and sahusanket August 2, 2023 07:29

Aryan-Verma reviewed Aug 4, 2023

View reviewed changes

sahusanket requested changes Aug 4, 2023

View reviewed changes

sahusanket reviewed Aug 4, 2023

View reviewed changes

Vipinofficial11 assigned bharatgulati Aug 4, 2023

Aryan-Verma approved these changes Aug 4, 2023

View reviewed changes

itsankit-google reviewed Aug 8, 2023

View reviewed changes

itsankit-google added the build Trigger unit test build label Aug 8, 2023

itsankit-google reviewed Aug 8, 2023

View reviewed changes

itsankit-google requested changes Aug 8, 2023

View reviewed changes

Vipinofficial11 added build Trigger unit test build and removed build Trigger unit test build labels Aug 9, 2023

itsankit-google reviewed Aug 9, 2023

View reviewed changes

itsankit-google approved these changes Aug 10, 2023

View reviewed changes

sahusanket approved these changes Aug 10, 2023

View reviewed changes

BQ e2e updated tests

23c9479

bharatgulati force-pushed the bigquery_e2e_tests branch from 16c091f to 23c9479 Compare August 10, 2023 14:58

itsankit-google merged commit aca754d into data-integrations:develop Aug 16, 2023
8 of 9 checks passed

		validateSourceBQToTargetBQRecord("E2E_SOURCE_32bee2c2_8e92_46ab_86af_9f9333371f58",
		"E2E_TARGET_9905c77f_c04d_4958_accc_de3f02de1bba");

BQ e2e updated tests #1273

BQ e2e updated tests #1273

Conversation

priyabhatnagar25 commented Jul 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsankit-google left a comment

Choose a reason for hiding this comment

bharatgulati commented Aug 9, 2023

itsankit-google commented Aug 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharatgulati Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsankit-google left a comment

Choose a reason for hiding this comment

bharatgulati commented Aug 10, 2023

itsankit-google commented Aug 10, 2023

bharatgulati commented Aug 11, 2023 • edited Loading

bharatgulati Aug 9, 2023 •

edited

Loading

bharatgulati commented Aug 11, 2023 •

edited

Loading