Add export tasks #2450

inv-jishnu · 2025-01-06T09:27:35Z

Description

In this PR I have added classes and dtos for export tasks which processes the data to be exported for supported export file formats (JSON, JSONLines and CSV).

Related issues and/or PRs

Please review and merge this PR once the following PRs are merged

Changes made

Added producer tasks to generate export output files

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

Road map to merge remaining data loader core files. Current status

General
- Add ScalarDB Dao and related files - Add ScalarDB Dao and related files #2417
- TableMetadataService(partially replaced by ConsensusUtils): Add table metadata service #2434
Export
- Export options and validations: Add export options validator #2435
- ProducerTasks: Add export tasks #2450
Import
- Dto classes and utilities: Add data chunk and task result enums and dtos #2442
- Import processor and task code: 2-3 PRs incoming
  - Add dtos and other classes for task #2446
- Code for Import transaction batch and data chunk import: 1 PR Incoming
- ControlFile related Dtos: Add Control file module files and validation #2445
- Import logger: 1 PR incoming
- Import Manager: 1 PR incoming

Release notes

NA

komamitsu · 2025-02-13T01:51:59Z

...oader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/JsonLineExportManager.java

+  @Override
+  void processHeader(ExportOptions exportOptions, TableMetadata tableMetadata, Writer writer)
+      throws IOException {
+    System.out.println("SSSSSSSSSSSSSSSSSSSSSSSS");


This is for debug?

Yes, I forgot to remove it.
Removed it now.
Thank you.

komamitsu · 2025-02-13T01:52:08Z

...oader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/JsonLineExportManager.java

+  @Override
+  void processFooter(ExportOptions exportOptions, TableMetadata tableMetadata, Writer writer)
+      throws IOException {
+    System.out.println("SSSSSSSSSSSSSSSSSSSSSSSS");


Same as above

Removed it now.
Thank you.

komamitsu · 2025-02-13T02:39:51Z

data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataexport/ExportManager.java

+          // TODO: handle this
+        }
+        processFooter(exportOptions, tableMetadata, bufferedWriter);
+        bufferedWriter.flush();


BufferedWriter has its own buffer to contain tentative content to write. Even if the original Writer instance is closed, the buffered content can be lost if any exception is thrown before calling bufferedWriter.flush(). I think this should be called in finally block.

komamitsu

LGTM, thank you!

Torch3333 · 2025-02-14T00:50:13Z

...er/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTask.java

+      case BOOLEAN:
+        value = Boolean.toString(result.getBoolean(columnName));
+        break;
+      case TEXT:


This is a reminder just in case: please add support for the new data types DATE, TIME, TIMESTAMP, and TIMESTAMPTZ in the Data Loader at some point.

CC: @ypeckstadt

Torch3333 · 2025-02-14T00:51:59Z

...ore/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/ProducerTaskFactory.java

+  private final boolean prettyPrintJson;
+
+  /**
+   * *


A part of the Javadoc seems to be missing.

Torch3333

LGTM, thank you!

brfrn169

Left several comments. Please take a look when you have time!

brfrn169 · 2025-02-14T01:58:20Z

...er/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTask.java

+      case BOOLEAN:
+        value = Boolean.toString(result.getBoolean(columnName));
+        break;
+      case TEXT:


CC: @ypeckstadt

brfrn169 · 2025-02-14T02:03:26Z

...er/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/CsvProducerTask.java

+      default:
+        logger.error(
+            CoreError.DATA_LOADER_CONVERT_TO_STRING_FAILED.buildMessage(dataType.toString()));


We can throw an AssertionError since this code should never be reached.

Suggested change

default:

logger.error(

CoreError.DATA_LOADER_CONVERT_TO_STRING_FAILED.buildMessage(dataType.toString()));

default:

throw new AssertionError("Unkown data type:" + dataType);

brfrn169 · 2025-02-14T02:03:44Z

...re/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/JsonLineProducerTask.java

+      default:
+        logger.error(
+            CoreError.DATA_LOADER_CONVERT_TO_STRING_FAILED.buildMessage(dataType.toString()));
+        break;


Diito. We can throw an AssertionError since this code should never be reached.

brfrn169 · 2025-02-14T02:03:58Z

...r/core/src/main/java/com/scalar/db/dataloader/core/dataexport/producer/JsonProducerTask.java

+      default:
+        logger.error(
+            CoreError.DATA_LOADER_CONVERT_TO_STRING_FAILED.buildMessage(dataType.toString()));


Ditto. We can throw an AssertionError since this code should never be reached.

brfrn169

Left a comment. Other than that, LGTM. Thank you!

brfrn169 · 2025-02-20T05:35:42Z

core/src/main/java/com/scalar/db/common/error/CoreError.java

@@ -1079,6 +1087,12 @@ public enum CoreError implements ScalarDbError {
      "Something went wrong while scanning. Are you sure you are running in the correct transaction mode? Details: %s",
      "",
      ""),
+  DATA_LOADER_CONVERT_TO_STRING_FAILED(


It seems that this message is no longer needed.

I have removed it in 2828244.
Thank you.

Co-authored-by: Peckstadt Yves <[email protected]>

inv-jishnu and others added 30 commits December 4, 2024 15:59

Util classes for data loader

753618b

Fix spotbug issue

8d39d02

Removed error message and added core error

bf94c49

Applied spotless

47be388

Fixed unit test failures

913eb1c

Merge branch 'master' into feat/data-loader/utils

1f204b8

Basic data import enum and exception

6cfa83a

Removed exception class for now

d381b2b

Added DECIMAL_FORMAT

67f2474

Path util class updated

14e3593

Feedback changes

a096d51

Merge branch 'master' into feat/data-loader/utils

dbf1940

Merge branch 'master' into feat/data-loader/utils

cd8add9

Changes

52890c8

Merge branch 'master' into feat/data-loader/import-data-1

5114639

Merge branch 'feat/data-loader/utils' into feat/data-loader/scaladb-dao

4f9cd75

Added ScalarDB Dao

1997eb8

Merge branch 'master' into feat/data-loader/scaladb-dao

91e6310

Remove unnecessary files

8a7338b

Initial commit [skip ci]

2b52eeb

Changes

e206073

Changes

26d3144

spotbugs exclude

b86487d

spotbugs exclude -2

818a2b4

Added a file [skip ci]

90c4105

Added unit test files [skip ci]

3d5d3e0

Spotbug fixes

6495202

Removed use of List.of to fix CI error

90abd9e

Merged changes from master after resolving conflict

ba2b3dd

Merge branch 'master' into feat/data-loader/metadata-service

b1b811b

inv-jishnu requested a review from komamitsu February 12, 2025 10:47

komamitsu reviewed Feb 13, 2025

View reviewed changes

changes

a3a17f8

inv-jishnu requested a review from komamitsu February 13, 2025 04:32

komamitsu approved these changes Feb 13, 2025

View reviewed changes

inv-jishnu added 3 commits February 13, 2025 16:36

Resolved conflicts and merged latest changes from main

de52b56

Core error formatting

a4b33d2

Core error class formatted

4b550bd

Torch3333 approved these changes Feb 14, 2025

View reviewed changes

brfrn169 approved these changes Feb 14, 2025

View reviewed changes

brfrn169 self-requested a review February 14, 2025 02:05

Feedback changes

15b0df7

brfrn169 approved these changes Feb 20, 2025

View reviewed changes

inv-jishnu and others added 2 commits February 20, 2025 11:20

Removed unused error code

2828244

Merge branch 'master' into feat/data-loader/export-tasks

10a546d

ypeckstadt merged commit 75f8a9c into master Feb 20, 2025
48 checks passed

ypeckstadt deleted the feat/data-loader/export-tasks branch February 20, 2025 08:01

feeblefakie pushed a commit that referenced this pull request Feb 20, 2025

Add export tasks (#2450)

179991b

Co-authored-by: Peckstadt Yves <[email protected]>

feeblefakie mentioned this pull request Feb 20, 2025

Backport to branch(3.15) : Add export tasks #2544

Merged

feeblefakie pushed a commit that referenced this pull request Feb 20, 2025

Add export tasks (#2450)

3920ab2

Co-authored-by: Peckstadt Yves <[email protected]>

feeblefakie mentioned this pull request Feb 20, 2025

Backport to branch(3) : Add export tasks #2545

Merged

feeblefakie pushed a commit that referenced this pull request Feb 20, 2025

Add export tasks (#2450)

c794044

Co-authored-by: Peckstadt Yves <[email protected]>

feeblefakie mentioned this pull request Feb 20, 2025

Backport to branch(3.14) : Add export tasks #2546

Merged

feeblefakie pushed a commit that referenced this pull request Feb 20, 2025

Add export tasks (#2450)

8f467e9

Co-authored-by: Peckstadt Yves <[email protected]>

feeblefakie mentioned this pull request Feb 20, 2025

Backport to branch(3.13) : Add export tasks #2547

Merged

feeblefakie pushed a commit that referenced this pull request Feb 20, 2025

Add export tasks (#2450)

940ad73

Co-authored-by: Peckstadt Yves <[email protected]>

This was referenced Feb 20, 2025

Backport to branch(3.12) : Add export tasks #2548

Merged

Backport to branch(3.11) : Add export tasks #2549

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add export tasks #2450

Add export tasks #2450

inv-jishnu commented Jan 6, 2025 •

edited

Loading

komamitsu Feb 13, 2025

inv-jishnu Feb 13, 2025

komamitsu Feb 13, 2025

inv-jishnu Feb 13, 2025

komamitsu Feb 13, 2025

komamitsu left a comment

Torch3333 Feb 14, 2025

brfrn169 Feb 14, 2025

Torch3333 Feb 14, 2025

Torch3333 left a comment

brfrn169 left a comment

brfrn169 Feb 14, 2025

brfrn169 Feb 14, 2025

brfrn169 Feb 14, 2025

brfrn169 Feb 14, 2025

brfrn169 left a comment

brfrn169 Feb 20, 2025

inv-jishnu Feb 20, 2025

Add export tasks #2450

Add export tasks #2450

Conversation

inv-jishnu commented Jan 6, 2025 • edited Loading

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

komamitsu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Torch3333 left a comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brfrn169 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inv-jishnu commented Jan 6, 2025 •

edited

Loading