Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug fix for using set-type when it's the first directive in the recipie #688

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

bug fix for using set-type when it's the first directive in the recipie #688

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c686ae
fix for using set-type when it's the first directive in the recipie
minurajeeve Dec 5, 2023
6ffcfaa
review comments -0
minurajeeve Jan 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 44 additions & 35 deletions wrangler-core/src/main/java/io/cdap/directives/column/SetType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,33 +134,12 @@ public Schema getOutputSchema(SchemaResolutionContext context) {
if (field.getName().equals(col)) {
Integer outputScale = scale;
Integer outputPrecision = precision;
Schema fieldSchema = field.getSchema().getNonNullable();
Pair<Integer, Integer> scaleAndPrecision = getPrecisionAndScale(fieldSchema);
Integer inputSchemaScale = scaleAndPrecision.getSecond();
Integer inputSchemaPrecision = scaleAndPrecision.getFirst();

if (scale == null && precision == null) {
outputScale = inputSchemaScale;
outputPrecision = inputSchemaPrecision;
} else if (scale == null && inputSchemaScale != null) {
if (precision - inputSchemaScale < 1) {
throw new DirectiveParseException(String.format(
"Cannot set scale as '%s' and precision as '%s' when "
+ "given precision - scale is less than 1 ", inputSchemaScale,
precision));
}
outputScale = inputSchemaScale;
outputPrecision = precision;

} else if (precision == null && inputSchemaPrecision != null) {
if (inputSchemaPrecision - scale < 1) {
throw new DirectiveParseException(String.format(
"Cannot set scale as '%s' and precision as '%s' when "
+ "given precision - scale is less than 1 ", scale,
inputSchemaPrecision));
}
outputScale = scale;
outputPrecision = inputSchemaPrecision;
if (type.equalsIgnoreCase("decimal") && field.getSchema().isNullable()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when schema is non nullable but set type directive is applied with 'decimal' as the type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the schema is non nullable it will skip this if condition and will go to the return statement in line no:167. By default outputScale and outputPrecision are the scale and precision specified by the user. In this case we don't validate if the user specified scale and precision values clash with the original scale and precision values of that column.

Schema fieldSchema = field.getSchema().getNonNullable();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I think this giant block that calculates precision and scale to use can be moved to a different function. Can refactor the function used in .map()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Pair<Integer, Integer> scaleAndPrecision = getValidatedPrecisionAndScale(
fieldSchema, precision, scale);
outputScale = scaleAndPrecision.getSecond();
outputPrecision = scaleAndPrecision.getFirst();
}
return Schema.Field.of(col, ColumnConverter.getSchemaForType(type,
outputScale, outputPrecision));
Expand All @@ -171,20 +150,50 @@ public Schema getOutputSchema(SchemaResolutionContext context) {
}
}
)
.collect(Collectors.toList())
.collect(Collectors.toList())
);
}

/**
* extracts precision and scale from schema string
*/
public static Pair<Integer, Integer> getPrecisionAndScale(Schema fieldSchema) {
Integer precision = null;
Integer scale = null;
if (fieldSchema.getLogicalType() == LogicalType.DECIMAL) {
precision = fieldSchema.getPrecision();
scale = fieldSchema.getScale();
public static Pair<Integer, Integer> getValidatedPrecisionAndScale(Schema fieldSchema,
Integer precision, Integer scale)
throws DirectiveParseException { //check precision and scale
Integer outputPrecision = precision;
Integer outputScale = scale;
Integer inputSchemaPrecision = null;
Integer inputSchemaScale = null;

if (fieldSchema.getLogicalType() == LogicalType.DECIMAL) {
inputSchemaPrecision = fieldSchema.getPrecision();
inputSchemaScale = fieldSchema.getScale();
}

if (scale == null && precision == null) {
outputScale = inputSchemaScale;
outputPrecision = inputSchemaPrecision;
} else if (scale == null && inputSchemaScale != null) {
if (precision - inputSchemaScale < 1) {
throw new DirectiveParseException(String.format(
"Cannot set scale as '%s' and precision as '%s' when "
+ "given precision - scale is less than 1 ", inputSchemaScale,
precision));
}
return new Pair<Integer, Integer>(precision, scale);
outputScale = inputSchemaScale;
outputPrecision = precision;

} else if (precision == null && inputSchemaPrecision != null) {
if (inputSchemaPrecision - scale < 1) {
throw new DirectiveParseException(String.format(
"Cannot set scale as '%s' and precision as '%s' when "
+ "given precision - scale is less than 1 ", scale,
inputSchemaPrecision));
}
outputScale = scale;
outputPrecision = inputSchemaPrecision;
}

return new Pair<Integer, Integer>(outputPrecision, outputScale);
}
}
Loading