Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CSV data support for Ballerina #1

Closed
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1dcb5c4
initial commit
SasinduDilshara Feb 15, 2024
bf83a37
Add toCsvString function
SasinduDilshara Feb 16, 2024
11481e7
Update `fromStringWithType` function
SasinduDilshara Feb 16, 2024
45f77d2
Update fromStringWithType function
SasinduDilshara Feb 16, 2024
3353ad4
Update json variables into csv variables
SasinduDilshara Feb 18, 2024
41270d7
Update csv var names
SasinduDilshara Feb 18, 2024
ee61184
Update error diagnostic codes
SasinduDilshara Feb 18, 2024
0f79b57
Remove nodestack from csv traversal
SasinduDilshara Feb 18, 2024
1dfcb0f
Update type of field hierachy and the rest type
SasinduDilshara Feb 18, 2024
f65b2ce
Remove unwanted fields in parser
SasinduDilshara Feb 18, 2024
0e81e8a
Refactor header states
SasinduDilshara Feb 18, 2024
a9d571a
Refactor csv parser
SasinduDilshara Feb 18, 2024
855f202
Add configs
SasinduDilshara Feb 19, 2024
03667ef
Refactor the code with user configurations
SasinduDilshara Feb 21, 2024
940fd33
Refactor the traversal function
SasinduDilshara Feb 21, 2024
4c9f9ad
Add usere configs and basic tests
SasinduDilshara Mar 6, 2024
ed3cbf4
Add tests for tuple source values
SasinduDilshara Mar 15, 2024
d9f3753
Add tests for csvWithType Functiojn
SasinduDilshara Mar 18, 2024
b7a93cd
Add record related tests for fromcsvstring with type function
SasinduDilshara Mar 20, 2024
23a025b
Add tests for arrays & maps for fromStringWithType
SasinduDilshara Mar 22, 2024
b6a379e
Add tests for config headers and custom headers
SasinduDilshara Mar 24, 2024
9074ecf
Add tests for several configs of header params
SasinduDilshara Mar 24, 2024
b1284ed
Update user configs
SasinduDilshara Mar 29, 2024
f89691c
Add test files for basic functionality
SasinduDilshara May 13, 2024
7bc30a5
Add user config tests
SasinduDilshara May 14, 2024
9b8c7f7
Add tests for comments and null configs
SasinduDilshara Jun 6, 2024
f2a4a1a
Add tests for customheaders
SasinduDilshara Jun 6, 2024
04388da
Add basic text enclosure property
SasinduDilshara Jun 9, 2024
090aec4
Add tests for escape characters and delimiters
SasinduDilshara Jun 10, 2024
31f7ab9
Add tests for line terminator
SasinduDilshara Jun 10, 2024
6c4444c
Add tests for nil config and data projection
SasinduDilshara Jun 13, 2024
a70c2a8
Add tests for name annotation
SasinduDilshara Jun 13, 2024
45399fe
Add Name config tests for records
SasinduDilshara Jun 13, 2024
f4b386d
Add type compatibility for string tuples
SasinduDilshara Jun 16, 2024
b0ee27c
Add intersection support
SasinduDilshara Jun 18, 2024
360ea4d
Add constraint validations
SasinduDilshara Jun 20, 2024
1f2c816
Add tests for internal union types
SasinduDilshara Jun 24, 2024
5bcc0d9
Add tests for union & intertsection types
SasinduDilshara Jun 25, 2024
e291761
Add tests for finite types
SasinduDilshara Jun 26, 2024
30fb9e4
Add implementation and tests for Singleton and subtypes
SasinduDilshara Jun 27, 2024
8b2f215
Add implementation for compiler plugin
SasinduDilshara Jul 2, 2024
5eb0918
[Automated] Update the native jar versions
SasinduDilshara Jul 2, 2024
1d21659
[Automated] Update the native jar versions
SasinduDilshara Jul 2, 2024
cfd8d2d
Add compiler plugin tests
SasinduDilshara Jul 2, 2024
61169f2
Update error type
SasinduDilshara Jul 5, 2024
41f02a7
Removed resolved comments in ballerina files
SasinduDilshara Jul 5, 2024
fec4c7b
Remove unwanted debug functions and logs
SasinduDilshara Jul 5, 2024
fb6d13b
[Automated] Update the native jar versions
SasinduDilshara Jul 5, 2024
85352b2
Add API documentations for public types
SasinduDilshara Jul 5, 2024
a1924d1
Remove the ReadMe description
SasinduDilshara Jul 5, 2024
d604598
Merge branch 'development' of https://github.com/ballerina-platform/m…
SasinduDilshara Jul 5, 2024
e4235af
Update git attributes for windows
SasinduDilshara Jul 5, 2024
58c4dbf
[Automated] Update the native jar versions
SasinduDilshara Jul 8, 2024
7ead985
Restructure the ballerina integration test structure
SasinduDilshara Jul 9, 2024
e4ae6d2
[Automated] Update the native jar versions
SasinduDilshara Jul 10, 2024
c7d0602
[Automated] Update the native jar versions
SasinduDilshara Jul 10, 2024
be14fe2
Fix new line character related issu in wiondows build
SasinduDilshara Jul 10, 2024
0ca601b
[Automated] Update the native jar versions
SasinduDilshara Jul 10, 2024
fe50cfa
[Automated] Update the native jar versions
SasinduDilshara Jul 11, 2024
f20fd85
[Automated] Update the native jar versions
SasinduDilshara Jul 11, 2024
f4f07ca
[Automated] Update the native jar versions
SasinduDilshara Jul 12, 2024
1eb6cf9
Add tests for byte and byte streams
SasinduDilshara Jul 12, 2024
916eff1
[Automated] Update the native jar versions
SasinduDilshara Jul 12, 2024
fba132a
Add tests for array indexes
SasinduDilshara Jul 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update error diagnostic codes
  • Loading branch information
SasinduDilshara committed Feb 18, 2024
commit ee61184fa18d0b85788331f43e07cace6b1d9907
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ static BMap<BString, Object> initRootMapValue(Type expectedType) {
case TypeTags.MAP_TAG:
return ValueCreator.createMapValue((MapType) expectedType);
default:
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType, "map type");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType);
}
}

Expand All @@ -61,7 +61,7 @@ static BArray initArrayValue(Type expectedType) {
case TypeTags.ARRAY_TAG:
return ValueCreator.createArrayValue((ArrayType) expectedType);
default:
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType, "list type");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType);
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@

package io.ballerina.stdlib.data.csvdata.csv;

import io.ballerina.runtime.api.PredefinedTypes;
import io.ballerina.runtime.api.TypeTags;
import io.ballerina.runtime.api.creators.ValueCreator;
import io.ballerina.runtime.api.flags.SymbolFlags;
Expand Down Expand Up @@ -161,7 +160,7 @@ private String value() {
public Object execute(Reader reader, Type type) throws BError {
Type referredType = TypeUtils.getReferredType(type);
if (referredType.getTag() != TypeTags.ARRAY_TAG) {
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, type, PredefinedTypes.TYPE_ANYDATA_ARRAY);
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, type);
} else {
rootCsvNode = ValueCreator.createArrayValue((ArrayType) type);
expectedArrayElementType = ((ArrayType) TypeUtils.getReferredType(referredType)).getElementType();
Expand All @@ -184,7 +183,7 @@ public Object execute(Reader reader, Type type) throws BError {
break;
// TODO: Check to add Union types as well
default:
throw DiagnosticLog.error(DiagnosticErrorCode.CSV_PARSER_EXCEPTION);
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedArrayElementType);
}
boolean header = true;
State currentState;
Expand All @@ -204,9 +203,9 @@ public Object execute(Reader reader, Type type) throws BError {
}
return rootCsvNode;
} catch (IOException e) {
throw DiagnosticLog.error(DiagnosticErrorCode.CSV_PARSER_EXCEPTION, e.getMessage());
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TOKEN, e.getMessage(), line, column);
} catch (CsvParserException e) {
throw DiagnosticLog.error(DiagnosticErrorCode.CSV_PARSER_EXCEPTION, e.getMessage(), line, column);
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TOKEN, e.getMessage(), line, column);
}
}

Expand Down Expand Up @@ -301,7 +300,7 @@ private void finalizeHeaders(StateMachine sm) throws CsvParserException {
for (Field field : recordType.getFields().values()) {
if (!sm.headers.contains(field.getFieldName())) {
if (SymbolFlags.isFlagOn(field.getFlags(), SymbolFlags.REQUIRED)) {
throw DiagnosticLog.error(DiagnosticErrorCode.REQUIRED_FIELD_NOT_PRESENT, field.getFieldName());
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_FIELD_IN_CSV, field.getFieldName());
}
}
}
Expand All @@ -313,15 +312,15 @@ private void finalizeHeaders(StateMachine sm) throws CsvParserException {
//TODO: Replace arraysize -1 with
if (size != -1 && size > sm.headers.size()) {
// TODO: Can remove using fillers
throw DiagnosticLog.error(DiagnosticErrorCode.REQUIRED_FIELD_NOT_PRESENT, "Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_EXPECTED_ARRAY_SIZE, sm.headers.size());
}
} else if (expType instanceof MapType) {
//ignore
} else if (expType instanceof TupleType) {
TupleType tupleType = (TupleType) expType;
if (tupleType.getRestType() != null && tupleType.getTupleTypes().size() > sm.headers.size()) {
// TODO: Can remove using fillers
throw DiagnosticLog.error(DiagnosticErrorCode.REQUIRED_FIELD_NOT_PRESENT, "Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_EXPECTED_TUPLE_SIZE, sm.headers.size());
}
} else {
throw new CsvParserException("Invalid expected type");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,10 @@ public Object traverseCsv(BArray csv, BMap<BString, Object> config, Type type) {
int expectedArraySize = ((ArrayType) referredType).getSize();
long sourceArraySize = csv.getLength();
if (expectedArraySize > sourceArraySize) {
throw DiagnosticLog.error(DiagnosticErrorCode.ARRAY_SIZE_MISMATCH);
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_EXPECTED_ARRAY_SIZE, sourceArraySize);
}
if (TypeUtils.getReferredType(expectedArrayElementType).getTag() == TypeTags.UNION_TAG) {
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, type, PredefinedTypes.TYPE_ANYDATA);
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedArrayElementType);
}
this.config = config;
if (TypeUtils.getReferredType(expectedArrayElementType).getTag() == TypeTags.RECORD_TYPE_TAG
Expand Down Expand Up @@ -136,7 +136,7 @@ public Object traverseCsvElementWithArray(Object csvElement, Type expectedType)
traverseArrayValue(csvElement, csvMember, true);
return csvMember;
}
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType, PredefinedTypes.TYPE_ANYDATA);
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType);
}

private void traverseArrayValue(Object csvElement, BArray csvMember, boolean tuple) {
Expand Down Expand Up @@ -212,7 +212,7 @@ private void addValuesToArrayType(Object csvElement, BArray csvMember, Type arra
csvMember.add(index, convertToBasicType(csvElement, arrayElementType));
break;
default:
DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, arrayElementType, PredefinedTypes.TYPE_ANYDATA);
DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, arrayElementType);
}
}

Expand All @@ -233,13 +233,12 @@ public Object traverseCsvElementWithMapOrRecord(Object csvElement, Type expected
int arrayElementTypeTag = arrayType.getElementType().getTag();
// TODO: Only allow string[]
if (arrayElementTypeTag != TypeTags.STRING_TAG) {
DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE,
expectedType, PredefinedTypes.TYPE_ANYDATA_ARRAY);
DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType);
}
traverseCsvElementWithArrayAsExpectedType(csvElement,expectedType, arrayType);
break;
default:
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType, PredefinedTypes.TYPE_ANYDATA);
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE, expectedType);
}
return currentCsvNode;
}
Expand Down Expand Up @@ -276,10 +275,7 @@ private void traverseCsvMap(Object csvElement, Type expectedType, boolean mappin
} else if (csvElement instanceof BArray) {
traverseArrayValueWithMapAsExpectedType((BArray) csvElement, parentJsonNode, mappingType, expectedType);
} else {
if (fieldNames.isEmpty()) {
throw DiagnosticLog.error(DiagnosticErrorCode.INCOMPATIBLE_TYPE, expectedType, csvElement);
}
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE_FOR_FIELD, "Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_CSV_DATA_FORMAT);
}
nodesStack.pop();
}
Expand Down Expand Up @@ -362,8 +358,7 @@ private void traverseArrayValueWithMapAsExpectedType(BArray csvElement, Object p
}
boolean conversion = checkExpectedTypeMatchWithHeaders(expectedType, headers, csvElement, arraySize);
if (!conversion) {
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE_FOR_FIELD, headers.toString(),
csvElement.toString(), "Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_CAST, csvElement, expectedType);
}
// TODO: Add headers from config
for(int i = 1; i <= arraySize; i++) {
Expand Down Expand Up @@ -433,8 +428,8 @@ private void addCurrentFieldValue(Type currentFieldType, Object mapValue, BStrin
case TypeTags.MAP_TAG:
// TODO: Check
if (!checkTypeCompatibility(((MapType) currentFieldType).getConstrainedType(), mapValue)) {
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_TYPE_FOR_FIELD, mapValue,
currentFieldType, "Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_CAST, mapValue,
currentFieldType);
}
((BMap<BString, Object>) currentCsvNode).put(StringUtils.fromString(fieldNames.pop()),
mapValue);
Expand Down Expand Up @@ -498,12 +493,12 @@ private boolean checkTypeCompatibility(Type constraintType, Object csv) {
private void checkOptionalFieldsAndLogError(Map<String, Field> currentField) {
currentField.values().forEach(field -> {
if (SymbolFlags.isFlagOn(field.getFlags(), SymbolFlags.REQUIRED)) {
throw DiagnosticLog.error(DiagnosticErrorCode.REQUIRED_FIELD_NOT_PRESENT, field.getFieldName());
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_FIELD_IN_CSV, field.getFieldName());
}
// TODO: Handle this properly
if (!(SymbolFlags.isFlagOn(field.getFlags(), SymbolFlags.REQUIRED) &&
SymbolFlags.isFlagOn(field.getFlags(), SymbolFlags.OPTIONAL))) {
throw DiagnosticLog.error(DiagnosticErrorCode.REQUIRED_FIELD_NOT_PRESENT, field.getFieldName());
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_FIELD_IN_CSV, field.getFieldName());
}
});
}
Expand All @@ -519,11 +514,7 @@ private Object convertToBasicType(Object csv, Type targetType) {
}
return value;
} catch (Exception e) {
if (fieldNames.isEmpty()) {
throw DiagnosticLog.error(DiagnosticErrorCode.INCOMPATIBLE_TYPE, targetType, csv);
}
throw DiagnosticLog.error(DiagnosticErrorCode.INCOMPATIBLE_TYPE, csv, targetType,
"Test");
throw DiagnosticLog.error(DiagnosticErrorCode.INVALID_CAST, csv, targetType);
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,12 @@ public static Object fromCsvStringWithType(Environment env, Object csv, BMap<BSt
DataReaderThreadPool.EXECUTOR_SERVICE.submit(task);
return null;
} else {
return DiagnosticLog.error(DiagnosticErrorCode.UNSUPPORTED_TYPE, expType);
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_EXPECTED_TYPE, expType);
}
} catch (BError e) {
return e;
} catch (Exception e) {
return DiagnosticLog.error(DiagnosticErrorCode.CSV_PARSE_ERROR, e.getMessage());
return DiagnosticLog.error(DiagnosticErrorCode.INVALID_CAST, csv, type);
}
}

Expand Down
2 changes: 1 addition & 1 deletion native/src/main/resources/error.properties
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ error.invalid.expected.type=\
expected a array type, found ''{0}''

error.invalid.token.while.reading.the.csv.data=\
error while reading the csv data ''{0}''
error while reading the csv data ''{0}'' in line: ''{1}'', column: ''{2}''

error.invalid.csv.data.format=\
invalid csv data format
Expand Down