Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data.yaml module #4

Merged
merged 99 commits into from
Aug 2, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
56b7c0d
Add data.yaml initial implementation
LakshanWeerasinghe Feb 19, 2024
16e8ba4
Impl yaml lexer
LakshanWeerasinghe Feb 22, 2024
cf4eddb
Impl yaml parser
LakshanWeerasinghe Apr 21, 2024
7a49666
Add tests for basic types
LakshanWeerasinghe Apr 22, 2024
5e05d50
Fix issue with apply projection
LakshanWeerasinghe Apr 24, 2024
bef0b7d
[Automated] Update the native jar versions
LakshanWeerasinghe Apr 24, 2024
7ade15b
Add tests for yaml document parsing
LakshanWeerasinghe Apr 29, 2024
c46037c
Implement toYamlString api
LakshanWeerasinghe May 2, 2024
d2d80a9
Handle union type as expected type
LakshanWeerasinghe May 2, 2024
4c6914c
Add JsonTraverse to create values from json
LakshanWeerasinghe May 4, 2024
858a6a6
Add tests for union type as expected type
LakshanWeerasinghe May 4, 2024
92c443a
Support yaml input as byte[] stream
LakshanWeerasinghe May 4, 2024
216068f
Handle runtime errors as BErrors
LakshanWeerasinghe May 5, 2024
fb20398
Add negative tests
LakshanWeerasinghe May 6, 2024
0a30bde
Merge branch 'to-yaml-string' into yaml-data-impl
LakshanWeerasinghe May 7, 2024
30911de
Add options for parsing
LakshanWeerasinghe May 7, 2024
9d9fac6
Add test after disabling projection
LakshanWeerasinghe May 7, 2024
db5d271
Support anchors for maps and sequences
LakshanWeerasinghe May 8, 2024
9ee33b9
Add tests with anchors
LakshanWeerasinghe May 8, 2024
cc72c28
Add tag resolution for json and core schemas
LakshanWeerasinghe May 8, 2024
bde792e
Add tests for tag resolution
LakshanWeerasinghe May 8, 2024
d0cf4cb
Add tests for escaped characters
LakshanWeerasinghe May 8, 2024
bac9a4c
Fix issue with handling directives
LakshanWeerasinghe May 9, 2024
15aa7da
Support intersection types and yaml stream parsing
LakshanWeerasinghe May 13, 2024
816eaa8
Impl union and intersection exp type yaml streams
LakshanWeerasinghe May 13, 2024
2c32d86
Fix issue with handling indents
LakshanWeerasinghe May 14, 2024
2641189
Fix issue handling espcaed white spaces
LakshanWeerasinghe May 14, 2024
e78c360
Add tests for parsing yaml streams
LakshanWeerasinghe May 14, 2024
a3dac22
Add tests for toYamlString
LakshanWeerasinghe May 14, 2024
1e2b0dc
[Automated] Update the native jar versions
LakshanWeerasinghe May 14, 2024
8097874
Add Name annotation
LakshanWeerasinghe May 15, 2024
1ee0272
Impl compiler plugin
LakshanWeerasinghe May 15, 2024
4db611e
Fix validating var declrations
LakshanWeerasinghe May 15, 2024
4b51713
Add compiler plugin tests
LakshanWeerasinghe May 15, 2024
b7d3d60
Fix issue handling union expected types
LakshanWeerasinghe May 15, 2024
813b7e5
Add tests for yaml stream parsing
LakshanWeerasinghe May 15, 2024
abc82d1
Add licence headers
LakshanWeerasinghe May 16, 2024
12fd41d
Refactor code
LakshanWeerasinghe May 16, 2024
cc34639
Merge branch 'yaml-data-impl' into yml-data
LakshanWeerasinghe May 17, 2024
00c3f68
[Automated] Update the native jar versions
LakshanWeerasinghe May 17, 2024
fdceef7
Update .gitattributes file
LakshanWeerasinghe May 20, 2024
db993a7
Add codecov.yml file
LakshanWeerasinghe May 20, 2024
4153e1a
Fix issue handling new line characters
LakshanWeerasinghe May 24, 2024
9f1557d
Add graalvm resource config
LakshanWeerasinghe May 28, 2024
177fbdc
Support parsing yaml stream using a single api
LakshanWeerasinghe May 29, 2024
aae0d27
Support tuple member re-order for yaml streams
LakshanWeerasinghe May 30, 2024
f787d68
Fix handling union types for yaml streams
LakshanWeerasinghe Jun 3, 2024
5ff8ca2
Fix handling union types in tuple mem
LakshanWeerasinghe Jun 4, 2024
18ee860
Add tests for tuple expected type
LakshanWeerasinghe Jun 4, 2024
d573cd6
[Automated] Update the native jar versions
LakshanWeerasinghe Jun 4, 2024
43849a2
Refactor code
LakshanWeerasinghe Jun 4, 2024
f209fc0
Refactor code
LakshanWeerasinghe Jun 4, 2024
9437104
Refactor yaml stream composing
LakshanWeerasinghe Jun 4, 2024
7ee195b
Add Module.md and Package.md files
LakshanWeerasinghe Jun 4, 2024
d62c994
Update READEME.md file
LakshanWeerasinghe Jun 4, 2024
a05ef8e
[Automated] Update the native jar versions
LakshanWeerasinghe Jun 5, 2024
5dc871f
Update ballerina version
LakshanWeerasinghe Jun 5, 2024
499bf19
Add more tests for projection options
LakshanWeerasinghe Jun 6, 2024
7123aac
Remove unreachable functions
LakshanWeerasinghe Jun 6, 2024
15de097
Add compiler plugin tests
LakshanWeerasinghe Jun 7, 2024
5388c3a
Add tests for parsing tag handles
LakshanWeerasinghe Jun 7, 2024
610124a
Remove empty files
LakshanWeerasinghe Jun 10, 2024
f169ff1
Merge pull request #2 from LakshanWeerasinghe/yml-data
LakshanWeerasinghe Jun 10, 2024
6f236fc
Disable failing compiler-plugin test
LakshanWeerasinghe Jun 10, 2024
cf25f61
Add more tests for toYamlString
LakshanWeerasinghe Jun 10, 2024
df8e2b6
Add tag directive tests
LakshanWeerasinghe Jun 10, 2024
f0f19b6
Add tests for escaped characters
LakshanWeerasinghe Jun 10, 2024
5a78c05
Add private constructors for utils
LakshanWeerasinghe Jun 10, 2024
173f5d7
Add more tests
LakshanWeerasinghe Jun 10, 2024
b680098
Add more tag resolution tests
LakshanWeerasinghe Jun 10, 2024
5a3ca95
Fix issue with strict tuple ordering
LakshanWeerasinghe Jun 11, 2024
3658e38
Add more negative tests
LakshanWeerasinghe Jun 12, 2024
2b96a07
Fix block header issue and add tests
LakshanWeerasinghe Jun 12, 2024
4248895
Add tests for handling empty lines
LakshanWeerasinghe Jun 12, 2024
79e825c
Add tests for handling quoted scalars
LakshanWeerasinghe Jun 12, 2024
8fe8114
Add negative tests
LakshanWeerasinghe Jun 16, 2024
783dc2c
Add more tests
LakshanWeerasinghe Jun 17, 2024
6108aef
Add tests for streams with basic types
LakshanWeerasinghe Jun 18, 2024
d8d1338
Address review suggestions for compiler plugin
LakshanWeerasinghe Jun 18, 2024
c80e000
Add more compiler plugin tests
LakshanWeerasinghe Jun 18, 2024
36f9fad
Improve utils functions
LakshanWeerasinghe Jun 18, 2024
95506c8
Address review suggestion
LakshanWeerasinghe Jul 9, 2024
b85242d
Add api examples
LakshanWeerasinghe Jul 9, 2024
a8824aa
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 22, 2024
6e4f3ab
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 22, 2024
2207ef5
Add constraint validation support
LakshanWeerasinghe Jul 22, 2024
4082f7f
Move ModuleUtils to base package
LakshanWeerasinghe Jul 24, 2024
8b27ffb
Add tests by disabling constrain validation
LakshanWeerasinghe Jul 24, 2024
4cad13d
Improve record documentation
LakshanWeerasinghe Jul 29, 2024
9874cf5
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 29, 2024
6c10c63
Address review suggestion
LakshanWeerasinghe Jul 29, 2024
f90c207
Remove TypeUtils
LakshanWeerasinghe Jul 29, 2024
4e29be0
Add maven package publishing task
LakshanWeerasinghe Jul 30, 2024
8c5997e
Refactor options utils
LakshanWeerasinghe Jul 30, 2024
1f68831
Address review suggestions
LakshanWeerasinghe Jul 31, 2024
2b1ea84
Rename tupleReOrder config
LakshanWeerasinghe Jul 31, 2024
1258057
Updated the yaml stream parsing
LakshanWeerasinghe Jul 31, 2024
1c95d30
[Automated] Update the native jar versions
LakshanWeerasinghe Aug 1, 2024
31da9de
[Automated] Update the native jar versions
LakshanWeerasinghe Aug 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add options for parsing
  • Loading branch information
LakshanWeerasinghe committed May 7, 2024
commit 30911de6393897447d9d30e286738b638a2940b9
10 changes: 8 additions & 2 deletions ballerina/yaml_api.bal
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

import ballerina/jballerina.java;

# Converts YAML string to subtype of anydata.
Expand Down Expand Up @@ -57,7 +56,7 @@ public isolated function toYamlString(anydata yamlValue, WriteConfig config) ret
}

isolated function toYamlStringArray(anydata yamlValue, WriteConfig config)
returns string[]|Error = @java:Method {'class: "io.ballerina.stdlib.data.yaml.Native" } external;
returns string[]|Error = @java:Method {'class: "io.ballerina.stdlib.data.yaml.Native"} external;

# Represents the YAML schema available for the parser.
#
Expand All @@ -75,10 +74,17 @@ public enum YAMLSchema {
# + schema - field description
# + allowAnchorRedefinition - field description
# + allowMapEntryRedefinition - field description
# + allowDataProjection - Enable or disable projection
public type Options record {
YAMLSchema schema = CORE_SCHEMA;
boolean allowAnchorRedefinition = true;
boolean allowMapEntryRedefinition = false;
record {
# If `true`, nil values will be considered as optional fields in the projection.
boolean nilAsOptionalField = false;
# If `true`, absent fields will be considered as nilable types in the projection.
boolean absentAsNilableType = false;
}|false allowDataProjection = {};
};

# Configurations for writing a YAML document.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ public class Native {

public static Object parseString(BString yaml, BMap<BString, Object> options, BTypedesc typed) {
try {
return YamlParser.parse(new StringReader(yaml.getValue()), typed.getDescribingType());
return YamlParser.parse(new StringReader(yaml.getValue()), options, typed.getDescribingType());
} catch (BError e) {
return e;
}
Expand All @@ -58,7 +58,7 @@ public static Object parseString(BString yaml, BMap<BString, Object> options, BT
public static Object parseBytes(BArray yaml, BMap<BString, Object> options, BTypedesc typed) {
try {
return YamlParser.parse(new InputStreamReader(new ByteArrayInputStream(yaml.getBytes())),
typed.getDescribingType());
options, typed.getDescribingType());
} catch (BError e) {
return e;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ public void run() {
ResultConsumer<Object> resultConsumer = new ResultConsumer<>(future);
try (var byteBlockSteam = new BallerinaByteBlockInputStream(env, iteratorObj, resolveNextMethod(iteratorObj),
resolveCloseMethod(iteratorObj), resultConsumer)) {
Object result = YamlParser.parse(new InputStreamReader(byteBlockSteam), typed.getDescribingType());
Object result = YamlParser.parse(new InputStreamReader(byteBlockSteam), options, typed.getDescribingType());
future.complete(result);
} catch (Exception e) {
future.complete(DiagnosticLog.getYamlError("Error occurred while reading the stream: " + e.getMessage()));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import io.ballerina.runtime.api.PredefinedTypes;
import io.ballerina.runtime.api.TypeTags;
import io.ballerina.runtime.api.creators.ValueCreator;
import io.ballerina.runtime.api.flags.SymbolFlags;
import io.ballerina.runtime.api.types.ArrayType;
import io.ballerina.runtime.api.types.Field;
Expand All @@ -21,10 +22,12 @@
import io.ballerina.stdlib.data.yaml.lexer.LexerState;
import io.ballerina.stdlib.data.yaml.lexer.Token;
import io.ballerina.stdlib.data.yaml.lexer.YamlLexer;
import io.ballerina.stdlib.data.yaml.utils.Constants;
import io.ballerina.stdlib.data.yaml.utils.DiagnosticErrorCode;
import io.ballerina.stdlib.data.yaml.utils.DiagnosticLog;
import io.ballerina.stdlib.data.yaml.utils.Error;
import io.ballerina.stdlib.data.yaml.utils.JsonTraverse;
import io.ballerina.stdlib.data.yaml.utils.OptionsUtils;

import java.io.Reader;
import java.util.ArrayDeque;
Expand Down Expand Up @@ -78,11 +81,7 @@ public static class ComposerState {
private final ParserState parserState;
private final Map<String, String> anchorBuffer = new HashMap<>();
private boolean documentTerminated = false;
private boolean allowMapEntryRedefinition = false;
private boolean allowAnchorRedefinition = false;

Object currentYamlNode;

Field currentField;
Deque<Object> nodesStack = new ArrayDeque<>();
Stack<Map<String, Field>> fieldHierarchy = new Stack<>();
Expand All @@ -95,9 +94,21 @@ public static class ComposerState {
Stack<ParserContext> parserContexts = new Stack<>();
int unionDepth = 0;
boolean rootValueInitialized = false;

public ComposerState(ParserState parserState) {
final Types.YAMLSchema schema;
final boolean allowAnchorRedefinition;
final boolean allowMapEntryRedefinition;
final boolean allowDataProjection;
final boolean nilAsOptionalField;
final boolean absentAsNilableType;

public ComposerState(ParserState parserState, OptionsUtils.ReadConfig readConfig) {
this.parserState = parserState;
this.schema = readConfig.schema();
this.allowAnchorRedefinition = readConfig.allowAnchorRedefinition();
this.allowMapEntryRedefinition = readConfig.allowMapEntryRedefinition();
this.allowDataProjection = readConfig.allowDataProjection();
this.nilAsOptionalField = readConfig.nilAsOptionalField();
this.absentAsNilableType = readConfig.absentAsNilableType();
}

public int getLine() {
Expand Down Expand Up @@ -177,7 +188,17 @@ public Object verifyAndConvertToUnion(Object json) {
if (unionDepth > 0) {
return json;
}
return JsonTraverse.traverse(json, expectedTypes.peek());
BMap<BString, Object> options = ValueCreator.createMapValue();
BMap<BString, Object> allowDataProjectionMap = ValueCreator.createMapValue();
if (!allowDataProjection) {
options.put(Constants.ALLOW_DATA_PROJECTION, false);
} else {
allowDataProjectionMap.put(Constants.NIL_AS_OPTIONAL_FIELD, nilAsOptionalField);
allowDataProjectionMap.put(Constants.ABSENT_AS_NILABLE_TYPE, absentAsNilableType);
options.put(Constants.ALLOW_DATA_PROJECTION, allowDataProjectionMap);
}

return JsonTraverse.traverse(json, options, expectedTypes.peek());
}

private void finalizeObject() {
Expand Down Expand Up @@ -270,12 +291,14 @@ public enum ParserContext {
* Parses the contents in the given {@link Reader} and returns subtype of anydata value.
*
* @param reader reader which contains the YAML content
* @param options represent the options that can be used to modify the behaviour of conversion
* @param expectedType Shape of the YAML content required
* @return subtype of anydata value
* @throws BError for any parsing error
*/
public static Object parse(Reader reader, Type expectedType) throws BError {
ComposerState composerState = new ComposerState(new ParserState(reader, expectedType));
public static Object parse(Reader reader, BMap<BString, Object> options, Type expectedType) throws BError {
OptionsUtils.ReadConfig readConfig = OptionsUtils.resolveReadConfig(options);
ComposerState composerState = new ComposerState(new ParserState(reader, expectedType), readConfig);
composerState.handleExpectedType(expectedType);
try {
return parseDocument(composerState);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,9 @@ public class Constants {
public static final BString SCHEMA = StringUtils.fromString("schema");
public static final BString IS_STREAM = StringUtils.fromString("isStream");
public static final BString FLOW_STYLE = StringUtils.fromString("flowStyle");
public static final BString ALLOW_ANCHOR_REDEFINITION = StringUtils.fromString("allowAnchorRedefinition");
public static final BString ALLOW_MAP_ENTRY_REDEFINITION = StringUtils.fromString("allowMapEntryRedefinition");
public static final BString ALLOW_DATA_PROJECTION = StringUtils.fromString("allowDataProjection");
public static final BString NIL_AS_OPTIONAL_FIELD = StringUtils.fromString("nilAsOptionalField");
public static final BString ABSENT_AS_NILABLE_TYPE = StringUtils.fromString("absentAsNilableType");
}
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,19 @@ public class JsonTraverse {

private static final ThreadLocal<JsonTree> tlJsonTree = ThreadLocal.withInitial(JsonTree::new);

public static Object traverse(Object json, Type type) {
public static Object traverse(Object json, BMap<BString, Object> options, Type type) {
JsonTree jsonTree = tlJsonTree.get();
try {
Object allowDataProjection = options.get(Constants.ALLOW_DATA_PROJECTION);
if (allowDataProjection instanceof Boolean) {
jsonTree.allowDataProjection = false;
} else if (allowDataProjection instanceof BMap<?, ?>) {
jsonTree.allowDataProjection = true;
jsonTree.absentAsNilableType =
(Boolean) ((BMap<?, ?>) allowDataProjection).get(Constants.ABSENT_AS_NILABLE_TYPE);
jsonTree.nilAsOptionalField =
(Boolean) ((BMap<?, ?>) allowDataProjection).get(Constants.NIL_AS_OPTIONAL_FIELD);
}
return jsonTree.traverseJson(json, type);
} finally {
jsonTree.reset();
Expand All @@ -49,13 +59,18 @@ private static class JsonTree {
Deque<String> fieldNames = new ArrayDeque<>();
Type rootArray;
boolean allowDataProjection = true;
boolean nilAsOptionalField = false;
boolean absentAsNilableType = false;

void reset() {
currentField = null;
fieldHierarchy.clear();
restType.clear();
fieldNames.clear();
rootArray = null;
allowDataProjection = false;
nilAsOptionalField = false;
absentAsNilableType = false;
}

private Object traverseJson(Object json, Type type) {
Expand Down Expand Up @@ -162,7 +177,7 @@ private Object traverseMapValue(BMap<BString, Object> map, Object currentJsonNod
int currentFieldTypeTag = currentFieldType.getTag();
Object mapValue = map.get(key);

if (!currentFieldType.isNilable() && mapValue == null
if (nilAsOptionalField && !currentFieldType.isNilable() && mapValue == null
&& SymbolFlags.isFlagOn(currentField.getFlags(), SymbolFlags.OPTIONAL)) {
continue;
}
Expand Down Expand Up @@ -250,7 +265,7 @@ private void addRestField(Type restFieldType, BString key, Object jsonMember, Ob

private void checkOptionalFieldsAndLogError(Map<String, Field> currentField) {
currentField.values().forEach(field -> {
if (field.getFieldType().isNilable()) {
if (field.getFieldType().isNilable() && absentAsNilableType) {
return;
}
if (SymbolFlags.isFlagOn(field.getFlags(), SymbolFlags.REQUIRED)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,26 @@ public static WriteConfig resolveWriteOptions(BMap<BString, Object> options) {
((Boolean) canonical), ((Boolean) useSingleQuotes), ((Boolean) forceQuotes),
Types.YAMLSchema.valueOf(((BString) schema).getValue()), ((Boolean) isStream), ((Boolean) flowStyle));
}

public record ReadConfig(Types.YAMLSchema schema, boolean allowAnchorRedefinition,
boolean allowMapEntryRedefinition, boolean allowDataProjection,
boolean nilAsOptionalField, boolean absentAsNilableType) {
}

public static ReadConfig resolveReadConfig(BMap<BString, Object> options) {
Object schema = options.get(Constants.SCHEMA);
Object allowAnchorRedefinition = options.get(Constants.ALLOW_ANCHOR_REDEFINITION);
Object allowMapEntryRedefinition = options.get(Constants.ALLOW_MAP_ENTRY_REDEFINITION);
Object allowDataProjection = options.get(Constants.ALLOW_DATA_PROJECTION);
if (allowDataProjection instanceof Boolean) {
return new ReadConfig(Types.YAMLSchema.valueOf(((BString) schema).getValue()),
((Boolean) allowAnchorRedefinition), ((Boolean) allowMapEntryRedefinition), false, false, false);
}
Object nilAsOptionalField = ((BMap<?, ?>) allowDataProjection).get(Constants.NIL_AS_OPTIONAL_FIELD);
Object absentAsNilableType = ((BMap<?, ?>) allowDataProjection).get(Constants.ABSENT_AS_NILABLE_TYPE);

return new ReadConfig(Types.YAMLSchema.valueOf(((BString) schema).getValue()),
((Boolean) allowAnchorRedefinition), ((Boolean) allowMapEntryRedefinition), true,
((Boolean) nilAsOptionalField), ((Boolean) absentAsNilableType));
}
}