Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data.yaml module #4

Merged
merged 99 commits into from
Aug 2, 2024
Merged
Show file tree
Hide file tree
Changes from 96 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
56b7c0d
Add data.yaml initial implementation
LakshanWeerasinghe Feb 19, 2024
16e8ba4
Impl yaml lexer
LakshanWeerasinghe Feb 22, 2024
cf4eddb
Impl yaml parser
LakshanWeerasinghe Apr 21, 2024
7a49666
Add tests for basic types
LakshanWeerasinghe Apr 22, 2024
5e05d50
Fix issue with apply projection
LakshanWeerasinghe Apr 24, 2024
bef0b7d
[Automated] Update the native jar versions
LakshanWeerasinghe Apr 24, 2024
7ade15b
Add tests for yaml document parsing
LakshanWeerasinghe Apr 29, 2024
c46037c
Implement toYamlString api
LakshanWeerasinghe May 2, 2024
d2d80a9
Handle union type as expected type
LakshanWeerasinghe May 2, 2024
4c6914c
Add JsonTraverse to create values from json
LakshanWeerasinghe May 4, 2024
858a6a6
Add tests for union type as expected type
LakshanWeerasinghe May 4, 2024
92c443a
Support yaml input as byte[] stream
LakshanWeerasinghe May 4, 2024
216068f
Handle runtime errors as BErrors
LakshanWeerasinghe May 5, 2024
fb20398
Add negative tests
LakshanWeerasinghe May 6, 2024
0a30bde
Merge branch 'to-yaml-string' into yaml-data-impl
LakshanWeerasinghe May 7, 2024
30911de
Add options for parsing
LakshanWeerasinghe May 7, 2024
9d9fac6
Add test after disabling projection
LakshanWeerasinghe May 7, 2024
db5d271
Support anchors for maps and sequences
LakshanWeerasinghe May 8, 2024
9ee33b9
Add tests with anchors
LakshanWeerasinghe May 8, 2024
cc72c28
Add tag resolution for json and core schemas
LakshanWeerasinghe May 8, 2024
bde792e
Add tests for tag resolution
LakshanWeerasinghe May 8, 2024
d0cf4cb
Add tests for escaped characters
LakshanWeerasinghe May 8, 2024
bac9a4c
Fix issue with handling directives
LakshanWeerasinghe May 9, 2024
15aa7da
Support intersection types and yaml stream parsing
LakshanWeerasinghe May 13, 2024
816eaa8
Impl union and intersection exp type yaml streams
LakshanWeerasinghe May 13, 2024
2c32d86
Fix issue with handling indents
LakshanWeerasinghe May 14, 2024
2641189
Fix issue handling espcaed white spaces
LakshanWeerasinghe May 14, 2024
e78c360
Add tests for parsing yaml streams
LakshanWeerasinghe May 14, 2024
a3dac22
Add tests for toYamlString
LakshanWeerasinghe May 14, 2024
1e2b0dc
[Automated] Update the native jar versions
LakshanWeerasinghe May 14, 2024
8097874
Add Name annotation
LakshanWeerasinghe May 15, 2024
1ee0272
Impl compiler plugin
LakshanWeerasinghe May 15, 2024
4db611e
Fix validating var declrations
LakshanWeerasinghe May 15, 2024
4b51713
Add compiler plugin tests
LakshanWeerasinghe May 15, 2024
b7d3d60
Fix issue handling union expected types
LakshanWeerasinghe May 15, 2024
813b7e5
Add tests for yaml stream parsing
LakshanWeerasinghe May 15, 2024
abc82d1
Add licence headers
LakshanWeerasinghe May 16, 2024
12fd41d
Refactor code
LakshanWeerasinghe May 16, 2024
cc34639
Merge branch 'yaml-data-impl' into yml-data
LakshanWeerasinghe May 17, 2024
00c3f68
[Automated] Update the native jar versions
LakshanWeerasinghe May 17, 2024
fdceef7
Update .gitattributes file
LakshanWeerasinghe May 20, 2024
db993a7
Add codecov.yml file
LakshanWeerasinghe May 20, 2024
4153e1a
Fix issue handling new line characters
LakshanWeerasinghe May 24, 2024
9f1557d
Add graalvm resource config
LakshanWeerasinghe May 28, 2024
177fbdc
Support parsing yaml stream using a single api
LakshanWeerasinghe May 29, 2024
aae0d27
Support tuple member re-order for yaml streams
LakshanWeerasinghe May 30, 2024
f787d68
Fix handling union types for yaml streams
LakshanWeerasinghe Jun 3, 2024
5ff8ca2
Fix handling union types in tuple mem
LakshanWeerasinghe Jun 4, 2024
18ee860
Add tests for tuple expected type
LakshanWeerasinghe Jun 4, 2024
d573cd6
[Automated] Update the native jar versions
LakshanWeerasinghe Jun 4, 2024
43849a2
Refactor code
LakshanWeerasinghe Jun 4, 2024
f209fc0
Refactor code
LakshanWeerasinghe Jun 4, 2024
9437104
Refactor yaml stream composing
LakshanWeerasinghe Jun 4, 2024
7ee195b
Add Module.md and Package.md files
LakshanWeerasinghe Jun 4, 2024
d62c994
Update READEME.md file
LakshanWeerasinghe Jun 4, 2024
a05ef8e
[Automated] Update the native jar versions
LakshanWeerasinghe Jun 5, 2024
5dc871f
Update ballerina version
LakshanWeerasinghe Jun 5, 2024
499bf19
Add more tests for projection options
LakshanWeerasinghe Jun 6, 2024
7123aac
Remove unreachable functions
LakshanWeerasinghe Jun 6, 2024
15de097
Add compiler plugin tests
LakshanWeerasinghe Jun 7, 2024
5388c3a
Add tests for parsing tag handles
LakshanWeerasinghe Jun 7, 2024
610124a
Remove empty files
LakshanWeerasinghe Jun 10, 2024
f169ff1
Merge pull request #2 from LakshanWeerasinghe/yml-data
LakshanWeerasinghe Jun 10, 2024
6f236fc
Disable failing compiler-plugin test
LakshanWeerasinghe Jun 10, 2024
cf25f61
Add more tests for toYamlString
LakshanWeerasinghe Jun 10, 2024
df8e2b6
Add tag directive tests
LakshanWeerasinghe Jun 10, 2024
f0f19b6
Add tests for escaped characters
LakshanWeerasinghe Jun 10, 2024
5a78c05
Add private constructors for utils
LakshanWeerasinghe Jun 10, 2024
173f5d7
Add more tests
LakshanWeerasinghe Jun 10, 2024
b680098
Add more tag resolution tests
LakshanWeerasinghe Jun 10, 2024
5a3ca95
Fix issue with strict tuple ordering
LakshanWeerasinghe Jun 11, 2024
3658e38
Add more negative tests
LakshanWeerasinghe Jun 12, 2024
2b96a07
Fix block header issue and add tests
LakshanWeerasinghe Jun 12, 2024
4248895
Add tests for handling empty lines
LakshanWeerasinghe Jun 12, 2024
79e825c
Add tests for handling quoted scalars
LakshanWeerasinghe Jun 12, 2024
8fe8114
Add negative tests
LakshanWeerasinghe Jun 16, 2024
783dc2c
Add more tests
LakshanWeerasinghe Jun 17, 2024
6108aef
Add tests for streams with basic types
LakshanWeerasinghe Jun 18, 2024
d8d1338
Address review suggestions for compiler plugin
LakshanWeerasinghe Jun 18, 2024
c80e000
Add more compiler plugin tests
LakshanWeerasinghe Jun 18, 2024
36f9fad
Improve utils functions
LakshanWeerasinghe Jun 18, 2024
95506c8
Address review suggestion
LakshanWeerasinghe Jul 9, 2024
b85242d
Add api examples
LakshanWeerasinghe Jul 9, 2024
a8824aa
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 22, 2024
6e4f3ab
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 22, 2024
2207ef5
Add constraint validation support
LakshanWeerasinghe Jul 22, 2024
4082f7f
Move ModuleUtils to base package
LakshanWeerasinghe Jul 24, 2024
8b27ffb
Add tests by disabling constrain validation
LakshanWeerasinghe Jul 24, 2024
4cad13d
Improve record documentation
LakshanWeerasinghe Jul 29, 2024
9874cf5
[Automated] Update the native jar versions
LakshanWeerasinghe Jul 29, 2024
6c10c63
Address review suggestion
LakshanWeerasinghe Jul 29, 2024
f90c207
Remove TypeUtils
LakshanWeerasinghe Jul 29, 2024
4e29be0
Add maven package publishing task
LakshanWeerasinghe Jul 30, 2024
8c5997e
Refactor options utils
LakshanWeerasinghe Jul 30, 2024
1f68831
Address review suggestions
LakshanWeerasinghe Jul 31, 2024
2b1ea84
Rename tupleReOrder config
LakshanWeerasinghe Jul 31, 2024
1258057
Updated the yaml stream parsing
LakshanWeerasinghe Jul 31, 2024
1c95d30
[Automated] Update the native jar versions
LakshanWeerasinghe Aug 1, 2024
31da9de
[Automated] Update the native jar versions
LakshanWeerasinghe Aug 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 74 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,78 @@
Ballerina YAML Data Library
===================
# Ballerina YAML Data Library

[![Build](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/build-timestamped-master.yml/badge.svg)](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/build-timestamped-master.yml)
[![codecov](https://codecov.io/gh/ballerina-platform/module-ballerina-data.yaml/branch/main/graph/badge.svg)](https://codecov.io/gh/ballerina-platform/module-ballerina-data.yaml)
[![Trivy](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/trivy-scan.yml/badge.svg)](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/trivy-scan.yml)
[![GraalVM Check](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/build-with-bal-test-graalvm.yml/badge.svg)](https://github.com/ballerina-platform/module-ballerina-data.yaml/actions/workflows/build-with-bal-test-graalvm.yml)
[![GitHub Last Commit](https://img.shields.io/github/last-commit/ballerina-platform/module-ballerina-data.yaml.svg)](https://github.com/ballerina-platform/module-ballerina-data.yaml/commits/master)
[![Github issues](https://img.shields.io/github/issues/ballerina-platform/ballerina-standard-library/module/data.yaml.svg?label=Open%20Issues)](https://github.com/ballerina-platform/ballerina-standard-library/labels/module%2Fdata.yaml)

LakshanWeerasinghe marked this conversation as resolved.
Show resolved Hide resolved
The Ballerina data.yaml library provides robust and flexible functionalities for working with YAML data within
Ballerina applications.
This library enables developers to seamlessly integrate YAML processing capabilities,
ensuring smooth data interchange and configuration management.

## Key Features

- **Versatile Input Handling**: Convert YAML input provided as strings, byte arrays, or streams of byte arrays into
Ballerina's anydata sub-types, facilitating flexible data processing.
- **Data Projection**: Efficiently project data from YAML documents and YAML streams,
allowing for precise data extraction and manipulation.
- **Ordered Data Representation**: Employ tuples to preserve the order of elements when dealing with
YAML document streams of unknown order, ensuring the integrity of data sequences.
- **Serialization**: Serialize Ballerina values into YAML-formatted strings, enabling easy generation of YAML content
from Ballerina applications for configuration files, data storage, or data exchange purposes.

## Usage

### Converting external YAML document to a record value

For transforming YAML content from an external source into a record value,
the `parseString`, `parseBytes`, `parseStream` functions can be used.
This external source can be in the form of a string or a byte array/byte-block-stream that houses the YAML data.
This is commonly extracted from files or network sockets. The example below demonstrates the conversion of an
YAML value from an external source into a record value.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
type Book record {|
string name;
string author;
int year;
|};
public function main() returns error? {
string yamlContent = check io:fileReadString("path/to/file.yaml");
Book book = check yaml:parseString(yamlContent);
io:println(book);
}
```

Make sure to handle possible errors that may arise during the file reading or YAML to anydata conversion process.
The `check` keyword is utilized to handle these errors,
but more sophisticated error handling can be implemented as per your requirements.

## Serialize anydata value to YAML

The serialization of anydata value into YAML-formatted strings can be done in the below way.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
public function main() returns error? {
json content = {
name: "Clean Code",
author: "Robert C. Martin",
year: 2008
};
string yamlString = check yaml:toYamlString(content);
io:println(yamlString);
}
```

The Ballerina YAML Data Library is a comprehensive toolkit designed to facilitate the handling and manipulation of
YAML data within Ballerina applications.

## Issues and projects

Expand Down
11 changes: 9 additions & 2 deletions ballerina/Ballerina.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,20 @@ authors = ["Ballerina"]
keywords = ["yaml"]
repository = "https://github.com/ballerina-platform/module-ballerina-data.yaml"
license = ["Apache-2.0"]
distribution = "2201.8.1"
distribution = "2201.10.0"

[platform.java17]
graalvmCompatible = true

[[platform.java17.dependency]]
groupId = "io.ballerina.stdlib"
groupId = "io.ballerina.lib"
artifactId = "yaml-native"
version = "0.1.0"
path = "../native/build/libs/data.yaml-native-0.1.0-SNAPSHOT.jar"


[[platform.java17.dependency]]
groupId = "io.ballerina.stdlib"
artifactId = "constraint-native"
version = "1.5.0"
path = "./lib/constraint-native-1.5.0.jar"
6 changes: 6 additions & 0 deletions ballerina/CompilerPlugin.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[plugin]
id = "constraint-compiler-plugin"
class = "io.ballerina.lib.data.yaml.compiler.YamlDataCompilerPlugin"

[[dependency]]
path = "../compiler-plugin/build/libs/data.yaml-compiler-plugin-0.1.0-SNAPSHOT.jar"
103 changes: 101 additions & 2 deletions ballerina/Dependencies.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,48 @@

[ballerina]
dependencies-toml-version = "2"
distribution-version = "2201.9.0"
distribution-version = "2201.10.0-20240722-112100-1b113f29"

[[package]]
org = "ballerina"
name = "constraint"
version = "1.5.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"}
]
modules = [
{org = "ballerina", packageName = "constraint", moduleName = "constraint"}
]

[[package]]
org = "ballerina"
name = "data.yaml"
version = "0.1.0"
dependencies = [
{org = "ballerina", name = "jballerina.java"}
{org = "ballerina", name = "constraint"},
{org = "ballerina", name = "io"},
{org = "ballerina", name = "jballerina.java"},
{org = "ballerina", name = "test"},
{org = "ballerina", name = "time"}
]
modules = [
{org = "ballerina", packageName = "data.yaml", moduleName = "data.yaml"}
]

[[package]]
org = "ballerina"
name = "io"
version = "1.6.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"},
{org = "ballerina", name = "lang.value"}
]
modules = [
{org = "ballerina", packageName = "io", moduleName = "io"}
]

[[package]]
org = "ballerina"
name = "jballerina.java"
Expand All @@ -26,3 +55,73 @@ modules = [
{org = "ballerina", packageName = "jballerina.java", moduleName = "jballerina.java"}
]

[[package]]
org = "ballerina"
name = "lang.__internal"
version = "0.0.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"},
{org = "ballerina", name = "lang.object"}
]

[[package]]
org = "ballerina"
name = "lang.array"
version = "0.0.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"},
{org = "ballerina", name = "lang.__internal"}
]

[[package]]
org = "ballerina"
name = "lang.error"
version = "0.0.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"}
]

[[package]]
org = "ballerina"
name = "lang.object"
version = "0.0.0"
scope = "testOnly"

[[package]]
org = "ballerina"
name = "lang.value"
version = "0.0.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"}
]

[[package]]
org = "ballerina"
name = "test"
version = "0.0.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"},
{org = "ballerina", name = "lang.array"},
{org = "ballerina", name = "lang.error"}
]
modules = [
{org = "ballerina", packageName = "test", moduleName = "test"}
]

[[package]]
org = "ballerina"
name = "time"
version = "2.4.0"
scope = "testOnly"
dependencies = [
{org = "ballerina", name = "jballerina.java"}
]
modules = [
{org = "ballerina", packageName = "time", moduleName = "time"}
]

67 changes: 67 additions & 0 deletions ballerina/Module.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Ballerina YAML Data Library

The Ballerina data.yaml library provides robust and flexible functionalities for working with YAML data within
Ballerina applications.
This library enables developers to seamlessly integrate YAML processing capabilities,
ensuring smooth data interchange and configuration management.

## Key Features

- **Versatile Input Handling**: Convert YAML input provided as strings, byte arrays, or streams of byte arrays into
Ballerina's anydata sub-types, facilitating flexible data processing.
- **Data Projection**: Efficiently project data from YAML documents and YAML streams,
allowing for precise data extraction and manipulation.
- **Ordered Data Representation**: Employ tuples to preserve the order of elements when dealing with
YAML document streams of unknown order, ensuring the integrity of data sequences.
- **Serialization**: Serialize Ballerina values into YAML-formatted strings, enabling easy generation of YAML content
from Ballerina applications for configuration files, data storage, or data exchange purposes.

## Usage

### Converting external YAML document to a record value

For transforming YAML content from an external source into a record value,
the `parseString`, `parseBytes`, `parseStream` functions can be used.
This external source can be in the form of a string or a byte array/byte-block-stream that houses the YAML data.
This is commonly extracted from files or network sockets. The example below demonstrates the conversion of an
YAML value from an external source into a record value.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
type Book record {|
string name;
string author;
int year;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int year;
string year;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo better to have string with constraints or unsinged integers

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having simpler examples for these sort of documentation will be better.

|};
public function main() returns error? {
string yamlContent = check io:fileReadString("path/to/<file-name>.yaml");
Book book = check yaml:parseString(yamlContent);
io:println(book);
}
```

Make sure to handle possible errors that may arise during the file reading or YAML to anydata conversion process.
The `check` keyword is utilized to handle these errors,
but more sophisticated error handling can be implemented as per your requirements.

## Serialize anydata value to YAML

The serialization of anydata value into YAML-formatted strings can be done in the below way.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
public function main() returns error? {
json content = {
name: "Clean Code",
author: "Robert C. Martin",
year: 2008
};
string yamlString = check yaml:toYamlString(content);
io:println(yamlString);
}
```
72 changes: 67 additions & 5 deletions ballerina/Package.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,67 @@
# module-ballerina-data.yaml
The Ballerina YAML Data Library is a comprehensive toolkit designed to facilitate the handling and manipulation of
YAML data within Ballerina applications.
It streamlines the process of converting YAML data to native Ballerina data types,
enabling developers to work with YAML content seamlessly and efficiently.
# Ballerina YAML Data Library

The Ballerina data.yaml library provides robust and flexible functionalities for working with YAML data within
Ballerina applications.
This library enables developers to seamlessly integrate YAML processing capabilities,
ensuring smooth data interchange and configuration management.

## Key Features

- **Versatile Input Handling**: Convert YAML input provided as strings, byte arrays, or streams of byte arrays into
Ballerina's anydata sub-types, facilitating flexible data processing.
- **Data Projection**: Efficiently project data from YAML documents and YAML streams,
allowing for precise data extraction and manipulation.
- **Ordered Data Representation**: Employ tuples to preserve the order of elements when dealing with
YAML document streams of unknown order, ensuring the integrity of data sequences.
- **Serialization**: Serialize Ballerina values into YAML-formatted strings, enabling easy generation of YAML content
from Ballerina applications for configuration files, data storage, or data exchange purposes.

## Usage

### Converting external YAML document to a record value

For transforming YAML content from an external source into a record value,
the `parseString`, `parseBytes`, `parseStream` functions can be used.
This external source can be in the form of a string or a byte array/byte-block-stream that houses the YAML data.
This is commonly extracted from files or network sockets. The example below demonstrates the conversion of an
YAML value from an external source into a record value.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
type Book record {|
string name;
string author;
int year;
|};
public function main() returns error? {
string yamlContent = check io:fileReadString("path/to/<file-name>.yaml");
Book book = check yaml:parseString(yamlContent);
io:println(book);
}
```

Make sure to handle possible errors that may arise during the file reading or YAML to anydata conversion process.
The `check` keyword is utilized to handle these errors,
but more sophisticated error handling can be implemented as per your requirements.

## Serialize anydata value to YAML

The serialization of anydata value into YAML-formatted strings can be done in the below way.

```ballerina
import ballerina/data.yaml;
import ballerina/io;
public function main() returns error? {
json content = {
name: "Clean Code",
author: "Robert C. Martin",
year: 2008
};
string yamlString = check yaml:toYamlString(content);
io:println(yamlString);
}
```
Loading
Loading