Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comparing changes from develop to framework-refactor-entry-types #168

Closed
wants to merge 38 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
09417cb
correcting typo in index.md
costero-e Jun 21, 2023
4c5527e
doc comma-contatenation
mbaudis Jul 3, 2023
4d8c40b
re-activating mermaid2 plugin
mbaudis Jul 10, 2023
597d171
fixing website mkdocs action
mbaudis Jul 10, 2023
fb8bfc5
Update framework.md
mbaudis Jul 11, 2023
89e3b9d
Update framework.md
mbaudis Jul 11, 2023
a793c8c
FAQ, Boolean (non-)use
mbaudis Aug 8, 2023
898b22b
Update FAQ.md
mbaudis Aug 8, 2023
8680e9f
Update FAQ.md
mbaudis Aug 8, 2023
c09c3ef
updating branch organization description
costero-e Aug 17, 2023
8c1340d
Removing the AND exception note
mbaudis Sep 20, 2023
394a409
Update filters.md
mbaudis Sep 20, 2023
a45ccc7
provide "required" properties for the "age" and "ageRange" schemas
redmitry Oct 6, 2023
1f6fb0b
comment changes
redmitry Oct 10, 2023
7c64fdf
Merge pull request #113 from redmitry/invalid-timeElement-fix
costero-e Oct 10, 2023
401a0cb
Update filters.md
mbaudis Oct 10, 2023
c1e8f5f
Update variant-queries.md
mbaudis Oct 31, 2023
4921810
BeaconGeneQuery-graphics
mbaudis Nov 9, 2023
f1f6265
Update BeaconGeneQuery-graphics.png
mbaudis Nov 12, 2023
c1ccadc
query options first order nav links
mbaudis Nov 14, 2023
edf561e
filtering terms results example listed
costero-e Mar 20, 2024
1447418
fix "examples" beaconFilteringTermsResults.json
redmitry Mar 20, 2024
ce1c82d
Merge pull request #119 from ga4gh-beacon/filteringTerms_filteringTer…
redmitry Mar 21, 2024
db307ba
Merge branch 'main' into develop
costero-e Mar 21, 2024
75850c3
Merge pull request #118 from ga4gh-beacon/develop
costero-e Mar 21, 2024
d3c2692
link fix and different TOC style
mbaudis Mar 22, 2024
aaf2028
Fix file naming conflict error in schemas-md on macOS APFS (case-inse…
mrueda Mar 26, 2024
f2f08ca
Remove clashing files
mrueda Mar 26, 2024
9aefbed
Merge pull request #122 from mrueda/website-docs
mbaudis Mar 26, 2024
b154dda
complexValue required relocated to correct level
costero-e Apr 11, 2024
afaf76a
Merge pull request #124 from ga4gh-beacon/hotfix_complexValue-required
costero-e Apr 12, 2024
f5be698
adding github actions demo file
costero-e May 3, 2024
7de0b56
Merge branch 'develop' into website-docs
costero-e May 3, 2024
58e68af
Merge pull request #127 from ga4gh-beacon/website-docs
mbaudis May 3, 2024
d2ee94b
Merge branch 'develop' into main
mbaudis Jul 6, 2024
cfcc85d
Hot Fix to align with json
mbaudis Jul 6, 2024
4db7f83
Merge pull request #132 from ga4gh-beacon/complex-value-required-src-…
costero-e Jul 10, 2024
efec4c3
Merge pull request #131 from ga4gh-beacon/main
costero-e Jul 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 0 additions & 18 deletions .github/workflows/github-actions-demo-yml

This file was deleted.

19 changes: 19 additions & 0 deletions .github/workflows/mk-docs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: mk-progenetix-docs
on:
push:
branches:
- website-docs
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.x
- run: pip install mkdocs-material
- run: pip install mkdocs-macros-plugin
- run: pip install pymdown-extensions
- run: pip install mkdocs-mermaid2-plugin
- run: pip install mdx_gh_links
- run: mkdocs gh-deploy --force
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,3 @@ site
.DS_Store
models/.DS_Store
/.vs
docs/schemas-md
2 changes: 1 addition & 1 deletion bin/SCHEMAS2MD.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ _NB:_ The script was built to work with the Beacon v2 Model schemas and the auth

_NB:_ The decission to take YAMLs (and not JSON) as an input is deliberate and made by the author.

_NB:_ The script only processes the `Terms` nested **up to 3 degrees of hierarchy**. Before Adoption of VRS/PHX that limit was OK.
_NB:_ The script only processes the `Terms` nested **up to 3 degrees of hierarchy**. Before Adoption of VRS/PXF that limit was OK.

_NB:_ The script also includes the Beacon v2 Models examples from [beacon-v2 repo](https://github.com/ga4gh-beacon/beacon-v2) in JSON format.

Expand Down
File renamed without changes.
60 changes: 36 additions & 24 deletions bin/beacon_yaml2md.pl
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
#
# Script to convert Beacon v2 Models schemas to Markdown tables
#
# Last Modified: May/05/2022
# Last Modified: Mar/26/2024
#
# Version 2.0.0
#
# Copyright (C) 2021-2022 Manuel Rueda (manuel.rueda@crg.eu)
# Copyright (C) 2021-2024 Manuel Rueda (manuel.rueda@cnag.eu)
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
Expand Down Expand Up @@ -236,6 +236,10 @@ sub yaml2md_obj {
# We parse $yaml to get paths and more...
my ( $base, $dir, $ext ) = fileparse( $yaml, '.yaml' );
$ext =~ s/\.yaml/.md/;

# Ad hoc fix for two files that have same namex except for uc/lc
# AgeRange == ageRange and Value == value on MacOS cwAPFS (Case insensitive)
$base = $base . '_PXF' if ( $base eq 'AgeRange' || $base eq 'Value' );
my $file = catfile( $mo_dir, $base . $ext ); # Note -> $base.$ext
write_file( $file, $out_str );

Expand Down Expand Up @@ -278,11 +282,11 @@ sub yaml_slicer {
# one YAML file for each property and then re-use code from the 'main' schema

##########################################
# **** Note about VRS / PHX adoption *** #
# **** Note about VRS / PXF adoption *** #
##########################################

# The adoption of those standards had technical implications. The script expects objects to have
# <key> for the object and then <properties>. VRS/PHX follow JSON schemas that include /oneOf allOf anyOf/
# <key> for the object and then <properties>. VRS/PXF follow JSON schemas that include /oneOf allOf anyOf/
# plus other complex intructions such as <if:> <else:>.
# This becomes a real challenge with $ref as, for instance, in <g_v.variation> we can not find the key for
# 'MolecularVariation', 'SystemicVariation', 'LegacyVariation'
Expand Down Expand Up @@ -352,7 +356,7 @@ sub yaml_slicer {
sub table_content {

my ( $yaml_properties, $ra_properties, $headers, $obj, $link ) = @_;
my @lc_headers = map { lc } @$headers; # Copy array uc to avoid modifying original $ref
my @lc_headers = map { lc } @$headers; # Copy array uc to avoid modifying original $ref
my $out_str = '';

#---------------------------------------------------------|
Expand Down Expand Up @@ -394,10 +398,10 @@ sub table_content {
if $header eq 'example';

# Slice differentely if $object->{type} eq 'array'
if ($object->{type} eq 'array' ) {
for ('description', 'properties'){
$value_header = $object->{items}{$_} if $header eq $_;
}
if ( $object->{type} eq 'array' ) {
for ( 'description', 'properties' ) {
$value_header = $object->{items}{$_} if $header eq $_;
}
}

# Now convert data structure to string
Expand Down Expand Up @@ -454,7 +458,7 @@ sub ref2str {

# string or undef
else {
$out_str = defined $data->[0] ? join ', ', @$data : 'NA'; # Note ', ' to allow HTML column rendering
$out_str = defined $data->[0] ? join ', ', @$data : 'NA'; # Note ', ' to allow HTML column rendering
}
}
elsif ( ref $data eq 'HASH' ) {
Expand All @@ -480,15 +484,20 @@ sub add_external_links {
my ( $tmp_str, $key ) = @_;

# Note: This is an ad hoc solution to fix errors with deeply-nested data
my @phx = qw( typedQuantities days weeks Quantity high low);
my @vrs = qw(_id state type CURIE Location);
my @pxf = qw( typedQuantities days weeks Quantity high low);
my @vrs = qw(_id state type CURIE Location);
my @framework = ("ontologyTerm");
return ( any { ( $_ eq $key ) } @phx )

return ( any { ( $_ eq $key ) } @pxf )
? "[$key](https://phenopacket-schema.readthedocs.io/en/latest/building-blocks.html)"
: ( any { ( $_ eq $key ) } @vrs )
? "[$key](https://vrs.ga4gh.org/en/stable/terms_and_model.html#$key)"
: ( any { ( $_ eq $key ) } @framework )
? "[$key](https://github.com/ga4gh-beacon/beacon-v2/blob/main/framework/src/common/$key.yaml)"
: ( any { ( $_ eq $key ) } @framework )
? "[$key](https://github.com/ga4gh-beacon/beacon-v2/blob/main/framework/src/common/$key.yaml)"

# NB: Ad hoc solution for properties having equal name (lc)
: ( $key eq 'AgeRange' || $key eq 'Value' )
? "[$key]($tmp_str/${key}_PXF.md)"
: "[$key]($tmp_str/$key.md)";
}

Expand Down Expand Up @@ -588,7 +597,7 @@ sub create_str_yaml {

## ontologyTerm.yaml is needed due to a bug with jsonref2json.js that overrided "parent" <description> field

my $str_ontologyTerm = <<EOF;
my $str_ontologyTerm = <<EOF;
---
additionalProperties: true
description: Definition of an ontology term.
Expand Down Expand Up @@ -676,10 +685,10 @@ sub parse_json_keywords {
'variation' =>
[ 'MolecularVariation', 'SystemicVariation', 'LegacyVariation' ],
'SystemicVariation' => ['CopyNumber'],
'MolecularVariation' => [ 'Allele', 'Haplotype' ],
'location' => [ 'CURIE', 'Location' ],
'MolecularVariation' => [ 'Allele', 'Haplotype' ],
'location' => [ 'CURIE', 'Location' ],
'state' => [ 'SequenceState', 'SequenceExpression' ],
'Value' => [ 'Quantity', 'ontologyTerm' ]
'Value' => [ 'Quantity', 'ontologyTerm' ]
};

# We'll be checking <oneOf allOf anyOf>
Expand All @@ -699,14 +708,17 @@ sub parse_json_keywords {
# my $const = $pointer->get("/$keyword/$property/$count/properties/type/const");
# $tmp_hash->{properties}{$const} = $elements;
#} else{
my $tmp_term = ( $pointer->contains("/$keyword/$count/title") && $pointer->get("/$keyword/$count/title") ne 'Ontology Term' )
my $tmp_term =
( $pointer->contains("/$keyword/$count/title")
&& $pointer->get("/$keyword/$count/title") ne
'Ontology Term' )
? $pointer->get("/$keyword/$count/title")
: @{ $terms->{$property} }[$count];
$tmp_hash->{properties}{$tmp_term} = $elements if $tmp_term; # Ad-hoc some terms appear duplicated and come empty....
#}
$tmp_hash->{properties}{$tmp_term} = $elements if $tmp_term; # Ad-hoc some terms appear duplicated and come empty....
#}
$count++;
}
$data = $tmp_hash; # Adding new reference
$data = $tmp_hash; # Adding new reference
}
}
return $data;
Expand Down Expand Up @@ -872,7 +884,7 @@ =head1 HOW TO RUN BEACON_YAML2MD

I<NB:> The decission to take YAMLs (and not JSON) as an input is deliberate and made by the author.

I<NB:> The script only processes the C<Terms> nested B<up to 3 degrees of hierarchy>. Before Adoption of VRS/PHX that limit was OK.
I<NB:> The script only processes the C<Terms> nested B<up to 3 degrees of hierarchy>. Before Adoption of VRS/PXF that limit was OK.

I<NB:> The script also includes the Beacon v2 Models examples from L<beacon-v2 repo|https://github.com/ga4gh-beacon/beacon-v2> in JSON format.

Expand Down
11 changes: 6 additions & 5 deletions bin/transform_json2md.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
#
# Script to convert Beacon v2 Models to Markdown
#
# Last Modified: Jul/20/2022
# Last Modified: Mar/26/2022
#
# Version 2.0.0
#
# Copyright (C) 2021-2022 Manuel Rueda (manuel.rueda@crg.eu)
# Copyright (C) 2021-2024 Manuel Rueda (manuel.rueda@cnag.eu)
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
Expand All @@ -26,11 +26,12 @@
set -eu
mod_dir=../models/json/beacon-v2-default-model
fwk_dir=../framework/json
adhoc_url='https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/bin/adhoc'
#adhoc_url='https://raw.githubusercontent.com/g4gh-beacon/beacon-v2/main/bin/adhoc'
adhoc_url='https://raw.githubusercontent.com/mrueda/beacon-v2/main/bin/adhoc'
out_dir=./deref_schemas
jsonref='node ./jsonref2json.js'
yaml2md=./beacon_yaml2md.pl
yaml2json='perl -MYAML -MJSON -0777 -wnl -e'
yaml2json='perl -MYAML::XS -MJSON::XS -0777 -wnl -e'

mkdir -p $out_dir/obj

Expand Down Expand Up @@ -75,7 +76,7 @@ do
rm $out_dir/$schema/defaultSchema.mod.json

echo "Transforming $schema JSON to YAML ..."
$yaml2json 'print YAML::Dump(decode_json($_))' $out_dir/$schema/defaultSchema.json | perl -pe 's/ \*(\d+)$/ $1/' > $out_dir/$schema/defaultSchema.yaml
$yaml2json 'print YAML::XS::Dump(decode_json($_))' $out_dir/$schema/defaultSchema.json | perl -pe 's/ \*(\d+)$/ $1/' > $out_dir/$schema/defaultSchema.yaml

echo "---"
done
Expand Down
11 changes: 11 additions & 0 deletions docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,5 +149,16 @@
representations and will be adopted by Beacon v2.n after reaching a release state.


??? faq "Does the Beacon protocol support Boolean expressions?<a id="boolean-logic"> </a>"

No (...but). Beacon queries as of v2 always assume a logical **AND** between query parameters
and individual filters, _i.e._ all conditions have to be met. There is currently
no support for Boolean expressions.
However, a logical exception is the use of multiple filters for the same parameter which
a Beacon implementation should treat as a logical **OR** since they otherwise
would fail in most instances. E.g. the query using `NCIT:C3493` and `NCIT:C2926`
(mapped against `biosample.histological_diagnosis.id`) would match both
_Lung Non-Small Cell Carcinoma_ (NCIT:C2926) and _Lung Squamous Cell Carcinoma_
(NCIT:C3493) which are exclusive diagnoses.


112 changes: 111 additions & 1 deletion docs/code-organization.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,117 @@ Changes to the Markdown files in the `/docs` directory (and its children) will i

The `gh-pages` branch is generated from the `/docs` directory through its `mkdocs` workflow and contains the website itself. **Do not edit**

## Topic branches
## Hotfix branches

These are the branches that are meant to fix some bugs that break the specifications and need an urgent fix. The branches are directly deployed towards the `main` branch.

### `entry-type-definitions-cleanup`

This branch aims to redefine the wording of the entry types in a way that makes more clear what are the entry types. As the old definitions could mislead to some confusions, this is a sensible change that is needed to directly affect the current modification.

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

### `hotfix_filteringTermsResults`

As the beaconFilteringTermsResults.json "type" property of the FilteringTerm is ambiguous, this branch has the objective to fix this and make filtering terms object to be operative and ready to point to the filtering terms type.

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

### `schema-urgent-fixes`

Some of the instances of the schema were missing attributes and other aspects that are required to make a beacon work. This is mandatory to be urgently fixed and this is the purpose of this branch deployment.

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

## Feature and subfeature branches

The feature branches are the branches that bring a lot of changes together to change some specific part of the specifications. They can be composed by different subfeature branches that commit to them or just have one single working branch. The feature branches commit to the `develop` branch as they are the changes that will lead beacon to be upgraded to a new version. The subfeature branches commit to their parent feature branch, as they are a microchange of all the aspects that have to change in a new feature that is being developed. The branches are named as the main purpose of them, so it is made very clear what is the working area of them and the subfeature branches add the name of the feature branch they belong as a prefix followed by an underscore. The list of these feature branches with their subfeature branches is the one shown next:

### `clean-up`

#### `clean-up_sticky_modified_files`

<b>Current status:</b><span style="color:blue">Last commit by @jrambla.</span>

#### `clean-up_renaming_entity_to_entry_type`

<b>Current status:</b><span style="color:blue">Added in framework-refactor-entry-type-definitions branch by @mbaudis and waiting for a PR to be accepted.</span>

#### `clean-up_decouple-model-framework-refs`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

#### `clean-up_move-CURIE-to-beaconCommonComponents`

<b>Current status:</b><span style="color:blue">This branch has not had any commit.</span>

#### `clean-up_de-snakify-token-names`

<b>Current status:</b><span style="color:blue">This branch has not had any commit.</span>

#### `clean-up_refactor-analysys-pipeline-info`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

### `resultsets`

#### `resultsets_remove-requirement-results-resultsCount`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

### `requestParameters`

#### `requestParameters_genomicVariations-fix`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

#### `requestParameters_list-parameters-with-comma`

<b>Current status:</b><span style="color:blue">This branch has not had any commit.</span>

### `phenopackets`

#### `phenopackets_standards-alignment`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

### `network`

#### `network_aggregator-and-networks-support`

<b>Current status:</b><span style="color:blue">This branch has not had any commit.</span>

#### `network_meta-add-aggregator-flag`

<b>Current status:</b><span style="color:blue">Last commit by @mbaudis.</span>

### `anyOf`

#### `anyOf_filteringTerms`

<b>Current status:</b><span style="color:blue">This branch has not had any commit.</span>

### `genomicVariations`

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

### `received-filters`

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

### `response-meta-add-warnings`

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>

### `framework-refactor-entry-type-definitions`

<b>Current status:</b><span style="color:blue">Waiting for PR to be accepted.</span>







==TBD==

Loading
Loading