Platform fixes, documentation updates and other small stuff (#405)

* Use and configure license-maven-plugin (org.honton.chas)

* First setup of distribution verification integration test

* Use Java 17 for compilation, updates of test dependencies, update license validation config

* Update comment on CacioTest annotation

* Cleanup

* Add generating fat jars for WhiteRabbit and RabbitInAHat; lock hsqldb version for Java 1.8

* Enforce Java 1.8 for distributed dependencies

* Update main.yml

Project now requires Java 17 to build. Should still produce java 8 (1.8) compatible artifacts though.

* Bump org.apache.avro:avro from 1.11.2 to 1.11.3 in /rabbit-core

Bumps org.apache.avro:avro from 1.11.2 to 1.11.3.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Use jdk8 classifier for hsqldb 2.7.x

* Exclude older version of hsqldb

* Fix image crop when using stem table

* Update stem table image

* Decrease size of table panel when using stem table.

Without this change, the table panel height is always higher than
needed (when using stem table), because the stem table is counted
as one of the items in the components list. It is however shown
separately at the top, which is already accounted for by the
stem table margin.

* Add snowflake support (#37)

* Refactor RichConnection into separate classes, and add an abstraction for the JDBC connection. Implement a Snowflake connection with this abstraction

* Add unit tests for SnowflakeConnector

* Added Snowflake support for SourceDataScan; added minimal test for it; some refactorings to move database responsibility to rabbit-core/databases

* Move more database details to rabbit-core/databases

* Clearer name for method

* Ignore snowflake.env

* Create PostgreSQL container in the TestContainers way

* Refactored Snowflake tests + a bit of documentation

* Fix Snowflake test for Java 17, and make it into an automated integration test instead of a unit test

* Remove duplicate postgresql test

* Make TestContainers based database tests into automated integration tests

* Suppress some warnings when generating fat jars

* Let autimatic integration tests fail when docker is not available

* Allow explicit skipping of Snowflake integration tests

* Added tests for Snowflake, delimited text files

* Switch to fully verifying the scan results against a reference version (v0.10.7)

* Working integration test for Snowflake, and some refactorings

* Some proper logging, small code improvements and cleanup

* Remove unused interface

* Added tests, some changes to support testing

* Make automated test work reliably (way too many changes, sorry)

* Rudimentary support for Snowflake authenticator parameter (untested)

* review xmlbeans dependencies, remove conflict

* extend integration test for distribution

* Restructuring database configuration. Work in process, but unit and integration tests all OK

* Restructuring database configuration 2/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 3/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 4/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 5/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 6/x. Still work in process, but unit and integration tests all OK

* Restructuring database configuration 7/x. Still work in process, but unit and integration tests all OK

* Intermezzo: get rid of the package naming error (upper case R in whiteRabbit)

* Intermezzo: code cleanup

* Snowflake is now working from the GUI. And many small refactorings, like logging instead of printing to stout/err

* Refactor DbType into an enum, get rid of DBChoice

* Move DbType and DbSettings classes into configuration subpackage

* Avoid using a manually destructured DbSettings object when creating a RochConnection object

* Code cleanup, remove unneeded Snowflake references

* Refactoring, code cleanup

* More refactoring, code cleanup

* More refactoring, code cleanup and documentation

* Make sure that order of databases in pick list in GUI is the same as before, and enforce completeness of that list in a test

* Add/update copyright headers

* Add line to verify that a tooltip is shown for a DBConnectionInterface implementing class

* Test distribution for Snowflake JDBC issue with Java 17

* cleanup of build files

* Add verification that all JDBC drivers are in the distributed package

* Add/improve error reporting for Snowflake

* Disable screenshottaker in GuiTestExtension, hoping that that is what blocks the build on github. Fingers crossed

* Better(?) naming for database interface and implementing class

* Use our own GUITestExtension class

---------

Co-authored-by: Jan Blom <[email protected]>

* Add mysql test (#38)

* Fixed a bug in the comparison for sort; let comparison report report all differences before failing

* Allow the user to specify the port for a MySQL server

* Add tests for a MySQL source database

* Add sas test (#39)

* Add automated regression tests for SAS files

* Fix problems with comparisons of test results to references

* create bypass for value mismatch that only shows up in github actions so far

* create bypass for value mismatch that only shows up in github actions so far, 2nd

* Pom updates to enable building on MacOS

* Prepare release (#40)

* Add warehouse/database handling to StorageHandler class

* Show stdout/stderr from distribution verification when there are errors

* Pom updates to enable building on MacOS

* Update dependencies as far as possible without code changes

* Update README.md

---------

Co-authored-by: Jan Blom <[email protected]>

* Update whiterabbit/src/main/java/org/ohdsi/whiterabbit/WhiteRabbitMain.java

The sample size should start disabled, as the calculateNumericStats checkbox is unchecked by default.

Co-authored-by: Maxim Moinat <[email protected]>

* Fixes from windows (#41)

* Fix problems blocking verification on Windows

* Avoid using bind mounts for TestContainers, copy files instead

* Remove file copy (was for debugging purposes)

* Oracle Tests: use the actual TestContainer hostname/ip address instead of localhost

* Remove debug print statement and stale imports

* Remove commented code

---------

Co-authored-by: Jan Blom <[email protected]>

* Use The Hyve fork of the caciocavello project (#42)

* Use The Hyve fork of the caciocavello project (Swing virtual graphics environment for testing) until the parent project has been fixed for JDK 18+

* Use updated cacio-tta version, should run fine when headless

* For developemnt, JDK versions 17-21 are supported

* Update docs (#44)

* Update documentation for Snowflake

* Add Snowflake.ini example file

* Add password field in Snowflake example

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Jan Blom <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Spayralbe <[email protected]>
Co-authored-by: Maxim Moinat <[email protected]>

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 White Rabbit and Rabbit in a Hat are structured as a Maven package. Contributions are welcome.
 
-While the software in the project can be executed with Java 8 (1.8), for development Java 17 is needed.
+While the software in the project can be executed with Java 8 (1.8), for development Java 17 (or higher, currently tested upto version 21) is needed.
 This has to do with test and verification dependencies that are not available in a version compatible with Java 8 .
 
 Please note that when using an IDE for development, source and target release must still be Java 8 (1.8) . This is enforced
@@ -14,11 +14,13 @@ To generate the files ready for distribution, run `mvn install`.
 
 When creating a pull request, please make sure the verification phase (`mvn verify`) does not fail.
 
+When contributing code, please make sure that `mvn verify` runs without errors. 
+
 ### Testing
 
 A number of unit and integration tests exist. The integration tests run only in the maven verification phase,
-(`mvn verify`) and depend on docker being available to the user running the verification. If docker is not available, the
-integration tests will fail.
+(`mvn verify`) and depend on Docker being available to the user running the verification. If Docker is not available, the
+integration tests will fail. 
 
 When adding test, please follow these conventions:
 
@@ -47,6 +49,12 @@ These are used for testing of the main White Rabbit and Rabbit in a Hat features
 | `riah_input`   | An example mapping file used to create the Rabbit in a Hat outputs.                                                                                                                                                                                                                                                                                      |
 | `riah_output`  | All export formats created by Rabbit in a Hat: as word, html, markdown, sql skeleton and the R TestFramework.<br> These are all generated from `riah_input/riah_mapping_example.gz`.                                                                                                                                                                     |
 
+### Database support
+
+If you are considering adding support for a new type of database, it is recommended to follow the pattern as used
+by the SnowflakeHandler class, which extends the StorageHandler interface. This way, the brand/database specific code
+is isolated into one class, instead of through the code paths that implement support for the 
+databases that were added earlier. This will lead to clearer code, that will also be easier to test and debug.
 
 ### Snowflake
 
@@ -68,4 +76,4 @@ and do not relate in any way to any production environment.
 The schema should not contain any tables when the test is started.
 
 It is possible to skip the Snowflake tests without failing the build by passing
-`-Dohdsi.org.whiterabbit.skip_snowflake_tests=1` to maven.
+`-Dohdsi.org.whiterabbit.skip_snowflake_tests=1` to maven.

README.md

@@ -44,7 +44,7 @@ Requires Java 1.8 or higher for running, and read access to the database to be s
 
 Dependencies
 ============
-For the distributable packages, the only requirement is Java 8. For building the package, Java 17 and Maven are needed.
+For the distributable packages, the only requirement is Java 8. For building the package, Java 17+ and Maven are needed.
 
 Getting Started
 ===============

docs/ReadMe.html

@@ -377,24 +377,14 @@
 
 <div id="white-rabbit-documentation-readme" class="section level1">
 <h1>White Rabbit Documentation Readme</h1>
-<p>This folder contains the raw (<code>.md</code>) and rendered
-(<code>.html</code>) documentation of WhiteRabbit. The documentation is
-renderd with the R package <code>rmarkdown</code> and used for the <a
-href="https://ohdsi.github.io/WhiteRabbit/">github.io page</a>.</p>
+<p>This folder contains the raw (<code>.md</code>) and rendered (<code>.html</code>) documentation of WhiteRabbit. The documentation is renderd with the R package <code>rmarkdown</code> and used for the <a href="https://ohdsi.github.io/WhiteRabbit/">github.io page</a>.</p>
 <div id="contribute" class="section level2">
 <h2>Contribute</h2>
-<p>Contributions to the documentation are very welcome and even a must
-when new features are implemented. To update the documentation, edit one
-of the following markdown files or create a new markdown file: - <a
-href="/docs/WhiteRabbit.md">WhiteRabbit.md</a> - <a
-href="/docs/RabbitInAHat.md">RabbitInAHat.md</a> - <a
-href="/docs/riah_test_framework.md">riah_test_framework.md</a> - <a
-href="/docs/best_practices.md">best_practices.md</a></p>
+<p>Contributions to the documentation are very welcome and even a must when new features are implemented. To update the documentation, edit one of the following markdown files or create a new markdown file: - <a href="/docs/WhiteRabbit.md">WhiteRabbit.md</a> - <a href="/docs/RabbitInAHat.md">RabbitInAHat.md</a> - <a href="/docs/riah_test_framework.md">riah_test_framework.md</a> - <a href="/docs/best_practices.md">best_practices.md</a></p>
 </div>
 <div id="render-html" class="section level2">
 <h2>Render html</h2>
-<p>To generate the site from markdown files, run the following R code
-with the <code>./docs</code> folder as working directory.</p>
+<p>To generate the site from markdown files, run the following R code with the <code>./docs</code> folder as working directory.</p>
 <pre class="r"><code>#devtools::install_github(&quot;ropenscilabs/icon&quot;)
 library(rmarkdown)
 rmarkdown::render_site()</code></pre>