Synchronizes your data collection with GitHub and GitLab.
- Sync data collection from Git to DB
- Deal with multiple repositories
- Incremental updates
- Works with private or public repositories
Tuttle uses Gulp as its build tool which itself builds on NPM. To initialize the project and load dependencies run
npm i
Note: the
installcommands below assume that you have a local eXist-db running on port 8080. However the database connection can be modified in.existdb.json.
| Run | Description |
|---|---|
npm run build |
to just build Tuttle. |
npm run deploy |
To build and install Tuttle in one go |
The resulting xar(s) are found in the root of the project.
To run the local test suite you need an
- instance of eXist running on
localhost:8080and npmto be available in your path- a GitHub personal access token with read access to public repositories
- a gitlab personal access token with read access to public repositories
In CI these access tokens are read from environment variables. You can do the same with
export tuttle_token_tuttle_sample_data=<GITHUB_PAT>; \
export tuttle_token_gitlab_sample_data=<GITLAB_PAT>; \
path/to/startup.shAlternatively, you can modify /db/apps/tuttle/data/tuttle.xml and test/fixtures/alt-tuttle.xml to include your tokens. But remember to never commit them!
Run tests with npm test
Tuttle is configured in data/tuttle.xml.
New with version 2.0.0:
A commented example configuration is available data/tuttle-example-config.xml.
If you want to update tuttle your modified configuration file will be backed up to
/db/tuttle-backup/tuttle.xml and restored on installation of the new version.
Otherwise, when no back up of an existing config-file is found, the example configuration is copied to data/tuttle.xml.
Tip
When migrating from an earlier version you can copy your existing configuration to the backup location:
xmldb:copy-resource('/db/apps/tuttle/data', 'tuttle.xml', '/db/tuttle-backup', 'tuttle.xml')
The repositories to keep in sync with a gitservice are all listed under the repos-element.
The name-attribute refers to the destination collection also known as the target collection.
An example: <collection name="tuttle-sample-data">
The collection /db/apps/tuttle-sample-data is now considered to be kept in sync with a git repository.
<collection name="tuttle-sample-data">
<default>true</default>
<type>github</type>
<baseurl>https://api.github.com/</baseurl>
<repo>tuttle-sample-data</repo>
<owner>tuttle-sample-data</owner>
<token>a-personal-access-token</token>
<ref>a-branch</ref>
<hookuser>a-exist-user</hookuser>
<hookpasswd>that-users-password</hookpasswd>
</collection><type>gitlab</type>There are two supported git services at the moment github and gitlab
<baseurl>https://api.server/</baseurl>- For github the baseurl is
https://api.github.com/or your github-enterprise API endpoint - For gitlab the baseurl is
https://gitlab.com/api/v4/but can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'
- For github you have to specify the owner and the repo
- For gitlab you have to specify the project-id of the repository
<ref>main</ref>Defines the branch you want to track.
If a token is specified Tuttle authenticates against GitHub or GitLab. When a token is not defined, Tuttle assumes a public repository without any authentication.
Note
Be aware of the rate limits for unauthenticated requests GitHub allows 60 unauthenticated requests per hour but 5,000 for authenticated requests
Tip
It is also possible to pass the token via an environment variable. The name of the variable have to be tuttle_token_ + collection (all dashes must be replaces by underscore). Example: tuttle_token_tuttle_sample_data
At this stage of development, the API keys must be generated via the API endpoint /git/apikey or for a specific collection /git/{collection}/apikey.
In the configuration tuttle.xml the "hookuser" is used to define the dbuser which executes the update.
Example configuration for GitHub:
- 'Payload URL': https://existdb:8443/exist/apps/tuttle/git/hook
- 'Content type': application/json
Example configuration for GitLab:
The dashboard can trigger a full deployment or an incremental update.
Full deployment clones the repository from git and install it as a .xar file.
With incremental update only the changes to the database collection are applied.
Note
Tuttle is built to keep track of data collections
Note
Tuttle is does not run pre- or post install scripts nor change the index configuration on incremental updates!
- customize the configuration (
data/tuttle.xml) - click on 'full' to trigger a full deployment from git to existdb
- now you can update your collection with a click on 'incremental'
Repositories from which a valid XAR (existing expath-pkg.xml and repo.xml) package can be generated are installed as a package, all others are created purely on the DB.
Note
Note that there may be index problems if a collection is not installed as a package.
The page below is reachable via api.html in your installed tuttle app.
Calling the API without {collection} config:default-collection() is chosen.
GET ~/tuttle/{collection}/git
With this most basic endpoint the complete data repository is pulled from the gitservice. The data will not directly update the target collection but be stored in a staging collection.
To update the target collection use another POST request to /tuttle/git.
The data collection is stored in /db/app/sample-collection-staging.
POST ~/tuttle/{collection}/git
The staging collection /db/app/sample-collection-staging is deployed to /db/app/sample-collection. All permissions are set and a pre-install function is called if needed.
POST ~/tuttle/{collection}/git
All commits since the last update are applied.To ensure the integrity of the collection, all commits are deployed individually.
GET ~/tuttle/{collection}/hash
Reports the GIT hashed of all participating collections and the hash of the remote repository.
GET ~/tuttle/{collection}/commits
Displays all commits with commit message of the repository.
GET ~/tuttle/{collection}/hook
The webhook is usually triggered by GitHub or GitLab. An incremental update is triggered. Authentication is done by APIKey. The APIKey must be set in the header of the request.
curl --header 'X-Gitlab-Token: RajWFNCILBuQ8SWRfAAAJr7pHxo7WIF8Fe70SGV2Ah' http://127.0.0.1:8080/exist/apps/tuttle/git/hook
GET ~/tuttle/{collection}/apikey
The APIKey is generated and displayed once. If forgotten, it must be generated again.
GET ~/tuttle/config
Displays the configuration and the state of the git repository.
States:
- uptodate: Collection is up to date with GIT
- behind: Collection is behind GIT and need an update
- new: Collection is not a tuttle collection, full deployment is needed
<tuttle>
<default>sample-collection-github</default>
<repos>
<repo type="github" url="https://github.com/Jinntec/tuttle-demo" ref="master" collection="sample-collection-github" status="uptodate"/>
<repo type="gitlab" url="https://gitlab.com/tuttle-test/tuttle-demo.git" ref="master" collection="sample-collection-gitlab" status="uptodate"/>
</repos>
</tuttle>POST ~/tuttle/{collection}/lockfile
Remove lockfile after anything goes wrong.
GET ~/tuttle/{collection}/lockfile
The running task is stored in the lockfile. It ensures that two tasks do not run at the same time.
To talk to the configured gitservice Tuttle needs an access token. These can be obtained from the respective service.
- see Creating a personal access token for github
- see Personal access tokens for Gitlab
The key for the gitservice must be configured in Gitservice configuration as shown above.
- DB to Git
Horace Parnell Tuttle - American astronomer
Archibald "Harry" Tuttle - Robert de Niro in Terry Gilliams' 'Brazil'