Bareos backup server

[phillxnet]

BareOS is a fully AGPLv3 fork of bacula, please see the related discussion for a bacula rockon in issue #93
Copying my comments from that issue in here:
"I'm rather favouring we adopt @pauldsmyth move and consider Bareos given they have a far more current binary download for a bunch of linux distros and win.

Also from their web page:

"Bareos is a 100% open source fork of the backup project from bacula.org. The fork is in development since late 2010, it has a lot of new features.
The source has been published on github, licensed AGPLv3."

"Backup Archiving Recovery Open Sourced"
https://github.com/bareos/

see also:
https://www.bareos.org/en/
https://www.bareos.org/en/download.html

[EDIT] further to exploring barcus's docker hub images in:

• BareOS server set #94 #383

I have concluded that we should instead roll our own docker images. Predominantly as I would rather use unmodified upstream provided packages, and to also, in-time, provide for their stable branch subscription supported efforts.

It is also important to have up-to-date packages which barcus's were not: being dependant on 3rd / 4th parties to maintain on an upstream unsupported OS base.

Plus there are [EDIT: versions] up to date (by major version number) BareOS (16.2) docker images for all components with excellent (rather than no) documentation by barcus on dockerhub: [EDIT: above version info pertains to March 2017 issue creation date ; as of Aug 2024 we have continued maintenance of the barcus repo on dockerhub but with tagged versions of 20 (EOL), 21 (oldest of 3 non-EOL versions available: 21, 22, 23) while the latest tag looks to correspond to version 22].

https://hub.docker.com/u/barcus/

On the grounds that bacula would in comparison appear almost abandoned from the client binary point of view and has very few docker image options I'm changing this Issue to BareOS from Bacula. [EDIT] However there are also few options for BareOS docker images, with many being minor edits of barcus's. This is further encouragement of the new direction this issue has taken in late 2024 re our roll-our-own docker images. Having learned a lot from barcus's many years of building them for their particular setup.

If anyone has a working knowledge of BareOS please feel free to contribute to this issue.

---

Upstream Docs Quick Start: https://docs.bareos.org/IntroductionAndTutorial/WhatIsBareos.html#quick-start

[alazare619]

2 things I see that might be problems. The first being that the SQL DB password is passed as an enviromental variable. This can be seen in top...the other being that all the components use docker-compose which I'm not certain how to achieve this with Rockstor.

[phillxnet]

Still interested in this, but have to focus elsewhere at the moment.

Any takers on this one.

[phillxnet]

Overview of components:

From: https://www.bareos.com/learn/documentation/

• Director (daemon/service) - Central control program for all the other daemons. A scheduler/supervisor for all backup, restore, verify, and archive operations.
• Console - the bconsole app is a TTY interface to the Director. For administration.
• File (daemon/service) - Client software installed on the machines to be backed up. Instructed by the Director to find & send file data to the allotted Storage deamon/service for the client concerned. Similarly responsible for instantiating file creation/modification during Director instructed restore procedures, in this case receiving file data from the directed Storage deamon/service. See note: 1.
• Storage (daemon/service) - Instructed/entrusted by the Director as responsible for accepting file data from associated File deamons/services and archiving this to the associated backing storage/physical backup media. Similarly retrieves file data/attributes during restore under Director request, so that it can be sent to a relevant File deamon/service. Multiple Storage deamons/services can be associated/controlled by a single Director. See note: 1.
• Catalog (PostgreSQL DB) - services comprising the maintenance of file/volume indexes/databases for all files backed up per client, providing user and admin alike an overview of an environments current back-up status.

The Catalog services sets Bareos apart from simple backup programs like tar and bru, because the catalog maintains a record of all Volumes used, all Jobs run, and all Files saved, permitting efficient restoration and Volume management. Bareos supports PostgreSQL.

Notes:

1. The communication between a File deamon/service and a Storage deamon/service during a back-up or restore is known as a Session: https://docs.bareos.org/IntroductionAndTutorial/WhatIsBareos.html#terminology:

A session has a one-to-one correspondence to a Bareos Job ...

Ergo: Save & Restore operations require a minimum of 4 daemons to be co-configured and running:

• Director
• File
• Storage
• PostgreSQL (Catalog host).

[phillxnet]

#Release versions
Here we concern ourselves with the .org variations: https://download.bareos.org/, i.e.:

"These binaries are UNSUPPORTED by bareos.com."

• Current 23.0.3
• Next looks to be 24.0.0 pre *

and the GitHub releases page: https://github.com/bareos/bareos/releases

4th June 2024

• https://github.com/bareos/bareos/releases/tag/Release%2F21.1.10
• https://github.com/bareos/bareos/releases/tag/Release%2F22.1.5
• https://github.com/bareos/bareos/releases/tag/Release%2F23.0.3

However our indicated, and long running, https://hub.docker.com/u/barcus/ has versions 20, and 21 only (updated 3 days ago as of this comment).

From Official Bareos Subscription Repository (.com) https://download.bareos.com/bareos/release/ we have 21, 22, & 23 as NOT EOL.

Version Interoperabiliy info

https://download.bareos.org/current/BareosMainReference/IntroductionAndTutorial/WhatIsBareos.html#bareos-version-numbers-and-releases

Bareos Director and Bareos Storage Daemon are guaranteed to be backwards-compatible to the Bareos File Daemon of the previous two major releases.

BareOS Binary release policy

https://download.bareos.org/current/BareosMainReference/IntroductionAndTutorial/WhatIsBareos.html#bareos-binary-release-policy

1. Bareos Community binaries on https://download.bareos.org/:
   Publicly accessible

• Latest build of the current Bareos stable branch at https://download.bareos.org/current/
• Latest build of the Bareos master branch at https://download.bareos.org/next/

2. Bareos Subscription binaries on https://download.bareos.com/

• Available in the Bareos Subscription customers repositories.
• The last three major releases are maintained in https://download.bareos.com/bareos/release/
• The repostories contain the whole binary history of each release, so that downgrades are possible if required.
• Repositories of discontinued versions stay available.

[phillxnet]

DB setup

We have to establish/document how we are to perform the PostgreSQL DB setup to host BareOS's Catalogue:

https://docs.bareos.org/IntroductionAndTutorial/InstallingBareos.html#prepare-bareos-database
subsection: https://docs.bareos.org/IntroductionAndTutorial/InstallingBareos.html#other-platforms

su postgres -c /usr/lib/bareos/scripts/create_bareos_database
su postgres -c /usr/lib/bareos/scripts/make_bareos_tables
su postgres -c /usr/lib/bareos/scripts/grant_bareos_privileges

[EDIT: in our proposed docker images we have, if DB_INIT=true is set on the director image, the above executed via the docker entry script: https://github.com/barcus/bareos/blob/master/director-pgsql/22-alpine/docker-entrypoint.sh

https://github.com/barcus/bareos/blob/db8b993d129d8a2cf0e0a8a7069ec04ccd45c28b/director-pgsql/22-alpine/docker-entrypoint.sh#L108-L120

]

We have the ability to perform 'special' operations per Rock-on, on install etc: but this is to be avoided as it's inevitably fragile given upstream changes. Better to reference upstream docs, and document specific setups for custom Rock-ons or Rock-on combinations such as we may be looking at there: i.e. we previously hosted various versions of PostgreSQL, via Rock-ons; but these have since been removed as they exceeded EOL: #314

There is also the issue of an accidental Rock-ons reinstall re-setting a BareOS Catalogue host DB if we for example automate the above create_bareos_database: if that script ends up first dropping existing DBs. Althought BareOS does have the ability to re-create Catalogue DBs via bscan functions of existing Volumes (Backup archives of file or tape).

For our proposed BareOS docker images (newer alphine variants) and more recent BareOS, our Catalogue host must be PostreSQL and at least version: 14 (assumed as https://github.com/bareos/bareos/blob/master/core/platforms/packaging/bareos.spec has no version info, and our proposed docker image docs have: "Bareos v20+ -> PostgreSQL v14 or less" see: https://hub.docker.com/r/barcus/bareos-director)

Catalogue Maintenance

Upstream BareOS docs on DB setup:
https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#catalog-maintenance

Manual Catalogue DB setup docs

https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#manual-configuration

Incidental note: our pending how-to for his Rock-on/Rock-on set should advise, by default, accurate mode:
https://docs.bareos.org/Configuration/Director.html#config-Dir_Job_Accurate

[phillxnet]

Simplicity/Usability/Accessibility vs Complexity/Flexibility

We have the choice here to present a host of Rock-ons each expressing/encompassing/encapsulating a BareOS service, interlinked by-hand/duplicated-config & rock-nets: https://rockstor.com/docs/interface/overview.html#joining-rocknets, or to attempt a simplification/set. I favour the latter.

• Director is master controller: and absolutely needs a Catalogue (PostgreSQL DB). But is also useless without a Storage component.

As such - given Rockstor's (and Rock-ons) nature: Appliance in essence; I favour the compromised multi-docker image Rock-on approach here: with an integrated Web-UI via barcus/bareos-webui:latest image: upstream doc reference:
https://docs.bareos.org/master/IntroductionAndTutorial/BareosWebui.html#bareos-webui

N.B. we also have from BareOS docs: https://docs.bareos.org/Appendix/HardwareSizing.html#director

It is recommended to run the Bareos Director service together with the database server on the same machine, which minimizes the latency and overhead of the communication with the database.

Experimentation to follow on the feasibility on this approach. We may end up having to host individual Rock-ons for some, or all, of these BareOS deamons, but the associated user overhead complexity may well rival a bare-metal install on our host OS: voiding the Appliance nature we are aiming at here. Especially given advanced users could implement BareOS on our host OS anyway - as-is.

Opinions welcome, but regardless I'll try some mock-ups and try to present proof of concept PR's for this issue.

[phillxnet]

On our preferred PostgreSQL version we have from https://www.postgresql.org/support/versioning/ that v12 is getting its final release on November 14th, 2024 and is the last version currently supported. Ergo we go for at least 14 given the prior reference in #94 (comment) that BareOS v20+ is PostgreSQL 14 compatible. Althought our proposed docker hub image provider barcus indicates a v12 (postgres:12) in their docker compose example.

BareOS doc entry re PostgreSQl version: https://docs.bareos.org/Appendix/HardwareSizing.html#postgresql-version

It is always advisable to use the latest currently supported PostgreSQL on your platform.

[phillxnet]

Postgres docker hub

https://hub.docker.com/_/postgres

Given the initial plan here is to have a multi image Rock-on they would need to use the docker --link capability: as per our existing Seafile v7+ Rock-on: https://github.com/rockstor/rockon-registry/blob/master/seafile7%2B.json. However --link is deprecationed: https://docs.docker.com/network/links/

@FroggyFlox has already enabled within Rock-ons the partial replacement of user-defined networks: known as rocknets within our implementation.

@FroggyFlox As I read this, we appear to now be cornered regarding the initial plan here for a multi-image Rock-on. And have-to instead implement all BareOS docker images required as individual Rock-ons; and document tying them together via your rocknets. Does that sound right to your?

[EDIT] from the below PR we have @FroggyFlox's following comment:

In point 1, this PR follows @flukejones' idea and simply uses the existing container_links object in the rock-on json to create a dedicated docker network linking the two containers defined in the container_link.

Which is used in the previously referenced Seafile v7+ [EDIT] Zabbis-XXL multi-container Rock-on: https://github.com/rockstor/rockon-registry/blob/master/zabbix-xxl.json

---

Link to PR that addressed the deprecation of docker --link via docker networks (rocknets):
rockstor/rockstor-core#2207

As detailed in #1982, our current inter-containers communication in multi-containers rock-ons is not functional due to the deprecation of containers links by Docker. As originally proposed by @flukejones, we should switch to using docker networks instead.

[FroggyFlox]

@phillxnet ,
All below is from memory but I'm fairly certain it is still accurate (even if only in general).

Yes docker has deprecated links a long time ago now, but Rockstor still understands that concepts and uses docker networks to create the equivalent. We indeed have 2 types of rocknets:

• user-created: these would be those that the user explicitly creates in Rockstor's Web UI (be it at the Rock-on post install customization or via the Networks page)
• Rock-on-created: these are created by Rockstor when the json file contains the "link" object

So you should still be able to use that "link" object and Rockstor will automatically create the required docker networks and attach the containers in question to each of these networks.

Another option would be for Rockstor to support a "network" object in the JSON that would define network name and containers to attach but that's for another day.

[phillxnet]

More updated docker-compose info from the GitHub origin repo for our proposed image author on docke hub of barcus/bareos.
See: https://github.com/barcus/bareos/blob/master/docker-compose-alpine-pgsql.yml

I.e. port: parameter in docker hub for db access pertains to MySQL default of 3306, not the now only supported PostgreSQL default of 5432,

[phillxnet]

From our proposed upstream barcus docker github repo we have in:

https://github.com/barcus/bareos/blob/master/director-pgsql/22-alpine/docker-entrypoint.sh

That it curl retrieves from it's upstream of bareos's github repo, example config files:

- for console admin: https://raw.githubusercontent.com/bareos/bareos/master/webui/install/bareos/bareos-dir.d/console/admin.conf.example
- for web-ui: https://raw.githubusercontent.com/bareos/bareos/master/webui/install/bareos/bareos-dir.d/profile/webui-admin.conf

Which the docker script then edits according to environmental variables.

[phillxnet]

Client side (BareOS File deamons/services)

This issue pertains to an all-in-one BareOS server set: as such it is proposed that we do not include a BareOS client (File deamon/server) within this docker-image set. This way folks with existing BareOS setups could install just the BareOS Client Rock-on - and configure it to server their existing BareOS director. Similarly folks could use this same proposed BareOS File Rock-on but configure it to work with this issues BareOS server set.

---

See complementary Client side Rock-on proposal: #378

[phillxnet]

Current state of associated development branch:

So we have parsing compliance sufficient for uninstalled / available display (using local file repo for now).

[phillxnet]

@FroggyFlox & @Hooverdan96
I've now remarked out my prior notes here regarding barcus's docker work as it was not a good fit for us, and did not use packages distributed by BareOS themselves, or an OS that was supported by BareOS. But instead preferred the Apline base favoured for many docker installs that to my mind throws away work done by upstream re compatibility.

From what I've learned from barcus's work I have begun rolling our own docker images using an openSUSE base via their official docker image as a base, and in-turn installing packages directly downloaded from bareos.org (for their non subscription variant initially). And hope to later include their bareos.com images what required a subscription to access.

Closing my second draft attempt at this server/service set to embark on hopefully the final PR ready for wider input/review.

[phillxnet]

Noting a new release blog posted as of 15th November: https://www.bareos.com/blog/

• Bareos 23.1.0 is now available
• Changelog: https://github.com/bareos/bareos/blob/Release/23.1.0/CHANGELOG.md