SHIELD BOSH Release v8.0.4

This BOSH release packages version 8.0.4 of SHIELD

Fixes

• The shield-agent job can now be deployed via bosh create-env,
  since it no longer uses the -%> ERB template delimiter.

• The shield-agent job now expects agent.key to be the SSH
  authorized key (ssh-rsa AAA...) format, instead of the private
  key. This is a change from previous 8.x versions, but the
  warnings in the template rendering should guide you well.

Deployment

releases:
- name: shield
  version: 8.0.4
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v8.0.4/shield-8.0.4.tgz
  sha1: 4ffe8f8f8bd0051ab8dc24c48e94f3ff82a1e620

v8.0.3 (BETA)

SHIELD 8.x is still beta software; not recommended for production deployment

Note: you cannot upgrade to 8.0.3 from any of the previous 8.x releases; a database change is included in the v4 migration which 8.0.1 and 8.0.2 (also both BETA) would have already migrated to. Apologies for the inconvenience.

shield

Bumped https://github.com/starkandwayne/shield to v8.0.3

Deployment

releases:
- name: shield
  version: 8.0.3
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v8.0.3/shield-8.0.3.tgz
  sha1: 4d1a6f3fb823d42f0550df217ab6831e3c352a93

v8.0.2 (BETA)

SHIELD 8.x is still beta software; not recommended for production deployment

Bug Fixes

• Handle non-v2 BOSH Directors
• Fix import errand

Deployment

releases:
- name: shield
  version: 8.0.2
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v8.0.2/shield-8.0.2.tgz
  sha1: 78f8379d4cf24a9419c5134c01137994f97b4a93

v8.0.1 (BETA)

SHIELD 8.x is still beta software; not recommended for production deployment

Bug Fixes

• The nginx frontend now properly proxies normal HTTP and
  WebSocket requests through to the backend, avoiding a stall
  in the core scheduler brought on by dead broadcast receivers.

• Fixed an issue with some jobs that would overwrite PID files
  when trying to start jobs that were already running.

• Fixed an issue where the nginx proxy was up and ready before the
  shieldd process, leading to HTML and 502 Bad Gateway errors in
  the authorized_keys file during /v1/meta/pubkey provisioning.

shield

Bumped https://github.com/starkandwayne/shield to v8.0.1

Deployment

releases:
- name: shield
  version: 8.0.1
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v8.0.1/shield-8.0.1.tgz
  sha1: f0ac324f723b777e35617adac8920de6f0ddb052

v8.0.0 (BETA)

This release packages SHIELD v8.
SHIELD v8 is a dramatic overhaul of the SHIELD Data Protection
System, and this BOSH release version is a severe departure from
the previous methods of deploying SHIELD.

SHIELD 8.x is still beta software; not recommended for production deployment

Upgrading from v6.x / v7.x

This section details deployment manifest changes that operators
will need to apply in order to migate from v6 / v7 of SHIELD to
this (v8) release.

Changes to the shield-daemon job (now core)

The shield-daemon job is now just core.

The name property is gone. In its place are the following
properties for identifying your SHIELD:

• core.env - The name of the environment, like "sandbox" or
  "production", or "a testing shield instance", or "fred".
• core.color - A CSS color name, or hexadecimal RGB color to
  use for the environment name in the new web UI. yellow
  and green look nice.
• core.motd - A (possible multi-line) message that will be
  displayed to users logging into SHIELD. Useful for whatever
  messages of the day are generally useful for (compliance,
  advertising maintenance windows, etc.)

workers has been renamed to core.workers, but otherwise retains
its semantic meaning.

max_timeout has been renamed to core.task-timeout, but
otherwise retains its semantic meaning.

ssl.key, ssl.crt, and ssl.timeout have been renamed to
tls.key, tls.certificate, and tls.reuse-after, respectively.
The default value of tls.reuse-after was dropped from 12 (hours)
to 2 (hours).

The ssh_private_key has been renamed to agent.key, because
it's not used for SSH in the same sense as most SSH (RSA) keys.
Its value should stay the same for a smooth upgrade.

The database.* properties have been removed; SHIELD v8 has its
own internal data store that does not need to be configured by the
operator. See the Database Migration subsection, later, for
details on migrating your data into this new data store.

The auth.oauth.* properties have been removed; SHIELD v8
supports multiple (possibly OAuth2-based) authentication
providers. These are configured under the new top-level
authentication key.

The auth.username and auth.password properties have been
removed; SHIELD v8 no longer supports simple HTTP Basic
Authentication. Instead, it features a robust user authentication
system backed by an internal local user database. Two new
properties, failsafe.username and failsafe.password kind of
take over for these deprecated properties. They specify a
username and (cleartext) password that SHIELD will insert into the
local user database if it boots up and finds that no users exist
yet. This "failsafe" is intended to provide a secure way of
bootstrapping a SHIELD environment, without being stuck with a
user whose password is in a BOSH manifest somewhere.
Administrators are free to delete the failsafe user once they have
set up other accounts.

The auth.api_keys property has been removed; SHIELD v8 does not
support API Keys in the same fashion as its predecessors.
Instead, user accounts are free to issue Auth Tokens that behave
a stand-ins for their issuer (not unlike Github Personal Access
Tokens).

nginx.worker_processes has been shortened to nginx.workers.

nginx.worker_connections has been shortened to
nginx.connections.

nginx.keepalive_timeout has been shortened to nginx.keepalive.

The log_level property has been renamed to log-level.

Changes to the shield-agent job

This job is still called shield-agent, since it needs to be
unique across a wide variety of other deployments.

name is a new property for specifying the name this agent will
use when registering with the SHIELD Core.

autoprovision has been removed. Its usage was always
problematic, and with the introduction of proper BOSH links, we
only need to specify where and how to talk to the SHIELD Core in
the event that our Core lives on another BOSH director (which is
rare).

shield-url is a new property that kind of takes the place of
autoprovision, by allowing operators to identify where their
SHIELD Core lives, as a full URL (i.e.
"https://shield.example.com")

require-shield-core is a new property that lets operators ignore
an error condition whereby an agent is unable to communicate with
the SHIELD Core. In theory, that is a show-stopping problem, but
in practice, we found that it held up too many deployments for
legitimate reasons, ranging from simple network connectivity
issues and firewalling to more mundane problems like "we haven't
deployed SHIELD itself yet."

The daemon_public_key property has been removed. In its place
is the new agent.key property. The most important difference
between these two properties is that daemon_public_key was the
authorized_keys-formatted public key, and agent.key is the
private key that the SHIELD Core also specifies as
agent.key. Internally, the BOSH release will extract the
correctly formatted public fingerprint from the private key.

Note that if the shield link is in use, you don't need to
explicitly set agent.key -- the agent startup scripts will just
retrieve the public key from the SHIELD Core automagically. This
allows SHIELD site operators to rotate that key with minimal fuss.

The recovery.* properties have been removed, since SHIELD v8's
new encryption feature makes it difficult to restore backups
outside of the watchful eye of a running SHIELD Core.

For SHIELD Agents that need to operate behind HTTP proxies, three
new env.* properties were added. env.http_proxy and
env.https_proxy allow you to specify the full URL for an
upstream proxy that will handle (respectively) cleartext HTTP
requests and TLS-encrypted HTTPS requests. The env.no_proxy
property is a list of FQDNs, domain fragments, and IP addresses
that will be flattened and joined by commas to fashion an
exclusion list to put in the $no_proxy environment variable.

The new env.path, env.libs, and env.auto properties control
how the SHIELD agent process will set up its environment, for the
benefit of executed plugins.

env.path is a list of auxiliary paths to bin/ and sbin/
directories that you want to manually inject into the $PATH of
the running shield-agent / plugins.

env.libs is a list of auxiliary paths to lib/ directories that
you want / need in your $LD_LIBRARY_PATH for dynamic shared
object runtime loading.

env.auto is a boolean; if set, the shield-agent job will go
looking for installed BOSH packages named shield-addon-*, add
any bin/ and sbin/ directories to $PATH, and add any lib/
directories to $LD_LIBRARY_PATH. This allows you to augment an
agent with additional command-line tools it might need, like
specific versions of psql, or xtrabackup. env.auto is on by
default.

The auto-provisioning properties stores, targets,
retention-policies, and jobs have all been removed, in favor
of the new buckler import-based import errand.

The log_level property has been renamed to log-level.

Removed Jobs

The agent-mysql and xtrabackup jobs have been removed. If you
need to augment a SHIELD agent with MySQL / MariaDB tools, you can
try the nee [SHIELD MySQL Addon][mysql-addon], which contains all
of these packages.

The agent-pgtools job has been removed. If you need to augment
a SHIELD agent with PostgreSQL tools, you can try the new [SHIELD
PostgreSQL Addon][postgres-addon].

The mongo-tools3.2 and mongo-tools3.4 jobs have been removed.
They too have moved into a separate BOSH release, the [SHIELD
MongoDB Addon][mongodb-addon].

The postgres and mariadb jobs have been removed. SHIELD v8
now leverages a standalone, dedicated database that is baked into
the new core job. See the subsection Database Migration,
below, for details on migrating your SHIELD data.

The New Import Errand

Previous versions of the SHIELD BOSH release used a post-start
script and shield-agent properties to facilitate a form of
configuration auto-provisioning.

In v8, this has all been replaced by the new import errand,
which drives the much more powerful and flexible buckler import
command-line tool.

The import errand takes a single property, import, which is a
full recipe of things to import into SHIELD, as understood by the
buckler tool's import sub-command.

Here's an example that sets up a bunch of stuff:

- name: import
  lifecycle: errand
  instances: 1
  azs: [z1]
  vm_type:         default
  stemcell:        default
  networks: {name: default}
  jobs:
    - name:    import
      release: shield
      properties:
        import:
          core:  https://shield.example.com
          token: ... # an auth token, from `buckler create-auth-token`

          global:
            storage:
              - name: S3 Cloud Storage
                summary: |
                  Public S3 cloud storage for all SHIELD tenants to use
                agent:  127.0.0.1:5444
                plugin: s3
                config:
                  access_key_id:     AKI12
                  secret_access_key: secret

            policies:
              - name: Long-Term Storage
                days: 90

          users:
            - name:     James Hunt
              username: jhunt
              password: sekrit
              sysrole:  admin
              tenants:
                - name: Stark & Wayne
                  role: admin

          tenants:
            - name: CF Community
              members:
                - user: jhunt@local
                  role: admin
              storage:
                - name: Scality
                  a...

v7.0.3

• shield-agent job has new properties enabling backup and restore behind a proxy

  env.http_proxy:		
    description: The URL of an upstream HTTP proxy for proxying all unencrypted web communications.		
  env.https_proxy:		
    description: The URL of an upstream HTTPS proxy for proxying all encrypted web communications.		
  env.no_proxy:		
    description: A list of domains, partial domains (i.e. ".example.com"), and IP addresses that should not be routed through env.http_proxy and env.https_proxy.
  

Deployment

releases:
- name: shield
  version: 7.0.3
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v7.0.3/shield-7.0.3.tgz
  sha1: a1f705e0ad0130ac78d851332d97905a3f8cbce2

v7.0.2

• shield-daemon job shield-db link is now optional (thanks @karampok)

• shield-daemon job has two new properties that can be overridden if you need to tweak these settings (thanks #90 from @karampok)

  workers:
    description: Max number of concurrent tasks in running state.
    default: 5
  max_timeout:
    description: Duration in hours after which a running task is timed out.
    default: 12
  

• shield-server blob now includes version number

Deployment

releases:
- name: shield
  version: 7.0.2
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v7.0.2/shield-7.0.2.tgz
  sha1: db19190e1a21336bdee5ec7ed16d4b57e456cae7

v7.0.1

Major version v7

NOTE: Please ignore v7.0.0 which was accidentally cut and is a duplicate of v6.9.0. v7.0.1 is the first release of v7.

v7 is the first release of SHIELD BOSH release to drastically simplify deployment using BOSH v2 (bosh2 CLI), Credentials generation using bosh2 or Credhub, Cloud Config, Links and other loveliness.

See #76 for background to this major version.

NOTE: There are no upgrade docs from v6 to v7 yet. If you do go thru the upgrade process and survive, please help us to write some documentation or blog posts, thanks!

Deployment

releases:
- name: shield-boshrelease
  version: 7.0.1
  url: https://github.com/starkandwayne/shield-boshrelease/releases/download/v7.0.1/shield-7.0.1.tgz
  sha1: 59523e113c2276a54eca308aa1ca944fd6e3a99c

S.H.I.E.L.D. BOSH Release v6.9.0

shield

Bumped https://github.com/starkandwayne/shield to v0.10.9

S.H.I.E.L.D. BOSH Release v6.8.0

shield

Bumped https://github.com/starkandwayne/shield to v0.10.8

New Features

• Added Google Cloud Storage plugin

Bug Fixes

• Fixes some post-start script issues when creating schedules, policies, stores and/or targets & jobs at the same time