Skip to content

Commit

Permalink
Refactor to new architecture. Handle ceilometer metrics/events and co…
Browse files Browse the repository at this point in the history
…llectd metrics/events (#22)

* Refactor to new architecture. Handle ceilometer metrics/events and collectd metrics/events

Co-authored-by: pleimer <pleimer@redhat.com>

Co-authored-by: Leif Madsen <leif@leifmadsen.com>
Co-authored-by: Martin Magr <mmagr@redhat.com>
  • Loading branch information
3 people authored Feb 24, 2021
1 parent ec31fcb commit 18edc86
Show file tree
Hide file tree
Showing 70 changed files with 4,529 additions and 6,093 deletions.
16 changes: 10 additions & 6 deletions .github/workflows/main.yml → .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,21 @@ jobs:
runs-on: ubuntu-20.04

steps:
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: '1.14'
- name: Checkout code
uses: actions/checkout@v2

- name: Run unit testing and code coverage
run: |
docker run -eCOVERALLS_TOKEN -uroot --network host -i --volume $GITHUB_WORKSPACE:/go/src/github.com/infrawatch/sg-core:z --workdir /go/src/github.com/infrawatch/sg-core registry.access.redhat.com/ubi8 /bin/sh -c 'sh ./build/test-framework/run_tests.sh'
- name: Run unit tests and code coverage
run: go test -v -coverprofile=profile.cov ./...
- name: Send coverage
uses: shogo82148/actions-goveralls@v1
with:
path-to-profile: profile.cov
- name: Verify image builds
run: |
docker build --tag infrawatch/sg-core:latest --file build/Dockerfile .
- name: List images
run: |
docker images
29 changes: 29 additions & 0 deletions .github/workflows/updates.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Coveralls Badge
on: pull_request

jobs:
coveralls_badge:
name: Coveralls Badge
# if: |
# (
# (github.event.issue.author_association == 'OWNER') ||
# (github.event.issue.author_association == 'COLLABORATOR') ||
# (github.event.issue.author_association == 'CONTRIBUTOR') ||
# (github.event.issue.author_association == 'MEMBER')
# )
runs-on: ubuntu-20.04
steps:
- name: update PR with coveralls badge
uses: actions/github-script@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
var BRANCH_NAME = process.env.BRANCH_NAME;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `[![Coverage Status](https://coveralls.io/repos/github/${context.repo.owner}/${context.repo.repo}/badge.svg?branch=${BRANCH_NAME})](https://coveralls.io/github/${context.repo.owner}/${context.repo.repo}?branch=${BRANCH_NAME})`
})
env:
BRANCH_NAME: ${{ github.event.pull_request.head.ref }}
110 changes: 102 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,111 @@
# SA-Benchmark and prototype
[![Coverage Status](https://coveralls.io/repos/github/infrawatch/sg-core/badge.svg?branch=master)](https://coveralls.io/github/infrawatch/sg-core?branch=master)

## GOLANG Server
# Plugins
Default plugins exist in /plugins. Plugins can also be hosted as separate projects.

UDP / Unix Socket gateway
## Types
### Transport
Transports listen on an external protocol for incoming messages.

### Build
### Handler
Handlers receive message blobs from a transport plugin and parse them
to be placed on the internal metrics or events bus. A handler can either be a
metric or event handler, but not both.


### Application
Applications receive both metrics and events and decide what to do
with them. Most application plugins interact with a storage backend such as
Prometheus.

# Build
```bash
go build cmd/server/server.go
# build sg-core and plugins. Places plugin binaries in ./bin
./build.sh
```

### Usage
# Configuration
Administrators must specify 3 sections in the yaml config:

```bash
./server net -port 30000
1. Sg-core configuration options
2. Transport plugins and configurations
3. Application plugins and configurations

``` yaml
#1 sg-core configs
plugindir:
loglevel:

#2 Transports
transports:
- name: <transport-0>
handlers:
- <handler-0>
[...]
- <handler-n>
config:
# plugin specific configuration
[...]

- name: <transport-n>
handlers:
[...]
config:

#3 Applications
applications:
- name: <application-0>
config:
# application plugin specific configurations
[...]

- name: <application-n>
config:
```
Section one describes sg-core specific configurations.
Section two describes any number of transport plugins that should be configured
in a list. Each transport plugin can bind any number of message handlers to itself.
Keep in mind that at this time, all handlers receive every message arriving on
the transport. Thus, it is generally only viable to bind either all metric handlers
or all event handlers to any one transport. The `config` block is specific to
the plugin being described. For example, TCP transport may require an address to
be specified here.

Section three describes application plugins. Just like the transport, more than
one application can be configured to run. Each application block contains a
config block specific to that plugin.

## Example Configuration
This configuration assumes both a QPID Dispatch Router and Prometheus instance
are running on the localhost and listens for incoming messages on a unix socket
at `/tmp/smartgateway`. The setup expects incoming messages to be collectd
metrics, as can be seen by the type of handler bound to the socket transport.

```yaml
plugindir: bin/
loglevel: debug
transports:
- name: socket
handlers:
- collectd-metrics
config:
address: /tmp/smartgateway
applications:
- name: prometheus
config:
host: localhost
port: 8081
withtimestamp: false
```

## Run
`./sg-core -config <path to config>`

## Docker/Podman
Build:
`podman build -t sg-core -f build/Dockerfile .`

Run:
`podman run -d -v /path/to/sg-core.conf.yaml:/etc/sg-core.conf.yaml:z --name sg-core sg-core`
32 changes: 32 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash

base=$(pwd)

PLUGIN_DIR=${PLUGIN_DIR:-"/tmp/plugins/"}
CONTAINER_BUILD=${CONTAINER_BUILD:-false}

for i in plugins/transport/*; do
cd "$base/$i"
echo "building $(basename $i).so"
go build -o "$PLUGIN_DIR$(basename $i).so" -buildmode=plugin
done
cd "$base"
for i in plugins/handler/*; do
cd "$base/$i"
echo "building $(basename $i).so"
go build -o "$PLUGIN_DIR$(basename $i).so" -buildmode=plugin
done
cd "$base"
for i in plugins/application/*; do
cd "$base/$i"
echo "building $(basename $i).so"
go build -o "$PLUGIN_DIR$(basename $i).so" -buildmode=plugin
done
cd "$base"

if $CONTAINER_BUILD; then
echo "building sg-core for container"
go build -o /tmp/sg-core cmd/*.go
else
go build -o sg-core cmd/*.go
fi
10 changes: 6 additions & 4 deletions build/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ ENV D=/go/src/github.com/infrawatch/sg-core
WORKDIR $D
COPY . $D/

RUN dnf install golang git -y --setopt=tsflags=nodocs && \
go build -o /tmp/smart_gateway cmd/server/server.go
RUN dnf install golang git -y --setopt=tsflags=nodocs
RUN go get -d ./...
RUN CONTAINER_BUILD=true ./build.sh

# --- end build, create smart gateway layer ---
FROM registry.access.redhat.com/ubi8
Expand All @@ -16,6 +17,7 @@ LABEL io.k8s.display-name="Smart Gateway" \
io.k8s.description="A component of the Service Telemetry Framework on the server side that ingests data from AMQP 1.x and provides a metrics scrape endpoint for Prometheus, and forwards events to ElasticSearch" \
maintainer="Leif Madsen <leif+smartgatewayoperator@redhat.com>"

COPY --from=builder /tmp/smart_gateway /
COPY --from=builder /tmp/sg-core /
COPY --from=builder /tmp/plugins/*.so /usr/lib64/sg-core/

ENTRYPOINT ["/smart_gateway"]
ENTRYPOINT ["/sg-core"]
4 changes: 2 additions & 2 deletions build/test-framework/pre-commit
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ DEPCHECK_OK=1
# TODO: uncomment each of these tests during future pull-requests with
# resolving code changes
main() {
mv_vendor_out
# mv_vendor_out
run_gofmt
run_golint
mv_vendor_in
# mv_vendor_in
run_govet
run_staticcheck
check_results
Expand Down
8 changes: 4 additions & 4 deletions build/test-framework/run_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
set -ex

# bootstrap
mkdir -p /go/bin /go/src /go/pkg
export GOPATH=/go
mkdir -p go/bin go/src go/pkg
export GOPATH="$PWD/go"
export PATH=$PATH:$GOPATH/bin

# get dependencies
Expand All @@ -24,7 +24,7 @@ go test -v ./...

set +e
echo " *** Running code coverage tooling"
go test ./... -race -covermode=atomic -coverprofile=coverage.txt
go test ./... -race -covermode=atomic -coverprofile=coverage.out

echo " *** Running Coveralls test coverage report"
goveralls -coverprofile=coverage.txt
goveralls -coverprofile=coverage.out
Loading

0 comments on commit 18edc86

Please sign in to comment.