Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add constructs and configuration options to enable Roadie Agent driven Tech Insights Data Sources #4

Closed
wants to merge 6 commits into from
Closed

Add constructs and configuration options to enable Roadie Agent driven Tech Insights Data Sources #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bd4146e
Add constructs and configuration options to enable Roadie Agent drive…
Xantier Jan 27, 2023
56b7be5
Fix names in README.md
Xantier Feb 10, 2023
574e2c3
Move schema registration to Roadie app.
Xantier Feb 10, 2023
f2868c8
Fix import
Xantier Feb 10, 2023
d4d24d7
Fix logging to point to the correct place.
Xantier Feb 10, 2023
0499c73
Finish full end to end custom data source
Xantier Feb 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 212 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,38 @@

## What is Roadie Agent

Roadie Agent library is functionality utilizing Snyk Broker to achieve a secure connection between customer infrastructure and running Roadie application. It utilizes a broker connection between a Broker client initialized by the Roadie Agent and a broker server running in Roadie infrastructure.
Roadie Agent library is functionality utilizing Snyk Broker to achieve a secure connection between customer
infrastructure and running Roadie application. It utilizes a broker connection between a Broker client initialized by
the Roadie Agent and a broker server running in Roadie infrastructure.

Roadie agent relies heavily on the Snyk Broker client library in its core. The main flow of data is the same as with Snyk broker where the _Broker Client_ initiates a connection to the _Broker Server_ which is running on the Roadie instance. After the connection between the client and server is successful, data can flow securely via that connection to other destinations reachable by their respective endpoints.
Roadie agent relies heavily on the Snyk Broker client library in its core. The main flow of data is the same as with
Snyk broker where the _Broker Client_ initiates a connection to the _Broker Server_ which is running on the Roadie
instance. After the connection between the client and server is successful, data can flow securely via that connection
to other destinations reachable by their respective endpoints.

The library creates a wrapper around the Snyk Broker library and uses this connection to provide users of the library the possibility to use this brokered connection to run custom agent code in their own infrastructure. That is achieved by creating an additional web server and configuring the _Broker client_ to forward all requests to this, agent embedded, webserver. This local webserver can also send specific subset of requests back via the same brokered connection, thus giving the library users a clear, secure interface they can use to provide data to a running Roadie instance.

The agent webserver runs by default in `http://localhost:7044` and has a specified set of endpoints that are initialized only based on agent configurations. All other endpoints return 404.
The library creates a wrapper around the Snyk Broker library and uses this connection to provide users of the library
the possibility to use this brokered connection to run custom agent code in their own infrastructure. That is achieved
by creating an additional web server and configuring the _Broker client_ to forward all requests to this, agent
embedded, webserver. This local webserver can also send specific subset of requests back via the same brokered
connection, thus giving the library users a clear, secure interface they can use to provide data to a running Roadie
instance.

The agent webserver runs by default in `http://localhost:7044` and has a specified set of endpoints that are initialized
only based on agent configurations. All other endpoints return 404.

## Installation

1. Run `npm install @roadiehq/roadie-agent` or `yarn add @roadiehq/roadie-agent` (or equivalent) depending on your preferred choice of package manager.
1. Run `npm install @roadiehq/roadie-agent` or `yarn add @roadiehq/roadie-agent` (or equivalent) depending on your
preferred choice of package manager.
2. Run `npm install` or `yarn` (or equivalent).

(With default settings) Create a folder called `config` at the root of your project and copy `provider-accept.json` file from the installed packages config folder into it. Alternatively the default configuration can be copy-pasted from below:
(With default settings) Create a folder called `config` at the root of your project and copy `provider-accept.json` file
from the installed packages config folder into it. Alternatively the default configuration can be copy-pasted from
below:
<details>

<summary>Click to expand</summary>


```json
{
"private": [
Expand Down Expand Up @@ -49,16 +61,25 @@ Standard use case is to use the agent as a library. The minimal example to get s
```javascript

// myfile.js
const { RoadieAgent, createRoadieAgentEntityProvider } = require('@roadiehq/roadie-agent')
const myEntityProviderHandler = require('./myEntityProvider')
const { RoadieAgent, createRoadieAgentEntityProvider } = require("@roadiehq/roadie-agent");
const myEntityProviderHandler = require("./myEntityProvider");
const { myDataSourceHandler } = require("./myDataSource");

RoadieAgent.fromConfig()
const roadieAgent = RoadieAgent.fromConfig({
identifier: "my-roadie-agent-token"
})
.addEntityProvider(
createRoadieAgentEntityProvider({
name: 'testprovider',
name: "testprovider",
handler: myEntityProviderHandler
}),
)
.addTechInsightsDataSource(
createRoadieAgentTechInsightsDataSource({
name: "my-data-source",
handler: myDataSourceHandler
}),
)
.start();

```
Expand All @@ -69,61 +90,71 @@ And then running the file with `node myfile.js`

In reality the Agent initialization call needs some configuration items to work correctly.

| Config name | Default | Description |
|-------------|-----------------------|-----------------------------------------------------------------------------------------------------------------|
| server | http://localhost:7341 | Broker server URL. This will usually be something like `https://<tenant-name>.broker.roadie.so` |
| identifier | example | Identifier of the broker client-server connection, also known as broker token. This can be arbitrarily chosen. |
| port | 7342 | Port to use for the local broker client receiver web server |
| agentPort | 7044 | Port to use for the local Roadie Agent that handles events brokered via the broker client |
| accept | config/accept.json | Path to the broker client accept file to use |

| Config name | Default | Description |
|-------------|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| server | http://localhost:7341 | Broker server URL. This will usually be something like `https://<tenant-name>.broker.roadie.so` |
| identifier | example | Identifier of the broker client-server connection, also known as broker token. This is configured in Roadie Agent settings in Roadie. |
| port | 7342 | Port to use for the local broker client receiver web server |
| agentPort | 7044 | Port to use for the local Roadie Agent that handles events brokered via the broker client |
| accept | config/accept.json | Path to the broker client accept file to use |

<details>

<summary>See example</summary>


```js
RoadieAgent.fromConfig({
server: 'https://myroadie.broker.roadie.so',
identifier: 'my-dev-cluster-roadie-agent',
accept: '/etc/config/my-modified-accept.json',
})
server: 'https://myroadie.broker.roadie.so',
identifier: 'my-dev-cluster-roadie-agent',
accept: '/etc/config/my-modified-accept.json',
})
.addEntityProvider(
createRoadieAgentEntityProvider({
name: 'testprovider',
handler: myEntityProviderHandler
}),
)
.addTechInsightsDataSource(
createRoadieAgentTechInsightsDataSource({
name: 'my-data-source',
handler: myDataSourceHandler
}),
)
.start();
```

</details>

### Handler types

#### Entity Provider

You can create a Roadie Agent driven Entity Provider by using the `createRoadieAgentEntityProvider` helper function. This function expects two arguments, one naming the agent itself and a handler to provide the actual entities. Note that the `name` defined in here **needs to match** the one configured in your Roadie instance
You can create a Roadie Agent driven Entity Provider by using the `createRoadieAgentEntityProvider` helper function.
This function expects two arguments, one naming the agent itself and a handler to provide the actual entities. Note that
the `name` defined in here **needs to match** the one configured in your Roadie instance

Entity provider handler is a function that receives an `emit` callback. This callback can be called to emit new entity mutations to your Roadie instance. The shape of the payload to be emitted is based on `EntityProviderMutation` type defined in `@backstage/plugin-catalog-node` package.
Entity provider handler is a function that receives an `emit` callback. This callback can be called to emit new entity
mutations to your Roadie instance. The shape of the payload to be emitted is based on `EntityProviderMutation` type
defined in `@backstage/plugin-catalog-node` package.
<details>
<summary>Show type</summary>

```typescript
export declare type EntityProviderMutation = {
type: 'full';
entities: DeferredEntity[];
type: 'full';
entities: DeferredEntity[];
} | {
type: 'delta';
added: DeferredEntity[];
removed: DeferredEntity[];
type: 'delta';
added: DeferredEntity[];
removed: DeferredEntity[];
};

export declare type DeferredEntity = {
entity: Entity;
locationKey?: string;
};
```

</details>


Expand All @@ -132,7 +163,6 @@ Example entity provider with a hardcoded entity definition would look like the f
<details>
<summary>Show example hardcoded entity handler</summary>


```typescript
const fakePayload = {
type: 'full',
Expand Down Expand Up @@ -164,10 +194,158 @@ const fakePayload = {
],
};

const myEntityHandler = async (emit) => {
export const myEntityHandler = async (emit) => {
await emit(fakePayload);
}
```

</details>

#### Tech Insights Data Source

You can create a custom Roadie Agent driven Tech Insights Data Source by using
the `createRoadieAgentTechInsightsDataSource` helper function. This function expects two arguments, one naming the data
source driven by the agent and a handler to provide the actual facts for entities. Note that the `name` defined in here
**needs to match** the corresponding Data Source configuration created in your Roadie instance.

Roadie Agent drive Data Source handler is a function that receives an `emit` callback. This callback can be called to
emit collections of entity-facts pairs to your Roadie instance. The shape of the payload to be emitted is an array
of `TechInsightFact` type defined in `@backstage/plugin-tech-insights-node` package.

<details>
<summary>Show type</summary>

```typescript
declare type TechInsightFact = {
/**
* Entity reference that this fact relates to
*/
entity: {
namespace: string;
kind: string;
name: string;
};
/**
* A collection of fact values as key value pairs.
*
* Key indicates fact name as it is defined in FactSchema
*/
facts: Record<string, number | string | boolean | DateTime | number[] | string[] | boolean[] | DateTime[] | JsonValue>;
/**
* Optional timestamp value which can be used to override retrieval time of the fact row.
* Otherwise when stored into data storage, defaults to current time
*/
timestamp?: DateTime;
};
```

</details>

Example Roadie Agent driven data source with a hardcoded entity and fact data would look like the following:

<details>
<summary>Show example hardcoded tech insights data source handler</summary>

```typescript
const fakePayload = [{
entity: {
name: 'my-entity',
namespace: 'default',
kind: 'component'
},
facts: {
'myDataSource.integerFact': 4,
'myDataSource.stringFact': "3.40.1",
'myDataSource.datetimeFact': "2023-01-27T14:55:45.289Z",
}
}, {
entity: {
name: 'my-other-entity',
namespace: 'default',
kind: 'component'
},
facts: {
'myDataSource.integerFact': 42,
'myDataSource.stringFact': "0.9.1",
'myDataSource.datetimeFact': "2021-11-14T19:25:32.425Z",
}
}];

const myDataSourceHandler = async (emit) => {
await emit(fakePayload);
}

createRoadieAgentTechInsightsDataSource({
name: 'my-data-source',
handler: myDataSourceHandler,
})

```
</details>


Roadie Agent driven data source can be triggered by calling the `triggerDataSource` method on the constructed RoadieAgent object. This method expects the name of the registered Roadie Agent datasource to trigger the handler function. See an example of the full flow below:



<details>
<summary>Show example Express web server triggering Tech Insights custom Data Source</summary>

```typescript
const fakePayload = [{
entity: {
name: "my-entity",
namespace: "default",
kind: "component"
},
facts: {
"myDataSource.integerFact": 4,
"myDataSource.stringFact": "3.40.1",
"myDataSource.datetimeFact": "2023-01-27T14:55:45.289Z"
}
}, {
entity: {
name: "my-other-entity",
namespace: "default",
kind: "component"
},
facts: {
"myDataSource.integerFact": 42,
"myDataSource.stringFact": "0.9.1",
"myDataSource.datetimeFact": "2021-11-14T19:25:32.425Z"
}
}];

const myDataSourceHandler = async (emit) => {
await emit(fakePayload);
};


const { RoadieAgent, createRoadieAgentEntityProvider } = require("@roadiehq/roadie-agent");
const myEntityProviderHandler = require("./myEntityProvider");

const roadieAgent = await RoadieAgent.fromConfig({
identifier: "my-roadie-agent-token"
})
.addTechInsightsDataSource(
createRoadieAgentTechInsightsDataSource({
name: "my-data-source",
handler: myDataSourceHandler
})
)
.start();


const app = express();

app.get("/", async (req, res) => {
const response = await roadieAgent.trigger("jussi-test-local");
res.send(response);
});

app.listen(3000);


```
</details>

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
"dependencies": {
"@backstage/plugin-catalog-backend": "^1.5.1",
"@backstage/plugin-catalog-node": "^1.2.1",
"@backstage/plugin-tech-insights-node": "^0.3.9",
"@backstage/catalog-model": "^1.1.3",
"express": "^4.17.1",
"node-fetch": "^2.6.7",
Expand Down
Loading