Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Origo] Adds support to mark storage-node/distrubtion-node under maintenance #4793

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8a10f93
add protobuf schemas for operational status feat
zeeshanakram3 Jun 21, 2023
bb0b5f5
add QN schemas for operational status feat
zeeshanakram3 Jun 21, 2023
407cdeb
add QN mappings for operational status feat.
zeeshanakram3 Jun 21, 2023
ca8a495
add CLI commands to set operational status in Argus
zeeshanakram3 Jun 21, 2023
e53f696
add CLI commands to set operational status in Colossus
zeeshanakram3 Jun 21, 2023
21bc807
argus: generate-docs
zeeshanakram3 Jun 21, 2023
a27f656
Merge remote-tracking branch 'upstream/master' into origo_argus_colos…
zeeshanakram3 Jul 15, 2023
33997c4
add inquirer dependency in storage-node package.json
zeeshanakram3 Jul 18, 2023
b07bbec
Merge remote-tracking branch 'upstream/master' into origo_argus_colos…
zeeshanakram3 Aug 1, 2023
d133ffb
bump package versions & update changelog
zeeshanakram3 Aug 1, 2023
ba24cfe
bump QN package version
zeeshanakram3 Aug 2, 2023
4860658
Merge remote-tracking branch 'upstream/master' into origo_argus_colos…
zeeshanakram3 Feb 26, 2024
6ff4531
address requested changes
zeeshanakram3 Feb 29, 2024
fc04793
update generated storage-squid schema
zeeshanakram3 Feb 29, 2024
b95ae04
[Argus] ensure storage node only syncs from operationally active nodes
zeeshanakram3 Feb 29, 2024
4dd076f
update colossus docs
zeeshanakram3 Feb 29, 2024
b6329f7
[Argus] ensure distributor node only connects with operationally acti…
zeeshanakram3 Feb 29, 2024
172bcb7
revert QN changes (i.e. remove mappings and schema changes)
zeeshanakram3 Feb 29, 2024
16aaa48
update graphql queries
zeeshanakram3 Feb 29, 2024
73dc634
Merge remote-tracking branch 'upstream/master' into origo_argus_colos…
zeeshanakram3 Mar 25, 2024
971aaae
address requested changes
zeeshanakram3 Mar 27, 2024
0537ee7
fix: only filter currently under maintenance nodes
zeeshanakram3 Mar 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions distributor-node/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
### 2.1.0

- Updates `operator:set-metadata` CLI command to set distributor-node's operational status along with other metadata.
- Adds `leader:set-node-operational-status` CLI command to set operational status of any distributor-node by Lead.

## 1.5.1

- Added parsing of Axios errors on logger level so that we never log the whole Axios client instance (which is a circular object and causes the node to crash)
Expand Down
28 changes: 28 additions & 0 deletions distributor-node/docs/commands/leader.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Commands for performing Distribution Working Group leader on-chain duties (like
* [`joystream-distributor leader:remove-bucket-operator`](#joystream-distributor-leaderremove-bucket-operator)
* [`joystream-distributor leader:set-bucket-family-metadata`](#joystream-distributor-leaderset-bucket-family-metadata)
* [`joystream-distributor leader:set-buckets-per-bag-limit`](#joystream-distributor-leaderset-buckets-per-bag-limit)
* [`joystream-distributor leader:set-node-operational-status`](#joystream-distributor-leaderset-node-operational-status)
* [`joystream-distributor leader:update-bag`](#joystream-distributor-leaderupdate-bag)
* [`joystream-distributor leader:update-bucket-mode`](#joystream-distributor-leaderupdate-bucket-mode)
* [`joystream-distributor leader:update-bucket-status`](#joystream-distributor-leaderupdate-bucket-status)
Expand Down Expand Up @@ -210,6 +211,33 @@ OPTIONS

_See code: [src/commands/leader/set-buckets-per-bag-limit.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/set-buckets-per-bag-limit.ts)_

## `joystream-distributor leader:set-node-operational-status`

Set/update distribution node operational status. Requires distribution working group leader permissions.

```
USAGE
$ joystream-distributor leader:set-node-operational-status

OPTIONS
-B, --bucketId=bucketId (required) Distribution bucket ID in
{familyId}:{bucketIndex} format.

-c, --configPath=configPath [default: ./config.yml] Path to config
JSON/YAML file (relative to current working
directory)

-o, --operationalStatus=(Normal|NoService|NoServiceFrom|NoServiceDuring) Operational status of the operator

-w, --workerId=workerId (required) ID of the operator (distribution
group worker)

-y, --yes Answer "yes" to any prompt, skipping any
manual confirmations
```

_See code: [src/commands/leader/set-node-operational-status.ts](https://github.com/Joystream/joystream/blob/master/src/commands/leader/set-node-operational-status.ts)_

## `joystream-distributor leader:update-bag`

Add/remove distribution buckets from a bag.
Expand Down
20 changes: 13 additions & 7 deletions distributor-node/docs/commands/operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,24 @@ USAGE
$ joystream-distributor operator:set-metadata

OPTIONS
-B, --bucketId=bucketId (required) Distribution bucket ID in {familyId}:{bucketIndex} format.
-B, --bucketId=bucketId (required) Distribution bucket ID in
{familyId}:{bucketIndex} format.

-c, --configPath=configPath [default: ./config.yml] Path to config JSON/YAML file (relative to current working
directory)
-c, --configPath=configPath [default: ./config.yml] Path to config
JSON/YAML file (relative to current working
directory)

-e, --endpoint=endpoint Root distribution node endpoint
-e, --endpoint=endpoint Root distribution node endpoint

-i, --input=input Path to JSON metadata file
-i, --input=input Path to JSON metadata file

-w, --workerId=workerId (required) ID of the operator (distribution group worker)
-o, --operationalStatus=(Normal|NoService|NoServiceFrom|NoServiceDuring) Operational status of the operator

-y, --yes Answer "yes" to any prompt, skipping any manual confirmations
-w, --workerId=workerId (required) ID of the operator (distribution
group worker)

-y, --yes Answer "yes" to any prompt, skipping any
manual confirmations

DESCRIPTION
Requires active distribution bucket operator worker role key.
Expand Down
5 changes: 3 additions & 2 deletions distributor-node/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@joystream/distributor-cli",
"description": "Joystream distributor node CLI",
"version": "2.0.0",
"version": "2.1.0",
"author": "Joystream contributors",
"bin": {
"joystream-distributor": "./bin/run"
Expand All @@ -10,7 +10,7 @@
"dependencies": {
"@apollo/client": "^3.2.5",
"@elastic/ecs-winston-format": "^1.1.0",
"@joystream/metadata-protobuf": "^2.8.0",
"@joystream/metadata-protobuf": "^2.9.0",
"@joystream/opentelemetry": "1.0.0",
"@joystream/storage-node-client": "^4.0.0",
"@joystream/types": "^2.0.0",
Expand All @@ -34,6 +34,7 @@
"graphql": "^15.3.0",
"graphql-tag": "^2.12.6",
"inquirer": "^8.1.2",
"inquirer-datepicker": "2.0.2",
"js-image-generator": "^1.0.3",
"jsonwebtoken": "^8.5.1",
"lodash": "^4.17.21",
Expand Down
1 change: 1 addition & 0 deletions distributor-node/src/@types/inquirer-datepicker/index.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
declare module 'inquirer-datepicker'
27 changes: 22 additions & 5 deletions distributor-node/src/command-base/default.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import Command, { flags as oclifFlags } from '@oclif/command'
import inquirer from 'inquirer'
import ExitCodes from './ExitCodes'
import { ReadonlyConfig } from '../types/config'
import { ConfigParserService } from '../services/parsers/ConfigParserService'
import { LoggingService } from '../services/logging'
import inquirer, { DistinctQuestion } from 'inquirer'
import inquirerDatepicker from 'inquirer-datepicker'
import { Logger } from 'winston'
import { LoggingService } from '../services/logging'
import { BagIdParserService } from '../services/parsers/BagIdParserService'
import { BucketIdParserService } from '../services/parsers/BucketIdParserService'
import { ConfigParserService } from '../services/parsers/ConfigParserService'
import { ReadonlyConfig } from '../types/config'
import ExitCodes from './ExitCodes'

export const flags = {
...oclifFlags,
Expand Down Expand Up @@ -75,6 +76,7 @@ export default abstract class DefaultCommandBase extends Command {
this.logging = LoggingService.withCLIConfig()
this.logger = this.logging.createLogger('CLI')
this.autoConfirm = !!(process.env.AUTO_CONFIRM === 'true' || parseInt(process.env.AUTO_CONFIRM || '') || yes)
inquirer.registerPrompt('datepicker', inquirerDatepicker)
}

public log(message: string, ...meta: unknown[]): void {
Expand All @@ -98,6 +100,21 @@ export default abstract class DefaultCommandBase extends Command {
}
}

async datePrompt(question: DistinctQuestion): Promise<Date> {
const { result } = await inquirer.prompt([
{
...question,
type: 'datepicker',
name: 'result',
clearable: true,
default: new Date('2017-09-28 17:36:05').toISOString(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried the date picker, works really nicely in the terminal!

Perhaps a more reasonable default would be the current date new Date().toISOString(), so less fiddling with inital year and month for the user.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

},
])

const date = new Date(result)
return date
}

async finally(err: unknown): Promise<void> {
if (!err) this.exit(ExitCodes.OK)
if (process.env.DEBUG === 'true') {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import {
INodeOperationalStatusMetadata,
ISetNodeOperationalStatus,
NodeOperationalStatusMetadata,
SetNodeOperationalStatus,
} from '@joystream/metadata-protobuf'
import AccountsCommandBase from '../../command-base/accounts'
import DefaultCommandBase, { flags } from '../../command-base/default'
import { NODE_OPERATIONAL_STATUS_OPTIONS, NodeOperationalStatus } from '../../types/metadata'

export default class LeadSetNodeOperationalStatus extends AccountsCommandBase {
static description = `Set/update distribution node operational status. Requires distribution working group leader permissions.`

static flags = {
bucketId: flags.bucketId({
required: true,
}),
workerId: flags.integer({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggested that worker id is not needed in metadata. If we agree on that, then this workerId can be dropped.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see #4793 (comment)

char: 'w',
description: 'ID of the operator (distribution group worker)',
required: true,
}),
operationalStatus: flags.enum<NodeOperationalStatus>({
char: 'o',
options: [...NODE_OPERATIONAL_STATUS_OPTIONS],
required: false,
description: 'Operational status of the operator',
}),
...DefaultCommandBase.flags,
}

async run(): Promise<void> {
const { bucketId, workerId, operationalStatus: statusType } = this.parse(LeadSetNodeOperationalStatus).flags
const leadKey = await this.getDistributorLeadKey()

let operationalStatus: INodeOperationalStatusMetadata = {}
switch (statusType) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

statusType input is not required, should we default to Normal?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes more sense to make it required, the command name doesn't imply what the default value should be.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes more sense to make it required, the command name doesn't imply what the default value should be.

Done

case 'Normal': {
operationalStatus = { status: NodeOperationalStatusMetadata.OperationalStatus.NORMAL }
break
}
case 'NoService': {
operationalStatus = { status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE }
break
}
case 'NoServiceFrom': {
operationalStatus = {
status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE,
noServiceFrom: (await this.datePrompt({ message: 'Enter No Service period start date' })).toISOString(),
}
break
}
case 'NoServiceDuring': {
operationalStatus = {
status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE,
noServiceFrom: (await this.datePrompt({ message: 'Enter No Service period start date' })).toISOString(),
noServiceTo: (await this.datePrompt({ message: 'Enter No Service period end date' })).toISOString(),
}
}
}

this.log(`Setting node operational status...`, {
bucketId: bucketId.toHuman(),
workerId,
operationalStatus,
})

const metadata: ISetNodeOperationalStatus = {
workerId: workerId.toString(),
bucketId: `${bucketId.distributionBucketFamilyId}:${bucketId.distributionBucketIndex}`,
operationalStatus,
}
await this.sendAndFollowTx(
await this.getDecodedPair(leadKey),
this.api.tx.distributionWorkingGroup.leadRemark(
'0x' + Buffer.from(SetNodeOperationalStatus.encode(metadata).finish()).toString('hex')
)
)
this.log('Bucket operator metadata successfully set/updated!')
}
}
66 changes: 60 additions & 6 deletions distributor-node/src/commands/operator/set-metadata.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
import {
DistributionBucketOperatorMetadata,
IDistributionBucketOperatorMetadata,
INodeOperationalStatusMetadata,
NodeOperationalStatusMetadata,
} from '@joystream/metadata-protobuf'
import fs from 'fs'
import AccountsCommandBase from '../../command-base/accounts'
import DefaultCommandBase, { flags } from '../../command-base/default'
import { ValidationService } from '../../services/validation/ValidationService'
import { DistributionBucketOperatorMetadata, IDistributionBucketOperatorMetadata } from '@joystream/metadata-protobuf'
import { NODE_OPERATIONAL_STATUS_OPTIONS, NodeOperationalStatus } from '../../types/metadata'

export default class OperatorSetMetadata extends AccountsCommandBase {
static description = `Set/update distribution bucket operator metadata.
Expand All @@ -22,6 +28,12 @@ export default class OperatorSetMetadata extends AccountsCommandBase {
description: 'Root distribution node endpoint',
exclusive: ['input'],
}),
operationalStatus: flags.enum<NodeOperationalStatus>({
char: 'o',
options: [...NODE_OPERATIONAL_STATUS_OPTIONS],
required: false,
description: 'Operational status of the operator',
}),
input: flags.string({
char: 'i',
description: 'Path to JSON metadata file',
Expand All @@ -31,18 +43,60 @@ export default class OperatorSetMetadata extends AccountsCommandBase {
}

async run(): Promise<void> {
const { bucketId, workerId, input, endpoint } = this.parse(OperatorSetMetadata).flags
const { bucketId, workerId, input, endpoint, operationalStatus: statusType } = this.parse(OperatorSetMetadata).flags
const workerKey = await this.getDistributorWorkerRoleKey(workerId)

let operationalStatus: INodeOperationalStatusMetadata = {}
switch (statusType) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question about default status here

case 'Normal': {
operationalStatus = { status: NodeOperationalStatusMetadata.OperationalStatus.NORMAL }
break
}
case 'NoService': {
operationalStatus = { status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE }
break
}
case 'NoServiceFrom': {
operationalStatus = {
status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE,
noServiceFrom: (await this.datePrompt({ message: 'Enter No Service period start date' })).toISOString(),
}
break
}
case 'NoServiceDuring': {
operationalStatus = {
status: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE,
noServiceFrom: (await this.datePrompt({ message: 'Enter No Service period start date' })).toISOString(),
noServiceTo: (await this.datePrompt({ message: 'Enter No Service period end date' })).toISOString(),
}
}
}

const validation = new ValidationService()
const metadata: IDistributionBucketOperatorMetadata = input
? validation.validate('OperatorMetadata', JSON.parse(fs.readFileSync(input).toString()))
: { endpoint }
let metadata: IDistributionBucketOperatorMetadata
if (input) {
const params = validation.validate('OperatorMetadata', JSON.parse(fs.readFileSync(input).toString()))
metadata = {
...params,
operationalStatus: params.operationalStatus
? {
...params.operationalStatus,
status:
params.operationalStatus?.status === 'Normal'
? NodeOperationalStatusMetadata.OperationalStatus.NORMAL
: NodeOperationalStatusMetadata.OperationalStatus.NO_SERVICE,
}
: {},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this override the current status? Or skip the update?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should skip the update but previously it was overriding, I have fixed it

}
} else {
metadata = { endpoint, operationalStatus }
}

this.log(`Setting bucket operator metadata...`, {
bucketId: bucketId.toHuman(),
workerId,
metadata,
operationalStatus,
})
await this.sendAndFollowTx(
await this.getDecodedPair(workerKey),
Expand All @@ -52,6 +106,6 @@ export default class OperatorSetMetadata extends AccountsCommandBase {
'0x' + Buffer.from(DistributionBucketOperatorMetadata.encode(metadata).finish()).toString('hex')
)
)
this.log('Bucket operator metadata succesfully set/updated!')
this.log('Bucket operator metadata successfully set/updated!')
}
}
10 changes: 10 additions & 0 deletions distributor-node/src/schemas/operatorMetadataSchema.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { JSONSchema4 } from 'json-schema'
import { NODE_OPERATIONAL_STATUS_OPTIONS } from '../types/metadata'

export const operatorMetadataSchema: JSONSchema4 = {
type: 'object',
Expand All @@ -21,6 +22,15 @@ export const operatorMetadataSchema: JSONSchema4 = {
},
},
},
operationalStatus: {
type: 'object',
properties: {
status: { type: 'string', enum: [...NODE_OPERATIONAL_STATUS_OPTIONS] },
noServiceFrom: { type: 'string', format: 'date-time' },
noServiceTo: { type: 'string', format: 'date-time' },
rationale: { type: 'string' },
},
},
extra: { type: 'string' },
},
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ query getDistributionBucketsWithBagsByIds($ids: [String!]) {
}
}

query getDistributionBucketsWithBagsByWorkerId($workerId: Int!) {
query getDistributionBucketsWithBagsByWorkerId($workerId: BigInt!) {
distributionBuckets(where: { operators_some: { workerId_eq: $workerId, status_eq: ACTIVE } }) {
...DistributionBucketWithBags
}
Expand All @@ -107,7 +107,17 @@ fragment StorageBucketOperatorFields on StorageBucket {
}

query getActiveStorageBucketOperatorsData {
storageBuckets(where: { operatorStatus: { isTypeOf_eq: "StorageBucketOperatorStatusActive" } }) {
storageBuckets(
where: {
operatorStatus: { isTypeOf_eq: "StorageBucketOperatorStatusActive" }
operatorMetadata: {
OR: [
{ nodeOperationalStatus_isNull: true }
{ nodeOperationalStatus: { isTypeOf_eq: "NodeOperationalStatusNormal" } }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This filter would skip any nodes with future maintenance as well. We probably should get all the operators and later only filter those that are under maintenance currently

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0537ee7

]
}
}
) {
...StorageBucketOperatorFields
}
}
Expand Down
Loading
Loading