Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for migrations. #267

Closed
wants to merge 37 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
b78c95f
feat: Add migrationVariation method. (#212)
kinyoklion Jul 24, 2023
b48f847
feat: Add migration configuration and basic migration. (#213)
kinyoklion Jul 26, 2023
72c0f53
feat: Add support for payloads to read and write methods. (#215)
kinyoklion Jul 28, 2023
36fef4f
feat: Add support for exclude from summaries. (#216)
kinyoklion Jul 28, 2023
59b8b6f
Merge branch 'main' into feat/node-migrations
kinyoklion Jul 31, 2023
bee4e74
chore: Prettier.
kinyoklion Jul 31, 2023
2a1eb6f
feat: Add migration operation input event and tracker. (#214)
kinyoklion Aug 1, 2023
87095c9
feat: Implement migration op event and connect to tracking. (#218)
kinyoklion Aug 1, 2023
a2154cc
feat: Adjustments to spec and enhanced event validation. (#222)
kinyoklion Aug 3, 2023
ababe74
feat: Add configuration overrides and metrics data kinds. (#220)
kinyoklion Aug 3, 2023
f750bd5
feat: Add custom event support to tracker. (#227)
kinyoklion Aug 4, 2023
9a6bd82
Merge branch 'main' into feat/node-migrations
kinyoklion Aug 16, 2023
02e92a4
feat: Remove custom events. (#243)
kinyoklion Aug 16, 2023
f9b4e6e
feat: Event sampling. (#245)
kinyoklion Aug 23, 2023
ab61937
chore: Add execution order support for contract tests. (#247)
kinyoklion Aug 23, 2023
bd95697
feat: Change migration variation to support forwarding the sampling r…
kinyoklion Aug 23, 2023
ee62381
feat: New data kinds for edge SDKs. (#260)
kinyoklion Aug 28, 2023
ee4ebbf
feat: Add invoked measurement. (#258)
kinyoklion Aug 30, 2023
760567d
chore: Support migration payload contract tests. (#262)
kinyoklion Aug 30, 2023
e12068a
chore: Add initial documentation. (#263)
kinyoklion Aug 30, 2023
01fd110
Merge branch 'main' into rlamb/merge-perf-updates-migrations
kinyoklion Aug 31, 2023
f7ed7eb
fix sampling
kinyoklion Aug 31, 2023
10c4875
fix merging event sampling
kinyoklion Aug 31, 2023
7c08e82
feat: Refactor variation method and consistency tracking. (#264)
kinyoklion Aug 31, 2023
e56e18f
Merge branch 'feat/node-migrations' into rlamb/merge-perf-updates-mig…
kinyoklion Aug 31, 2023
d7b8170
Merge branch 'main' into feat/node-migrations
kinyoklion Aug 31, 2023
cadc90b
fix: Fix double call when platform support performance API. (#268)
kinyoklion Sep 5, 2023
11179ba
fix: Fix log messages for failed migration creation. (#274)
kinyoklion Sep 11, 2023
5ff85e4
chore: Use new granular categories for event sampling. (#277)
kinyoklion Sep 13, 2023
d1f7197
fix: Handle exceptions thrown in the comparison function. (#278)
kinyoklion Sep 15, 2023
c120948
feat: Do not generate an event if the measurements are inconsistent. …
kinyoklion Sep 22, 2023
9c5f402
feat: Add flag version to migration op event. (#281)
kinyoklion Sep 22, 2023
f3481a4
feat: Use a factory method to create migrations. (#283)
kinyoklion Sep 25, 2023
dda6b37
feat: No migration op event for empty flag key. (#287)
kinyoklion Sep 27, 2023
8e96a52
feat: Add typed variation methods. (#288)
kinyoklion Sep 27, 2023
837c4e1
fix: Include flag version for WRONG_TYPE migrations. (#290)
kinyoklion Oct 2, 2023
c3f2d88
feat: Remove index and custom event sampling. (#289)
kinyoklion Oct 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions contract-tests/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ app.get('/', (req, res) => {
'tags',
'big-segments',
'user-type',
'migrations',
'event-sampling',
'config-override-kind',
'metric-kind',
'strongly-typed',
],
});
});
Expand Down
155 changes: 151 additions & 4 deletions contract-tests/sdkClientEntity.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
import ld from 'node-server-sdk';
import got from 'got';
import ld, {
LDConcurrentExecution,
LDExecutionOrdering,
LDMigrationError,
LDMigrationSuccess,
LDSerialExecution,
createMigration,
} from 'node-server-sdk';

import BigSegmentTestStore from './BigSegmentTestStore.js';
import { Log, sdkLogger } from './log.js';
Expand Down Expand Up @@ -54,6 +62,30 @@ export function makeSdkConfig(options, tag) {
return cf;
}

function getExecution(order) {
switch (order) {
case 'serial': {
return new LDSerialExecution(LDExecutionOrdering.Fixed);
}
case 'random': {
return new LDSerialExecution(LDExecutionOrdering.Random);
}
case 'concurrent': {
return new LDConcurrentExecution();
}
default: {
throw new Error('Unsupported execution order.');
}
}
}

function makeMigrationPostOptions(payload) {
if (payload) {
return { body: payload };
}
return {};
}

export async function newSdkClientEntity(options) {
const c = {};
const log = Log(options.tag);
Expand Down Expand Up @@ -92,10 +124,30 @@ export async function newSdkClientEntity(options) {
case 'evaluate': {
const pe = params.evaluate;
if (pe.detail) {
return await client.variationDetail(pe.flagKey, pe.context || pe.user, pe.defaultValue);
switch(pe.valueType) {
case "bool":
return await client.boolVariationDetail(pe.flagKey, pe.context || pe.user, pe.defaultValue);
case "int": // Intentional fallthrough.
case "double":
return await client.numberVariationDetail(pe.flagKey, pe.context || pe.user, pe.defaultValue);
case "string":
return await client.stringVariationDetail(pe.flagKey, pe.context || pe.user, pe.defaultValue);
default:
return await client.variationDetail(pe.flagKey, pe.context || pe.user, pe.defaultValue);
}

} else {
const value = await client.variation(pe.flagKey, pe.context || pe.user, pe.defaultValue);
return { value };
switch(pe.valueType) {
case "bool":
return {value: await client.boolVariation(pe.flagKey, pe.context || pe.user, pe.defaultValue)};
case "int": // Intentional fallthrough.
case "double":
return {value: await client.numberVariation(pe.flagKey, pe.context || pe.user, pe.defaultValue)};
case "string":
return {value: await client.stringVariation(pe.flagKey, pe.context || pe.user, pe.defaultValue)};
default:
return {value: await client.variation(pe.flagKey, pe.context || pe.user, pe.defaultValue)};
}
}
}

Expand Down Expand Up @@ -126,6 +178,101 @@ export async function newSdkClientEntity(options) {
case 'getBigSegmentStoreStatus':
return await client.bigSegmentStoreStatusProvider.requireStatus();

case 'migrationVariation':
const migrationVariation = params.migrationVariation;
const res = await client.migrationVariation(
migrationVariation.key,
migrationVariation.context,
migrationVariation.defaultStage,
);
return { result: res.value };

case 'migrationOperation':
const migrationOperation = params.migrationOperation;
const readExecutionOrder = migrationOperation.readExecutionOrder;

const migration = createMigration(client, {
execution: getExecution(readExecutionOrder),
latencyTracking: migrationOperation.trackLatency,
errorTracking: migrationOperation.trackErrors,
check: migrationOperation.trackConsistency ? (a, b) => a === b : undefined,
readNew: async (payload) => {
try {
const res = await got.post(
migrationOperation.newEndpoint,
makeMigrationPostOptions(payload),
);
return LDMigrationSuccess(res.body);
} catch (err) {
return LDMigrationError(err.message);
}
},
writeNew: async (payload) => {
try {
const res = await got.post(
migrationOperation.newEndpoint,
makeMigrationPostOptions(payload),
);
return LDMigrationSuccess(res.body);
} catch (err) {
return LDMigrationError(err.message);
}
},
readOld: async (payload) => {
try {
const res = await got.post(
migrationOperation.oldEndpoint,
makeMigrationPostOptions(payload),
);
return LDMigrationSuccess(res.body);
} catch (err) {
return LDMigrationError(err.message);
}
},
writeOld: async (payload) => {
try {
const res = await got.post(
migrationOperation.oldEndpoint,
makeMigrationPostOptions(payload),
);
return LDMigrationSuccess(res.body);
} catch (err) {
return LDMigrationError(err.message);
}
},
});

switch (migrationOperation.operation) {
case 'read': {
const res = await migration.read(
migrationOperation.key,
migrationOperation.context,
migrationOperation.defaultStage,
migrationOperation.payload,
);
if (res.success) {
return { result: res.result };
} else {
return { result: res.error };
}
}
case 'write': {
const res = await migration.write(
migrationOperation.key,
migrationOperation.context,
migrationOperation.defaultStage,
migrationOperation.payload,
);

if (res.authoritative.success) {
return { result: res.authoritative.result };
} else {
return { result: res.authoritative.error };
}
}
}
return undefined;

default:
throw badCommandError;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ export class EdgeFeatureStore implements LDFeatureStore {
callback(item.segments[dataKey]);
break;
default:
throw new Error(`Unsupported DataKind: ${namespace}`);
callback(null);
}
} catch (err) {
this.logger.error(err);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ import {
} from '../../../src/api/subsystem';
import { EventProcessor, InputIdentifyEvent } from '../../../src/internal';
import { EventProcessorOptions } from '../../../src/internal/events/EventProcessor';
import shouldSample from '../../../src/internal/events/sampling';

jest.mock('../../../src/internal/events/sampling', () => ({
__esModule: true,
default: jest.fn(() => true),
}));

const user = { key: 'userKey', name: 'Red' };
const userWithFilteredName = {
Expand Down Expand Up @@ -195,6 +201,8 @@ describe('given an event processor', () => {
beforeEach(() => {
eventSender = new MockEventSender();
contextDeduplicator = new MockContextDeduplicator();
// @ts-ignore
shouldSample.mockImplementation(() => true);

eventProcessor = new EventProcessor(
eventProcessorConfig,
Expand Down Expand Up @@ -286,6 +294,8 @@ describe('given an event processor', () => {
value: 'value',
trackEvents: true,
default: 'default',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand All @@ -302,6 +312,70 @@ describe('given an event processor', () => {
]);
});

it('uses sampling ratio for feature events', async () => {
Date.now = jest.fn(() => 1000);
eventProcessor.sendEvent({
kind: 'feature',
creationDate: 1000,
context: Context.fromLDContext(user),
key: 'flagkey',
version: 11,
variation: 1,
value: 'value',
trackEvents: true,
default: 'default',
samplingRatio: 2,
withReasons: true,
});

await eventProcessor.flush();
const request = await eventSender.queue.take();
expect(shouldSample).toHaveBeenCalledWith(2);

expect(request.data).toEqual([
{
kind: 'index',
creationDate: 1000,
context: { ...user, kind: 'user' },
},
{ ...makeFeatureEvent(1000, 11), samplingRatio: 2 },
makeSummary(1000, 1000, 1, 11),
]);
});

it('excludes feature events that are not sampled', async () => {
// @ts-ignore
shouldSample.mockImplementation((ratio) => ratio !== 2);
Date.now = jest.fn(() => 1000);
eventProcessor.sendEvent({
kind: 'feature',
creationDate: 1000,
context: Context.fromLDContext(user),
key: 'flagkey',
version: 11,
variation: 1,
value: 'value',
trackEvents: true,
default: 'default',
samplingRatio: 2,
withReasons: true,
});

await eventProcessor.flush();
const request = await eventSender.queue.take();
expect(shouldSample).toHaveBeenCalledWith(2);
expect(shouldSample).toHaveBeenCalledWith(1);

expect(request.data).toEqual([
{
kind: 'index',
creationDate: 1000,
context: { ...user, kind: 'user' },
},
makeSummary(1000, 1000, 1, 11),
]);
});

it('handles the version being 0', async () => {
Date.now = jest.fn(() => 1000);
eventProcessor.sendEvent({
Expand All @@ -314,6 +388,8 @@ describe('given an event processor', () => {
value: 'value',
trackEvents: true,
default: 'default',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -344,6 +420,8 @@ describe('given an event processor', () => {
trackEvents: false,
debugEventsUntilDate: 2000,
default: 'default',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -373,6 +451,8 @@ describe('given an event processor', () => {
trackEvents: true,
debugEventsUntilDate: 2000,
default: 'default',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -409,6 +489,8 @@ describe('given an event processor', () => {
trackEvents: false,
debugEventsUntilDate: 1500,
default: 'default',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -438,6 +520,8 @@ describe('given an event processor', () => {
value: 'value',
trackEvents: true,
default: 'default',
samplingRatio: 1,
withReasons: true,
});
eventProcessor.sendEvent({
kind: 'feature',
Expand All @@ -449,6 +533,8 @@ describe('given an event processor', () => {
value: 'carrot',
trackEvents: true,
default: 'potato',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -511,6 +597,8 @@ describe('given an event processor', () => {
value: 'value',
trackEvents: false,
default: 'default',
samplingRatio: 1,
withReasons: true,
});
eventProcessor.sendEvent({
kind: 'feature',
Expand All @@ -522,6 +610,8 @@ describe('given an event processor', () => {
value: 'carrot',
trackEvents: false,
default: 'potato',
samplingRatio: 1,
withReasons: true,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -577,6 +667,7 @@ describe('given an event processor', () => {
context: Context.fromLDContext(user),
key: 'eventkey',
data: { thing: 'stuff' },
samplingRatio: 1,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -607,6 +698,7 @@ describe('given an event processor', () => {
context: Context.fromLDContext(anonUser),
key: 'eventkey',
data: { thing: 'stuff' },
samplingRatio: 1,
});

await eventProcessor.flush();
Expand Down Expand Up @@ -639,6 +731,7 @@ describe('given an event processor', () => {
key: 'eventkey',
data: { thing: 'stuff' },
metricValue: 1.5,
samplingRatio: 1,
});

await eventProcessor.flush();
Expand Down
Loading