Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: local file metadata provider #56

Merged
merged 4 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/metadata/src/exceptions/fileNotFound.exception.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
export class FileNotFound extends Error {
constructor(path: string) {
super(`File not found at path: ${path}`);
}
}
1 change: 1 addition & 0 deletions packages/metadata/src/exceptions/index.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export * from "./invalidSchema.exception.js";
export * from "./fetchError.exception.js";
export * from "./fileNotFound.exception.js";
8 changes: 6 additions & 2 deletions packages/metadata/src/external.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
export type { IMetadataProvider } from "./internal.js";

export { InvalidSchema, FetchError } from "./internal.js";
export { InvalidSchema, FetchError, FileNotFound } from "./internal.js";

export { StaticMetadataProvider, GithubMetadataProvider } from "./internal.js";
export {
StaticMetadataProvider,
GithubMetadataProvider,
LocalFileMetadataProvider,
} from "./internal.js";
24 changes: 24 additions & 0 deletions packages/metadata/src/interfaces/metadata.interface.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,30 @@
import { Token, TokenType, ZKChainMetadata } from "@zkchainhub/shared";

/**
* Represents a metadata provider that retrieves chains and tokens metadata.
*/
export interface IMetadataProvider {
/**
* Retrieves the metadata for ZK chains of the ecosystem
*
* @returns A promise that resolves to the metadata of ZK chains.
*
* @throws {FetchError}
* If there is an issue with the network request.
*
*
* @throws {InvalidSchema}
* If the response data is invalid or cannot be parsed.
*/
getChainsMetadata(): Promise<ZKChainMetadata>;

/**
* Retrieves metadata for tokens of the ecosystem
*
* @returns A promise that resolves to an array of token metadata.
*
* @throws {FetchError} If there is an issue with the network request.
* @throws {InvalidSchema} If the response data is invalid or cannot be parsed.
*/
getTokensMetadata(): Promise<Token<TokenType>[]>;
}
80 changes: 52 additions & 28 deletions packages/metadata/src/providers/githubMetadata.provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import axios, { AxiosInstance } from "axios";
import { z } from "zod";

import {
Cache,
ILogger,
Token,
TokenType,
Expand All @@ -13,6 +14,8 @@ import { IMetadataProvider } from "../interfaces/index.js";
import { FetchError, InvalidSchema } from "../internal.js";
import { ChainSchema, TokenSchema } from "../schemas/index.js";

export const GITHUB_METADATA_PREFIX = "github-metadata";

/**
* Represents a provider for retrieving metadata from GitHub.
*/
Expand All @@ -22,6 +25,7 @@ export class GithubMetadataProvider implements IMetadataProvider {
private readonly tokenJsonUrl: string,
private readonly chainJsonUrl: string,
private readonly logger: ILogger,
private readonly cache: Cache,
) {
this.axios = axios.create({
headers: {
Expand All @@ -30,44 +34,64 @@ export class GithubMetadataProvider implements IMetadataProvider {
});
}

/** @inheritdoc */
async getChainsMetadata(): Promise<ZKChainMetadata> {
const { data } = await this.axios.get(this.chainJsonUrl).catch((e) => {
this.logger.error(
`Failed to fetch chains metadata from ${this.chainJsonUrl}: ${e.message}`,
);
throw new FetchError(`Failed to fetch chains metadata: ${e.message}`);
});
let cachedData = await this.cache.get<ZKChainMetadata | undefined>(
`${GITHUB_METADATA_PREFIX}-chains`,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use some sort of unique id here ? like UUID?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any suggestions here ? @0xyaco

Copy link
Collaborator

@0xyaco 0xyaco Aug 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess there would be a possibility of, given a shared Cache instance between different classes, to overwrite a cached value by accident if someone uses the same key in multiple classes.

I can't come up with anything simple as a workaround to that issue right now lol so I think we can keep it as is.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an alternative im thinking now would be a wrapper around the cache, like BrandedCache or smth like that:

BrandedCache is Cache {
constructor(cache: Cache, prefix: string)
...wraps calls
}

and that this class wraps the cache methods and automatically adds the prefix to the key

but seems too much at this point

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it's too much, let's just be aware of this issue when defining cache keys.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets keep it as it is nigiri, no need for uuid

);
if (!cachedData) {
const { data } = await this.axios.get(this.chainJsonUrl).catch((e) => {
this.logger.error(
`Failed to fetch chains metadata from ${this.chainJsonUrl}: ${e.message}`,
);
throw new FetchError(`Failed to fetch chains metadata: ${e.message}`);
});

const validatedData = z.array(ChainSchema).safeParse(data);

const validatedData = z.array(ChainSchema).safeParse(data);
if (!validatedData.success) {
this.logger.error(`Invalid ZKChain metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid ZKChain metadata");
}

if (!validatedData.success) {
this.logger.error(`Invalid ZKChain metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid ZKChain metadata");
cachedData = validatedData.data.reduce((acc, chain) => {
const { chainId, ...rest } = chain;
const chainIdBn = BigInt(chainId);
acc.set(chainIdBn, { ...rest, chainId: chainIdBn });
return acc;
}, new Map<bigint, ZKChainMetadataItem>());

await this.cache.set(`${GITHUB_METADATA_PREFIX}-chains`, cachedData);
}

return validatedData.data.reduce((acc, chain) => {
const { chainId, ...rest } = chain;
const chainIdBn = BigInt(chainId);
acc.set(chainIdBn, { ...rest, chainId: chainIdBn });
return acc;
}, new Map<bigint, ZKChainMetadataItem>());
return cachedData;
}

/** @inheritdoc */
async getTokensMetadata(): Promise<Token<TokenType>[]> {
const { data } = await this.axios.get(this.tokenJsonUrl).catch((e) => {
this.logger.error(
`Failed to fetch chains metadata from ${this.chainJsonUrl}: ${e.message}`,
);
throw new FetchError(`Failed to fetch chains metadata: ${e.message}`);
});
let cachedData = await this.cache.get<Token<TokenType>[] | undefined>(
`${GITHUB_METADATA_PREFIX}-tokens`,
);

const validatedData = z.array(TokenSchema).safeParse(data);
if (!cachedData) {
const { data } = await this.axios.get(this.tokenJsonUrl).catch((e) => {
this.logger.error(
`Failed to fetch chains metadata from ${this.chainJsonUrl}: ${e.message}`,
);
throw new FetchError(`Failed to fetch chains metadata: ${e.message}`);
});

if (!validatedData.success) {
this.logger.error(`Invalid Token metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid Token metadata");
}
const validatedData = z.array(TokenSchema).safeParse(data);

if (!validatedData.success) {
this.logger.error(`Invalid Token metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid Token metadata");
}

return validatedData.data;
cachedData = validatedData.data;

await this.cache.set(`${GITHUB_METADATA_PREFIX}-tokens`, cachedData);
}
return cachedData;
}
}
1 change: 1 addition & 0 deletions packages/metadata/src/providers/index.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export * from "./githubMetadata.provider.js";
export * from "./staticMetadata.provider.js";
export * from "./localFileMetadata.provider.js";
91 changes: 91 additions & 0 deletions packages/metadata/src/providers/localFileMetadata.provider.ts
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was wondering if we want to index the token list by address like { [key:address] : Token<"erc20">} for example...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought about it too but i would leave that task for when we refactor the pricing service, so now everything still works as it is, wdyt?

Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
import { existsSync, readFileSync } from "fs";
import { z } from "zod";

import {
ILogger,
Token,
TokenType,
ZKChainMetadata,
ZKChainMetadataItem,
} from "@zkchainhub/shared";

import { FileNotFound, IMetadataProvider, InvalidSchema } from "../internal.js";
import { ChainSchema, TokenSchema } from "../schemas/index.js";

export const LOCALFILE_METADATA_PREFIX = "local-metadata";

/**
* Represents a provider that retrieves metadata from local files.
* Note: Files are read only once and saved to instance variables.
*/
export class LocalFileMetadataProvider implements IMetadataProvider {
private readonly chainMetadata: ZKChainMetadata;
private readonly tokenMetadata: Token<TokenType>[];

/**
* Constructs a new instance of the LocalFileMetadataProvider class.
* @param tokenJsonPath The path to the token JSON file.
* @param chainJsonPath The path to the chain JSON file.
* @param logger The logger instance.
* @throws {FileNotFound} if any of the files is not found.
*/
constructor(
private readonly tokenJsonPath: string,
private readonly chainJsonPath: string,
private readonly logger: ILogger,
) {
if (!existsSync(tokenJsonPath)) {
throw new FileNotFound(tokenJsonPath);
}

if (!existsSync(chainJsonPath)) {
throw new FileNotFound(chainJsonPath);
}

this.tokenMetadata = this.readAndParseTokenMetadata();
this.chainMetadata = this.readAndParseChainMetadata();
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add natspec on IMetadataProvider and reuse it on the classes that implement the interface

/** @inheritdoc */
async getChainsMetadata(): Promise<ZKChainMetadata> {
return Promise.resolve(this.chainMetadata);
}

readAndParseChainMetadata() {
const jsonData = readFileSync(this.chainJsonPath, "utf-8");
const parsed = JSON.parse(jsonData);

const validatedData = z.array(ChainSchema).safeParse(parsed);

if (!validatedData.success) {
this.logger.error(`Invalid ZKChains metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid ZKChains metadata");
}

return validatedData.data.reduce((acc, chain) => {
const { chainId, ...rest } = chain;
const chainIdBn = BigInt(chainId);
acc.set(chainIdBn, { ...rest, chainId: chainIdBn });
return acc;
}, new Map<bigint, ZKChainMetadataItem>());
}

/** @inheritdoc */
async getTokensMetadata(): Promise<Token<TokenType>[]> {
return Promise.resolve(this.tokenMetadata);
}

readAndParseTokenMetadata() {
const jsonData = readFileSync(this.tokenJsonPath, "utf-8");
const parsed = JSON.parse(jsonData);

const validatedData = z.array(TokenSchema).safeParse(parsed);

if (!validatedData.success) {
this.logger.error(`Invalid Tokens metadata: ${validatedData.error.errors}`);
throw new InvalidSchema("Invalid Tokens metadata");
}

return validatedData.data;
}
}
6 changes: 6 additions & 0 deletions packages/metadata/src/providers/staticMetadata.provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,16 @@ import { Token, tokens, TokenType, ZKChainMetadata, zkChainsMetadata } from "@zk

import { IMetadataProvider } from "../interfaces/index.js";

/**
* Represents a provider that retrieves metadata from static data of mainnet.
*/
export class StaticMetadataProvider implements IMetadataProvider {
/** @inheritdoc */
async getChainsMetadata(): Promise<ZKChainMetadata> {
return structuredClone(zkChainsMetadata);
}

/** @inheritdoc */
async getTokensMetadata(): Promise<Token<TokenType>[]> {
return Array.from(tokens);
}
Expand Down
6 changes: 4 additions & 2 deletions packages/metadata/test/fixtures/metadata.fixtures.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import { Token, TokenType } from "@zkchainhub/shared";

export const tokenJsonUrl = "https://example.com/tokens.json";
export const chainJsonUrl = "https://example.com/chains.json";
export const mockTokenData = [
export const mockTokenData: Token<TokenType>[] = [
{
name: "Ethereum",
symbol: "ETH",
Expand All @@ -20,7 +22,7 @@ export const mockTokenData = [
type: "erc20",
decimals: 18,
},
];
] as const;
export const mockChainData = [
{
chainId: 324,
Expand Down
Loading