Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve membersURI #812

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
a386393
improve membersURI
Rashmi-278 Feb 26, 2024
3e5bf86
add limit, pagination and uniqueness check for members
Rashmi-278 Mar 6, 2024
231b6b3
updated approach, get all unique addresses from DB
Rashmi-278 Apr 22, 2024
b828ed3
remove angle brackets from context link
Rashmi-278 Apr 22, 2024
51c4294
add context
Rashmi-278 Apr 22, 2024
7ebe8c7
Update .gitignore
Rashmi-278 May 1, 2024
ac8669b
Update src/eip4824.ts
Rashmi-278 May 1, 2024
9d41dd3
lint
Rashmi-278 May 1, 2024
47e6eb0
requested changes
Rashmi-278 May 1, 2024
a35202b
Merge remote changes into improve-membersURI
Rashmi-278 May 1, 2024
458e794
remove pnpm
Rashmi-278 May 2, 2024
9849c0b
optimize query for speedy retrival
Rashmi-278 Jun 17, 2024
e342770
Exclude known addresses to optimize for speed
Rashmi-278 Jun 17, 2024
252acc3
remove comments
Rashmi-278 Jun 17, 2024
1df3050
Update src/helpers/spaces.ts
Rashmi-278 Jun 18, 2024
1da8ee9
Update src/helpers/spaces.ts
Rashmi-278 Jun 18, 2024
b2c10b8
cursor conditional
Rashmi-278 Jul 1, 2024
0ce6028
Merge branch 'snapshot-labs:master' into improve-membersURI
Rashmi-278 Jul 1, 2024
240ed25
apply changes
Rashmi-278 Jul 1, 2024
1cd13f3
apply changes
Rashmi-278 Jul 1, 2024
ad86432
cursor conditions
Rashmi-278 Jul 1, 2024
3ca9efa
query improved
Rashmi-278 Aug 21, 2024
560a182
Merge branch 'master' into improve-membersURI
Rashmi-278 Aug 21, 2024
d285d3b
Merge branch 'snapshot-labs:master' into improve-membersURI
Rashmi-278 Aug 21, 2024
f96bf1d
Update eip4824.ts
Rashmi-278 Aug 21, 2024
84d905e
lint
Rashmi-278 Aug 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ coverage

# Remove some common IDE working directories
.idea
.vscode
.vscode
74 changes: 53 additions & 21 deletions src/eip4824.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import express from 'express';
import { getSpace } from './helpers/spaces';
import { getCombinedMembersAndVoters, getSpace } from './helpers/spaces';
import db, { sequencerDB } from './helpers/mysql';

const router = express.Router();
const context = '<http://www.daostar.org/schemas>';
const context = ['https://snapshot.org', 'https://daostar.org/schemas'];

router.get('/:space', async (req, res) => {
let space: any = {};
Expand Down Expand Up @@ -34,29 +34,61 @@ router.get('/:space', async (req, res) => {
});

router.get('/:space/members', async (req, res) => {
let space: any = {};

try {
space = await getSpace(req.params.space);
const spaceId = req.params.space;
const cursor = req.query.cursor || null;
const pageSize = 500; // Default page size

const space = await getSpace(spaceId);
if (!space.verified) {
return res.status(400).json({
error: 'INVALID_SPACE',
message: 'The specified space is not verified.'
});
}

if (!space.verified) return res.status(400).json({ error: 'INVALID' });
} catch (e) {
return res.status(404).json({ error: 'NOT_FOUND' });
}
// Includes space.members, space.admins, space.moderators and voters
const combinedMembersResult = await getCombinedMembersAndVoters(
spaceId,
cursor,
pageSize
);

const members = [...space.admins, ...space.moderators, ...space.members].map(
ChaituVR marked this conversation as resolved.
Show resolved Hide resolved
member => ({
const members = combinedMembersResult.members.map(address => ({
type: 'EthereumAddress',
id: member
})
);

return res.json({
'@context': context,
type: 'DAO',
name: space.name,
members
});
id: address
}));

const responseObject = {
'@context': context,
type: 'DAO',
name: space.name,
members: members,
nextCursor: combinedMembersResult.nextCursor
};

return res.json(responseObject);
} catch (e) {
const error = e as Error;
console.error(error);

if (error.message.includes('database')) {
return res.status(500).json({
error: 'DATABASE_ERROR',
message: 'Failed to retrieve data from the database.'
});
} else if (error.message.includes('parameter')) {
return res.status(400).json({
error: 'INVALID_PARAMETER',
message: 'Invalid or missing parameter.'
});
} else {
return res.status(500).json({
error: 'INTERNAL_SERVER_ERROR',
message: 'An unexpected error occurred.'
});
}
}
});

router.get('/:space/proposals', async (req, res) => {
Expand Down
51 changes: 51 additions & 0 deletions src/helpers/spaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,57 @@ async function getVotes() {
return await db.queryAsync(query);
}

export async function getCombinedMembersAndVoters(
spaceId,
cursor = null,
pageSize
) {
const params = [spaceId, spaceId, spaceId, spaceId];
const query = `
SELECT address
FROM (
(SELECT DISTINCT JSON_UNQUOTE(JSON_EXTRACT(settings, '$.admins[*]')) AS address
FROM spaces
WHERE id = ?)
UNION
(SELECT DISTINCT JSON_UNQUOTE(JSON_EXTRACT(settings, '$.moderators[*]')) AS address
FROM spaces
WHERE id = ?)
UNION
(SELECT DISTINCT JSON_UNQUOTE(JSON_EXTRACT(settings, '$.members[*]')) AS address
FROM spaces
WHERE id = ?)
UNION
(SELECT DISTINCT voter AS address
FROM votes
WHERE space = ?)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should limit to 500 voters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have placed the limit to the Outer Query, using the param, pageSize.
Placing a limit on the inner query would mess up the pagination, wouldn't it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but this is taking a lot of time, especially for spaces with a lot of votes, we don't need to read all the distinct voters here, we just need 500 of them,

you can remove pagination from the outer query

Also for admins, moderators and members -> all this data is already available on space object, we don't need to read them from database again,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We made an improvement on this, now the with cursor filter, if cursor is present, admins, moderators, and members are excluded from the distinct voter list. Also regardless we are not querying admins, moderators and members.

Query now only queries the Votes Table hence eliminating the need for subqueries.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how to use cursor now? you have an example request? i tried few ways but couldn't get it working

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A request should be something like /ens.eth/members?cursor=983434323 ( Cursor Example )

Could you share what message do you get?
Could we get on a call and debug this? As I don't have a way to see whats wrong, its taking longer to close this issue

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have to pass address to cursor now, something like http://localhost:3000/api/eip4824/ens.eth/members?cursor=0xB8cb5Bbf2DdD552145336436b2A0FD490aBf88Ca works

) AS combined_addresses
WHERE (? IS NULL OR address > ?)
ORDER BY address
LIMIT ?
`;
ChaituVR marked this conversation as resolved.
Show resolved Hide resolved

if (cursor) {
params.push(cursor, cursor, pageSize);
} else {
params.push(null, null, pageSize);
}

const results = await db.queryAsync(query, params);

if (!results || results.length === 0) {
return Promise.reject(new Error('NOT_FOUND'));
}

const nextCursor =
results.length === pageSize ? results[results.length - 1].address : null;

return {
members: results.map(row => row.address),
nextCursor: nextCursor
};
}

async function getFollowers() {
const query = `
SELECT space, COUNT(id) as count,
Expand Down