Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement API to manage indexing queues #437

Open
msm-cert opened this issue Nov 18, 2024 · 0 comments · May be fixed by #444
Open

Implement API to manage indexing queues #437

msm-cert opened this issue Nov 18, 2024 · 0 comments · May be fixed by #444
Assignees
Labels
next-sprint Scheduled for work (CERT.PL internal) type:feature Add a new feature zone:backend Backend oriented tasks
Milestone

Comments

@msm-cert
Copy link
Member

We plan to implement an automated file indexing in the future.

Right now (after #434), there is a database object representing indexed files, but there's no way to manipulate them - add files, remove them, etc.

In this issue I propose to add three endpoints to mquery:

POST /api/queue/{ursadb_id}

Takes a JSON body with a list of file paths on the ursadb server to be indexed.

For example, POST to /api/queue/default with ["/mnt/samples/12345"] will tell ursadb with ID default that a file /mnt/samples/12345 should be indexed.

This endpoint should support all QueuedFile fields, i.e. tags and index types, etc.

JSON body can look like this:

[
    { "path": "...", "types": [...], "tags": [...] },
    { "path": "...", "types": [...], "tags": [...] },
]

or:

{
    "paths": ["..."],
    "types": [...],
    "tags": [...] 
]

whichever is easier to implement.

GET /api/queue/{ursadb_id}

Returns a JSON with enough information to show the queue status in the admin panel. Probably for now this should be just something like:

[
    {
        "ursadb_id": "default",
        "size": 1330
        # I can't think of anything more useful - maybe date of oldest and newest file?
    },
    ...
]

POST /api/queue/{ursadb_id}/clear

Or just DELETE /api/queue/{name} if you're feeling RESTy - to be decided by implementer.
Takes no parameters, and removes all files from the specified queue (i.e. removes all QueuedFiles with the given ursadb_id from the database).

Finally, all endpoints must be hidden behind appropriate permissions. We need two new user roles: can_view_queues and can_manage_queues. I think every user should be able to see queue status, and admin should be able to manage them by default. Add the permissions to the groups by default (see expand_role function).

This should most likely be implemented more than one PR, to avoid one huge PR

@msm-cert msm-cert added type:feature Add a new feature next-sprint Scheduled for work (CERT.PL internal) zone:backend Backend oriented tasks labels Nov 18, 2024
@msm-cert msm-cert added this to the v1.7.0 milestone Dec 16, 2024
@michalkrzem michalkrzem linked a pull request Dec 31, 2024 that will close this issue
4 tasks
@michalkrzem michalkrzem linked a pull request Dec 31, 2024 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
next-sprint Scheduled for work (CERT.PL internal) type:feature Add a new feature zone:backend Backend oriented tasks
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants