Add the ability to export events for a subset of projects and from a starting offset #4701
Conversation
| ) | ||
| } yield ExportResult(exportFile, exportSuccess, start, end) | ||
|
|
||
| semaphore.permit.use { _ => exportIO } |
There was a problem hiding this comment.
We use a semaphore so that exports don't consume all connections
| _ <- exportToFile(q, exportFile) | ||
| end <- clock.realTimeInstant | ||
| exportSuccess = targetDirectory / s"$start.success" | ||
| _ <- Files[IO].createFile(exportSuccess) |
There was a problem hiding this comment.
This marker file allows to know that the export succeeded.
If we plan to sync those files, this allows to know the state without an additional api call
There was a problem hiding this comment.
Maybe this file could contain the input payload + when it started / ended? So that we know what projects are included at a glance
There was a problem hiding this comment.
Good idea to put the query in it.
The starting time is encoded in the name and the end will be the creation time of that file.
Ok to just include the query for now ?
|
|
||
| import scala.annotation.nowarn | ||
|
|
||
| final case class ExportEventQuery(id: Label, projects: NonEmptyList[ProjectRef], offset: Offset) |
There was a problem hiding this comment.
Is id anything other than the folder containing export files? Might be clearer to name it that
| _ <- exportToFile(q, exportFile) | ||
| end <- clock.realTimeInstant | ||
| exportSuccess = targetDirectory / s"$start.success" | ||
| _ <- Files[IO].createFile(exportSuccess) |
There was a problem hiding this comment.
Maybe this file could contain the input payload + when it started / ended? So that we know what projects are included at a glance
delta/sdk/src/main/scala/ch/epfl/bluebrain/nexus/delta/sdk/permissions/Permissions.scala
Outdated
Show resolved
Hide resolved
delta/app/src/main/scala/ch/epfl/bluebrain/nexus/delta/wiring/ExportModule.scala
Outdated
Show resolved
Hide resolved
| extractCaller { implicit caller => | ||
| (post & pathEndOrSingleSlash & entity(as[ExportEventQuery])) { query => | ||
| authorizeFor(AclAddress.Root, Permissions.exporter.run).apply { | ||
| emit(StatusCodes.Accepted, exporter.events(query).start.void) |
There was a problem hiding this comment.
It might be interesting later to have a way of knowing a failure in the export occurred?
There was a problem hiding this comment.
Yes, to be discussed like reporting in general
Fixes #4662