Add a dev doc for resuming SDG runs #136
Conversation
Document the design for how we plan to resume interrupted SDG runs. Signed-off-by: Ben Browning <[email protected]>
Signed-off-by: Ben Browning <[email protected]>
Signed-off-by: Ben Browning <[email protected]>
bbrowning
force-pushed
the
sdg-resume
branch
from
e6b3d88
to
d9cca57
Compare
Signed-off-by: Ben Browning <[email protected]>
Signed-off-by: Ben Browning <[email protected]>
|
|
||
| ## Non-Goals | ||
|
|
||
| This is not replacing checkpoints. Checkpoints are how we save each |
There was a problem hiding this comment.
It might be helpful to give the context of how long a batch might take for comparison.
There was a problem hiding this comment.
Yes, that's a good idea. And I also need to talk about how checkpoints persist across individual data generation runs, while this is focused on the ability to resume a specific interrupted SDG run.
|
This pull request has been automatically marked as stale because it has not had activity within 30 days. It will be automatically closed if no further activity occurs within 7 days. |
|
This is not abandoned, but how we resume SDG runs is going to change as we integrate the context-aware chunking and docling. I was planning to revisit this once we get the former integrated. Just updating status here for visibility and for the stale bot. |
Document the design for how we plan to resume interrupted SDG runs.