Measure Hydration/Transformation/Indexing Times #153
Conversation
Container Scanning Status: ✅ Success |
|
This is looking good. Do you think we shouldn't use a span, then? |
I looked at that, but I couldn't figure out how to wrap something around a process that happens over a long period of time. Like I guess I could spawn a task that spins and waits...but I think just sending an event for the duration is just as good. |
tpendragon
force-pushed
the
index_metrics_tracker_poc
branch
from
ecdbb32
to
b24d354
Compare
I'm not sure this is useful still - the full hydration test definitely takes more than a second to run, but since we're just measuring what it takes to produce those last messages it's less than a second. Maybe the differential isn't a big deal in production.
tpendragon
force-pushed
the
index_metrics_tracker_poc
branch
from
b538ad0
to
7950503
Compare
tpendragon
changed the title
There was a problem hiding this comment.
This is very good and cool, the IndexMetricsTracker genserver is a bit hard for me to follow; maybe we could look at it together and then I could make some suggestions around organization and documentation that might help me or someone looking at it in the future?
| @@ -0,0 +1,129 @@ | |||
| defmodule DpulCollections.IndexingPipeline.DashboardPage do | |||
There was a problem hiding this comment.
I feel like this should be in DpulCollectionsWeb somewhere, is there a particular reason it's here?
There was a problem hiding this comment.
I put it here because it didn't really seem like it was part of the app, but happy to move it.
| def handle_call({:poll_started, source}, _, state) do | ||
| if get_in(state, [source.processor_marker_key(), :start_time]) != nil && | ||
| get_in(state, [source.processor_marker_key(), :end_time]) == nil do | ||
| state = put_in(state, [source.processor_marker_key(), :request_end], true) |
There was a problem hiding this comment.
not sure what :request_end is, not entirely sure what state should / could look like
| :indexer | ||
| end | ||
|
|
||
| defp handle_ack_received([:database_producer, :ack, :done], _measurements, metadata, _config) do |
There was a problem hiding this comment.
this one is split from the others?
| def mount(_params, _session, socket) do | ||
| socket = | ||
| assign(socket, | ||
| hydration_times: IndexMetricsTracker.index_times(HydrationProducerSource), |
There was a problem hiding this comment.
maybe we could rename the index_times function
There was a problem hiding this comment.
Oh interesting, I tried index_durations - it's the index part that's confusing?
tpendragon
force-pushed
the
index_metrics_tracker_poc
branch
from
2bd2e78
to
efbe0f5
Compare
| |> put_in([:acked_count], old_acked_count + new_acked_count) | ||
| end | ||
|
|
||
| defp handle_ack_received([:database_producer, :ack, :done], _measurements, metadata, _config) do |
There was a problem hiding this comment.
Because I should probably call that function something different
Tracks index metrics for Hydrator, Transformer, and Indexer.
This stores the index times and acknowledged records in the database when a fresh process is started, and ends the measurement the first time acknowledgements empty out after polling is hit.
The dashboard has a page to display these metrics along with our lower and upper bound of times.
Closes #115
