Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow bulk writing and reading of time series #1077

Merged
merged 2 commits into from
Mar 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/src/modeler_guide/time_series.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,27 @@ components that share the time series data.
This function stores a single copy of the data. Each component will store a
reference to that data.

### Adding time series in bulk

By default, the call to `add_time_series!` will open the HDF5 file, write the data to the file,
and close the file. Opening and closing the file has overhead. If you will add thousands of time
series arrays, consider using `open_time_series_store!`as shown in the example below. All arrays
will be written with one file handle.

This example assumes that there are arrays of components and time series stored in the variables
`components` and `single_time_series`, respectively.

```julia
open_time_series_store!(sys, "r+") do
for (component, ts) in zip(components, single_time_series)
add_time_series!(sys, component, ts)
end
end
```

You can also use this function to make reads faster. Change the mode from `"r+"` to `"r"` to open
the file read-only.

## Removing time series data

Time series instances can be removed from a system like this:
Expand Down
1 change: 1 addition & 0 deletions src/PowerSystems.jl
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,7 @@ export TimeSeriesCounts
export get_dynamic_components

export parse_file
export open_time_series_store!
export add_time_series!
export remove_time_series!
export check_time_series_consistency
Expand Down
28 changes: 28 additions & 0 deletions src/base.jl
Original file line number Diff line number Diff line change
Expand Up @@ -704,6 +704,34 @@ function add_service!(
return
end

"""
Open the time series store for bulk additions or reads. This is recommended before calling
add_time_series! many times because of the overhead associated with opening and closing an
HDF5 file.

This is not necessary for an in-memory time series store.

# Examples
```julia
# Assume there is a system with an array of components and SingleTimeSeries
open_time_series_store!(sys, "r+") do
for (component, ts) in zip(components, single_time_series)
add_time_series!(sys, component, ts)
end
end
```
julia>
"""
function open_time_series_store!(
func::Function,
sys::System,
mode = "r",
args...;
kwargs...,
)
IS.open_time_series_store!(func, sys.data, mode, args...; kwargs...)
end

"""
Add time series data from a metadata file or metadata descriptors.

Expand Down
33 changes: 33 additions & 0 deletions test/test_system.jl
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,39 @@ end
end
end

@testset "Test bulk add of time series" begin
sys = System(100.0)
bus = ACBus(nothing)
bus.bustype = ACBusTypes.REF
add_component!(sys, bus)
components = []
len = 2
component = ThermalStandard(nothing)
component.name = "gen"
component.bus = bus
add_component!(sys, component)
initial_time = Dates.DateTime("2020-09-01")
resolution = Dates.Hour(1)
len = 24
timestamps = range(initial_time; length = len, step = resolution)
arrays = [TimeSeries.TimeArray(timestamps, rand(len)) for _ in 1:5]
ts_name = "test"

open_time_series_store!(sys, "r+") do
for (i, ta) in enumerate(arrays)
ts = SingleTimeSeries(; data = ta, name = "$(ts_name)_$(i)")
add_time_series!(sys, component, ts)
end
end

open_time_series_store!(sys, "r") do
for (i, expected_array) in enumerate(arrays)
ts = IS.get_time_series(IS.SingleTimeSeries, component, "$(ts_name)_$(i)")
@test ts.data == expected_array
end
end
end

@testset "Test set_name! of system component" begin
sys = System(100.0)
bus = ACBus(nothing)
Expand Down
Loading