Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move some of the active checks earlier in the workflow #962

Merged
merged 1 commit into from
Dec 12, 2024
Merged

Move some of the active checks earlier in the workflow #962

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 6 additions & 36 deletions src/constraints/create.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,8 @@ function _create_constraints_tables(connection)
FROM t_highest_all_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type = 'consumer';
asset.type = 'consumer';
",
)

Expand All @@ -78,13 +73,8 @@ function _create_constraints_tables(connection)
FROM t_highest_all_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type = 'hub';
asset.type = 'hub';
",
)

Expand All @@ -98,13 +88,8 @@ function _create_constraints_tables(connection)
FROM t_highest_in_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type in ('storage')",
asset.type in ('storage')",
)

DuckDB.query(
Expand All @@ -117,13 +102,8 @@ function _create_constraints_tables(connection)
FROM t_highest_out_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type in ('producer', 'storage', 'conversion')",
asset.type in ('producer', 'storage', 'conversion')",
)

DuckDB.query(
Expand All @@ -136,13 +116,8 @@ function _create_constraints_tables(connection)
FROM t_highest_assets_and_out_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type in ('producer', 'conversion')
asset.type in ('producer', 'conversion')
AND asset.unit_commitment = true;
",
)
Expand All @@ -157,13 +132,8 @@ function _create_constraints_tables(connection)
FROM t_highest_out_flows AS t_high
LEFT JOIN asset
ON t_high.asset = asset.asset
LEFT JOIN asset_both
ON t_high.asset = asset_both.asset
AND t_high.year = asset_both.milestone_year
AND t_high.year = asset_both.commission_year
WHERE
asset_both.active = true
AND asset.type in ('producer', 'storage', 'conversion')",
asset.type in ('producer', 'storage', 'conversion')",
)

DuckDB.query(
Expand Down
29 changes: 22 additions & 7 deletions src/tmp.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,8 +375,17 @@ function tmp_create_lowest_resolution_table(connection)
current_group = ("", 0, 0)
@timeit to "append $table_name rows" for row in DuckDB.query(
connection,
"SELECT * FROM $union_table
ORDER BY asset, year, rep_period, time_block_start
"SELECT t_union.* FROM $union_table AS t_union
LEFT JOIN asset_both
ON asset_both.asset = t_union.asset
AND asset_both.milestone_year = t_union.year
AND asset_both.commission_year = t_union.year
WHERE asset_both.active = true
ORDER BY
t_union.asset,
t_union.year,
t_union.rep_period,
t_union.time_block_start
",
)
if (row.asset, row.year, row.rep_period) != current_group
Expand Down Expand Up @@ -418,28 +427,34 @@ function tmp_create_highest_resolution_table(connection)
# - keep all unique time_block_start
# - create corresponing time_block_end

# filtering by active
for union_table in
("t_union_" * x for x in ("in_flows", "out_flows", "assets_and_out_flows", "all_flows"))
table_name = replace(union_table, "t_union" => "t_highest")
DuckDB.execute(
connection,
"CREATE OR REPLACE TABLE $table_name AS
SELECT
asset,
t_union.asset,
t_union.year,
t_union.rep_period,
time_block_start,
lead(time_block_start - 1, 1, rep_periods_data.num_timesteps)
OVER (PARTITION BY asset, t_union.year, t_union.rep_period ORDER BY time_block_start)
t_union.time_block_start,
lead(t_union.time_block_start - 1, 1, rep_periods_data.num_timesteps)
OVER (PARTITION BY t_union.asset, t_union.year, t_union.rep_period ORDER BY t_union.time_block_start)
AS time_block_end
FROM (
SELECT DISTINCT asset, year, rep_period, time_block_start
FROM $union_table
) AS t_union
LEFT JOIN asset_both
ON asset_both.asset = t_union.asset
AND asset_both.milestone_year = t_union.year
AND asset_both.commission_year = t_union.year
LEFT JOIN rep_periods_data
ON t_union.year = rep_periods_data.year
AND t_union.rep_period = rep_periods_data.rep_period
ORDER BY asset, t_union.year, t_union.rep_period, time_block_start
WHERE asset_both.active = true
Comment on lines +449 to +456
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my understanding, the JOIN here is to have available in the table the active parameter. So, when we take out active from the Tulipa files in #960 we don't have to do this JOIN. Is that correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stored table itself won't have the active parameter because I haven't selected it, but the intermediary table resulting from the join will, which is then filtered in this WHERE. So the JOIN is only so that we have active available for filtering.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, deleting the active also will also delete this extra join and make the code more simple 😄

Thanks!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During #960, this will also be removed as there will be no mention of active anywhere. I was thinking about where the removal of active will happen, and the earliest point would be not having active at all in the input of TulipaEnergyModel. That means that during the workflow processing, at some point we check for active and remove. In the current workflow, it would be at the read_csv_folder function in TulipaIO. It's not clear yet the implications, and it's not urgent, so I'm not actively thinking about it.
For this PR, I noticed the constraint creations had a bunch of active checks, and the constraints could look a bit simpler if they didn't, so I shifted left.

ORDER BY t_union.asset, t_union.year, t_union.rep_period, time_block_start
",
)
end
Expand Down
Loading