Index hydro units by their location

[davide-f]

Closes #1331

Changes proposed in this Pull Request

This PR aims at resolving indexing conflicts with hydro units and is a first step into revising alternative clustering.
This PR:

• revise the indexing of hydro profiles to be hydro-powerplant specific, so no longer relying on the regions_onshore file
• drops the use of alternative_clustering if/else clauses for hydro

This, however, means that we no longer have the time series for locations that do not have hydro units.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

This PR is architectural and requires review.

@yerbol-akhmetov , it would be nice if you may have time for a revision too.
I'll continue working on top of this PR in another branch to revise the use of alternative clustering.

Checks have been performed on the "tutorial" configuration with/without alternative clustering.
I've compared the inflows for the elec.nc and the pre-solved network. Inflows agree [except for the multiplier] as shown below

legend:

• nb: base configuration [obtained with current main]
• nm: tutorial configuration using this PR
• na: tutorial configuration with alternative clustering using this PR

[yerbol-akhmetov]

This PR is architectural and requires review.

@yerbol-akhmetov , it would be nice if you may have time for a revision too. I'll continue working on top of this PR in another branch to revise the use of alternative clustering.

Checks have been performed on the "tutorial" configuration with/without alternative clustering. I've compared the inflows for the elec.nc and the pre-solved network. Inflows agree [except for the multiplier] as shown below

legend:

• nb: base configuration [obtained with current main]
• nm: tutorial configuration using this PR
• na: tutorial configuration with alternative clustering using this PR

Hi, @davide-f. Great thanks for trying to resolve the issue. I have run the PR locally for Nigeria (not tutorial) for Voronoi and alternative clustering. Here is the comparison of inflows:

This is what I got for Nigeria simulation only. Vor PR and Vor AC PR mean the results for Voronoi and alternative clustering using this PR. Vor up means using Voronoi with alternative clustering using upstream commit. The last three entries are for elec.nc for respective networks. As the double multiplier was removed, new inflow data matches the old/1.1.

With the new PR, the inflow data is assigned to plant represented by buses. My next plan is to run for US and see the dispatch.

[yerbol-akhmetov]

@davide-f, what can I check for Nigeria additionally?

[davide-f]

This PR is architectural and requires review.
@yerbol-akhmetov , it would be nice if you may have time for a revision too. I'll continue working on top of this PR in another branch to revise the use of alternative clustering.
Checks have been performed on the "tutorial" configuration with/without alternative clustering. I've compared the inflows for the elec.nc and the pre-solved network. Inflows agree [except for the multiplier] as shown below
legend:

• nb: base configuration [obtained with current main]
• nm: tutorial configuration using this PR
• na: tutorial configuration with alternative clustering using this PR

Hi, @davide-f. Great thanks for trying to resolve the issue. I have run the PR locally for Nigeria (not tutorial) for Voronoi and alternative clustering. Here is the comparison of inflows:

This is what I got for Nigeria simulation only. Vor PR and Vor AC PR mean the results for Voronoi and alternative clustering using this PR. Vor up means using Voronoi with alternative clustering using upstream commit. The last three entries are for elec.nc for respective networks. As the double multiplier was removed, new inflow data matches the old/1.1.

With the new PR, the inflow data is assigned to plant represented by buses. My next plan is to run for US and see the dispatch.

Great! Seems you are reproducing the results and the numbers are consistent.

I would see no problems in merging this PR given the above results. A test on US would be interesting but not necessary.
Do you expect your US problems would be solved with this PR?

[yerbol-akhmetov]

This PR is architectural and requires review.
@yerbol-akhmetov , it would be nice if you may have time for a revision too. I'll continue working on top of this PR in another branch to revise the use of alternative clustering.
Checks have been performed on the "tutorial" configuration with/without alternative clustering. I've compared the inflows for the elec.nc and the pre-solved network. Inflows agree [except for the multiplier] as shown below
legend:

• nb: base configuration [obtained with current main]
• nm: tutorial configuration using this PR
• na: tutorial configuration with alternative clustering using this PR

Hi, @davide-f. Great thanks for trying to resolve the issue. I have run the PR locally for Nigeria (not tutorial) for Voronoi and alternative clustering. Here is the comparison of inflows:
This is what I got for Nigeria simulation only. Vor PR and Vor AC PR mean the results for Voronoi and alternative clustering using this PR. Vor up means using Voronoi with alternative clustering using upstream commit. The last three entries are for elec.nc for respective networks. As the double multiplier was removed, new inflow data matches the old/1.1.
With the new PR, the inflow data is assigned to plant represented by buses. My next plan is to run for US and see the dispatch.

Great! Seems you are reproducing the results and the numbers are consistent.

I would see no problems in merging this PR given the above results. A test on US would be interesting but not necessary. Do you expect your US problems would be solved with this PR?

I am not sure if it will resolve the issues in the US model. I was not able to test the PR for the US yet. I will try that close to weekends.

[yerbol-akhmetov]

Hi, @davide-f. I have tried the PR for the US with alternative clustering. Unfortunately, the total inflow and generation were the same as in the upstream, but I think it was expected, as you said that this was the first step towards reorganiztion of alternative clustering method. AC PR oldcost - is the results of the PR with alternative clustering. stable - means commit before linopy integration.

[davide-f]

That's a pity! though it seems this PR creates no further harm.
We may consider merging this; it would be good to discuss on Thursday; I won't be able to make it but if you align with the others we can then proceed.

FYI @ekatef

[ekatef]

Thanks a lot for great work @davide-f and @yerbol-akhmetov !

@davide-f Do I understand properly the PR is ready for review? If so, happy (and very curious) to have a look. At the first glance, it appears that the new implementation it a great step to improve the clarity and quality of the code