Equal ratio allocation objective

[SouthEndMusic]

Fixes #1348.

The objective function I found that leads to equal fraction allocation has terms of the form

$$ d\left(1 - \frac{F}{d}\right)^2 = \left(\sqrt{d} - \frac{F}{\sqrt{d}}\right)^2 = \frac{F^2}{d} - 2F + d. $$

I can't explain yet why this works.

[jarsarasty]

Explanation of this formula: Since d is the demand and F is the flow to the user to be optimized, your objective (as stated in the formula above) is to make F as close as possible to the value of d, allowing for deviations above and below d, with a quadratic penalty for those deviations. The parameter d is also used as a weighting factor when multiplying the quadratic term.

[SouthEndMusic]

@jarsarasty ⁠thanks for your comment. (1 - F/d)^2 is our original quadratic_relative objective function, and I found that although I expected it at first, this does not lead to the same fraction F/d for each user after optimization. What I found is that that does happen when I add the additional factor d to obtain d*(1 - F/d)^2 but I do not understand why that is. For me intuitively it has to do with the maximum amount the objective function can be reduced by distributing the same amount of flow in different ways over the users, and apparently these extra weights d shift that balance precisely (at least in my test model) so that the minimum of the objective function coincides with this universal fraction distribution.

[jarsarasty]

Indeed, the result is counterintuitive at first sight. However, if we consider that there is a capacity in the total flow (maximum flow), the original formulation of the objective function tends to favour the demands that are easier to satisfy (i.e. the demands with a lower value) to the detriment of the other demands. This is because in the objective function all quadratic_relative terms $(1 - F/d)^2$ have the same weight, and so deviations in a large demand are over-compensated in the objective value by almost completely covering the smaller demands with such deviations from the large demands.

By including the demands as weighting factors in the objective function, each unit of demand (e.g. m3/s) is given the same weight, and because the objective is quadratic, the optimal solution tends to allocate the same fraction to all demands (users). However, the fractions cannot be expected to be equal in a more general case, as other constraints may come into play.

For example, in the case with demands 10, 10, and 100, and a maximum flow of 100, optimizing the original objective function yields the following results:

	$d$	$F$	$(1-F/d)$	$(1-F/d)^2$
	10	9.80392	0.01961	0.00038
	10	9.80392	0.01961	0.00038
	100	80.39216	0.19608	0.03845
Sum	120	100.00000	0.23529	0.03922

whereas for the second case, we have:

	$d$	$F$	$(1-F/d)$	$(1-F/d)^2$	$d (1-F/d)^2$
	10.0	8.3333	0.167	0.03	0.278
	10.0	8.3333	0.167	0.03	0.278
	100.0	83.3333	0.167	0.03	2.778
Sum	120.0	100.0	0.500	0.083	3.333

Clearly, in the first case, the large demand gets a smaller allocation than in the second case (80.39 vs. 83.33), but the sum of the quadratic fractions in the first case is much smaller than in the second case ((0.039 vs. 0.083), where the solution is optimized for the weighted sum of the quadratic terms.

[SouthEndMusic]

Hi @jarsarasty, thanks a lot for your in depth reply. There a some things that I do not yet fully understand, maybe you can elaborate:

This is because in the objective function all quadratic terms have the same weight

Before the added weight, the quadratic terms look like $\frac{F^2}{d^2}$, so the weight of these terms differ because the demans differ right?

By including the demands as weighting factors in the objective function, each unit of demand (e.g. m3/s) is given the same weight

Is this true? Would this indeed incentivize the equal fraction allocation?

[visr]

Thanks @jarsarasty!
Because I like pictures, here is a case with 1.2 supply and, 1.5 demand spread over two nodes A with demand 1.0 and B with demand 0.5. Equal ratio is 1.2 / 1.5 = 0.8, which gets the lowest objective:

[jarsarasty]

Hi @SouthEndMusic,

These are very good questions. I realized that I need to make some clarifications

This is because in the objective function all quadratic terms have the same weight

Before the added weight, the quadratic terms look like F2d2, so the weight of these terms differ because the demans differ right?

I was referring to the quadratic_relative term of your original objective function. In such an objective function, each of the quadratic_relative terms had the same weight, and so as shown in my example above, the model tends to satisfy low demand users at the expense of high demand users. For clarity, I have included the name quadratic_relative in my comment above.

By including the demands as weighting factors in the objective function, each unit of demand (e.g. m3/s) is given the same weight

Is this true?

By multiplying the quadratic_relative term by $d$, you give weight to the quadratic_relative terms in linear proportion to their demand $d$, and so each unit of demand (m3/s) has the same weight.

Would this indeed incentivize the equal fraction allocation?

The factor $d$ is not sufficient to incentivize equal allocation. The incentive for equal allocation comes from the quadratic objective, as mentioned in my comment:

and because the objective is quadratic, the optimal solution tends to allocate the same fraction to all demands (users).

[visr]

Looks great! The discontinuity across demands of 1e-5 m3/s feels a little uneasy:

Add an objective term demand * (1 - flow/demand)^2. If the absolute
value of the demand is very small, this would lead to huge coefficients,
so in that case a term of the form (flow - demand)^2 is used.

Though this is less than 1 m3/day, so it shouldn't be an issue except in exceptional cases.

[SouthEndMusic]

TLDR: I found my proof 👌

Objective function:

$$ e(F_1,\ldots,F_n) = \sum_{i=1}^n d_i\left(1-\frac{F_i}{d_i}\right)^2. $$

Used flow:

$$ \sum_{i=1}^n F_i = Q. $$

The used flow defines a $(n-1)$-dimensional hyperplane which can be parameterized as follows:

$$ \mathbf{F}(c_1, \ldots, c_{n-1}) = \frac{Q}{n}(1,\ldots,1)^T + \sum_{i=1}^{n-1} c_i (\mathbf{e}_{i+1} - \mathbf{e}_i). $$

Here the $\mathbf{e}_i$ define the standard orthonormal basis of $\mathbb{R}^n$. From this we obtain the relationship

$$ F_i = \frac{Q}{n} - c_i + c_{i-1}. $$

Writing the error in terms of the coefficients $c_i$ yields

$$ \hat{e}(c_1, \ldots, c_{n-1}) = \sum_{i=1}^n d_i\left(1-\frac{1}{d_i}\left[\frac{Q}{n} - c_i + c_{i-1} \right]\right)^2 $$

where $c_0 = c_n = d_0 = 0$. Then for the minimum we look at

$$ 0 = \frac{\partial \hat{e}}{\partial c_j} = 2\left(1-\frac{1}{d_j}\left[\frac{Q}{n} - c_j + c_{j-1} \right]\right) - 2\left(1-\frac{1}{d_{j+1}}\left[\frac{Q}{n} - c_{j+1} + c_j \right]\right) = 2\left(\frac{F_{j+1}}{d_{j+1}}-\frac{F_j}{d_j}\right), $$

from which we conclude that all allocated fractions $\frac{F_i}{d_i}$ are the same at the minimum of the objective function. This fraction $f$ is given by

$$ \sum_{i=1}^n fd_i = Q \quad \Rightarrow \quad f = \frac{Q}{\Sigma_{i=1}^n d_i}. $$

[jarsarasty]

TLDR

Great, @SouthEndMusic!

It's worh mentioning the assumptions: i) arc capacities are not binding ii) $d_i >0$ for each demand $i$.

By the way, a proof can also be derived using the KKT optimality conditions.