StatisticsV2: statistics framework initial redesign for Datafusion

[Fly-Style]

Rationale for this change

https://synnada.notion.site/Redesigning-and-Enhancing-the-Statistics-Framework-in-Datafusion-16bf46d2dab180448272dbbd1d1f7cea

What changes are included in this PR?

This patch presents a Statistics v.2 framework with the following main points:

• introduction to enum-based struct to support multiple distribution types, which initially include:
  • Uniform distribution (range)
  • Gaussian distribution (also known as normal)
  • Exponential distribution
  • Bernoulli distribution (holds probability, is used as resulting distribution of comparison operators),
  • Unknown distribution, abstracts any non-represented distribution.
• tree-based stats calculation is divided into bottom-up evaluation and top-down propagation;

Tables of statistic execution and propagation rules for PhysicalExpr-s:

Definitions

UF = Uniform UN = Unknown EXP = Exponential GSS = Gaussian BRN = Bernoulli

Binary arithmetical operators, evaluation.

Input 1	Input 2	(+)	(-)	(*)	(/)
UF [a,b]	UF [c,d]	UN	UN	UN	UN
UF [a,b]	GSS N(μ,σ²)	UN	UN	UN	UN
UF [a,b]	EXP Exp(λ)	UN	UN	UN	UN
UF [a,b]	BRN	UN	UN	UN	UN
UF [a,b]	UN	UN	UN	UN	UN
GSS N(μ₁,σ₁²)	GSS N(μ₂,σ₂²)	GSS	GSS	GSS	UN
GSS N(μ,σ²)	EXP Exp(λ)	UN	UN	UN	UN
GSS N(μ,σ²)	BRN	UN	UN	UN	UN
GSS N(μ,σ²)	UN	UN	UN	UN	UN
EXP Exp(λ₁)	EXP Exp(λ₂)	UN (gamma dist)	UN	UN	UN
EXP Exp(λ)	BRN	UN	UN	UN	UN
EXP Exp(λ)	UN	UN	UN	UN	UN
BRN Exp(λ)	BRN	UN	UN	UN	UN
BRN Exp(λ)	UN	UN	UN	UN	UN
UN	UN	UN	UN	UN	UN

Comparison operators, evaluation.

| Input 1 | Input 2 | Comparison (=, ≠) | Inequality (>, >=) | Inequality (<, <=) | |-------------------|-------------------|-------------------|--------------------|--------------------| | **UF** [a,b] | **UF** [c,d] | BRN | BRN | BRN | | **GSS** N(μ₁,σ₁²) | **GSS** N(μ₂,σ₂²) | UN | UN(Ф-func) | UN(Ф-func) | | **EXP** Exp(λ₁) | **EXP** Exp(λ₂) | UN | UN | UN | | **BRN** | **BRN** | BRN | BRN | BRN | | **UF** [a,b] | **GSS** N(μ,σ²) | UN | UN | UN | | **UF** [a,b] | **EXP** Exp(λ) | UN | UN | UN | | **UF** [a,b] | **BRN** | UN? | UN? | UN? | | **UF** [a,b] | **UN** | BRN? | BRN? | BRN? | | **GSS** N(μ,σ²) | **EXP** Exp(λ) | UN | UN | UN | | **GSS** N(μ,σ²) | **BRN** | BRN?/UN | BRN?/UN | BRN? /UN | | **GSS** N(μ,σ²) | **UN** | UN | UN | UN | | **EXP** Exp(λ) | **UN** | UN, estimate | UN, estimate | UN, estimate | | **EXP** Exp(λ) | **BRN** | UN, estimate | UN, estimate | UN, estimate | | **UN** | **BRN** | BRN/UN | BRN/UN | BRN/UN | | **UN** | **UN** | BRN/UN | BRN/UN | BRN/UN | Unit tests are included as well as integration tests.