Optimize primary shard balance

[bugmakerrrrrr]

Is your feature request related to a problem? Please describe

Today, we utilize the allocation constraint mechanism to achieve primary shard balance. One constraint is known as AllocationConstraints, which is applied during shard allocation. Another constraint is RebalanceConstraints, which comes into play during shard rebalance. It is worth noting that both of these constraints currently utilize the same predicates in their implementation, which may cause uneven shards distribution.

OpenSearch/server/src/main/java/org/opensearch/cluster/routing/allocation/ConstraintTypes.java

Lines 69 to 88 in 56825f6

	public static Predicate<Constraint.ConstraintParams> isPerIndexPrimaryShardsPerNodeBreached() {
	return (params) -> {
	int perIndexPrimaryShardCount = params.getNode().numPrimaryShards(params.getIndex());
	int perIndexAllowedPrimaryShardCount = (int) Math.ceil(params.getBalancer().avgPrimaryShardsPerNode(params.getIndex()));
	return perIndexPrimaryShardCount >= perIndexAllowedPrimaryShardCount;
	};
	}
	/**
	* Defines a predicate which returns true when a node contains more than average number of primary shards with added buffer. This
	* constraint is used in weight calculation during allocation/rebalance both. When breached a high weight {@link ConstraintTypes#CONSTRAINT_WEIGHT}
	* is assigned to node resulting in lesser chances of node being selected as allocation/rebalance target
	*/
	public static Predicate<Constraint.ConstraintParams> isPrimaryShardsPerNodeBreached(float buffer) {
	return (params) -> {
	int primaryShardCount = params.getNode().numPrimaryShards();
	int allowedPrimaryShardCount = (int) Math.ceil(params.getBalancer().avgPrimaryShardsPerNode() * (1 + buffer));
	return primaryShardCount >= allowedPrimaryShardCount;
	};
	}

For example, suppose we have a cluster withe 3 data nodes, and create a index with 10 shards, 1 replica. A possible shards distribution shown below.

Node	Primary	Replica
Node1	3	5
Node2	4	0
Node3	3	5

As you can see, Node1 and Node2 have double shards than Node2. This is because Node2 is assigned a high weight during rebalance. When evaluating the isPerIndexPrimaryShardsPerNodeBreached predicate, the perIndexPrimaryShardCount of Node2 is 4, and the perIndexAllowedPrimaryShardCount is 4 (Math.ceil(10/3)), so the constraint is breached and a high weight is assigned. And the isPrimaryShardsPerNodeBreached predicate is not breached thanks to the exist of the parameter buffer. However, the parameter buffer works as a proportion of the avgPrimaryShardsPerNode, which may also cause uneven distribution when theavgPrimaryShardsPerNode is large.

Describe the solution you'd like

1. When avgPrimaryShardsPerNode equals perIndexPrimaryShardCount, we should treat the isPerIndexPrimaryShardsPerNodeBreached as not breached during rebalance. This is because it is bound to happen when the number of shards is not divisible by the number of nodes, and this distribution is in fact balanced. Which is addressed in Using gt to make constraint test during rebalance #17324. When the change is applied, the distribution in the above example becomes as follows.

Node	Primary	Replica
Node1	3	4
Node2	4	2
Node3	3	4

2. The parameter buffer should be an integer that stays constant as the number of shards in the cluster increases.

Related component

ShardManagement:Placement

Describe alternatives you've considered

No response

Additional context

No response