Revert "operator: always delete terminated pending nodes (#2545)"

[3u13r]

This reverts commit 5267ad0.

Context

Proposed change(s)

• We need to revert this. With this patch we've observed 30+ nodes being scheduled during the upgrade of a 1:1 cluster on GCP. This presumably happens because if a node is not found then it's state within the operator is "Terminated". This likely happens in the first seconds of the new node being created. With is patch the "Terminated" node already has reached its goal. It's true goal was joining the cluster though.

The idea before (and after) this patch is that the joiningNode has a deadline set until it must join the cluster. Otherwise it's goal is set to "Leaving" which is reached if the node never joined the cluster because it was shut down. This should've already handled the AWS case.

Additional info

Checklist

• Add labels (e.g., for changelog category)
• Link to Milestone

[netlify]

✅ Deploy Preview for constellation-docs canceled.

Name	Link
🔨 Latest commit	c2e2ae3
🔍 Latest deploy log	https://app.netlify.com/sites/constellation-docs/deploys/65526e036c278b00084f3205

[malt3]

I agree that reverting is better. However, we should not forget about the AWS bug, where we will now schedule too many nodes. The AWS case is not handled because of weird behavior of the AWS state machine (the per instance state can be unexpected).

[msanft]

approving here while workaround for AWS should be implemented separately