Alert when sites are stuck building (build timeout)

[ChristopherChudzicki]

We recently had a situation where:

1. Course author published a site to draft.
2. Concourse was stuck indefinitely. Concourse was stuck discovering inputs. Input discover does not have a timeout. In this case, it was "Waiting for suitable version" of webpack-json". Manually running fly check-resource on the offending pipeline/resource managed to get the thing unstuck.
3. Course author alerted us.

Effectively, the site silently failed to publish and the Studio UI was stuck in "Pending". (Concourse did not consider itself failing, but it was never going to succeed.)

We should raise an alert in this scenario so that the failure is not silent.

[ChristopherChudzicki]

One idea: Studio triggers the build, and site builds are fast, so Studio could issue an alert if the site fails to build within X minutes.