services/horizon: add metrics for ingestions failures and alert on them

[mollykarcher]

What problem does your feature solve?

There's been 2 incidents in the past year where ingestion in horizon halted (for different reasons):

• https://docs.google.com/document/d/1rzuvcXFhXHVf_L_1UBxmCbADD7reEKxNhPnZOj2_ZDg/edit#heading=h.b5xdh739mlq
• https://docs.google.com/document/d/1gnjeNOiQ8W6DNYicT5nkZ_X6f4Zr0Uw6Tdom9Ab6RHw/edit

During the resolution process, we had a lot of general health metric alerts firing but it wasn't immediately clear it was an ingestion halt until we looked at error logs.

What would you like to see?

An explicit metric that tracks ingestion failures. When a single instance of this occurs, we should alert (critical) on it. This will make it clearer to the responding engineer immediately what the root cause of the issue is.