Fix maintenance of internal state related to snapshot submissions

[anomit]

Describe the bug

While debugging the Telegram bot alerts on malfunctioning nodes, it was discovered that the internal state is not being correctly maintained in the snapshotter node regarding submissions which were in turn shared in the Telegram alert messages. The values against each of these 3 fields are being reported as 0.

snapshotter-lite-v2/snapshotter/utils/models/data_models.py

Lines 137 to 142 in e75fc20

	class SnapshotterStatus(BaseModel):
	totalSuccessfulSubmissions: int = 0
	totalIncorrectSubmissions: int = 0
	totalMissedSubmissions: int = 0
	consecutiveMissedSubmissions: int = 0
	projects: List[ProjectStatus]

To Reproduce

Affected versions:

Steps to reproduce the behavior:

1. Let snapshots fail by blocking RPC calls or supplying bad RPC URLs etc
2. Configure the Telgram Chat ID in env
3. Check Telegram messages sent by bot

Expected behavior

The fields in the data model linked above should be set correctly regarding total, missed, incorrect and other categories of submission data.

The telgram alert data model for the same:

snapshotter-lite-v2/snapshotter/utils/models/message_models.py

Lines 93 to 95 in e75fc20

	class TelegramSnapshotterReportMessage(TelegramMessage):
	issue: SnapshotterIssue
	status: SnapshotterStatus

Proposed Solution
NA

Caveats
NA

Additional context
NA

[Seth-Schmidt]

The fix has been implemented on this issue branch: https://github.com/PowerLoom/snapshotter-lite-v2/tree/113-fix-maintenance-of-internal-state-related-to-snapshot-submissions

It changes the missed snapshot tracking pattern by bubbling up exceptions and catching a failed snapshot process at the top level instead of tracking individual exceptions as they pop up. Will need to test a bit to ensure that all exceptions are now being handled correctly.

E: The preloader failures were also not being tracked properly which is likely the cause of the fields being reported as 0. Most failures due to RPC issues will fail at the preloader level.