Configuration validation for Discovery

[MeltyBot]

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/296

Originally created by @vischous on 2021-12-23 17:34:00

---

https://gitlab.com/meltano/sdk/-/blob/main/singer_sdk/tap_base.py#L439-441

I'm using config for definitions of some semi complicated things.

Example of use case:

1. https://gitlab.com/autoidm/autoidm-tap-bamboohr/-/merge_requests/7 (This is merged into the master branch as well if you want to take a look there)
   • For this scneario f you define the configurations wrong the required=True validation gets hit, but doesn't throw an error during discovery.
   • lets say I misname the configuration for name to namee

2021-12-23T17:31:09.595670Z [info     ] time=2021-12-23 12:31:09 name=tap-bamboohr level=INFO message=Skipping parse of env var settings... name=tap-bamboohr stdio=stderr type=discovery                                                    2021-12-23T17:31:09.595782Z [info     ] time=2021-12-23 12:31:09 name=tap-bamboohr level=WARNING message=Config validation failed: 'name' is a required property name=tap-bamboohr stdio=stderr type=discovery                               2021-12-23T17:31:09.595863Z [info     ] JSONSchema was: {'type': 'object', 'properties': {'auth_token': {'type': ['string'], 'description': 'Token gathered from BambooHR, instructions are [here](https://documentation.bamboohr.com/docs#section-authentication)'}, 'subdomain': {'type': ['string'], 'description': 'subdomain from BambooHR'}, 'custom_reports': {'type': ['array', 'null'], 'items': {'type': 'object', 'properties': {'name': {'type': ['string']}, 'filters': {'type': 'object', 'properties': {'lastChanged': {'type': ['object', 'null'], 'properties': {'includeNull': {'type': ['string', 'null']}, 'value': {'type': ['string', 'null']}}}}}, 'fields': {'type': 'array', 'items': {'type': ['string']}}}, 'required': ['name', 'filters', 'fields']}, 'description': 'CustomReport full body definition, example in meltano.yml, same format as the Body for the POST request [here](https://documentation.bamboohr.com/reference/request-custom-report-1)'}}, 'required': ['auth_token', 'subdomain']} name=tap-bamboohr stdio=stderr type=discovery
2021-12-23T17:31:09.595961Z [info     ] time=2021-12-23 12:31:09 name=root level=INFO message=Operator '__else__=None' was not found. Unmapped streams will be included in output. name=tap-bamboohr stdio=stderr type=discovery             2021-12-23T17:31:09.596680Z [info     ] Traceback (most recent call last): name=tap-bamboohr stdio=stderr type=discovery                                                                                                                     2021-12-23T17:31:09.596763Z [info     ]   File "<string>", line 1, in <module> name=tap-bamboohr stdio=stderr type=discovery                                                                                                                 2021-12-23T17:31:09.596830Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__ name=tap-bamboohr stdio=stderr type=discovery                                          2021-12-23T17:31:09.596895Z [info     ]     return self.main(*args, **kwargs) name=tap-bamboohr stdio=stderr type=discovery
2021-12-23T17:31:09.596959Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/click/core.py", line 1053, in main name=tap-bamboohr stdio=stderr type=discovery                                              2021-12-23T17:31:09.597027Z [info     ]     rv = self.invoke(ctx)      name=tap-bamboohr stdio=stderr type=discovery                                                                                                                         2021-12-23T17:31:09.597091Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke name=tap-bamboohr stdio=stderr type=discovery                                            2021-12-23T17:31:09.597156Z [info     ]     return ctx.invoke(self.callback, **ctx.params) name=tap-bamboohr stdio=stderr type=discovery                                                                                                     2021-12-23T17:31:09.597224Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke name=tap-bamboohr stdio=stderr type=discovery
2021-12-23T17:31:09.597292Z [info     ]     return __callback(*args, **kwargs) name=tap-bamboohr stdio=stderr type=discovery                                                                                                                 2021-12-23T17:31:09.597386Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 460, in cli name=tap-bamboohr stdio=stderr type=discovery                                       2021-12-23T17:31:09.597463Z [info     ]     tap = cls(  # type: ignore  # Ignore 'type not callable' name=tap-bamboohr stdio=stderr type=discovery                                                                                           2021-12-23T17:31:09.597529Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 82, in __init__ name=tap-bamboohr stdio=stderr type=discovery                                   2021-12-23T17:31:09.597595Z [info     ]     self.input_catalog or self._singer_catalog name=tap-bamboohr stdio=stderr type=discovery
2021-12-23T17:31:09.597660Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 218, in _singer_catalog name=tap-bamboohr stdio=stderr type=discovery                           2021-12-23T17:31:09.597730Z [info     ]     for stream in self.streams.values() name=tap-bamboohr stdio=stderr type=discovery                                                                                                                2021-12-23T17:31:09.597794Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 108, in streams name=tap-bamboohr stdio=stderr type=discovery                                   2021-12-23T17:31:09.597858Z [info     ]     for stream in self.load_streams(): name=tap-bamboohr stdio=stderr type=discovery                                                                                                                 2021-12-23T17:31:09.597923Z [info     ]   File "/home/visch/git/tap-bamboohr/.venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 247, in load_streams name=tap-bamboohr stdio=stderr type=discovery
2021-12-23T17:31:09.597987Z [info     ]     for stream in self.discover_streams(): name=tap-bamboohr stdio=stderr type=discovery                                                                                                             2021-12-23T17:31:09.598051Z [info     ]   File "/home/visch/git/tap-bamboohr/tap_bamboohr/tap.py", line 57, in discover_streams name=tap-bamboohr stdio=stderr type=discovery                                                                2021-12-23T17:31:09.598115Z [info     ]     custom_report = CustomReport(tap=self, name=report["name"], custom_report_http_post_body=report) name=tap-bamboohr stdio=stderr type=discovery                                                   2021-12-23T17:31:09.598203Z [info     ] KeyError: 'name'               name=tap-bamboohr stdio=stderr type=discovery                                                                                                                         2021-12-23T17:31:09.616971Z [info     ]                                name=tap-bamboohr stdio=stderr type=discovery                  

As you can see here an error occurs in the code when we try to do a lookup on the configuration dict, and name does not exist. Instead failing at configuration validation is what I actually want here.

[MeltyBot]

View 7 previous comments from the original issue on GitLab

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

May be fixed by #937