feat: expand exploreDirectory algorithm

[JKRhb]

This PR contains a rough first draft for expanding the exploreDirectory algorithm.

---

Preview | Diff

[zolkis]

This should be expanded into more concise sub-steps.

[JKRhb]

Started the split into sub-steps, however, there is still some more refinement needed.

[zolkis]

The algorithm drafted here looks more like an internal implementation detail. It is hard to standardize it in order to expose these details to scripts. Or, in other words, it's hard to distillate simple APIs that cover the programmatic side of discovery, without going too much into protocol details -- since then a script could just use the network/REST API.

I am thinking if we could make examples on how to fetch TD directories using the Discovery spec / network API, as depicted here. Those will help figuring out if we can distillate a good programmatic API on top of that.

Currently the WoT Discovery introduction mechanisms are definitely encapsulated in the Scripting implementation.

Moreover, the Authentication layer and the Exploration layer, too, are encapsulated. The Scripting API presents another (programmatic) exploration API that connects to the middleman (the Scripting implementation), provisioned as an endpoint in the given WoT solution, and maybe represented (in the future) as an internal slot to the implementation (to the WoT API object). True, we are missing a standard provisioning API, but more about this later.

Since currently we are not co-hosting multiple solutions in a single WoT runtime (Scripting implementation, i.e. WoT API object), this works quite well, separating the complexities of the underlying WoT Discovery spec.

We can specify a separate API for implementing the details of the WoT Discovery spec, and that should be in a different conformance class, with its own Security section, same level as WoT Provisioning. In my view, these belong to the next level down in the stack. If we were to standardize these (Provisioning, Discovery introduction + exploration), I would support it, but eventually in a separate deliverable (spec, impl + tests) in the WG.

If we were to co-host and support multiple WoT/IoT solutions with a single runtime and WoT object and API, then it needs to be transparent from identification/authentication point of view as well, therefore pulling the need for more low level configuration functions -- therefore going one level down as mentioned above.

Another way to do that is to expose multiple WoT objects for multiple solutions, i.e. we need an additional root-level API to create (async) a WoT API object, with the given provisioning/initialization routines, including discovery introduction and exploration, resulting in a configured discovery "middleman" internal slot. We could use the WoT object which exposes the current API functionality the same way as now, making this proposal backwards compatible, but also (re)configurable wrt provisioning, discovery etc.

I can draft an experimental API for this in a separate issue.

[JKRhb]

Thank you, @zolkis, for your comment and the proposal in #558! I think we can probably close this PR then.

Should we maybe also close #535 since the discussion is going in a different direction now?