Pipeline for policy generation from DDS discovery data

[ruffsl]

Related: #112

[mikaelarguedas]

Some things that came up when spot comparing the generated files with the ones from https://github.com/mikaelarguedas/tb3_demo/blob/857e1c62ba06476cda4ba359fdf59d31ba488f91/policies/sqlite_playground/extract_policy_from_discovery_db.py

[ruffsl]

How's it now? See example output:
https://pastebin.com/JhCvvYVh

[mikaelarguedas]

How's it now?

Better! less unnecessary permissions and it validates the schema!

Few surprising things in the result file:

• empty profiles:
  Profiles are generated for namespaces that don't belong to them and this results in a lot of empty profiles.

    <profile node="gazebo" ns="/global_costmap"/>
    <profile node="gazebo" ns="/local_costmap"/>

• There are some topics listed in the database that don't appear in the generated file, for example tf doesnt appear for any node in the generated policies o_O

• clock appears sometimes absolute and sometimes relative. I believe it makes sense as when the namespace is just / the script has no way to know which one of the two it is. Something to keep in mind when adding logic to collapse permissions to higher level common objects

[ruffsl]

• empty profiles:
  Profiles are generated for namespaces that don't belong to them and this results in a lot of empty profiles.

Empty profiles were unintendedly added with an erroneous combinatorial across nodes and namespaces: address in 368f4c5 . I've also added a check for when the discovery user_data from a foreign QoS configuration: see commit message in 3533ec2

• There are some topics listed in the database that don't appear in the generated file, for example tf doesnt appear for any node in the generated policies o_O

The template can be debugged better commenting back the secondary transforms:

sros2/sros2/sros2/policy/templates/dds/demangle.xsl

Lines 204 to 215 in 1854b7e

	<xsl:variable name="profile_deduplicated">
	<xsl:apply-templates mode="deduplicate"
	select="ext:node-set($profile)"/>
	</xsl:variable>
	<xsl:variable name="profile_namespaced">
	<xsl:apply-templates mode="namespace"
	select="ext:node-set($profile_deduplicated)"/>
	</xsl:variable>
	<xsl:apply-templates mode="compress"
	select="ext:node-set($profile_namespaced)"/>

Diffing the consecutive chained output from different iterations of commiting, I found some copy and paste typos in the compression. See 1854b7e . I'd like to make this a little more DRY, but it's tricky when only using XSL v1.0. Let me know if you see a better expression for things.

• clock appears sometimes absolute and sometimes relative. I believe it makes sense as when the namespace is just / the script has no way to know which one of the two it is. Something to keep in mind when adding logic to collapse permissions to higher level common objects

Yea, I thought about this, like adding a lookup for common root level topics in the xst for such cases, but it's probably better instead to ensure the added logic to collapse permissions to higher level common objects is robust enough to handle both appearances, perhaps by expanding the fqn topic namespace using rcl logic when comparing candidate substitutions?

[ruffsl]

You can diff from last pastebin using the fix output here: https://pastebin.com/WYvxNTbg

[mikaelarguedas]

Deduplicating the topics in the raw intermediate representation now generate the exact same topic list as https://github.com/mikaelarguedas/tb3_demo/blob/857e1c62ba06476cda4ba359fdf59d31ba488f91/policies/sqlite_playground/policy_dump_from_sqlite.xml#L1094 🎉

diff --git a/sros2/scripts/dds_sql_to_sros2_policy.py b/sros2/scripts/dds_sql_to_sros2_policy.py
index 6c602cd..801a919 100755
--- a/sros2/scripts/dds_sql_to_sros2_policy.py
+++ b/sros2/scripts/dds_sql_to_sros2_policy.py
@@ -91,9 +91,12 @@ def df_to_dds_policy(df):
                 if not pd.isna(mode):
                     topics = etree.SubElement(profile, 'raws')
                     topics.set(mode, "ALLOW")
+                    raw_topics = []
                     for dds_topic in df['dds_topic'].loc[namespace, name, mode]:
-                        topic = etree.SubElement(topics, 'raw')
-                        topic.text = dds_topic
+                        if dds_topic not in raw_topics:
+                            raw_topics.append(dds_topic)
+                            topic = etree.SubElement(topics, 'raw')
+                            topic.text = dds_topic
     return dds_policy

Comparing the second stage is a bit trickier but spot checking a few nodes everything seems to work as expected 👍

I'd like to make this a little more DRY, but it's tricky when only using XSL v1.0. Let me know if you see a better expression for things.

Yeah that would be good indeed, but like you said it looks tricky to do with XSL 1.0. We can do more investigation after we get everything running well.

like adding a lookup for common root level topics in the xst for such cases, but it's probably better instead to ensure the added logic to collapse permissions to higher level common objects is robust enough to handle both appearances, perhaps by expanding the fqn topic namespace using rcl logic when comparing candidate substitutions?

Yep that's the approach I had in mind (using FQNs) before comparing to common assets.
Not sure if rcl logic needs to be leveraged here as the concatenation of namespace and names already happens in the xsl in some places.

[ruffsl]

@mikaelarguedas and I chatted offline about adding a <raw> type element to the policy schema. Though I think we'll hold off on that for now until we have a more concrete use case for it. So I've just relegated that schema check to its own transport folder for now. a67dbc7