antsibull-docs: process routing metadata

[felixfontein]

This PR adds code to antsibull-docs to process routing metadata (meta/runtime.yml for collections, and lib/ansible/config/ansible_builtin_runtime.yml for ansible-base/-core).

1. It removes duplicate module doc pages for redirects:
   • Before: https://docs.ansible.com/ansible/latest/collections/community/crypto/, see acme_account_facts and acme_account_info: the _facts module is a deprecated link to the _info module.
   • After: https://ansible.fontein.de/collections/community/crypto/index.html#plugins-in-community-crypto, the index only contains the non-deprecated acme_account_info, and https://ansible.fontein.de/collections/community/crypto/acme_account_facts_module.html still exists and informs about the deprecated redirect.
2. It adds stubs for all redirects; for example, in community.general 2.0.0, docker_container will be redirected to community.docker.docker_container.
   • The listing https://ansible.fontein.de/collections/community/general/ no longer contains docker_container
   • The file https://ansible.fontein.de/collections/community/general/docker_container_module.html still exists and contains information about the redirect.

This is similar to the stub pages created for the 2.9 docsite (see for example https://docs.ansible.com/ansible/2.9/modules/acme_account_facts_module.html; also there, acme_account_facts did not show up on https://docs.ansible.com/ansible/2.9/modules/list_of_crypto_modules.html or the global module index).

One "disadvantage" of the current form of the PR: it will also create redirect stub pages for all redirects in https://github.com/ansible/ansible/blob/devel/lib/ansible/config/ansible_builtin_runtime.yml. This is a huge amount of files, and I'm not sure whether we really want it. On the other hand, if ansible-core will ever move more content from itself to collections, this will make sure that when users switch in the version selector say from 2.10 to 4.0.0 on a plugin page where the plugin was moved in Ansible 4.0.0, the user sees that the module was moved.

This PR also makes switching between 2.10 and 3 friction-free: if a module has been moved somewhere else in Ansible 3, or has been removed, the user will directly see that (from the stub page they end up on). (For things already moved using the process in ansible-collections/overview#128, you can also switch back from 3 to 2.10 in the version selector and end up with a valid page.)

[felixfontein]

The latest commit (9f28573) makes sure that only new stubs are created for ansible.builtin. For this, antsibull/data/ansible_2_10_routing.yml contains a list of all plugins that are redirected somewhere in ansible-base 2.10. This file never has to be adjusted, but we need it to be sure which redirects were added later (say, for plugins/modules that will be moved in ansible-core 2.11 or later).

[felixfontein]

With this change:

$ ls rst/collections/ansible/builtin/
add_host_module.rst
advanced_host_list_inventory.rst
apt_key_module.rst
apt_module.rst
apt_repository_module.rst
assemble_module.rst
assert_module.rst
async_status_module.rst
auto_inventory.rst
bigip_asm_policy_module.rst
bigip_facts_module.rst
bigip_gtm_facts_module.rst
blockinfile_module.rst
cmd_shell.rst
command_module.rst
config_lookup.rst
constructed_inventory.rst
copy_module.rst
cron_module.rst
csvfile_lookup.rst
debconf_module.rst
debug_module.rst
debug_strategy.rst
default_callback.rst
dict_lookup.rst
dnf_module.rst
dpkg_selections_module.rst
env_lookup.rst
expect_module.rst
fail_module.rst
fetch_module.rst
fileglob_lookup.rst
file_lookup.rst
file_module.rst
find_module.rst
first_found_lookup.rst
free_strategy.rst
gather_facts_module.rst
generator_inventory.rst
getent_module.rst
get_url_module.rst
git_module.rst
group_by_module.rst
group_module.rst
host_group_vars_vars.rst
host_list_inventory.rst
hostname_module.rst
host_pinned_strategy.rst
import_playbook_module.rst
import_role_module.rst
import_tasks_module.rst
include_module.rst
include_role_module.rst
include_tasks_module.rst
include_vars_module.rst
indexed_items_lookup.rst
index.rst
ini_inventory.rst
ini_lookup.rst
inventory_hostnames_lookup.rst
iptables_module.rst
items_lookup.rst
jsonfile_cache.rst
junit_callback.rst
known_hosts_module.rst
linear_strategy.rst
lineinfile_module.rst
lines_lookup.rst
list_lookup.rst
local_connection.rst
memory_cache.rst
meta_module.rst
minimal_callback.rst
nested_lookup.rst
oneline_callback.rst
package_facts_module.rst
package_module.rst
paramiko_ssh_connection.rst
password_lookup.rst
pause_module.rst
ping_module.rst
pipe_lookup.rst
pip_module.rst
powershell_shell.rst
psrp_connection.rst
random_choice_lookup.rst
raw_module.rst
reboot_module.rst
replace_module.rst
rpm_key_module.rst
runas_become.rst
script_inventory.rst
script_module.rst
sequence_lookup.rst
service_facts_module.rst
service_module.rst
set_fact_module.rst
set_stats_module.rst
setup_module.rst
shell_module.rst
sh_shell.rst
slurp_module.rst
ssh_connection.rst
stat_module.rst
su_become.rst
subelements_lookup.rst
subversion_module.rst
sudo_become.rst
systemd_module.rst
sysvinit_module.rst
tempfile_module.rst
template_lookup.rst
template_module.rst
together_lookup.rst
toml_inventory.rst
tree_callback.rst
unarchive_module.rst
unvault_lookup.rst
uri_module.rst
url_lookup.rst
user_module.rst
varnames_lookup.rst
vars_lookup.rst
wait_for_connection_module.rst
wait_for_module.rst
winrm_connection.rst
yaml_inventory.rst
yum_module.rst
yum_repository_module.rst

(https://ansible.fontein.de/collections/ansible/builtin/index.html)

[gundalow]

Thank you for doing this. The shorter list on https://ansible.fontein.de/collections/community/crypto/index.html#plugins-in-community-crypto is a great improvement

https://ansible.fontein.de/collections/community/crypto/acme_account_facts_module.html

1. Wonder if we should break the paragraph into a bullet list. Feels slightly like a wall of text
2. Could we make the call to action clearer (maybe this should be the first point)
3. For redirects should To use it in a playbook, specify be removed, or maybe use the new name?

[felixfontein]

@gundalow I've adjusted the template in c9ca790, see https://ansible.fontein.de/collections/community/crypto/acme_account_facts_module.html for the result.

Could we make the call to action clearer (maybe this should be the first point)

I've moved it up.

For redirects should To use it in a playbook, specify be removed, or maybe use the new name?

For now I've kept it for redirects, and removed it for deprecated redirects. Maybe it should say "To use this redirect in a playbook, specify: foo.bar.old_name. To directly reference the new location, specify: baz.bar.new_name". What do you think?

[felixfontein]

Example of a tombstoned module: https://ansible.fontein.de/collections/community/general/docker_image_facts_module.html

[abadger]

I think we discussed this at length in a previous PR and eventually we settled on "be liberal in what you accept and conservative in what you emit"... So this should use t.DefaultDict and t.Dict instead of t.Mapping.

I think the same thing applies to CollectionRoutingT when used as a return value. I think we created a *ResultT or *ReturnT or something when we had a nested data structure that we both returned and used as a parameter.

[felixfontein]

We did discuss it here: #200 (comment) and in IRC (on November 30th, just read up the logs). I'm not sure we really settled on something, I still think it's better to not let the implementation leak into the return value. But in the end it doesn't matter that much to me, I've adjusted the return types in 5ae211f.

[abadger]

As much as possible, I like to create functions rather than procedures that operate by side effect. I think this one definitely can return its value rather than modifying plugin_type_routing. I have a feeling that add_symlinks() can also return its value but I'm not as sure there. returning the value would mean moving some logic to the higher level and I haven't quite decided if that would be a good thing or not.

Reasons for liking functions:

• Side effects are harder to keep in mind than return values.
• Unit testing is more straightforward for functions than procedures which operate by side effect.

[abadger]

I don't really like that this function both returns plugin_record and mutates plugin_record. But I'm not sure if there's an easy way to fix that. It would be preferable as an API if it returned plugin_record but that would require copying plugin_record when it's not needed. OTOH, mutating the plugin_record without returning it here won't fit in with the way we're using it in load_collection_routing.

[felixfontein]

@abadger thanks for your work on this!