Refactor sphinx directories

[rkent]

This is a fairly significant refactoring of how directories are used. Let me try to explain.

Previously, the sphinx project was organized into a sphinx project folder with conf.py (either a generated default folder, or the user's /doc folder), sphinx-apidoc was run there (and put its output there). conf.py is wrapped, and placed in a wrapped folder under doc_build, along with any content from the user's /doc folder. Then sphinx-build was run from the wrapped folder, with its output placed in a 'sphinx_output' folder there. Yes this is confusing, at least to me.

This also has the side effect, that if the user defines a custom conf.py and index.rst in their doc folder, and there is c++ in the package, the /doc folder of the package had a 'generated' subdirectory created (and possibly more if python was also present. See ffmpeg_image_transport_tools as an example where running rosdoc2 injects a 'generated' folder). This was going to get worse as I started generating other content into that folder, such as the interface definitions. I was also adding these interface definitions fairly early, only into the 'default' directory if it was created.

It's hard to explain all of this, because it is confusing. It was made more confusing, because many of the relevant subdirectories had ambiguous or changing names in the code.

How is this changed? Generated content is placed immediately within the "wrapped_sphinx_directory" where sphinx-build will be called. No build content is placed into the user's folders. sphinx-apidoc, for example, though it runs in the user's python folder, puts no output there, instead puts it directly into the wrapped_sphinx_directory. The interface documents content, which previously generated content into the default sphinx folder to be copied later into the wrapped_sphinx_directory, is now placed immediately into the wrapped_sphinx_folder. This required re-ordering some operations, like the default index.rst is generated later and placed directly into the wrapped_sphinx_folder.

The PR is divided into three different commits with different purposes. These could be consecutive independent PRs if desired.

The first commit, 'clarify directory naming', only renames variables to be easier to understand. Here are the renames and reasons:

• package_src_directory -> python_src_directory: 'package' is ambiguous, elsewhere it means the ROS package, but there it meant the sphinx package (the python directory).
• directory -> doc_build_directory in a function call: Don't rename a variable in a function, if the function is single-use.
• sourcedir -> sphinx_project_directory: 'sourcedir' is ambiguous
• user_sourcedir -> sphinx_project_directory. Again, source is ambiguous, 'user' does not help. Also don't rename in a single use function call.

The second commit, 'refactor to write directly into wrapped directory' changes how the directories are used.

The third commit, 'Add basic C++ test with custom conf.py' just adds a new test case with a custom conf.py and index.rst, also checking that the 'generated' folder does not get created like it did before.

I have considerably more generated content waiting to be PR'd, so it is important the we clarify now where such content should be placed.

[tfoote]

Overall this makes a lot of sense to restructure it to build out of source. The way we got to the current setup was mostly doing a series of decisions versus a strategic choice.

I tried to run this and got a failure with the wrapped directory

[rosdoc2] [INFO] Running Sphinx-build: 'sphinx-build docs_build/tf2_ros/tf2_ros/wrapped_sphinx_directory /home/tullyfoote/ws/ros-infrastructure/rosdoc2/docs_build/tf2_ros/tf2_ros/wrapped_sphinx_directory/sphinx_output' in 'docs_build/tf2_ros/tf2_ros/wrapped_sphinx_directory'

Application error:
Cannot find source directory (/home/tullyfoote/ws/ros-infrastructure/rosdoc2/docs_build/tf2_ros/tf2_ros/wrapped_sphinx_directory/docs_build/tf2_ros/tf2_ros/wrapped_sphinx_directory)

The directory tree inside the docs_build folder appears to be duplicated in the path when trying to invoke sphinx.

I'm doing this with invoking rosdoc2 from the rosdoc2 directory and pointing it elsewhere on my filesystem for the tf2_ros checkout.

[rkent]

Yes I can duplicate this. There is probably a conversion to abs directory missing somewhere. I'll figure it out.

[rkent]

I believe this fixes it.

One thing I want to do is an audit of how relative vs fixed directories are used in rosdoc2. There is no real plan for when one or the other makes sense.

[tfoote]

Thanks refactoring this out of the source tree makes a lot of sense.

I've tested message packages, c++ and python packages all look to be working.