docs(doc_builder.py) overhaul, remove bugs, clean up, modularize

[vwheeler63]

This PR has been long in the making. Firstly, doc_builder.py was a module that was partly a package that dealt with Doxygen XML output as well as logic that generated API pages and links to those pages for build.py -- and these two logically separate functions were literally mixed inside the file, making it difficult to understand and maintain. Finally, it had a significant amount of duplicated functionality, undocumented and/or obfuscated (difficult-to-comprehend) code, and several important bugs, all of which have been fixed herein.

• doc_builder.py was split into 2 cleanly-designed modules:

  • doxygen_xml.py is now the module that deals with Doxygen XML output (also used by gen_json.py, which still generates what it previously generated), and
  • api_doc_builder.py is now much cleaner and better logic that:
    • generates API pages, and
    • adds hyperlinks to those pages to eligible docs in the intermediate directory.

• 3 bugs were found and fixed in doxygen_xml.py that was previously causing about 11.7KB of documentation to be erroneously omitted from the JSON output by gen_json.py, and simultaneously omitted linkage of certain elements in the data structure back to their source files. Both of these were fixed and documented.

• api_doc_builder.py now takes into account NOT generating API pages for source code that Doxygen was told to ignore (e.g. thorvg and other 3rd-party library code that is not part of the LVGL API). Simultaneously, this eliminates a large number of Sphinx errors from the doc build.

• Issue Generated API-Page Link Problems #7068 spells out illogic in auto-generating hyperlinks to the API pages, that especially did not age well when documents were renamed or restructured. This has been thoroughly handled herein.

• The API-page hyperlinks added to eligible documents are now of better quality:

  • For a document to be eligible for adding API-page links to, it must have an API section at the end.
  • Writers/editors can add their own API hyperlinks of they wish. These will remain at the top of the list and will not be repeated.
  • Per Issue Generated API-Page Link Problems #7068, writers/editors can now add guiding API-hyperlink directives to documents that remains robust regardless of documents being renamed and/or restructured.
  • Previously lv_types_h was in virtually every hyperlink list (useless). No more.
  • lv_xxxx_private_h links are now included. Reason: advanced users are going to need this documentation, and it is important for all users to understand that these files exist even when that user is not directly accessing the fields in "private" structs. Better to err on side of thoroughness.
  • API-hyperlink lists are now ordered with a special sort algorithm that helps readers more quickly find links in long lists.

• Functionality that previously prepared the Doxyfile and ran Doxygen is factored out from 2 places to 1 place, now handled inside doxygen_xml.py.

• Functionality that handled non-trivial log-type announcements to STDOUT during the build processes factored out from many places to 1 place in new announce.py module.

• doxygen_xml.py now uses the new doxygen_config.py module to gracefully and reliably manage Doxyfile configuration contents, so the need for the Doxyfile to carry place-holder values for the doc-build process (that make it unusable to an end user) is no longer needed. The result is that Doxyfile is now set up so it is syntactically correct and logical for its location, so any user can

  $ cd ./docs/
  $ doxygen

  and it will generate HTML output in ./docs/doxygen/html or ./docs/doxygen/xxxx where xxxx is any other format the user turns on in the Doxyfile. The local .gitignore file has been modified to allow for this without making Git think new files have arrived. It's not that this is needed, but it is sometimes nice be able to reference Doxygen HTML when verifying API documentation in source files is being done correctly. This is about 50X faster than doing a whole doc-build to find out. This is very useful when you are "tweaking" API documentation in source files to ensure Doxygen understands it correctly. Further, it seemed somewhat unprofessional to have a Doxyfile sitting there carrying invalid Doxygen syntax and thereby being useless, even if someone wanted to use it—he could not change it lest it would previously break the doc-build process. This restriction (thankfully) no longer applies.

As usual, http://crystal-clear-research.com/for_gabor/demo/docs.lvgl.io/master/ is a reflection of what the documentation will look like with this PR merged.

Resolves #7068

cc: @kisvegabor

I attest this is okay to merge.

Notes

• Update the Documentation if needed. Done.
• Add Examples if relevant. n/a
• Add Tests if applicable. n/a
• If you added new options to lv_conf_template.h run lv_conf_internal_gen.py and update Kconfig. n/a
• Run scripts/code-format.py (astyle v3.4.12 needs to installed by running cd scripts; ./install_astyle.sh) and follow the Code Conventions. n/a
• Mark the Pull request as Draft while you are working on the first version, and mark is as Ready when it's ready for review. Done.
• When changes were requested, re-request review to notify the maintainers. Done.
• Help us to review this Pull Request! Anyone can approve or request changes.

[kisvegabor]

Hi Victor,

Im on embedded world and could not immerse in it deeply. In principle in looks great, and happy that these long outstanding error with thorvg (and others) are resolved. @liamHowatt please test it on your end.

[vwheeler63]

In principle in looks great, and happy that these long outstanding error with thorvg (and others) are resolved.

cc: @liamHowatt

Indeed. By the way, this upgrade now makes several things possible that were hidden problems before:

• Doc renaming and restructuring can now be done without losing valuable API links.
• A number of docs incorrectly have API links to things like thorvg files, when they are not actually part of the LVGL API. There are 7 of them, which I am planning to handle after this merge is complete.
• Refine API links. Some documents like draw.rst have an unusually long list of API links and at least half of them (if not 2/3) are not really applicable -- all because the word "draw" was found as the "word" after "lv_..." (or "_lv_...") in 1 or more C symbols—which is the incomplete algorithm this upgrade gets us away from. There are quite a few places this can occur.

Anyway, my point is to let the two of you know these things are coming (with a "wrap-up" PR) and have been prevented (up to this point) by that incomplete algorithm being in the way—remedied herein.