feat: automatically generate docs for all rst files in the repo

[thezaeemaanwar]

Generate docs for all rst files in edx-platform

Implementation Issue: #33950
Planning Issue: #33047

How it works?

The method find_rst_files finds and returns all the rest files in repository. Another method writes all these file paths in the form of a toctree in a custom_toctree.rst file. This custom toctree is included in index.rst which acts as the entry point while building.

[feanil]

@thezaeemaanwar I tried to build the docs locally with this and got the following when I ran make html in the docs directory.

❯ make html
sphinx-build -M html "." "_build"  
Running Sphinx v6.2.1
Unable to load XBlock 'staffgradedxblock'
Traceback (most recent call last):
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/xblock/plugin.py", line 141, in load_classes
    yield (class_.name, cls._load_class_entry_point(class_))
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/xblock/plugin.py", line 70, in _load_class_entry_point
    class_ = entry_point.load()
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2516, in load
    return self.resolve()
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2522, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/staff_graded/__init__.py", line 3, in <module>
    from .staff_graded import StaffGradedXBlock
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/staff_graded/staff_graded.py", line 38, in <module>
    from bulk_grades.api import ScoreCSVProcessor, get_score, set_score
  File "/home/feanil/.virtualenvs/edx-platform/lib/python3.8/site-packages/bulk_grades/api.py", line 18, in <module>
    from openedx.core.djangoapps.course_groups.cohorts import get_cohort
ImportError: cannot import name 'get_cohort' from partially initialized module 'openedx.core.djangoapps.course_groups.cohorts' (most likely due to a circular import) (/home/feanil/src/openedx/edx-platform/openedx/core/djangoapps/course_groups/cohorts.py)
WARNING: html_static_path entry '_static' does not exist
WARNING: The config value `featuretoggles_source_path' has type `Path', defaults to `str'.
WARNING: The config value `settings_source_path' has type `Path', defaults to `str'.
WARNING: The pre-Sphinx 1.0 'intersphinx_mapping' format is deprecated and will be removed in Sphinx 8. Update to the current format as described in the documentation. Hint: "intersphinx_mapping = {'<name>': ('https://docs.python.org/2.7', None)}".https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html#confval-intersphinx_mapping
loading pickled environment... done
loading intersphinx inventory from https://docs.python.org/2.7/objects.inv...
loading intersphinx inventory from https://docs.djangoproject.com/en/1.11/_objects/...

Extension error:
Handler <function on_init at 0x7effd05df040> for event 'builder-inited' threw an exception (exception: generate_toctree() takes 0 positional arguments but 1 was given)
make: *** [Makefile:22: html] Error 2

Do you have this working locally? Did I miss some steps?

[thezaeemaanwar]

Resolved and fixed all the issues @feanil
Do we need to add the option to exclude any rst file?

[feanil]

@thezaeemaanwar yea, we'll want the option to exclude specific files, but we can add that when we need it, maybe just leave a comment in conf.py near where you think it will make the most sense to add that logic.

[farhan]

Closing on account of
#34405