updated docs and example notebooks

Squashed commit of the following: commit c3f1087 Author: AndreFCruz <[email protected]> Date: Fri Jul 5 20:53:40 2024 +0200 removed dev branch from docs workflow commit b0b8027 Author: AndreFCruz <[email protected]> Date: Fri Jul 5 20:22:49 2024 +0200 minor notebook update commit 7f7ac05 Author: AndreFCruz <[email protected]> Date: Fri Jul 5 20:11:02 2024 +0200 update docs requirements commit e4b4e9b Author: AndreFCruz <[email protected]> Date: Fri Jul 5 20:06:23 2024 +0200 update docs requirements commit 5d172b3 Author: AndreFCruz <[email protected]> Date: Fri Jul 5 19:57:02 2024 +0200 set docs to track dev branch as well commit 6c0cb80 Author: AndreFCruz <[email protected]> Date: Fri Jul 5 19:55:03 2024 +0200 updated documentation commit 085409f Author: AndreFCruz <[email protected]> Date: Fri Jul 5 18:51:26 2024 +0200 updated notebooks commit 0e5fc76 Author: AndreFCruz <[email protected]> Date: Thu Jul 4 17:54:57 2024 +0200 updated gitignore commit 203182f Author: AndreFCruz <[email protected]> Date: Thu Jul 4 16:02:20 2024 +0200 minor readme update commit 1d1c8bf Author: AndreFCruz <[email protected]> Date: Thu Jul 4 14:44:54 2024 +0200 updated feature importance plots commit 37f9483 Author: AndreFCruz <[email protected]> Date: Thu Jul 4 14:26:45 2024 +0200 updated feature importance notebooks commit d465024 Author: AndreFCruz <[email protected]> Date: Wed Jul 3 13:21:12 2024 +0200 updated notebooks commit ce428da Author: AndreFCruz <[email protected]> Date: Tue Jul 2 12:08:55 2024 +0200 updated ACS benchmark results commit d357258 Author: AndreFCruz <[email protected]> Date: Tue Jul 2 11:48:06 2024 +0200 minor notebook update
socialfoundations · Jul 9, 2024 · 4b978b0 · 4b978b0
1 parent d67fad5
commit 4b978b0
Show file tree

Hide file tree

Showing 22 changed files with 3,267 additions and 558 deletions.
diff --git a/.github/workflows/python-docs.yml b/.github/workflows/python-docs.yml
@@ -26,8 +26,13 @@ jobs:
         run: |
           pip install -e ".[docs]"
 
+      - name: Install pandoc (external dependency)
+        run: |
+          sudo apt-get update && sudo apt-get install -y pandoc
+
       - name: Build Documentation
         run: |
+          sphinx-apidoc -o docs/source ../folktexts tests/
           sphinx-build docs _build
 
       - name: Deploy to GitHub Pages

diff --git a/.gitignore b/.gitignore
@@ -163,10 +163,15 @@ cython_debug/
 # MacOS
 .DS_Store
 ._.DS_Store
+._*
 
 # HTCondor submission files
 *.sub
 
 # Logs
 *.err
 *.out
+
+# sphinx docs
+docs/_build/
+docs/notebooks/
diff --git a/README.md b/README.md
@@ -7,10 +7,8 @@
 ![PyPI - License](https://img.shields.io/pypi/l/folktexts)
 ![Python compatibility](https://badgen.net/pypi/python/folktexts)
 
-
-Folktexts is a python package to compute and evaluate classification risk scores
-using large language models.
-It enables using any transformers model as a classifier for tabular data tasks.
+Folktexts is a python package to evaluate statistical properties of LLMs as classifiers.
+It enables computing and evaluating classification _risk scores_ for tabular prediction tasks using LLMs.
 
 Several benchmark tasks are provided based on data from the American Community Survey.
 Namely, each prediction task from the popular 
@@ -23,6 +21,7 @@ Package documentation can be found [here](https://socialfoundations.github.io/fo
 - [Installing](#installing)
 - [Basic setup](#basic-setup)
 - [Example usage](#example-usage)
+- [Evaluating feature importance](#evaluating-feature-importance)
 - [Benchmark options](#benchmark-options)
 - [License and terms of use](#license-and-terms-of-use)
 
@@ -111,6 +110,23 @@ clf.predict(dataset)
 LLMClassifier (maybe the above code is fine for this), the benchmark, and
 creating a custom ACS prediction task -->
 
+## Evaluating feature importance
+
+By evaluating LLMs on tabular classification tasks, we can use standard feature importance methods to assess which features the model uses to compute risk scores.
+
+You can do so yourself by calling `folktexts.cli.eval_feature_importance` (add `--help` for a full list of options).
+
+Here's an example for the Llama3-70B-Instruct model on the ACSIncome task (*warning: takes 24h on an Nvidia H100*):
+```
+python -m folktexts.cli.eval_feature_importance --model 'meta-llama/Meta-Llama-3-70B-Instruct' --task ACSIncome --subsampling 0.1
+```
+<div style="text-align: center;">
+<img src="docs/_static/feat-imp_meta-llama--Meta-Llama-3-70B-Instruct.png" alt="feature importance on llama3 70b it" width="50%">
+</div>
+
+This script uses sklearn's [`permutation_importance`](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html#sklearn.inspection.permutation_importance) to assess which features contribute the most for the ROC AUC metric (other metrics can be assessed using the `--scorer [scorer]` parameter).
+
+
 ## Benchmark options
 
 ```

diff --git a/docs/_static/custom.js b/docs/_static/custom.js
@@ -0,0 +1,14 @@
+// File: custom.js
+
+/**
+ * This function will make sure all links in the document open as separate tabs
+ * instead of opening on the current user window.
+ */
+window.onload = function () {
+    let links = document.getElementsByTagName('a');
+    for (let l of links) {
+        if (l.classList.contains("reference") && l.classList.contains("external")) {
+            l.target = '_blank'; // Open links in a new tab
+        }
+    }
+};
diff --git a/docs/_static/feat-imp_meta-llama--Meta-Llama-3-70B-Instruct.png b/docs/_static/feat-imp_meta-llama--Meta-Llama-3-70B-Instruct.png
diff --git a/docs/conf.py b/docs/conf.py
@@ -12,15 +12,19 @@
 author = 'AndreFCruz'
 
 # Import package version programmatically
+import os
 import sys
-from pathlib import Path
 
-sys.path.insert(0, Path(__file__).parent.as_posix())
+sys.path.insert(0, os.path.abspath('../'))
 from folktexts._version import __version__
 
 release = __version__
 version = __version__
 
+# Copy notebooks folder to the documentation folder
+import shutil
+shutil.copytree(src="../notebooks", dst="notebooks", dirs_exist_ok=True)
+
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -32,19 +36,34 @@
     'sphinx.ext.napoleon',
     'sphinx.ext.autosummary',
     'sphinx.ext.autosectionlabel',
+    'sphinx_autodoc_typehints',  # needs to be AFTER napoleon
+    'numpydoc',
     'sphinx_copybutton',
     # 'sphinx_autopackagesummary',
     'myst_parser',
     'sphinx.ext.viewcode',
-    'sphinxemoji.sphinxemoji',
+    'sphinx.ext.githubpages',
+    'nbsphinx',             # for rendering jupyter notebooks
+    'sphinx_gallery.load_style',
+    'IPython.sphinxext.ipython_console_highlighting',   # work-around for syntax-highlighting on jupyter notebooks
 ]
 
 templates_path = ['_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
+autoclass_content = 'both'
 
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 html_theme = 'sphinx_rtd_theme'
 html_static_path = ['_static']
+html_js_files = [
+    'custom.js',    # custom JS file
+]
+
+# nbsphinx configuration
+nbsphinx_execute = 'never'  # Set to 'always' if you want to execute the notebooks during the build process
+
+# numpydoc configuration
+numpydoc_show_class_members = False
diff --git a/docs/index.rst b/docs/index.rst
@@ -3,6 +3,53 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
+.. error-parity documentation master file, created by
+   sphinx-quickstart on Thu Nov 23 16:53:42 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+
+Welcome to :code:`folktexts`' documentation!
+============================================
+
+The :code:`folktexts` package enables you to benchmark and evaluate LLMs as risk
+scores.
+
+We encode unrealizable tabular prediction tasks as natural-language text,
+and prompt LLMs for the probability of target variable being true.
+The correct solutions for each task often require expressing uncertainty, as the
+target variable is not uniquely determined by the input features.
+
+Folktexts is compatible with any huggingface transformer model.
+
+Five tabular data tasks are provided out-of-the-box, using the American
+Community Survey as a data source: `ACSIncome`, `ACSMobility`, `ACSTravelTime`, 
+`ACSEmployment`, and `ACSPublicCoverage`. These tasks follow the same name, feature 
+columns, and target columns as those put forth by `Ding et al. (2021)`_ in the
+`folktables`_ python package.
+
+
+Full code available on the `GitHub repository`_, 
+including various `jupyter notebook examples`_ .
+
+Check out the following sub-pages:
+
+.. toctree::
+   :maxdepth: 1
+
+   Readme file <readme>
+   API reference <source/modules>
+   Example notebooks <notebooks>
+
+
+Indices
+=======
+
+* :ref:`genindex`
+* :ref:`modindex`
+
 
-.. include:: ../README.md
-   :parser: myst_parser.sphinx_
+.. _folktables: https://github.com/socialfoundations/folktables
+.. _Ding et al. (2021): https://arxiv.org/abs/2108.04884
+.. _GitHub repository: https://github.com/socialfoundations/folktexts
+.. _jupyter notebook examples: https://github.com/socialfoundations/folktexts/tree/main/notebooks
diff --git a/docs/notebooks.rst b/docs/notebooks.rst
@@ -0,0 +1,6 @@
+Notebooks Gallery
+=================
+
+.. nbgallery::
+   notebooks/minimal-example.ipynb
+   notebooks/run-benchmark.ipynb
diff --git a/docs/readme.rst b/docs/readme.rst
@@ -0,0 +1,5 @@
+README.md
+=========
+
+.. include:: ../README.md
+   :parser: myst_parser.sphinx_
diff --git a/docs/source/folktexts.acs.rst b/docs/source/folktexts.acs.rst
@@ -0,0 +1,53 @@
+folktexts.acs package
+=====================
+
+Submodules
+----------
+
+folktexts.acs.acs\_columns module
+---------------------------------
+
+.. automodule:: folktexts.acs.acs_columns
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.acs.acs\_dataset module
+---------------------------------
+
+.. automodule:: folktexts.acs.acs_dataset
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.acs.acs\_questions module
+-----------------------------------
+
+.. automodule:: folktexts.acs.acs_questions
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.acs.acs\_tasks module
+-------------------------------
+
+.. automodule:: folktexts.acs.acs_tasks
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.acs.acs\_thresholds module
+------------------------------------
+
+.. automodule:: folktexts.acs.acs_thresholds
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: folktexts.acs
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/source/folktexts.cli.rst b/docs/source/folktexts.cli.rst
@@ -0,0 +1,61 @@
+folktexts.cli package
+=====================
+
+Submodules
+----------
+
+folktexts.cli.download\_models module
+-------------------------------------
+
+.. automodule:: folktexts.cli.download_models
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.cli.eval\_feature\_importance module
+----------------------------------------------
+
+.. automodule:: folktexts.cli.eval_feature_importance
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.cli.experiments module
+--------------------------------
+
+.. automodule:: folktexts.cli.experiments
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.cli.launch\_experiments\_htcondor module
+--------------------------------------------------
+
+.. automodule:: folktexts.cli.launch_experiments_htcondor
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.cli.rerun\_experiment module
+--------------------------------------
+
+.. automodule:: folktexts.cli.rerun_experiment
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+folktexts.cli.run\_acs\_benchmark module
+----------------------------------------
+
+.. automodule:: folktexts.cli.run_acs_benchmark
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: folktexts.cli
+   :members:
+   :undoc-members:
+   :show-inheritance: