Reduce the BSK wheel size

.github/workflows/pull-request.yml

@@ -152,6 +152,7 @@ jobs:
         run: |
           source .venv/bin/activate
           pip install . -v
+          bskLargeData
       - name: "Run Python Tests"
         run: |
           source .venv/bin/activate

.gitignore

@@ -42,7 +42,7 @@ externalTools/fswAuto/autowrapper/wraps/*
 src/moduleTemplates/autoCModule/*
 src/moduleTemplates/autoCppModule/*
 
-supportData/EphemerisData/de430.bsp
+supportData/EphemerisData/*.bsp
 
 *.fdb_latexmk
 *.fls

CONTRIBUTING.md

@@ -44,7 +44,7 @@ disabled). Always use the "Merge" strategy.
 
 ## Coding Conventions
 
-A [coding conventions](https://hanspeterschaub.info/basilisk/Support/Developer/CodingGuidlines.html) document exists to
+A [coding conventions](https://avslab.github.io/basilisk/Support/Developer/CodingGuidlines.html) document exists to
 explain peculiarities and assist in onboarding.
 
 -  All development should correspond to a GitHub ticket, and branch names and PRs should include the ticket name.

conanfile.py

@@ -69,7 +69,7 @@ def is_running_virtual_env():
 
 class BasiliskConan(ConanFile):
     name = "Basilisk"
-    homepage = "http://hanspeterschaub.info/basilisk"
+    homepage = "https://avslab.github.io/basilisk/"
     f = open('docs/source/bskVersion.txt', 'r')
     version = f.read()
     f.close()

docs/source/Install/pipInstall.rst

@@ -44,6 +44,11 @@ Build options (as passed to ``conanfile.py`` and described in :ref:`configureBui
     installations (``pip install -e .``) are not currently supported. Please follow the standard
     :ref:`configureBuild` process.
 
+After installing
+
+Building Basilisk ``wheel`` File
+--------------------------------
+
 On its own, there is no significant benefit to installing Basilisk in this way. However, supporting standard Python
 packaging tools means that Basilisk can now be built into a pre-compiled `"wheel" (.whl) file
 <https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#wheels>`_ that can be shared
@@ -61,6 +66,30 @@ The main benefit of this approach will come in the future, when a set of pre-com
 allowing most users to easily ``pip install Basilisk`` without compilation, in the same way that packages like
 ``numpy``, ``scipy``, and ``pandas`` are available.
 
+To keep the wheel size smaller, the large BSK data files are not installed by default.  If the user
+wants to use script that assumes they are included into the Basilisk python package, then go to the
+command line, change the current directory to be inside the environment where Basilisk was ``pip`` installed,
+and run the command::
+
+    bskLargeData
+
+This command runs a python file stored in the ``src/utilities`` folder.
+The ``pip install`` process automatically
+creates this console command in the current python environment to call this python file.  The file
+directly downloads the missing large BSK data files and put them into the local Basilisk python
+package installation.
+
+.. note::
+
+    If the computer does not have local internet access and the ``pip install`` is done via
+    a local wheel, then these missing Spice ``*.bsp`` data files can be manually added to::
+
+        .../.venv/lib/python3.11/site-packages/Basilisk/supportData/EphemerisData
+
+If installing Basilisk via a wheel the user does not have direct access to the full Basilisk source
+folder which contains the ``examples`` folder.  The Terminal command ``bskExamples``
+will download a copy of the examples folder into the local directory.
+
 Alternatively, if you download a zip'd folder of the Basilisk source code you can install it via ``pip``
 using::
 

docs/source/Support/bskReleaseNotes.rst

@@ -49,6 +49,12 @@ Version  |release|
 - Updated :ref:`scenarioMonteCarloAttRW` to include the new ``useBokeh`` feature in the ``run()`` method
 - Updated :ref:`scenarioMonteCarloAttRW` to use more pythonic OOP for Monte Carlo data retention
 - Updated :ref:`scenarioMonteCarloSpice` to use more pythonic OOP for Monte Carlo data retention
+- Decreased the Basilisk wheel size by keeping large data files out of the wheel.
+- The wheel installs the local command tool ``bskLargeData`` to execute :ref:`bskLargeData`.
+  The purpose is to install the large Basilisk data files into the local Basilisk
+  python package.
+- The wheel installs the local command tool ``bskExamples`` to execute :ref:`bskExamples`.  This
+  tool downloads the Basilisk repo ``examples`` folder into the local folder
 - Removed the now deprecated ``datashader_utilities.py`` in favor of the new bokeh plotting features in ``AnalysisBaseClass.py``
 - Upgraded protoc compiler to v3.20.0, added ``protobuf`` to optional package install list
 - Created unit tests for protobuffer packing and saving in :ref:`vizInterface`

pyproject.toml

@@ -21,15 +21,32 @@ license     = {file = "LICENSE"}
 description = "Basilisk: an Astrodynamics Simulation Framework"
 
 [project.urls]
-homepage = 'https://hanspeterschaub.info/basilisk/'
+homepage = 'https://avslab.github.io/basilisk/'
+source = "https://github.com/AVSLab/basilisk"
+tracker = "https://github.com/AVSLab/basilisk/issues"
+
+# Define console scripts here
+[project.scripts]
+bskLargeData = "Basilisk.utilities.bskLargeData:main"
+bskExamples = "Basilisk.utilities.bskExamples:main"
 
 [tool.setuptools]
 packages = []  # XXX: Leave blank, populated automatically by setup.py.
 include-package-data = true
 
 [tool.setuptools.package-data]
 "*" = ["*.so", "*.dll", "*.lib", "*.pyd", "*.a", "*.dylib"]  # Include all built objects.
-Basilisk = ["supportData/**/*"]            # Include all support data.
+Basilisk = [
+    "supportData/AlbedoData/*",
+    "supportData/AtmosphereData/*",
+    "supportData/DentonGEO/*",
+    "supportData/EphemerisData/*.tpc",
+    "supportData/EphemerisData/*.dat",
+    "supportData/EphemerisData/*.tls",
+    "supportData/EphemerisData/*.tsc",
+    "supportData/LocalGravData/*",
+    "supportData/MagneticField/*"
+]
 
 [tool.setuptools.dynamic]
 version = {file = "docs/source/bskVersion.txt"}

requirements.txt

@@ -7,3 +7,4 @@ Pillow
 pytest
 pytest-html
 pytest-xdist
+requests

setup.py

@@ -141,6 +141,8 @@ def run(self) -> None:
         )
     ],
 
+    url="https://avslab.github.io/basilisk/",  # Ensure this field is populated
+
     # XXX: Override build_ext with ConanExtension builder.
     cmdclass={'build_ext': BuildConanExtCommand},
 )

src/CMakeLists.txt

@@ -354,24 +354,53 @@ FetchContent_MakeAvailable(googletest)
 include(GoogleTest)
 enable_testing()
 
-# Check if the large Spice files are present.  If not, download from JPL site directly
-if(NOT EXISTS "${CMAKE_SOURCE_DIR}/../supportData/EphemerisData/de430.bsp")
-  message(STATUS "File de430.bsp not found, downloading file:")
-  file(DOWNLOAD
-        "https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/de430.bsp"
-        # Download it to the build directory (in case the download gets interrupted).
-        "${CMAKE_BINARY_DIR}/data/de430.bsp"
-        SHOW_PROGRESS
-        STATUS DOWNLOAD_RESULT_CODE
+
+# Function to download and move files
+function(download_and_move_file url destination)
+    get_filename_component(filename "${destination}" NAME)
+    set(temp_file "${CMAKE_BINARY_DIR}/data/${filename}")
+
+    if(NOT EXISTS "${destination}")
+        message(STATUS "File ${filename} not found, downloading from ${url}...")
+
+        # Download the file and show a progress bar
+        file(DOWNLOAD
+            "${url}"
+            "${temp_file}"
+            SHOW_PROGRESS
+            STATUS DOWNLOAD_RESULT_CODE
         )
-  if(NOT DOWNLOAD_RESULT_CODE EQUAL 0)
-        message(FATAL_ERROR "Failed downloading de430.bsp! Error: ${DOWNLOAD_RESULT}.")
-  endif()
-  # Move the SPICE file into the appropriate directory.
-  file(RENAME "${CMAKE_BINARY_DIR}/data/de430.bsp" "${CMAKE_SOURCE_DIR}/../supportData/EphemerisData/de430.bsp")
-else()
-  message(STATUS "Found Spice files.")
-endif ()
+
+        # Check for download errors
+        if(NOT DOWNLOAD_RESULT_CODE EQUAL 0)
+            message(FATAL_ERROR "Failed downloading ${filename}! Error code: ${DOWNLOAD_RESULT_CODE}.")
+        endif()
+
+        # Move the file into the appropriate directory
+        file(RENAME "${temp_file}" "${destination}")
+        message(STATUS "Downloaded and moved ${filename} to ${destination}.")
+    else()
+        message(STATUS "File ${filename} already exists.")
+    endif()
+endfunction()
+
+# Define download URLs and target paths
+set(SPICE_FILES
+    "https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/de430.bsp::${CMAKE_SOURCE_DIR}/../supportData/EphemerisData/de430.bsp"
+    "https://naif.jpl.nasa.gov/pub/naif/HST/kernels/spk/hst_edited.bsp::${CMAKE_SOURCE_DIR}/../supportData/EphemerisData/hst_edited.bsp"
+    "https://naif.jpl.nasa.gov/pub/naif/pds/data/nh-j_p_ss-spice-6-v1.0/nhsp_1000/data/spk/nh_pred_od077.bsp::${CMAKE_SOURCE_DIR}/../supportData/EphemerisData/nh_pred_od077.bsp"
+)
+
+# Process each large data file
+message(STATUS "Checking large Basilisk data files")
+foreach(spice_file IN LISTS SPICE_FILES)
+    string(REPLACE "::" ";" spice_parts "${spice_file}")
+    list(GET spice_parts 0 url)
+    list(GET spice_parts 1 destination)
+    download_and_move_file("${url}" "${destination}")
+endforeach()
+
+
 
 # TODO: Remove the global link-libraries call
 find_package(Eigen3 CONFIG REQUIRED)

src/utilities/bskExamples.py

@@ -0,0 +1,68 @@
+#
+#  ISC License
+#
+#  Copyright (c) 2024, Autonomous Vehicle Systems Lab, University of Colorado at Boulder
+#
+#  Permission to use, copy, modify, and/or distribute this software for any
+#  purpose with or without fee is hereby granted, provided that the above
+#  copyright notice and this permission notice appear in all copies.
+#
+#  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+#  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+#  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+#  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+#  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+#  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+#  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+
+import os
+import requests
+
+# Function to download a single file
+def download_file(file_url, dest_folder):
+    """Download a Basilisk example file"""
+    if not os.path.exists(dest_folder):
+        os.makedirs(dest_folder)
+
+    local_filename = os.path.join(dest_folder, file_url.split("/")[-1])
+    with requests.get(file_url, stream=True) as response:
+        response.raise_for_status()
+        with open(local_filename, "wb") as file:
+            for chunk in response.iter_content(chunk_size=8192):
+                file.write(chunk)
+    print(f"Downloaded: {local_filename}")
+
+# Function to process a folder or file via GitHub API
+def process_github_folder(api_url, dest_folder):
+    """Process the BSK GitHub examples folder recursively and download all files."""
+    response = requests.get(api_url)
+    if response.status_code != 200:
+        print(f"Failed to fetch the URL: {api_url}")
+        print(f"Response: {response.text}")
+        return
+
+    items = response.json()
+    for item in items:
+        if item["type"] == "file":
+            print(f"Downloading file: {item['name']}")
+            download_file(item["download_url"], dest_folder)
+        elif item["type"] == "dir":
+            print(f"Entering folder: {item['name']}")
+            subfolder_api_url = item["url"]
+            subfolder_dest = os.path.join(dest_folder, item["name"])
+            process_github_folder(subfolder_api_url, subfolder_dest)
+
+def main():
+    """
+    Script to download all GitHub Basilisk examples into the local folder called ``examples``.
+    """
+    # GitHub API URL for the target folder
+    github_api_url = "https://api.github.com/repos/AVSLab/basilisk/contents/examples"
+    destination_folder = "./examples"
+
+    process_github_folder(github_api_url, destination_folder)
+
+# Main execution
+if __name__ == "__main__":
+    main()

src/utilities/bskLargeData.py

@@ -0,0 +1,109 @@
+#
+#  ISC License
+#
+#  Copyright (c) 2024, Autonomous Vehicle Systems Lab, University of Colorado at Boulder
+#
+#  Permission to use, copy, modify, and/or distribute this software for any
+#  purpose with or without fee is hereby granted, provided that the above
+#  copyright notice and this permission notice appear in all copies.
+#
+#  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+#  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+#  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+#  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+#  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+#  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+#  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+
+import os
+import sys
+import requests
+import importlib.util
+from tqdm import tqdm
+
+def download_file(url, destination_path):
+    """Download a file from a URL with a progress bar."""
+    try:
+        print(f"Downloading file from {url}...")
+        response = requests.get(url, stream=True, timeout=10)
+        response.raise_for_status()  # Raise an error for bad HTTP responses
+
+        # Get the total file size from headers
+        total_size = int(response.headers.get("content-length", 0))
+
+        os.makedirs(os.path.dirname(destination_path), exist_ok=True)  # Ensure destination folder exists
+
+        # Download the file with tqdm progress bar
+        with open(destination_path, "wb") as file, tqdm(
+            desc=f"Downloading {os.path.basename(destination_path)}",
+            total=total_size,
+            unit="B",
+            unit_scale=True,
+            unit_divisor=1024,
+        ) as bar:
+            for chunk in response.iter_content(chunk_size=1024):
+                file.write(chunk)
+                bar.update(len(chunk))
+
+        print(f"File downloaded and saved to {destination_path}.")
+    except requests.exceptions.RequestException as e:
+        print(f"Error downloading file: {e}")
+        sys.exit(1)
+    except Exception as e:
+        print(f"Unexpected error: {e}")
+        sys.exit(1)
+
+def main():
+    """
+    Large BSK data files are downloaded directly from their web server
+    and are then installed in the local Basilisk python package.
+    For example, if using a python 3.11 virtual environment, the Spice ``*.bsp`` files
+    will be stored in::
+
+        .../.venv/lib/python3.11/site-packages/Basilisk/supportData/EphemerisData
+
+    This function is useful if Basilisk is installed via a wheel which does not contain
+    these large BSK data file to keep the wheel file size reasonable.  Calling this
+    python file allows these files to be installed in an automated manner.
+
+    If internet access is not available, these large BSK data files can be
+    included in the above python package installation directly as well.
+
+    """
+
+    # Display the task message
+    print("\033[93mTask:\033[0m Downloading large BSK data files")
+
+    # Step 1: Determine the file location of the Basilisk package
+    try:
+        spec = importlib.util.find_spec("Basilisk")
+        if spec is None:
+            print("Basilisk package not found. Ensure it is installed.")
+            sys.exit(1)
+        basilisk_path = os.path.dirname(spec.origin)
+    except Exception as e:
+        print(f"Error locating Basilisk package: {e}")
+        sys.exit(1)
+
+    # Step 2: Define the download URLs and destination paths
+    files_to_download = {
+        "https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/de430.bsp": os.path.join(
+            basilisk_path, "supportData", "EphemerisData", "de430.bsp"
+        ),
+        "https://naif.jpl.nasa.gov/pub/naif/HST/kernels/spk/hst_edited.bsp": os.path.join(
+            basilisk_path, "supportData", "EphemerisData", "hst_edited.bsp"
+        ),
+        "https://naif.jpl.nasa.gov/pub/naif/pds/data/nh-j_p_ss-spice-6-v1.0/nhsp_1000/data/spk/nh_pred_od077.bsp": os.path.join(
+            basilisk_path, "supportData", "EphemerisData", "nh_pred_od077.bsp"
+        ),
+    }
+
+    # Step 3: Download each file
+    for url, destination_path in files_to_download.items():
+        download_file(url, destination_path)
+
+    print("All files downloaded successfully.")
+
+if __name__ == "__main__":
+    main()