Support a local development upload pipeline

[Nyumat]

Overview

In local development environments for BeavsAI, files stored in Cloudflare R2 are not preserved consistently across different instances. This means that each developer has their own isolated file storage when the database is reset or spun up and down. To improve the development experience, a local upload pipeline needs to be implemented to keep uploaded files synchronized across local environments.

Objective

Implement a local development upload pipeline that allows developers to work with a shared or consistent file storage solution that aligns with local database instances.

Details

Core Features

• Local File Storage Option: Implement an option for local file storage instead of Cloudflare R2 when in a local development environment.
• Configurable Storage Path: Allow developers to configure the file storage path in their local environment for better flexibility.
• Synchronization Mechanism: Develop a method to synchronize uploaded files with the local database to maintain consistency across instances.

Technical Requirements

• Environment Detection: Add logic to detect when the application is running in a local environment and switch to local file storage.
• File Handling: Implement file handling using Node.js file system (fs) or a lightweight storage service such as multer for local file management.
• Database Linking: Ensure that uploaded files in local storage are properly linked and referenced in the local database.
• Fallback Mechanism: Create a fallback to Cloudflare R2 for non-local environments to maintain the current production pipeline.

UI/UX Changes

• Admin Panel (Optional): Add an admin panel or page for local developers to manage and review locally stored files.

Data and Privacy Considerations

• Local Only: Ensure that files stored locally are used strictly for development purposes and are not shared publicly.
• Clean-Up Scripts: Provide clean-up scripts for developers to easily manage local file storage and avoid clutter.

Acceptance Criteria

• Developers can upload files locally without relying on Cloudflare R2.
• Files stored locally are synchronized with the local database for consistent development experience.
• The system detects and switches to local storage when in a development environment.
• Configuration for local file storage path is documented and easy to set up.