Nicolas/514oss 1394 fix plan comparison with remote

[callicles]

This pull request to apps/framework-cli includes several changes to improve the handling of Kafka topics and streamline the infrastructure components. The most important changes include renaming topic-related fields, updating the API update listener to handle namespaced topics, and refactoring the Redpanda client usage.

Kafka Topic Handling Improvements:

• Renamed topic_name to kafka_topic_name in RouteMeta struct and updated all associated references in local_webserver.rs [1] [2] [3].
• Updated the spawn_api_update_listener method to handle namespaced topics using RedpandaStreamConfig [1] [2].

Redpanda Client Refactoring:

• Refactored the Redpanda client usage to use the client module for creating producers and consumers [1] [2].
• Updated methods to fetch topics using the new Redpanda client module and handle topic names without namespaces.

API Change Handling:

• Modified the spawn_api_update_listener method and related functions to pass the InfrastructureMap along with ApiChange [1] [2].

Plan Validation:

• Updated plan validation functions to include the project parameter [1] [2].

Documentation:

• Added a new documentation module for the framework/core/infrastructure components to provide better context and usage guidelines.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
framework-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 25, 2025 6:32pm
framework-landing	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 25, 2025 6:32pm
moose-logs-frontend	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 25, 2025 6:32pm
moose-product-analytics	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 25, 2025 6:32pm

[Copilot]

Pull Request Overview

This pull request refactors the Kafka topic handling and Redpanda client integration while introducing several API signature changes and improving documentation. Key changes include renaming topic fields (topic_name → kafka_topic_name), updating the API update listener and plan validation functions to pass along the project and infrastructure map, and refactoring the Redpanda client usage under the new client module.

Reviewed Changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
apps/framework-cli/src/framework/core/infrastructure.rs	Added module documentation and definitions for infrastructure components.
apps/framework-cli/src/framework/versions.rs	Extended version handling with improved documentation for semantic versioning.
apps/framework-cli/src/cli/local_webserver.rs	Updated route metadata field names and modified listener logic to handle namespaced topics.
apps/framework-cli/src/infrastructure.rs	Removed unused ingest module.
apps/framework-cli/src/framework/python/wrappers/streaming_function_runner.py	Adapted to new KafkaTopicConfig parameters for streaming configurations.
apps/framework-cli/src/cli/routines.rs, plan_validator.rs, execute.rs, etc.	Applied corresponding API changes and refactoring across the codebase.

Comments suppressed due to low confidence (2)

apps/framework-cli/src/cli/local_webserver.rs:1052

• The function signature now includes a project parameter; ensure that all invocations (including tests) are updated to pass the required project reference for consistency.

pub async fn spawn_api_update_listener(

apps/framework-cli/src/framework/core/infrastructure/function_process.rs:100

• [nitpick] Once all deployments are fully migrated to the new schema, consider removing the backward compatibility mappings (for source_columns, target_topic_config, and target_columns) to simplify the code.

// Reverse compatibility with old code

[Copilot]

Pull Request Overview

This PR updates the framework’s streaming and infrastructure components by renaming topic-related fields and transitioning from Redpanda to Kafka-based configurations. Key changes include:

• Renaming the RouteMeta field from “topic_name” to “kafka_topic_name” and updating associated references.
• Updating the API update listener to pass an (InfrastructureMap, ApiChange) tuple and refactoring consumer/producer creation to use Kafka.
• Replacing all occurrences of Redpanda configuration with the new Kafka configuration in both Rust and Python modules.

Reviewed Changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated no comments.

File	Description
apps/framework-cli/src/cli/local_webserver.rs	Updated RouteMeta and API listener to use kafka_topic_name and pass project with KafkaStreamConfig
apps/framework-cli/src/framework/core/infrastructure/function_process.rs	Renamed topic fields to use “_id” suffix and removed legacy fields for better clarity
apps/framework-cli/src/framework/python/wrappers/streaming_function_runner.py	Updated argument parsing to use KafkaTopicConfig over legacy topic strings
Other files (e.g., versions.rs, code_loader.rs, routines/*, etc.)	Replaced Redpanda configuration and module references with Kafka-based implementations

Comments suppressed due to low confidence (4)

apps/framework-cli/src/cli/local_webserver.rs:1050

• The updated function signature now takes a 'project' parameter and returns a channel of type (InfrastructureMap, ApiChange). Ensure that related variable names, documentation, and usage across the codebase are updated consistently.

pub async fn spawn_api_update_listener(

apps/framework-cli/src/framework/core/infrastructure/function_process.rs:24

• [nitpick] The renaming from 'source_topic' to 'source_topic_id' (and similarly for target) improves clarity; verify that all references, tests, and documentation reflect this change consistently.

pub source_topic_id: String,

apps/framework-cli/src/framework/python/streaming.rs:5

• [nitpick] The update replacing RedpandaConfig with KafkaConfig requires ensuring that all Kafka-related operations and imports are consistent across the module.

use crate::infrastructure::stream::{kafka::models::KafkaConfig, StreamConfig};

apps/framework-cli/src/cli/routines/ls.rs:14

• [nitpick] Ensure that the new Kafka module import is used consistently and that any lingering Redpanda references have been fully removed.

use crate::infrastructure::stream::kafka::{self},