Interface for Queues #220
Interface for Queues #220
Conversation
WalkthroughThis pull request introduces new classes for managing message queues across multiple cloud platforms, including AWS SQS, Azure Storage Queues, and Google Cloud Pub/Sub. It adds a new dependency for Google Cloud Pub/Sub in the requirements file and establishes an abstract base class for queues, ensuring a consistent interface for queue operations. Each cloud-specific class implements methods for creating, deleting, sending, and receiving messages, enhancing the project's capability to handle asynchronous messaging in a serverless architecture. Changes
Possibly related PRs
Poem
Tip New featuresWalkthrough comment now includes:
Notes:
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
| self._subscription_client = pubsub_v1.SubscriberClient() | ||
|
|
||
| self._topic_name = 'projects/{project_id}/topics/{topic}'.format( | ||
| project_id=os.getenv('GOOGLE_CLOUD_PROJECT'), |
There was a problem hiding this comment.
This variable should be stored already in config.resources -> no need to retrieve it again.
There was a problem hiding this comment.
I can't see another mention of GOOGLE_CLOUD_PROJECT in the code - I believe it may have only been added in config.resources by your NoSQL PR?
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (5)
- requirements.gcp.txt (1 hunks)
- sebs/aws/queue.py (1 hunks)
- sebs/azure/queue.py (1 hunks)
- sebs/faas/queue.py (1 hunks)
- sebs/gcp/queue.py (1 hunks)
Additional comments not posted (3)
sebs/faas/queue.py (1)
8-11: EnumQueueTypedefined correctly.The enumeration
QueueTypeis well-defined and provides clear options for queue types, which enhances readability and maintainability.sebs/aws/queue.py (1)
7-117: Review of classSQS:
Properties (
queue_urlandqueue_arn):
- These properties correctly expose the private attributes
_queue_urland_queue_arn. This encapsulation is good practice.- Ensure that these attributes are always initialized before being accessed.
Constructor (
__init__):
- The constructor initializes the AWS SQS client and sets up queue URL and ARN if provided. This is a flexible design allowing for both new and existing queues.
- Consider validating the
region,queue_url, andqueue_arnparameters to ensure they meet AWS requirements.Queue Operations (
create_queue,remove_queue,send_message,receive_message):
- The
create_queuemethod checks if the queue exists before trying to create it, which is efficient. However, it should handle potential exceptions from AWS operations.- The
remove_queuemethod should also handle exceptions, especially since deleting a non-existent queue could raise an error.- The
send_messageandreceive_messagemethods are straightforward but should include error handling for AWS service exceptions.Logging:
- Logging is used extensively, which is good for debugging and monitoring. Ensure that the logging level is appropriate for production (e.g., using
infofor routine operations anddebugfor detailed debugging information).Serialization (
serialize,deserialize):
- These methods facilitate the conversion of queue objects to and from dictionaries, which is useful for configurations or inter-service communication.
- Ensure that all necessary attributes are included in the serialization and that the deserialization method correctly re-creates the queue object.
Overall, the class is well-structured and covers the necessary functionality for interacting with AWS SQS. Ensure thorough testing, especially for error handling and edge cases.
Consider adding unit tests to cover:
- Initialization with and without existing queue details.
- Exception handling in all AWS operations.
- Serialization and deserialization integrity.
sebs/gcp/queue.py (1)
10-124: Review of classGCPQueue:
Properties (
topic_name,subscription_name,subscription_client):
- These properties correctly expose the private attributes
_topic_name,_subscription_name, and_subscription_client. This encapsulation is good practice.- Ensure that these attributes are always initialized before being accessed.
Constructor (
__init__):
- The constructor initializes the Google Cloud Pub/Sub client and sets up topic and subscription names based on the environment variables. This design is flexible and allows for easy configuration.
- Consider validating the
regionparameter and the environment variables to ensure they are set correctly.Queue Operations (
create_queue,remove_queue,send_message,receive_message):
- The
create_queuemethod handles the creation of both the topic and the subscription, which is necessary for Google Cloud Pub/Sub. It correctly handles theAlreadyExistsexception.- The
remove_queuemethod should handle exceptions, especially since deleting non-existent resources could raise an error.- The
send_messageandreceive_messagemethods include retry mechanisms and message acknowledgment, which are best practices for reliable messaging.Logging:
- Logging is used extensively, which is good for debugging and monitoring. Ensure that the logging level is appropriate for production.
Serialization (
serialize,deserialize):
- These methods facilitate the conversion of queue objects to and from dictionaries, which is useful for configurations or inter-service communication.
- Ensure that all necessary attributes are included in the serialization and that the deserialization method correctly re-creates the queue object.
Overall, the class is well-structured and covers the necessary functionality for interacting with Google Cloud Pub/Sub. Ensure thorough testing, especially for error handling and edge cases.
Consider adding unit tests to cover:
- Initialization with and without existing topic and subscription details.
- Exception handling in all Google Cloud operations.
- Serialization and deserialization integrity.
This PR introduces interfacing support for queues, namely AWS SQS, GCP Pub/Sub and Azure Queue Storage. Their infrastructure is standardised by an abstract base class
Queue.Support exists for:
The queues will primarily gravitate around triggers - used either as triggers for a function or as a means of collecting cloud-side measurements after the function has run (since for queue- or storage-triggered functions, we cannot simply return an HTTP response back to SeBS).