Runs on Cloud

AutoMQ is a cloud-native reimagining of the Kafka release, making full use of cloud services such as EBS and S3. The following sections will introduce the support for AutoMQ across major cloud providers.

Dependency on EBS Services

AutoMQ leverages EBS as Write-Ahead-Log (WAL) storage, accessing it as a raw device without the need to mount a file system. EBS is a core product of cloud providers and offers extremely high data durability, often utilizing triple replication or erasure coding for multiple replicas. Services like AWS Nitro and Alibaba Cloud ECS Bare Metal Instance use local hardware acceleration, and EBS server clusters leverage RDMA technology, making EBS latency comparable to local NVME SSD. Additionally, EBS uses dedicated network channels that do not consume the host machine’s network bandwidth. When used appropriately, EBS is highly cost-effective, such as AWS EBS GP3 at 10GB/3000IOPS/125MBps costing less than $1 per month.

Therefore, EBS is an excellent alternative to traditional data storage technologies for WAL.

AutoMQ accesses EBS using standard POSIX APIs, ensuring great compatibility with all mainstream cloud providers' EBS services.

Dependency on Object Storage Services

AutoMQ accesses object storage services using the AWS S3 SDK. Most cloud providers support the standard S3 protocol, with the only exception being Azure. The AutoMQ team will soon release a separate adaptation for Azure, so stay tuned.

The Community Edition of AutoMQ uses the following AWS S3 SDK to access object storage.

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.20.127</version>
</dependency>

Dependency on Access Control Policies

When accessing object storage, to ensure security, you can restrict access by setting minimal permissions. The configuration methods are similar across major cloud providers. Refer to Cluster Deployment on Linux▸ for detailed steps.

Testing Object Storage Compatibility

After creating a storage bucket, you can use the following commands to test if the object storage is compatible with AutoMQ.

bin/automq-kafka-admin.sh generate-s3-url \ 
--s3-access-key=xxx  \ 
--s3-secret-key=yyy \ 
--s3-region=cn-northwest-1  \ 
--s3-endpoint=s3.cn-northwest-1.amazonaws.com.cn \ 
--s3-data-bucket=automq-test \ 
--s3-ops-bucket=automq-test

List of Supported Cloud Providers

Cloud Providers	EBS Compatibility	Object Storage Compatibility	Description
AWS	Compatible	Compatible	-
Azure	Compatible	Not Compatible	Azure's object storage network protocol is entirely different from AWS S3, requiring separate adaptation.
Google Cloud	Compatible	Not Compatible	The deleteObjects and uploadPartCopy interfaces of Google Cloud are not compatible with AWS S3 and require separate adaptation.
Alibaba Cloud	Compatible	Compatible	-
Oracle Cloud	Compatible	Compatible	-
DigitalOcean	Compatible	Compatible	-
Cloudflare	Compatible	Compatible	-
Tencent Cloud	Compatible	Compatible	-
Huawei Cloud	Compatible	Compatible	-
Baidu Cloud	Compatible	Compatible	-
Volcanic Cloud	Compatible	Pending Testing	-
China Mobile Cloud	Compatible	Pending Testing	-
Chinatelecom Cloud	Compatible	Pending Testing	-

Deploy AutoMQ

Please refer to Cluster Deployment on Linux▸