Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: update kubernetes deployment Run. #2919

Merged
merged 39 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
b546cc9
build: k8s improve.
mo3et Nov 8, 2024
2909896
refactor: update docker image contents.
mo3et Nov 12, 2024
dc48098
rename seq file.
mo3et Nov 12, 2024
201f1f0
build: update k8s origin deploys.
mo3et Nov 13, 2024
ce27a4a
update check logic.
mo3et Nov 13, 2024
300778f
update magefile
mo3et Nov 13, 2024
6143a5c
update image name.
mo3et Nov 13, 2024
34236ad
update readme
mo3et Nov 13, 2024
ac20cc9
update Kubernetes Discovery.
mo3et Nov 13, 2024
36ac5e0
revert pkg.
mo3et Nov 13, 2024
2d5eb51
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Nov 21, 2024
f3f0fd0
update create in k8s
mo3et Nov 21, 2024
e767d2c
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Dec 3, 2024
99865d7
update service image release CI.
mo3et Dec 4, 2024
387554f
update deployment image source.
mo3et Dec 4, 2024
08edbf2
update mage contents.
mo3et Dec 4, 2024
ca5a9f9
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Dec 4, 2024
84c8c0e
update pkg source.
mo3et Dec 4, 2024
e882519
update go get pkg.
mo3et Dec 4, 2024
1995cef
fix test file.
mo3et Dec 4, 2024
1c5cba6
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Dec 7, 2024
03cf860
update discovery register.
mo3et Dec 7, 2024
dbe44b6
update
mo3et Dec 7, 2024
8acd160
update deploy yaml.
mo3et Dec 7, 2024
66b27b6
update replica.
mo3et Dec 7, 2024
0a10fa2
update deployment.
mo3et Dec 10, 2024
248ce35
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Dec 10, 2024
472f4a2
remove notfication config.
mo3et Dec 10, 2024
a8fad30
remove notification and zookeeper.
mo3et Dec 10, 2024
fbaa779
update discovery in kubernetes.
mo3et Dec 12, 2024
4cd8914
build: improve kubernetes deployment.
mo3et Dec 12, 2024
e7a127b
update config field in discovery.
mo3et Dec 12, 2024
ea6d1c2
update ReadMe in deployments.
mo3et Dec 12, 2024
cfa2150
Merge branch 'main' of github.com:openimsdk/open-im-server into build…
mo3et Dec 12, 2024
248dc2d
update go mod.
mo3et Dec 12, 2024
2923b62
update const quote.
mo3et Dec 12, 2024
744b3e9
fix test fields.
mo3et Dec 12, 2024
a8c11c5
remove unused method.
mo3et Dec 12, 2024
c719c69
remove unused contents.
mo3et Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions .github/workflows/docker-build-and-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
name: Build and release services Docker Images

on:
push:
branches:
- release-*
release:
types: [published]
workflow_dispatch:
inputs:
tag:
description: "Tag version to be used for Docker image"
required: true
default: "v3.8.3"

jobs:
build-and-push:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Log in to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Log in to Aliyun Container Registry
uses: docker/login-action@v2
with:
registry: registry.cn-hangzhou.aliyuncs.com
username: ${{ secrets.ALIREGISTRY_USERNAME }}
password: ${{ secrets.ALIREGISTRY_TOKEN }}

- name: Extract metadata for Docker (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
tags: |
type=ref,event=tag
type=schedule
type=ref,event=branch
type=semver,pattern={{version}}
type=semver,pattern=v{{version}}
# type=semver,pattern={{major}}.{{minor}}
type=semver,pattern=release-{{raw}}
type=sha
type=raw,value=${{ github.event.inputs.tag }}

- name: Build and push Docker images
run: |
ROOT_DIR="build/images"
for dir in "$ROOT_DIR"/*/; do
# Find Dockerfile or *.dockerfile in a case-insensitive manner
dockerfile=$(find "$dir" -maxdepth 1 -type f \( -iname 'dockerfile' -o -iname '*.dockerfile' \) | head -n 1)

if [ -n "$dockerfile" ] && [ -f "$dockerfile" ]; then
IMAGE_NAME=$(basename "$dir")
echo "Building Docker image for $IMAGE_NAME with tags:"

# Initialize tag arguments
tag_args=()

# Read each tag and append --tag arguments
while IFS= read -r tag; do
tag_args+=(--tag "${{ secrets.DOCKER_USERNAME }}/$IMAGE_NAME:$tag")
tag_args+=(--tag "ghcr.io/${{ github.repository_owner }}/$IMAGE_NAME:$tag")
tag_args+=(--tag "registry.cn-hangzhou.aliyuncs.com/openimsdk/$IMAGE_NAME:$tag")
done <<< "${{ steps.meta.outputs.tags }}"

# Build and push the Docker image with all tags
docker buildx build --platform linux/amd64,linux/arm64 \
--file "$dockerfile" \
"${tag_args[@]}" \
--push "$dir"
else
echo "No valid Dockerfile found in $dir"
fi
done
2 changes: 1 addition & 1 deletion build/images/openim-crontask/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-crontask"]
ENTRYPOINT ["sh", "-c", "_output/openim-crontask"]
2 changes: 1 addition & 1 deletion build/images/openim-msggateway/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-msggateway"]
ENTRYPOINT ["sh", "-c", "_output/openim-msggateway"]
2 changes: 1 addition & 1 deletion build/images/openim-msgtransfer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-msgtransfer"]
ENTRYPOINT ["sh", "-c", "_output/openim-msgtransfer"]
2 changes: 1 addition & 1 deletion build/images/openim-push/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-push"]
ENTRYPOINT ["sh", "-c", "_output/openim-push"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-auth/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-auth"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-auth"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-conversation/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-conversation"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-conversation"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-friend/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-friend"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-friend"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-group/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-group"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-group"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-msg/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-msg"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-msg"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-third/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-third"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-third"]
2 changes: 1 addition & 1 deletion build/images/openim-rpc-user/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,4 @@ COPY --from=builder $SERVER_DIR/_output $SERVER_DIR/_output
# COPY --from=builder $SERVER_DIR/config $SERVER_DIR/config

# Set the command to run when the container starts
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-user"]
ENTRYPOINT ["sh", "-c", "_output/openim-rpc-user"]
2 changes: 1 addition & 1 deletion config/discovery.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ rpcService:
friend: friend-rpc-service
msg: msg-rpc-service
push: push-rpc-service
messageGateway: messageGateway-rpc-service
messageGateway: messagegateway-rpc-service
group: group-rpc-service
auth: auth-rpc-service
conversation: conversation-rpc-service
Expand Down
1 change: 0 additions & 1 deletion config/openim-rpc-auth.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ rpc:
# It will only take effect when autoSetPorts is set to false.
ports: [ 10200 ]


prometheus:
# Enable or disable Prometheus monitoring
enable: true
Expand Down
174 changes: 1 addition & 173 deletions deployments/Readme.md
Original file line number Diff line number Diff line change
@@ -1,175 +1,3 @@
# OpenIM Application Containerization Deployment Guide

OpenIM supports a variety of cluster deployment methods, including but not limited to `helm`, `sealos`, `kustomize`

Various contributors, as well as previous official releases, have provided some referenceable solutions:

+ [k8s-jenkins Repository](https://github.com/OpenIMSDK/k8s-jenkins)
+ [open-im-server-k8s-deploy Repository](https://github.com/openimsdk/open-im-server-k8s-deploy)
+ [openim-charts Repository](https://github.com/OpenIMSDK/openim-charts)
+ [deploy-openim Repository](https://github.com/showurl/deploy-openim)

### Dependency Check

```bash
Kubernetes: >= 1.16.0-0
Helm: >= 3.0
```

### Minimum Configuration

The recommended minimum configuration for a production environment is as follows:

```yaml
CPU: 4
Memory: 8G
Disk: 100G
```

## Configuration File Generation

We have automated all the files, making the generation of configuration files optional for OpenIM. However, if you desire custom configurations, you can follow the steps below:

```bash
$ make init
# Alternatively, use script:
# ./scripts/init-config.sh
```

At this point, configuration files will be generated under `deployments/openim/config`, which you can modify as per your requirements.

## Cluster Setup

If you already have a `kubernetes` cluster, or if you wish to build a `kubernetes` cluster from scratch, you can skip this step.

For a quick start, I used [sealos](https://github.com/labring/sealos) to rapidly set up the cluster, with sealos also being a wrapper for kubeadm at its core:

```bash
$ SEALOS_VERSION=`curl -s https://api.github.com/repos/labring/sealos/releases/latest | grep -oE '"tag_name": "[^"]+"' | head -n1 | cut -d'"' -f4` && \
curl -sfL https://raw.githubusercontent.com/labring/sealos/${SEALOS_VERSION}/scripts/install.sh |
sh -s ${SEALOS_VERSION} labring/sealos
```

**Supported Versions:**

+ docker: `labring/kubernetes-docker`:(v1.24.0~v1.27.0)
+ containerd: `labring/kubernetes`:(v1.24.0~v1.27.0)

#### Cluster Installation:

Cluster details are as follows:

| Hostname | IP Address | System Info |
| -------- | ---------- | ------------------------------------------------------------ |
| master01 | 10.0.0.9 | `Linux VM-0-9-ubuntu 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux` |
| node01 | 10.0.0.4 | Similar to master01 |
| node02 | 10.0.0.10 | Similar to master01 |

```bash
$ export CLUSTER_USERNAME=ubuntu
$ export CLUSTER_PASSWORD=123456
$ sudo sealos run labring/kubernetes:v1.25.0 labring/helm:v3.8.2 labring/calico:v3.24.1 \
--masters 10.0.0.9 \
--nodes 10.0.0.4,10.0.0.10 \
-u "$CLUSTER_USERNAME" \
-p "$CLUSTER_PASSWORD"
```

> **Node** Uninstallation method: using `kubeadm` for uninstallation does not remove `etcd` and `cni` related configurations. Manual clearance or using `sealos` for uninstallation is needed.
>
> ```bash
> $ sealos reset
> ```

If you are local, you can also use Kind and Minikube to test, for example, using Kind:

```bash
$ GO111MODULE="on" go get sigs.k8s.io/[email protected]
$ kind create cluster
```

### Installing helm

Helm simplifies the deployment and management of Kubernetes applications to a large extent by offering version control and release management through packaging.

**Using Script:**

```bash
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
```

**Adding Repository:**

```bash
$ helm repo add brigade https://openimsdk.github.io/openim-charts
```

### OpenIM Image Strategy

Automated offerings include aliyun, ghcr, docker hub: [Image Documentation](https://github.com/openimsdk/open-im-server/blob/main/docs/contrib/images.md)

**Local Test Build Method:**

```bash
$ make image
```

> This command assists in quickly building the required images locally. For a detailed build strategy, refer to the [Build Documentation](https://github.com/openimsdk/open-im-server/blob/main/build/README.md).

## Installation

Explore our Helm-Charts repository and read through: [Helm-Charts Repository](https://github.com/openimsdk/helm-charts)


Using the helm charts repository, you can ignore the following configuration, but if you want to just use the server and scale on top of it, you can go ahead:

**Use the Helm template to generate the deployment yaml file: `openim-charts.yaml`**

**Gen Image:**

```bash
../scripts/genconfig.sh ../scripts/install/environment.sh ./templates/helm-image.yaml > ./charts/generated-configs/helm-image.yaml
```

**Gen Charts:**

```bash
for chart in ./charts/*/; do
if [[ "$chart" == *"generated-configs"* || "$chart" == *"helmfile.yaml"* ]]; then
continue
fi

if [ -f "${chart}values.yaml" ]; then
helm template "$chart" -f "./charts/generated-configs/helm-image.yaml" -f "./charts/generated-configs/config.yaml" -f "./charts/generated-configs/notification.yaml" >> openim-charts.yaml
else
helm template "$chart" >> openim-charts.yaml
fi
done
```

**Use Helmfile:**

```bash
GO111MODULE=on go get github.com/roboll/helmfile@latest
```

```bash
export MONGO_ADDRESS=im-mongo
export MONGO_PORT=27017
export REDIS_ADDRESS=im-redis-master
export REDIS_PORT=6379
export KAFKA_ADDRESS=im-kafka
export KAFKA_PORT=9092
export OBJECT_APIURL="https://openim.server.com/api"
export MINIO_ENDPOINT="http://im-minio:9000"
export MINIO_SIGN_ENDPOINT="https://openim.server.com/im-minio-api"

mkdir ./charts/generated-configs
../scripts/genconfig.sh ../scripts/install/environment.sh ./templates/config.yaml > ./charts/generated-configs/config.yaml
cp ../config/notification.yaml ./charts/generated-configs/notification.yaml
../scripts/genconfig.sh ../scripts/install/environment.sh ./templates/helm-image.yaml > ./charts/generated-configs/helm-image.yaml
```

```bash
helmfile apply
```
view deploy [README](./deploy/README.md)
34 changes: 0 additions & 34 deletions deployments/charts/helmfile.yaml

This file was deleted.

Loading
Loading