Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker optimised #1676

Merged
merged 1 commit into from
Dec 24, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
FROM amazonlinux:latest

ENV KAIRON_HOME=/app PYTHON_VERSION=3.10.13 TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT RASA_TELEMETRY_ENABLED=false
ENV KAIRON_HOME=/app PYTHON_VERSION=3.10.16 TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT RASA_TELEMETRY_ENABLED=false
ENV PYENV_ROOT=/root/.pyenv
ENV PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
Comment on lines +3 to +5
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Optimize Docker layers by combining ENV instructions

Multiple ENV instructions create unnecessary layers. Consider combining them for better optimization.

-ENV KAIRON_HOME=/app PYTHON_VERSION=3.10.16 TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT RASA_TELEMETRY_ENABLED=false
-ENV PYENV_ROOT=/root/.pyenv
-ENV PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
+ENV KAIRON_HOME=/app \
+    PYTHON_VERSION=3.10.16 \
+    TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT \
+    RASA_TELEMETRY_ENABLED=false \
+    PYENV_ROOT=/root/.pyenv \
+    PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ENV KAIRON_HOME=/app PYTHON_VERSION=3.10.16 TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT RASA_TELEMETRY_ENABLED=false
ENV PYENV_ROOT=/root/.pyenv
ENV PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
ENV KAIRON_HOME=/app \
PYTHON_VERSION=3.10.16 \
TEMPLATE_DIR_DEFAULT=./template/use-cases/Hi-Hello-GPT \
RASA_TELEMETRY_ENABLED=false \
PYENV_ROOT=/root/.pyenv \
PATH=$PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH


WORKDIR ${KAIRON_HOME}

RUN yum update -y
RUN dnf update -y rpm --releasever 2023.4.20240319
RUN dnf update -y curl expat --releasever 2023.4.20240401

RUN yum update -y && yum -y install wget make gcc zlib-devel tar xz gzip openssl openssl-devel bzip2-devel sqlite-devel libffi-devel openblas-devel ca-certificates
RUN wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz && \
tar -xf Python-${PYTHON_VERSION}.tgz && \
rm ./Python-${PYTHON_VERSION}.tgz && \
cd Python-${PYTHON_VERSION}/ && \
./configure --enable-optimizations && \
make altinstall && \
rm -rf Python-${PYTHON_VERSION} && \
update-alternatives --install /usr/bin/python python /usr/bin/python2 50 && \
update-alternatives --install /usr/bin/python python /usr/local/bin/python3.10 60 && \
RUN yum install -y wget make gcc zlib-devel tar xz gzip openssl openssl-devel bzip2-devel sqlite-devel libffi-devel openblas-devel ca-certificates git

Comment on lines +10 to +11
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Optimize package installation and reduce image size

The current setup uses multiple RUN instructions and doesn't clean up package manager cache, leading to larger image size.

-RUN yum update -y
-RUN yum install -y wget make gcc zlib-devel tar xz gzip openssl openssl-devel bzip2-devel sqlite-devel libffi-devel openblas-devel ca-certificates git
+RUN yum update -y && \
+    yum install -y \
+        wget \
+        make \
+        gcc \
+        zlib-devel \
+        tar \
+        xz \
+        gzip \
+        openssl \
+        openssl-devel \
+        bzip2-devel \
+        sqlite-devel \
+        libffi-devel \
+        openblas-devel \
+        ca-certificates \
+        git && \
+    yum clean all && \
+    rm -rf /var/cache/yum
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
RUN yum install -y wget make gcc zlib-devel tar xz gzip openssl openssl-devel bzip2-devel sqlite-devel libffi-devel openblas-devel ca-certificates git
RUN yum update -y && \
yum install -y \
wget \
make \
gcc \
zlib-devel \
tar \
xz \
gzip \
openssl \
openssl-devel \
bzip2-devel \
sqlite-devel \
libffi-devel \
openblas-devel \
ca-certificates \
git && \
yum clean all && \
rm -rf /var/cache/yum

RUN curl https://pyenv.run | bash
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc && \
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc && \
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

RUN source ~/.zshrc && \
pyenv install ${PYTHON_VERSION} && \
pyenv global ${PYTHON_VERSION} && \
pyenv rehash && \
Comment on lines +12 to +20
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

⚠️ Potential issue

Security and optimization concerns in pyenv installation

Several issues need attention:

  1. Executing scripts directly from curl poses security risks
  2. Using .zshrc in amazonlinux (which typically uses bash)
  3. Multiple RUN instructions creating unnecessary layers

Consider this safer and more optimized approach:

-RUN curl https://pyenv.run | bash
-RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc && \
-    echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc && \
-    echo 'eval "$(pyenv init -)"' >> ~/.zshrc
-
-RUN source ~/.zshrc && \
-    pyenv install ${PYTHON_VERSION} && \
-    pyenv global ${PYTHON_VERSION} &&  \
-    pyenv rehash &&  \
+RUN set -ex && \
+    curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer -o pyenv-installer && \
+    chmod +x pyenv-installer && \
+    ./pyenv-installer && \
+    rm pyenv-installer && \
+    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
+    echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc && \
+    echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
+    . ~/.bashrc && \
+    pyenv install ${PYTHON_VERSION} && \
+    pyenv global ${PYTHON_VERSION} && \
+    pyenv rehash && \
     python -m pip install --upgrade pip && \
     mkdir data_generator training_data testing_data models /home/cache .rasa
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
RUN curl https://pyenv.run | bash
RUN echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc && \
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc && \
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
RUN source ~/.zshrc && \
pyenv install ${PYTHON_VERSION} && \
pyenv global ${PYTHON_VERSION} && \
pyenv rehash && \
RUN set -ex && \
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer -o pyenv-installer && \
chmod +x pyenv-installer && \
./pyenv-installer && \
rm pyenv-installer && \
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc && \
echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
. ~/.bashrc && \
pyenv install ${PYTHON_VERSION} && \
pyenv global ${PYTHON_VERSION} && \
pyenv rehash && \
python -m pip install --upgrade pip && \
mkdir data_generator training_data testing_data models /home/cache .rasa

python -m pip install --upgrade pip && \
mkdir data_generator training_data testing_data models /home/cache .rasa

Expand All @@ -37,10 +37,10 @@ RUN python -m pip install pyston_lite_autoload
COPY . .

RUN rm -rf ${TEMPLATE_DIR_DEFAULT}/models/* && \
rasa train --data ${TEMPLATE_DIR_DEFAULT}/data --config ${TEMPLATE_DIR_DEFAULT}/config.yml --domain ${TEMPLATE_DIR_DEFAULT}/domain.yml --out ${TEMPLATE_DIR_DEFAULT}/models
python -m rasa train --data ${TEMPLATE_DIR_DEFAULT}/data --config ${TEMPLATE_DIR_DEFAULT}/config.yml --domain ${TEMPLATE_DIR_DEFAULT}/domain.yml --out ${TEMPLATE_DIR_DEFAULT}/models

RUN cp kairon/shared/rule_policy.py /usr/local/lib/python3.10/site-packages/rasa/core/policies/rule_policy.py
RUN cp kairon/shared/schemas/domain.yml /usr/local/lib/python3.10/site-packages/rasa/shared/utils/schemas/domain.yml
RUN cp kairon/shared/rule_policy.py /root/.pyenv/versions/${PYTHON_VERSION}/lib/python3.10/site-packages/rasa/core/policies/rule_policy.py
RUN cp kairon/shared/schemas/domain.yml /root/.pyenv/versions/${PYTHON_VERSION}/lib/python3.10/site-packages/rasa/shared/utils/schemas/domain.yml
Comment on lines +42 to +43
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Avoid direct modification of installed packages

Copying files directly into the Python packages directory is risky and can break during version updates. This approach bypasses package management and can lead to maintenance issues.

Consider these alternatives:

  1. Create proper patches and apply them during package installation
  2. Fork and maintain your own version of the packages
  3. Use Python's site-packages directory for custom overrides:
-RUN cp kairon/shared/rule_policy.py /root/.pyenv/versions/${PYTHON_VERSION}/lib/python3.10/site-packages/rasa/core/policies/rule_policy.py
-RUN cp kairon/shared/schemas/domain.yml /root/.pyenv/versions/${PYTHON_VERSION}/lib/python3.10/site-packages/rasa/shared/utils/schemas/domain.yml
+# Create a custom package directory
+RUN mkdir -p /app/custom_overrides/rasa/{core/policies,shared/utils/schemas}
+
+# Copy your custom files
+COPY kairon/shared/rule_policy.py /app/custom_overrides/rasa/core/policies/
+COPY kairon/shared/schemas/domain.yml /app/custom_overrides/rasa/shared/utils/schemas/
+
+# Add the custom directory to Python path
+ENV PYTHONPATH=/app/custom_overrides:$PYTHONPATH

Committable suggestion skipped: line range outside the PR's diff.


ENV HF_HOME="/home/cache" SENTENCE_TRANSFORMERS_HOME="/home/cache"

Loading