DAGS WHERE NOT SHOWING IN AIRFLOW UI

[Prithivi-Krishna-pk]

Hi Community,

We have stucked in problem.

We have installed a new airflow setup in different node and need to migrate all the dags running existing node.

What we did:

we took the tar copy of the dags in current running airflow and moved to new node. after untar. we couldn't able to see all the dags in airflow UI.

We have tried multiple workaround but didn't work. Any idea to resolve this ?

OUR OS : RHEL 8.9 (oopta)
AIRFLOW REPOSITORY : MYSQL

[Prithivi-Krishna-pk]

post db init we are able to see dags but next schedule not running automatic and in scheduler.log we are hettig continue this error msg
airflow 2.2.4 with mysql
celery with 2 worker node
mysql 8.0.23
rhel 8.10

2025-02-14 16:02:01,326] {manager.py:163} INFO - Launched DagFileProcessorManager with pid: 1242023
[2025-02-14 16:02:01,337] {settings.py:55} INFO - Configured default timezone Timezone('UTC')
[2025-02-14 16:02:51,998] {manager.py:331} ERROR - DagFileProcessorManager (PID=1242023) last sent a heartbeat 50.67 seconds ago! Restarting it
[2025-02-14 16:02:52,022] {process_utils.py:120} INFO - Sending Signals.SIGTERM to group 1242023. PIDs of all processes in the group: [1242023]
[2025-02-14 16:02:52,022] {process_utils.py:75} INFO - Sending the signal Signals.SIGTERM to group 1242023
[2025-02-14 16:02:52,235] {process_utils.py:70} INFO - Process psutil.Process(pid=1242023, status='terminated', exitcode=0, started='16:02:01') (1242023) terminated with exit code 0

[potiuk]

Follow https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code -> looks like your DAGs are taking a long time to parse and this is something that best practices explained there are very explicit about. There are docs and examples and ways how you can check and make sure your DAGs are following best practices.

[Prithivi-Krishna-pk]

Hi potiuk,

Thank you, for your support.

At last we managed to rectify the issue. we found out that the mount point attached to the our new node found improper
after unmount and remount the issue rectified automatically.

[bobir01]

@Prithivi-Krishna-pk i am facing the same issue, my dags are visible in initially, then i trig one random dag, all my dags get dissappeared immediately, i am using airflow 2.10.5
in airflow dags list: it is showing i have 89 dags , but in UI only 1 running dag is shown

here my Docker file:

FROM apache/airflow:2.10.5-python3.10

USER root

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

USER airflow

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install --no-cache-dir "apache-airflow==2.10.5" -r requirements.txt

COPY --chown=airflow:root ./dags /opt/airflow/dags

and defaiult docker compose shown in the documentation

the strange this i wanted to upgrade the airflow since my old version was 2.8.0 and it worked perfectly, then i completely removed all docker and volumes and up the new version docker

did anyone have this kind of issue, what do to

[Prithivi-Krishna-pk]

Hi @bobir01,

We are hoisting our airflow in rhel directly. No idea about docker. have did airflow db init ? try once.

[bobir01]

@Prithivi-Krishna-pk thanks for reply didnt help, basically there is one container it applies init and migrations, but no way didnt help