*** Log file does not exist: Missing Airflow Logs in Docker Desktop (Windows 10 Pro)
Troubleshooting Airflow in Docker Desktop; Note(s) to myself.
Airflow DAG run was successful. But when I tried to check the log, I got the following error message, Docker Desktop (run - not as adminstrator):
*** Log file does not exist: /opt/airflow/logs/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:31:30.529636+00:00/1.log
*** Fetching from: http://064fb98cfc4e:8793/log/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:31:30.529636+00:00/1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://064fb98cfc4e:8793/log/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:31:30.529636+00:00/1.log
For more information check: https://httpstatuses.com/403ls into the /opt/airflow/logs folder also shows the folder is empty.
downloaded file is also not there.
Docker desktop run as admin:
*** Log file does not exist: /opt/airflow/logs/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:40:28.860222+00:00/1.log
*** Fetching from: http://bde2d5de6902:8793/log/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:40:28.860222+00:00/1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://bde2d5de6902:8793/log/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:40:28.860222+00:00/1.log
For more information check: https://httpstatuses.com/403But after browsing to the location, the file is there:
airflow@bde2d5de6902:/opt/airflow/logs/a_MMM_HPA_DAG/fhfa_uad_download/2023-01-06T12:40:28.860222+00:00$ cat 1.log
[2023-01-06 12:40:32,264] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: a_MMM_HPA_DAG.fhfa_uad_download manual__2023-01-06T12:40:28.860222+00:00 [queued]>
[2023-01-06 12:40:32,313] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: a_MMM_HPA_DAG.fhfa_uad_download manual__2023-01-06T12:40:28.860222+00:00 [queued]>
[2023-01-06 12:40:32,314] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2023-01-06 12:40:32,314] {taskinstance.py:1242} INFO - Starting attempt 1 of 1
[2023-01-06 12:40:32,314] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2023-01-06 12:40:32,336] {taskinstance.py:1262} INFO - Executing <Task(PythonOperator): fhfa_uad_download> on 2023-01-06 12:40:28.860222+00:00
[2023-01-06 12:40:32,347] {standard_task_runner.py:52} INFO - Started process 126 to run task
[2023-01-06 12:40:32,354] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'a_MMM_HPA_DAG', 'fhfa_uad_download', 'manual__2023-01-06T12:40:28.860222+00:00', '--job-id', '57', '--raw', '--subdir', 'DAGS_FOLDER/HPI_dag.py', '--cfg-path', '/tmp/tmppnt4ryti', '--error-file', '/tmp/tmp_xnh3l37']
[2023-01-06 12:40:32,355] {standard_task_runner.py:77} INFO - Job 57: Subtask fhfa_uad_download
[2023-01-06 12:40:32,427] {logging_mixin.py:109} INFO - Running <TaskInstance: a_MMM_HPA_DAG.fhfa_uad_download manual__2023-01-06T12:40:28.860222+00:00 [running]> on host bde2d5de6902
[2023-01-06 12:40:32,497] {taskinstance.py:1429} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=Mahdi Moosa
AIRFLOW_CTX_DAG_ID=a_MMM_HPA_DAG
AIRFLOW_CTX_TASK_ID=fhfa_uad_download
AIRFLOW_CTX_EXECUTION_DATE=2023-01-06T12:40:28.860222+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-01-06T12:40:28.860222+00:00
[2023-01-06 12:40:35,794] {HPI_dag.py:33} INFO - To be saved as data/raw_data/FHFA-UAD/UADAggs_tract.zip. Total size to be written is: 72.02 MB
[2023-01-06 12:40:54,515] {HPI_dag.py:41} INFO - File download succesful for UADAggs_tract.zip
[2023-01-06 12:40:54,516] {python.py:152} INFO - Done. Returned value was: None
[2023-01-06 12:40:54,534] {taskinstance.py:1280} INFO - Marking task as SUCCESS. dag_id=a_MMM_HPA_DAG, task_id=fhfa_uad_download, execution_date=20230106T124028, start_date=20230106T124032, end_date=20230106T124054
[2023-01-06 12:40:54,586] {local_task_job.py:154} INFO - Task exited with return code 0
[2023-01-06 12:40:54,631] {local_task_job.py:264} INFO - 1 downstream tasks scheduled from follow-on schedule checkNote: Simply running “docker-compose up” in a command shell run as admin did not solve the problem. Docker desktop needs to run as admin.
Conclusion: Run docker desktop as admin. It is likely that user-privilege descrepancy between Windows OS and Linux OS might be causing the problem.
How to check the location manually:
Step 1: Get the Container ID. This can be obtained the following command:
docker container lsStep 2: Connect to the desired docker container. Command:
docker exec -t -i bde2d5de6902 /bin/bash
Replace "bde2d5de6902" with the container id from step 1.System details:
airflow version
2.2.1
docker version
Client:
Cloud integration: v1.0.29
Version: 20.10.21
API version: 1.41
Go version: go1.18.7
Git commit: baeda1f
Built: Tue Oct 25 18:08:16 2022
OS/Arch: windows/amd64
Context: default
Experimental: true
Server: Docker Desktop 4.15.0 (93002)
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 3056208
Built: Tue Oct 25 18:00:19 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.10
GitCommit: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0