AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Airflow kubernetes executor logs8/18/2023 Here are the config variables I am setting, note these env variables get set in my webserver/scheduler and executor pods: #. So it seems that when I cat from within the executor pod, it is causing the logs to be flushed in some way, so that it is available everywhere. Weirdly, on the other hand, when I exec into the executor pod as it is running, and I cat the exact same log file in the shared EFS, I am able to see the correct logs up until that point in the job, and when I immediately cat from the scheduler or check the UI, I can also see the logs up until that point. Once the executor finishes however, I can see the full logs both when I cat the file and in the airflow UI. However, when I cat the log file, I do not see the logs as long as the executor is still running. log file of the currently running job because of the shared EFS. When I exec into my scheduler pod, while an executor pod is running, I am able to see the. However, I want to be able to see the logs of my jobs as they are running for long running ones. I am also using s3 remote logging.Īll the logging is working perfectly fine after a dag completes. I have started using KubernetesExecutor and I have set up a PV/PVC with an AWS EFS to store logs for my dags.
0 Comments
Read More
Leave a Reply. |