어떤 기능인가요?
추가하려는 기능에 대해 간결하게 설명해주세요
DAG 파일, docker-compose.yml, Dockerfile 간의 크롬 드라이버 버전 통합 및 호환성 확인
작업 상세 내용
참고할만한 자료(선택)
*** Found local files:
*** * /opt/airflow/logs/dag_id=crawl_and_upload_stock_data_dag/run_id=manual__2025-01-09T06:40:19.857886+00:00/task_id=crawl_stock_data_task/attempt=1.log
[2025-01-09, 15:40:26 +09] {taskinstance.py:1159} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [queued]>
[2025-01-09, 15:40:26 +09] {taskinstance.py:1159} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [queued]>
[2025-01-09, 15:40:26 +09] {taskinstance.py:1361} INFO - Starting attempt 1 of 2
[2025-01-09, 15:40:26 +09] {taskinstance.py:1382} INFO - Executing <Task(PythonOperator): crawl_stock_data_task> on 2025-01-09 06:40:19.857886+00:00
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:57} INFO - Started process 724 to run task
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:84} INFO - Running: ['', 'tasks', 'run', 'crawl_and_upload_stock_data_dag', 'crawl_stock_data_task', 'manual__2025-01-09T06:40:19.857886+00:00', '--job-id', '4', '--raw', '--subdir', 'DAGS_FOLDER/naverFinanceExtract_dag.py', '--cfg-path', '/tmp/tmpdwv23b3e']
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:85} INFO - Job 4: Subtask crawl_stock_data_task
[2025-01-09, 15:40:26 +09] {task_command.py:416} INFO - Running <TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [running]> on host 38332b978bcd
[2025-01-09, 15:40:27 +09] {taskinstance.py:1662} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='' AIRFLOW_CTX_DAG_ID='crawl_and_upload_stock_data_dag' AIRFLOW_CTX_TASK_ID='crawl_stock_data_task' AIRFLOW_CTX_EXECUTION_DATE='2025-01-09T06:40:19.857886+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2025-01-09T06:40:19.857886+00:00'
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - ====== WebDriver manager ======
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - About to download new driver from https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - Driver downloading response is 200
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Driver has been saved in cache [/home//.wdm/drivers/chromedriver/linux64/114.0.5735.90]
[2025-01-09, 15:40:29 +09] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/dags/naverFinanceExtract_dag.py", line 49, in crawl_stock_data
driver = webdriver.Chrome(
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in init
super().init(
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 55, in init
self.service.start()
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 108, in start
self.assert_process_still_running()
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 121, in assert_process_still_running
raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /home/airflow/.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 209, in execute_callable
return self.python_callable(self.op_args, self.op_kwargs)
File "/opt/airflow/dags/naverFinanceExtract_dag.py", line 54, in crawl_stock_data
raise Exception(f"ChromeDriver 초기화 실패: {e}")
Exception: ChromeDriver 초기화 실패: Message: Service /home//.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
[2025-01-09, 15:40:29 +09] {taskinstance.py:1400} INFO - Marking task as UP_FOR_RETRY. dag_id=crawl_and_upload_stock_data_dag, task_id=crawl_stock_data_task, execution_date=20250109T064019, start_date=20250109T064026, end_date=20250109T064029
[2025-01-09, 15:40:29 +09] {standard_task_runner.py:104} ERROR - Failed to execute job 4 for task crawl_stock_data_task (ChromeDriver 초기화 실패: Message: Service /home//.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
; 724)
[2025-01-09, 15:40:29 +09] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2025-01-09, 15:40:29 +09] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check
어떤 기능인가요?
작업 상세 내용
참고할만한 자료(선택)
*** Found local files:
*** * /opt/airflow/logs/dag_id=crawl_and_upload_stock_data_dag/run_id=manual__2025-01-09T06:40:19.857886+00:00/task_id=crawl_stock_data_task/attempt=1.log
[2025-01-09, 15:40:26 +09] {taskinstance.py:1159} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [queued]>
[2025-01-09, 15:40:26 +09] {taskinstance.py:1159} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [queued]>
[2025-01-09, 15:40:26 +09] {taskinstance.py:1361} INFO - Starting attempt 1 of 2
[2025-01-09, 15:40:26 +09] {taskinstance.py:1382} INFO - Executing <Task(PythonOperator): crawl_stock_data_task> on 2025-01-09 06:40:19.857886+00:00
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:57} INFO - Started process 724 to run task
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:84} INFO - Running: ['', 'tasks', 'run', 'crawl_and_upload_stock_data_dag', 'crawl_stock_data_task', 'manual__2025-01-09T06:40:19.857886+00:00', '--job-id', '4', '--raw', '--subdir', 'DAGS_FOLDER/naverFinanceExtract_dag.py', '--cfg-path', '/tmp/tmpdwv23b3e']
[2025-01-09, 15:40:26 +09] {standard_task_runner.py:85} INFO - Job 4: Subtask crawl_stock_data_task
[2025-01-09, 15:40:26 +09] {task_command.py:416} INFO - Running <TaskInstance: crawl_and_upload_stock_data_dag.crawl_stock_data_task manual__2025-01-09T06:40:19.857886+00:00 [running]> on host 38332b978bcd
[2025-01-09, 15:40:27 +09] {taskinstance.py:1662} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='' AIRFLOW_CTX_DAG_ID='crawl_and_upload_stock_data_dag' AIRFLOW_CTX_TASK_ID='crawl_stock_data_task' AIRFLOW_CTX_EXECUTION_DATE='2025-01-09T06:40:19.857886+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2025-01-09T06:40:19.857886+00:00'
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - ====== WebDriver manager ======
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - About to download new driver from https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
[2025-01-09, 15:40:27 +09] {logger.py:11} INFO - Driver downloading response is 200
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Get LATEST chromedriver version for google-chrome
[2025-01-09, 15:40:29 +09] {logger.py:11} INFO - Driver has been saved in cache [/home//.wdm/drivers/chromedriver/linux64/114.0.5735.90]
[2025-01-09, 15:40:29 +09] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/dags/naverFinanceExtract_dag.py", line 49, in crawl_stock_data
driver = webdriver.Chrome(
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in init
super().init(
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 55, in init
self.service.start()
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 108, in start
self.assert_process_still_running()
File "/home/airflow/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 121, in assert_process_still_running
raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service /home/airflow/.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 192, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/operators/python.py", line 209, in execute_callable
return self.python_callable(self.op_args, self.op_kwargs)
File "/opt/airflow/dags/naverFinanceExtract_dag.py", line 54, in crawl_stock_data
raise Exception(f"ChromeDriver 초기화 실패: {e}")
Exception: ChromeDriver 초기화 실패: Message: Service /home//.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
[2025-01-09, 15:40:29 +09] {taskinstance.py:1400} INFO - Marking task as UP_FOR_RETRY. dag_id=crawl_and_upload_stock_data_dag, task_id=crawl_stock_data_task, execution_date=20250109T064019, start_date=20250109T064026, end_date=20250109T064029
[2025-01-09, 15:40:29 +09] {standard_task_runner.py:104} ERROR - Failed to execute job 4 for task crawl_stock_data_task (ChromeDriver 초기화 실패: Message: Service /home//.wdm/drivers/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127
; 724)
[2025-01-09, 15:40:29 +09] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2025-01-09, 15:40:29 +09] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check