如何解决气流DAG失败-错误-HTTP错误404:找不到 DAG PythonOperator-covid_cases.py
我对Apache Airflow不满意。从字面上看才刚刚开始,我遇到了一个错误。我写了我的第一个数据,并且正在调用Python脚本。当我设置它时,它成功运行并开始工作,并且我计划每天运行一次。我今天来检查它,但dag失败并显示错误消息-HTTP错误404:找不到。
对我来说,一切都是新奇的,因此很抱歉,如果这很容易解决,但是我不明白为什么会出现404错误。我尝试重新启动docker,看看这是否是网络服务器问题,但没有运气。
感谢您的帮助
DAG
from airflow.models import DAG
from datetime import datetime,timedelta
from airflow.operators.python_operator import PythonOperator
from covid_cases import covid_data
default_args = {
'owner': 'airflow','start_date': datetime(2020,10,4),'retries': 2,'retry_delay': timedelta(seconds=20)}
dag = DAG(dag_id = 'covid_updates',default_args = default_args,schedule_interval = "0 4 * * *")
t1 = PythonOperator(task_id = 'covid_update',python_callable = covid_data,dag = dag)
t1
PythonOperator-covid_cases.py
def covid_data():
"""
Overview:
---------
Downloads the USA COVID data directly from John Hopkins CSSEGISandData.
This function merges all data from most current date to earliest date (2020-4-11).
Using this function a user can conduct time series analysis in how COVID
increases/decreases in various states.
Output:
-------
One uncleaned .csv file called "usa_covid_cases.csv"
"""
from datetime import datetime,timedelta
import pandas as pd
from urllib.error import HTTPError
# Set starting index
i = 1
# Earliest dataset available on GitHub
start_date = datetime.strptime('2020-4-11','%Y-%m-%d').date()
# Pulling today's date minus 1 day due to delay posting on GitHub
today = datetime.now().date() - timedelta(days=i)
# Setting llist to store dataframe file names
file_names = []
# Looping until date is equal to earlist date = Start Date
while not (start_date.day == today.day and start_date.month == today.month and start_date.year == today.year):
# Extracting variables from current date
day = today.day
month = today.month
year = today.year
# Cleaning and converting values for formatting on GitHub URL link
if day < 10:
day = '0' + str(day)
if month < 10:
month = '0' + str(month)
# Setting variable for each url
url = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports_us/{}-{}-{}.csv'\
.format(month,day,str(year))
try:
# Reading each url as a datafra,df = pd.read_csv(url,error_bad_lines=False)
except HTTPError as e:
# handle the error (print,log,etc)
continue
finally:
# Code moved here to prevent an endless loop
# Subtracting the new index to increase 1 less day from the current date
today = datetime.now().date() - timedelta(days=i)
# Saving each dataframe into the empty list
file_names.append(df)
# Increasing the index by 1
i += 1
# Once while loop ends - concat all the files into a single dataframe
new_df = pd.concat(file_names)
# Save output into new csv file
new_df.to_csv('usa_covid_cases.csv')
解决方法
从日志中,当您尝试从URL读取数据并且该URL不存在时会发生错误。
pd.read_csv(url,error_bad_lines=False) #Line 50 of covid_data.py
,
我刚刚看过GitHub repo,最早的数据集来自 2020年12月12日。
为防止DAG无法从可能的已删除数据集中失败,您可以将pd.read_csv()
包装在try except
块中,如下所示:
# Import the HTTPError (error from you screenshot)
from urllib.error import HTTPError
...
try:
# Reading each url as a datafra,df = pd.read_csv(url,error_bad_lines=False)
except HTTPError as e:
# handle the error (print,log,etc)
continue
finally:
# Code moved here to prevent an endless loop
# Subtracting the new index to increase 1 less day from the current date
today = datetime.now().date() - timedelta(days=i)
# Saving each dataframe into the empty list
file_names.append(df)
# Increasing the index by 1
i += 1
...
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。