Kubernetes上的气流：Errno 13-权限被拒绝：'/ opt / airflow / logs / scheduler

分类：编程问答

如何解决Kubernetes上的气流：Errno 13-权限被拒绝：'/ opt / airflow / logs / scheduler

我正在stable helm chart的Kubernetes上运行Airflow。我正在AWS环境中运行它。无论是否安装任何外部卷来存储日志，都存在此错误。我试图将[logs]部分的配置设置为指向我创建的EFS卷。 PV通过PVC进行安装，但是由于以下错误，我的容器崩溃了（调度程序和Web）：

*** executing Airflow initdb...
Unable to load the config,contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/logging/config.py",line 565,in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.6/logging/config.py",line 738,in configure_handler
    result = factory(**kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py",line 50,in __init__
    os.makedirs(self._get_log_directory())
  File "/usr/local/lib/python3.6/os.py",line 220,in makedirs
    mkdir(name,mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow",line 25,in <module>
    from airflow.configuration import conf
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py",line 47,in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py",line 374,in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py",line 68,in configure_logging
    raise e
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py",line 63,in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.6/logging/config.py",line 802,in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.6/logging/config.py",line 573,in configure
    '%r: %s' % (name,e))
ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2020-08-20'

持久体积（不是通过稳定/气流图手动创建的）

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"efs-pv"},"spec":{"accessModes":["ReadWriteMany"],"capacity":{"storage":"5Gi"},"csi":{"driver":"efs.csi.aws.com","volumeHandle":"fs-e476a166"},"persistentVolumeReclaimPolicy":"Retain","storageClassName":"efs-sc","volumeMode":"Filesystem"}}
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2020-08-20T15:47:21Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: efs-pv
  resourceVersion: "49476860"
  selfLink: /api/v1/persistentvolumes/efs-pv
  uid: 45d9f5ea-66c1-493e-a2f5-03e17f397747
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: efs-claim
    namespace: airflow
    resourceVersion: "49476857"
    uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-e476a166
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  volumeMode: Filesystem
status:
  phase: Bound

日志的持久体积声明（不是通过稳定/气流图手动创建的）：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","name":"efs-claim","namespace":"airflow"},"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"efs-sc"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2020-08-20T15:47:46Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: efs-claim
  namespace: airflow
  resourceVersion: "49476866"
  selfLink: /api/v1/namespaces/airflow/persistentvolumeclaims/efs-claim
  uid: 354103ea-f8a9-47f1-a7cf-8f449f9a2e8b
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: efs-sc
  volumeMode: Filesystem
  volumeName: efs-pv
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  phase: Bound

我下面的values.yaml：

airflow:
  image:
    repository: apache/airflow
    tag: 1.10.10-python3.6
    ## values: Always or IfNotPresent
    pullPolicy: IfNotPresent
    pullSecret: ""

  executor: KubernetesExecutor

  fernetKey: "XXXXXXXXXHIVb8jK6lfmSAvx4mO6Arehnc="

  config:
    AIRFLOW__CORE__REMOTE_LOGGING: "True"
    AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://mybucket/airflow/logs"
    AIRFLOW__CORE__REMOTE_LOG_CONN_ID: "MyS3Conn"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY: "apache/airflow"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG: "1.10.10-python3.6"
    AIRFLOW__KUBERNETES__WORKER_CONTAINER_IMAGE_PULL_POLICY: "IfNotPresent"
    AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE: "10"
    AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "efs-claim"
    AIRFLOW__KUBERNETES__GIT_REPO: "git@github.com:org/myrepo.git"
    AIRFLOW__KUBERNETES__GIT_BRANCH: "develop"
    AIRFLOW__KUBERNETES__GIT_DAGS_FOLDER_MOUNT_POINT: "/opt/airflow/dags"
    AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH: "repo/"
    AIRFLOW__KUBERNETES__GIT_SSH_KEY_SECRET_NAME: "airflow-git-keys"
    AIRFLOW__KUBERNETES__NAMESPACE: "airflow"
    AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "True"
    AIRFLOW__KUBERNETES__RUN_AS_USER: "50000"
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "60"
    AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME: "airflow"

  podAnnotations: {}
  extraEnv: []
  extraConfigmapMounts: []
  extraContainers: []
  extraPipPackages: []
  extraVolumeMounts: []
  extraVolumes: []
scheduler:
  resources: {}
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  podDisruptionBudget:
    enabled: true
    maxUnavailable: "100%"
    minAvailable: ""
  connections:
    - id: MyS3Conn
      type: aws
      extra: |
        {
        "aws_access_key_id": "XXXXXXXXX","aws_secret_access_key": "XXXXXXXX","region_name":"us-west-1"
        }

  refreshConnections: true
  variables: |
    {}

  pools: |
    {}

  numRuns: -1
  initdb: true
  preinitdb: false
  initialStartupDelay: 0
  extraInitContainers: []
web:
  resources: {}
  replicas: 1
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  service:
    annotations: {}
    sessionAffinity: "None"
    sessionAffinityConfig: {}
    type: ClusterIP
    externalPort: 8080
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePort:
      http: ""

  baseUrl: "http://localhost:8080"
  serializeDAGs: false
  extraPipPackages: []
  initialStartupDelay: 0
  minReadySeconds: 5
  readinessProbe:
    enabled: false
    scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3

  livenessProbe:
    enabled: true
    scheme: HTTP
    initialDelaySeconds: 300
    periodSeconds: 30
    timeoutSeconds: 3
    successThreshold: 1
    failureThreshold: 2

  secretsDir: /var/airflow/secrets
  secrets: []
  secretsMap:

workers:
  enabled: false
  resources: {}
  replicas: 1
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  autoscaling:
    enabled: false
    maxReplicas: 2
    metrics: []
  initialStartupDelay: 0
  celery:
    instances: 1
    gracefullTermination: false
    gracefullTerminationPeriod: 600
  terminationPeriod: 60
  secretsDir: /var/airflow/secrets
  secrets: []
  secretsMap:

flower:
  enabled: false
  resources: {}
  nodeSelector: {}
  affinity: {}
  tolerations: []
  labels: {}
  podLabels: {}
  annotations: {}
  podAnnotations: {}
  basicAuthSecret: ""
  basicAuthSecretKey: ""
  urlPrefix: ""
  service:
    annotations: {}
    type: ClusterIP
    externalPort: 5555
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    nodePort:
      http: ""

  initialStartupDelay: 0
  extraConfigmapMounts: []

logs:
  path: /opt/airflow/logs
  persistence:
    enabled: true
    existingClaim: efs-claim
    subPath: ""
    storageClass: efs-sc
    accessMode: ReadWriteMany
    size: 1Gi
dags:
  path: /opt/airflow/dags
  doNotPickle: false
  installRequirements: false
  persistence:
    enabled: false
    existingClaim: ""
    subPath: ""
    storageClass: ""
    accessMode: ReadOnlyMany
    size: 1Gi
  git:
    url: git@github.com:org/myrepo.git
    ref: develop
    secret: airflow-git-keys
    sshKeyscan: false
    privateKeyName: id_rsa
    repoHost: github.com
    repoPort: 22
    gitSync:
      enabled: true
      resources: {}
      image:
        repository: alpine/git
        tag: latest
        pullPolicy: Always
      refreshTime: 60
  initContainer:
    enabled: false
    resources: {}
    image:
      repository: alpine/git
      tag: latest
      pullPolicy: Always
    mountPath: "/dags"
    syncSubPath: ""
ingress:
  enabled: false
  web:
    annotations: {}
    path: ""
    host: ""
    livenessPath: ""
    tls:
      enabled: false
      secretName: ""
    precedingPaths: []
    succeedingPaths: []
  flower:
    annotations: {}
    path: ""
    host: ""
    livenessPath: ""
    tls:
      enabled: false
      secretName: ""
rbac:
  create: true
serviceAccount:
  create: true
  name: ""
  annotations: {}
extraManifests: []

postgresql:

  enabled: true
  postgresqlDatabase: airflow
  postgresqlUsername: postgres
  postgresqlPassword: airflow
  existingSecret: ""
  existingSecretKey: "postgresql-password"
  persistence:
    enabled: true
    storageClass: ""
    accessModes:
      - ReadWriteOnce
    size: 5Gi

externalDatabase:
  type: postgres
  host: localhost
  port: 5432
  database: airflow
  user: airflow
  passwordSecret: ""
  passwordSecretKey: "postgresql-password"

redis:
  enabled: false
  password: airflow
  existingSecret: ""
  existingSecretKey: "redis-password"
  cluster:
    enabled: false
    slaveCount: 1
  master:
    resources: {}
    persistence:
      enabled: false
      storageClass: ""
      accessModes:
        - ReadWriteOnce

      size: 8Gi

  slave:
    resources: {}
    persistence:
      enabled: false
      storageClass: ""
      accessModes:
        - ReadWriteOnce

      size: 8Gi

externalRedis:
  host: localhost
  port: 6379
  databaseNumber: 1
  passwordSecret: ""
  passwordSecretKey: "redis-password"

serviceMonitor:
  enabled: false
  selector:
    prometheus: kube-prometheus
  path: /admin/metrics
  interval: "30s"

prometheusRule:
  enabled: false
  additionalLabels: {}
  groups: []

如果有人知道如何解决权限错误，我不太确定该怎么做。

解决方法

您可以使用带有调度程序的 extraInitContainers 来更改权限，就像这样。

 extraInitContainers:
    - name: volume-logs
      image: busybox
      command: ["sh","-c","chown -R 50000:50000 /opt/airflow/logs/"]
      volumeMounts:
        - mountPath: /opt/airflow/logs/
          name: logs-data

这会改变挂载点的权限。

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

导入项目后报错问题
依赖报错 idea导入项目后依赖报错，解决方案：https://blog.csdn.net/weixin_42420249/article/details/81191861 依赖版本报错：更换其他版本 无法下载依赖可参考：https://blog.csdn.net/weixin_42628809/a
idea不能识别yaml文件
使用mybatis plus常见错误
错误1：代码生成器依赖和mybatis依赖冲突 启动项目时报错如下 2021-12-03 13:33:33.927 ERROR 7228 [ main] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPL
gradle常见问题与错误
错误1：gradle项目控制台输出为乱码 # 解决方案：https://blog.csdn.net/weixin_43501566/article/details/112482302 # 在gradle-wrapper.properties 添加以下内容 org.gradle.jvmargs=-Df
Mybatis Plus传入参数0不起作用
错误还原：在查询的过程中，传入的workType为0时，该条件不起作用 &lt;select id=&quot;xxx&quot;&gt; SELECT di.id, di.name, di.work_type, di.updated... &lt;where&gt; &lt;if test=&qu
linux中make编译源码包失败
报错如下，gcc版本太低 ^ server.c:5346:31: 错误：‘struct redisServer’没有名为‘server_cpulist’的成员 redisSetCpuAffinity(server.server_cpulist); ^ server.c: 在函数‘hasActiveC
微服务启动错误：Command line is too long. Shorten command line for
解决方案1 1、改项目中.idea/workspace.xml配置文件，增加dynamic.classpath参数 2、搜索PropertiesComponent，添加如下 &lt;property name=&quot;dynamic.classpath&quot; value=&quot;tru
vue常见错误
删除根组件app.vue中的默认代码后报错：Module Error (from ./node_modules/eslint-loader/index.js): 解决方案：关闭ESlint代码检测，在项目根目录创建vue.config.js，在文件中添加 module.exports = { lin
spark常见错误
查看spark默认的python版本 [root@master day27]# pyspark /home/software/spark-2.3.4-bin-hadoop2.7/conf/spark-env.sh: line 2: /usr/local/hadoop/bin/hadoop: No s
matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot