使用 Terraform 销毁 EKS 集群时出现“等待互联网网关分离时出错”

如何解决使用 Terraform 销毁 EKS 集群时出现“等待互联网网关分离时出错”

我这样部署集群：

provider "aws" {
  region = "us-east-1"
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

variable "cluster_name" {
  default = "my-cluster"
}

variable "instance_type" {
  default = "t2.large"
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "~> 1.11"
}

data "aws_availability_zones" "available" {
}


module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.0.0"

  name                 = "k8s-${var.cluster_name}-vpc"
  cidr                 = "172.16.0.0/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["172.16.1.0/24","172.16.2.0/24","172.16.3.0/24"]
  public_subnets       = ["172.16.4.0/24","172.16.5.0/24","172.16.6.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                    = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"           = "1"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "12.2.0"

  cluster_name    = "${var.cluster_name}"
  cluster_version = "1.18"
  subnets         = module.vpc.private_subnets

  vpc_id = module.vpc.vpc_id

  worker_groups = [
   {
     name                          = "worker-group-1"
     instance_type                 = "t3.small"
     additional_userdata           = "echo foo bar"
     asg_desired_capacity          = 2
     #additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
   },{
     name                          = "worker-group-2"
     instance_type                 = "t3.small"
     additional_userdata           = "echo foo bar"
     #additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
     asg_desired_capacity          = 1
   },]
  

  write_kubeconfig   = true
  config_output_path = "./"

  workers_additional_policies = [aws_iam_policy.worker_policy.arn]
}

resource "aws_iam_policy" "worker_policy" {
  name        = "worker-policy-${var.cluster_name}"
  description = "Worker policy for the ALB Ingress"

  policy = file("iam-policy.json")
}

我还通过 Helm 部署了一个 Nginx Ingress 控制器。

部署运行完美，但运行 terraform destroy 时出现此错误：

module.dev_cluster.module.vpc.aws_subnet.public[1]: Still destroying... [id=subnet-0bfa486ee8fd490fc,19m40s elapsed]
module.dev_cluster.module.vpc.aws_subnet.public[1]: Still destroying... [id=subnet-0bfa486ee8fd490fc,19m50s elapsed]
module.dev_cluster.module.vpc.aws_subnet.public[0]: Still destroying... [id=subnet-00179aea4aeb2711e,20m0s elapsed]
module.dev_cluster.module.vpc.aws_subnet.public[1]: Still destroying... [id=subnet-0bfa486ee8fd490fc,20m0s elapsed]

Error: Error waiting for internet gateway (igw-0ede95704f16bf459) to detach: timeout while waiting for state to become 'detached' (last state: 'detaching',timeout: 15m0s)

Error: error deleting subnet (subnet-00179aea4aeb2711e): timeout while waiting for state to become 'destroyed' (last state: 'pending',timeout: 20m0s)

Error: Unauthorized

Error: error deleting subnet (subnet-0bfa486ee8fd490fc): timeout while waiting for state to become 'destroyed' (last state: 'pending',timeout: 20m0s)

我觉得我在某处缺乏依赖，让 Terraform 有权删除所有内容，直到删除结束，但我看不到在哪里。

我希望能够通过 terraform destroy 命令完全销毁我的集群，而无需通过 AWS 控制台手动删除资源。

编辑：如果我不安装 NGINX Ingress 控制器，terraform destroy 命令可以正常工作。只有在我像这样安装 Ingress 后它才会失败：helm install ingress-nginx ingress-nginx/ingress-nginx。所以 Terraform 不知道这个安装。

是否可以通过添加与 Ingress 相关的策略来解决此问题？

编辑：通过 Terraform 而不是命令行部署 Ingress 来修复：

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
    load_config_file       = false
  }
}

resource "helm_release" "ingress-nginx" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx/"
  chart      = "ingress-nginx"
}