Operator3-设计一个operator

分类：菜鸟笔记作者：对你无可奈何

背景：

前置知识Operator-1初识Operator，Operator-2从pod开始简单operator。

先拿一个个人的工作环境来设计吧，应用有十多个微服务，恩各种类型job deployment statefulset service ingress pv pvc configmap这些资源准备模仿eck:https://github.com/elastic/cloud-on-k8s来设计！

image.png

image.png

image.png

创建一个自己的operator

goland 创建项目

image.png

image.png

命名规则

关于应用命名一下吧：

就拿月份来作应用名称吧：

应用1：Jan

应用2：feb

应用3：mar

应用4：apr

应用5：may

也没有想好具体的代表什么，下面了边作边看......

kubebuilder init

[zhangpeng@zhangpeng develop-operator]$ kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"

image.png

开启支持多接口组

模仿目录结构：

image.png

设置multigroup=true，忘了这是在哪个地方搜到的了，应该是思否一篇文章

[zhangpeng@zhangpeng develop-operator]$ kubebuilder edit --multigroup=true
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group jan --version v1 --kind Jan

image.png
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group feb --version v1 --kind Feb

image.png
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group mar --version v1 --kind Mar
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group apr --version v1 --kind Apr
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group may --version v1 --kind May

image.png

从Jan开始

jan应用为一个deployment应用，参照https://www.qikqiak.com/post/k8s-operator-101/,创建一个deployment并与Operator-2从pod开始简单operator中对比一下pod 与deployment的区别！

注意：以下代码都是抄写自阳明大佬，些许修改......，比如有个& 还有关于service的修改

定义jan_type

apis/jan/v1/jan_type.go

type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	Size      *int32                      `json:"size"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
}

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	appsv1.DeploymentStatus `json:",inline"`
}

image.png

make install

make install 失败，继续拆解命令：

[zhangpeng@zhangpeng develop-operator]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

继续发布到集群：

[zhangpeng@zhangpeng develop-operator]$ kustomize build config/crd | kubectl apply -f -

不知道有没有单独发布的办法，一整都一起发布了......

image.png
[zhangpeng@zhangpeng develop-operator]$ kubectl describe crd jans.jan.zhangpeng.com

image.png

describe crd的内容与config/crd/bases/jan.zhangpeng.com_jans.yaml 内容是一样的，强调一下......

创建deployment pod service的方法

对应文件都偷懒了，直接放在controllers/jan目录下了：

image.png

cat jan_helper.go

package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Size,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
	}

}
func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}
func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  corev1.ServiceTypeNodePort,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

jan_controller.go Reconcile

基本阳明大佬的博客抄来的,Reconcile调谐函数：

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"reflect"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}

	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}
	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}
		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)
		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}

		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}

	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}

	if !reflect.DeepEqual(instance.Spec, oldspec) {
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}

		return reconcile.Result{}, nil
	}

	return reconcile.Result{}, nil

}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

强调一下：

json.Unmarshal

image.png

https://www.qikqiak.com/post/k8s-operator-101/#%E9%A1%B9%E7%9B%AE%E7%BB%93%E6%9E%84是这样写的，but maker run就报错了，看了一下别人有人写了&就加了一下

image.png

make run and test

image.png

测试yaml就用config/samples/jan_v1_jan.yaml去测试了：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  size: 2
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30002
[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created

image.png

修改副本数为3：

image.png

继续改造

修改Service Type

en 对比前一节的pod operator我想输出更多的内容，get Jan也想输出数量：继续改造(还有Jan服务可以输入Type我不喜欢nodePort的方式)

image.png
[zhangpeng@zhangpeng develop-operator]$ make install
[zhangpeng@zhangpeng develop-operator]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[zhangpeng@zhangpeng develop-operator]$  kustomize build config/crd | kubectl apply -f -

image.png

kubecelt delete -f config/samples/jan_v1_jan.yaml

重新编辑文件如下：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  size: 3
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
//     nodePort: 30002
    type: ClusterIP

controllers/jan/jan_helper.go NewService修改Type如下：Type: app.Spec.Type,

image.png

make run and kubectl apply

[zhangpeng@zhangpeng develop-operator]$ kubectl delete -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com "jan-sample" deleted
[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created
[zhangpeng@zhangpeng develop-operator]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
jan-sample   ClusterIP   10.99.27.123   <none>        80/TCP    1s
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   19d
[zhangpeng@zhangpeng develop-operator]$ kubectl get jan
NAME         AGE
jan-sample   5m21s

image.png

继续模仿

eck 有更多的输出阿 咱们的输出现在只有AGE......想输出更多

image.png

注意：comon我还是没有用到，创建了就创建了吧，后面看看还是否用的到！

人家eck有个公用的comon?咱也搞一个

image.png
kubebuilder create api --group common --version v1 --kind Common

image.png

继续狗一下

image.png

这里懒得写了 网上搜到一个博客：https://qingwave.github.io/how-to-write-a-k8s-operator就按照他写的改一下了：

apis/jan/v1/jan_type.go

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// JanSpec defines the desired state of Jan
type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	//+kubebuilder:default:=1
	//+kubebuilder:validation:Minimum:=1
	Replicas  *int32                      `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
	Type      corev1.ServiceType          `json:"type,omitempty"`
}

const (
	Running  = "Running"
	Pending  = "Pending"
	NotReady = "NotReady"
	Failed   = "Failed"
)

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	// Phase is the phase of guestbook
	Phase string `json:"phase,omitempty"`
	// replicas is the number of Pods created by the StatefulSet controller.
	Replicas int32 `json:"replicas"`

	// readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.
	ReadyReplicas int32 `json:"readyReplicas"`

	// LabelSelector is label selectors for query over pods that should match the replica count used by HPA.
	LabelSelector string `json:"labelSelector,omitempty"`
}

//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.labelSelector
//+kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase",description="The phase of game."
//+kubebuilder:printcolumn:name="DESIRED",type="integer",JSONPath=".spec.replicas",description="The desired number of pods."
//+kubebuilder:printcolumn:name="CURRENT",type="integer",JSONPath=".status.replicas",description="The number of currently all pods."
//+kubebuilder:printcolumn:name="READY",type="integer",JSONPath=".status.readyReplicas",description="The number of pods ready."
//+kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp",description="CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC."
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Jan is the Schema for the jans API
type Jan struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   JanSpec   `json:"spec,omitempty"`
	Status JanStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// JanList contains a list of Jan
type JanList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Jan `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Jan{}, &JanList{})
}

make install

image.png

controllers/jan/jan_helper.go

其实我就修改了一下type，我可不想一直写nodeport......一般都司clasterip

package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Replicas,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
		Status: appv1.DeploymentStatus{},
	}

}

func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}

func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  app.Spec.Type,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

jan_controller.go

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"reflect"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps,resources=deployments/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=networking,resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	defer utilruntime.HandleCrash()
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}
	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}

	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}

		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}
	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}
	if !reflect.DeepEqual(instance.Spec, oldspec) {
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}
		return reconcile.Result{}, nil
	}
	newStatus := janv1.JanStatus{
		Replicas:      *instance.Spec.Replicas,
		ReadyReplicas: instance.Status.Replicas,
	}

	if newStatus.Replicas == newStatus.ReadyReplicas {
		newStatus.Phase = janv1.Running
	} else {
		newStatus.Phase = janv1.NotReady
	}
	if !reflect.DeepEqual(instance.Status, newStatus) {
		instance.Status = newStatus
		log.FromContext(ctx).Info("update game status", "name", instance.Name)
		err := r.Client.Status().Update(ctx, instance)
		return reconcile.Result{}, err

	}
	return reconcile.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

偷懒抄来的：

image.png

make run :

清空原来的Jan应用

[zhangpeng@zhangpeng develop-operator]$ kubectl delete jan jan-sample
jan.jan.zhangpeng.com "jan-sample" deleted
apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 2
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png
[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created
[zhangpeng@zhangpeng develop-operator]$ kubectl get jan
NAME         PHASE     DESIRED   CURRENT   READY   AGE
jan-sample   Running   2         2         2       1s

image.png

问题又来了：

image.png

这样还是不能更新状态status阿？没有什么实际意义。所以这个哥们写的也是一个纯看的demo....还是找个成熟的应用去看吧！

image.png

偷懒秘籍：

更新资源这里我是不是可以重新生成一下Annotations？那旧的数据不就变成新的了？

image.png

尝试一下：

image.png

自己骗自己算是成功了

image.png

但是还是有问题：

修改image

image.png

image.png

image.png

pod更新中这样是否能接受呢？

image.png

看了一眼deployment也是这样，我接受了.......

最终代码

上面感觉还是缺少点东西，什么呢？ingress我是否也可以封过来？

增加ingress相关字段

apis/jan/v1/jan_type.go

image.png
/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// JanSpec defines the desired state of Jan
type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	//+kubebuilder:default:=1
	//+kubebuilder:validation:Minimum:=1
	Replicas  *int32                      `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
	Type      corev1.ServiceType          `json:"type,omitempty"`
	Host      string                      `json:"host,omitempty"`
}

const (
	Running  = "Running"
	Pending  = "Pending"
	NotReady = "NotReady"
	Failed   = "Failed"
)

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	// Phase is the phase of guestbook
	Phase string `json:"phase,omitempty"`
	// replicas is the number of Pods created by the StatefulSet controller.
	Replicas int32 `json:"replicas"`

	// readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.
	ReadyReplicas int32 `json:"readyReplicas"`

	// LabelSelector is label selectors for query over pods that should match the replica count used by HPA.
	LabelSelector string `json:"labelSelector,omitempty"`
}

//+kubebuilder:printcolumn:name="Host",type="string",JSONPath=".spec.host",description="The host address."
//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.labelSelector
//+kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase",description="The phase of game."
//+kubebuilder:printcolumn:name="DESIRED",type="integer",JSONPath=".spec.replicas",description="The desired number of pods."
//+kubebuilder:printcolumn:name="CURRENT",type="integer",JSONPath=".status.replicas",description="The number of currently all pods."
//+kubebuilder:printcolumn:name="READY",type="integer",JSONPath=".status.readyReplicas",description="The number of pods ready."
//+kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp",description="CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC."
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Jan is the Schema for the jans API
type Jan struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   JanSpec   `json:"spec,omitempty"`
	Status JanStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// JanList contains a list of Jan
type JanList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Jan `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Jan{}, &JanList{})
}

创建NewIngress方法

jan_helper.go 中，先写死这个了port了

image.png
package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	v1 "k8s.io/api/networking/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Replicas,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
		Status: appv1.DeploymentStatus{},
	}

}

func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}

func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  app.Spec.Type,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

const (
	port = 80
)

func NewIngress(app *janv1.Jan) *v1.Ingress {
	pathType := v1.PathTypePrefix
	return &v1.Ingress{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Ingress",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
		},
		Spec: v1.IngressSpec{
			IngressClassName: nil,
			Rules: []v1.IngressRule{
				{
					Host: app.Spec.Host,
					IngressRuleValue: v1.IngressRuleValue{
						HTTP: &v1.HTTPIngressRuleValue{
							Paths: []v1.HTTPIngressPath{{
								Path:     "/",
								PathType: &pathType,
								Backend: v1.IngressBackend{
									Service: &v1.IngressServiceBackend{
										Name: app.Name,
										Port: v1.ServiceBackendPort{
											Number: int32(port),
										},
									},
									Resource: nil,
								},
							},
							}}},
				},
			},
		},
	}
}

jan_controller.go中增加ingress相关

image.png
/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"reflect"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps,resources=deployments/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=networking,resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	defer utilruntime.HandleCrash()
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}
	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}

	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}

		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 创建 Ingress
		ingress := NewIngress(instance)
		if err := r.Client.Create(context.TODO(), ingress); err != nil {
			return reconcile.Result{}, err
		}
		// 4. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}
	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}

	if !reflect.DeepEqual(instance.Spec, oldspec) {
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}
		return reconcile.Result{}, nil
	}
	newStatus := janv1.JanStatus{
		Replicas:      *instance.Spec.Replicas,
		ReadyReplicas: instance.Status.Replicas,
	}

	if newStatus.Replicas == newStatus.ReadyReplicas {
		newStatus.Phase = janv1.Running
	} else {
		newStatus.Phase = janv1.NotReady
	}
	if !reflect.DeepEqual(instance.Status, newStatus) {
		instance.Status = newStatus
		log.FromContext(ctx).Info("update game status", "name", instance.Name)
		err = r.Client.Status().Update(ctx, instance)
		if err != nil {
			return reconcile.Result{}, err
		}
	}
	return reconcile.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

实验一下：

make install make run

image.png

删除清空jan应用

[zhangpeng@zhangpeng develop-operator]$ kubectl delete -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com "jan-sample" deleted

修改config/samples/jan_v1_jan.yaml 如下：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 3
  image: nginx:1.17.6
  host: www.zhangpeng.com
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png

接着修改config/samples/jan_v1_jan.yaml 副本数与host

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 2
  image: nginx:1.17.6
  host: www1.zhangpeng.com
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png

基本实现了第一步的需求了！

总结一下：

  1. operator要解决的是什么 自己还是没有搞明确，也没有想好怎么去设计一个operator。只是简单的实现了一些基本的功能，还没有体会到更多的便利性。
  2. 本来想照着eck写，但是对我这种初学者还是有点难，一步一步 去完善写吧......

原文地址：https://cloud.tencent.com/developer/article/2059806

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

如何学习编程？新手菜鸟如何学编程？
学习编程是顺着互联网的发展潮流，是一件好事。新手如何学习编程？其实不难，不过在学习编程之前你得先了解你的目的是什么？这个很重要，因为目的决定你的发展方向、决定你的发展速度。
it行业都是什么工作？it行业主要做什么？
IT行业是什么工作做什么？IT行业的工作有：产品策划类、页面设计类、前端与移动、开发与测试、营销推广类、数据运营类、运营维护类、游戏相关类等，根据不同的分类下面有细分了不同的岗位。
女生学Java好就业吗？女生适合学Java编程吗？
女生学Java好就业吗？女生适合学Java编程吗？目前有不少女生学习Java开发，但要结合自身的情况，先了解自己适不适合去学习Java，不要盲目的选择不适合自己的Java培训班进行学习。只要肯下功夫钻研，多看、多想、多练
mysql-connect-confused
Can’t connect to local MySQL server through socket \'/var/lib/mysql/mysql.sock问题 1.进入mysql路径
oracle基本命令
oracle基本命令 一、登录操作 1.管理员登录 # 管理员登录 sqlplus / as sysdba 2.普通用户登录
shell判断服务是否启动
一、背景 因为项目中需要通北京网络，所以需要连vpn，但是服务器有时候会断掉，所以写个shell脚本每五分钟去判断是否连接，于是就有下面的shell脚本。
SQL BETWEEN 操作符
BETWEEN 操作符选取介于两个值之间的数据范围内的值。这些值可以是数值、文本或者日期。
​​windows上传ipa到开发者中心app store的方法
假如你已经使用过苹果开发者中心上架app，你肯定知道在苹果开发者中心的web界面，无法直接提交ipa文件，而是需要使用第三方工具，将ipa文件上传到构建版本，开...
SQL 入门篇之什么是别名？
下面的 SQL 语句指定了两个别名，一个是 name 列的别名，一个是 country 列的别名。**提示：**如果列名称包含空格，要求使用双引号或方括号：
windows上传ipa文件到app store的方法​
在使用H5混合开发的app打包后，需要将ipa文件上传到appstore进行发布，就需要去苹果开发者中心进行发布。​
SQL AND & OR 运算符
+----+--------------+---------------------------+-------+---------+
C++ 数组
数组的声明并不是声明一个个单独的变量，比如 number0、number1、...、number99，而是声明一个数组变量，比如 numbers，然后使用 nu...
最全iOS 应用上架流程提交到AppStore
第一步：到appuploader官网下载辅助工具和iCloud驱动，使用前面创建的AppID登录。
SQL ALTER TABLE 语句
如需删除表中的列，请使用下面的语法（请注意，某些数据库系统不允许这种在数据库表中删除列的方式）：
dism操作巨慢
前不久在制作win11pe，制作了一版，1.26GB，太大了，不满意，想再裁剪下，发现这次dism mount正常，commit或discard巨慢，以前都很快...
三种方法卸载Symantec Endpoint Protection 赛门铁克
赛门铁克各个版本概览：https://knowledge.broadcom.com/external/article?legacyId=tech163829
python3.6、python3.10用什么版本的pip
实测Python 3.6.6用pip 21.3.1，再高就报错了，Python 3.10.7用pip 22.3.1是可以的
博通和诺顿
Broadcom Corporation （博通公司，股票代号AVGO）是全球领先的有线和无线通信半导体公司。其产品实现向家庭、 办公室和移动环境以及在这些环境...
高版本c4d在低版本windows系统上打开后不显示窗格
发现个问题，server2016上安装了c4d这些版本，低版本的正常显示窗格，但红色圈出的高版本c4d打开后不显示窗格，
基础网络windows不要使用TAT、如何卸载TAT
TAT：https://cloud.tencent.com/document/product/1340
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot