Kubernetes
开源文化 ThingsBoard 开源中间件 Kubernetes DevOps KubeEdge EdgeX Foundry Node-RED
Documentation > Spring Cloud > Helm部署方案

On this page

Helm部署方案

一、概述

1.微服务部署方案

1.架构图

2.部署图

2.微服务部署规划

1.微服务部署方式

微服务 部署方式 实例数 备注
Ingress NodePort    
Eureka StatefulSet 3 有状态服务
Gateway Deployment 3  
Zipkin Deployment 3  
XXL-JOB Deployment 3  
Sentinel Deployment 1 只能部署1个实例
msa-deploy-producer Deployment 3  
msa-deploy-consumer Deployment 3  
msa-deploy-job Deployment 3  

2.微服务功能

微服务 功能 备注
Gateway 服务接口通过网关调用  
msa-deploy-producer 实现日志、调用链、Prometheus  
msa-deploy-consumer 实现日志、调用链、Prometheus、openfeign  
msa-deploy-job 分布式任务调度平台(XXL-JOB)、日志  

3.微服务访问信息

微服务 内部端口 外部端口 访问地址
注册中心 10001 30001 http://172.51.216.81:30001/
网关 8888   curl 10.99.231.140:8888/dconsumer/hello
curl 10.99.231.140:8888/dconsumer/say
curl 10.99.231.140:8888/dproducer/hello
curl 10.99.231.140:8888/dproducer/say
流量控制 8858 30858 访问地址:http://172.51.216.81:30858/#/login
账户密码:sentinel/sentinel
调用链监控 9411 30411 http://172.51.216.81:30411/zipkin
分布式任务调度平台 8080 30880 访问地址:http://172.51.216.81:30880/xxl-job-admin
账户密码:admin/123456
生产者 8911   curl 10.110.15.159:8911/hello
curl 10.110.15.159:8911/say
消费者 8912   curl 10.108.141.96:8912/hello
curl 10.108.141.96:8912/say
Ingress   80:31208
443:32099
zipkin
http://zipkin.k8s.com:31208/zipkin

sentinel
http://sentinel.k8s.com:31208/#/login
账户密码:sentinel/sentinel

xxl-job
http://xxljob.k8s.com:31208/xxl-job-admin
账户密码:admin/123456

# eureka
http://eureka.k8s.com:31208/

# gateway
http://gateway.k8s.com:31208/dconsumer/hello
http://gateway.k8s.com:31208/dconsumer/say
http://gateway.k8s.com:31208/dproducer/hello
http://gateway.k8s.com:31208/dproducer/say
Prometheus     消费者 msa-deploy-consumer
http://gateway.k8s.com:31208/dconsumer/actuator/prometheus

生产者 msa-deploy-producer
http://gateway.k8s.com:31208/dproducer/actuator/prometheus

3.准备工作

3.1.创建名字空间

1
2
3
4
5
6
[root@k8s-master ~]# kubectl create ns dev
namespace/dev created

[root@k8s-master ~]# kubectl get ns
...
dev                    Active   4s

3.2.Harbor创建项目

在私有镜像仓库创建springcloud项目,存储微服务镜像

3.3.创建私有镜像密钥Secret

K8S所有工作节点登陆Harbor镜像仓库

1
2
3
4
5
6
7
8
9
10
11
12
# 登陆 admin/admin
# docker login http://172.51.216.85:8888


[root@k8s-node1 ~]# docker login http://172.51.216.85:8888
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

提示登录成功。

登录过程创建或更新一个包含授权令牌的config.json文件。 查看config.json文件:

1
cat ~/.docker/config.json

输出包含类似以下内容的部分:

1
2
3
4
5
6
7
{
	"auths": {
		"172.51.216.85:8888": {
			"auth": "YWRtaW46YWRtaW4="
		}
	}
}

注意:如果您使用Docker凭据存储,您将看不到该auth条目,而是看到一个以存储名称为值的credsstore条目。

在master节点操作

1
2
3
4
5
6
# 对秘钥文件进行base64加密
cat ~/.docker/config.json |base64 -w 0


[root@k8s-master harbor]# cat ~/.docker/config.json |base64 -w 0
ewoJImF1dGhzIjogewoJCSIxNzIuNTEuMjE2Ljg1Ojg4ODgiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2WVdSdGFXND0iCgkJfQoJfQp9

k8s创建secret秘钥

注意:必须在相同的命名空间,默认在default

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# secret
harbor-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: harborsecret
  namespace: dev
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxNzIuNTEuMjE2Ljg1Ojg4ODgiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2WVdSdGFXND0iCgkJfQoJfQp9


[root@k8s-master harbor]# kubectl apply -f harbor-secret.yaml 
secret/harborsecret created

[root@k8s-master springcloud]# kubectl get secret -n dev
NAME                  TYPE                                  DATA   AGE
default-token-kjvf5   kubernetes.io/service-account-token   3      51m
harborsecret          kubernetes.io/dockerconfigjson        1      37s

3.4.Dockerfile标准配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
FROM openjdk:8-jdk-alpine

VOLUME /temp

ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"

ENV JAVA_POS=""

ENV ACTIVE="-Dspring.profiles.active=test"

ADD *.jar app.jar

ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}

ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone


#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-app:1.0.0 .

二、微服务部署

1.微服务配置

1.1.注册中心(eureka-server)

msa-eureka.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      name: 
      targetPort: 
      nodePort:  #对外暴露30001端口
  selector:
    app: 

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: dev
  name: 
spec:
  serviceName: 
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name:    #对应创建私有镜像密钥Secret
      containers:
        - name: 
          imagePullPolicy: 
          image: :
          ports:
          - containerPort: 
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: EUREKA_SERVER
              value: 
            - name: EUREKA_INSTANCE_HOSTNAME
              value: ${MY_POD_NAME}.msa-eureka.dev
            - name: ACTIVE
              value: 
  podManagementPolicy: "Parallel"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-eureka
  labels:
    app: msa-eureka
spec:
  type: NodePort
  ports:
    - port: 10001
      name: msa-eureka
      targetPort: 10001
      nodePort: 30001 #对外暴露30001端口
  selector:
    app: msa-eureka

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: dev
  name: msa-eureka
spec:
  serviceName: msa-eureka
  replicas: 3
  selector:
    matchLabels:
      app: msa-eureka
  template:
    metadata:
      labels:
        app: msa-eureka
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-eureka
          image: 172.51.216.85:8888/springcloud/msa-eureka:2.0.0
          ports:
          - containerPort: 10001
          env:
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: EUREKA_INSTANCE_HOSTNAME
              value: ${MY_POD_NAME}.msa-eureka.dev
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
  podManagementPolicy: "Parallel"

1.2.网关(msa-gateway)

msa-gateway.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name:    #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: :
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: EUREKA_SERVER
              value:  
            - name: ACTIVE
              value: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-gateway
  labels:
    app: msa-gateway
spec:
  type: ClusterIP
  ports:
    - port: 8888
      targetPort: 8888
  selector:
    app: msa-gateway

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: msa-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: msa-gateway
  template:
    metadata:
      labels:
        app: msa-gateway
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-gateway
          image: 172.51.216.85:8888/springcloud/msa-gateway:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 8888
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"

1.3.流量控制(Sentinel)

sentinel-server.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
      nodePort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      containers:
        - name: 
          image: bladex/sentinel-dashboard:
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: sentinel-server
  labels:
    app: sentinel-server
spec:
  type: NodePort
  ports:
    - port: 8858
      targetPort: 8858
      nodePort: 30858 #对外暴露30858端口
  selector:
    app: sentinel-server

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: sentinel-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sentinel-server
  template:
    metadata:
      labels:
        app: sentinel-server
    spec:
      containers:
        - name: sentinel-server
          image: bladex/sentinel-dashboard:1.7.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8858

1.4.调用链监控(Zipkin)

1.RabbitMQ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# rabbitmq.yml


apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-svc
  namespace: dev
spec:
  ports:
  - port: 5672
    targetPort: 5672
    protocol: TCP

---
apiVersion: v1
kind: Endpoints
metadata:
  name: rabbitmq-svc
  namespace: dev
subsets:
  - addresses:
    - ip: 172.51.216.98
    ports:
    - port: 5672

2.Elasticsearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# elasticsearch.yml


apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-svc
  namespace: dev
spec:
  ports:
  - port: 9200
    targetPort: 9200
    protocol: TCP

---
apiVersion: v1
kind: Endpoints
metadata:
  name: elasticsearch-svc
  namespace: dev
subsets:
  - addresses:
    - ip: 172.51.216.98
    ports:
    - port: 9200

3.部署Zipkin服务

zipkin-server.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
      nodePort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      containers:
        - name: 
          image: openzipkin/zipkin:
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: RABBIT_ADDRESSES
              value: 
            - name: RABBIT_USER
              value: 
            - name: RABBIT_PASSWORD
              value: 
            - name: STORAGE_TYPE
              value: 
            - name: ES_HOSTS
              value: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: zipkin-server
  labels:
    app: zipkin-server
spec:
  type: NodePort
  ports:
    - port: 9411
      targetPort: 9411
      nodePort: 30411 #对外暴露30411端口
  selector:
    app: zipkin-server

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: zipkin-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: zipkin-server
  template:
    metadata:
      labels:
        app: zipkin-server
    spec:
      containers:
        - name: zipkin-server
          image: openzipkin/zipkin:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9411
          env:
            - name: RABBIT_ADDRESSES
              value: "rabbitmq-svc.dev.svc.cluster.local:5672"
            - name: RABBIT_USER
              value: "guest"
            - name: RABBIT_PASSWORD
              value: "guest"
            - name: STORAGE_TYPE
              value: "elasticsearch"
            - name: ES_HOSTS
              value: "elasticsearch-svc.dev.svc.cluster.local:9200"

1.5.分布式任务调度平台(XXL-JOB)

1.mysql

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# mysql.yml


apiVersion: v1
kind: Service
metadata:
  name: mysql-svc
  namespace: dev
spec:
  ports:
  - port: 3306
    targetPort: 3306
    protocol: TCP

---
apiVersion: v1
kind: Endpoints
metadata:
  name: mysql-svc
  namespace: dev
subsets:
  - addresses:
    - ip: 172.51.216.98
    ports:
    - port: 3306

2.部署XXL-JOB服务

xxl-job.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
      nodePort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      containers:
        - name: 
          image: xuxueli/xxl-job-admin:
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: PARAMS
              value:  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: xxl-job
  labels:
    app: xxl-job
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30880 #对外暴露30880端口
  selector:
    app: xxl-job

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: xxl-job
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xxl-job
  template:
    metadata:
      labels:
        app: xxl-job
    spec:
      containers:
        - name: xxl-job
          image: xuxueli/xxl-job-admin:2.3.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: PARAMS
              value:  "--spring.datasource.username=root --spring.datasource.password=root --spring.datasource.url=jdbc:mysql://mysql-svc.dev.svc.cluster.local:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai"

1.6.部署XXL-JOB执行器(msa-deploy-job)

msa-deploy-job.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name:    #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: :
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: ACTIVE
              value: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-deploy-job
  labels:
    app: msa-deploy-job
spec:
  type: ClusterIP
  ports:
    - port: 8913
      targetPort: 8913
  selector:
    app: msa-deploy-job

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: msa-deploy-job
spec:
  replicas: 3
  selector:
    matchLabels:
      app: msa-deploy-job
  template:
    metadata:
      labels:
        app: msa-deploy-job
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-deploy-job
          image: 172.51.216.85:8888/springcloud/msa-deploy-job:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 8913
          env:
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"

1.7.生产者(msa-deploy-producer)

msa-deploy-producer.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name:    #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: :
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: EUREKA_SERVER
              value:  
            - name: ACTIVE
              value: 
            - name: DASHBOARD
              value: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-deploy-producer
  labels:
    app: msa-deploy-producer
spec:
  type: ClusterIP
  ports:
    - port: 8911
      targetPort: 8911
  selector:
    app: msa-deploy-producer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: msa-deploy-producer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: msa-deploy-producer
  template:
    metadata:
      labels:
        app: msa-deploy-producer
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-deploy-producer
          image: 172.51.216.85:8888/springcloud/msa-deploy-producer:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 8911
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
            - name: DASHBOARD
              value: "sentinel-server.dev.svc.cluster.local:8858"

1.8.消费者(msa-deploy-consumer)

msa-deploy-consumer.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name:    #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: :
          imagePullPolicy:  #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: EUREKA_SERVER
              value:  
            - name: ACTIVE
              value: 
            - name: DASHBOARD
              value: 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-deploy-consumer
  labels:
    app: msa-deploy-consumer
spec:
  type: ClusterIP
  ports:
    - port: 8912
      targetPort: 8912
  selector:
    app: msa-deploy-consumer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: msa-deploy-consumer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: msa-deploy-consumer
  template:
    metadata:
      labels:
        app: msa-deploy-consumer
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-deploy-consumer
          image: 172.51.216.85:8888/springcloud/msa-deploy-consumer:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 8912
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
            - name: DASHBOARD
              value: "sentinel-server.dev.svc.cluster.local:8858"

1.9.Ingress

调用链监控(Zipkin)、流量控制(Sentinel)、分布式任务调度平台(XXL-JOB)、注册中心(eureka-server)、网关(msa-gateway)。

Http代理

创建 msa-http.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: msa-http
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: zipkin.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: zipkin-server
                port:
                  number: 9411
    - host: sentinel.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sentinel-server
                port:
                  number: 8858
    - host: xxljob.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: xxl-job
                port:
                  number: 8080
    - host: eureka.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: msa-eureka
                port:
                  number: 10001
    - host: gateway.k8s.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: msa-gateway
                port:
                  number: 8888

1.10.values

values.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
global:
  namespace: "dev"
  imagePullSecrets: "harborsecret"
  repository: "172.51.216.85:8888/springcloud/"
  env:
    eureka: "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"


eureka:
  name: "msa-eureka"
  labels: "msa-eureka"
  service:
    type: "NodePort"
    port: 10001
    targetPort: 10001
    nodePort: 30001
  replicaCount: 3
  image:
    name: "msa-eureka"
    pullPolicy: "IfNotPresent"
    tag: "2.0.0"
    containerPort: 10001
  env:
    profiles: "-Dspring.profiles.active=k8s"


gateway:
  name: "msa-gateway"
  labels: "msa-gateway"
  service:
    type: "ClusterIP"
    port: 8888
    targetPort: 8888
  replicaCount: 3
  image:
    name: "msa-gateway"
    pullPolicy: "IfNotPresent"
    tag: "2.0.0"
    containerPort: 8888
  env:
    profiles: "-Dspring.profiles.active=k8s"


sentinel:
  name: "sentinel-server"
  labels: "sentinel-server"
  service:
    type: "NodePort"
    port: 8858
    targetPort: 8858
    nodePort: 30858
  replicaCount: 1
  image:
    name: "sentinel-server"
    pullPolicy: "IfNotPresent"
    tag: "1.7.2"
    containerPort: 8858


zipkin:
  name: "zipkin-server"
  labels: "zipkin-server"
  service:
    type: "NodePort"
    port: 9411
    targetPort: 9411
    nodePort: 30411
  replicaCount: 3
  image:
    name: "zipkin-server"
    pullPolicy: "IfNotPresent"
    tag: "latest"
    containerPort: 9411
  env:
    addresses: "rabbitmq-svc.dev.svc.cluster.local:5672"
    user: "guest"
    password: "guest"
    storage: "elasticsearch"
    eshosts: "elasticsearch-svc.dev.svc.cluster.local:9200"


xxljob:
  name: "xxl-job"
  labels: "xxl-job"
  service:
    type: "NodePort"
    port: 8080
    targetPort: 8080
    nodePort: 30880
  replicaCount: 3
  image:
    name: "xxl-job"
    pullPolicy: "IfNotPresent"
    tag: "2.3.0"
    containerPort: 8080
  env:
    params: "--spring.datasource.username=root --spring.datasource.password=root --spring.datasource.url=jdbc:mysql://mysql-svc.dev.svc.cluster.local:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai"


job:
  name: "msa-deploy-job"
  labels: "msa-deploy-job"
  service:
    type: "ClusterIP"
    port: 8913
    targetPort: 8913
  replicaCount: 3
  image:
    name: "msa-deploy-job"
    pullPolicy: "IfNotPresent"
    tag: "2.0.0"
    containerPort: 8913
  env:
    profiles: "-Dspring.profiles.active=k8s"


producer:
  name: "msa-deploy-producer"
  labels: "msa-deploy-producer"
  service:
    type: "ClusterIP"
    port: 8911
    targetPort: 8911
  replicaCount: 3
  image:
    name: "msa-deploy-producer"
    pullPolicy: "IfNotPresent"
    tag: "2.0.0"
    containerPort: 8911
  env:
    profiles: "-Dspring.profiles.active=k8s"
    dashboard: "sentinel-server.dev.svc.cluster.local:8858"


consumer:
  name: "msa-deploy-consumer"
  labels: "msa-deploy-consumer"
  service:
    type: "ClusterIP"
    port: 8912
    targetPort: 8912
  replicaCount: 3
  image:
    name: "msa-deploy-consumer"
    pullPolicy: "IfNotPresent"
    tag: "2.0.0"
    containerPort: 8912
  env:
    profiles: "-Dspring.profiles.active=k8s"
    dashboard: "sentinel-server.dev.svc.cluster.local:8858"

2.部署服务

创建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# 创建 msa
[root@k8s-master springcloud-helm]# helm create cloud -n dev
Creating cloud


[root@k8s-master test]# ll
drwxr-xr-x 4 root root 93 Nov 11 13:50 cloud


[root@k8s-master test]# tree
.
└── cloud
    ├── charts
    ├── Chart.yaml
    ├── templates
    │   ├── deployment.yaml
    │   ├── _helpers.tpl
    │   ├── hpa.yaml
    │   ├── ingress.yaml
    │   ├── NOTES.txt
    │   ├── serviceaccount.yaml
    │   ├── service.yaml
    │   └── tests
    │       └── test-connection.yaml
    └── values.yaml

4 directories, 10 files


--------------------------------------------
# templates删除所有文件
# templates创建配置文件

[root@k8s-master cloud]# tree
.
├── charts
├── Chart.yaml
├── templates
│   ├── elasticsearch.yml
│   ├── msa-deploy-consumer.yaml
│   ├── msa-deploy-job.yaml
│   ├── msa-deploy-producer.yaml
│   ├── msa-eureka.yaml
│   ├── msa-gateway.yaml
│   ├── msa-http.yaml
│   ├── mysql.yml
│   ├── NOTES.txt
│   ├── rabbitmq.yml
│   ├── sentinel-server.yaml
│   ├── xxl-job.yaml
│   └── zipkin-server.yaml
└── values.yaml

2 directories, 15 files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#模板文件

# Chart.yaml
[root@k8s-master cloud]# vim Chart.yaml

apiVersion: v2
name: cloud
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.0.0"


# values.yaml 暂时不用
[root@k8s-master cloud]# vim values.yaml


# NOTES.txt
[root@k8s-master cloud]# vim NOTES.txt

Spring Cloud!!!

安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# 查看实际的模板被渲染过后的资源文件
# helm get manifest web
# helm install web nginx/

# 生成的最终的资源清单文件打印出来,而不会真正的去部署一个release实例
# helm install --name mychart --dry-run --debug -f global.yaml ./mychart/
# helm install --name mychart --dry-run --debug --set course="k8s" ./mychart/


# 调试
# Helm也提供了--dry-run --debug调试参数,帮助你验证模板正确性。在执行helm install时候带上这两个参数就可以把对应的values值和渲染的资源清单打印出来,而不会真正的去部署一个release。

[root@k8s-master springcloud-helm]#  helm install msa cloud --dry-run --debug -n dev


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION

----------------------------------------------------------------------
# 安装
[root@k8s-master test]# helm install msa cloud -n dev
NAME: msa
LAST DEPLOYED: Thu Nov 11 14:25:55 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master test]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-11 14:25:55.000327298 +0800 CST	deployed	cloud-0.1.0	1.0.0   


[root@k8s-master springcloud-helm]# helm install msa cloud -n dev
NAME: msa
LAST DEPLOYED: Thu Nov 11 22:56:20 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


# 检索已经发布的 release 的资源文件
[root@k8s-master springcloud-helm]# helm get manifest msa -n dev


# 打包
# 修改Chart.yaml中的helm chart配置信息,然后使用下列命令将chart打包成一个压缩文件。
# 打包出cloud-0.1.0.tgz文件。

[root@k8s-master springcloud-helm]# helm package cloud
Successfully packaged chart and saved it to: /k8s/springcloud-helm/cloud-0.1.0.tgz

[root@k8s-master springcloud-helm]# ll
total 8
drwxr-xr-x 4 root root   93 Nov 11 22:50 cloud
-rw-r--r-- 1 root root 2733 Nov 11 23:00 cloud-0.1.0.tgz
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# 查看资源
[root@k8s-master springcloud-helm]# kubectl get all -n dev
NAME                                       READY   STATUS    RESTARTS   AGE
pod/msa-deploy-consumer-6b75cf55d-dj4vm    1/1     Running   0          5m8s
pod/msa-deploy-consumer-6b75cf55d-dzgtc    1/1     Running   0          5m8s
pod/msa-deploy-consumer-6b75cf55d-rx6sm    1/1     Running   0          5m8s
pod/msa-deploy-job-7c999dd76-87cpr         1/1     Running   0          5m8s
pod/msa-deploy-job-7c999dd76-wgt5v         1/1     Running   0          5m8s
pod/msa-deploy-job-7c999dd76-z5hp2         1/1     Running   0          5m8s
pod/msa-deploy-producer-7965c98bbf-hz6nj   1/1     Running   0          5m8s
pod/msa-deploy-producer-7965c98bbf-r8rs4   1/1     Running   0          5m8s
pod/msa-deploy-producer-7965c98bbf-s65qh   1/1     Running   0          5m8s
pod/msa-deploy-producer-7965c98bbf-sbndc   1/1     Running   0          5m8s
pod/msa-deploy-producer-7965c98bbf-w4bkf   1/1     Running   0          5m8s
pod/msa-eureka-0                           1/1     Running   0          5m8s
pod/msa-eureka-1                           1/1     Running   0          5m8s
pod/msa-eureka-2                           1/1     Running   0          5m8s
pod/msa-gateway-597494c7f4-fq47k           1/1     Running   0          5m8s
pod/msa-gateway-597494c7f4-hvj2r           1/1     Running   0          5m8s
pod/msa-gateway-597494c7f4-ws8f7           1/1     Running   0          5m8s
pod/sentinel-server-5697bdc87b-xlgzg       1/1     Running   0          5m8s
pod/xxl-job-57649955bd-9lcpw               1/1     Running   0          5m8s
pod/zipkin-server-84c5fdcf5d-76vlw         1/1     Running   0          5m8s
pod/zipkin-server-84c5fdcf5d-t4jjh         1/1     Running   0          5m8s

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
service/elasticsearch-svc     ClusterIP   10.100.36.157    <none>        9200/TCP          5m8s
service/msa-deploy-consumer   ClusterIP   10.108.247.253   <none>        8912/TCP          5m8s
service/msa-deploy-job        ClusterIP   10.97.206.223    <none>        8913/TCP          5m8s
service/msa-deploy-producer   ClusterIP   10.102.227.63    <none>        8911/TCP          5m8s
service/msa-eureka            NodePort    10.111.13.62     <none>        10001:30001/TCP   5m8s
service/msa-gateway           ClusterIP   10.103.80.16     <none>        8888/TCP          5m8s
service/mysql-svc             ClusterIP   10.102.78.175    <none>        3306/TCP          5m8s
service/rabbitmq-svc          ClusterIP   10.110.218.166   <none>        5672/TCP          5m8s
service/sentinel-server       NodePort    10.111.141.18    <none>        8858:30858/TCP    5m8s
service/xxl-job               NodePort    10.102.102.253   <none>        8080:30880/TCP    5m8s
service/zipkin-server         NodePort    10.102.220.51    <none>        9411:30411/TCP    5m8s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/msa-deploy-consumer   3/3     3            3           5m8s
deployment.apps/msa-deploy-job        3/3     3            3           5m8s
deployment.apps/msa-deploy-producer   5/5     5            5           5m8s
deployment.apps/msa-gateway           3/3     3            3           5m8s
deployment.apps/sentinel-server       1/1     1            1           5m8s
deployment.apps/xxl-job               1/1     1            1           5m8s
deployment.apps/zipkin-server         2/2     2            2           5m8s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/msa-deploy-consumer-6b75cf55d    3         3         3       5m8s
replicaset.apps/msa-deploy-job-7c999dd76         3         3         3       5m8s
replicaset.apps/msa-deploy-producer-7965c98bbf   5         5         5       5m8s
replicaset.apps/msa-gateway-597494c7f4           3         3         3       5m8s
replicaset.apps/sentinel-server-5697bdc87b       1         1         1       5m8s
replicaset.apps/xxl-job-57649955bd               1         1         1       5m8s
replicaset.apps/zipkin-server-84c5fdcf5d         2         2         2       5m8s

NAME                          READY   AGE
statefulset.apps/msa-eureka   3/3     5m8s

3.测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# 测试
在本机添加域名解析
C:\Windows\System32\drivers\etc
在hosts文件添加域名解析(master主机的iP地址)
172.51.216.81 		zipkin.k8s.com
172.51.216.81 		sentinel.k8s.com
172.51.216.81 		xxljob.k8s.com
172.51.216.81       eureka.k8s.com
172.51.216.81       gateway.k8s.com


[root@k8s-master ingress]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx                        NodePort    10.97.245.122    <none>        80:31208/TCP,443:32099/TCP   57d
ingress-nginx-controller             NodePort    10.101.238.213   <none>        80:30743/TCP,443:31410/TCP   57d
ingress-nginx-controller-admission   ClusterIP   10.100.142.101   <none>        443/TCP                      57d


# ingress地址
# 在本机浏览器输入:
http://zipkin.k8s.com:31208/
http://sentinel.k8s.com:31208/
http://xxljob.k8s.com:31208/
http://eureka.k8s.com:31208/
http://gateway.k8s.com:31208/


# 访问地址:
# zipkin
http://172.51.216.81:30411/zipkin
#sentinel
http://172.51.216.81:30858/#/login
账户密码:sentinel/sentinel
# xxl-job
http://172.51.216.81:30880/xxl-job-admin
账户密码:admin/123456


# 测试地址
# zipkin
http://zipkin.k8s.com:31208/zipkin
#sentinel
http://sentinel.k8s.com:31208/#/login
账户密码:sentinel/sentinel
# xxl-job
http://xxljob.k8s.com:31208/xxl-job-admin
账户密码:admin/123456
# eureka
http://eureka.k8s.com:31208/

# gateway
http://gateway.k8s.com:31208/dconsumer/hello
http://gateway.k8s.com:31208/dconsumer/say
http://gateway.k8s.com:31208/dproducer/hello
http://gateway.k8s.com:31208/dproducer/say
1
2
3
4
5
6
7
8
9
10
11
12
13
# 查看日志
[root@k8s-master msa-deploy-job]# kubectl logs -f --tail=10 msa-deploy-job-7c999dd76-87cpr -n dev
----job one 1---Wed Nov 10 09:30:50 CST 2021
----job one 1---Wed Nov 10 09:30:53 CST 2021
----job one 1---Wed Nov 10 09:30:56 CST 2021
----job one 1---Wed Nov 10 09:30:59 CST 2021
----job one 1---Wed Nov 10 09:31:02 CST 2021
----job one 1---Wed Nov 10 09:31:05 CST 2021
----job one 1---Wed Nov 10 09:31:08 CST 2021
----job one 1---Wed Nov 10 09:31:11 CST 2021
----job one 1---Wed Nov 10 09:31:14 CST 2021
----job one 1---Wed Nov 10 09:31:17 CST 2021
----job one 1---Wed Nov 10 09:31:20 CST 2021

4.升级

通过修改模板yaml升级

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
# 升级方式
# 修改生产者服务(msa-deploy-producer)接口两次,形成两个新版本的docker镜像
# 分别升级两次版本


# 升级前
[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-11 22:56:20.805228313 +0800 CST	deployed	cloud-0.1.0	1.0.0  


[root@k8s-master springcloud-helm]# helm status msa -n dev
NAME: msa
LAST DEPLOYED: Thu Nov 11 22:56:20 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS  	CHART      	APP VERSION	DESCRIPTION     
1       	Thu Nov 11 22:56:20 2021	deployed	cloud-0.1.0	1.0.0      	Install complete


[root@k8s-master templates]# kubectl get all -n dev
NAME                                       READY   STATUS    RESTARTS   AGE
pod/msa-deploy-consumer-6b75cf55d-dj4vm    1/1     Running   0          10h
pod/msa-deploy-consumer-6b75cf55d-dzgtc    1/1     Running   0          10h
pod/msa-deploy-consumer-6b75cf55d-rx6sm    1/1     Running   0          10h
pod/msa-deploy-job-7c999dd76-87cpr         1/1     Running   0          10h
pod/msa-deploy-job-7c999dd76-wgt5v         1/1     Running   0          10h
pod/msa-deploy-job-7c999dd76-z5hp2         1/1     Running   0          10h
pod/msa-deploy-producer-7965c98bbf-hz6nj   1/1     Running   0          10h
pod/msa-deploy-producer-7965c98bbf-r8rs4   1/1     Running   0          10h
pod/msa-deploy-producer-7965c98bbf-s65qh   1/1     Running   0          10h
pod/msa-deploy-producer-7965c98bbf-sbndc   1/1     Running   0          10h
pod/msa-deploy-producer-7965c98bbf-w4bkf   1/1     Running   0          10h
pod/msa-eureka-0                           1/1     Running   0          10h
pod/msa-eureka-1                           1/1     Running   0          10h
pod/msa-eureka-2                           1/1     Running   0          10h
pod/msa-gateway-597494c7f4-fq47k           1/1     Running   0          10h
pod/msa-gateway-597494c7f4-hvj2r           1/1     Running   0          10h
pod/msa-gateway-597494c7f4-ws8f7           1/1     Running   0          10h
pod/sentinel-server-5697bdc87b-xlgzg       1/1     Running   0          10h
pod/xxl-job-57649955bd-9lcpw               1/1     Running   0          10h
pod/zipkin-server-84c5fdcf5d-76vlw         1/1     Running   0          10h
pod/zipkin-server-84c5fdcf5d-t4jjh         1/1     Running   0          10h

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
service/elasticsearch-svc     ClusterIP   10.100.36.157    <none>        9200/TCP          10h
service/msa-deploy-consumer   ClusterIP   10.108.247.253   <none>        8912/TCP          10h
service/msa-deploy-job        ClusterIP   10.97.206.223    <none>        8913/TCP          10h
service/msa-deploy-producer   ClusterIP   10.102.227.63    <none>        8911/TCP          10h
service/msa-eureka            NodePort    10.111.13.62     <none>        10001:30001/TCP   10h
service/msa-gateway           ClusterIP   10.103.80.16     <none>        8888/TCP          10h
service/mysql-svc             ClusterIP   10.102.78.175    <none>        3306/TCP          10h
service/rabbitmq-svc          ClusterIP   10.110.218.166   <none>        5672/TCP          10h
service/sentinel-server       NodePort    10.111.141.18    <none>        8858:30858/TCP    10h
service/xxl-job               NodePort    10.102.102.253   <none>        8080:30880/TCP    10h
service/zipkin-server         NodePort    10.102.220.51    <none>        9411:30411/TCP    10h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/msa-deploy-consumer   3/3     3            3           10h
deployment.apps/msa-deploy-job        3/3     3            3           10h
deployment.apps/msa-deploy-producer   5/5     5            5           10h
deployment.apps/msa-gateway           3/3     3            3           10h
deployment.apps/sentinel-server       1/1     1            1           10h
deployment.apps/xxl-job               1/1     1            1           10h
deployment.apps/zipkin-server         2/2     2            2           10h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/msa-deploy-consumer-6b75cf55d    3         3         3       10h
replicaset.apps/msa-deploy-job-7c999dd76         3         3         3       10h
replicaset.apps/msa-deploy-producer-7965c98bbf   5         5         5       10h
replicaset.apps/msa-gateway-597494c7f4           3         3         3       10h
replicaset.apps/sentinel-server-5697bdc87b       1         1         1       10h
replicaset.apps/xxl-job-57649955bd               1         1         1       10h
replicaset.apps/zipkin-server-84c5fdcf5d         2         2         2       10h

NAME                          READY   AGE
statefulset.apps/msa-eureka   3/3     10h


# 升级1
[root@k8s-master springcloud-helm]# helm upgrade msa cloud -n dev
Release "msa" has been upgraded. Happy Helming!
NAME: msa
LAST DEPLOYED: Fri Nov 12 09:31:55 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	2       	2021-11-12 09:31:55.408522116 +0800 CST	deployed	cloud-0.1.0	1.0.0  


# 升级2
[root@k8s-master springcloud-helm]# helm upgrade msa cloud -n dev
Release "msa" has been upgraded. Happy Helming!
NAME: msa
LAST DEPLOYED: Fri Nov 12 09:35:03 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	3       	2021-11-12 09:35:03.053485256 +0800 CST	deployed	cloud-0.1.0	1.0.0  


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
1       	Thu Nov 11 22:56:20 2021	superseded	cloud-0.1.0	1.0.0      	Install complete
2       	Fri Nov 12 09:31:55 2021	superseded	cloud-0.1.0	1.0.0      	Upgrade complete
3       	Fri Nov 12 09:35:03 2021	deployed  	cloud-0.1.0	1.0.0      	Upgrade complete


[root@k8s-master springcloud-helm]# helm get manifest msa -n dev

5.回滚

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# 如果在发布后没有达到预期的效果,则可以使用helm rollback回滚到之前的版本。
# 例如将应用回滚到第一个版本:
helm rollback web 1


# 回滚前
[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	3       	2021-11-12 09:35:03.053485256 +0800 CST	deployed	cloud-0.1.0	1.0.0


[root@k8s-master springcloud-helm]# helm status msa -n dev
NAME: msa
LAST DEPLOYED: Fri Nov 12 09:35:03 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
1       	Thu Nov 11 22:56:20 2021	superseded	cloud-0.1.0	1.0.0      	Install complete
2       	Fri Nov 12 09:31:55 2021	superseded	cloud-0.1.0	1.0.0      	Upgrade complete
3       	Fri Nov 12 09:35:03 2021	deployed  	cloud-0.1.0	1.0.0      	Upgrade complete


# 回滚到版本1
[root@k8s-master springcloud-helm]# helm rollback msa 1 -n dev
Rollback was a success! Happy Helming!


# 回滚后的状态
[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	4       	2021-11-12 09:41:31.271565007 +0800 CST	deployed	cloud-0.1.0	1.0.0  

[root@k8s-master springcloud-helm]# helm status msa -n dev
NAME: msa
LAST DEPLOYED: Fri Nov 12 09:41:31 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
1       	Thu Nov 11 22:56:20 2021	superseded	cloud-0.1.0	1.0.0      	Install complete
2       	Fri Nov 12 09:31:55 2021	superseded	cloud-0.1.0	1.0.0      	Upgrade complete
3       	Fri Nov 12 09:35:03 2021	superseded	cloud-0.1.0	1.0.0      	Upgrade complete
4       	Fri Nov 12 09:41:31 2021	deployed  	cloud-0.1.0	1.0.0      	Rollback to 1 

6.删除

1
2
3
4
5
6
7
8
9
10
11
卸载发行版,请使用以下helm uninstall命令:
# helm uninstall web -n dev
# helm delete web -n dev


[root@k8s-master springcloud-helm]# helm uninstall msa -n dev
release "msa" uninstalled


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION

7.使用包安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 使用 cloud-0.1.0.tgz 安装

[root@k8s-master springcloud-helm]# helm install msa cloud-0.1.0.tgz -n dev
NAME: msa
LAST DEPLOYED: Fri Nov 12 09:51:09 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-12 09:51:09.530677806 +0800 CST	deployed	cloud-0.1.0	1.0.0

8.配置values

8.1.修改配置信息

values.yaml

1
2
3
4
5
6
7
8
9
10
global:
  namespace: "dev"

producer:
  name: "msa-deploy-producer"
  serviceType: "ClusterIP"
  port: 8911
  targetPort: 8911
  replicas: 4
  imageVersion: "2.0.0"

生产者

msa-deploy-producer.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: 172.51.216.85:8888/springcloud/:
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
            - name: DASHBOARD
              value: "sentinel-server.dev.svc.cluster.local:8858"

原配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: msa-deploy-producer
  labels:
    app: msa-deploy-producer
spec:
  type: ClusterIP
  ports:
    - port: 8911
      targetPort: 8911
  selector:
    app: msa-deploy-producer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: msa-deploy-producer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: msa-deploy-producer
  template:
    metadata:
      labels:
        app: msa-deploy-producer
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: msa-deploy-producer
          image: 172.51.216.85:8888/springcloud/msa-deploy-producer:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 8911
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
            - name: DASHBOARD
              value: "sentinel-server.dev.svc.cluster.local:8858"

8.2.部署服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@k8s-master test]# helm install msa cloud --dry-run --debug -n dev


# 安装
[root@k8s-master springcloud-helm]# 
[root@k8s-master springcloud-helm]# helm install msa cloud -n dev
NAME: msa
LAST DEPLOYED: Fri Nov 12 10:16:35 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-12 10:16:35.753166106 +0800 CST	deployed	cloud-0.2.0	2.0.0  


# 检索已经发布的 release 的资源文件
[root@k8s-master test]# helm get manifest msa -n dev

8.3.升级

通过修改values.yaml升级

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# 通过修改values.yaml升级,拉取不同镜像版本

# 升级前
[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-12 10:16:35.753166106 +0800 CST	deployed	cloud-0.2.0	2.0.0  


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS  	CHART      	APP VERSION	DESCRIPTION     
1       	Fri Nov 12 10:16:35 2021	deployed	cloud-0.2.0	2.0.0      	Install complete


# 修改文件
# 修改成副本
[root@k8s-master cloud]# vim values.yaml
global:
  namespace: "dev"

producer:
  name: "msa-deploy-producer"
  serviceType: "ClusterIP"
  port: 8911
  targetPort: 8911
  replicas: 8
  imageVersion: "2.2.0"


[root@k8s-master springcloud-helm]# helm upgrade msa cloud -n dev
Release "msa" has been upgraded. Happy Helming!
NAME: msa
LAST DEPLOYED: Fri Nov 12 10:22:37 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Spring Cloud!!!



[root@k8s-master springcloud-helm]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	2       	2021-11-12 10:22:37.210875583 +0800 CST	deployed	cloud-0.2.0	2.0.0  


[root@k8s-master springcloud-helm]# helm history msa -n dev
REVISION	UPDATED                 	STATUS    	CHART      	APP VERSION	DESCRIPTION     
1       	Fri Nov 12 10:16:35 2021	superseded	cloud-0.2.0	2.0.0      	Install complete
2       	Fri Nov 12 10:22:37 2021	deployed  	cloud-0.2.0	2.0.0      	Upgrade complete


[root@k8s-master test]# helm get manifest msa -n dev

9.Chart Hooks

9.1.Hook配置

Hooks在资源清单中的metadata部分用annotations注解的方式进行声明:

1
2
3
4
5
6
7
8
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded
......

9.2.生产者加Hook注解

msa-deploy-producer.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: v1
kind: Service
metadata:
  namespace: 
  name: 
  labels:
    app: 
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  type: 
  ports:
    - port: 
      targetPort: 
  selector:
    app: 

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: 
  name: 
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  replicas: 
  selector:
    matchLabels:
      app: 
  template:
    metadata:
      labels:
        app: 
    spec:
      imagePullSecrets:
        - name: harborsecret   #对应创建私有镜像密钥Secret
      containers:
        - name: 
          image: 172.51.216.85:8888/springcloud/:2.0.0
          imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
          ports:
          - containerPort: 
          env:
            - name: EUREKA_SERVER
              value:  "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
            - name: ACTIVE
              value: "-Dspring.profiles.active=k8s"
            - name: DASHBOARD
              value: "sentinel-server.dev.svc.cluster.local:8858"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[root@k8s-master test]# helm install msa cloud --dry-run --debug -n dev


# 安装
[root@k8s-master test]# helm install msa cloud -n dev
NAME: msa
LAST DEPLOYED: Fri Nov 12 11:36:58 2021
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Spring Cloud!!!


[root@k8s-master test]# helm list -n dev
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART      	APP VERSION
msa 	dev      	1       	2021-11-12 11:36:58.339808934 +0800 CST	deployed	cloud-0.5.0	1.0.5 


# 检索已经发布的 release 的资源文件
[root@k8s-master test]# helm get manifest msa -n dev


# 卸载
[root@k8s-master test]# helm uninstall msa -n dev
release "msa" uninstalled

# Hook资源需要自己删除
[root@k8s-master test]# kubectl get all -n dev
NAME                                       READY   STATUS             RESTARTS   AGE
pod/msa-deploy-producer-786f6dff44-4tn6h   0/1     ImagePullBackOff   0          5m24s
pod/msa-deploy-producer-786f6dff44-7wcbw   0/1     ImagePullBackOff   0          5m24s
pod/msa-deploy-producer-786f6dff44-b56r8   0/1     ImagePullBackOff   0          5m24s

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/msa-deploy-producer   ClusterIP   10.101.112.240   <none>        8911/TCP   5m24s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/msa-deploy-producer   0/3     3            0           5m24s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/msa-deploy-producer-786f6dff44   3         3         0       5m24s