Pika在Kubeblocks上部署细节

背景

Helm 用于在 Kubernetes 集群中安装 Helm Charts, 它是一种打包和分发 Kubernetes 资源的方式, helm install 命令用于将 Helm Chart 部署到 Kubernetes 集群中，并创建一个运行应用程序的 Kubernetes Release

在 kubeblocks_helm目录下有 pika 和 pika-cluster两个目录，在 pika-cluster目录下的 value.yaml定义了集群组件部署个数

  1. Upgrade or update kubeblocks to 0.7.1-beta.1

    1
    kbcli kubeblocks install --version 0.7.1-beta.1

  2. install the cluster

    1
    helm install pika ./pika && helm install pika-cluster ./pika-cluster

    pika-cluster/values.yaml

    这里定义了部署时集群的个数以及资源的分配情况 (group，etcd，codisproxy，codisdashboard，codisproxy)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    # Default values for redis-cluster.
    # This is a YAML-formatted file.
    # Declare variables to be passed into your templates.

    nameOverride: ""
    fullnameOverride: ""

    groupCount: 2

    slaveCount: 1

    etcdReplicaCount: 3

    codisProxyReplicaCount: 2

    codisFeReplicaCount: 1

    codisDashboardReplicaCount: 1

    terminationPolicy: Delete

    clusterVersionOverride: ""

    monitor:
      enabled: false

    switchPolicy:
      type: Noop
      
      resources:
      # We usually recommend not to specify default resources and to leave this as a conscious
      # choice for the user. This also increases chances charts run on environments with little
      # resources, such as Minikube. If you do want to specify resources, uncomment the following
      # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      pikaGroup:
        limits:
          cpu: 500m
          memory: 3Gi
        requests:
          cpu: 500m
          memory: 1Gi
      etcd:
        limits:
          cpu: 500m
          memory: 3Gi
        requests:
          cpu: 500m
          memory: 1Gi
      codisProxy:
        limits:
          cpu: 500m
          memory: 3Gi
        requests:
          cpu: 500m
          memory: 1Gi
      codisFe:
        limits:
          cpu: 500m
          memory: 3Gi
        requests:
          cpu: 500m
          memory: 1Gi
      codisDashboard:
        limits:
          cpu: 500m
          memory: 3Gi
        requests:
          cpu: 500m
          memory: 1Gi

    persistence:
      enabled: true
      pikaData:
        storageClassName:
        size: 10Gi
      etcdData:
        storageClassName:
        size: 1Gi

    topologyKeys:
      - kubernetes.io/hostname

    ## @param tolerations
    ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
    ##
    tolerations: [ ]

    #enabledLogs:
    #  - running

    # The RBAC permission used by cluster component pod, now include event.create
    serviceAccount:
      name: ""

    etcdAddrs: "etcd-0.etcd-headless.default.svc.cluster.local:2379,etcd-1.etcd-headless.default.svc.cluster.local:2379,etcd-2.etcd-headless.default.svc.cluster.local:2379"

    pika/value.yaml

    这里定义了拉取 Pika，Codis，Etcd 镜像的路径

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    pika:
      version: v3.5.2
    image:
      pika:
        registry: docker.io
        repository: pikadb/pika
        tag: v3.5.2
        pullPolicy: IfNotPresent
      codis:
        registry: docker.io
        repository: pikadb/codis
        tag: v3.5.2
        pullPolicy: IfNotPresent
      etcd:
        registry: docker.io
        repository: bitnami/etcd
        tag: 3.5.9
        pullPolicy: IfNotPresent
        
     roleProbe:
      pika:
        failureThreshold: 2
        periodSeconds: 1
        timeoutSeconds: 1
      codis-proxy:
        failureThreshold: 2
        periodSeconds: 1
        timeoutSeconds: 1
      codis-dashboard:
        failureThreshold: 2
        periodSeconds: 1
        timeoutSeconds: 1
      codis-fe:
        failureThreshold: 2
        periodSeconds: 1
        timeoutSeconds: 1
      etcd:
        failureThreshold: 2
        periodSeconds: 1
        timeoutSeconds: 1

    clusterVersionOverride: 3.5.2
    nameOverride: ""
    fullnameOverride: ""

    clusterDomain: ".cluster.local"

    kubeblocks_helm/pika/templates/clusterdefinition.yaml

    这里明确了 Pika-Group 的部署方式，可以看到是通过执行 /script/admin.sh --rebalance 脚本去实现的

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    spec:
      type: pika
      connectionCredential:
        username: default
        password: "$(RANDOM_PASSWD)"
        endpoint: "$(SVC_FQDN):$(SVC_PORT_pika)"
        host: "$(SVC_FQDN)"
        port: "$(SVC_PORT_pika)"
      componentDefs:
        - name: pika-group
          workloadType: Stateful
          characterType: pika
          service:
            ports:
              - name: pika
                port: 9221
                targetPort: pika
          configSpecs:
            - name: pika-config
              templateRef: pika-conf-template
              namespace: {{ .Release.Namespace }}
              volumeName: config
          scriptSpecs:
            - name: pika-script
              templateRef: pika-script-template
              namespace: {{ .Release.Namespace }}
              volumeName: script
              defaultMode: 0555
          postStartSpec:
            cmdExecutorConfig:
              image: {{ .Values.image.pika.registry | default "docker.io" }}/{{ .Values.image.pika.repository }}:{{ .Values.image.pika.tag }}
              command:
                - /script/pika-group-post-start.sh
            scriptSpecSelectors:
              - name: pika-script
          podSpec:
            containers:
              - name: pika
                ports:
                  - name: pika
                    containerPort: 9221
                volumeMounts:
                  - name: config
                    mountPath: /etc/pika
                command:
                  - "/pika/bin/pika"
                args:
                  - "-c"
                  - "/etc/pika/pika.conf"
              - name: codis-admin
                volumeMounts:
                  - name: script
                    mountPath: /script
                command:
                  - "/bin/bash"
                args:
                  - "-c"
                  - "/script/admin.sh --rebalance" ## 自动rebalance
                        podSpec:
            containers:
              - name: pika
                ports:
                  - name: pika
                    containerPort: 9221
                volumeMounts:
                  - name: config
                    mountPath: /etc/pika
                command:
                  - "/pika/bin/pika"
                args:
                  - "-c"
                  - "/etc/pika/pika.conf"
              - name: codis-admin
                volumeMounts:
                  - name: script
                    mountPath: /script
                command:
                  - "/bin/bash"
                args:
                  - "-c"
                  - "/script/admin.sh --register-server;tail -f /dev/null"

    kubeblocks_helm/pika/script/admin.sh

    这个脚本文件里面写了 Pika 集群的部署方式

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    #! /bin/bash
    set -x

    # set instance role
    set_instance_role() {
      POD_ID=${HOSTNAME##*-}
      echo "POD_ID: "${POD_ID}
    }

    # set group id
    set_group_id() {
      GROUP_ID=${KB_CLUSTER_COMP_NAME##*-}
      echo "GROUP_ID: "${GROUP_ID}
    }

    # set codis dashboard
    set_codis_dashboard() {
      CODIS_DASHBOARD="${KB_CLUSTER_NAME}-codis-dashboard"
      echo "CODIS_DASHBOARD: "${CODIS_DASHBOARD}
      CODIS_ADMIN="/codis/bin/codis-admin --dashboard=${CODIS_DASHBOARD}:18080"
      echo "CODIS_ADMIN: "${CODIS_ADMIN}
    }

    wait_server_running() {
      until nc -z 127.0.0.1 9221; do
        echo waiting for pika
        sleep 2
      done
    }

    wait_dashboard_running() {
      until nc -z ${CODIS_DASHBOARD} 18080; do
        echo waiting for codis dashboard
        sleep 2
      done
    }

    wait_master_registered() {
      until $CODIS_ADMIN --list-group | jq -r '.[] | select(.id == '${GROUP_ID}') | .servers[] | select(.role == "master") | .server'; do
        echo waiting for master registered
        sleep 2
      done
    }

    # confirm group has the max index of all groups
    confirm_max_group() {
        max_group_id=0
        for component in ${KB_CLUSTER_COMPONENT_LIST//,/ }; do
            if [[ ${component} =~ pika-group-([0-9]+) ]]; then
            group_id=${BASH_REMATCH[1]}
            if [ ${group_id} -gt ${max_group_id} ]; then
                max_group_id=${group_id}
            fi
            fi
        done
        if [ ${GROUP_ID} -ne ${max_group_id} ]; then
            echo "Exit: group id ${GROUP_ID} is not max group id ${max_group_id}"
            exit 0
        fi

    }

    reload_until_success() {
      until $CODIS_ADMIN --reload 1>/dev/null 2>&1; do
        echo waiting for reload success
        sleep 2
      done
    }

    register_server() {
      reload_until_success
      if [ ${POD_ID} -gt 0 ]; then wait_master_registered; fi
      $CODIS_ADMIN --create-group --gid=${GROUP_ID} 1>/dev/null 2>&1
      $CODIS_ADMIN --group-add --gid=${GROUP_ID} --addr=${KB_POD_FQDN}:9221
      $CODIS_ADMIN --sync-action --create --addr=${KB_POD_FQDN}:9221 1>/dev/null 2>&1
    }

    remove_server() {
      $CODIS_ADMIN --reload
      if [ $? != 0 ]; then exit 1; fi
      gid=${GROUP_ID}
      sleep 5
      $CODIS_ADMIN --group-del --gid=${GROUP_ID} --addr=${KB_POD_FQDN}:9221
    }

    reblance() {
      $CODIS_ADMIN --rebalance --confirm
      if [ $? != 0 ]; then
          echo "Error: rebalance failed"
          exit 1
      fi
    }

    set_group_id
    set_instance_role
    set_codis_dashboard

    if [ $# -eq 1 ]; then
      case $1 in
      --help)
        echo "Usage: $0 [options]"
        echo "Options:"
        echo "  --help                show help information"
        echo "  --register-server     register server to dashboard"
        echo "  --remove-server       remove server from dashboard"
        exit 0
        ;;
      --register-server)
        set_instance_role
        wait_dashboard_running
        wait_server_running
        register_server
        exit 0
        ;;
      --remove-server)
        set_instance_role
        remove_server
        exit 0
        ;;
      --rebalance)
        wait_dashboard_running
        confirm_max_group
        wait_master_registered
        rebalance
        exit 0
        ;;
      *)
        echo "Error: invalid option '$1'"
        exit 1
        ;;
      esac
    fi

    kubeblocks_helm/pika-cluster/templates/cluster.yaml

    通过更改groupcount的值，部署一个新的 pika-group component，component ready 后会触发上面的自动rebalance脚本，启动一个 job ，执行 admin.sh –rebalance

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    spec:
      clusterDefinitionRef: pika  # ref clusterDefinition.name
      clusterVersionRef: pika-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} # ref clusterVersion.name
      terminationPolicy: {{ .Values.terminationPolicy }}
      affinity:
        {{- with .Values.topologyKeys }}
        topologyKeys: {{ . | toYaml | nindent 6 }}
        {{- end }}
      {{- with $.Values.tolerations }}
      tolerations: {{ . | toYaml | nindent 4 }}
      {{- end }}
      componentSpecs:
        {{- range $i := until (int .Values.groupCount) }} ## groupCount循环遍历创造出Pika-group
        - name: pika-group-{{ add ($i) 1 }} # user-defined
          componentDefRef: pika-group # ref clusterDefinition componentDefs.name
          monitor: {{ $.Values.monitor.enabled | default false }}
          enabledLogs: {{ $.Values.enabledLogs | toJson | indent 4 }}
          replicas: {{ add (int $.Values.slaveCount) 1 | default 2 }}
          serviceAccountName: {{ include "pika-cluster.serviceAccountName" $ }}
          switchPolicy:
            type: {{ $.Values.switchPolicy.type}}
          {{- with  $.Values.resources.pikaGroup }}
          resources:
            limits:
              cpu: {{ .limits.cpu | quote }}
              memory: {{ .limits.memory | quote }}
            requests:
              cpu: {{ .requests.cpu | quote }}
              memory: {{ .requests.memory | quote }}
          {{- end }}
          {{- if $.Values.persistence.enabled }}
          volumeClaimTemplates:
            {{- with  $.Values.persistence.pikaData }}
            - name: data # ref clusterdefinition components.containers.volumeMounts.name
              spec:
                storageClassName: {{ .storageClassName }}
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: {{ .size }}
            {{- end }}
          {{- end }}
        {{- end }}

  3. Verify the cluster status

    1
    kubectl port-forward svc/pika-cluster-codis-fe 8080

  4. Scale out

    1
    helm upgrade pika-cluster ./pika-cluster

    ​