搭建起本地的 DevOps 环境

目录

动机

自己作为独立开发者,也想体验那种写完代码效果就出来的感觉,不用又当个运维人员。

想当初 Spring Boot 程序,得手动编译,然后手动复制到目标机器,然后重启服务,可麻烦了。

因此,本地跑一套 DevOps 或者 GitOps 的系统应该会很有趣。

Untitled

方案

整体方案就是通过 CNCF 等的开源软件。

包括,但不限于:

  • microk8s
  • Argo CD:遵循声明式 GitOps 理念的持续交付(continuous delivery, CD)工具
  • Argo Workflows:CI 工具,开源的容器原生的工作流引擎,替代 Jenkins
  • GitLab

microk8s

windows 启动 3 台 hyper v 虚拟机安装

这个方式,是我在 mac 启动 3 台 vmware 虚拟机安装 之后尝试的。

原因是一台 mac 带不动了。

想起还有一台 windows 台式机,想拿来用用。

虽然官方 microk8s 的 windows 版就是快速启动 1 台 hyper v 虚拟机,但我还是想自己尝试手动创建。

使用的是 ubuntu 22.04.1 的 server 镜像。

Untitled

注意设置桥接之后,主机可能就上不了网了,去网络适配器配置,提示说 IP 已经分配给了图中 #2 的适配器。解决办法就是把那个适配器给删除!然后虚拟机和主机都能正常访问了!

Untitled

然后怎么安装,就是直接在 Ubuntu 里面操作了,请参见 mac 启动 3 台 vmware 虚拟机安装。

如果之后虚拟机 ping 不通主机,我想要不把 IP 与 MAC 绑定下?ARP 协议

VM虚拟机桥接模式下连接路由器与主机PING不通_Happy_Enger的博客-CSDN博客_vm主机ping不通桥接win7

mac 启动 3 台 vmware 虚拟机安装

如果遇到 Disk Pressure,注意进行虚拟机 Linux 磁盘扩容,从物理卷到逻辑卷。

在我学习 k8s 的时候做过,ubuntu 的 lts 版系统直接 snap 装。

就一条命令:sudo snap install microk8s --classic --channel=1.27/stable

  • 装完可能没启动起来,因为 k8s 必须设置科学上网代理:

  • sudo vim /var/snap/microk8s/current/args/containerd-env

    HTTP_PROXY=http://10.130.158.145:7890
    HTTPS_PROXY=http://10.130.158.145:7890
    NO_PROXY=10.1.0.0/16,10.152.183.0/24
    
    microk8s.stop
    microk8s.start
    microk8s.status
    
  • 看看 pod 情况 kubectl get po -n kube-system,如果有 pod 不正常,要查看:sudo kubectl describe pods metrics-server-5f8f64cb86-8fpp4 -n kube-system

  • 设置别名:

    sudo snap alias microk8s.kubectl kubectl

    sudo snap alias microk8s.ctr ctr

    sudo snap alias microk8s.helm helm

    sudo snap alias microk8s.helm3 helm3

  • 本地启动了 3 台 VMware 虚拟机,都装有 microk8s(克隆完,还得配置静态 IP + /etc/hosts + /etc/hostname 都得改)

    • 特别注意:静态IP 的配置在 /etc/netplan/00-xxx.yml 文件

    • 其中,gateway4 非常重要,配错了就上不了网了,连不上外部了

      Untitled

    • 我的 mbp 在 vmware 网络下地址为 192.168.124.1 ,而对应的网关其实是 192.168.124.**2** 这个地址在 mac 的 vmware fusion 还看不出来,我是之前在 windows 下的 vmware workstation 中看到的

      Untitled

    • 改 /etc/hosts + /etc/hostname:

    Untitled

    Untitled

组网

构造器一主,二工作的格局

  • 查看所有 Pods 运行情况:kubectl get pods --all-namespaces

  • 声明主节点:microk8s add-node,(选择性的将上面日志中输出的三种命令,分别在另外两台虚拟机 spud和 murphy中执行)此时 microk8s status 多了一行“datastore master nodes“

    Untitled

    • 如果报 “Connection failed. The hostname (c221) of the joining node does not resolve to the IP “10.130.159.221”. Refusing join (400).“ 说明得配置 /etc/hosts

      Untitled

    • 此外,如果 microk8s 权限不过,记得 sudo:microk8s join 10.130.158.191:25000/3c039d6cd09f44ecdd87d86e56d7e2cd/bd6ec9df0b3a --worker

  • 注意 token 是一次性的,若我们需要添加多个工作节点,需要重复在第一个节点中执行 microk8s add-node,获得带有不同的 token参数的命令。

    Untitled

  • 删除节点:kubectl get nodes,

  • 组网结束,去主节点:kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName --all-namespaces

    Untitled

  • dashboard :microk8s enable dns dashboard

  • 生成证书:docker run --rm -it -v pwd/ssl:/ssl -e "FOR_K8S=on" -e "CERT_DNS=*.yuhan.tech" -e "CERT_CN=yuhan" soulteary/certs-maker

    • 注意 Docker share 磁盘问题

      Untitled

  • 替换证书:kubectl -n kube-system delete secret kubernetes-dashboard-certs

    • 注意文件名不要带 * ,记得改过来 regex used for validation is ‘[-._a-zA-Z0-9]+’
    • 生成 secret:kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=ssl/yuhan.tech.crt --from-file=ssl/yuhan.tech.key
  • 编辑 K8s 配置,让 K8s 不再自动生成证书

    • kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

    • 找到 --auto-generate-certificates,将其注释掉,并添加我们刚刚导入的证书

      Untitled

  • 查看 certs 配置kubectl -n kube-system get secrets kubernetes-dashboard-certs -o yaml

  • 外部访问:microk8s dashboard-proxy

    • 用于登录的 token 获取:microk8s kubectl describe secret -n kube-system microk8s-dashboard-token

    Untitled

后序 ingress 配置:

Untitled

Warning: annotation “kubernetes.io/ingress.class” is deprecated, please use ‘spec.ingressClassName’ instead

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard
  namespace: kube-system
  annotations:
    # use the shared ingress-nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: public
  # https://kubernetes.io/docs/concepts/services-networking/ingress/
  # https://kubernetes.github.io/ingress-nginx/user-guide/tls/
  tls:
  - hosts:
    - k8s.yuhan.tech
    secretName: all-yuhan-tech-tls
  rules:
  - host: k8s.yuhan.tech
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

Untitled

常见问题

Node Capacity 是 110 ?能不能调大些?

大规模集群的注意事项

Kubernetes v1.26 单个集群支持的最大节点数为 5,000。 更具体地说,Kubernetes 旨在适应满足以下所有标准的配置:

  • 每个节点的 Pod 数量不超过 110
  • 节点数不超过 5,000
  • Pod 总数不超过 150,000
  • 容器总数不超过 300,000

从图形界面改(好像不生效)

Untitled

Update node’s pod capacity

https://github.com/kubernetes/kubernetes/blob/54706661ad72d62ea0b494112a74e0467093c9f4/cmd/kubelet/app/server.go#L317

又对应这篇文章:

The Kubernetes 110 pod limit per node

Untitled

/var/snap/microk8s/current/args/kubelet

Add line to the bottom (limit to 200 pods):

  • --max-pods=200
  • -resolv-conf=/run/systemd/resolve/resolv.conf (已经有了啊)

And restart the service

相当有效!!!!!

Untitled

登录能不能方便一点?

microk8s kubectl describe secret -n kube-system microk8s-dashboard-token

eyJhbGciOiJSUzI1NiIsImtpZCI6IllJVm5ZclljcDE5WUxxT1dMYS01d3FkbG44WEc4TVZrSzRyY1pOMDlmdFEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtaWNyb2s4cy1kYXNoYm9hcmQtdG9rZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImFiYWMxMGIwLTQ3MDYtNGU5OS05MjRjLWNhNThiNjQxYWQ5MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpkZWZhdWx0In0.GhXt1HsJ1QrNYhGyAHw67FucOFRxkFEzdVX6Sz-iWVjvVy9Es6zIip19DFQjN20KnL00Jki8vclGNqKiOfTpw9UbGjIUOknUwhbmmGjbHjIESKIGQKoOjYx3_q9FnOU8re03x_g74HN2vB-PAKRCpvOM4wMx-HZpCUIjWRnCDpaSsn8RfwgEMbDVa8Ccl0oANE5TuycCoLmnsxGxxBgtmzhFlOtKwYDaKM8TIJdc3CHcCefz2mrnUTsm55gzQCPpOiUS5iBfeIJOFDDoZZdw5PvyiBdTBxZAqFTTUb44CMybZCjyzjZhEwq9wykGQeWoKvyIW5VGZ-TFesZ2zGoKbA

设置 master 节点不参与调度

kubectl cordon haha

取消则是:kubectl uncordon haha

CoeDNS 已知问题

调试 DNS 问题

有些 Linux 发行版本(比如 Ubuntu)默认使用一个本地的 DNS 解析器(systemd-resolved)。 systemd-resolved 会用一个存根文件(Stub File)来覆盖 /etc/resolv.conf 内容, 从而可能在上游服务器中解析域名产生转发环(forwarding loop)。 这个问题可以通过手动指定 kubelet 的 --resolv-conf 标志为正确的 resolv.conf(如果是 systemd-resolved, 则这个文件路径为 /run/systemd/resolve/resolv.conf)来解决。 kubeadm 会自动检测 systemd-resolved 并对应的更改 kubelet 的命令行标志。

microk8s 已解决,但我想换上游 DNS为 8.8.8.8,参考:2022学习0616【K8S coredns日志报错】_qq_43264065的博客-CSDN博客kubectl edit cm coredns -n kube-system -o yaml,然后立马好使:

Untitled

kubectl logs --namespace=kube-system -l k8s-app=kube-dns

Untitled

Untitled

cert-manager

后期 gitlab 好像是需要的。

使用 cert-manager 签发免费证书 | kubernetes 学习笔记

microk8s enable cert-manager

采用 DNS 验证方式。

首先必须配置这个 DNS,登录你的 DNS 提供商后台,配置域名的 DNS A 记录,指向你需要证书的后端服务对外暴露的 IP 地址。(不管这个 IP 是局域网还是广域网)

Untitled

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    privateKeySecretRef:
      name: letsencrypt-account-key
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - dns01:
        cloudflare:
          email: my-cloudflare-acc@example.com # 替换成你的 cloudflare 邮箱账号,API Token 方式认证非必需,API Keys 认证是必需
          apiTokenSecretRef:
            key: api-token
            name: cloudflare-api-token-secret # 引用保存 cloudflare 认证信息的 Secret

Untitled

点到 My Profile > API Tokens > Create Token来创建 Token,选择 zone DNS

Cloudflare

Untitled

  • Permissions:
    • Zone - DNS - Edit
    • Zone - Zone - Read
  • Zone Resources:
    • Include - All Zones

Token 需要保存到 Secret 中:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager  # 需要在这个命名空间
type: Opaque
stringData:
  api-token: <API Token> # 粘贴 Token 到这里,不需要 base64 加密。

创建证书(单独颁发证书,其实是一种手动的方式,一般不这么使用,但这却能加深我们对这整套流程的理解。):

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-mydomain-com
  namespace: default
spec:
  dnsNames:
  - test.mydomain.com # 要签发证书的域名
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt # 引用 ClusterIssuer,指示采用 dns01 方式进行校验
  secretName: test-mydomain-com-tls # 最终签发出来的证书会保存在这个 Secret 里面

创建好证书 后,等一小会儿,我们可以 kubectl 查看是否签发成功(前提域名要创建好):

kubectl get certificate -A

如果 READY 为 False 表示失败,可以通过 describe 查看 event 来排查失败原因:

kubectl describe **cert** test-mydomain-com

更多的时候我们不需要手动创建 Certificate,只需要在创建 Ingress 时作好对应的注释,cert-manager 就将自动为我们创建。对应的 yaml 文件大致是长这样的:

Securing Ingress Resources

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/cluster-issuer: nameOfClusterIssuer
  name: myIngress
  namespace: myIngress
spec:
  rules:
  - host: example.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: myservice
            port:
              number: 80
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames
  - hosts:
    - example.com
    secretName: myingress-cert # < cert-manager will store the created certificate in this secret.

Untitled

创建泛域名证书

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: all-yuhan-tech-tls
spec:
  dnsNames:
  - yuhan.tech
  - "*.yuhan.tech"
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  secretName: all-yuhan-tech-tls

遇到问题了怎么排查?

Troubleshooting

kubectl get certificate

kubectl get certificaterequest

Untitled

Untitled

哦吼,再看 cluster issuer:

kubectl get clusterissuer

Untitled

Untitled

感觉又是代理的锅:

Untitled

果然,IP 又被挤掉了:

Untitled

把老 cert 删除,然后 k8s 会自动新建:

k delete cert k8s-yuhan-tech-tls -n kube-system

Untitled

大功告成!如果之前改过 /etc/hosts 的域名, 记得删除哦

Untitled

volume

得有个本地持久化卷!(但我感觉用 NFS 更好呢)

local 卷所代表的是某个被挂载的本地存储设备,例如磁盘、分区或者目录。

local 卷只能用作静态创建的持久卷。不支持动态配置。

👉注意,fdisk /dev/sdamkfs.xfs 之后需要 mount /dev/sda6 /data,但重启系统之后之前的 mount 就没了,因此需开机自动挂载!blkid 获取 UUID ,然后编写 /etc/fstab 文件。 详情查看 《Linux 速查表》有关磁盘管理的“开机自动挂载分区”一节。

NFS 服务器启动

Service - NFS | Ubuntu

sudo apt install nfs-kernel-server

👀注意各个节点上都得有 nfs-common,不然 mount 不上: sudo apt install nfs-common

sudo systemctl start nfs-kernel-server.service

把路径加到 /etc/exports 文件即可共享,该文件内容类似:

/srv     *(ro,sync,subtree_check)
/home    *.hostname.com(rw,sync,no_subtree_check)
/scratch *(rw,async,no_subtree_check,no_root_squash)

# "insecure"必须有,否则mac上的finder会报找不到服务器的错误。
/data *.yuhan.tech(rw,sync,no_subtree_check,no_root_squash,insecure)

sudo exportfs -a 使配置生效!

注意 *.hostname.com将匹配  foo.hostname.com但不匹配 foo.bar.my-domain.com

Untitled

no_root_squash adds a convenience to allow root-owned files to be modified by any client system’s root user

subtree_check and no_subtree_check enables or disables a security verification that subdirectories a client attempts to mount for an exported filesystem are ones they’re permitted to do so.

使用 service 命令无法看到 nfs 运行状态,毕竟它其实是内核应用:

NFS server active (exited)

mac 上试试:

showmount -e k8s.yuhan.tech

Untitled

mount: /Users/yuhanliu/data failed with 13,失败

但是另一个 ubuntu 可以(用umount取消挂载):

Untitled

本地测试也没问题:

sudo mount -t nfs localhost:/data /nfs

To check whether the kernel NFS server is actually running, run ps axf or htop and search for kthreads named “[nfsd]”. You can also check information found in /proc/fs/nfsd:

Untitled

nfs卷的内容在删除 Pod 时会被保存,卷只是被卸载。 这意味着 nfs卷可以被预先填充数据,并且这些数据可以在 Pod 之间共享。

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: registry.k8s.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /my-nfs-data
      name: test-volume
  volumes:
  - name: test-volume
    nfs:
      server: my-nfs-server.example.com
      path: /my-nfs-volume
      readOnly: true

建立一个 storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: haha-nfs
provisioner: example.com/external-nfs
parameters:
  server: nfs.yuhan.tech
  path: /data
  readOnly: "false"

创建一个 PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: haha-nfs-claim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 200Gi
  storageClassName: haha-nfs

然后可这么使用:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: mypd
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: haha-nfs-claim

但还需要 provisioner ,NFS 驱动呢。

所以一般我们不手动搞,只要有一个 NFS server 就好了。

NFS 驱动搞成 provisioner 的服务

存储类

nfs-subdir-external-provisioner/README.md at master · kubernetes-sigs/nfs-subdir-external-provisioner

Untitled

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner **--set nfs.server=nfs.yuhan.tech --set nfs.path=/data**


helm upgrade nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner -f nfs-subdir-external-provisioner-0.yml

helm upgrade nfs-subdir-external-provisioner-1 nfs-subdir-external-provisioner/nfs-subdir-external-provisioner -f nfs-subdir-external-provisioner-1.yml

自动就创建了存储类、pod:

kubectl get storageclass

Untitled

Untitled

PostgreSQL

Kubernetes环境PostgreSQL部署与应用

前提:上面的 nfs-client 弄好

helm repo add bitnami https://charts.bitnami.com/bitnami

# 创建密码,按指定的规则随机生成密码
kubectl create secret generic postgresql-secret \
  --from-literal=postgres-password=$(openssl rand -base64 18)

# 安装,nfs 这就用上了
~~helm upgrade --install \
  --set auth.existingSecret=postgresql-secret \
  --set primary.persistence.enabled=true \
  --set primary.persistence.size=50Gi \
  --set primary.persistence.storageClass=nfs-client \
  postgresql bitnami/postgresql~~

helm show values bitnami/postgresql > postgresql.yml

helm install my-postgresql bitnami/postgresql -f postgresql.yml

配置文件主要修改了:

  • auth.existingSecret
  • primary.persistence.size: 50Gi
  • primary.persistence.storageClass: “nfs-client”
  • primary.persistence.existingClaim: “postgresql-pvc”
  • primary.service.type: NodePort
  • primary.service.nodePorts.postgresql: “30101”

自己要提前创建 PVC:

apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: postgresql-pvc 
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 50Gi

测试外部连接

Untitled

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    my-postgresql.default.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace default my-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

To connect to your database run the following command:

    kubectl run my-postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:15.3.0-debian-11-r24 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host my-postgresql -U postgres -d postgres -p 5432

    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"

To connect to your database from outside the cluster execute the following commands:

    export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
    export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services my-postgresql)
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host $NODE_IP --port $NODE_PORT -U postgres -d postgres

按照上面的命令,成功从外部连上:

Untitled

用户和数据库的创建

额,我比较喜欢用图形界面。

Untitled

# 创建密码,按指定的规则随机生成密码
kubectl create secret generic postgresql-gitlab-secret \
  --from-literal=postgres-password=$(openssl rand -base64 18)

# 查看密码
kubectl get secret postgresql-gitlab-secret -o jsonpath="{.data.postgres-password}" | base64 -d

就用上面的密码,创建一个 gitlab 用户和 gitlabhq_production 数据库:

CREATE ROLE "gitlab" CREATEDB CREATEROLE LOGIN REPLICATION BYPASSRLS PASSWORD 'tO+Qo4jvFoyDGpgef44j69+r';

CREATE DATABASE "gitlabhq_production"
WITH
  OWNER = "gitlab"
;

测试 StatefulSet 卷绑定的不变性

我们删除这个应用再用之前的 pvc 和 secret 重新部署:

helm uninstall my-postgresql

helm install my-postgresql bitnami/postgresql -f postgresql.yml

然后原来创建的一切都在,真的太好了:

Untitled

Redis

前提有 nfs-client 的 storageClass,然后创建一个 pvc:

apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: redis-pvc 
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 20Gi

---
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: redis-pvc-replica
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 40Gi

然后修改默认的配置,主要加上:existingClaim: "redis-pvc"storageClass: "nfs-client",以及 replica 的要加上:existingClaim: "redis-pvc-replica"storageClass: "nfs-client"

helm repo add bitnami https://charts.bitnami.com/bitnami

~~helm install my-redis bitnami/redis --version 17.8.0  --set persistence.storageClass=nfs-client~~

helm delete my-redis

~~~~helm show values bitnami/redis > redis.yml

helm install my-redis bitnami/redis -f redis.yml

Untitled

my-redis-master.default.svc.cluster.local for read/write operations (port 6379)
my-redis-replicas.default.svc.cluster.local for read-only operations (port 6379)

export REDIS_PASSWORD=$(kubectl get secret --namespace default my-redis -o jsonpath="{.data.redis-password}" | base64 -d)

kubectl run --namespace default redis-client --restart='Never'  --env REDIS_PASSWORD=$REDIS_PASSWORD  --image docker.io/bitnami/redis:7.0.8-debian-11-r13 --command -- sleep infinity

kubectl exec --tty -i redis-client \
   --namespace default -- bash

REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h my-redis-master
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h my-redis-replicas

kubectl port-forward --namespace default svc/my-redis-master 6379:6379 &
    REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h 127.0.0.1 -p 6379

Untitled

然后就可以装 GitLab 了。

通过 helm delete my-redis 命令可卸载。

后期

改 master 的 NodePort 访问。

k edit svc my-redis-master (是 service 啊喂,不是 statefulset)

Untitled

添加用户,因为有人不是用的 “default” 用户名,而是 “redis”:

Untitled

> ACL GENPASS 64
5fd1203ac95a0424

> ACL SETUSER redis on >5fd1203ac95a0424 ~* +@all
OK

Redis 7 中的用户权限管理功能

Gitea

helm3 repo add bitnami https://charts.bitnami.com/bitnami

helm install my-gitea bitnami/gitea --version 0.3.6

helm show values bitnami/gitea > gitea.yml

helm install my-gitea bitnami/gitea -f gitea.yml
1. Get the Gitea URL:

  Or running:

  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services my-gitea)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo "Gitea URL: http://$NODE_IP:$NODE_PORT/"

2. Get your Gitea login credentials by running:

  echo Username: admin
  echo Password: $(kubectl get secret --namespace default my-gitea -o jsonpath="{.data.admin-password}" | base64 -d)
  • 注意 ReadWriteMany 不仅要在 pvc 中配置,gitea.yml 中也要

apiVersion: v1 
kind: PersistentVolumeClaim
metadata: 
  name: gitea-pvc
spec: 
  accessModes: 
  - ReadWriteMany
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 10Gi

------
persistence:
  ## @param persistence.enabled Enable persistence using PVC
  ##
  enabled: true
  ## @param persistence.storageClass PVC Storage Class for Gitea volume
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: ""
  ## @param persistence.accessModes PVC Access Mode for Gitea volume
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  ##
  accessModes:
    - ReadWriteMany
  ## @param persistence.size PVC Storage Request for Gitea volume
  ##
  size: 10Gi
  ## @param persistence.dataSource Custom PVC data source
  ##
  dataSource: {}
  ## @param persistence.existingClaim A manually managed Persistent Volume Claim
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  ##
  existingClaim: "gitea-pvc"
  ## @param persistence.hostPath If defined, the gitea-data volume will mount to the specified hostPath.
  ## Requires persistence.enabled: true
  ## Requires persistence.existingClaim: nil|false
  ## Default: nil.
  ##
service:
  ## @param service.type Kubernetes Service type
  ##
  type: NodePort
	nodePorts:
	    http: 30005
	    ssh: 30006
	  ## @param service.externalTrafficPolicy Enable client source IP preservation
	  ## ref https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
	  ##
	  externalTrafficPolicy: Local

minio

这是个对象存储服务器,安装 GitLab 需要。

MinIO(R) is an object storage server, compatible with Amazon S3 cloud storage service, mainly used for storing unstructured data (such as photos, videos, log files, etc.).

helm show values bitnami/minio  > minio-helm-values.yml

helm install my-minio bitnami/minio -f minio-helm-values.yml 
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: minio-pvc 
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 60Gi

主要修改了:

  • persistence.storageClass: “nfs-client”

  • persistence.size: 50Gi

  • persistence.existingClaim: “minio-pvc”

  • ingress.enabled: false

  • ingress.enabled: true

  • ingress.apiVersion: “networking.k8s.io/v1"

  • ingress.ingressClassName: “public”

  • ingress.hostname: objectstorage.yuhan.tech

  • ingress.annotations: {

    cert-manager.io/cluster-issuer: letsencrypt }

  • ingress.tls: true


** Please be patient while the chart is being deployed **

MinIO&reg; can be accessed via port  on the following DNS name from within your cluster:

   my-minio.default.svc.cluster.local

To get your credentials run:

   export ROOT_USER=$(kubectl get secret --namespace default my-minio -o jsonpath="{.data.root-user}" | base64 -d)
   export ROOT_PASSWORD=$(kubectl get secret --namespace default my-minio -o jsonpath="{.data.root-password}" | base64 -d)

To connect to your MinIO&reg; server using a client:

- Run a MinIO&reg; Client pod and append the desired command (e.g. 'admin info'):

   kubectl run --namespace default my-minio-client \
     --rm --tty -i --restart='Never' \
     --env MINIO_SERVER_ROOT_USER=$ROOT_USER \
     --env MINIO_SERVER_ROOT_PASSWORD=$ROOT_PASSWORD \
     --env MINIO_SERVER_HOST=my-minio \
     --image docker.io/bitnami/minio-client:2023.2.16-debian-11-r1 -- admin info minio

To access the MinIO&reg; web UI:

- Get the MinIO&reg; URL:

   echo "MinIO&reg; web URL: http://127.0.0.1:9001/minio"
   kubectl port-forward --namespace default svc/my-minio 9001:9001

手动暴露服务

它自动生成的 Ingress 好像有点问题,我们自己弄一个:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: default-ingress
  namespace: default
  annotations:
    # use the shared ingress-nginx
    # kubernetes.io/ingress.class: public
    # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: public  # 最新用法,不是在注解里面了
  # https://kubernetes.io/docs/concepts/services-networking/ingress/
  # https://kubernetes.github.io/ingress-nginx/user-guide/tls/
  tls:
  - hosts:
    - objectstorage.yuhan.tech
    - objectstorage-web.yuhan.tech
    secretName: objectstorage-yuhan-tech-tls
  rules:
  - host: objectstorage.yuhan.tech
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: my-minio
            port:
              number: 9000  # 这是 API 不是网页,网页是 9001 需要另外配置
  - host: objectstorage-web.yuhan.tech
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: my-minio
            port:
              number: 9001

排查证书问题:https://cert-manager.io/docs/troubleshooting/acme/

kubectl get certificate -A

kubectl describe cert objectstorage-yuhan-tech-tls

k delete cert objectstorage-yuhan-tech-tls

kubectl get certificaterequestk get cr

k describe order objectstorage-yuhan-tech-tls-cm84m-1673293443

k describe challenge objectstorage-yuhan-tech-tls-cm84m-1673293443-876957053

Untitled

Untitled

说明 CNAME 还没生效,必须等!我们去 cloudflare 查看:

Untitled

确实如此,剩下的交给时间,你看这就好了!

Untitled

环境变量配置(必须)

extraEnvVars:  # 必填,否则地址就是 localhost 了 https://github.com/minio/console/issues/1632
  - name: MINIO_SERVER_URL
    value: https://objectstorage.yuhan.tech
  - name: MINIO_BROWSER_REDIRECT_URL
    value: https://objectstorage.yuhan.tech

Ingress 代理:最大上传 Size 配置(必须)

Annotations - NGINX Ingress Controller

ingress.kubernetes.io/proxy-body-size: 10000m  # 老 k8s 版本用
nginx.ingress.kubernetes.io/proxy-body-size: 10000m  # 最新版本使用

同时也要注意 harbor 的相关 size 配置!minio 也得配置!

最终才完美:

Untitled

外部客户端连接

关于外部连接,minio 与 AWS S3 完全兼容:

AWS CLI with MinIO Server — MinIO Object Storage for Linux

AWS CLI is a unified tool to manage AWS services. It is frequently the tool used to transfer data in and out of AWS S3. It works with any S3 compatible cloud storage service.

先登录网页:

用户名:echo $(kubectl get secret --namespace default my-minio -o jsonpath="{.data.root-user}" | base64 -d)

密码:echo $(kubectl get secret --namespace default my-minio -o jsonpath="{.data.root-password}" | base64 -d)

Untitled

然后创建 AccessToken:

H3vCj0L3LIBHWvbh

LevG9vCPLvNguatIMJfQdGZhKRUgNl0h

之后还可以设置服务的 region 是 cn:

Untitled

然后 GitLab 的 S3 存储可以有了,所以 Secret 参考 S3 进行配置, Secret 的 key 对应一个 yaml 文件, 文件参考为:

Object storage | GitLab

provider: AWS
# Specify the region
region: cn
# Specify access/secret keys
aws_access_key_id: H3vCj0L3LIBHWvbh
aws_secret_access_key: LevG9vCPLvNguatIMJfQdGZhKRUgNl0h
# The below settings are for S3 compatible endpoints
#   See https://docs.gitlab.com/ee/administration/job_artifacts.html#s3-compatible-connection-settings
aws_signature_version: 4
host: objectstorage.yuhan.tech
endpoint: "https://objectstorage.yuhan.tech"
path_style: true
## https://docs.gitlab.com/charts/charts/globals#lfs-artifacts-uploads-packages-external-mr-diffs-and-dependency-proxy
object_store:
  enabled: true
  proxy_download: true
  storage_options: {}
    # server_side_encryption:
    # server_side_encryption_kms_key_id
  connection:
    secret: **minio-storage-secret**
    key: **connection

# 记得创建对应的桶 git-lfs gitlab-artifacts gitlab-uploads gitlab-packages gitlab-backups

# When consolidated object storage is enabled, for each item `bucket` must be specified and the `connection` must be empty. Check the following object storage configuration(s): artifacts,lfs,uploads,packages**

Untitled

kubectl create secret generic **minio-storage-secret** --from-file=**connection**=./minio-storage-secret.ym

已经不建议用命令式,建议声明式!

尝试用 mac 的 client 连接:

Untitled

成功!

Argo CD

argo-cd 5.23.1 · argoproj/argo

按照上面的 INSTALL 提示:

helm repo add argo https://argoproj.github.io/argo-helm

helm install my-argo-cd argo/argo-cd --version 5.23.1

# 可选自行设置,包括 ingress 的配置等
helm show values argo/argo-cd > argo.yml
helm install my-argo-cd argo/argo-cd -f argo.yml

我自行设置主要修改了:

  • ingress 配置:Enable an ingress resource for the Argo CD server

    • 使得使用泛域名证书 (之后 都自建 ingress )

    • 使得使用 argo.yuhan.tech/cd 路径访问,这需要参考下面这个注释:

      Untitled

  • externalRedis 配置:External Redis parameters,注意禁用自带的

NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/my-argo-cd-argocd-server -n default 8080:443

    and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
      - Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
      - Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts

After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n default get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)

在这里 add 应该 pon 一下,即开个 export 代理。

# 代理设置
url=socks5://127.0.0.1:7890
# 如果代理失效的话直接运行 poff 即可断开 proxy
alias poff='unset http_proxy;unset https_proxy'
# 快捷方式打开
alias pon='export http_proxy=$url; export https_proxy=$url'

Untitled

上面有命令做端口转发,但我偏要修改 yml 文件,用 NodePort 而不是 ClusterIP

kubectl edit svc my-argo-cd-argocd-server

Untitled

或者自建 Ingress (一定要是 443,因为• 80 - HTTP (redirects to HTTPS)):

Ingress Configuration - Argo CD - Declarative GitOps CD for Kubernetes

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: default-tls-ingress
  namespace: default
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    # cert-manager.io/cluster-issuer: letsencrypt
spec:
  # https://kubernetes.io/docs/concepts/services-networking/ingress/
  # https://kubernetes.github.io/ingress-nginx/user-guide/tls/
  tls:
  - hosts:
    - argo.yuhan.tech
    secretName: all-yuhan-tech-tls
  rules:
  - host: argo.yuhan.tech
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: my-argo-cd-argocd-server
            port:
              number: 443

访问 https 网页:

Untitled

用户名:admin

登录密码:kubectl -n default get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

第一件事改密码:

Untitled

Argo Workflows

argo-workflows 0.22.12 · argoproj/argo

按照上面的 INSTALL 指示来安装:

helm repo add argo https://argoproj.github.io/argo-helm

~~helm install my-argo-workflows argo/argo-workflows --version 0.22.12~~

helm show values argo/argo-workflows > argo-workflows.yml

helm install my-argo-workflows argo/argo-workflows -f argo-workflows.yml

Untitled

上面虽然有命令 kubectl --namespace default get services -o wide | grep my-argo-workflows-server,但是无法外部访问。

同 Argo CD ,我们配置下 NodePort。

kubectl edit svc my-argo-workflows-server

Untitled

注意此时配置的是 HTTP 端口,用 http 访问。

Untitled

登录有三种方式 SSO 我肯定不会配置,那么就 client 模式,需要 token:

https://argoproj.github.io/argo-workflows/access-token/#token-creation

方法一,进入容器运行 argo auth token 命令:

Untitled

方法二,添加用户啥的就如链接所说,好麻烦

直接粘贴 Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjZEaHJ2ZzRfdkctNFdaN2c3MUJ6VV9LTWhoV3VKd29ucy1BdnAtS1B5LXMifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcwODc4MDI0MiwiaWF0IjoxNjc3MjQ0MjQyLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJwb2QiOnsibmFtZSI6Im15LWFyZ28td29ya2Zsb3dzLXNlcnZlci03NjY1ODQ1OTk1LW05N201IiwidWlkIjoiZTg1ZTBiOGUtZDRhYy00OWI3LTk1ZTEtNDFjYjZjOGFiNjk4In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJteS1hcmdvLXdvcmtmbG93cy1zZXJ2ZXIiLCJ1aWQiOiJjMzkyODg3NC00MzVmLTQ4MDAtODRhMC0xNmVjMmY1YTdmZjgifSwid2FybmFmdGVyIjoxNjc3MjQ3ODQ5fSwibmJmIjoxNjc3MjQ0MjQyLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpteS1hcmdvLXdvcmtmbG93cy1zZXJ2ZXIifQ.J2EoxuIvWj6F-bCKnpDoWiBgaezNm4-pSkVlC9kIIJUbkzk6GgjcL7toxzFNOVK129kXJxtD1-5h_9mBTeJRfA47D0hOZPD2ZWlg3PbvpzR-WgKK2afuJQfrJo9kDuLnDnZtEdi09bIWu5iNcVPYF0nX2BKR4ctxn1QSpMD8zLtKhpIAnEWMXyux5dUuXmWy0s5a5Wgug8MpmudNpqvTu_y-kpJrWT9T6ALW5MGLyZfPuBcMgoFO0D5g0wJVey7XU1nDc1oRMk8XeDy05rnHd7LSpzuOcf7NYKo9upqogDx7R6ahBnkwfSgklbn7_jlqhnlJ7Gz7o4nVVojIcg9_ng

这一串就能登录:

Untitled

Quick Start - Argo Workflows - The workflow engine for Kubernetes

选择 CI/CD 进入指导页面,提示下载 Argo Workflows CLI ,release 页面有介绍怎么弄,不赘述了,详见 https://github.com/argoproj/argo-workflows/releases

提交一个实例 workflow:

argo submit --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml

但报错:invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

解决:https://github.com/canonical/microk8s/issues/2022 sudo microk8s.kubectl config view --raw > $HOME/.kube/config

Untitled

argo list

argo auth token 现在也能直接用了:

Bearer bUNlSTM1REZJaEpOSWVEeWJyS1pqV25oWEI4T1M4SDB6MEwzaXFsUUdqYz0K

清空 namespace 的 undefined 就可以看到之前部署的 workflow 了:

Untitled

后期

show values 配置,主要修改:

  • ingress
  • artifactRepository 的 minio s3 配置

Configuring Your Artifact Repository - Argo Workflows - The workflow engine for Kubernetes

GitLab

前提:Redis、PostgreSQL

GitLab chart prerequisites | GitLab

Deploy the GitLab Helm chart | GitLab

根据上面的 INSTALL 提示进行安装:

helm repo add gitlab http://charts.gitlab.io/  # 还是得 pon 开代理哦

~~helm install my-gitlab gitlab/gitlab --version 6.9.1  # 但是失败了,得看文档了

helm repo update
helm upgrade --install gitlab gitlab/gitlab \
  --timeout 600s \
  --set global.hosts.domain=gitlab.yuhan.tech \
  --set global.hosts.externalIP=10.130.158.191 \
  --set certmanager-issuer.email=1106741606@qq.com \
  --set postgresql.image.tag=15.2.0~~

GitLab Helm chart deployment options | GitLab

使用 Helm 安装 Gitlab 时, 默认的配置会安装所有的依赖, 但是对于已有数据库等功能的场景来说, 这样做并不是很合适, 所以需要调整 values.yaml 文件让 Gitlab 使用已有的外部依赖.

使用 Helm 部署 Gitlab 家庭实例

# 导出默认配置到文件
helm show values gitlab/gitlab > gitlab.yml

# 修改这个配置
# 过程中注意设置数据库用户、对象存储等等
# 再安装

helm install my-gitlab gitlab/gitlab --version 6.9.1 -f gitlab.yml

关闭这下依赖的安装:install: false

certmanager

Ingress

Redis Configure charts using globals | GitLab

Postgresql

Untitled

Untitled

配置文件主要做了以下修改

  • global.hosts.domain: gitlab.yuhan.tech

  • global.psql.password.useSecret: true

  • global.psql.password.secret: postgresql-gitlab-secret # 之前创建数据库生成过

  • global.psql.password.key: postgres-password

  • global.psql.host: my-postgresql.default.svc.cluster.local

  • global.psql.port: 5432

  • global.psql.username: gitlab # 之前的

  • global.psql.database: gitlabhq_production # 之前的

  • global.redis.password.secret: my-redis

  • global.redis.password.key: redis-password

  • global.redis.host: my-redis-master.default.svc.cluster.local

  • global.time_zone: Asia/Shanghai

  • certmanager-issuer.email: dfface@sina.com

  • certmanager.installCRDs: false

  • certmanager.install: false

  • nginx-ingress.enabled: false

  • redis.install: false

  • redis.existingSecret: my-redis

  • redis.existingSecretKey: redis-password

  • postgresql.install: false

  • global.minio.enabled: false

  • global.appConfig.object_store.enabled: true

  • global.appConfig.object_store.connection.secret: minio-storage-secret

  • global.appConfig.object_store.connection.key: connection

  • global.ingress.configureCertmanager: false

  • ingress 全部禁用,后期我们自己搞(自己配不太行啊,配不准,只配一个不行)

    # 就用它自带的
    ingress:
        apiVersion: ""
        configureCertmanager: false
        class: public
        provider: nginx    # class:
        annotations: {}
        enabled: true
        tls: 
          enabled: true
          secretName: all-yuhan-tech-tls
        path: /
        pathType: Prefix
    
  • gitlab-runner. install: false

  • global.edition: ce # md 默认居然是企业版……我要社区版

对象存储配置

GitLab 依靠对象存储来存储 Kubernetes 中的高可用持久数据。默认情况下,名为 minio  的 S3 兼容存储解决方案与 chart 一起部署,但对于生产环境质量的部署,我们建议使用托管对象存储解决方案,如 Google Cloud Storage 或 AWS S3。

在安装 minio(它是 S3 兼容的) 时有提到:

Object storage | GitLab

大功告成

=== NOTICE
The minimum required version of PostgreSQL is now 12. See https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/doc/installation/upgrade.md for more details.

=== NOTICE
You've installed GitLab Runner without the ability to use 'docker in docker'.
The GitLab Runner chart (gitlab/gitlab-runner) is deployed without the `privileged` flag by default for security purposes. This can be changed by setting `gitlab-runner.runners.privileged` to `true`. Before doing so, please read the GitLab Runner chart's documentation on why we
chose not to enable this by default. See https://docs.gitlab.com/runner/install/kubernetes.html#running-docker-in-docker-containers-with-gitlab-runners
Help us improve the installation experience, let us know how we did with a 1 minute survey:https://gitlab.fra1.qualtrics.com/jfe/form/SV_6kVqZANThUQ1bZb?installation=helm&release=15-9

=== NOTICE
kas:
    The configuration of `gitlab.kas.privateApi.tls.enabled` has moved. Please use `global.kas.tls.enabled` instead.
    Other components of the GitLab chart other than KAS also need to be configured to talk to KAS via TLS.
    With a global value the chart can take care of these configurations without the need for other specific values.

排错

在Kubernetes上部署Gitlab集群 - 哈希

  • 第一一定手动 Ingress
  • 第二不安装 runner,我貌似在官方文档看过 helm 部署的 runner 有问题来着,但是我已经找不到链接了

再提示 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims.

说明 pvc 没有绑定,这边建议手动建立,跟他们同名的:

apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: my-gitlab-prometheus-server
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 50Gi

---
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: repo-data-my-gitlab-gitaly-0
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 50Gi

PVC 它不会自动删除,还留下来了!这就是 statefulset 卷绑定可知!

登录

账号:root

密码:echo $(kubectl get secret --namespace default my-gitlab-gitlab-initial-root-password -o jsonpath="{.data.password}" | base64 -d)

页面错误

说明你还有两个 ingress 没配!

rules:
  - host: kas.yuhan.tech
    http:
      paths:
      - backend:
          service:
            name: my-gitlab-kas
            port:
              number: 8154
        path: /k8s-proxy/
        pathType: Prefix
      - backend:
          service:
            name: my-gitlab-kas
            port:
              number: 8150
        path: /
        pathType: Prefix

- host: registry.yuhan.tech
    http:
      paths:
      - backend:
          service:
            name: my-gitlab-registry
            port:
              number: 5000
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - registry.yuhan.tech
    secretName: all-yuhan-tech-tls

- host: gitlab.yuhan.tech
    http:
      paths:
      - backend:
          service:
            name: my-gitlab-webservice-default
            port:
              number: 8181
        path: /
        pathType: Prefix

Untitled

Untitled

后期

我还尝试用过中文版 极狐:

helm install my-gitlab gitlab-jh/gitlab -f gitlab-jh.yml

后期我还开启了 runner ,然后更新配置,丝滑得很:

helm **upgrade** my-gitlab gitlab-jh/gitlab -f gitlab-jh.yml

Untitled

取消使用 gitlab 的 runner 、registry、prometheus,因为没用……

开启 SSH 克隆

由于默认 my-gitlab-gitlab-shell 也是用的 22 端口,不能直接映射到主机的 22 端口。

所以,我们映射到 30122 端口。

然后再 clone 的时候,填完整地址:

  • ssh:// 开头
  • gitlab.yuhan.tech**:30122** 表明端口
  • /kgd/ 是群组名
git clone **ssh://**git@gitlab.yuhan.tech**:30122/kgd**/dataacquire.git

在 ssh key 的配置上:

Host gitlab
  User git
  Hostname gitlab.yuhan.tech:30122
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/id_ed25519_gitlab_dfface

ssh -T 你仍然会失败:

Untitled

但仓库已经可 clone:

Untitled

git remote get origin

Untitled

Untitled

添加仓库,也要这样写,不能是它提示的那种:

git remote add origin ssh://git@gitlab.yuhan.tech:30122/kgd/umlfigure.git

如果 ssh 方式实在连不上,请改用 http 方式

https://gitlab.yuhan.tech/kgd/dataacquire.git

Jenkins Webhook

显示不允许访问本地的

Untitled

Untitled

https://gitlab.com/gitlab-org/gitlab/-/issues/26845

Untitled

But returned 403

Untitled

Gitlab Jenkins Webhook executed successfully but returned HTTP 403 (Authentication required)

注意,配置 Jenkins 任务的时候,要选择高级,然后生成 Secret Token:

But I already fixed it: I simply added the authentication parameters in the url like this: https://USERNAME:PASSWORD@Serverjenkins.com/github-webhook

Untitled

我新建了 gitlab 用户。

SonarQube

Untitled

暂不上

Hashicorp Vault

暂不上

Istio

暂不上

FAQ

DNS 解析异常

如果是 Ubuntu Desktop 默认会用 NetworkManager 来管理网络。

使用 nmcli 来设置 DNS:

sudo nmcli con

Untitled

sudo nmcli con mod "有线连接 1" ipv4.dns "114.114.114.114 8.8.8.8”

service NetworkManager restart

RocketMQ

https://github.com/itboon/rocketmq-helm

# 集群名
clusterName: "cluster-production"

image:
  repository: "apache/rocketmq"
  tag: "4.9.4"

broker:
  # 3 个 master 节点,每个 master 具有1个副节点,共6个 broker 节点
  size:
    master: 1
    replica: 1

  persistence:
    enabled: true
    size: 8Gi
    storageClass: nfs-client

  # 主节点资源分配
  master:
    brokerRole: ASYNC_MASTER
    jvmMemory: " -Xms4g -Xmx4g -Xmn1g "
    resources:
      limits:
        cpu: 4
        memory: 12Gi
      requests:
        cpu: 200m
        memory: 6Gi
  
  # 副节点资源分配
  replica:
    jvmMemory: " -Xms1g -Xmx1g -Xmn256m "
    resources:
      limits:
        cpu: 4
        memory: 8Gi
      requests:
        cpu: 50m
        memory: 2Gi

nameserver:
  replicaCount: 3

  # 修改 nameserver 容器镜像地址和版本
  #image:
  #  repository: "apacherocketmq/rocketmq-nameserver"
  #  tag: "4.5.0-alpine-operator-0.3.0"

  resources:
    limits:
      cpu: 4
      memory: 8Gi
    requests:
      cpu: 50m
      memory: 1Gi
  
  persistence:
    enabled: true
    size: 8Gi
    #storageClass: gp2

dashboard:
  enabled: true
  replicaCount: 1

  ingress:
    enabled: false
    className: "nginx"
    hosts:
      - host: rocketmq.yuhan.tech
        paths:
          - path: /
            pathType: ImplementationSpecific
    #tls:
    #  - secretName: example-com-tls
    #    hosts:
    #      - rocketmq.example.com
- host: rocketmq.yuhan.tech
    http:
      paths:
      - backend:
          service:
            name: my-rocketmq-dashboard
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific

helm install my-rocketmq -f examples/production.yaml ./charts/rocketmq****

兼容官方镜像:https://hub.docker.com/r/apache/rocketmq

你得等等才能访问网页:

Untitled

MongoDB

helm show values bitnami/mongodb > mongodb-helm-values.yml

  • 主要是 persistence
apiVersion: v1 
kind: PersistentVolumeClaim 
metadata: 
  name: mongodb-pvc 
spec: 
  accessModes: 
  - ReadWriteOnce 
  storageClassName: nfs-client 
  resources:  
    requests: 
      storage: 20Gi
** Please be patient while the chart is being deployed **

MongoDB&reg; can be accessed on the following DNS name(s) and ports from within your cluster:

    my-mongodb.default.svc.cluster.local

To get the root password run:

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default my-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 -d)

To connect to your database, create a MongoDB&reg; client container:

    kubectl run --namespace default my-mongodb-client --rm --tty -i --restart='Never' --env="MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD" --image docker.io/bitnami/mongodb:6.0.4-debian-11-r10 --command -- bash

Then, run the following command:
    mongosh admin --host "my-mongodb" --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/my-mongodb 27017:27017 &
    mongosh --host 127.0.0.1 --authenticationDatabase admin -p $MONGODB_ROOT_PASSWORD

后期去重,new bing 可太好用了:

Untitled

Neo4j

helm repo add neo4j-helm-charts https://neo4j.github.io/helm-charts/

helm show values neo4j-helm-charts/neo4j > neo4j-helm-values.yml

  • 主要是 volume 的持久化

helm install my-neo4j neo4j-helm-charts/neo4j -f neo4j-helm-values.yml

helm upgrade my-neo4j neo4j-helm-charts/neo4j -f neo4j-helm-values.yml

Thank you for installing neo4j.

Your release "my-neo4j" has been installed
 .

The neo4j user's password has been set to "YmkGAFYzieUlA8".To view the progress of the rollout try:

  $ kubectl rollout status --watch --timeout=600s statefulset/my-neo4j

Once rollout is complete you can log in to Neo4j at "neo4j://my-neo4j.default.svc.cluster.local:7687". Try:

  $ kubectl run --rm -it --image "neo4j:5.5.0" cypher-shell \
     -- cypher-shell -a "neo4j://my-neo4j.default.svc.cluster.local:7687" -u neo4j -p "or6saMeRDcZyL1"

Graphs are everywhere!

WARNING: Passwords set using 'neo4j.password' will be stored in plain text in the Helm release ConfigMap.
Please consider using 'neo4j.passwordFromSecret' for improved security.

修改 svc 为 NodePort:

Untitled

Untitled

Untitled

密码这样看:

k get secret my-neo4j-auth -o jsonpath={.data.NEO4J_AUTH} | base64 --decode

ETL 工具备份

在 mac 上有:

brew install cypher-shell

我看文档还有 neo4j-admin 工具呢

如何在 k8s 中备份数据库:

https://community.neo4j.com/t/how-to-backup-a-db-running-in-kubernetes/16724/5

kubectl run -it --rm neo4j-utils
--image=neo4j:4.0.3 --restart=Never --namespace neo4j --command -- bash

./bin/neo4j-admin

用 Desktop 的 ETL 工具好像很不错:

Untitled

Untitled

安装 data science 库

本来这个库是需要手动下载放到 plugins 文件夹下的,但是 Docker 的话可配置环境变量,自动下载。

Neo4j Docker - Neo4j Graph Data Science

docker run -it --restart always \
  --publish=7474:7474 --publish=7687:7687 \
  --user="$(id -u):$(id -g)" \
  -e NEO4J_AUTH=none \
  --env NEO4JLABS_PLUGINS='["graph-data-science"]' \
  neo4j:4.4

用 helm 的话(我手动 neo4j stop 之后它再自动重启就有了!可能需要时间下载!):

Untitled

helm upgrade my-neo4j neo4j-helm-charts/neo4j -f neo4j-helm-values.yml

Untitled

Harbor

容器存储服务。

~~helm repo add bitnami https://charts.bitnami.com/bitnami

helm show values bitnami/harbor > harbor.yml

helm install my-harbor bitnami/harbor -f harbor.yml

helm upgrade my-harbor bitnami/harbor -f harbor.yml~~

helm repo add harbor https://helm.goharbor.io
helm show values harbor/harbor > harbor.yml
helm install my-harbor  harbor/harbor -f harbor.yml

主要修改了:

  • s3 的存储、persistence

  • External PostgreSQL: 创建用户 harbor 密码 echo $(openssl rand -base64 18)、创建系列数据库

    ; 举例
    CREATE DATABASE "harbor_notary_signer"
    WITH
    OWNER = "harbor"
    
  • External Redis: 得新建一个账户 redis:ACL SETUSER redis on >5fd1203ac95a0424 ~* +@all,密码就是 5fd1203ac95a0424

    @lebseu @darklight147 are you using the default user or a custom ACL user?If you’re using the default user, make sure the set the username to “default” or just leave it empty.

  • ingress

** Please be patient while the chart is being deployed **

1. Get the Harbor URL:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace default -w my-harbor'
    export SERVICE_IP=$(kubectl get svc --namespace default my-harbor --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
    echo "Harbor URL: http://$SERVICE_IP/"

2. Login with the following credentials to see your Harbor application

  echo Username: "admin"
  echo Password: $(kubectl get secret --namespace default my-harbor-core-envvars -o jsonpath="{.data.HARBOR_ADMIN_PASSWORD}" | base64 -d)

# 账户 admin 密码 Harbor12345
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://harbor.yuhan.tech
For more details, please visit https://github.com/goharbor/harbor

bitnami 排查问题

我居然在创建数据库的时候名字写错了!!!harbor_notary_erver 掉了 s ……

Untitled

现在又是我的数据库版本太高了……

Untitled

只好放弃让它自己安装一个 postgresql。

Untitled

外部 Redis 端口号要指定

登录成功

注意要用 sudo docker

Untitled

注意:key 和 secret 都是明文:

Untitled

Untitled

推送镜像:

Untitled

注意上传大小限制:

ingress.kubernetes.io/proxy-body-size: 10000m  # 老 k8s 版本用
nginx.ingress.kubernetes.io/proxy-body-size: 10000m  # 最新版本使用

同时也要注意 harbor 的相关 size 配置!minio 也得配置!

fluentd

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-fluentd bitnami/fluentd --version 5.5.15

helm show values bitnami/fluentd > fluentd.yml
helm install my-fluentd bitnami/fluentd -f fluentd.yml
  • global.storageClass: “nfs-client”
  • persistence true
  • ingress:
    • 不使用 all-yuhan-teck-tls,因为说Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: Secret “all-yuhan-tech-tls” in namespace “default” exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key “app.kubernetes.io/managed-by”: must be set to “Helm”; annotation validation error: missing key “meta.helm.sh/release-name”: must be set to “my-fluentd”; annotation validation error: missing key “meta.helm.sh/release-namespace”: must be set to “default”
    • 直接用 cert-manager
** Please be patient while the chart is being deployed **

  To verify that Fluentd has started, run:

    kubectl get all -l "app.kubernetes.io/name=fluentd,app.kubernetes.io/instance=my-fluentd"

  Logs are captured on each node by the forwarder pods and then sent to the aggregator pods. By default, the aggregator pods send the logs to the standard output.
  You can see all the logs by running this command:

    kubectl logs -l "app.kubernetes.io/component=aggregator"

  You can mount your own configuration files to the aggregators and the forwarders. For example, this is useful if you want to forward the aggregated logs to Elasticsearch or another service.

但这个东西,不太行,没有 web 界面诶。

Jaeger

Jaeger is a distributed tracing system. It is used for monitoring and troubleshooting microservices-based distributed systems.

Jenkins


helm repo add jenkins https://charts.jenkins.io
helm repo update

helm show values jenkins/jenkins > jenkins.yml
helm install my-jenkins jenkins/jenkins -f jenkins.yml

helm upgrade my-jenkins jenkins/jenkins -f jenkins.yml

主要修改:

  • ingress
  • pvc
NOTES:
1. Get your 'admin' user password by running:
  kubectl exec --namespace default -it svc/my-jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo
2. Visit https://jenkins.yuhan.tech

3. Login with the password from step 1 and the username: admin
4. Configure security realm and authorization strategy
5. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: https://jenkins.yuhan.tech/configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine

For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/

NOTE: Consider using a custom image with pre-installed plugins

Jenkins详细教程

安装插件

  • Docker
  • Docker steps
  • CloudBees Docker Build and Publish
  • Docker Pipeline
  • Node and Label parameter

Host key verification failed

需要手动登入容器,保存 host

Untitled

全局安全配置 → Git Host Key 改成 Accept first connection

Untitled

Untitled

Jenkins Could not find the cloud this project was built on

需配置 Docker Builder:

Untitled

先在 Ubuntu 上安装 Docker:

https://docs.docker.com/engine/install/ubuntu/

如果是 snap install docker 安装的~~,注意:systemctl status snap.docker.dockerd.service 找到位置:/etc/systemd/system[无效,自动生成]~~

Adding this line:

"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],

to my daemon.json file in /var/snap/docker/current/config solved the issue for me but maybe the container option explained on the github would of worked also.

How to enable remote API with Docker that’s been installed with snap?

然后开启 Docker URL REST API 访问:

dockerd

Run Docker in daemon mode:

~~sudo dockerd -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 &~~

Untitled

Configure remote access for Docker daemon

sudo systemctl edit docker.service

# 修改默认 nano 为 vim
sudo update-alternatives --config editor

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375

sudo systemctl daemon-reload

sudo systemctl restart docker.service

Untitled

Untitled

用 mac 上的 Docker 验证下,能不能连上:

Untitled

jenkins 配上,成功:

Untitled

但是这好像不太行,不是那个 build/publish docker 的插件,它好像需要在节点管理中添加 Cloud Server:

Untitled

agent 端口暴露与配置

全局安全配置,我指定为 30007,然后让 agent 服务的 NodePort 与之一致。

Untitled

Untitled

Gitlab 插件(弃用,连不上)

Jenkins Gitlab插件的使用 - 掘金

先注册 jenkins 账户,再登录这个账户,再创建这个账户的访问令牌:

Untitled

glpat-nFD3qfZfqYEZd5uCnSyX

然后在 Jenkins 中创建 Gitlab API token 类型的凭据:

Untitled

新建节点:连宿主机

注意 root 要在 sshd 中配置 PermitRootLogin yes。

还要 sudo passwd root 为 root 用户配置密码。

然后注意新节点的 Number of executors 别是 0

Untitled

宿主机 docker 配置

如何优雅的给 Docker 配置网络代理 - 腾讯云开发者社区-腾讯云

需要代理上网,否则拉不下来 docker 容器库中的依赖

sudo systemctl edit docker.service

sudo systemctl daemon-reload

sudo systemctl restart docker.service

Untitled

注意,宿主机一定要登录上 dockerhub,不然推送也会失败(看控制台日志)

节点 + Docker 完美构建

Untitled

安装 Blue Ocean 插件

全新的界面啊

Untitled

安装 vscode Jenkinsfile 插件

Untitled

Untitled

参考文章

轻量高可用的 K8s 集群搭建方案:MicroK8s

microk8s 设置代理

Argo CD 入门教程

argo workflow 登录 - 掘金

k8s argo workflow获取登录token的命令_学亮编程手记的博客-CSDN博客

云原生流水线 Argo Workflows 的安装、使用以及个人体验

How to manage DNS in NetworkManager via console (nmcli)?

后记

致谢:

标签 :
comments powered by Disqus

相关文章

图数据库 Cypher 查询语言的子查询 CALL (subquery)

子查询允许将查询组合起来,这在使用UNION或聚合时特别有用。 子查询与封闭查询交互的方式有一些限制: 子查询只能引用外部查询中显式导入的变量。 子查询不能返回与外围查询中变量名称相同的变量。 从子查询返回的

阅读更多

如何在没有U盘的情况下,重新安装操作系统?无U盘也能装Ubuntu!

动机 windows11 是在是跑不起来了,卡的要死 咱还是装个 Ubuntu 好让我跑 microk8s ! easyUEFI 没有U盘安装ubuntu18(linux),EasyUEFI安装ubuntu_大蜻科的博客-CSDN博客_无u盘安装ubuntu18 使用eas

阅读更多

Remarkable2 配置中文字体与其他 Tips

动机 每次更新系统之后,我的字体就没了,我哭😭 主旨 导入文档 下面尝试导入 PDF、EPUB 来看看阅读效果。首先肯定要去官方的支持网站,看看用户手册。 官方的导入方案中,总的来说就两种,一个是通过它的云同步服务

阅读更多