Astrisk Blog

Gcp Shadowsocks Server Install

• kubernetes-docker

更新源和升级系统&&安装shadowsocks

$ sudo su
# apt update
# apt upgrade
# echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf
# echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf
# sysctl -p
# lsmod | grep bbr
# apt-get update
# apt-get install python-pip
# pip install shadowsocks

创建配置文件

# vim /etc/ss-conf.json

{
"server":"0.0.0.0",
"server_port":8838,
"local_address":"127.0.0.1",
"local_port":1080,
"password":"123456",
"timeout":600,
"method":"aes-256-cfb"
}

创建启动脚本

# vim /etc/init.d/shadowsocks.sh

#!/bin/bash  
# command content  
/usr/bin/python /usr/local/bin/ssserver -c /etc/ss-conf.json -d start
exit 0
# chmod +x /etc/init.d/shadowsocks.sh

容器化DEVOPS-弹性CI平台系列之Jenkins Gitlab集成

• kubernetes-docker

我们已经部署好了Jenkins cluster,现在是时候把Jenkins和代码做集成。这样开发提交代码,就可以自动触发Jenkins的自动化构建jobs。 我这里的代码仓库选用的是Gitlab(现在很多公司都部署了内网的Gitlab).

Jenkins插件安装配置

在Jenkins左上角: Jenkins -> 系统管理 -> 插件管理页面, 搜索中输入:gitlab, 然后找到gitlab插件安装即可。

jenkins2

gitlab

jenkins中Gitlab的配置在Jenkins ->系统管理 -> 系统设置

gitlab2

完成后,可以点击TestConnection,如果显示Success,即可。

Jenkins新建job

在JenkinsUI中新建一个job,在构建触发器部分,勾选:Build when a change is pushed to GitLab. GitLab CI Service URL: http://x.x.x.x:8080/project/test(拷贝这个URL,等会配置gitlab用),保存退出。

jenkins3

Gitlab repo配置

容器化DEVOPS-弹性CI平台系列之gradle android自动化

• kubernetes-docker

在以前的blog中,已经把SoapUI自动化放到我们的弹性Jenkins集群中,现在我们再实现一个业务需求:在弹性Jenkins集群中利用gradle运行Android自动化。

基础镜像

都知道Android的基础构建环境是Java+Android SDK,以前用ant来做构建工具,现在基本上都已经使用gradle这个第三代的构建工具。为了遵循我们的docker images构建原则,先构建通用的基础镜像。 这次我们换成ubuntu Linux来作为base images,下面是ubuntu-jnlp-slave的Dockerfile

容器化DEVOPS-弹性CI平台系列之kubernetes-nfs动态卷

• kubernetes-docker

无路是线上还是线下的集群,都是有很多的持久存储需求,比如为了提高CI效率就会存储一些构建环境数据。虽然可以动过手动挂载或者手动创建PV和PVC方式来实现,但是都没有动态卷来得方便。kubernetes支持的动态卷存储有很多,比如之前提到的GlusterFS,Ceph。由于分布式存储比较复杂,所以我的线下集群选择用NFS的动态卷。 下面就来看看如何在kubernetes中实现NFS的动态卷,并使用。后面也会用NFS的动态卷来存储Mysql,PostgreSQL数据库等。 后面我会介绍如何把代码质量平台Sonarqube也放到docker中并部署到kubernetes里,并数据存储到基于nfs动态卷的PostgreSQL数据库中

NFS服务器安装

需要安装nfs服务,具体可以参考KUBERNETES 存储 NFS 部署

RBAC

由于kubernetes开启了RBAC功能,首先需要创建rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  namespace: mysql
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: mysql
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io

nfs-client-provisioner

创建服务

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nfs-client-provisioner
  namespace: mysql
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: x.x.x.x
            - name: NFS_PATH
              value: /home/nfs/mariadb/mariadb00
      volumes:
        - name: nfs-client-root
          nfs:
            server: x.x.x.x
            path: /home/nfs/mariadb/mariadb00

创建nfs storageclass class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  namespace: mysql
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'

测试

创建test PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  namespace: mysql
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

创建 test pod

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
  namespace: mysql
spec:
  containers:
  - name: test-pod
    image: gcr.io/google_containers/busybox:1.24
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

mysql deployment中使用

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mariadb
  namespace: mysql
spec:
  serviceName: "mariadb"
  replicas: 1
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mariadb
          image: mariadb:10.1.22@sha256:21afb9ab191aac8ced2e1490ad5ec6c0f1c5704810d73451dd124670bcacfb14
          ports:
            - containerPort: 3306
              name: mysql
            - containerPort: 4444
              name: sst
            - containerPort: 4567
              name: replication
            - containerPort: 4567
              protocol: UDP
              name: replicationudp
            - containerPort: 4568
              name: ist
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: rootpw
            - name: MYSQL_INITDB_SKIP_TZINFO
              value: "yes"
          args:
            - --character-set-server=utf8mb4
            - --collation-server=utf8mb4_unicode_ci
            # Remove after first replicas=1 create
            - --wsrep-new-cluster
          volumeMounts:
            - name: mysql
              mountPath: /var/lib/mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
            - name: initdb
              mountPath: /docker-entrypoint-initdb.d
      volumes:
        - name: conf
          configMap:
            name: conf-d
        - name: initdb
          emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: mysql
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs-storage"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 30Gi

创建成功后,在kubernetes dashboard 页面可以看到自动创建的PV,并且状态是Bound,如下图。

nfs

容器化DEVOPS-弹性CI平台系列之maven slave运行SoapUI自动化用例

• kubernetes-docker

在前面的blog中,我们已经在kubernetes中部署自动伸缩的Jenkins集群。既然基础设施已经齐全,那就让我们把业务的东西放进去,让我们的基础设施发挥作用。在这篇blog中,我们会做一个maven的slave,然后利用这个maven slave来运行我们的SoapUI自动化用例。

jenkins jnlp maven slave

用maven来构建Java程序时,需要先下载各种依赖的jar包,而且这个jar的量还比较大。为了利用maven的缓存,我们把.m2目录挂在在nfs服务器目录中,这样只需要在第一次构建时,从网络中下载依赖jar包,以后就可以利用本地存储的jar包,以此来提高CI的构建效率。 下面直接给出maven slave的Dockerfile

FROM fangruo/jenkins-slave-apline-base
MAINTAINER Qiusheng Zhang <fang_ruo@163.com>

USER root

RUN apk add --no-cache curl tar bash

ARG MAVEN_VERSION=3.3.9
ARG USER_HOME_DIR="/home/jenkins"
ARG SHA=6e3e9c949ab4695a204f74038717aa7b2689b1be94875899ac1b3fe42800ff82
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries

RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
  && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
  && echo "${SHA}  /tmp/apache-maven.tar.gz" | sha256sum -c - \
  && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
  && rm -f /tmp/apache-maven.tar.gz \
  && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn

ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"

VOLUME "$USER_HOME_DIR/.m2"

USER jenkins

Jenkins配置maven slave

在Jenkins管理界面左上角 Jenkins -> 系统管理 -> 系统设置 -> Kubernetes Cloud中增加slave。

apiVersion: "v1" 
kind: "Pod" 
metadata: 
  name: "maven-slave" 
  labels: 
    name: "maven-slave" 
spec: 
  containers: 
  - name: "maven-slave" 
    image: "fangruo/jenkins-snlp-maven:latest" 
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /home/jenkins/.m2
      name: mavenhome
    env: 
    - name: "JENKINS_URL" 
      value: "http://jenkins.jenkins:8080"
  volumes:
    - name: mavenhome
      nfs:
        server: x.x.x.x
        path: "/home/nfs/.m2"  

参考下图

maven

测试

在Jenkins中新建一个job,在Restrict where this project can be run 中填写maven slave对应的label(我的是maven3.3.9)即可

maven2

maven3

maven4

kubernetes动态卷-GlusterFS踩坑

• kubernetes-docker

之前按照《kubernetes权威指南》在kubernetes中部署GlusterFS总是不成功,最后各种Google,才搞成功。这之间踩了不少坑,这里把主要的坑记录下。

在kubernetes中部署GlusterFS有两个须知: 1.需要3个或以上kubernetes node节点。 2.用来部署GlusterFS的硬盘,必须是裸盘(没有文件系统),如果安装失败,要重新安装,必须删除安装的vg 和pv(如何删除,下面清理资源部分会提到)。

install GlusterFS client on all nodes

$ yum install -y glusterfs glusterfs-fuse

更新apiserver启动参数

容器化DEVOPS-弹性CI平台系列之docker build docker

• kubernetes-docker

由于我们所有的技术都是基于docker,所以会经常构建docker images,之前我们都是手工构建,并推送到docker register。现在是时候把构建docker images做成自动化,并集成的CI管道。懒人的目标是:提交完代码(或者merge完代码),CI系统就会自动拉取代码,构建包括镜像,推送到docker images到docker register。 有了docker images,就可以持续部署了。关于部署,我们会用到别的技术,暂时不在这里介绍。

选择构建docker 方式

用docker来构建docker有以下两种方式或技术:

我们主要是CI中构建docker images,所以我选择用docker outside docker的方式来构建docker images。

DOD-jenkins slave base

现在我们需要在jnlp slave的基础上做一个能够构建docker的Jenkins jnlp slave。具体细节大家可以自己进入docker container里,尝试install docker,我这里直接给出Dockerfile

$ vim Dockerfile

FROM jenkinsci/slave:2.62

MAINTAINER Qiusheng Zhang <fang_ruo@163.com>

USER root

RUN echo "Asia/Shanghai" > /etc/timezone && \
    dpkg-reconfigure -f noninteractive tzdata && \
    apt-get update && \
    apt-get -y install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \
     software-properties-common && \
     curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | apt-key add - && \
     add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
   $(lsb_release -cs) \
   stable" && \
     apt-get update && apt-get -y install docker-ce

VOLUME /var/run/docker.sock

WORKDIR /home/jenkins

COPY jenkins-slave /usr/local/bin/jenkins-slave

ENTRYPOINT ["jenkins-slave"]

jenkins-slave文件

#!/usr/bin/env bash

# The MIT License
#
#  Copyright (c) 2015, CloudBees, Inc.
#
#  Permission is hereby granted, free of charge, to any person obtaining a copy
#  of this software and associated documentation files (the "Software"), to deal
#  in the Software without restriction, including without limitation the rights
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#  copies of the Software, and to permit persons to whom the Software is
#  furnished to do so, subject to the following conditions:
#
#  The above copyright notice and this permission notice shall be included in
#  all copies or substantial portions of the Software.
#
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
#  THE SOFTWARE.

# Usage jenkins-slave.sh [options] -url http://jenkins SECRET SLAVE_NAME
# Optional environment variables :
# * JENKINS_TUNNEL : HOST:PORT for a tunnel to route TCP traffic to jenkins host, when jenkins can't be directly accessed over network
# * JENKINS_URL : alternate jenkins URL

if [ $# -eq 1 ]; then

  # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
  exec "$@"

else

  # if -tunnel is not provided try env vars
  if [[ "$@" != *"-tunnel "* ]]; then
    if [ ! -z "$JENKINS_TUNNEL" ]; then
      TUNNEL="-tunnel $JENKINS_TUNNEL"
    fi
  fi

  if [ ! -z "$JENKINS_URL" ]; then
    URL="-url $JENKINS_URL"
  fi

  if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
    echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
    JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
  fi

  exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL "$@"
fi

Jenkins中配置DOD slave

具体如何配置,在之前的blog,Jenkins 集成kubernetes有介绍,具体如下图 dood

测试

在Jenkins中建个测试的job,简单运行docker run hello-world 或docker ps命令,如果运行成功,就表示已经可以用CI系统来构建docker。下图我直接从gitlab repo拉代码,利用Jenkins 构建了docker images,并push到私有的docker register。

dod1

dod2

容器化DEVOPS-弹性CI平台系列之Jenkins kubernetes集成

• kubernetes-docker

之前的blog中已经把Jenkins master 部署到kubernetes集群中,在浏览器输入http://nodeip:38080 就可以看到如下界面

start

在nfs服务器或者进入Jenkins master的container中获取secret,就可以继续安装通用插件

secrets

容器化DEVOPS-弹性CI平台系列之部署Jenkins master

• kubernetes-docker

在之前的文章中,我们已经构建好了Jenkins的相关docker images,并且推送到私有register中。现在我们把Jenkins部署到kubernetes集群中。

创建k8s Jenkins资源文件

容器化DEVOPS-弹性CI平台系列之Jenkins docker images

• kubernetes-docker

docker images 制作比较简单,只要熟悉一下相关的命令多操作几遍就行。具体可以参考

这里简单提下最基本的几点:

按照以上几点,我们就可以编写需要的几个Jenkins Dockerfile。

Jenkins master

FROM jenkins:2.46.3-alpine

MAINTAINER Qiusheng Zhang <fang_ruo@163.com>

USER root

RUN apk --no-cache add tzdata  && \
    ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

USER jenkins