{"id":3455,"date":"2021-12-09T10:09:56","date_gmt":"2021-12-09T10:09:56","guid":{"rendered":"https:\/\/en.pingcap.com\/?p=3455"},"modified":"2024-08-20T06:46:24","modified_gmt":"2024-08-20T13:46:24","slug":"implementing-chaos-engineering-in-k8s-chaos-mesh-principle-analysis-and-control-plane-development","status":"publish","type":"post","link":"https:\/\/www.pingcap.com\/ko\/blog\/implementing-chaos-engineering-in-k8s-chaos-mesh-principle-analysis-and-control-plane-development\/","title":{"rendered":"Implementing Chaos Engineering in K8s: Chaos Mesh Principle Analysis and Control Plane Development"},"content":{"rendered":"\n

Author:<\/strong> Mayo Cream<\/a> (Kubernetes Member, CNCF Security TAG Member, OSS Contributor)<\/p>\n\n\n\n

Chaos Mesh<\/a> is an open-source, cloud-native Chaos Engineering platform built on Kubernetes (K8s) custom resource definitions (CRDs). Chaos Mesh can simulate various types of faults and has an enormous capability to orchestrate fault scenarios. You can use Chaos Mesh to conveniently simulate various abnormalities that might occur in development, testing, and production environments and find potential problems in the system.<\/p>\n\n\n\n

In this article, I’ll explore the practice of Chaos Engineering in Kubernetes clusters, discuss important Chaos Mesh features through analysis of its source code, and explain how to develop Chaos Mesh’s control plane with code examples.<\/p>\n\n\n\n

If you’re not familiar with Chaos Mesh, please review the Chaos Mesh documentation<\/a> to get a basic knowledge of Chaos Mesh’s architecture.<\/p>\n\n\n\n

For the test code in this article, see the mayocream\/chaos-mesh-controlpanel-demo<\/a> repository on GitHub.<\/p>\n\n\n\n

<\/a>How Chaos Mesh creates chaos<\/h2>\n\n\n\n

Chaos Mesh is a Swiss army knife for implementing Chaos Engineering on Kubernetes. This section introduces how it works.<\/p>\n\n\n\n

<\/a>Privileged mode<\/h3>\n\n\n\n

Chaos Mesh runs privileged containers in Kubernetes to create failures. Chaos Daemon’s Pod runs as DaemonSet<\/code> and adds additional capabilities<\/a> to the Pod’s container runtime via the Pod’s security context.<\/p>\n\n\n\n

apiVersion: apps\/v1\nkind: DaemonSet\nspec:\n  template:\n    metadata: ...\n    spec:\n      containers:\n        - name: chaos-daemon\n          securityContext:\n            {{- if .Values.chaosDaemon.privileged }}\n            privileged: true\n            capabilities:\n              add:\n                - SYS_PTRACE\n            {{- else }}\n            capabilities:\n              add:\n                - SYS_PTRACE\n                - NET_ADMIN\n                - MKNOD\n                - SYS_CHROOT\n                - SYS_ADMIN\n                - KILL\n                # CAP_IPC_LOCK is used to lock memory\n                - IPC_LOCK\n            {{- end }}<\/code><\/pre>\n\n\n\n
The Linux capabilities grant containers privileges to create and access the \/dev\/fuse<\/code> Filesystem in Userspace (FUSE) pipe. FUSE is the Linux userspace filesystem interface. It lets non-privileged users create their own file systems without editing the kernel code.<\/p>\n\n\n\n
According to pull request #1109<\/a> on GitHub, the DaemonSet<\/code> program uses cgo to call the Linux makedev<\/code> function to create a FUSE pipe.<\/p>\n\n\n\n
\/\/ #include <sys\/sysmacros.h>\n\/\/ #include <sys\/types.h>\n\/\/ \/\/ makedev is a macro, so a wrapper is needed\n\/\/ dev_t Makedev(unsigned int maj, unsigned int min) {\n\/\/   return makedev(maj, min);\n\/\/ }\n\/\/ EnsureFuseDev ensures \/dev\/fuse exists. If not, it will create one\nfunc EnsureFuseDev() {\n    if _, err := os.Open(\"\/dev\/fuse\"); os.IsNotExist(err) {\n        \/\/ 10, 229 according to https:\/\/www.kernel.org\/doc\/Documentation\/admin-guide\/devices.txt\n        fuse := C.Makedev(10, 229)\n        syscall.Mknod(\"\/dev\/fuse\", 0o666|syscall.S_IFCHR, int(fuse))\n    }\n}<\/code><\/pre>\n\n\n\n
In pull request #1453<\/a>, Chaos Daemon enables privileged mode by default; that is, it sets privileged: true<\/code> in the container’s SecurityContext<\/code>.<\/p>\n\n\n\n
<\/a>Killing Pods<\/h3>\n\n\n\n
PodKill<\/code>, PodFailure<\/code>, and ContainerKill<\/code> belong to the PodChaos<\/code> category. PodKill<\/code> randomly kills a Pod. It calls the API server to send the kill command.<\/p>\n\n\n\n
import (\n    \"context\"\n    v1 \"k8s.io\/api\/core\/v1\"\n    \"sigs.k8s.io\/controller-runtime\/pkg\/client\"\n)\ntype Impl struct {\n    client.Client\n}\nfunc (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {\n    ...\n    err = impl.Get(ctx, namespacedName, &pod)\n    if err != nil {\n        \/\/ TODO: handle this error\n        return v1alpha1.NotInjected, err\n    }\n    err = impl.Delete(ctx, &pod, &client.DeleteOptions{\n        GracePeriodSeconds: &podchaos.Spec.GracePeriod, \/\/ PeriodSeconds has to be set specifically\n    })\n    ...\n    return v1alpha1.Injected, nil\n}<\/code><\/pre>\n\n\n\n
The GracePeriodSeconds<\/code> parameter lets Kubernetes forcibly terminate a Pod<\/a>. For example, if you need to delete a Pod immediately, use the kubectl delete pod --grace-period=0 --force<\/code> command.<\/p>\n\n\n\n
PodFailure<\/code> patches the Pod object resource to replace the image in the Pod with a wrong one. Chaos only modifies the image<\/code> fields of containers<\/code> and initContainers<\/code>. This is because most of the metadata about a Pod is immutable. For more details, see Pod update and replacement<\/a>.<\/p>\n\n\n\n
func (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) {\n    ...\n    pod := origin.DeepCopy()\n    for index := range pod.Spec.Containers {\n        originImage := pod.Spec.Containers[index].Image\n        name := pod.Spec.Containers[index].Name\n        key := annotation.GenKeyForImage(podchaos, name, false)\n        if pod.Annotations == nil {\n            pod.Annotations = make(map[string]string)\n        }\n        \/\/ If the annotation is already existed, we could skip the reconcile for this container\n        if _, ok := pod.Annotations[key]; ok {\n            continue\n        }\n        pod.Annotations[key] = originImage\n        pod.Spec.Containers[index].Image = config.ControllerCfg.PodFailurePauseImage\n    }\n    for index := range pod.Spec.InitContainers {\n        originImage := pod.Spec.InitContainers[index].Image\n        name := pod.Spec.InitContainers[index].Name\n        key := annotation.GenKeyForImage(podchaos, name, true)\n        if pod.Annotations == nil {\n            pod.Annotations = make(map[string]string)\n        }\n        \/\/ If the annotation is already existed, we could skip the reconcile for this container\n        if _, ok := pod.Annotations[key]; ok {\n            continue\n        }\n        pod.Annotations[key] = originImage\n        pod.Spec.InitContainers[index].Image = config.ControllerCfg.PodFailurePauseImage\n    }\n    err = impl.Patch(ctx, pod, client.MergeFrom(&origin))\n    if err != nil {\n        \/\/ TODO: handle this error\n        return v1alpha1.NotInjected, err\n    }\n    return v1alpha1.Injected, nil\n}<\/code><\/pre>\n\n\n\n
The default container image that causes failures is gcr.io\/google-containers\/pause:latest<\/code>.<\/p>\n\n\n\n
PodKill<\/code> and PodFailure<\/code> control the Pod lifecycle through the Kubernetes API server. But ContainerKill<\/code> does this through Chaos Daemon that runs on the cluster node. ContainerKill<\/code> uses Chaos Controller Manager to run the client to initiate gRPC calls to Chaos Daemon.<\/p>\n\n\n\n
func (b *ChaosDaemonClientBuilder) Build(ctx context.Context, pod *v1.Pod) (chaosdaemonclient.ChaosDaemonClientInterface, error) {\n    ...\n    daemonIP, err := b.FindDaemonIP(ctx, pod)\n    if err != nil {\n        return nil, err\n    }\n    builder := grpcUtils.Builder(daemonIP, config.ControllerCfg.ChaosDaemonPort).WithDefaultTimeout()\n    if config.ControllerCfg.TLSConfig.ChaosMeshCACert != \"\" {\n        builder.TLSFromFile(config.ControllerCfg.TLSConfig.ChaosMeshCACert, config.ControllerCfg.TLSConfig.ChaosDaemonClientCert, config.ControllerCfg.TLSConfig.ChaosDaemonClientKey)\n    } else {\n        builder.Insecure()\n    }\n    cc, err := builder.Build()\n    if err != nil {\n        return nil, err\n    }\n    return chaosdaemonclient.New(cc), nil\n}<\/code><\/pre>\n\n\n\n
When Chaos Controller Manager sends commands to Chaos Daemon, it creates a corresponding client based on the Pod information. For example, to control a Pod on a node, it creates a client by getting the ClusterIP<\/code> of the node where the Pod is located. If the Transport Layer Security (TLS) certificate configuration exists, Controller Manager adds the TLS certificate for the client.<\/p>\n\n\n\n
When Chaos Daemon starts, if it has a TLS certificate it attaches the certificate to enable gRPCS. The TLS configuration option RequireAndVerifyClientCert<\/code> indicates whether to enable mutual TLS (mTLS) authentication.<\/p>\n\n\n\n
func newGRPCServer(containerRuntime string, reg prometheus.Registerer, tlsConf tlsConfig) (*grpc.Server, error) {\n    ...\n    if tlsConf != (tlsConfig{}) {\n        caCert, err := ioutil.ReadFile(tlsConf.CaCert)\n        if err != nil {\n            return nil, err\n        }\n        caCertPool := x509.NewCertPool()\n        caCertPool.AppendCertsFromPEM(caCert)\n        serverCert, err := tls.LoadX509KeyPair(tlsConf.Cert, tlsConf.Key)\n        if err != nil {\n            return nil, err\n        }\n        creds := credentials.NewTLS(&tls.Config{\n            Certificates: []tls.Certificate{serverCert},\n            ClientCAs:    caCertPool,\n            ClientAuth:   tls.RequireAndVerifyClientCert,\n        })\n        grpcOpts = append(grpcOpts, grpc.Creds(creds))\n    }\n    s := grpc.NewServer(grpcOpts...)\n    grpcMetrics.InitializeMetrics(s)\n    pb.RegisterChaosDaemonServer(s, ds)\n    reflection.Register(s)\n    return s, nil\n}<\/code><\/pre>\n\n\n\n
Chaos Daemon provides the following gRPC interfaces to call:<\/p>\n\n\n\n
\/\/ ChaosDaemonClient is the client API for ChaosDaemon service.\n\/\/\n\/\/ For semantics around ctx use and closing\/ending streaming RPCs, please refer to https:\/\/godoc.org\/google.golang.org\/grpc#ClientConn.NewStream.\ntype ChaosDaemonClient interface {\n    SetTcs(ctx context.Context, in *TcsRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    FlushIPSets(ctx context.Context, in *IPSetsRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    SetIptablesChains(ctx context.Context, in *IptablesChainsRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    SetTimeOffset(ctx context.Context, in *TimeRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    RecoverTimeOffset(ctx context.Context, in *TimeRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    ContainerKill(ctx context.Context, in *ContainerRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    ContainerGetPid(ctx context.Context, in *ContainerRequest, opts ...grpc.CallOption) (*ContainerResponse, error)\n    ExecStressors(ctx context.Context, in *ExecStressRequest, opts ...grpc.CallOption) (*ExecStressResponse, error)\n    CancelStressors(ctx context.Context, in *CancelStressRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n    ApplyIOChaos(ctx context.Context, in *ApplyIOChaosRequest, opts ...grpc.CallOption) (*ApplyIOChaosResponse, error)\n    ApplyHttpChaos(ctx context.Context, in *ApplyHttpChaosRequest, opts ...grpc.CallOption) (*ApplyHttpChaosResponse, error)\n    SetDNSServer(ctx context.Context, in *SetDNSServerRequest, opts ...grpc.CallOption) (*empty.Empty, error)\n}<\/code><\/pre>\n\n\n\n
<\/a>Network failure injection<\/h3>\n\n\n\n
From pull request #41<\/a>, we know that Chaos Mesh injects network failures this way: it calls pbClient.SetNetem<\/code> to encapsulate parameters into a request and send the request to the Chaos Daemon on the node for processing.<\/p>\n\n\n\n
The network failure injection code is shown below as it appeared in 2019. As the project developed, the functions were distributed among several files.<\/p>\n\n\n\n
func (r *Reconciler) applyPod(ctx context.Context, pod *v1.Pod, networkchaos *v1alpha1.NetworkChaos) error {\n    ...\n    pbClient := pb.NewChaosDaemonClient(c)\n    containerId := pod.Status.ContainerStatuses[0].ContainerID\n    netem, err := spec.ToNetem()\n    if err != nil {\n        return err\n    }\n    _, err = pbClient.SetNetem(ctx, &pb.NetemRequest{\n        ContainerId: containerId,\n        Netem:       netem,\n    })\n    return err\n}<\/code><\/pre>\n\n\n\n
In the pkg\/chaosdaemon<\/code> package, we can see how Chaos Daemon processes requests.<\/p>\n\n\n\n
func (s *Server) SetNetem(ctx context.Context, in *pb.NetemRequest) (*empty.Empty, error) {\n    log.Info(\"Set netem\", \"Request\", in)\n    pid, err := s.crClient.GetPidFromContainerID(ctx, in.ContainerId)\n    if err != nil {\n        return nil, status.Errorf(codes.Internal, \"get pid from containerID error: %v\", err)\n    }\n    if err := Apply(in.Netem, pid); err != nil {\n        return nil, status.Errorf(codes.Internal, \"netem apply error: %v\", err)\n    }\n    return &empty.Empty{}, nil\n}\n\/\/ Apply applies a netem on eth0 in pid related namespace\nfunc Apply(netem *pb.Netem, pid uint32) error {\n    log.Info(\"Apply netem on PID\", \"pid\", pid)\n    ns, err := netns.GetFromPath(GenNetnsPath(pid))\n    if err != nil {\n        log.Error(err, \"failed to find network namespace\", \"pid\", pid)\n        return errors.Trace(err)\n    }\n    defer ns.Close()\n    handle, err := netlink.NewHandleAt(ns)\n    if err != nil {\n        log.Error(err, \"failed to get handle at network namespace\", \"network namespace\", ns)\n        return err\n    }\n    link, err := handle.LinkByName(\"eth0\") \/\/ TODO: check whether interface name is eth0\n    if err != nil {\n        log.Error(err, \"failed to find eth0 interface\")\n        return errors.Trace(err)\n    }\n    netemQdisc := netlink.NewNetem(netlink.QdiscAttrs{\n        LinkIndex: link.Attrs().Index,\n        Handle:    netlink.MakeHandle(1, 0),\n        Parent:    netlink.HANDLE_ROOT,\n    }, ToNetlinkNetemAttrs(netem))\n    if err = handle.QdiscAdd(netemQdisc); err != nil {\n        if !strings.Contains(err.Error(), \"file exists\") {\n            log.Error(err, \"failed to add Qdisc\")\n            return errors.Trace(err)\n        }\n    }\n    return nil\n}<\/code><\/pre>\n\n\n\n
Finally, the vishvananda\/netlink<\/code> library<\/a> operates the Linux network interface to complete the job.<\/p>\n\n\n\n
From here, NetworkChaos<\/code> manipulates the Linux host network to create chaos. It includes tools such as iptables and ipset.<\/p>\n\n\n\n
In Chaos Daemon’s Dockerfile, you can see the Linux tool chain that it depends on:<\/p>\n\n\n\n
RUN apt-get update && \\ \n    apt-get install -y tzdata iptables ipset stress-ng iproute2 fuse util-linux procps curl && \\\n    rm -rf \/var\/lib\/apt\/lists\/*<\/code><\/pre>\n\n\n\n
<\/a>Stress test<\/h3>\n\n\n\n
Chaos Daemon also implements StressChaos<\/code>. After the Controller Manager calculates the rules, it sends the task to the specific Daemon<\/code>. The assembled parameters are shown below. They are combined into command execution parameters and appended to the stress-ng<\/code> command for execution.<\/p>\n\n\n\n
\/\/ Normalize the stressors to comply with stress-ng\nfunc (in *Stressors) Normalize() (string, error) {\n    stressors := \"\"\n    if in.MemoryStressor != nil && in.MemoryStressor.Workers != 0 {\n        stressors += fmt.Sprintf(\" --vm %d --vm-keep\", in.MemoryStressor.Workers)\n        if len(in.MemoryStressor.Size) != 0 {\n            if in.MemoryStressor.Size[len(in.MemoryStressor.Size)-1] != '%' {\n                size, err := units.FromHumanSize(string(in.MemoryStressor.Size))\n                if err != nil {\n                    return \"\", err\n                }\n                stressors += fmt.Sprintf(\" --vm-bytes %d\", size)\n            } else {\n                stressors += fmt.Sprintf(\" --vm-bytes %s\",\n                    in.MemoryStressor.Size)\n            }\n        }\n        if in.MemoryStressor.Options != nil {\n            for _, v := range in.MemoryStressor.Options {\n                stressors += fmt.Sprintf(\" %v \", v)\n            }\n        }\n    }\n    if in.CPUStressor != nil && in.CPUStressor.Workers != 0 {\n        stressors += fmt.Sprintf(\" --cpu %d\", in.CPUStressor.Workers)\n        if in.CPUStressor.Load != nil {\n            stressors += fmt.Sprintf(\" --cpu-load %d\",\n                *in.CPUStressor.Load)\n        }\n        if in.CPUStressor.Options != nil {\n            for _, v := range in.CPUStressor.Options {\n                stressors += fmt.Sprintf(\" %v \", v)\n            }\n        }\n    }\n    return stressors, nil\n}<\/code><\/pre>\n\n\n\n
The Chaos Daemon server side processes the function’s execution command to call the official Go package os\/exec<\/code>. For details, see the pkg\/chaosdaemon\/stress_server_linux.go<\/code><\/a> file. There is also a file with the same name that ends with darwin. *_darwin<\/code> files prevent possible errors when the program is running on macOS.<\/p>\n\n\n\n
The code uses the shirou\/gopsutil<\/code><\/a> package to obtain the PID process status and reads the stdout and stderr standard outputs. I’ve seen this processing mode in hashicorp\/go-plugin<\/code><\/a>, and go-plugin does this better.<\/p>\n\n\n\n
<\/a>I\/O fault injection<\/h3>\n\n\n\n
Pull request #826<\/a> introduces a new implementation of IOChaos, without the use of sidecar injection. It uses Chaos Daemon to directly manipulate the Linux namespace through the underlying commands of the runc<\/a> container and runs the chaos-mesh\/toda<\/a> FUSE program developed by Rust to inject container I\/O chaos. The JSON-RPC 2.0<\/a> protocol is used to communicate between toda and the control plane.<\/p>\n\n\n\n
The new IOChaos implementation doesn’t modify the Pod resources. When you define the IOChaos chaos experiment, for each Pod filtered by the selector field, a corresponding PodIOChaos resource is created. PodIoChaos’ owner reference<\/a> is the Pod. At the same time, a set of finalizers<\/a> is added to PodIoChaos to release PodIoChaos resources before PodIoChaos is deleted.<\/p>\n\n\n\n
\/\/ Apply implements the reconciler.InnerReconciler.Apply\nfunc (r *Reconciler) Apply(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error {\n    iochaos, ok := chaos.(*v1alpha1.IoChaos)\n    if !ok {\n        err := errors.New(\"chaos is not IoChaos\")\n        r.Log.Error(err, \"chaos is not IoChaos\", \"chaos\", chaos)\n        return err\n    }\n    source := iochaos.Namespace + \"\/\" + iochaos.Name\n    m := podiochaosmanager.New(source, r.Log, r.Client)\n    pods, err := utils.SelectAndFilterPods(ctx, r.Client, r.Reader, &iochaos.Spec)\n    if err != nil {\n        r.Log.Error(err, \"failed to select and filter pods\")\n        return err\n    }\n    r.Log.Info(\"applying iochaos\", \"iochaos\", iochaos)\n    for _, pod := range pods {\n        t := m.WithInit(types.NamespacedName{\n            Name:      pod.Name,\n            Namespace: pod.Namespace,\n        })\n        \/\/ TODO: support chaos on multiple volume\n        t.SetVolumePath(iochaos.Spec.VolumePath)\n        t.Append(v1alpha1.IoChaosAction{\n            Type: iochaos.Spec.Action,\n            Filter: v1alpha1.Filter{\n                Path:    iochaos.Spec.Path,\n                Percent: iochaos.Spec.Percent,\n                Methods: iochaos.Spec.Methods,\n            },\n            Faults: []v1alpha1.IoFault{\n                {\n                    Errno:  iochaos.Spec.Errno,\n                    Weight: 1,\n                },\n            },\n            Latency:          iochaos.Spec.Delay,\n            AttrOverrideSpec: iochaos.Spec.Attr,\n            Source:           m.Source,\n        })\n        key, err := cache.MetaNamespaceKeyFunc(&pod)\n        if err != nil {\n            return err\n        }\n        iochaos.Finalizers = utils.InsertFinalizer(iochaos.Finalizers, key)\n    }\n    r.Log.Info(\"commiting updates of podiochaos\")\n    err = m.Commit(ctx)\n    if err != nil {\n        r.Log.Error(err, \"fail to commit\")\n        return err\n    }\n    r.Event(iochaos, v1.EventTypeNormal, utils.EventChaosInjected, \"\")\n    return nil\n}<\/code><\/pre>\n\n\n\n
<\/path><\/svg><\/div><\/div><\/div>\n\n\n\n
In the controller of the PodIoChaos resource, Controller Manager encapsulates the resource into parameters and calls the Chaos Daemon interface to process the parameters.<\/p>\n\n\n\n
\/\/ Apply flushes io configuration on pod\nfunc (h *Handler) Apply(ctx context.Context, chaos *v1alpha1.PodIoChaos) error {\n    h.Log.Info(\"updating io chaos\", \"pod\", chaos.Namespace+\"\/\"+chaos.Name, \"spec\", chaos.Spec)\n    ...\n    res, err := pbClient.ApplyIoChaos(ctx, &pb.ApplyIoChaosRequest{\n        Actions:     input,\n        Volume:      chaos.Spec.VolumeMountPath,\n        ContainerId: containerID,\n        Instance:  chaos.Spec.Pid,\n        StartTime: chaos.Spec.StartTime,\n    })\n    if err != nil {\n        return err\n    }\n    chaos.Spec.Pid = res.Instance\n    chaos.Spec.StartTime = res.StartTime\n    chaos.OwnerReferences = []metav1.OwnerReference{\n        {\n            APIVersion: pod.APIVersion,\n            Kind:       pod.Kind,\n            Name:       pod.Name,\n            UID:        pod.UID,\n        },\n    }\n    return nil\n}<\/code><\/pre>\n\n\n\n
The\u00a0pkg\/chaosdaemon\/iochaos_server.go<\/code>\u00a0file processes IOChaos. \u200b\u200bIn this file, a FUSE program needs to be injected into the container. As discussed in issue\u00a0#2305\u00a0on GitHub, the\u00a0\/usr\/local\/bin\/nsexec -l- p \/proc\/119186\/ns\/pid -m \/proc\/119186\/ns\/mnt - \/usr\/local\/bin\/toda --path \/tmp --verbose info<\/code>\u00a0command is executed to run the toda program under the same namespace as the Pod.<\/p>\n\n\n\n
func (s *DaemonServer) ApplyIOChaos(ctx context.Context, in *pb.ApplyIOChaosRequest) (*pb.ApplyIOChaosResponse, error) {\n    ...\n    pid, err := s.crClient.GetPidFromContainerID(ctx, in.ContainerId)\n    if err != nil {\n        log.Error(err, \"error while getting PID\")\n        return nil, err\n    }\n    args := fmt.Sprintf(\"--path %s --verbose info\", in.Volume)\n    log.Info(\"executing\", \"cmd\", todaBin+\" \"+args)\n    processBuilder := bpm.DefaultProcessBuilder(todaBin, strings.Split(args, \" \")...).\n        EnableLocalMnt().\n        SetIdentifier(in.ContainerId)\n    if in.EnterNS {\n        processBuilder = processBuilder.SetNS(pid, bpm.MountNS).SetNS(pid, bpm.PidNS)\n    }\n    ...\n    \/\/ Calls JSON RPC\n    client, err := jrpc.DialIO(ctx, receiver, caller)\n    if err != nil {\n        return nil, err\n    }\n    cmd := processBuilder.Build()\n    procState, err := s.backgroundProcessManager.StartProcess(cmd)\n    if err != nil {\n        return nil, err\n    }\n    ...\n}<\/code><\/pre>\n\n\n\n
The following code sample builds the running commands. These commands are the underlying namespace isolation implementation of runc:<\/p>\n\n\n\n
\/\/ GetNsPath returns corresponding namespace path\nfunc GetNsPath(pid uint32, typ NsType) string {\n    return fmt.Sprintf(\"%s\/%d\/ns\/%s\", DefaultProcPrefix, pid, string(typ))\n}\n\/\/ SetNS sets the namespace of the process\nfunc (b *ProcessBuilder) SetNS(pid uint32, typ NsType) *ProcessBuilder {\n    return b.SetNSOpt([]nsOption{{\n        Typ:  typ,\n        Path: GetNsPath(pid, typ),\n    }})\n}\n\/\/ Build builds the process\nfunc (b *ProcessBuilder) Build() *ManagedProcess {\n    args := b.args\n    cmd := b.cmd\n    if len(b.nsOptions) > 0 {\n        args = append([]string{\"--\", cmd}, args...)\n        for _, option := range b.nsOptions {\n            args = append([]string{\"-\" + nsArgMap[option.Typ], option.Path}, args...)\n        }\n        if b.localMnt {\n            args = append([]string{\"-l\"}, args...)\n        }\n        cmd = nsexecPath\n    }\n    ...\n}<\/code><\/pre>\n\n\n\n
<\/a>Control plane<\/h2>\n\n\n\n
Chaos Mesh is an open-source chaos engineering system under the Apache 2.0 protocol. As discussed above, it has rich capabilities and a good ecosystem. The maintenance team developed the chaos-mesh\/toda<\/code><\/a> FUSE based on the chaos system, the chaos-mesh\/k8s_dns_chaos<\/code><\/a> CoreDNS chaos plug-in, and Berkeley Packet Filter (BPF)-based kernel error injection chaos-mesh\/bpfki<\/code><\/a>.<\/p>\n\n\n\n
Now, I’ll describe the server side code required to build an end-user-oriented chaos engineering platform. This implementation is only an example\u2014not necessarily the best example. If you want to see the development practice on a real world platform, you can refer to Chaos Mesh’s Dashboard<\/a>. It uses the uber-go\/fx<\/code><\/a> dependency injection framework and the controller runtime’s manager mode.<\/p>\n\n\n\n
<\/a>Key Chaos Mesh features<\/h3>\n\n\n\n
As shown in the Chaos Mesh workflow below, we need to implement a server that sends YAML to the Kubernetes API. Chaos Controller Manager implements complex rule verification and rule delivery to Chaos Daemon. If you want to use Chaos Mesh with your own platform, you only need to connect to the process of creating CRD resources.<\/p>\n\n\n\n
<\/p>\n\n\n\n
\"Chaos<\/figure>\n\n\n\n
Chaos Mesh’s basic workflow<\/em><\/p>\n\n\n\n
<\/p>\n\n\n\n
Let’s take a look at the example on the Chaos Mesh website:<\/p>\n\n\n\n
import (\n    \"context\"\n    \"github.com\/pingcap\/chaos-mesh\/api\/v1alpha1\"\n    \"sigs.k8s.io\/controller-runtime\/pkg\/client\"\n)\nfunc main() {\n    ...\n    delay := &chaosv1alpha1.NetworkChaos{\n        Spec: chaosv1alpha1.NetworkChaosSpec{...},\n    }\n    k8sClient := client.New(conf, client.Options{ Scheme: scheme.Scheme })\n    k8sClient.Create(context.TODO(), delay)\n    k8sClient.Delete(context.TODO(), delay)\n}<\/code><\/pre>\n\n\n\n
Chaos Mesh provides APIs corresponding to all CRDs. We use the controller-runtime<\/a> developed by Kubernetes API Machinery SIG<\/a> to simplify the interaction with the Kubernetes API.<\/p>\n\n\n\n
<\/a>Inject chaos<\/h3>\n\n\n\n
Suppose we want to create a PodKill<\/code> resource by calling a program. After the resource is sent to the Kubernetes API server, it passes Chaos Controller Manager’s validating admission controller<\/a> to verify data. When we create a chaos experiment, if the admission controller fails to verify the input data, it returns an error to the client. For specific parameters, you can read Create experiments using YAML configuration files<\/a>.<\/p>\n\n\n\n
NewClient<\/code> creates a Kubernetes API client. You can refer to this example:<\/p>\n\n\n\n
package main\nimport (\n    \"context\"\n    \"controlpanel\"\n    \"log\"\n    \"github.com\/chaos-mesh\/chaos-mesh\/api\/v1alpha1\"\n    \"github.com\/pkg\/errors\"\n    metav1 \"k8s.io\/apimachinery\/pkg\/apis\/meta\/v1\"\n)\nfunc applyPodKill(name, namespace string, labels map[string]string) error {\n    cli, err := controlpanel.NewClient()\n    if err != nil {\n        return errors.Wrap(err, \"create client\")\n    }\n    cr := &v1alpha1.PodChaos{\n        ObjectMeta: metav1.ObjectMeta{\n            GenerateName: name,\n            Namespace:    namespace,\n        },\n        Spec: v1alpha1.PodChaosSpec{\n            Action: v1alpha1.PodKillAction,\n            ContainerSelector: v1alpha1.ContainerSelector{\n                PodSelector: v1alpha1.PodSelector{\n                    Mode: v1alpha1.OnePodMode,\n                    Selector: v1alpha1.PodSelectorSpec{\n                        Namespaces:     []string{namespace},\n                        LabelSelectors: labels,\n                    },\n                },\n            },\n        },\n    }\n    if err := cli.Create(context.Background(), cr); err != nil {\n        return errors.Wrap(err, \"create podkill\")\n    }\n    return nil\n}<\/code><\/pre>\n\n\n\n
The log output of the running program is:<\/p>\n\n\n\n
I1021 00:51:55.225502   23781 request.go:665] Waited for 1.033116256s due to client-side throttling, not priority and fairness, request: GET:https:\/\/***\n2021\/10\/21 00:51:56 apply podkill<\/code><\/pre>\n\n\n\n
Use kubectl to check the status of the PodKill<\/code> resource:<\/p>\n\n\n\n
$ k describe podchaos.chaos-mesh.org -n dev podkillvjn77\nName:         podkillvjn77\nNamespace:    dev\nLabels:       <none>\nAnnotations:  <none>\nAPI Version:  chaos-mesh.org\/v1alpha1\nKind:         PodChaos\nMetadata:\n  Creation Timestamp:  2021-10-20T16:51:56Z\n  Finalizers:\n    chaos-mesh\/records\n  Generate Name:     podkill\n  Generation:        7\n  Resource Version:  938921488\n  Self Link:         \/apis\/chaos-mesh.org\/v1alpha1\/namespaces\/dev\/podchaos\/podkillvjn77\n  UID:               afbb40b3-ade8-48ba-89db-04918d89fd0b\nSpec:\n  Action:        pod-kill\n  Grace Period:  0\n  Mode:          one\n  Selector:\n    Label Selectors:\n      app:  nginx\n    Namespaces:\n      dev\nStatus:\n  Conditions:\n    Reason:  \n    Status:  False\n    Type:    Paused\n    Reason:  \n    Status:  True\n    Type:    Selected\n    Reason:  \n    Status:  True\n    Type:    AllInjected\n    Reason:  \n    Status:  False\n    Type:    AllRecovered\n  Experiment:\n    Container Records:\n      Id:            dev\/nginx\n      Phase:         Injected\n      Selector Key:  .\n    Desired Phase:   Run\nEvents:\n  Type    Reason           Age    From          Message\n  ----    ------           ----   ----          -------\n  Normal  FinalizerInited  6m35s  finalizer     Finalizer has been inited\n  Normal  Updated          6m35s  finalizer     Successfully update finalizer of resource\n  Normal  Updated          6m35s  records       Successfully update records of resource\n  Normal  Updated          6m35s  desiredphase  Successfully update desiredPhase of resource\n  Normal  Applied          6m35s  records       Successfully apply chaos for dev\/nginx\n  Normal  Updated          6m35s  records       Successfully update records of resource<\/code><\/pre>\n\n\n\n
The control plane also needs to query and acquire Chaos resources, so that platform users can view all chaos experiments’ implementation status and manage them. To achieve this, we can call the REST<\/code> API to send the Get<\/code> or List<\/code> request. But in practice, we need to pay attention to the details. At our company, we’ve noticed that each time the controller requests the full amount of resource data, the load of the Kubernetes API server increases.<\/p>\n\n\n\n
I recommend that you read the How to use the controller-runtime client<\/a> (in Japanese) controller runtime tutorial. If you don’t understand Japanese, you can still learn a lot from the tutorial by reading the source code. It covers many details. For example, by default, the controller runtime reads kubeconfig, flags, environment variables, and the service account automatically mounted in the Pod from multiple locations. Pull request #21<\/a> for armosec\/kubescape<\/code><\/a> uses this feature. This tutorial also includes common operations, such as how to paginate, update, and overwrite objects. I haven’t seen any English tutorials that are so detailed.<\/p>\n\n\n\n
Here are examples of Get<\/code> and List<\/code> requests:<\/p>\n\n\n\n
package controlpanel\nimport (\n    \"context\"\n    \"github.com\/chaos-mesh\/chaos-mesh\/api\/v1alpha1\"\n    \"github.com\/pkg\/errors\"\n    \"sigs.k8s.io\/controller-runtime\/pkg\/client\"\n)\nfunc GetPodChaos(name, namespace string) (*v1alpha1.PodChaos, error) {\n    cli := mgr.GetClient()\n    item := new(v1alpha1.PodChaos)\n    if err := cli.Get(context.Background(), client.ObjectKey{Name: name, Namespace: namespace}, item); err != nil {\n        return nil, errors.Wrap(err, \"get cr\")\n    }\n    return item, nil\n}\nfunc ListPodChaos(namespace string, labels map[string]string) ([]v1alpha1.PodChaos, error) {\n    cli := mgr.GetClient()\n    list := new(v1alpha1.PodChaosList)\n    if err := cli.List(context.Background(), list, client.InNamespace(namespace), client.MatchingLabels(labels)); err != nil {\n        return nil, err\n    }\n    return list.Items, nil\n}<\/code><\/pre>\n\n\n\n
This example uses the manager. This mode prevents the cache mechanism from repetitively fetching large amounts of data. The following figure<\/a> shows the workflow:<\/p>\n\n\n\n
    \n
  1. Get the Pod.<\/li>\n\n\n\n
  2. Get the List<\/code> request’s full data for the first time.<\/li>\n\n\n\n
  3. Update the cache when the watch data changes.<\/li>\n<\/ol>\n\n\n\n
    <\/p>\n\n\n\n
    \"List<\/figure>\n\n\n\n
    List request<\/em><\/p>\n\n\n\n
    <\/p>\n\n\n\n
    <\/a>Orchestrate chaos<\/h3>\n\n\n\n
    The container runtime interface (CRI) container runtime provides strong underlying isolation capabilities that can support the stable operation of the container. But for more complex and scalable scenarios, container orchestration is required. Chaos Mesh also provides Schedule<\/code><\/a> and Workflow<\/code><\/a> features. Based on the set Cron<\/code> time, Schedule<\/code> can trigger faults regularly and at intervals. Workflow<\/code> can schedule multiple fault tests like Argo Workflows.<\/p>\n\n\n\n
    Chaos Controller Manager does most of the work for us. The control plane mainly manages these YAML resources. You only need to consider the features you want to provide to end users.<\/p>\n\n\n\n
    <\/a>Platform features<\/h3>\n\n\n\n
    The following figure shows Chaos Mesh Dashboard. We need to consider what features the platform should provide to end users.<\/p>\n\n\n\n
    <\/p>\n\n\n\n
    \"Chaos<\/figure>\n\n\n\n
    Chaos Mesh Dashboard<\/em><\/p>\n\n\n\n
    <\/p>\n\n\n\n
    From the Dashboard, we know that the platform may have these features:<\/p>\n\n\n\n