Deploying Elastic Cloud on Kubernetes (ECK): A Beginner’s Guide

Elastic Cloud on Kubernetes (ECK) is a tool designed to deploy and manage Elasticsearch, Kibana, and other components of the Elastic Stack on Kubernetes clusters.

ECK provides a native Kubernetes operator to automate the deployment, scaling, and management of these components, allowing users to easily set up and run a fully managed Elastic Stack environment on their Kubernetes infrastructure.

ECK also provides integration with other Kubernetes tools and features, such as custom resource definitions, persistent volume claims, and role-based access control, to make it easier for users to configure and manage their Elastic Stack deployment.

main components of Elastic Cloud on Kubernetes (ECK):

1. ECK Operator: The ECK Operator is a core component of ECK that runs in a Kubernetes cluster as a Kubernetes Operator. It automates the deployment and scaling of Elasticsearch, Kibana, APM Server, Beats, and Enterprise Search clusters in a Kubernetes environment.
2. Elasticsearch: Elasticsearch is the distributed search and analytics engine that stores and indexes data. It is a core component of the Elastic Stack and is designed to provide high availability and scalability.
3. Kibana: Kibana is the web-based data visualization and exploration tool that allows users to search, analyze, and visualize data stored in Elasticsearch.
4. APM Server: The Application Performance Monitoring (APM) Server collects and processes application performance metrics. It allows users to monitor and analyze the performance of their applications.
5. Elastic Agent: Agent are lightweight data shippers that can be used to send various types of data, such as logs, metrics, and network data, to Elasticsearch.
6. Enterprise Search: Enterprise Search is a set of tools that allow users to search and explore data from various sources, such as websites, databases, and file systems. It provides a unified interface for searching data and can be integrated with other components of the Elastic Stack, such as Elasticsearch and Kibana.

In our use case, we only use ECK operator, Elasticsearch, Kibana & Elastic Agent.

Below is the steps on how to setup our ECK cluster:

1. Install custom resource definitions:
   kubectl create -f https://download.elastic.co/downloads/eck/2.6.1/crds.yaml
2. Install the operator with its RBAC rules:
   kubectl apply -f https://download.elastic.co/downloads/eck/2.6.1/operator.yaml
3. Create a new Storage Class for allow volume expansion
   kubectl apply -f storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations: {}
  name: elasticsearch
parameters:
  fsType: ext4
  type: gp3
provisioner: kubernetes.io/aws-ebs
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

4. Create a Elastic Search Cluster with 3 nodes (master & data) with the new storage class
kubectl apply -f elasticsearch.yaml

# This sample sets up an Elasticsearch cluster with 3 nodes.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-qa
spec:
  version: 8.6.1
  volumeClaimDeletePolicy: DeleteOnScaledownOnly
  nodeSets:
  - name: default
    config:
      # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: false
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          env: qa
      spec:
        # this changes the kernel setting on the node to allow ES to use mmap
        # if you uncomment this init container you will likely also want to remove the
        # "node.store.allow_mmap: false" setting above
        # initContainers:
        # - name: sysctl
        #   securityContext:
        #     privileged: true
        #     runAsUser: 0
        #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        ###
        # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
        # automountServiceAccountToken: true
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 4Gi
              cpu: 1
          # env:
          # - name: ES_JAVA_OPTS
          #   value: "-Xms2g -Xmx2g"

        # To enforce a strict single node per host
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  elasticsearch.k8s.elastic.co/cluster-name: elasticsearch-qa
              topologyKey: kubernetes.io/hostname

        # schedule to elasticsearch node pool only 
        tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "qa-es"
          effect: "NoSchedule"

        nodeSelector:
          dedicated: qa-es

    count: 3
    # request 2Gi of persistent data storage for pods in this topology element
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: elasticsearch

5. Create Kibana with fleet kubernetes integration configuration
kubectl apply -f kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-qa
spec:
  version: 8.6.1
  count: 1
  elasticsearchRef:
    name: "elasticsearch-qa"
  config:
    monitoring.ui.ccs.enabled: false
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-qa-es-http.elastic-system.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic-system.svc:8220"]
    xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: fleet_server
      version: latest
    - name: kubernetes
      version: latest
    - name: log
      version: latest  
    xpack.fleet.agentPolicies:
    - name: Fleet Server on ECK policy
      id: eck-fleet-server
      namespace: default
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      is_default_fleet_server: true
      package_policies:
      - name: fleet_server-1
        id: fleet_server-1
        package:
          name: fleet_server
    - name: Elastic Agent on ECK policy
      id: eck-agent
      namespace: default
      monitoring_enabled:
      - logs
      - metrics
      unenroll_timeout: 900
      is_default: true
      package_policies:
      - package:
          name: kubernetes
        name: kubernetes-1
      - name: system-1
        id: system-1
        package:
          name: system
      - package:
          name: log
        name: log-1
        inputs:
        - type: logfile
          enabled: true
          streams:
          - data_stream:
              dataset: log.log
            enabled: true
            vars:
            - name: paths
              value:
              - '/var/log/containers/*${kubernetes.container.id}.log'

  podTemplate:
    metadata:
      labels:
        env: qa
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 500m

      # schedule to elasticsearch node pool only 
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "qa-es"
        effect: "NoSchedule"

      nodeSelector:
        dedicated: qa-es

6. Create Fleet server (Deployment) and Elastic Agent (Daemonset)
kubectl apply -f elasticagent.yaml

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
spec:
  version: 8.6.1
  kibanaRef:
    name: kibana-qa
  elasticsearchRefs:
  - name: elasticsearch-qa
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: fleet-server
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        # schedule to elasticsearch node pool only 
        tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "qa-es"
          effect: "NoSchedule"

        nodeSelector:
          dedicated: qa-es

---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata: 
  name: elastic-agent
spec:
  version: 8.6.1
  kibanaRef:
    name: kibana-qa
  fleetServerRef: 
    name: fleet-server
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        containers:
        - name: agent
          resources:
            limits:
              memory: 800Mi
              cpu: 200m
            requests:
              cpu: 200m
              memory: 500Mi
          volumeMounts:
          - mountPath: /var/lib/docker/containers
            name: varlibdockercontainers
          - mountPath: /var/log/containers
            name: varlogcontainers
          - mountPath: /var/log/pods
            name: varlogpods

        volumes:
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods

        serviceAccountName: elastic-agent
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0

        # schedule to elasticsearch node pool only 
        tolerations:
        - operator: "Exists"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-server
rules:
- apiGroups: [""]
  resources:
  - pods
  - namespaces
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fleet-server
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fleet-server
subjects:
- kind: ServiceAccount
  name: fleet-server
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: fleet-server
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""]
  resources:
  - pods
  - nodes
  - namespaces
  - events
  - services
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get
- apiGroups: ["extensions"]
  resources:
    - replicasets
  verbs: 
  - "get"
  - "list"
  - "watch"
- apiGroups:
  - "apps"
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - "get"
  - "list"
  - "watch"
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- apiGroups:
  - "batch"
  resources:
  - jobs
  verbs:
  - "get"
  - "list"
  - "watch"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

7. Next thing is expose the kibana service, there are multiple ways of doing it. For example, create an ingress, loadbalancer, self host nginx and etc.

In conclusion, setting up Elastic Cloud on Kubernetes (ECK) can seem like a daunting task at first, but with the right guidance and resources, it can be a manageable and rewarding process. ECK provides a powerful and flexible way to manage your Elastic Stack deployment, allowing you to scale resources up or down as needed, monitor performance, and automate common tasks. By following the steps outlined in this guide, you can gain the benefits of both Kubernetes and Elastic Stack, and simplify your infrastructure management in the process. With a little bit of patience and persistence, you’ll be up and running with ECK in no time.