Log Management

OpeNgine implements centralized log management functionality on top of EFK stack (Elasticsearch + Fluentd + Kibana). This documentation covers log collection and storage part of the solution, please refer to dashboard(s) configuration for information about Kibana settings.

OpeNgine provisions centralized storage and indexing service in one of clusters. Log collection agents are provisioned in multiple clusters. In the OpeNgine configuration file environment/vars.yml all log management settings are defined as single log_management: block containing multiple sub-sections - one sub-section per cluster.

Log management configuration in environment/vars.yml file looks like this:

log_management:
  cluster-identifier-1:
    agent:
      type: provision-fluentd
      role_name: "opengine-fluentd"
      account_name: "opengine-fluentd"
      index_prefix: "the-first-cluster-fluentd"
      helm:
        repo:
          name: "stable"
          url: "https://kubernetes-charts.storage.googleapis.com/"
        chart:
          app_name: "opengine-fluentd"
          port_name: "opengine-fld"
          name: "fluentd-elasticsearch"
          version: "2.0.7"
          appVersion: "2.3.2"
    storage:
      type: provision-elasticsearch
      helm:
        repo:
          name: "elasticsearch"
          url: "https://kubernetes-charts.storage.googleapis.com/"
        chart:
          app_name: "opengine-elasticsearch"
          name: "elasticsearch"
          version: "1.21.0"
          appVersion: "6.6.0"
          clusterName: "elasticsearch"
          timeout: 300
        master:
          minimum_master_nodes: 2
          replicas: 2
        client:
          replicas: 2
        data:
          replicas: 2
  cluster-identifier-N:
    agent:
      type: provision-fluentd
      role_name: "opengine-fluentd"
      account_name: "opengine-fluentd"
      index_prefix: "the-second-cluster-fluentd"
      helm:
        repo:
          name: "stable"
          url: "https://kubernetes-charts.storage.googleapis.com/"
        chart:
          app_name: "opengine-fluentd"
          port_name: "opengine-fld"
          name: "fluentd-elasticsearch"
          version: "2.0.7"
          appVersion: "2.3.2"

Element

Description

log_management

Parent element of log management configuration, contains list clusters to be configured.

cluster-identifier-1cluster-identifier-N

Kubernetes cluster identifier(s). Groups log management settings. OpeNgine will install and configure Elasticsearch and/or Fluentd on cluster(s) using these identifiers.

agent

Parent element for log collection agent settings.

agent.type

Currently one type is supported - provision-fluentd.

agent.role_name and agent.account_name

OpeNgine will create these role and account to run Fluentd.

agent.index_prefix

To aggregate/segregate logs by cluster origin, each agent should have unique index prefix. See dashboard(s) configuration for information on how these prefixes are used.

storage

Parent element for log storage settings.

storage.type

Currently one type is supported - provision-elasticsearch.

agent.helm and storage.helm

Parent element containing Helm chart details. Allows to control which version is installed and from which repository.

helm.repo.name and helm.repo.url

Chart name and repository location.

helm.chart

List of settings including: application version, chart version, appication name etc.