Try Kubernetes

Tasks

Step-by-step instructions for performing operations with Kubernetes.

Documentation for Kubernetes v1.9 is no longer actively maintained. The version you are currently viewing is a static snapshot. For up-to-date documentation, see the latest version.

Edit This Page

Auditing

FEATURE STATE: Kubernetes v1.9 beta

This feature is currently in a beta state, meaning:

  • The version names contain beta (e.g. v2beta3).
  • Code is well tested. Enabling the feature is considered safe. Enabled by default.
  • Support for the overall feature will not be dropped, though details may change.
  • The schema and/or semantics of objects may change in incompatible ways in a subsequent beta or stable release. When this happens, we will provide instructions for migrating to the next version. This may require deleting, editing, and re-creating API objects. The editing process may require some thought. This may require downtime for applications that rely on the feature.
  • Recommended for only non-business-critical uses because of potential for incompatible changes in subsequent releases. If you have multiple clusters that can be upgraded independently, you may be able to relax this restriction.
  • Please do try our beta features and give feedback on them! After they exit beta, it may not be practical for us to make more changes.

Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. It allows cluster administrator to answer the following questions:

Kube-apiserver performs auditing. Each request on each stage of its execution generates an event, which is then pre-processed according to a certain policy and written to a backend. You can find more details about the pipeline in the design proposal.

Audit Policy

Audit policy defines rules about what events should be recorded and what data they should include. When an event is processed, it’s compared against the list of rules in order. The first matching rule sets the audit level of the event. The audit policy object structure is defined in the audit.k8s.io API group.

You can pass a file with the policy to kube-apiserver using the --audit-policy-file flag. If the flag is omitted, no events are logged. Note: kind and apiVersion fields along with rules must be provided in the audit policy file. A policy with no (0) rules, or a policy that doesn’t provide valid apiVersion and kind values is treated as illegal.

Some example audit policy files:

audit-policy.yaml 
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]

  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]

  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]

  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"

  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]

  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]

  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.

  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

You can use a minimal audit policy file to log all requests at the Metadata level:

# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

The audit profile used by GCE should be used as reference by admins constructing their own audit profiles.

Audit backends

Audit backends implement exporting audit events to an external storage. Kube-apiserver out of the box provides two backends:

In both cases, audit events structure is defined by the API in the audit.k8s.io API group. The current version of the API is v1beta1.

Log backend

Log backend writes audit events to a file in JSON format. You can configure log audit backend using the following kube-apiserver flags:

Webhook backend

Webhook backend sends audit events to a remote API, which is assumed to be the same API as kube-apiserver exposes. You can configure webhook audit backend using the following kube-apiserver flags:

The webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it.

Log Collector Examples

Use fluentd to collect and distribute audit events from log file

Fluentd is an open source data collector for unified logging layer. In this example, we will use fluentd to split audit events by different namespaces.

  1. install fluentd, fluent-plugin-forest and fluent-plugin-rewrite-tag-filter in the kube-apiserver node

  2. create a config file for fluentd

    $ cat <<EOF > /etc/fluentd/config
# fluentd conf runs in the same host with kube-apiserver
<source>
    @type tail
    # audit log path of kube-apiserver
    path /var/log/audit
    pos_file /var/log/audit.pos
    format json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%N%z
    tag audit
</source>

<filter audit>
    #https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13
    type record_transformer
    enable_ruby
    <record>
     namespace ${record["objectRef"].nil? ? "none":(record["objectRef"]["namespace"].nil? ?  "none":record["objectRef"]["namespace"])}
    </record>
</filter>

<match audit>
    # route audit according to namespace element in context
    @type rewrite_tag_filter
    rewriterule1 namespace ^(.+) ${tag}.$1
</match>

<filter audit.**>
   @type record_transformer
   remove_keys namespace
</filter>

<match audit.**>
    @type forest
    subtype file
    remove_prefix audit
    <template>
        time_slice_format %Y%m%d%H
        compress gz
        path /var/log/audit-${tag}.*.log
        format json
        include_time_key true
    </template>
</match>

  3. start fluentd

    $ fluentd -c /etc/fluentd/config  -vv

  4. start kube-apiserver with the following options:

    --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/kube-audit --audit-log-format=json
  5. check audits for different namespaces in /var/log/audit-*.log

Use logstash to collect and distribute audit events from webhook backend

Logstash is an open source, server-side data processing tool. In this example, we will use logstash to collect audit events from webhook backend, and save events of different users into different files.

  1. install logstash

  2. create config file for logstash

    $ cat <<EOF > /etc/logstash/config
input{
    http{
        #TODO, figure out a way to use kubeconfig file to authenticate to logstash
        #https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html#plugins-inputs-http-ssl
        port=>8888
    }
}
filter{
    split{
        # Webhook audit backend sends several events together with EventList
        # split each event here.
        field=>[items]
        # We only need event subelement, remove others.
        remove_field=>[headers, metadata, apiVersion, "@timestamp", kind, "@version", host]
    }
    mutate{
        rename => {items=>event}
    }
}
output{
    file{
        # Audit events from different users will be saved into different files.
        path=>"/var/log/kube-audit-%{[event][user][username]}/audit"
    }
}

  3. start logstash

    $ bin/logstash -f /etc/logstash/config --path.settings /etc/logstash/

  4. create a kubeconfig file for kube-apiserver webhook audit backend

    $ cat <<EOF > /etc/kubernetes/audit-webhook-kubeconfig
apiVersion: v1
clusters:
- cluster:
    server: http://<ip_of_logstash>:8888
  name: logstash
contexts:
- context:
    cluster: logstash
    user: ""
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []
EOF

  5. start kube-apiserver with the following options:

    --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-webhook-config-file=/etc/kubernetes/audit-webhook-kubeconfig
  6. check audits in logstash node’s directories /var/log/kube-audit-*/audit

Note that in addition to file output plugin, logstash has a variety of outputs that let users route data where they want. For example, users can emit audit events to elasticsearch plugin which supports full-text search and analytics.

Legacy Audit

Note: Legacy Audit is deprecated and is disabled by default since Kubernetes 1.8. To fallback to this legacy audit, disable the advanced auditing feature using the AdvancedAuditing feature gate in kube-apiserver:

--feature-gates=AdvancedAuditing=false

In legacy format, each audit log entry contains two lines:

  1. The request line containing a unique ID to match the response and request metadata, such as the source IP, requesting user, impersonation information, resource being requested, etc.
  2. The response line containing a unique ID matching the request line and the response code.

Example output for admin user listing pods in the default namespace:

2017-03-21T03:57:09.106841886-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" ip="127.0.0.1" method="GET" user="admin" groups="\"system:masters\",\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="default" uri="/api/v1/namespaces/default/pods"
2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200"

Configuration

Kube-apiserver provides the following options which are responsible for configuring where and how audit logs are handled:

If an audit log file already exists, Kubernetes appends new audit logs to that file. Otherwise, Kubernetes creates an audit log file at the location you specified in audit-log-path. If the audit log file exceeds the size you specify in audit-log-maxsize, Kubernetes will rename the current log file by appending the current timestamp on the file name (before the file extension) and create a new audit log file. Kubernetes may delete old log files when creating a new log file; you can configure how many files are retained and how old they can be by specifying the audit-log-maxbackup and audit-log-maxage options.

Analytics

Create an Issue Edit this Page