MeshAccessLog

With the MeshAccessLog policy you can easily set up access logs on every data plane proxy in a mesh.

This policy uses a new policy matching algorithm. Do not combine with TrafficLog.

This guide assumes you have already configured your observability tools to work with Kuma. If you haven’t, see the observability docs.

`targetRef` support matrix

Sidecar

Builtin Gateway

Delegated Gateway

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `Dataplane`, `MeshSubset(deprecated)`
`to[].targetRef.kind`	`Mesh`, `MeshService`, `MeshExternalService`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshGateway`, `MeshGateway` with `tags`
`to[].targetRef.kind`	`Mesh`, `MeshService`
`from[].targetRef.kind`	`Mesh`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshSubset`
`to[].targetRef.kind`	`Mesh`, `MeshService`, `MeshExternalService`

To learn more about the information in this table, see the matching docs.

Configuration

Format

Kuma gives you full control over the format of the access logs.

The shape of a single log record is defined by a template string that uses command operators to extract and format data about a TCP connection or an HTTP request.

For example:

%START_TIME% %KUMA_SOURCE_SERVICE% => %KUMA_DESTINATION_SERVICE% %DURATION%

%START_TIME% and %KUMA_SOURCE_SERVICE% are examples of available command operators.

All command operators defined by Envoy are supported, along with additional command operators defined by Kuma:

Command Operator	Description
`%KUMA_MESH%`	Name of the mesh in which traffic is flowing.
`%KUMA_SOURCE_SERVICE%`	Name of a `service` that is the `source` of traffic.
`%KUMA_DESTINATION_SERVICE%`	Name of a `service` that is the `destination` of traffic.
`%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%`	Address of a `Dataplane` that is the `source` of traffic.
`%KUMA_TRAFFIC_DIRECTION%`	Direction of the traffic, `INBOUND`, `OUTBOUND`, or `UNSPECIFIED`.

All additional access log command operators are valid to use with both TCP and HTTP traffic.

Internally, Kuma determines traffic protocol based on the value of kuma.io/protocol tag on the inbound interface of a destination Dataplane.

There are two types of format, plain and json.

Plain accepts a string with command operators and produces a string output.

JSON accepts a list of key-value pairs that produces a valid JSON object.

It is up to the user to decide which format type to use. Some system will automatically parse JSON logs and allow you to filter and query based on available keys.

If a command operator is specific to HTTP traffic, such as %REQ(X?Y):Z% or %RESP(X?Y):Z%, in the case of TCP traffic it will be replaced by a symbol “-” for plain and a null value for json. You can set the format.omitEmptyValues boolean option to change this to "" for plain and omit them entirely for json.

Plain

The default format string for TCP traffic is:

[%START_TIME%] %RESPONSE_FLAGS% %KUMA_MESH% %KUMA_SOURCE_ADDRESS_WITHOUT_PORT%(%KUMA_SOURCE_SERVICE%)->%UPSTREAM_HOST%(%KUMA_DESTINATION_SERVICE%) took %DURATION%ms, sent %BYTES_SENT% bytes, received: %BYTES_RECEIVED% bytes

The default format string for HTTP traffic is:

[%START_TIME%] %KUMA_MESH% "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%KUMA_SOURCE_SERVICE%" "%KUMA_DESTINATION_SERVICE%" "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%" "%UPSTREAM_HOST%"

Example configuration:

format:
  type: Plain
  plain: '[%START_TIME%] %BYTES_RECEIVED%'

Example output:

[2016-04-15T20:17:00.310Z] 154

JSON

Example configuration:

format:
  type: Json
  json:
    - key: "start_time"
      value: "%START_TIME%"
    - key: "bytes_received"
      value: "%BYTES_RECEIVED%"

Example output:

{
  "start_time": "2016-04-15T20:17:00.310Z",
  "bytes_received": "154"
}

TCP configuration with default fields:

format:
  type: Json
  json:
    - key: "start_time"
      value: "%START_TIME%"
    - key: "response_flags"
      value: "%RESPONSE_FLAGS%"
    - key: "kuma_mesh"
      value: "%KUMA_MESH%"
    - key: "kuma_source_address_without_port"
      value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
    - key: "kuma_source_service"
      value: "%KUMA_SOURCE_SERVICE%"
    - key: "upstream_host"
      value: "%UPSTREAM_HOST%"
    - key: "kuma_destination_service"
      value: "%KUMA_DESTINATION_SERVICE%"
    - key: "duration_ms"
      value: "%DURATION%"
    - key: "bytes_sent"
      value: "%BYTES_SENT%"
    - key: "bytes_received"
      value: "%BYTES_RECEIVED%"

HTTP configuration with default fields:

format:
  type: Json
  json:
    - key: "start_time"
      value: "%START_TIME%"
    - key: "kuma_mesh"
      value: "%KUMA_MESH%"
    - key: 'method'
      value: '"%REQ(:METHOD)%'
    - key: "path"
      value: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
    - key: 'protocol'
      value: '%PROTOCOL%'
    - key: "response_code"
      value: "%RESPONSE_CODE%"
    - key: "response_flags"
      value: "%RESPONSE_FLAGS%"
    - key: "bytes_received"
      value: "%BYTES_RECEIVED%"
    - key: "bytes_sent"
      value: "%BYTES_SENT%"
    - key: "duration_ms"
      value: "%DURATION%"
    - key: "upstream_service_time"
      value: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
    - key: 'x_forwarded_for'
      value: '"%REQ(X-FORWARDED-FOR)%"'
    - key: 'user_agent'
      value: '"%REQ(USER-AGENT)%"'
    - key: 'request_id'
      value: '"%REQ(X-REQUEST-ID)%"'
    - key: 'authority'
      value: '"%REQ(:AUTHORITY)%"'
    - key: "kuma_source_service"
      value: "%KUMA_SOURCE_SERVICE%"
    - key: "kuma_destination_service"
      value: "%KUMA_DESTINATION_SERVICE%"
    - key: "kuma_source_address_without_port"
      value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
    - key: "upstream_host"
      value: "%UPSTREAM_HOST%"

Backends

A backend determines where the logs end up.

TCP

A TCP backend streams logs to a server via TCP protocol. You can configure a TCP backend with an address:

backends:
  - type: Tcp
    tcp:
      address: 127.0.0.1:5000

File

A file backend streams logs to a text file. You can configure a file backend with a path:

backends:
  - type: File
    file:
      path: /dev/stdout

OpenTelemetry

An OpenTelemetry (OTel) backend sends data to an OpenTelemetry server. You can configure an OpenTelemetry backend with an endpoint, attributes (which contain additional information about the log) and body (can be a string message, including multi-line, or it can be a structured data). Attributes and endpoints can use placeholders described in the format section.

backends:
  - type: OpenTelemetry
    openTelemetry:
      endpoint: otel-collector:4317
      body:
        kvlistValue:
          values:
            - key: "mesh"
              value:
                stringValue: "%KUMA_MESH%"
      attributes:
        - key: "start_time"
          value: "%START_TIME%"

Body

Body is of type any (defined here) and can be one of the following forms:

body:
  stringValue: "%KUMA_MESH%"

body:
  boolValue: true

body:
  intValue: 123

body:
  doubleValue: 1.2

body:
  bytesValue: aGVsbG8=

body:
  arrayValue:
    values:
      - stringValue: "%KUMA_MESH%"

body:
  kvlistValue:
    values:
      - key: "mesh"
        value:
          stringValue: "%KUMA_MESH%"

Examples

Log outgoing traffic from specific frontend version to a backend service

Kubernetes

Universal

Terraform

I am using MeshService

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: frontend-to-backend
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
      version: canary
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: frontend-to-backend
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
      version: canary
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

I am using MeshService

type: MeshAccessLog
name: frontend-to-backend
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
      version: canary
  to:
  - targetRef:
      kind: MeshService
      name: backend
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

type: MeshAccessLog
name: frontend-to-backend
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
      version: canary
  to:
  - targetRef:
      kind: MeshService
      name: backend
      sectionName: http
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration

resource "konnect_mesh_access_log" "frontend_to_backend" {
  provider = konnect-beta
  type = "MeshAccessLog"
  name = "frontend-to-backend"
  spec = {
    target_ref = {
      kind = "Dataplane"
      labels = {
        app = "frontend"
        version = "canary"
      }
    }
    to = [
      {
        target_ref = {
          kind = "MeshService"
          name = "backend"
          section_name = "http"
        }
        default = {
          backends = [
            {
              type = "File"
              file = {
                path = "/dev/stdout"
              }
            }
          ]
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}

Logging to multiple backends

This configuration logs to three backends: TCP, file and OpenTelemetry.

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: multiple-backends
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  rules:
  - default:
      backends:
      - type: Tcp
        tcp:
          address: 127.0.0.1:5000
          format:
            type: Json
            json:
            - key: start_time
              value: "%START_TIME%"
      - type: File
        file:
          path: "/dev/stdout"
          format:
            type: Plain
            plain: "[%START_TIME%]"
      - type: OpenTelemetry
        openTelemetry:
          endpoint: otel-collector:4317
          body:
            kvlistValue: 
            values:
            - key: mesh
              value:
                stringValue: "%KUMA_MESH%"
          attributes:
          - key: start_time
            value: "%START_TIME%"

    
      type: MeshAccessLog
name: multiple-backends
mesh: default
spec:
  rules:
  - default:
      backends:
      - type: Tcp
        tcp:
          address: 127.0.0.1:5000
          format:
            type: Json
            json:
            - key: start_time
              value: "%START_TIME%"
      - type: File
        file:
          path: "/dev/stdout"
          format:
            type: Plain
            plain: "[%START_TIME%]"
      - type: OpenTelemetry
        openTelemetry:
          endpoint: otel-collector:4317
          body:
            kvlistValue: 
            values:
            - key: mesh
              value:
                stringValue: "%KUMA_MESH%"
          attributes:
          - key: start_time
            value: "%START_TIME%"

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_access_log" "multiple_backends" {
  provider = konnect-beta
  type = "MeshAccessLog"
  name = "multiple-backends"
  spec = {
    rules = [
      {
        default = {
          backends = [
            {
              type = "Tcp"
              tcp = {
                address = "127.0.0.1:5000"
                format = {
                  type = "Json"
                  json = [
                    {
                      key = "start_time"
                      value = "%START_TIME%"
                    }
                  ]
                }
              }
            },
            {
              type = "File"
              file = {
                path = "/dev/stdout"
                format = {
                  type = "Plain"
                  plain = "[%START_TIME%]"
                }
              }
            },
            {
              type = "OpenTelemetry"
              open_telemetry = {
                endpoint = "otel-collector:4317"
                body = {
                  kvlist_value = ""
                  values = [
                    {
                      key = "mesh"
                      value = {
                        string_value = "%KUMA_MESH%"
                      }
                    }
                  ]
                }
                attributes = [
                  {
                    key = "start_time"
                    value = "%START_TIME%"
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Log all incoming and outgoing traffic

For this use case we recommend creating two separate policies. One for incoming traffic:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: all-incoming-traffic
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  rules:
  - default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

    
      type: MeshAccessLog
name: all-incoming-traffic
mesh: default
spec:
  rules:
  - default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_access_log" "all_incoming_traffic" {
  provider = konnect-beta
  type = "MeshAccessLog"
  name = "all-incoming-traffic"
  spec = {
    rules = [
      {
        default = {
          backends = [
            {
              type = "File"
              file = {
                path = "/dev/stdout"
              }
            }
          ]
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

And one for outgoing traffic:

        
          Kubernetes
        
      

        
          Universal
        
      

        
          Terraform
        
      

    
      apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: all-outgoing-traffic
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  to:
  - targetRef:
      kind: Mesh
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

    
      type: MeshAccessLog
name: all-outgoing-traffic
mesh: default
spec:
  to:
  - targetRef:
      kind: Mesh
    default:
      backends:
      - type: File
        file:
          path: "/dev/stdout"

    
      
Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration
resource "konnect_mesh_access_log" "all_outgoing_traffic" {
  provider = konnect-beta
  type = "MeshAccessLog"
  name = "all-outgoing-traffic"
  spec = {
    to = [
      {
        target_ref = {
          kind = "Mesh"
        }
        default = {
          backends = [
            {
              type = "File"
              file = {
                path = "/dev/stdout"
              }
            }
          ]
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}


    
  

Logging traffic going outside the Mesh

To target ExternalServices, use MeshService as the targetRef kind with name set to
the kuma.io/service value.

To target other non-mesh traffic, for example passthrough traffic, use Mesh as the targetRef kind. In this case, %KUMA_DESTINATION_SERVICE% is set to external.

Select a built-in gateway

You can select a built-in gateway using the kuma.io/service value. A current limitation is that traffic routed from a gateway to a service is logged by that gateway as having destination "*".