Collect Logs from a Self-Managed Kubernetes Cluster with Tencent Cloud CLS

Managed Kubernetes platforms usually make log collection feel like a checkbox. Self-managed Kubernetes clusters are different: you need to define what to collect, how to parse it, where to send it, and which node paths the collector can read.

Tencent Cloud CLS supports this model with a Kubernetes-native setup built around a LogConfig custom resource and three collection components:

• LogConfig defines the log source, parsing format, and target CLS topic.
• Log-Provisioner syncs the LogConfig rule to CLS.
• Log-Agent watches Kubernetes and resolves the real host path for container logs.
• LogListener reads, parses, and uploads the logs to CLS.

This guide walks through the deployment flow and the main configuration choices for self-managed Kubernetes log collection.

Prerequisites

Before deploying the collector, prepare:

• A Kubernetes 1.10+ cluster.
• A CLS logset and log topic.
• The target topic ID, usually referred to as topicId.
• The CLS regional endpoint, referred to as CLS_HOST.
• Tencent Cloud API credentials, referred to as TmpSecretId and TmpSecretKey.

If you are using Tencent Kubernetes Engine, the console-based log collection path is usually faster. This guide is for self-managed clusters where you need to deploy the collection components yourself.

The collection architecture

The architecture has two control-plane ideas and two data-plane ideas.

Component	Role
LogConfig	Kubernetes CRD object that says where logs come from, how they are parsed, and which CLS topic receives them.
Log-Provisioner	Watches LogConfig resources and syncs collection rules to CLS.
Log-Agent	Watches cluster and container changes, then computes the actual host path for container logs.
LogListener	Reads log files on the host, parses records, and uploads them to CLS.

In practical terms, you define a LogConfig, provide credentials, deploy the sync component, deploy the node-level collection components, and verify logs in the CLS console.

Deployment flow

The deployment sequence is:

1. Define the LogConfig resource type with a CRD.
2. Define a LogConfig object.
3. Create the LogConfig object in Kubernetes.
4. Create a ConfigMap for CLS authentication.
5. Deploy Log-Provisioner.
6. Deploy Log-Agent and LogListener.
7. Verify logs in the CLS search page.

The deployment examples below use /usr/local/ on the master node as the working path.

Step 1: define the LogConfig CRD

Download the CRD declaration and create the resource type:

wget https://mirrors.tencent.com/install/cls/k8s/CRD.yaml
kubectl create -f /usr/local/CRD.yaml

Once this exists, Kubernetes can store LogConfig objects as custom resources.

Step 2: define a LogConfig object

Download the sample declaration:

wget https://mirrors.tencent.com/install/cls/k8s/LogConfig.yaml

The LogConfig.yaml file has two main sections:

• clsDetail: parsing format and target CLS topicId.
• inputDetail: the log source to collect.

Always replace the sample topicId with the CLS topic you created.

Choose a parsing format

CLS supports several parsing modes. Pick the one that matches the shape of the log before you decide where the log comes from.

Log type	CLS value	When to use it
Single-line full text	minimalist_log	One line is one log event; store content under __CONTENT__.
Multi-line full text	multiline_log	Stack traces or multi-line records where a regex marks the first line.
Single-line full regex	fullregex_log	Structured logs where regex capture groups map to named fields.
Multi-line full regex	multiline_fullregex_log	Multi-line logs that also need field extraction.
JSON	json_log	JSON logs where first-level keys should become fields.
Delimiter	delimiter_log	Logs split by a delimiter such as ::: and mapped to named keys.

Single-line full text

Use minimalist_log when one line is one log event and you do not need field extraction.

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    logType: minimalist_log

Multi-line full text

Use multiline_log when a log event can span several lines, such as a Java stack trace. The beginningRegex marks the start of a new record.

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    logType: multiline_log
    extractRule:
      beginningRegex: \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s.+

Full regex extraction

Use fullregex_log when a single log line should be parsed into named fields.

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    logType: fullregex_log
    extractRule:
      logRegex: (\S+)[^\[]+(\[[^:]+:\d+:\d+:\d+\s\S+)\s"(\w+)\s(\S+)\s([^"]+)"\s(\S+)\s(\d+)\s(\d+)\s(\d+)\s"([^"]+)"\s"([^"]+)"\s+(\S+)\s(\S+).*
      keys: ['remote_addr','time_local','request_method','request_url','http_protocol','http_host','status','request_length','body_bytes_sent','http_referer','http_user_agent','request_time','upstream_response_time']

JSON logs

Use json_log when each log event is a JSON object and first-level keys should become searchable fields.

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    logType: json_log

Delimiter logs

Use delimiter_log when a log line is split by a fixed delimiter.

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    logType: delimiter_log
    extractRule:
      delimiter: ':::'
      keys: ['IP','time','request','host','status','length','bytes','referer']

Choose a log source

CLS supports three Kubernetes log-source categories: container stdout, container files, and host files.

Container stdout

Collect all container stdout logs in the default namespace:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_stdout
    containerStdout:
      namespace: default
      allContainers: true

Collect stdout from a specific deployment:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_stdout
    containerStdout:
      allContainers: false
      workloads:
        - namespace: production
          name: ingress-gateway
          kind: deployment

Collect stdout from pods selected by label:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_stdout
    containerStdout:
      namespace: production
      allContainers: false
      includeLabels:
        k8s-app: nginx

Container files

Collect a file inside a container selected by workload:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: production
      workload:
        name: ingress-gateway
        type: deployment
      container: nginx
      logPath: /data/nginx/log
      filePattern: access.log

Collect a container file from pods selected by label:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: container_file
    containerFile:
      namespace: production
      includeLabels:
        k8s-app: ingress-gateway
      container: nginx
      logPath: /data/nginx/log
      filePattern: access.log

Host files

Collect host files directly:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  inputDetail:
    type: host_file
    hostFile:
      logPath: /data
      filePattern: "*.log"

Step 3: create the LogConfig object

After editing LogConfig.yaml, create the object:

kubectl create -f /usr/local/LogConfig.yaml

Step 4: configure CLS authentication

Log upload requires authentication. Download the ConfigMap template:

wget https://mirrors.tencent.com/install/cls/k8s/ConfigMap.yaml

Set TmpSecretId and TmpSecretKey to your Tencent Cloud API key ID and key. Then create the ConfigMap:

kubectl create -f /usr/local/ConfigMap.yaml

Treat this file as sensitive. Avoid committing filled credential values into a shared repository.

Step 5: deploy Log-Provisioner

Log-Provisioner discovers and watches LogConfig resources, then syncs log topic IDs, collection rules, and file paths to CLS.

Download the declaration:

wget https://mirrors.tencent.com/install/cls/k8s/Log-Provisioner.yaml

Set the CLS_HOST environment variable to the endpoint of the region where the target log topic lives. Then deploy it:

kubectl create -f /usr/local/Log-Provisioner.yaml

Step 6: deploy Log-Agent and LogListener

Log-Agent calculates the real host path for container logs. LogListener reads the host-side log files, parses them, and uploads them to CLS.

Download the declaration:

wget https://mirrors.tencent.com/install/cls/k8s/Log-Agent.yaml

If the host Docker root is not /var/lib/docker, mount the actual Docker root into the agent container. For example, if Docker uses /data/docker, the host path must be exposed to Log-Agent; otherwise the agent may calculate paths that look correct inside Kubernetes metadata but do not exist on the node filesystem.

Docker root on the node	What the agent needs
/var/lib/docker	The default declaration can resolve container log paths.
/data/docker or another custom root	Mount that real host path into the agent container before deploying the DaemonSet.

Deploy Log-Agent and LogListener as a DaemonSet:

kubectl create -f /usr/local/Log-Agent.yaml

Verify logs in CLS

After the DaemonSet is running, open the CLS log search page and verify that logs are being ingested. Select the target topic, set a recent time range, and confirm both signals: the time chart should show incoming log volume, and the result table should contain matching Kubernetes log rows from the expected namespace, workload, container, or file path.

Troubleshooting checklist

• If no logs arrive, confirm the topicId is correct.
• If authentication fails, check TmpSecretId and TmpSecretKey in the ConfigMap.
• If the provisioner runs but rules do not take effect, confirm CLS_HOST matches the topic region.
• If container file logs are missing, check namespace, workload, labels, container name, logPath, and filePattern.
• If stdout logs are missing on a self-managed cluster, verify that the agent can resolve the container log host path.
• If Docker root is not /var/lib/docker, mount the actual root path into the agent container.
• If parsing fields are wrong, validate logType, regex capture groups, delimiter settings, and key order.

FAQ

What is LogConfig in CLS Kubernetes log collection?

LogConfig is the Kubernetes custom resource that defines the collection source, parsing format, and destination CLS topic.

Which component uploads logs to CLS?

LogListener reads and uploads logs. Log-Agent calculates host-side log paths, while Log-Provisioner syncs collection rules.

Can CLS collect both container stdout and files?

Yes. Use container_stdout for stdout, container_file for files inside selected containers, and host_file for files on the node.

Which parsing format should I start with?

Start with minimalist_log or json_log when possible. Move to regex or delimiter parsing when you need structured fields from plain-text logs.

What is the most common self-managed cluster caveat?

Host-path visibility. If Docker stores data somewhere other than /var/lib/docker, mount that actual Docker root into the agent container so it can find container logs.

Closing thought

Self-managed Kubernetes logging works best when you separate four decisions: the log source, the parsing mode, the target CLS topic, and the node paths the collector can read. Once those are explicit in LogConfig and the collector manifests, CLS can handle stdout logs, container files, and host files with the same operational model.