Reduce CLS Log Costs with Index, Retention, and Storage-Tier Controls
A practical cost-optimization playbook for Tencent Cloud CLS covering full-text index risk, low-frequency storage, topic design, and log-level routing.
Log platforms become expensive when teams store too much, retain it too long, index everything, and route every log level into the same high-cost path. Tencent Cloud CLS gives teams several controls for avoiding that outcome: reduce unnecessary logs, shorten retention, move colder logs to low-frequency storage, and avoid full-text indexing unless it is genuinely needed.
This playbook focuses on storage and traffic because the production example behind it found that those two categories represented 99.73% of a monthly log-topic cost.
The four cost levers
| Lever | Practical action |
|---|---|
| Reduce volume | Stop printing meaningless logs and keep only operationally useful records |
| Reduce retention | Keep logs for the shortest period that matches investigation and compliance needs |
| Move cold logs | Use low-frequency storage for logs that only need retrospective search |
| Avoid broad indexing | Prefer key-value indexes and be careful with full-text indexing and index rebuilds |
The important point is that cost control is not one setting. It is a topic design and log governance habit.
Understand log storage and index storage
CLS bills log storage and index storage separately, and indexed data is measured differently from compressed raw log storage.
Two storage concepts matter:
| Concept | Meaning | Cost implication |
|---|---|---|
| Log storage | Raw logs stored in CLS. LogListener collection has compression, commonly described as a 4-5x compression ratio in CLS usage guidance. | Charged on compressed storage size. |
| Index storage | Storage occupied after indexes are enabled on a topic. | Measured on uncompressed log size, so it can be much larger than raw log storage. |
That difference explains why a topic can look normal by raw log volume while the index portion dominates the bill.
Index configuration choices matter because full-text indexing can make index storage and traffic much larger than key-value indexing. The working rule is simple: use key-value indexes by default, and do not enable full-text indexing casually.
Cost example: full-text indexing makes index storage dominate
The production example uses a standard-storage topic with 30-day retention and full-text indexing enabled. That combination becomes expensive when index storage is several times larger than log storage.
The example values are:
| Metric | Value |
|---|---|
| Standard log storage | 7,867 GB |
| Standard index storage | 44,785 GB |
| Standard storage unit price used in the calculation | RMB 0.0115 / GB / day |
| Retention window | 30 days |
The calculation:
Standard log storage cost = 7,867 * 0.0115 * 30 = RMB 2,714
Standard index storage cost = 44,785 * 0.0115 * 30 = RMB 15,450
Monthly storage cost = 2,714 + 15,450 = RMB 18,164
The lesson is not only "this one topic is expensive". The lesson is that full-text indexing can make index storage many times larger than raw log storage.
Use low-frequency storage when logs are only queried retrospectively
Standard storage fits alerting, dashboards, SQL analysis, and fast query needs; low-frequency storage fits slower retrospective searches.
The storage choice should follow the usage pattern:
| Storage choice | Use when |
|---|---|
| Standard storage | Logs need alerts, dashboards, SQL analysis, short retention, or very fast query response |
| Low-frequency storage | Logs do not need SQL analysis, dashboards, or alerts, and slower retrospective search is acceptable |
| Standard then low-frequency | Recent logs need standard storage, while older logs can move to low-frequency storage after 7 days |
The cost example uses low-frequency storage at RMB 0.0025 / GB / day. Applying a 7-day standard plus 23-day low-frequency policy to the same data gives:
Log storage side = (7,867 * 0.0115 * 7) + (7,867 * 0.0025 * 23) = RMB 1,085
Index storage side = (44,785 * 0.0115 * 7) + (44,785 * 0.0025 * 23) = RMB 6,180
Monthly storage cost = 1,085 + 6,180 = RMB 7,265
Monthly saving = 18,164 - 7,265 = RMB 10,899
Use the same arithmetic in your own pricing calculator before changing a production topic. The important comparison is not only the lower unit price, but how many days remain in standard storage and how many days move to low-frequency storage.
For logs that only need retrieval and backtracking, low-frequency storage can reduce cost by roughly 60-70% in the described scenario.
Treat full-text indexing and index rebuilds as high-risk operations
Traffic cost has two practical optimization directions:
- Reduce original invalid logs at the source.
- Avoid full-text indexing when key-value indexing is enough.
The second point is easy to miss. Enabling full-text indexing converts raw logs into index traffic, and that traffic is measured on the original log size rather than the compressed storage size. Rebuilding indexes reprocesses historical raw logs and can generate additional index traffic cost.
Avoid casual index rebuilds. If a field is added and the team clicks "rebuild index" across historical data, CLS reprocesses the selected historical raw logs and generates index traffic cost. For large topics, that operation can be slow and expensive.
For shared sub-accounts, remove or tightly control the permission to rebuild indexes.
Design one topic per application
In CLS, a log topic should generally map to an application. A broad topic that mixes many applications makes indexing, retention, dashboards, and alerting harder to tune.
When a business team requests log collection, review these fields before creating the topic:
| Decision | Default policy from the playbook |
|---|---|
| Retention | Default to 3 days, usually no more than 7 days |
| Longer retention | More than 7 days should use low-frequency transition or direct low-frequency storage |
| More than 15 days | Require a special explanation |
| More than 30 days | Consider shipping to COS |
| Storage type | Default to low-frequency unless alerts, dashboards, or SQL analysis require standard storage |
| Index strategy | Disable full-text indexing by default; configure key-value indexes for known fields |
This topic design makes each application's cost easier to reason about.
Split log levels into standard and low-frequency topics
Log level is another strong cost-control lever. Suppose an application writes logs under /data/logs/ and separates by file:
user-api-info-2023-05-20.log
user-api-warn-2023-05-20.log
user-api-error-2023-05-20.log
If info logs produce about 10 GB per day while warn and error logs produce about 100 MB per day, route them differently:
| Topic | Collection rule | Storage type | Why |
|---|---|---|---|
user-api-base-log |
/data/logs/user-api-warn-*.log and /data/logs/user-api-error-*.log |
Standard storage | These logs need alerting and operational dashboards |
user-api-low-log |
/data/logs/user-api-info-*.log |
Low-frequency storage | These logs are useful for retrospective search but not always for alerts |
A single standard topic for all log levels can be more expensive than separating warning/error logs from high-volume info logs. Splitting standard and low-frequency topics can preserve operational needs while lowering monthly cost.
The example comparison saves:
RMB 5,979 - RMB 3,886 = RMB 2,090 per month
Even without an existing log-level convention, teams can create one:
/data/logs/user-api-base-2023-05-20.log
/data/logs/user-api-low-2023-05-20.log
The operations side then maps base logs to standard storage and low logs to low-frequency storage.
A compact CLS cost review checklist
Use this checklist before approving a new or existing CLS topic:
- Does the topic map to one application?
- Is full-text indexing disabled unless there is a clear need?
- Are key-value indexes configured for the fields people actually query?
- Is the retention period 3-7 days by default?
- Are logs older than 7 days moved to low-frequency storage when possible?
- Are high-volume
infologs separated fromwarnanderrorlogs? - Is rebuild-index permission restricted?
- Are invalid or low-value logs being reduced at the application source?
FAQ
Why can index storage be larger than log storage?
Raw log storage collected by LogListener can be compressed. Index storage is calculated from indexed data, which is measured differently and can be much larger, especially when full-text indexing is enabled.
Should full-text indexing ever be used?
Yes, but only when the search requirement needs it. The default should be key-value indexing for known fields.
When is low-frequency storage a good fit?
Use it for logs that only need retrieval or backtracking, do not require SQL analysis, do not drive dashboards, and do not need alerting.
Why separate log levels into different topics?
Different log levels have different operational value. warn and error logs often need fast alerting and dashboards, while high-volume info logs may only need lower-cost retrospective search.
What is the most dangerous button for cost?
For large topics, rebuilding indexes can be expensive because it reprocesses historical raw logs and generates index traffic. Restrict that permission.
