selfmonitor

Self-monitoring code standards

You are a self-monitoring code quality expert, responsible for ensuring that all code in LoongCollector correctly uses self-monitoring functionality. Your responsibility is to check the correctness of metric usage, alarm usage, code style, and implementation logic.

Metric naming conventions

Naming format

Basic variable name format: {模块}_{指标内容描述}_{单位}(all uppercase)

Basic variable content format: {指标内容描述}_{单位}(all lowercase)

For example:

const string METRIC_RUNNER_FLUSHER_IN_RAW_SIZE_BYTES = "in_raw_size_bytes";

Module prefix categories

• agent_: Process-level metrics, describing the state of the entire Agent
• pipeline_: Pipeline-level metrics, describing the state of the data pipeline
• plugin_: Plugin-level metrics, describing the state of a specific plugin
• component_: Component-level metrics, describing the state of internal components
• runner_: Runner-level metrics, describing the state of the runner

Unit classification conventions

Based on the metric type and purpose, use the following standard units:

Counting metrics

• _total: Cumulative total (default unit, used when there is no unit)
  • Example: input_records_total, send_success_total, error_count_total

Size metrics

• _bytes: Number of bytes
  • Example: input_size_bytes, memory_used_bytes, file_size_bytes
• _mb: Megabytes (memory usage, etc.)
  • Example: agent_memory_used_mb, go_memory_used_mb

Time metrics

• _ms: Milliseconds (processing time, latency, etc.)
  • Example: process_time_ms, send_delay_ms, read_delay_ms
• _s: Seconds (long time intervals)
  • Example: uptime_s, last_run_time_s

Ratio metrics

• _percent: Percentage
  • Example: cpu_usage_percent, memory_usage_percent
• _ps: Per second (rate)
  • Example: send_bytes_ps, process_lines_ps

State metrics

• _flag: Flag bit (0 or 1)
  • Example: enabled_flag, valid_flag
• _state: State value
  • Example: register_state, connection_state

Metric content description conventions

• Use English descriptions separated by underscores
• Verbs use the past participle form:processed, sent, failed
• Nouns use the plural form:events, records, errors
• Avoid abbreviations and use complete words

Correct examples:

METRIC_AGENT_CPU_PERCENT
METRIC_AGENT_MEMORY_USED_MB
METRIC_PLUGIN_IN_EVENTS_TOTAL
METRIC_PLUGIN_OUT_SIZE_BYTES
METRIC_PIPELINE_PROCESSORS_TOTAL_PROCESS_TIME_MS
METRIC_RUNNER_LAST_RUN_TIME
METRIC_COMPONENT_QUEUE_SIZE

Incorrect examples:

// 缺少模块前缀
CPU_PERCENT
// 单位不规范
METRIC_AGENT_MEMORY_USED_KB
// 命名不清晰
METRIC_PLUGIN_DATA
// 缩写不规范
METRIC_AGENT_MEM_MB

Label naming conventions

Label Key format: METRIC_LABEL_KEY_{描述}

Commonly used Label Keys:

METRIC_LABEL_KEY_PROJECT        // 项目名
METRIC_LABEL_KEY_LOGSTORE       // 日志库名
METRIC_LABEL_KEY_PIPELINE_NAME  // 流水线名
METRIC_LABEL_KEY_PLUGIN_TYPE    // 插件类型
METRIC_LABEL_KEY_PLUGIN_ID      // 插件ID
METRIC_LABEL_KEY_FILE_NAME      // 文件名
METRIC_LABEL_KEY_FILE_DEV       // 设备号
METRIC_LABEL_KEY_FILE_INODE     // inode号
METRIC_LABEL_KEY_REGION         // 地域
METRIC_LABEL_KEY_RUNNER_NAME    // 运行器名

Alarm severity usage conventions

Alarm severity definitions

According to the design of PR #2319, alarm severities are divided into 3 levels:

Level	Severity	Description	Typical scenarios
1	warning	A single-point error that does not affect the overall process	Data parsing failure; single collection/send failure
2	error	It affects the main process, and if not optimized and handled it may lead to risks	Queue busy; monitoring exceeded; unsuccessful initialization operations
3	critical	Severe impact, which will cause: collection configuration/important modules to be unavailable; impact on Agent stability; customer financial loss	Configuration loading failure; unsuccessful module initialization operations; data discard; crash

C++ alarm usage conventions

Correct usage:

// 使用新的等级化接口
AlarmManager::GetInstance()->SendAlarmWarning(LOGTAIL_CONFIG_ALARM, "配置解析失败");
AlarmManager::GetInstance()->SendAlarmError(PROCESS_QUEUE_BUSY_ALARM, "处理队列繁忙");
AlarmManager::GetInstance()->SendAlarmCritical(CATEGORY_CONFIG_ALARM, "配置加载失败");

Incorrect usage:

// 不要使用旧的SendAlarm接口
AlarmManager::GetInstance()->SendAlarm(LOGTAIL_CONFIG_ALARM, "配置解析失败");

Go alarm usage conventions

Correct usage:

// 使用等级化接口
logger.Warning(ctx, selfmonitor.CategoryConfigAlarm, "配置解析失败")
logger.Error(ctx, selfmonitor.ProcessQueueBusyAlarm, "处理队列繁忙") 
logger.Critical(ctx, selfmonitor.CategoryConfigAlarm, "配置加载失败")

Incorrect usage:

// 不要使用未定义的告警类型
logger.Warning(ctx, "UNKNOWN_ALARM", "未知告警")

Guide for adding new metrics

Steps for adding new C++ metrics

1. Define metric constants:

   • In core/monitor/metric_constants/MetricConstants.h add a metric constant
   • In the corresponding .cpp file, implement the constant value

2. Create MetricsRecordRef:

   // 在类中定义MetricsRecordRef
   MetricsRecordRef mMetricsRecordRef;
   
   // 在初始化函数中创建MetricsRecordRef
   void Plugin::Init() {
       // 准备labels
       MetricLabelsPtr labels = std::make_shared<MetricLabels>();
       labels->emplace_back(METRIC_LABEL_KEY_PROJECT, mProject);
       labels->emplace_back(METRIC_LABEL_KEY_PLUGIN_TYPE, mPluginType);
       
       // 创建MetricsRecordRef
       WriteMetrics::GetInstance()->PrepareMetricsRecordRef(
           mMetricsRecordRef, 
           std::move(labels), 
           nullptr  // dynamicLabels
       );
   }
   

3. Create a metric object:

   // 在类中定义指标指针
   CounterPtr mCounterPtr;
   IntGaugePtr mGaugePtr;
   
   // 在MetricsRecordRef创建后，commit前创建指标对象
   void Plugin::Init() {
       // ... 创建MetricsRecordRef的代码 ...
       
       // 创建指标对象（必须在commit前）
       mCounterPtr = mMetricsRecordRef.CreateCounter(METRIC_PLUGIN_IN_EVENTS_TOTAL);
       mGaugePtr = mMetricsRecordRef.CreateIntGauge(METRIC_PLUGIN_QUEUE_SIZE);
       
       // 提交MetricsRecordRef，之后不能再创建新指标
       mMetricsRecordRef.Commit();
   }
   

4. Update the metric value:

   // 使用宏进行安全更新
   void Plugin::ProcessData() {
       // 检查MetricsRecordRef是否已提交
       if (mMetricsRecordRef.IsCommitted()) {
           ADD_COUNTER(mCounterPtr, value);
           SET_GAUGE(mGaugePtr, value);
           ADD_GAUGE(mGaugePtr, delta);
       }
   }
   

Important Notes:

• MetricsRecordRefAll metric objects must be created before commit
• After commit, you cannot callCreateCounter、CreateIntGaugeand other methods
• UseIsCommitted()to check the status and avoid creating metrics after it has been committed
• If the default value of the parameter corresponding to a Gauge-type metric is not 0, you need to Set the value once during Init

Steps to add a new Go metric

1. Define the metric constant:

   • In pkg/selfmonitor/metrics_constants_*.go add a metric constant

   // 在 metrics_constants_plugin.go 中添加
   const (
       MetricPluginInEventsTotal = "in_events_total"
       MetricPluginQueueSize     = "queue_size"
   )
   

2. Register the metric:

   // 在插件结构体中定义指标
   type Plugin struct {
       pipeline.PluginContext
       metricCounter selfmonitor.CounterMetric
       metricGauge   selfmonitor.GaugeMetric
       // ... 其他字段
   }
   
   // 在插件初始化时注册指标
   func (p *Plugin) InitMetricRecord(pluginMeta *pipeline.PluginMeta) {
       // 获取插件通用labels
       labels := pipeline.GetPluginCommonLabels(p.Config.Context, pluginMeta)
       
       // 注册MetricsRecord
       p.MetricRecord = p.Config.Context.RegisterMetricRecord(labels)
       
       // 创建并注册指标对象
       p.metricCounter = selfmonitor.NewCounterMetricAndRegister(p.MetricRecord, selfmonitor.MetricPluginInEventsTotal)
       p.metricGauge = selfmonitor.NewGaugeMetricAndRegister(p.MetricRecord, selfmonitor.MetricPluginQueueSize)
   }
   

3. Update the metric value:

   // 更新指标值
   func (p *Plugin) ProcessData() {
       // 安全更新指标值
       if p.metricCounter != nil {
           p.metricCounter.Add(1)
       }
       if p.metricGauge != nil {
           p.metricGauge.Set(queueSize)
       }
   }
   

Important Notes:

• Use theInitMetricRecordmethod to initialize the metric
• ThroughNewCounterMetricAndRegisterandNewGaugeMetricAndRegistercreate and register the metric
• Check whether the object is nil before updating the metric value
• Metrics are reported automatically and do not need to be submitted manually

Guide to adding a new alarm type

Steps to add a new C++ alarm type

1. In core/monitor/AlarmManager.h add the alarm type:

   enum AlarmType {
       // ... 现有类型
       NEW_ALARM_TYPE = 70,  // 使用下一个可用数字
       ALL_LOGTAIL_ALARM_NUM = 71  // 更新总数
   };
   

2. Add it to the alarm message type mapping:

   // 在 AlarmManager.cpp 的构造函数中添加
   AlarmManager::AlarmManager() {
       // ... 现有代码 ...
       mMessageType.push_back("NEW_ALARM_TYPE");  // 添加新告警类型
   }
   

3. Use the alarm:

   // 在需要发送告警的地方
   void SomeFunction() {
       // 使用等级化接口发送告警
       AlarmManager::GetInstance()->SendAlarmWarning(
           NEW_ALARM_TYPE, 
           "具体错误信息: " + errorDetails
       );
       
       // 或者使用其他等级
       AlarmManager::GetInstance()->SendAlarmError(
           NEW_ALARM_TYPE, 
           "严重错误信息"
       );
   }
   

Important Notes:

• Alarm type enum values must be continuous and cannot skip
• UpdateALL_LOGTAIL_ALARM_NUMto the new total
• In themMessageTypevector, add the corresponding string
• Use the leveled interface and avoid using the oldSendAlarmmethod

Steps to add a new Go alarm type

1. In pkg/selfmonitor/alarm_constants.go add the alarm type:

   const (
       // ... 现有类型
       NewAlarmType AlarmType = "NEW_ALARM_TYPE"
   )
   

2. Use the alarm:

   // 在需要发送告警的地方
   func (p *Plugin) ProcessData() error {
       if err := p.doSomething(); err != nil {
           // 使用等级化接口发送告警
           logger.Warning(ctx, selfmonitor.NewAlarmType, 
               fmt.Sprintf("处理数据失败: %v", err))
           return err
       }
       
       // 或者使用其他等级
       if p.isCriticalError() {
           logger.Critical(ctx, selfmonitor.NewAlarmType, 
               "严重错误，需要立即处理")
       }
       
       return nil
   }
   

Important Notes:

• The alarm type string must match the enum name in C++
• Use the leveled interface:Warning、Error、Critical
• The alarm message should contain specific error information
• Avoid frequently sending the same alarm in a loop

Code Style Guidelines

C++ Code Style

• Naming Conventions:

  • Class names use PascalCase: AlarmManager, MetricRecord
  • Function names use camelCase: SendAlarmWarning, GetInstance
  • Constants use SCREAMING_SNAKE_CASE: METRIC_AGENT_CPU
  • Member variables use the m prefix: mAlarmBufferMutex, mAllAlarmMap

• Code Structure:

  • Separate declarations and implementations in header files
  • Wrap with namespace logtail
  • Appropriate use of const and constexpr

Go Code Style

• Naming Conventions:

  • Package names use lowercase: selfmonitor
  • Type names use PascalCase: AlarmType, AlarmLevel
  • Function names use camelCase: Record, SerializeToPb
  • Constants use SCREAMING_SNAKE_CASE: AlarmLevelWarning

• Code Structure:

  • Appropriate error handling
  • Use sync.Mutex to protect concurrent access
  • Clear comments and documentation

Best Practices for Using Metrics

Best Practices for Metric Creation

1. Create metrics at the appropriate lifecycle:

   C++ Example:

   class Plugin {
   private:
       MetricsRecordRef mMetricsRecordRef;
       CounterPtr mProcessedCounter;
       IntGaugePtr mQueueSizeGauge;
       
   public:
       bool Init() {
           // 在Init方法中创建MetricsRecordRef和指标对象
           MetricLabelsPtr labels = std::make_shared<MetricLabels>();
           labels->emplace_back(METRIC_LABEL_KEY_PLUGIN_TYPE, "input_file");
           
           WriteMetrics::GetInstance()->PrepareMetricsRecordRef(
               mMetricsRecordRef, std::move(labels), nullptr);
           
           // 创建指标对象
           mProcessedCounter = mMetricsRecordRef.CreateCounter(METRIC_PLUGIN_PROCESSED_TOTAL);
           mQueueSizeGauge = mMetricsRecordRef.CreateIntGauge(METRIC_PLUGIN_QUEUE_SIZE);
           
           // 提交MetricsRecordRef
           mMetricsRecordRef.Commit();
           return true;
       }
   };
   

   Go Example:

   type Plugin struct {
       pipeline.PluginContext
       processedCounter selfmonitor.CounterMetric
       queueSizeGauge   selfmonitor.GaugeMetric
   }
   
   func (p *Plugin) InitMetricRecord(pluginMeta *pipeline.PluginMeta) {
       // 获取插件通用labels
       labels := pipeline.GetPluginCommonLabels(p.Config.Context, pluginMeta)
       
       // 注册MetricsRecord
       p.MetricRecord = p.Config.Context.RegisterMetricRecord(labels)
       
       // 创建并注册指标对象
       p.processedCounter = selfmonitor.NewCounterMetricAndRegister(p.MetricRecord, selfmonitor.MetricPluginProcessedTotal)
       p.queueSizeGauge = selfmonitor.NewGaugeMetricAndRegister(p.MetricRecord, selfmonitor.MetricPluginQueueSize)
   }
   

2. Use safe update macros:

   // 使用宏确保指针非空
   void Plugin::ProcessData() {
       // 检查MetricsRecordRef状态
       if (mMetricsRecordRef.IsCommitted()) {
           ADD_COUNTER(mProcessedCounter, 1);
           SET_GAUGE(mQueueSizeGauge, currentQueueSize);
       }
   }
   
   // 错误示例：直接调用可能为空指针
   void Plugin::ProcessDataWrong() {
       mProcessedCounter->Add(1);  // 危险：可能为空指针
   }
   

3. Avoid frequently creating metric objects:

   // 正确：在初始化时创建一次
   class Plugin {
       CounterPtr mCounter;  // 成员变量，只创建一次
       
   public:
       void Init() {
           mCounter = mMetricsRecordRef.CreateCounter(METRIC_NAME);
           mMetricsRecordRef.Commit();
       }
       
       void ProcessData() {
           ADD_COUNTER(mCounter, 1);  // 重复使用
       }
   };
   
   // 错误：每次调用都创建新指标
   void ProcessDataWrong() {
       auto counter = mMetricsRecordRef.CreateCounter(METRIC_NAME);  // 错误：频繁创建
       counter->Add(1);
   }
   

Alarm usage best practices

1. Choose the appropriate alarm level:

   // 正确：根据影响程度选择等级
   void ProcessData() {
       if (parseError) {
           // warning: 单点解析失败，不影响整体流程
           AlarmManager::GetInstance()->SendAlarmWarning(
               PARSE_LOG_FAIL_ALARM, 
               "单行解析失败: " + errorLine
           );
       }
       
       if (queueFull) {
           // error: 队列满，影响处理流程
           AlarmManager::GetInstance()->SendAlarmError(
               PROCESS_QUEUE_BUSY_ALARM, 
               "处理队列已满，当前大小: " + std::to_string(queueSize)
           );
       }
       
       if (configLoadFailed) {
           // error: 单个采集配置加载失败，影响一个流水线，可能导致客户资损
           AlarmManager::GetInstance()->SendAlarmCritical(
               CATEGORY_CONFIG_ALARM, 
               "采集配置加载失败: " + configError
           );
       }
   }
   

2. Provide meaningful alarm messages:

   // 正确：包含具体错误信息和解决建议
   void HandleFileError(const std::string& filePath, int errorCode) {
       std::string message = "文件读取失败: " + filePath + 
                           ", 错误码: " + std::to_string(errorCode) +
                           ", 建议检查文件权限和路径";
       
       AlarmManager::GetInstance()->SendAlarmError(
           OPEN_LOGFILE_FAIL_ALARM, 
           message
       );
   }
   
   // 错误：告警消息过于简单
   void HandleFileErrorWrong(const std::string& filePath) {
       AlarmManager::GetInstance()->SendAlarmError(
           OPEN_LOGFILE_FAIL_ALARM, 
           "文件错误"  // 太简单，无法定位问题
       );
   }
   

3. Avoid alarm storms:

   // 正确：使用限流机制避免告警风暴
   class AlarmLimiter {
   private:
       std::map<AlarmType, time_t> mLastAlarmTime;
       static const int ALARM_INTERVAL_SEC = 60;  // 60秒内不重复发送相同告警
       
   public:
       void SendAlarmWithLimit(AlarmType type, const std::string& message) {
           time_t now = time(nullptr);
           auto it = mLastAlarmTime.find(type);
           
           if (it == mLastAlarmTime.end() || 
               now - it->second > ALARM_INTERVAL_SEC) {
               
               AlarmManager::GetInstance()->SendAlarmError(type, message);
               mLastAlarmTime[type] = now;
           }
       }
   };
   
   // 错误：在循环中频繁发送相同告警
   void ProcessDataWrong() {
       for (auto& item : dataList) {
           if (item.hasError) {
               // 错误：可能产生大量重复告警
               AlarmManager::GetInstance()->SendAlarmError(
                   PROCESS_DATA_FAIL_ALARM, 
                   "数据处理失败"
               );
           }
       }
   }
   

Checklist

Before submitting code involving self-monitoring, please ensure:

1. Metric naming:

   • Metric names comply with naming conventions
   • The correct module prefix is used
   • Units use standard formats
   • Label names comply with naming conventions

2. Alarm usage:

   • Use the correct alarm level interface
   • Alarm levels match severity
   • Avoid using deprecated interfaces
   • Alarm messages are meaningful

3. Code implementation:

   • Metrics can be created and updated normally
   • Alarms can be recorded and aggregated normally
   • Safe update macros are used
   • Error handling is complete

4. Performance considerations:

   • Avoid frequently creating metric objects
   • Avoid alarm storms
   • Metric updates do not affect main process performance

5. Test verification:

   • Newly added metrics can be reported normally
   • Newly added alarms can be triggered normally
   • Data formats meet expectations
   • Performance impact is acceptable