SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsoms-kafka-performance

oms-kafka-performance

Data & analytics

View repository

Updated 10-16 02:35

About this skill

OceanBase Community Edition → Kafka (OMS): performance troubleshooting and tuning

Use cases

  • Troubleshoot OMS sync to Kafka latency、insufficient throughput or components falling behind (e.g., Incr-Sync lagging behind Store).
  • Tune full import (Full-Import) or incremental sync (Incr-Sync) to Kafka.
  • Configure Kafka data source,KafkaSink read/write concurrency, RPS/BPS rate limiting, serialization, and partitioning rules.
  • Optimize OMS platform threads, batching, connection count, and JVM, or the OceanBase server and table schema to match the sync workload.

Division of labor with other skills

Requirement Do not use this skill instead Should use
Observer / election / rootservice logs This skill observer-log-analysis（gather_log / analyze_log）
OBProxy logs This skill obproxy-log-analysis
Single SQL plan, trace, locks This skill observer-sql-analysis
Overall ASH/AWR for a time period, host perf This skill obdiag-performance
Official documentation parameters, KafkaSink version differences Verbal only oceanbase-knowledge（query with OMS/OB version)

Tools and evidence collection (Agent)

  • OMS logs ≠ analyze_log：analyze_log only observer side. For OMS / Ghana / CDC use the Agent tool gather_oms_log(wraps obdiag gather oms_log), and after success, within the same round, for the pack directory file_list / file_read; if encountering .tar.gz / .zip first run_shell decompress (requires approval) — process aligned with obproxy-log-analysis。
  • config.yml the oms section: must be configured correctly, otherwise gather_oms_log will fail. CDC-type scopes (e.g., cdc、libobcdc) often require oms_component_id, format is IP-进程编号(e.g., 192.168.1.100-1); the specific value can be viewed in the OMS console "Component Management" or obdiag gather oms_log --help .
  • gather_oms_log failure troubleshooting: first check whether config.yml in oms section (host, port, SSH credentials) is filled in completely; second, confirm the OMS machine is SSH-reachable and the corresponding obdiag version supports it; if necessary, use run_shell to view the log path locally on the OMS machine.
  • If there is no Agent and only a terminal: you can manually execute obdiag gather oms_log ...; if obdiag_gather_pack_* already exists, directly use file_list / file_read。

Phase 1: performance troubleshooting approach

  • Check latency: focus on oms_sync_delay , etc.; if Incr-Sync lags behind Store, latency will gradually increase, so start with consumer-side concurrency and resources.
  • Check bottlenecks: distinguish source side (OceanBase) read pressure、OMS processing、target side (Kafka) write pressure; infer from monitoring and rate limiting configuration.
  • Check rate limiting: both full and incremental can be configured with RPS and BPS; actual throughput is also affected by source, target, and specifications, and rate limiting is mostly an upper bound and protection.

Completion criteria: Be able to state which side is currently more suspect (OB / OMS / Kafka), and give the next step (collect logs, check metrics, tuning suggestions).

Phase 2: OMS platform-side tuning (common to full/incremental)

  • Concurrency and threads: e.g., increase limitator.platform.threads.number(example: 32→64).
  • Batch processing: e.g., increase limitator.select.batch.max(example: 1200→2400),limitator.image.insert.batch.max(example: 200→400).
  • Number of connections: e.g., increase limitator.datasource.connections.max(example: 50→200).
  • JVM: Increase the heap according to machine resources (example: from -Xms16g -Xmx16g -Xmn8g to a larger specification).
  • Empirical magnitude: Single-concurrency network inflow is about 1–2M, RPS is about the thousands, depending on fields and row width.

Completion criteria: List the items already suggested and risk warnings (excessively high concurrency may overwhelm the source or Kafka).

Phase 3: Kafka synchronization dedicated configuration

  • Full: Read/write concurrency (maximum both 512, etc., subject to the documentation); full RPS/BPS rate limiting.
  • Incremental: Write concurrency and incremental RPS/BPS; pay attention to the message ordering strategy (see the official documentation).
  • KafkaSink: Serialization (Default, Canal, Debezium, Avro, etc.); partitioning rules (Hash, Table, One, etc.).
  • Note: Increase concurrency gradually and combine with monitoring.

Phase 4: OceanBase source side (in conjunction with high load)

  • Log compression, reduce clog:enable_clog_persistence_compress(off by default) it is recommended to enable.
  • Merge-related:merge_thread_count(recommended 4–16, depending on the number of cores),_mini_merge_concurrency(recommended 4–8),minor_merge_concurrency(recommended 4–8).
  • Dump/freeze and memory: pay attention to freeze_trigger_percentage(default 70) and memstore_limit_percentage(default 50); they can be appropriately increased during high writes.
  • Write throttling when necessary:writing_throttling_trigger_percentage(default 60; lowering it can alleviate write pressure).
  • Change reminder: All the above parameters require ALTER SYSTEM SET ... = ..., executed by the DBA in a maintenance window; confirm the current values before adjustment (can be done via db_query to check gv$ob_parameters）。

Description: The Agent defaults to db_query read-only; when involving changes such as ALTER SYSTEM , output suggested SQL / O&M steps, executed by the DBA in a change window.

Phase 5: Table structure and migration strategy

  • Full migration of partitioned tables: evaluate changing to non-partitioned (the upper limit for a single table is subject to official and on-site specifications); before full migration, secondary indexes can be temporarily dropped and rebuilt after completion.
  • Large tables, wide tables: pay attention to batch size and concurrency, avoid single-threaded or small-batch bottlenecks.

Phase 6: Documentation and community

  • For full/incremental tuning, Kafka data sources, and KafkaSink , refer to the current OMS version documentation .
  • For latency metrics, ordering, authentication (such as GSSAPI), etc., consult community Q&A and cloud vendor help documentation.
  • When parameter names change with versions, use query_oceanbase_knowledge_base to verify,specify the OMS and OB versions in the query。

Output and self-check

  • Output:Bottleneck judgment、Executable tuning items(by OMS / Kafka / OB),next-step evidence collection(log path, metrics, CLI).
  • Self-check：
    • Not used analyze_log to serve as OMS log analysis.
    • When the user asks for "collection + interpretation" of OMS logs, is it called within the same turn gather_oms_log + file_list/file_read(and decompressed if necessary)?
    • Do parameter-change recommendations indicate requires manual execution and risk？