When the user asks questions like "My id:xxx name:gis what problem does the service have now?", "Does the xxx_gis service have any problems?" or "Help me check the current status of the service with projectId=xxx, projectName=gis" and so on, wanting to query service monitoring metrics, please follow the 7 steps below to troubleshoot, and summarize and analyze the current status of the service based on the results (note to replace application).
-
Query the current CPU usage Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(rate(container_cpu_user_seconds_total{image!="",application="xxx_gis"}[30s])) by(application) * 100'Example result: "Latest value queried: 32.3487092130255" This means about 32% of CPU is being used (the maximum is not 100%, but the percentage of requested cores; for example, if 3 cores are requested, the maximum is 300%).
-
Query the current CPU request amount Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum((container_spec_cpu_quota{application="xxx_gis", image!=""}) /1000) by (application)'Example result: "Latest value queried: 6181" This means this service has requested a total of 6181% CPU.
-
Query the current Memory usage Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(container_memory_rss{image!="",application="xxx_gis"}) by (application)'Example result: "Latest value queried: 21611335680.0" (unit is byte)
-
Query the current Memory request amount Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(container_spec_memory_limit_bytes{image!="",application="xxx_gis"}) by (application)'Example result: "Latest value queried: 75161927680.0" (unit is byte)
-
Query the current Load Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(container_cpu_load_average_10s{application="xxx_gis"}) by (application)'Example result: "Latest value queried: 1242.0", note that this value should be compared with the CPU request amount; in theory, less than 80% of the request amount is optimal. For example, if the CPU request amount is 2000, then 1242 is not high.
-
Query the current HeapUsed usage Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(jvm_memory_used_bytes{application="xxx_gis"}) by (application)'Example result: "Latest value queried: 15085025864.0" (unit byte)
-
Query the current HeapMax maximum Construct the promql statement and run
prometheus.py, for example:uv run prometheus.py 'sum(jvm_memory_max_bytes{application="xxx_gis"}) by (application)'Example result: "Latest value queried: 100503912448.0" (unit byte)