ebpf-observability

eBPF Observability

eBPF (extended Berkeley Packet Filter) allows you to run sandboxed programs in the Linux kernel without modifying kernel source code or loading kernel modules. This skill covers using eBPF for deep observability, network monitoring, and security enforcement across cloud-native infrastructure.

---

2. Prerequisites

Kernel Version Requirements

Feature	Minimum Kernel	Recommended Kernel
Basic BPF maps & probes	4.9	5.10+
BPF CO-RE (BTF support)	5.2	5.10+
BPF ring buffer	5.8	5.10+
BPF LSM hooks	5.7	5.15+
Cilium full features	4.19	5.10+
Tetragon	4.19	5.13+

Verify Kernel Support

# Check kernel version
uname -r

# Verify BTF (BPF Type Format) is enabled -- required for CO-RE
ls /sys/kernel/btf/vmlinux

# Check BPF filesystem is mounted
mount | grep bpf

# If not mounted, mount it
sudo mount -t bpf bpf /sys/fs/bpf

# Verify BPF JIT is enabled
cat /proc/sys/net/core/bpf_jit_enable
# Should return 1; if not:
sudo sysctl net.core.bpf_jit_enable=1

Install Toolchain

# Ubuntu/Debian -- install bpftrace, bcc tools, and libbpf
sudo apt-get update
sudo apt-get install -y bpftrace bpfcc-tools libbpf-dev linux-headers-$(uname -r)

# Fedora/RHEL
sudo dnf install -y bpftrace bcc-tools libbpf-devel kernel-devel

# Verify bpftrace works
sudo bpftrace -e 'BEGIN { printf("eBPF is working\n"); exit(); }'

---

3. Cilium Setup

Cilium can optionally replace kube-proxy when configured for the cluster, providing identity-aware security and deep network observability via Hubble.

Install Cilium on Kubernetes

# Add the Cilium Helm repo
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install Cilium with Hubble enabled
helm install cilium cilium/cilium --version 1.16.4 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="${API_SERVER_IP}" \
  --set k8sServicePort="${API_SERVER_PORT}" \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enableOpenMetrics=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload}"

# Wait for Cilium to be ready
cilium status --wait

Install the Cilium CLI and Hubble CLI

# Install reviewed platform-specific Cilium/Hubble CLI releases through the current official process.
# Verify checksums/signatures and archive members before extraction into a private staging directory.
# Do not extract unverified latest archives directly into /usr/local/bin.

Hubble Network Observability

# Port-forward the Hubble Relay
cilium hubble port-forward &

# Observe all flows in real time
hubble observe --follow

# Filter flows by namespace
hubble observe --namespace production --follow

# Filter by verdict (dropped traffic)
hubble observe --verdict DROPPED --follow

# Filter by DNS queries
hubble observe --protocol DNS --follow

# Filter HTTP traffic to a specific service
hubble observe --to-label "app=api-server" --protocol HTTP --follow

# Export flows as JSON for ingestion into SIEM
hubble observe --output json --last 1000 > flows.json

Hubble UI Access

# Port-forward the Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

# Access at http://localhost:12000 -- provides a real-time service dependency map

---

4. Tetragon for Security

Tetragon is Cilium's runtime security enforcement engine. It uses eBPF to observe and enforce security policies at the kernel level with zero application changes.

Install Tetragon

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install tetragon cilium/tetragon \
  --namespace kube-system \
  --set tetragon.grpc.enabled=true \
  --set tetragon.exportFilename=/var/run/cilium/tetragon/tetragon.log

# Install a pinned reviewed tetra CLI using verified release artifacts and a private staging directory.

Process Execution Monitoring

# process-monitor.yaml -- TracingPolicy to monitor all process executions
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: process-execution-monitor
spec:
  kprobes: []
  tracepoints: []
  uprobes: []
  enforcers: []
  # process_exec and process_exit events are always emitted by default
  # Use tetra CLI to observe them:

# Watch all process executions cluster-wide
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --process-exec

# Filter to a specific namespace
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact \
  --namespace production

File Access Tracking

# file-access-policy.yaml -- detect reads/writes to sensitive files
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: sensitive-file-access
spec:
  kprobes:
    - call: "security_file_open"
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"
                - "/etc/kubernetes/pki"
                - "/var/run/secrets/kubernetes.io"
                - "/root/.ssh"

kubectl apply -f file-access-policy.yaml

# Observe file access events
kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact \
  | grep "sensitive-file-access"

Network Connection Enforcement

# restrict-egress.yaml -- block unexpected outbound connections
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: restrict-egress-connections
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchArgs:
            - index: 0
              operator: "DAddr"
              values:
                - "169.254.169.254"  # Block IMDS access
          matchActions:
            - action: Post  # observe only; enforcement requires separate reviewed scope
        - matchNamespaces:
            - namespace: Mnt
              operator: NotIn
              values:
                - "host_mnt"

kubectl apply -f restrict-egress.yaml

Privileged Escalation Detection

# detect-privilege-escalation.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: detect-privilege-escalation
spec:
  kprobes:
    - call: "__x64_sys_setuid"
      syscall: true
      args:
        - index: 0
          type: "int"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "0"
          matchActions:
            - action: Post
              rateLimit: "1m"
    - call: "__x64_sys_setns"
      syscall: true
      args:
        - index: 1
          type: "int"
      selectors:
        - matchActions:
            - action: Post

kubectl apply -f detect-privilege-escalation.yaml

---

5. bpftrace One-Liners

These are version-specific tracing sketches: validate in a test host and obtain production scope, duration, data and overhead limits before attachment.

Syscall Latency

# Trace read() syscall latency distribution (microseconds)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @usecs = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

# Top 10 slowest syscalls by total time
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @start[tid] = nsecs; }
  tracepoint:raw_syscalls:sys_exit /@start[tid]/ {
    @ns[args->id] = sum(nsecs - @start[tid]);
    delete(@start[tid]);
  } END { print(@ns, 10); }'

DNS Tracing

# Trace DNS queries via UDP port 53 sends
sudo bpftrace -e 'kprobe:udp_sendmsg {
    $sk = (struct sock *)arg0;
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    if ($dport == 53) {
      printf("%-8d %-16s DNS query to %s\n", pid, comm,
        ntop($sk->__sk_common.skc_daddr));
    }
  }'

# Count DNS queries by source process
sudo bpftrace -e 'kprobe:udp_sendmsg {
    $sk = (struct sock *)arg0;
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    if ($dport == 53) { @dns[comm] = count(); }
  }'

TCP Retransmits

# Trace TCP retransmits with source/destination
sudo bpftrace -e 'kprobe:tcp_retransmit_skb {
    $sk = (struct sock *)arg0;
    $daddr = ntop($sk->__sk_common.skc_daddr);
    $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    $dport = ($sk->__sk_common.skc_dport >> 8) | (($sk->__sk_common.skc_dport & 0xff) << 8);
    $sport = $sk->__sk_common.skc_num;
    printf("%-20s %-6d -> %-20s %-6d (%s)\n", $saddr, $sport, $daddr, $dport, comm);
  }'

Disk I/O Latency

# Block I/O latency histogram by device
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    @usecs[args->dev] = hist((nsecs - @start[args->dev, args->sector]) / 1000);
    delete(@start[args->dev, args->sector]);
  }'

# Top processes by disk I/O bytes
sudo bpftrace -e 'tracepoint:block:block_rq_issue {
    @bytes[comm] = sum(args->bytes);
  } interval:s:5 { print(@bytes, 10); clear(@bytes); }'

Container-Aware Tracing

# Trace process exec with cgroup IDs; this example has no container-only filter
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("%-8d %-8d %-16s %s\n", pid, cgroup, comm, str(args->filename));
  }'

# Memory allocation hotspots per container
sudo bpftrace -e 'kprobe:__alloc_pages { @pages[cgroup] = count(); }
  interval:s:10 { print(@pages, 10); clear(@pages); }'

---

Contents

• 6. Prometheus Integration
• 7. Network Observability
• 8. Security Monitoring
• 9. Performance Profiling
• 10. Troubleshooting

When to Use

Use eBPF-based observability when you need:

• Deep performance debugging -- trace kernel-level latency, syscall overhead, and scheduling delays that application-level metrics cannot reveal.
• Network observability without sidecars -- capture L3/L4/L7 flows, DNS queries, and TCP state transitions directly from the kernel, eliminating the CPU and memory overhead of sidecar proxies.
• Security monitoring at the kernel boundary -- detect container escapes, unexpected process execution, sensitive file access, and anomalous syscall patterns in real time.
• Continuous profiling -- measure host-specific CPU/memory/event overhead; no under1% guarantee is provided.
• Service mesh replacement or augmentation -- Cilium can replace kube-proxy and provide identity-aware network policies enforced at the kernel level.

Avoid eBPF when your kernel version is below 4.19, when you are running on managed platforms that restrict BPF capabilities, or when your debugging needs are fully met by application-level tracing.

---

Limitations

• Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
• Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.

Example

git status && git diff --stat
kubectl diff -f manifest.yaml

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.

Tracing and deployment limits

Kernel/version tables and Helm values are snapshots; support also depends on BTF, config, architecture, LSM and capabilities. The verifier is not a blanket safety guarantee: tracing can expose secrets and exhaust resources. CNI/kube-proxy replacement can disrupt the whole cluster; inventory current topology/versions, confirm management access and rollback, stage a reviewed chart release and measure connectivity. Observe-only Post policies are not network blocking; do not describe Sigkill as a safe packet deny.

Syscall kprobe naming and argument decoding vary by Tetragon/version/architecture; verify syscall:true expectations and current CRDs before loading. A daemonset kubectl exec/logs target generally selects one pod, not all cluster nodes. Empty TracingPolicy objects do not enable missing event telemetry. File-open events are not proof of file modification; rate limiting can suppress evidence.

Bound every follow/probe by duration/PID/cgroup and secure output permissions/retention. bpftrace examples need actual available fields/types: UDP connected-socket assumptions can miss sendto destinations, dev+sector is not a unique block-request identity, and kernel stack/probe labels do not prove user-level causality. No kernel program, package, cluster policy or live trace was executed.