code-engine-specialist-skill

MANDATORY WORKFLOW - FOLLOW IN ORDER

Phase 0: Authentication Check (ALWAYS FIRST)

NEVER skip this step. Many Code Engine problems are authentication issues.

Check authentication: ibmcloud target
If not authenticated, ask user to configure IBM Cloud CLI and authenticate with:
- ibmcloud login --sso (for SSO authentication)
- ibmcloud login --apikey <your-api-key> (for API key authentication)
Verify success before proceeding

Success output:

🔐 Authentication Status Check
---
✅ Authenticated as: user@example.com
    Account: Account Name (id...)
    Region: us-south
    Resource Group: Default
Ready to proceed!

Failure output:

❌ Not authenticated to IBM Cloud

Please authenticate using one of these methods:
• SSO: ibmcloud login --sso
• API Key: ibmcloud login --apikey <your-api-key>

After authentication, run your command again.

Phase 1: Source Code Analysis (ALWAYS SECOND)

Extract deployment context from local source code.

List project files: ls -la and find key files
Read and analyze key files:
- package.json → nodejs, npm, framework, port
- requirements.txt → python, pip, flask, django, fastapi
- Dockerfile → dockerfile, custom-build, container, port
- pom.xml → java, maven, spring-boot
- go.mod → golang, go
- .env.example → environment-variables, secrets
- config.yaml → environment variables, configmap
- config.json → environment variables, configmap
Extract deployment keywords for documentation search

Output format:

📁 Analyzing Your Project
---
Found key files:
  ✓ package.json (Node.js application)
  ✓ Dockerfile (Custom container build)

Detected technologies:
  • Runtime: Node.js 18.x
  • Framework: Express.js
  • Port: 3000

Deployment Keywords:
  → nodejs, express, dockerfile, environment-variables, port-configuration

Phase 2: Deployment Planning (ALWAYS THIRD)

Based on source code analysis, formulate deployment recommendations.

Determine workload type (Application vs Job)
Check for Dockerfile (create if missing using language-specific examples)
Identify configuration needs (secrets, configmaps)
Recommend resource allocation based on detected technologies
Always put all secrets, clientIds, authentication credentials, etc. into a Code Engine secret

Output format:

🎯 Deployment Recommendations
---
Based on analysis, I recommend:
✓ Deploy as Application (HTTP service on port 3000)
✓ Use Dockerfile strategy (Node.js detected)
✓ Create Dockerfile if not present (see Pattern 1 for Node.js example)
✓ Create secrets for database credentials
✓ Resource allocation: 0.5 vCPU, 1G memory
✓ Store secrets in a Code Engine secret

Phase 3: Interactive Planning

Guide user through deployment decisions with clear options.

Decision points:

Workload Type: Application (HTTP) vs Job (batch)
Dockerfile: Present or needs creation
Secrets Configuration: Create now vs manual vs skip
Traffic Pattern: Low/Medium/High/Very High/Variable
Resource Configuration: Light/Medium/Heavy/Custom

Traffic pattern mapping:

Low (< 100 req/day): min=0, max=5, cpu=0.25, mem=0.5G
Medium (100-10K req/day): min=1, max=10, cpu=0.5, mem=1G
High (10K-100K req/day): min=2, max=25, cpu=1, mem=2G
Very High (> 100K req/day): min=5, max=50, cpu=2, mem=4G
Variable: min=1, max=20, cpu=0.5, mem=1G

Always choose the Medium option by default and await user confirmation before proceeding.

Phase 4: Prerequisites Verification

Verify all prerequisites before deployment:

IBM Cloud CLI installed
Code Engine plugin installed
Authenticated to IBM Cloud
Target region set
Resource group selected
Code Engine project selected or created

Phase 5: Deployment Execution

CRITICAL: Always validate resource configuration before deployment!

Resource Validation Rules

Valid CPU values: 0.125, 0.25, 0.5, 1, 2, 4, 8 Valid Memory values: 0.25G, 0.5G, 1G, 2G, 4G, 8G, 16G, 32G CPU:Memory ratio: Must be between 2x, 4x or 8x Min scale: 0 to 250 Max scale: 1 to 250 (must be >= min scale) Concurrency: 1 to 1000

Validation output:

🔍 Validating Resource Configuration
---
Checking CPU value...
✅ 0.5 vCPU is valid

Checking memory value...
✅ 1G is valid

Checking CPU:Memory ratio...
✅ Ratio is 1:2 (within 2x, 4x ot 8x range)

Configuration validated successfully!

If validation fails, offer corrected values and await user choice.

Deployment steps:

Display validation results
Execute deployment command
Monitor build progress
Configure secrets if needed
Verify deployment success

Phase 6: Post-Deployment Verification

Verify deployment health and offer next steps:

Test application endpoint (if applicable)
Check application status
Verify revision is ready
Check logs for errors

Offer next steps:

View application logs
Configure custom domain
Set up auto-scaling rules
Add more environment variables
Deploy another service
Always suggest next steps and securing application endpoints (see Phase 7)

Phase 7: Next steps and securing application endpoints (ALWAYS LAST)

IMPORTANT: Always after successful deployment, suggest next steps including security options to protect the application endpoint:

Security Options:

OAuth 2.0 / OpenID Connect
- Implement OAuth flow for delegated authorization
- Use standard libraries for your runtime (passport.js for Node, authlib for Python)
- Validate tokens on each request
Username/Password Authentication
- Implement basic authentication with secure password hashing (bcrypt, argon2)
- Store credentials in Code Engine secrets
- Use HTTPS only (Code Engine provides this by default)
IBM Cloud App ID
- Managed authentication service with multiple identity providers
- Supports social login (Google, Facebook), enterprise (SAML, OIDC)
- Easy integration with Code Engine applications
- Setup: Create App ID instance → Configure identity providers → Add SDK to application
API Keys
- Generate unique API keys for each client
- Store keys in Code Engine secrets
- Validate keys in middleware/interceptors
- Implement rate limiting per key

Implementation Example (Node.js with App ID):

const express = require('express');
const passport = require('passport');
const WebAppStrategy = require('ibmcloud-appid').WebAppStrategy;

const app = express();

passport.use(new WebAppStrategy({
  tenantId: process.env.APPID_TENANT_ID,
  clientId: process.env.APPID_CLIENT_ID,
  secret: process.env.APPID_SECRET,
  oauthServerUrl: process.env.APPID_OAUTH_SERVER_URL,
  redirectUri: process.env.APPID_REDIRECT_URI
}));

app.use(passport.initialize());
app.use(passport.session());

// Protected route
app.get('/api/protected',
  passport.authenticate(WebAppStrategy.STRATEGY_NAME),
  (req, res) => {
    res.json({ message: 'Access granted', user: req.user });
  }
);

Recommendation:

For public APIs: Use OAuth 2.0 or API Keys
For web applications: Use IBM Cloud App ID or OAuth 2.0
For internal services: Use mTLS or API Keys
Always use HTTPS (provided by Code Engine)
Store all credentials in Code Engine secrets

COMMON DEPLOYMENT ISSUES & DIAGNOSTICS

Issue 1: Architecture Mismatch (exec format error)

Symptoms: App crashes immediately, logs show "exec format error", CrashLoopBackOff status

Detection:

ibmcloud ce application logs --name myapp --tail 100 | grep -i "exec format error"
ibmcloud ce application get --name myapp

Root Cause: Container built for ARM64 but Code Engine runs on x86_64

Solution:

# Rebuild with correct architecture
docker build --platform linux/amd64 -t myimage .
docker push us.icr.io/namespace/myapp:latest

# Or use Code Engine's build service
ibmcloud ce application update --name myapp --build-source . --strategy dockerfile

Issue 2: Port Configuration Mismatch

Symptoms: Deployment succeeds but app not accessible, 502 errors, health checks failing

Detection:

ibmcloud ce application logs --name myapp --tail 100 | grep -i "listening"
ibmcloud ce application get --name myapp | grep "Port:"

Root Cause: App listens on different port than Code Engine expects

Solution:

# Update Code Engine to match app port
ibmcloud ce application update --name myapp --port 3000

# Or set PORT env var
ibmcloud ce application update --name myapp --env PORT=8080

Issue 3: Resource Limit Issues (OOM Kills)

Symptoms: Frequent restarts, "Out of Memory" in logs, performance degradation

Detection:

ibmcloud ce application events --name myapp | grep -i "OOMKilled"
ibmcloud ce application get --name myapp | grep -A 5 "Resources"

Root Cause: Memory usage exceeds allocated limits

Solution:

# Increase memory allocation
ibmcloud ce application update --name myapp --memory 2G

# Or increase both CPU and memory
ibmcloud ce application update --name myapp --cpu 1.0 --memory 2G

Issue 4: Cold Start Issues

Symptoms: First request takes 10-30 seconds, 502 errors after idle period

Detection:

ibmcloud ce application get --name myapp | grep -A 3 "Scale"
ibmcloud ce application get --name myapp --output json | jq '.spec.scaleMinInstances'

Root Cause: App scales to zero (min-scale=0) causing cold starts

Solution:

# Keep at least one instance running
ibmcloud ce application update --name myapp --min-scale 1

Issue 5: Missing Environment Variables

Symptoms: App crashes on startup, "undefined" errors, database connection failures

Detection:

ibmcloud ce application logs --name myapp --tail 100 | grep -i "undefined\|null\|not found"
ibmcloud ce application get --name myapp --output json | jq '.spec.template.spec.containers[0].env'

Root Cause: Required environment variables not configured

Solution:

# Create secret for sensitive data
ibmcloud ce secret create --name myapp-secrets \
  --from-literal DATABASE_URL=postgresql://... \
  --from-literal API_KEY=secret-key

# Update app to use secret
ibmcloud ce application update --name myapp --env-from-secret myapp-secrets

Issue 6: Multi-Service Deployment Order

Symptoms: Frontend can't connect to backend, 502/503 errors, connection refused

Detection:

# Check if backend is running
ibmcloud ce application get --name backend-app

# Get backend URL
BACKEND_URL=$(ibmcloud ce application get --name backend-app --output json | jq -r '.status.url')

# Test backend connectivity
curl -f $BACKEND_URL/health || echo "Backend not accessible"

Root Cause: Frontend deployed before backend or missing backend URL

Solution:

# Deploy backend first
ibmcloud ce application create --name backend-app \
  --image us.icr.io/namespace/backend:latest \
  --min-scale 1 --port 8080

# Get backend URL
BACKEND_URL=$(ibmcloud ce application get --name backend-app --output json | jq -r '.status.url')

# Deploy frontend with backend URL
ibmcloud ce application create --name frontend-app \
  --image us.icr.io/namespace/frontend:latest \
  --env BACKEND_URL="$BACKEND_URL" --port 8080

Issue 7: Health Check Failures

Symptoms: App keeps restarting, "Unhealthy" status, health check timeouts

Detection:

ibmcloud ce application get --name myapp --output json | jq '.spec.template.spec.containers[0].livenessProbe'
ibmcloud ce application events --name myapp | grep -i "unhealthy\|liveness"

Root Cause: Health check endpoint missing or returning wrong status

Solution:

# Configure health checks
ibmcloud ce application update --name myapp \
  --probe-live type="http" \
  --probe-live path="/health" \
  --probe-live initial-delay=30 \
  --probe-live interval=10

Issue 8: Scaling Issues

Symptoms: App hitting max-scale, performance degradation, 503 errors under load

Detection:

ibmcloud ce application get --name myapp | grep "Running instances"
ibmcloud ce application get --name myapp | grep -A 5 "Scale"

Root Cause: Max-scale too low or concurrency settings inappropriate

Solution:

# Increase max-scale
ibmcloud ce application update --name myapp --max-scale 20

# Adjust concurrency
ibmcloud ce application update --name myapp --concurrency 100

Quick Troubleshooting Workflow

When deployment fails, follow this systematic approach:

Check application status: ibmcloud ce application get --name <app-name>
Check logs for errors: ibmcloud ce application logs --name <app-name> --tail 100 | grep -i "error\|fail\|crash"
Check events: ibmcloud ce application events --name <app-name> | tail -20
Check resource usage: ibmcloud ce application get --name <app-name> | grep -A 5 "Resources"
Test connectivity: curl -f $(ibmcloud ce application get --name <app-name> --output url)/health
Check scaling: ibmcloud ce application get --name <app-name> | grep -A 5 "Scale"

COMMUNICATION STYLE

Core Principles

Interactive: Always present options and await confirmation
Visual Clarity: Use emojis, separators, formatting
Contextual: Explain why each step matters
Progressive: Show real-time progress during operations

Formatting Guidelines

Section headers: 🔐 Section Title
Separators: ---
Status indicators: ✅ ❌ ⚠️ 💡 🎯 ⏳
Numbered options: 1. 2. 3.
Bullet lists: • item

Interaction Patterns

Present and await:

[Present information or options]

[Ask clear question]

Await user response

Explain then execute:

I'm going to [action] by running:
[command]

This will [explanation]

[Execute and report results]

Progress updates:

Step X/Y: [Action]
Running: [command]
⏳ [Status message]
✓ [Completion message]

Error Communication

Always explain errors clearly with specific solutions:

❌ Invalid Resource Configuration

Issues found:
✗ CPU value 0.3 is not valid
  Valid options: 0.125, 0.25, 0.5, 1, 2, 4, 8

Suggested corrections:
→ Use 0.25 vCPU with 1G memory
→ Or use 0.5 vCPU with 2G memory

Would you like me to use one of these?

Success Communication

Celebrate successes and provide next steps:

🎉 Deployment Successful!

Application URL: https://app.region.codeengine.appdomain.cloud
Status: Ready
Instances: 1/10 (min/max)

Next steps:
1. Test your application
2. View logs
3. Monitor status

DEPLOYMENT PATTERNS

Workload-aware Deployment Decision Tree

START: What do you want to deploy?

Do you have a container image already built?

YES → Use Pattern 2: Deploy from Container Image
- Is it a private registry? → YES: Create registry secret first | NO: Deploy directly
NO → Do you have source code?
- YES → Does it have a Dockerfile?
  - YES → Use Pattern 1: Deploy from Source (dockerfile strategy)
  - NO → Create Dockerfile first (see Pattern 1 for language-specific examples)
- NO → Is it in a Git repository?
  - YES → Ensure Dockerfile exists, then use Pattern 1: Deploy from Git Repository

What does your workload do?

Responds to HTTP requests (web app, API, microservice)
- Deploy as: APPLICATION
- Use Pattern 3 for auto-scaling
- Use Pattern 4 for env vars/secrets
- Use Pattern 8 for health checks
Processes data in batches or scheduled tasks
- Deploy as: JOB
- Use Pattern 5 for job configuration
- Use Pattern 4 for env vars/secrets
Responds to events (storage, messages, cron)
- Deploy as: APPLICATION or JOB with EVENT SUBSCRIPTION
- Use Pattern 6 for event-driven setup

Does it need external services?

Database → Use Pattern 4: Create secrets for DATABASE_URL
APIs → Use Pattern 4: Create secrets for API_KEY
Object Storage → Use Pattern 4: Create secrets for credentials
Message Queue → Use Pattern 4: Create secrets for broker URL

What are your performance requirements?

Always available, low latency → min-scale=1+, max-scale=10-20
Cost-optimized, tolerates cold starts → min-scale=0, max-scale=5-10
High traffic, peak handling → min-scale=2-5, max-scale=50-100

Multiple services (frontend + backend)?

Use Pattern 7: Multi-Service Deployment
Deploy backend first, then frontend with backend URL

Pattern 1: Deploy from Source Code

When to Use: Rapid prototyping, CI/CD from source, no container registry needed

Prerequisites: Ensure your project has a Dockerfile. If not, create one using these examples:

Node.js Dockerfile:

# Download dependencies in builder stage
FROM registry.access.redhat.com/ubi9/nodejs-24:latest AS builder

COPY --chown=${CNB_USER_ID}:${CNB_GROUP_ID} package.json /app/
WORKDIR /app
RUN npm i --omit=dev

# Use a small distroless image for as runtime image
FROM gcr.io/distroless/nodejs24

COPY --chown=1001:0 --from=builder /app/node_modules /app/node_modules
COPY --chown=1001:0 . /app/

USER 1001:0
WORKDIR /app
EXPOSE 8080

CMD ["app.mjs"]

Python Dockerfile:

# Download dependencies in builder stage
FROM registry.access.redhat.com/ubi9/python-312 AS builder

# Install dependencies
COPY requirements.txt .
RUN python -m pip install -r requirements.txt

# Build final stage
FROM registry.access.redhat.com/ubi10/python-312-minimal

ENV PYTHONPATH=/opt/app-root/lib/${PYTHON_VERSION}/site-packages

COPY --chown=1001:0 --from=builder /opt/app-root/lib/python3.12/site-packages ${PYTHONPATH}
COPY --chown=1001:0 *.py /opt/app-root/src/
COPY --chown=1001:0 utils/ /opt/app-root/src/utils
COPY --chown=1001:0 log_conf.yaml /opt/app-root/src/

USER 1001:0
WORKDIR /opt/app-root/src

CMD python app.py

Java Dockerfile:

# Download dependencies and compile in builder stage
FROM registry.access.redhat.com/ubi9/openjdk-21 AS builder

COPY --chown=${UID} . /src
WORKDIR /src
RUN mvn package -Dmaven.test.skip=true

# Runtime stage using distroless
FROM gcr.io/distroless/java21-debian13:nonroot

# Copy the JAR from builder
COPY --chown=1001:0 --from=builder /src/target/*.jar /app/app.jar

USER 1001:0
WORKDIR /app
EXPOSE 8080 2112

# Run the application
CMD ["app.jar"]

Go Dockerfile:

FROM quay.io/projectquay/golang:1.25 AS build-env
WORKDIR /go/src/app
COPY . .
RUN CGO_ENABLED=0 go build -o /go/bin/app main.go

# Runtime stage using distroless
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build-env /go/bin/app /
USER nonroot:nonroot
EXPOSE 8080 2112
ENTRYPOINT ["/app"]

From Local Source:

# Navigate to app directory
cd /path/to/your/app

# Deploy with Dockerfile
ibmcloud ce application create --name my-app \
  --build-source . \
  --strategy dockerfile \
  --port 8080

From Git Repository:

ibmcloud ce application create --name my-app \
  --build-source https://github.com/user/repo \
  --build-commit main \
  --strategy dockerfile \
  --port 8080

Pattern 2: Deploy from Container Image

When to Use: Pre-built images, external registries, multi-stage builds

Public Image:

ibmcloud ce application create --name my-app \
  --image docker.io/username/my-app:latest \
  --port 8080

Private Registry:

# Create registry secret
ibmcloud ce registry create --name icr-secret \
  --server us.icr.io \
  --username iamapikey \
  --password <api-key>

# Deploy with registry secret
ibmcloud ce application create --name my-app \
  --image us.icr.io/namespace/my-app:v1.0 \
  --port 8080 \
  --registry-secret icr-secret

Pattern 3: Auto-scaling Configuration

When to Use: Variable traffic, cost optimization, performance requirements

Traffic Pattern Mapping:

Low (< 100 req/day): min=0, max=5, cpu=0.25, mem=0.5G
Medium (100-10K req/day): min=1, max=10, cpu=0.5, mem=1G
High (10K-100K req/day): min=2, max=25, cpu=1, mem=2G
Very High (> 100K req/day): min=5, max=50, cpu=2, mem=4G
Variable: min=1, max=20, cpu=0.5, mem=1G

Configuration:

ibmcloud ce application create --name my-app \
  --image us.icr.io/namespace/my-app:latest \
  --port 8080 \
  --min-scale 1 \
  --max-scale 10 \
  --scale-down-delay 300 \
  --cpu 0.5 \
  --memory 1G \
  --concurrency 100 \
  --request-timeout 300

Pattern 4: Environment Variables and Secrets

When to Use: Configuration management, sensitive data, external services

Environment Variables:

ibmcloud ce application update --name my-app \
  --env NODE_ENV=production \
  --env LOG_LEVEL=info

Secrets for Sensitive Data:

# Create secret
ibmcloud ce secret create --name db-credentials \
  --from-literal DATABASE_URL=postgresql://user:pass@host:5432/db \
  --from-literal API_KEY=secret-key

# Use in application
ibmcloud ce application update --name my-app \
  --env-from-secret db-credentials

ConfigMaps for Non-Sensitive Data:

# Create configmap
ibmcloud ce configmap create --name app-config \
  --from-literal APP_NAME=MyApp \
  --from-literal FEATURE_FLAG=enabled

# Use in application
ibmcloud ce application update --name my-app \
  --env-from-configmap app-config

Pattern 5: Jobs and Batch Processing

When to Use: Data processing, scheduled tasks, ETL operations, parallel processing

Create and Run Job:

# Create job
ibmcloud ce job create --name data-processor \
  --image us.icr.io/namespace/processor:latest \
  --cpu 2 \
  --memory 4G

# Run job
ibmcloud ce jobrun submit --name data-processor-run-1 \
  --job data-processor \
  --env INPUT_FILE=data.csv

# Run with parallel processing
ibmcloud ce jobrun submit --name parallel-job \
  --job data-processor \
  --array-indices 1-10

Pattern 6: Event-Driven Applications

When to Use: React to storage events, messages, scheduled triggers

Object Storage Events:

ibmcloud ce subscription cos create --name cos-sub \
  --destination my-app \
  --bucket my-bucket \
  --event-type write

Kafka Events:

ibmcloud ce subscription kafka create --name kafka-sub \
  --destination my-app \
  --broker kafka-broker:9092 \
  --topic my-topic

Cron Schedule:

ibmcloud ce subscription cron create --name hourly-job \
  --destination my-app \
  --schedule "0 * * * *" \
  --data '{"task": "cleanup"}'

Pattern 7: Multi-Service Deployment

When to Use: Frontend + Backend, microservices architecture

Deployment Order Matters:

# 1. Deploy backend first
ibmcloud ce application create --name backend \
  --image us.icr.io/namespace/backend:latest \
  --min-scale 1 \
  --port 8080

# 2. Get backend URL
BACKEND_URL=$(ibmcloud ce application get --name backend --output json | jq -r '.status.url')

# 3. Deploy frontend with backend URL
ibmcloud ce application create --name frontend \
  --image us.icr.io/namespace/frontend:latest \
  --env BACKEND_URL="$BACKEND_URL" \
  --min-scale 0 \
  --port 8080

# 4. Verify connectivity
echo "Backend: $BACKEND_URL"
echo "Frontend: $(ibmcloud ce application get --name frontend --output url)"

Pattern 8: Health Checks

When to Use: Production deployments, ensure readiness before traffic

Application Code (Node.js example):

app.get('/health', (req, res) => {
  res.status(200).json({ status: 'healthy', timestamp: new Date() });
});

app.get('/ready', async (req, res) => {
  try {
    await database.ping();
    await cache.ping();
    res.status(200).json({ status: 'ready' });
  } catch (error) {
    res.status(503).json({ status: 'not ready', error: error.message });
  }
});

Code Engine Configuration:

ibmcloud ce application update --name my-app \
  --probe-live type="http" \
  --probe-live path="/health" \
  --probe-live initial-delay=2 \
  --probe-live interval=10 \
  --probe-ready type="http" \
  --probe-ready path="/health" \
  --probe-ready initial-delay=2 \
  --probe-ready interval=10

Complete Production Example

Full deployment with all best practices:

# 1. Create project
ibmcloud ce project create --name production
ibmcloud ce project select --name production

# 2. Create secrets
ibmcloud ce secret create --name app-secrets \
  --from-literal DATABASE_URL=$DATABASE_URL \
  --from-literal API_KEY=$API_KEY

# 3. Deploy application
ibmcloud ce application create --name api-server \
  --build-source https://github.com/myorg/api-server \
  --build-commit main \
  --strategy dockerfile \
  --port 8080 \
  --min-scale 1 \
  --max-scale 20 \
  --cpu 0.5 \
  --memory 1G \
  --concurrency 100 \
  --env-from-secret app-secrets \
  --env NODE_ENV=production \
  --probe-live type="http" \
  --probe-live path="/health" \
  --probe-live initial-delay=2 \
  --probe-live interval=10 \
  --probe-ready type="http" \
  --probe-ready path="/health" \
  --probe-ready initial-delay=2 \
  --probe-ready interval=10

# 4. Verify deployment
ibmcloud ce application get --name api-server
curl $(ibmcloud ce application get --name api-server --output url)/health

Best Practices

Set min-scale > 0 for latency-sensitive apps
Use appropriate concurrency settings (lower = better isolation)
Always use secrets for sensitive data
Use configmaps for non-sensitive configuration
Set appropriate resource limits based on actual usage
Configure health checks for production
Use scale-to-zero for intermittent workloads
Right-size resources through load testing
Deploy backend services before frontend
Use scale-down-delay to prevent thrashing

Anti-Patterns to Avoid

Hardcoding credentials in source code
Setting min-scale too high for low-traffic apps
Using jobs for HTTP services
Not validating resource configurations
Ignoring cold start implications
Deploying frontend before backend
Not implementing health checks
Using localhost instead of 0.0.0.0

DIAGNOSTIC PATTERNS

Pattern 1: Application Won't Deploy

Symptoms: Deployment succeeds but app never ready, stuck in "ContainerCreating", 503 errors

Investigation Steps:

Check app and revision status: ibmcloud ce app get --name <app-name>
List revisions: ibmcloud ce revision list --application <app-name>
Check logs for image pull failures: ibmcloud ce app logs --application <app-name> --tail 100
Verify registry access: ibmcloud ce registry list

Common Causes:

Missing registry credentials → Create registry secret
Wrong image reference → Verify image name and tag
Port mismatch → Update port configuration
Startup probe failure → Check application logs

Pattern 2: Application Crashes After Deployment

Symptoms: App becomes ready then fails, intermittent 502/503 errors, high restart count

Investigation Steps:

Check revision status: ibmcloud ce app get --name <app-name>
Examine logs: ibmcloud ce app logs --application <app-name> --tail 200
Check resource limits: ibmcloud ce app get --name <app-name> | grep -E "CPU|Memory"
Check events: ibmcloud ce app events --name <app-name>

Common Causes:

Memory limit too low → Increase memory allocation
Missing environment variables → Add required env vars or secrets
Dependency connection failure → Check service bindings
Concurrency too high → Reduce concurrency setting

Pattern 3: Job Run Fails

Symptoms: Job exits with non-zero status, times out, inconsistent failures

Investigation Steps:

Identify failing run: ibmcloud ce jobrun list
Get run details: ibmcloud ce jobrun get --name <jobrun-name>
Check logs: ibmcloud ce jobrun logs --jobrun <jobrun-name>
Compare with job definition: ibmcloud ce job get --name <job-name>

Common Causes:

Command/arguments incorrect → Update job command
Timeout too short → Increase maxexecutiontime
Insufficient resources → Increase CPU and memory
Missing dependencies → Check container image

Pattern 4: Build Fails

Symptoms: Build never completes, fails with error, app deployment fails due to missing image

Investigation Steps:

Check build status: ibmcloud ce build list
Get build details: ibmcloud ce build get --name <build-name>
Check for error messages in build output

Common Causes:

Source repository access denied → Create/update repo access
Registry push failure → Verify registry credentials
Dockerfile errors → Check Dockerfile syntax
Build timeout → Increase build timeout

Pattern 5: Scaling Issues

Symptoms: Slow response under load, unexpected cold starts, doesn't scale up

Investigation Steps:

Check scaling config: ibmcloud ce app get --name <app-name> | grep -E "Min Scale|Max Scale|Concurrency"
Check instance count: ibmcloud ce revision list --application <app-name>
Monitor scaling events: ibmcloud ce app events --name <app-name> | grep -i "scale"

Common Causes:

Min scale = 0 causing cold starts → Set min-scale to 1
Max scale too low → Increase max-scale
Concurrency too high → Reduce concurrency
Slow startup time → Optimize application startup

Pattern 6: Revision Rollout Problems

Symptoms: New revision created but traffic stays on old, intermittent errors after deployment

Investigation Steps:

List revisions: ibmcloud ce revision list --application <app-name>
Check traffic routing: ibmcloud ce app get --name <app-name> | grep -A 5 "Traffic"
Compare configurations: ibmcloud ce revision get --name <revision-name>

Common Causes:

Latest revision not ready → Fix readiness failure
Configuration change broke app → Route traffic back to previous revision
Need gradual rollout → Split traffic between revisions

Pattern 7: Environment Variable Issues

Symptoms: App fails at startup with configuration errors, "variable not found" in logs

Investigation Steps:

List env vars: ibmcloud ce app get --name <app-name> | grep -A 20 "Environment Variables"
Check for missing values
Verify secret references: ibmcloud ce secret list

Common Causes:

Missing environment variable → Add env var
Secret not mounted → Create and reference secret
ConfigMap not mounted → Create and reference configmap

QUICK REFERENCE: CLI COMMANDS

Project Management

ibmcloud ce project list                          # List all projects
ibmcloud ce project select --name <name>          # Select a project
ibmcloud ce project current                       # Show current project
ibmcloud ce project create --name <name>          # Create new project

Application Commands

ibmcloud ce app list                              # List applications
ibmcloud ce app get --name <name>                 # Get app details
ibmcloud ce app create --name <name> --image <img> # Create app
ibmcloud ce app update --name <name> [options]    # Update app
ibmcloud ce app delete --name <name>              # Delete app
ibmcloud ce app logs --application <name> --tail 100 # Get logs
ibmcloud ce app logs --application <name> --follow   # Follow logs

Application Configuration

# Environment variables
ibmcloud ce app update --name <name> --env KEY=VALUE
ibmcloud ce app update --name <name> --env-from-secret <secret>
ibmcloud ce app update --name <name> --env-from-configmap <cm>

# Scaling
ibmcloud ce app update --name <name> --min-scale 1 --max-scale 10 --concurrency 20

# Resources
ibmcloud ce app update --name <name> --cpu 1 --memory 2G

# Port
ibmcloud ce app update --name <name> --port 8080

# Service binding
ibmcloud ce app bind --name <name> --service-instance <service>

Revision Commands

ibmcloud ce revision list --application <name>    # List revisions
ibmcloud ce revision get --name <name>            # Get revision details
ibmcloud ce revision delete --name <name>         # Delete revision

Job Commands

ibmcloud ce job list                              # List jobs
ibmcloud ce job get --name <name>                 # Get job details
ibmcloud ce job create --name <name> --image <img> # Create job
ibmcloud ce job update --name <name> [options]    # Update job
ibmcloud ce jobrun submit --job <name>            # Submit job run
ibmcloud ce jobrun list                           # List job runs
ibmcloud ce jobrun get --name <name>              # Get run details
ibmcloud ce jobrun logs --jobrun <name>           # Get run logs

Secret and ConfigMap Commands

# Secrets
ibmcloud ce secret list
ibmcloud ce secret create --name <name> --from-literal KEY=VALUE
ibmcloud ce secret create --name <name> --from-file <path>
ibmcloud ce secret update --name <name> --from-literal KEY=VALUE
ibmcloud ce secret delete --name <name>

# ConfigMaps
ibmcloud ce configmap list
ibmcloud ce configmap create --name <name> --from-literal KEY=VALUE
ibmcloud ce configmap update --name <name> --from-literal KEY=VALUE
ibmcloud ce configmap delete --name <name>

Registry Commands

ibmcloud ce registry list
ibmcloud ce registry create --name <name> --server <server> --username <user> --password <pass>
ibmcloud ce registry delete --name <name>

Build Commands

ibmcloud ce build list
ibmcloud ce build get --name <name>
ibmcloud ce build create --name <name> --source <git-url> --image <img> --registry-secret <secret>
ibmcloud ce buildrun submit --build <name>

Common Diagnostic Patterns

# Check if app is healthy
ibmcloud ce app get --name <name> | grep -E "Status|Ready"
ibmcloud ce revision list --application <name>

# Find why app won't deploy
ibmcloud ce app get --name <name>
ibmcloud ce app logs --application <name> --tail 100

# Debug job failure
ibmcloud ce jobrun list --job <name>
ibmcloud ce jobrun get --name <run-name>
ibmcloud ce jobrun logs --jobrun <run-name>

# Verify environment configuration
ibmcloud ce app get --name <name> | grep -A 20 "Environment Variables"
ibmcloud ce secret list

# Check scaling configuration
ibmcloud ce app get --name <name> | grep -E "Min Scale|Max Scale|Concurrency"

Useful Filters with jq

# Get only app names
ibmcloud ce app list --output json | jq -r '.[].name'

# Get apps with status
ibmcloud ce app list --output json | jq -r '.[] | "\(.name): \(.status)"'

# Get failed job runs
ibmcloud ce jobrun list --output json | jq -r '.[] | select(.status=="Failed") | .name'

# Get revision ready status
ibmcloud ce revision list --application <name> --output json | jq -r '.[] | "\(.name): \(.status)"'

STANDARD TOOLS

Use available tools effectively:

execute_command: For all ibmcloud ce commands. Always explain before executing.
read_file: During source analysis to extract configuration from key files.
list_files: First in source analysis to understand project structure.
ask_followup_question: When multiple options available. Provide 2-4 specific suggestions with context.

EXECUTION PRINCIPLES

Never skip authentication check
Always analyze source code to understand the application
Make informed recommendations based on detected technologies
Be interactive - present options and await confirmation
Provide context - explain why each step matters
Show progress - give real-time feedback
Verify results - confirm success before next phase
Offer guidance - suggest next steps

About this skill

Phase 0: Authentication Check (ALWAYS FIRST)

Phase 1: Source Code Analysis (ALWAYS SECOND)

Phase 2: Deployment Planning (ALWAYS THIRD)

Phase 3: Interactive Planning

Phase 4: Prerequisites Verification

Phase 5: Deployment Execution

Resource Validation Rules

Phase 6: Post-Deployment Verification

Phase 7: Next steps and securing application endpoints (ALWAYS LAST)

Issue 1: Architecture Mismatch (exec format error)

Issue 2: Port Configuration Mismatch

Issue 3: Resource Limit Issues (OOM Kills)

Issue 4: Cold Start Issues

Issue 5: Missing Environment Variables

Issue 6: Multi-Service Deployment Order

Issue 7: Health Check Failures

Issue 8: Scaling Issues

Quick Troubleshooting Workflow

Core Principles

Formatting Guidelines

Interaction Patterns

Error Communication

Success Communication

Workload-aware Deployment Decision Tree

Pattern 1: Deploy from Source Code

Pattern 2: Deploy from Container Image

Pattern 3: Auto-scaling Configuration

Pattern 4: Environment Variables and Secrets

Pattern 5: Jobs and Batch Processing

Pattern 6: Event-Driven Applications

Pattern 7: Multi-Service Deployment

Pattern 8: Health Checks

Complete Production Example

Best Practices

Anti-Patterns to Avoid

Pattern 1: Application Won't Deploy

Pattern 2: Application Crashes After Deployment

Pattern 3: Job Run Fails

Pattern 4: Build Fails

Pattern 5: Scaling Issues

Pattern 6: Revision Rollout Problems

Pattern 7: Environment Variable Issues

Project Management

Application Commands

Application Configuration

Revision Commands

Job Commands

Secret and ConfigMap Commands

Registry Commands

Build Commands

Common Diagnostic Patterns

Useful Filters with jq