doris-docker-regression

Purpose

Run Doris docker-based regression tests from a clean package, avoiding contamination from local conf/ or startup script modifications.

Prerequisites

Python Environment

Requires Python 3. Install dependencies from docker/runtime/doris-compose/requirements.txt:

python -m pip install --user -r docker/runtime/doris-compose/requirements.txt

If installation fails (especially PyYAML conflicts), use specific versions:

python -m pip install --user pyyaml==5.3.1 docker==6.1.3
python -m pip install --user -r docker/runtime/doris-compose/requirements.txt

Alternatively, use a virtual environment:

python -m venv doris-compose-env
source doris-compose-env/bin/activate
pip install -r docker/runtime/doris-compose/requirements.txt

Docker Environment

docker run hello-world       # Docker works
docker compose version       # Compose v2
docker-compose version       # Should resolve to Compose v2

If docker-compose shows TypeError: kwargs_from_env(), make sure it is indeed Docker Compose V2; if not, you need to install and point to the correct version.

Check port availability:

lsof -nP -iTCP:8030,8040,8050,8060,8070,9010,9020,9030,9050,9060 -sTCP:LISTEN

If there is a conflict, ask the user how to handle it.

Build

./build.sh --fe --be -j<parallel_jobs> --output <output_directory>

Prepare Clean Package

Sanitize configs from git HEAD:

git show HEAD:conf/fe.conf > <output_directory>/fe/conf/fe.conf
git show HEAD:conf/be.conf > <output_directory>/be/conf/be.conf
git show HEAD:bin/start_fe.sh > <output_directory>/fe/bin/start_fe.sh
chmod 755 <output_directory>/fe/bin/start_fe.sh

Verify clean configs use default ports:

fe.conf: 8030, 9020, 9030, 9010, 8070
be.conf: 9060, 8040, 9050, 8060, 8050
start_fe.sh: no -agentlib:jdwp debug agent

Build Docker Image

docker build \
  --build-arg OUTPUT_PATH=<output_directory> \
  -f docker/runtime/doris-compose/Dockerfile \
  -t <image_name>:latest \
  .

Configure Regression

Create regression-test/conf/regression-conf-custom.groovy:

image = "<image_name>:latest"
excludeDockerTest = false
testGroups = "docker"

Run Regression

./run-regression-test.sh --run -d <directory> -s <suite_name>

Troubleshooting

Symptom	Cause	Fix
0 suites	`excludeDockerTest=true` or missing `testGroups="docker"`	Check `regression-conf-custom.groovy`
JDWP error	`start_fe.sh` has debug agent	Re-sanitize from `git show HEAD:bin/start_fe.sh`
Wrong ports	Configs have local edits	Re-sanitize from `git show HEAD:conf/fe.conf` / `be.conf`
Port conflict	Processes using default ports	`lsof` then kill

Debug Logs

Runtime cluster at /tmp/doris/<suite_name>/:

fe-1/log/fe.log, fe.warn.log, fe.out, health.out
be-1/log/be.INFO, be.WARNING, be.out, health.out
doris-compose.log

Full Command Sequence

# 1. Check environment
python --version
docker run hello-world
docker compose version
lsof -nP -iTCP:8030,8040,8050,8060,8070,9010,9020,9030,9050,9060 -sTCP:LISTEN

# 2. Build
./build.sh --fe --be -j60

# 3. Sanitize package configs
git show HEAD:conf/fe.conf > <output_directory>/fe/conf/fe.conf
git show HEAD:conf/be.conf > <output_directory>/be/conf/be.conf
git show HEAD:bin/start_fe.sh > <output_directory>/fe/bin/start_fe.sh
chmod 755 <output_directory>/fe/bin/start_fe.sh

# 4. Build image
docker build --build-arg OUTPUT_PATH=<output_directory> -f docker/runtime/doris-compose/Dockerfile -t <image_name>:latest .

# 5. Configure (create regression-conf-custom.groovy)
# 6. Run
./run-regression-test.sh --run -d <directory> -s <suite_name>