SuperAPI AI Directory

paddle-cross-ecosystem-custom-op

General & other

View repository

Updated 10-16 03:14

About this skill

Paddle cross-ecosystem custom operator migration

Task definition

The main task of this skill: get upstream PyTorch custom operator repositories running on Paddle along their original call paths, while preserving the ability to rebase / sync upstream later. After migration is complete, if the library needs to be integrated into PaddleFleet, follow PaddleFleet integration。

A complete output should cover four aspects:

  • Migration plan
  • Minimal modification boundary
  • Validation path
  • compat gap handling strategy

Core constraints

  • Minimal modification: do not perform extra formatting, optimization, or refactoring, and do not proactively change public APIs.
  • Upstream synchronization: all changes must take into account the convenience of subsequent rebase / sync upstream.
  • compat first: prioritize using Paddle's existing compat mechanisms, and let the compat layer take on compatibility responsibilities.
  • Gaps must be explicit: compat gaps must be clearly categorized, boundaries marked, and a minimal reproduction prepared.
  • Validation must be closed-loop: at least get one minimal build/test path running through.

Work order

  1. Identify the upstream repository, the current fork, the default branch, and the actual migration branch. The default branch of PFCCLab adaptation repositories is usually paddle, and confirm the parent and default branch before comparing.
  2. Divide the repository into four layers by control plane:
    • Framework-agnostic kernels / algorithms
    • Build and packaging
    • C++ compat API / registration
    • Python wrappers / runtime glue / tests
  3. If the task is to analyze multiple PFCCLab forks, and the user explicitly requests parallelism, split the parallel analysis by repository; each subtask must output the parent, comparison branches, four-layer diff categorization, and reusable patterns.
  4. First determine the locations for the first round of changes. The first patch round is usually concentrated in build, runtime glue, and device / stream / distributed boundaries.
  5. Advance validation step by step along the minimal path: build → import → minimal functional test → runtime comparison.

Default modification boundary

Parts that usually do not need to be changed:

  • CUDA/C++ core kernels and algorithm logic
  • Original schema definitions
  • Most TORCH_LIBRARY / pybind11 registration code
  • Upstream directory structure and Python package shape

Parts that usually need to be checked first:

  • setup.py / pyproject.toml
  • Entry scripts, test scripts, example scripts
  • torch.ops / torch.library / torch._dynamo / torch.profiler usage sites
  • device / stream / distributed / DLPack / custom op registration glue

Specific rules

  • setup.py / pyproject.toml: prefer adding paddle.enable_compat(), keep the original from torch.utils import cpp_extension style; only when the proxy path cannot cover it, minimally switch to paddle.utils.cpp_extension or locally adjust include / lib / flags.
  • TORCH_LIBRARY / TORCH_LIBRARY_IMPL / pybind11: by default keep them as-is first, and only locate the specific gap after compilation or runtime actually fails.
  • at::Tensor / c10::TensorOptions / torch::empty and other C++ APIs: prioritize relying on compat headers; when encountering gaps, bridge only a single API point.
  • Python entry points and tests: prioritize using paddle.enable_compat(scope={...}) Limit the proxy scope; short-lived build scripts can use the global paddle.enable_compat(). For PaddleFleet integration scenarios, follow the PaddleFleet integration existing template.
  • Distributed / stream / device: first connect the runtime context boundaries, then see whether it is necessary to go deeper into phi::GPUContext、ProcessGroup, DLPack, or stream wrapper.
  • Analyze the PFCCLab fork: the output must be distilled into reusable patterns, covering the four layers of build / C++ / Python / tests.

Read reference materials on demand

Select reference materials according to the current task:

Current task Read files
First understand the cross-ecosystem mechanisms and layering conventions Mechanism overview
Actually migrate a new repository Migration manual
Locate errors inside the Paddle repository Paddle internal anchors
Analyze the PFCCLab fork in the checklist First read the ecosystem library case index, then read only the corresponding case pointed to by the index
Reuse existing migration experience for a new repository First read the ecosystem library case index, then select at most one or two similar cases based on the control plane
Determine compat gaps, workarounds, issue MREs compat gap handling
build/import already works but runtime behavior is inconsistent Runtime debugging
Integrate a migrated ecosystem library into PaddleFleet PaddleFleet integration

Output requirements

  • Clearly list which files do not need to be changed, which files need to be changed, and which layer each change corresponds to.
  • If a workaround is needed, it must clearly state the coverage scope, removal conditions, and whether a Paddle issue needs to be filed.
  • If the problem reaches the runtime comparison stage, point out on which line the first difference appears, at which call site, and which layer it belongs to.
  • If analyzing an existing fork, summarize a reusable migration order and distill the diff into stable patterns.

Pre-completion checks

  • There is no irrelevant formatting, cleanup, or renaming.
  • The upstream directory structure and main API shapes are preserved.
  • The runtime enable_compat has been scoped as much as possible to scope; the global compat of the build script is only used at the build entry point.
  • At least one minimal path of build/test has been run successfully.
  • For compat gaps, an issue MRE has been prepared, or the gap and temporary workaround have been clearly stated in the results.