airflow-java-sdk

Source excerpt starting at line 1.

1---2name: airflow-java-sdk3description: >4  Guide for contributing to the Airflow Java SDK (AIP-108). Use this skill5  whenever a contributor is working in the `java-sdk/` directory or on the Java6  coordinator in `task-sdk/src/airflow/sdk/coordinators/java/` — whether they7  want to add a feature, write tests, fix a bug, understand the architecture, or8  prepare a PR. Trigger on phrases like "Java SDK", "JavaCoordinator",9  "java-sdk", "annotation processor", "Builder.Task", "BundleBuilder", or10  anything about running JVM tasks in Airflow.11---12 13<!-- SPDX-License-Identifier: Apache-2.014     https://www.apache.org/licenses/LICENSE-2.0 -->15 16# Airflow Java SDK contributor guide17 18The Java SDK lets Airflow tasks execute JVM code (Java, Kotlin, or any JVM language). You are helping19a contributor work in one or both of these locations:20 21- **`java-sdk/`** — the JVM-side library (Kotlin source, published to Maven)22- **`task-sdk/src/airflow/sdk/coordinators/java/`** — the Python coordinator that launches the JVM subprocess23 24Read these two documents early in every session — they contain the authoritative reference material:25 26- `airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst` — user-facing guide:27  annotation vs. interface API, XCom type mapping, Gradle/Maven steps, coordinator config.28- `java-sdk/README.md` — contributor guide: repository layout, detailed execution walkthrough,29  Gradle + Breeze test commands, coding conventions, common tasks, and PR checklist.30 31---32 33## SDK package architecture34 35The JVM-side library is split into two packages with distinct visibility rules:36 37- **`org.apache.airflow.sdk`** — public, user-facing API. Classes here (e.g. `Client`, `Bundle`,38  `BundleBuilder`, `Server`) are stable contracts that DAG authors and task implementers import39  directly. Changes to this package are breaking changes.40- **`org.apache.airflow.sdk.execution`** — internal implementation detail. Everything in this41  package (`CoordinatorComm`, `LogSender`, `Log`, `Client` in `execution/`, generated schema42  models, etc.) is not intended to be imported by users. It may change between releases without43  notice.44 45When reviewing or writing code, enforce this boundary: user task code and `BundleBuilder`46subclasses must only import from `org.apache.airflow.sdk`; any import of47`org.apache.airflow.sdk.execution.*` in user-facing API surface is a red flag.48 49---50 51## Bundle composition and coordinator discovery52 53A **bundle** is a directory of JAR files (typically `build/bundle/`) placed on the coordinator's54`jars_root`. The coordinator scans the directory at task-dispatch time to find:55 561. **`Main-Class`** (standard JAR manifest attribute) — the fully-qualified class name of the57   entry point that the coordinator invokes with `java -classpath … <Main-Class> --comm … --logs …`.58   This must be a class with a `public static void main(String[] args)` method; the Gradle plugin59   `org.apache.airflow.sdk` writes it automatically from `airflowBundle { mainClass = "…" }` and60   validates that the class exists and has the right signature at build time.61 622. **`Airflow-Supervisor-Schema-Version`** (Airflow-specific manifest attribute) — the wire63   protocol version the JVM side expects when talking to the Python supervisor. In fat-JAR mode64   (the default), the Gradle plugin reads this value from the `airflow-sdk` JAR in65   `runtimeClasspath` and copies it into the shadow JAR manifest. In thin-JAR mode (`fatJar =66   false`), the value stays in the `airflow-sdk` JAR deployed alongside the bundle JAR.67 68The Python coordinator (`JavaCoordinator`) scans every JAR under `jars_root` with69`_JarInfo.find()`, reads `META-INF/MANIFEST.MF` out of each ZIP, and collects `Main-Class` and70`Airflow-Supervisor-Schema-Version` from whichever JARs carry them. The resolved schema version71is then passed as the `schema_version` return value from `_build_execute_task_command`, which72the base `SubprocessCoordinator` uses to negotiate the supervisor wire protocol.73 74If `main_class` is set explicitly on the `JavaCoordinator` instance (via `[sdk] coordinators`75kwargs), the scan uses it as a filter; otherwise the first JAR with a `Main-Class` attribute76wins. Either way, `Airflow-Supervisor-Schema-Version` must be present in at least one JAR in77`jars_root` or startup fails.78 79---80 81## Key files to know82 83| File | Purpose |84|---|---|85| `java-sdk/sdk/.../Client.kt` | Public API (Variables, Connections, XCom) |86| `java-sdk/sdk/.../execution/Client.kt` | Supervisor wire calls |87| `java-sdk/sdk/.../execution/Comm.kt` | 4-byte-prefix MessagePack framing |88| `java-sdk/sdk/.../Server.kt` | Entry-point; drives the execution loop |89| `java-sdk/processor/.../BuilderProcessor.kt` | Kapt annotation processor |90| `java-sdk/plugin/.../AirflowSdkPlugin.kt` | Gradle bundle plugin |91| `task-sdk/.../coordinators/java/coordinator.py` | Python side — spawns the JVM |92| `task-sdk/.../schema/schema.json` | Wire protocol definition (both sides) |93 94---95 96## Running tests97 98Always use `./gradlew` from inside `java-sdk/`; never run Gradle via apt's `gradle`.99See `java-sdk/README.md#testing` for the full list of Gradle commands.100 101For the Python coordinator, use Breeze (never `pytest` directly on the host):102 103```bash104breeze testing task-sdk-tests -- task_sdk/coordinators/java105```106 107End-to-end test suite:108 109```bash110E2E_TEST_MODE=java_sdk uv run --project airflow-e2e-tests pytest \111    tests/airflow_e2e_tests/java_sdk_tests/ -xvs112```113 114---115 116## Updating the Python coordinator117 118`coordinator.py` extends `SubprocessCoordinator`. The only method subclasses must implement is119`_build_execute_task_command`, which returns `(argv, schema_version)`. Look at the existing120implementation for how `jars_root`, `java_executable`, `jvm_args`, and `main_class` are121assembled into the command. Do not reach into the JVM process from Python beyond what this122method provides.123 124---125 126## Upgrading Supervisor Schema client127 128When upgrading to a newer Supervisor Schema version:129 130- Regenerate models with `./gradlew generateJsonSchema2Pojo`131- Modify `execution/Client.kt` to handle changes132 133The `java-sdk/README.md#contributing` section walks through the full "adding a new Client134method" sequence step by step.135