Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 106 additions & 22 deletions .github/workflows/test-python-examples.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Runs examples/ against the `arcadedb-jvm` wheel. The wheel is py3-none-<platform tag> (setup.py): the `build` job builds it once for each
# platform, and the `test-examples` job installs that one file into each Python and runs the examples. Nothing is built per Python.
name: Test Python Examples

on:
Expand Down Expand Up @@ -115,13 +117,16 @@ jobs:
outputs:
arcadedb-tag: ${{ steps.jars.outputs.arcadedb-tag }}
matrix: ${{ steps.matrix.outputs.matrix }}
wheel-matrix: ${{ steps.matrix.outputs.wheel-matrix }}
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The platform x Python legs of this run (scripts/select_matrix.py): a small matrix on
# pull requests, the full one on every other event. The organisation shares one runner
# cap with ArcadeData/arcadedb, so a pull request does not run every combination.
# The wheel matrix is the platforms of those legs, one build each: the wheel is
# py3-none-<platform>, the same file for every Python.
# Done first, so a failure here costs no Docker pull.
- name: Select the test matrix
id: matrix
Expand All @@ -130,8 +135,11 @@ jobs:
EVENT_NAME: ${{ github.event_name }}
run: |
matrix="$(python3 scripts/select_matrix.py examples --event "$EVENT_NAME")"
wheel_matrix="$(python3 scripts/select_matrix.py examples --event "$EVENT_NAME" --wheels)"
echo "$matrix"
echo "$wheel_matrix"
echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
echo "wheel-matrix=$wheel_matrix" >> "$GITHUB_OUTPUT"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
Expand Down Expand Up @@ -176,28 +184,22 @@ jobs:
path: src/arcadedb/jars/*.jar
retention-days: 1

# Second job: Test examples on each platform
test-examples:
name: Test Python Examples (${{ matrix.platform }}, Python ${{ matrix.python-version }})
# Second job: build the wheel of each platform, once. The wheel is py3-none-<platform tag> (setup.py), so the file built here is
# the one every Python of the test-examples job below installs: nothing is built per Python. build.sh takes a Python only for the
# Linux container that builds and smoke-tests the wheel; any of them gives the same file. It also checks the size, the licences
# and the tag of the wheel, as in test-jvm.yml.
build:
name: Build wheel for examples (${{ matrix.platform }})
runs-on: ${{ matrix.runs-on }}
needs: download-jars
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
env:
BUILD_VERSION: ${{ inputs.build-version }}
# Pin the Hugging Face cache inside the workspace so it is cacheable with
# one path on every platform (the default differs per OS). The examples
# embed with sentence-transformers, and the full matrix is 16 jobs, so
# uncached that is 16 independent model downloads per release and 16
# chances for a transient hub outage to fail a release. It has happened.
HF_HOME: ${{ github.workspace }}/.hf-cache
strategy:
fail-fast: false
# Chosen by the "Select the test matrix" step of download-jars from the event that
# started the run (scripts/select_matrix.py holds the platform and Python lists):
# small on pull requests, the full 4 platforms x 4 Pythons on every other event.
# Every leg carries platform, runs-on and python-version, so the job names stay
# "Test Python Examples (<platform>, Python <version>)".
matrix: ${{ fromJSON(needs.download-jars.outputs.matrix) }}
# One leg per platform of the test matrix, from the "Select the test matrix" step of download-jars
# (scripts/select_matrix.py --wheels): {"include": [{"platform": ..., "runs-on": ...}, ...]}.
matrix: ${{ fromJSON(needs.download-jars.outputs.wheel-matrix) }}

steps:
- name: Checkout code
Expand All @@ -223,7 +225,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: ${{ matrix.python-version }}
python-version: "3.12"

- name: Setup UV package manager
shell: bash
Expand All @@ -242,6 +244,13 @@ jobs:
run: |
uv pip install --system build wheel setuptools

- name: Set UTF-8 encoding (Windows only)
if: matrix.platform == 'windows/amd64' || matrix.platform == 'windows/arm64'
shell: bash
run: |
echo "PYTHONIOENCODING=utf-8" >> $GITHUB_ENV
echo "PYTHONUTF8=1" >> $GITHUB_ENV

# Windows currently enabled, no symlink needed
# - name: Create python3 symlink (Windows only)
# if: matrix.platform == 'windows/amd64' || matrix.platform == 'windows/arm64'
Expand All @@ -258,17 +267,92 @@ jobs:
# the Docker build takes its base image by this tag, the jars from the artifact
ARCADEDB_TAG: ${{ needs.download-jars.outputs.arcadedb-tag }}
run: |
echo "🔨 Building arcadedb for ${{ matrix.platform }} with Python ${{ matrix.python-version }}..."
echo "🔨 Building arcadedb-jvm for ${{ matrix.platform }}..."
# The jars download-jars fetched, on every platform, so all wheels of this
# run carry the same engine build (and build.sh checks the integration jar)
./scripts/build.sh ${{ matrix.platform }} ${{ matrix.python-version }} src/arcadedb/jars
./scripts/build.sh ${{ matrix.platform }} 3.12 src/arcadedb/jars

# build.sh already ran this (and the size check); it is repeated here as in test-jvm.yml, so the tag shows in this job's log and
# summary and a change to build.sh cannot drop it unnoticed. The wheel must be py3-none-<platform>, the file every Python installs.
- name: Check the wheel tag
shell: bash
run: |
python scripts/verify_wheel_tag.py dist/*.whl
{
echo "## 📦 Wheel: arcadedb-jvm (${{ matrix.platform }})"
echo
for w in dist/*.whl; do
echo "- \`$(basename "$w")\`, $(du -m "$w" | cut -f1) MB"
done
} >> "$GITHUB_STEP_SUMMARY"

# One wheel per platform, for the test-examples legs of that platform. Named examples-wheel-<platform>, not wheel-<platform>:
# a release runs this workflow and test-jvm.yml in one run, and release-jvm.yml publishes every wheel-* artifact.
- name: Upload wheel artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: examples-wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}
path: dist/*.whl
if-no-files-found: error
retention-days: 1

# Third job: install the wheel of the platform into each Python and run the examples. Nothing is built here: the wheel is
# the file the build job made (py3-none-<platform tag>), so each Python runs the examples against the same file that is published.
test-examples:
name: Test Python Examples (${{ matrix.platform }}, Python ${{ matrix.python-version }})
runs-on: ${{ matrix.runs-on }}
needs: [download-jars, build]
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
env:
# Pin the Hugging Face cache inside the workspace so it is cacheable with
# one path on every platform (the default differs per OS). The examples
# embed with sentence-transformers, and the full matrix is 16 jobs, so
# uncached that is 16 independent model downloads per release and 16
# chances for a transient hub outage to fail a release. It has happened.
HF_HOME: ${{ github.workspace }}/.hf-cache
strategy:
fail-fast: false
# Chosen by the "Select the test matrix" step of download-jars from the event that
# started the run (scripts/select_matrix.py holds the platform and Python lists):
# small on pull requests, the full 4 platforms x 4 Pythons on every other event.
# Every leg carries platform, runs-on and python-version, so the job names stay
# "Test Python Examples (<platform>, Python <version>)".
matrix: ${{ fromJSON(needs.download-jars.outputs.matrix) }}

steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: ${{ matrix.python-version }}

- name: Setup UV package manager
shell: bash
run: |
# Retry transient network failures (a connection reset failed a whole examples job on #8897); the
# script is downloaded first so a retried transfer is never piped half-written into sh.
installer="$(mktemp)"
curl -LsSf --retry 5 --retry-all-errors --retry-delay 5 -o "$installer" https://astral.sh/uv/install.sh
sh "$installer"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
export PATH="$HOME/.local/bin:$PATH"
uv --version

# The wheel of this platform, built once by the build job and the same file for every Python.
- name: Download the wheel of ${{ matrix.platform }}
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: examples-wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}
path: test-install

# Note: Java is NOT required - arcadedb has bundled JRE!

- name: Install ArcadeDB Python bindings
shell: bash
run: |
uv pip install --system dist/arcadedb_jvm-*.whl
uv pip install --system test-install/arcadedb_jvm-*.whl

- name: Install example dependencies
shell: bash
Expand Down Expand Up @@ -763,7 +847,7 @@ jobs:
# Summary job that checks all platforms
test-examples-summary:
name: Examples Test Summary
needs: test-examples
needs: [build, test-examples]
runs-on: ubuntu-latest
# Skipped (not red) when the matrix was skipped for a draft PR.
if: always() && (github.event_name != 'pull_request' || github.event.pull_request.draft == false)
Expand All @@ -774,10 +858,10 @@ jobs:
echo "## 🎯 Overall Examples Test Summary" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY

if [ "${{ needs.test-examples.result }}" = "success" ]; then
if [ "${{ needs.build.result }}" = "success" ] && [ "${{ needs.test-examples.result }}" = "success" ]; then
echo "✅ **All platforms passed example testing!**" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "All examples ran successfully across all enabled platforms." >> $GITHUB_STEP_SUMMARY
echo "All examples ran successfully across all enabled platforms, against one wheel built for each platform." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Platforms tested**: linux/amd64, linux/arm64, darwin/arm64, windows/amd64" >> $GITHUB_STEP_SUMMARY
else
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ python -m bandit -c pyproject.toml -r examples --severity-level medium --confide

Each package has one test and one release workflow: `test-jvm.yml` and `release-jvm.yml` for `arcadedb-jvm`, `test-native.yml` and `release-native.yml` for `arcadedb` (the release workflows call the test workflows with `workflow_call`; the PyPI trusted publishers are keyed on the release file names and the environments `pypi-jvm` and `pypi`, so never rename them without changing PyPI first). `test-python-examples.yml` runs `examples/`.

`.github/workflows/test-jvm.yml` builds ONE wheel per platform (linux/amd64, linux/arm64, darwin/arm64, windows/amd64), tagged `py3-none-<platform>` by `setup.py` (`bdist_wheel.get_tag`; `scripts/verify_wheel_tag.py` checks it), and tests that wheel on Python 3.11–3.14; also runs nightly against `arcadedata/arcadedb:latest`. `requires-python` is `>=3.11,<3.15` because the tag no longer keeps pip off 3.15 and JPype has no 3.15 wheel; drop the cap with it. The org shares a runner cap with upstream, so the unit tests run 10 legs on every event (linux/amd64 on all four Pythons, the other three platforms on 3.11 and 3.14, the shape of test-native.yml), and the examples run all four platforms on 3.14 on a pull request (4 legs, 24 checks in all) and 16 legs on push to main, the nightly run, dispatch and the release (36 checks). `scripts/select_matrix.py` holds both lists (`PULL_REQUEST_LEGS`, keyed `jvm` and `examples`), the first job of each workflow runs it, and `tests/test_select_matrix.py` pins the full matrix; docs page `development/ci.adoc`. Keep the job names `Test arcadedb (<platform>, Python <version>)` unchanged. Both workflows accept an `arcadedb-tag` input to pin the image. `test-python-examples.yml` runs `examples/`. A dependency-floor audit resolves extras with `--resolution lowest-direct` for each Python classifier and runs pip-audit, so raise lower bounds when a floor has an advisory.
`.github/workflows/test-jvm.yml` builds ONE wheel per platform (linux/amd64, linux/arm64, darwin/arm64, windows/amd64), tagged `py3-none-<platform>` by `setup.py` (`bdist_wheel.get_tag`; `scripts/verify_wheel_tag.py` checks it), and tests that wheel on Python 3.11–3.14; also runs nightly against `arcadedata/arcadedb:latest`. `requires-python` is `>=3.11,<3.15` because the tag no longer keeps pip off 3.15 and JPype has no 3.15 wheel; drop the cap with it. The org shares a runner cap with upstream, so the unit tests run 10 legs on every event (linux/amd64 on all four Pythons, the other three platforms on 3.11 and 3.14, the shape of test-native.yml), and the examples run all four platforms on 3.14 on a pull request (4 legs, 28 checks in all) and 16 legs on push to main, the nightly run, dispatch and the release (40 checks). `scripts/select_matrix.py` holds both lists (`PULL_REQUEST_LEGS`, keyed `jvm` and `examples`), the first job of each workflow runs it, and `tests/test_select_matrix.py` pins the full matrix; docs page `development/ci.adoc`. Keep the job names `Test arcadedb (<platform>, Python <version>)` unchanged. Both workflows accept an `arcadedb-tag` input to pin the image. `test-python-examples.yml` runs `examples/`. A dependency-floor audit resolves extras with `--resolution lowest-direct` for each Python classifier and runs pip-audit, so raise lower bounds when a floor has an advisory.

`.github/workflows/release-jvm.yml` publishes `arcadedb-jvm` on a bare-version tag (`26.11.1.dev0`, `26.11.1`, `26.11.1.post1`; no `v`, never `aN`/`bN`/`rcN`). Its first job runs `scripts/check_version.py` (the devN/postN counting rule, read-only, fails closed); `scripts/after_tag_verify.sh` checks the finished release. The rule is stated once, on the docs page `development/release.adoc`.

Expand Down
6 changes: 3 additions & 3 deletions docs/modules/ROOT/pages/development/ci.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Each package has one test workflow and one release workflow:
| `release-native.yml`
|===

The release workflows call the test workflows (`workflow_call`) and then publish; see xref:development/release.adoc[Release]. Two workflows test the `arcadedb-jvm` package: `test-jvm.yml` builds a wheel and runs the unit tests, and `test-python-examples.yml` runs the scripts in `examples/`. Both build the wheels on a native runner of each platform. `test-native.yml` builds and tests the `arcadedb` (native) package; see <<_native_build,Native build>>.
The release workflows call the test workflows (`workflow_call`) and then publish; see xref:development/release.adoc[Release]. Two workflows test the `arcadedb-jvm` package: `test-jvm.yml` builds a wheel and runs the unit tests, and `test-python-examples.yml` runs the scripts in `examples/`. Both build the wheel once for each platform, on a native runner of that platform (`Build arcadedb-jvm wheel (<platform>)` and `Build wheel for examples (<platform>)`), and install that one file into each Python they test. `test-native.yml` builds and tests the `arcadedb` (native) package; see <<_native_build,Native build>>.

The package supports four platforms and four Python versions (3.11 to 3.14). That is 16 combinations. The organisation shares one runner limit with ArcadeData/arcadedb, so each macOS or Windows job delays the upstream jobs too, and not every combination runs.

Expand All @@ -31,7 +31,7 @@ The `requires-python` of the package is `>=3.11,<3.15`. The wheel tag no longer

== What runs when

The script `scripts/select_matrix.py` chooses the combinations. The first job of each workflow, `Download ArcadeDB JARs`, runs it and passes the result to the next jobs: the test legs, and for `test-jvm.yml` also the platforms to build (`--wheels`).
The script `scripts/select_matrix.py` chooses the combinations. The first job of each workflow, `Download ArcadeDB JARs`, runs it and passes the result to the next jobs: the test legs and the platforms to build (`--wheels`).

[cols="2,3,3"]
|===
Expand All @@ -46,7 +46,7 @@ The script `scripts/select_matrix.py` chooses the combinations. The first job of
| All 16 combinations.
|===

The unit tests run the same 10 legs on every event, like the native package (all Python versions on one Linux platform, the oldest and the newest elsewhere). They were 16 legs on `main`, the nightly run and a release before the wheel became `py3-none`; the extra legs tested the same file again. A pull request runs 24 checks: 14 test jobs (10 and 4), 4 wheel builds, the two `Download ArcadeDB JARs` jobs, the two summary jobs, `Bandit security scan` and `Dependency floor audit`. The other events run 36: the examples run all 16 combinations there.
The unit tests run the same 10 legs on every event, like the native package (all Python versions on one Linux platform, the oldest and the newest elsewhere). They were 16 legs on `main`, the nightly run and a release before the wheel became `py3-none`; the extra legs tested the same file again. A pull request runs 28 checks: 14 test jobs (10 and 4), 8 wheel builds (4 for each workflow), the two `Download ArcadeDB JARs` jobs, the two summary jobs, `Bandit security scan` and `Dependency floor audit`. The other events run 40: the examples run all 16 combinations there, from 4 wheel builds.

The names of the test jobs do not change: `Test arcadedb (<platform>, Python <version>)`. A check keeps its history when a pull request runs fewer of them.

Expand Down
1 change: 1 addition & 0 deletions scripts/select_matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
select_matrix.py jvm [--event NAME] # test-jvm.yml: the test legs
select_matrix.py jvm [--event NAME] --wheels # test-jvm.yml: the build legs, one for each platform of the test legs
select_matrix.py examples [--event NAME] # test-python-examples.yml
select_matrix.py examples [--event NAME] --wheels # test-python-examples.yml: the build legs, one for each platform

`--event` defaults to $GITHUB_EVENT_NAME. The output is one line of JSON for
`strategy.matrix`: {"include": [{"platform": ..., "runs-on": ..., "python-version": ...}, ...]}, and with
Expand Down
Loading
Loading