Skip to content

JVM wheels: py3-none-<platform>, one wheel per platform (4 per release, not 16) #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tae898 merged 2 commits into from
Oct 26, 2026
Merged

JVM wheels: py3-none-<platform>, one wheel per platform (4 per release, not 16) #92

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16a845a
JVM wheels: py3-none-<platform>, one wheel per platform
tae898 Oct 26, 2026
8e91598
tests: run test_log_redirect.py on the native build too (JVM-only JEP…
tae898 Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions .github/workflows/release-jvm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,9 @@ jobs:
TAG: ${{ github.ref_name }}
run: python3 -I scripts/check_version.py "$TAG" --git-remote origin --pypi "$DIST_NAME"

# Builds a wheel per platform and Python version and runs the unit tests on each. The called
# workflow uploads the wheels as wheel-<platform>-py<version> artifacts.
# Builds one wheel for each platform (py3-none-<platform tag>, the same file for every Python) and
# runs the unit tests on it, on the Pythons scripts/select_matrix.py picks. The called workflow
# uploads the wheels as wheel-<platform> artifacts.
test:
name: Unit tests and wheels
needs: validate-version
Expand Down Expand Up @@ -123,10 +124,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Download wheels for all tested Python versions
- name: Download the wheels of the four platforms
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: wheel-*-py*
pattern: wheel-*
path: dist/
merge-multiple: true

Expand Down Expand Up @@ -166,34 +167,31 @@ jobs:
run: |
ls -lh dist/
python3 -I - <<'PY'
import os, pathlib, re, subprocess, sys, tomllib, zipfile
import os, pathlib, re, subprocess, sys, zipfile

tag = os.environ["TAG"]
wheels = sorted(pathlib.Path("dist").glob("*.whl"))

# One wheel per platform and supported interpreter. The interpreters are the classifiers of
# pyproject.toml, which the full matrix of scripts/select_matrix.py mirrors (the release runs the full
# matrix, not the small pull request one); the four platforms are the matrix platforms. Adding either to
# the matrix changes the expected count here.
prefix = "Programming Language :: Python :: 3."
classifiers = tomllib.load(open("pyproject.toml", "rb"))["project"]["classifiers"]
pythons = [c.rsplit(" ", 1)[1] for c in classifiers if c.startswith(prefix)]
# One wheel for each platform, py3-none-<platform tag> (setup.py): nothing in the wheel is compiled against
# CPython, so the Python versions it installs on are requires-python and the classifiers of pyproject.toml,
# not extra files. The four platforms are the platforms of the build matrix (scripts/select_matrix.py --wheels).
# Adding a platform to the matrix changes the expected count here.
platforms = {
"linux/amd64": r"manylinux.*x86_64",
"linux/arm64": r"manylinux.*aarch64",
"darwin/arm64": r"macosx.*arm64",
"windows/amd64": r"win_amd64",
}
expected = len(pythons) * len(platforms)
print(f"wheels: {len(wheels)} (expected {expected} = {len(pythons)} Python versions x {len(platforms)} platforms)")
expected = len(platforms)
print(f"wheels: {len(wheels)} (expected {expected} = one for each of the {len(platforms)} platforms)")
problems = []
if len(wheels) != expected:
problems.append(f"expected {expected} wheels, got {len(wheels)}")
for name, pattern in platforms.items():
n = sum(1 for w in wheels if re.search(pattern, w.name))
print(f" {name}: {n}")
if n != len(pythons):
problems.append(f"{name}: expected {len(pythons)} wheels, got {n}")
if n != 1:
problems.append(f"{name}: expected 1 wheel, got {n}")

names = set()
for w in wheels:
Expand Down Expand Up @@ -222,10 +220,13 @@ jobs:
with open(os.environ["GITHUB_OUTPUT"], "a") as out:
out.write(f"import-name={imports[0]}\n")

# Size policy: every wheel stays under 100 MB (decimal); build.sh checks the same limit.
# Tag policy: every wheel is py3-none-<platform>, in its name and in its WHEEL file (a cpXY wheel would be one of sixteen).
# Size policy: every wheel stays under 100 MB (decimal); build.sh checks the same two limits.
# ARCADEDB_WHEEL_MAX_MB is deliberately not set: the release has no override.
rc = subprocess.run([sys.executable, "scripts/verify_wheel_size.py", *map(str, wheels)]).returncode
sys.exit(rc)
for script in ("verify_wheel_tag.py", "verify_wheel_size.py"):
rc = subprocess.run([sys.executable, f"scripts/{script}", *map(str, wheels)]).returncode
if rc:
sys.exit(rc)
PY
{
echo "## Wheels"
Expand Down
143 changes: 101 additions & 42 deletions .github/workflows/test-jvm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# The JVM build: the `arcadedb-jvm` distribution. release-jvm.yml calls this workflow on a version tag and publishes the wheels it builds.
# One wheel per platform, py3-none-<platform tag> (setup.py): the `build` job builds each once, and the `test` job installs that one file
# into every Python (3.11 to 3.14) and runs the suite. The jars and the bundled JRE do not depend on the Python, so nothing is built per Python.
# The native build, `arcadedb`, is test-native.yml / release-native.yml.
name: Test JVM (arcadedb-jvm)

Expand Down Expand Up @@ -200,13 +202,16 @@ jobs:
outputs:
arcadedb-tag: ${{ steps.jars.outputs.arcadedb-tag }}
matrix: ${{ steps.matrix.outputs.matrix }}
wheel-matrix: ${{ steps.matrix.outputs.wheel-matrix }}
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The platform x Python legs of this run (scripts/select_matrix.py): a small matrix on
# pull requests, the full one on every other event. The organisation shares one runner
# cap with ArcadeData/arcadedb, so a pull request does not run every combination.
# The wheel matrix is the platforms of those legs, one build each: the wheel is
# py3-none-<platform>, the same file for every Python.
# Done first, so a failure here costs no Docker pull.
- name: Select the test matrix
id: matrix
Expand All @@ -215,8 +220,11 @@ jobs:
EVENT_NAME: ${{ github.event_name }}
run: |
matrix="$(python3 scripts/select_matrix.py jvm --event "$EVENT_NAME")"
wheel_matrix="$(python3 scripts/select_matrix.py jvm --event "$EVENT_NAME" --wheels)"
echo "$matrix"
echo "$wheel_matrix"
echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
echo "wheel-matrix=$wheel_matrix" >> "$GITHUB_OUTPUT"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
Expand Down Expand Up @@ -264,9 +272,11 @@ jobs:
path: src/arcadedb/jars/*.jar
retention-days: 1

# Second job: Build and test on each platform
test:
name: Test arcadedb (${{ matrix.platform }}, Python ${{ matrix.python-version }})
# Second job: build the wheel of each platform, once. The wheel is py3-none-<platform tag> (setup.py), so the file built here is
# the one every Python of the test job below installs: nothing is built per Python. build.sh takes a Python only for the Linux
# container that builds and smoke-tests the wheel; any of them gives the same file.
build:
name: Build arcadedb-jvm wheel (${{ matrix.platform }})
runs-on: ${{ matrix.runs-on }}
needs: download-jars
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
Expand All @@ -275,12 +285,9 @@ jobs:
strategy:
# Don't cancel other matrix jobs if one fails
fail-fast: false
# Chosen by the "Select the test matrix" step of download-jars from the event that
# started the run (scripts/select_matrix.py holds the platform and Python lists):
# small on pull requests, the full 4 platforms x 4 Pythons on every other event.
# Every leg carries platform, runs-on and python-version, so the job names stay
# "Test arcadedb (<platform>, Python <version>)".
matrix: ${{ fromJSON(needs.download-jars.outputs.matrix) }}
# One leg per platform of the test matrix, from the "Select the test matrix" step of download-jars
# (scripts/select_matrix.py --wheels): {"include": [{"platform": ..., "runs-on": ...}, ...]}.
matrix: ${{ fromJSON(needs.download-jars.outputs.wheel-matrix) }}

steps:
- name: Checkout code
Expand All @@ -289,7 +296,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: ${{ matrix.python-version }}
python-version: "3.12"

- name: Setup UV package manager
shell: bash
Expand Down Expand Up @@ -342,23 +349,93 @@ jobs:
# ln -s "$PYTHON_DIR/python.exe" "$PYTHON_DIR/python3.exe" || true
# python3 --version

- name: Build and test arcadedb (${{ matrix.platform }})
- name: Build arcadedb (${{ matrix.platform }})
shell: bash
env:
# The engine version download-jars read from the ARCADEDB_IMAGE_TAG jars;
# the Docker build takes its base image by this tag, the jars from the artifact
ARCADEDB_TAG: ${{ needs.download-jars.outputs.arcadedb-tag }}
run: |
echo "🔨 Building arcadedb for ${{ matrix.platform }} with Python ${{ matrix.python-version }}..."
echo "🔨 Building arcadedb-jvm for ${{ matrix.platform }}..."
# The jars download-jars fetched, on every platform, so all wheels of this
# run carry the same engine build (and build.sh checks the integration jar)
./scripts/build.sh ${{ matrix.platform }} ${{ matrix.python-version }} src/arcadedb/jars
./scripts/build.sh ${{ matrix.platform }} 3.12 src/arcadedb/jars

- name: Extract wheel for additional testing
# build.sh already ran this; it is repeated here so the tag shows in this job's log and summary, and so a change to build.sh
# cannot drop it unnoticed. The wheel must be py3-none-<platform>, the file the test job installs on every Python.
- name: Check the wheel tag
shell: bash
run: |
mkdir -p test-install
cp dist/*.whl test-install/
python scripts/verify_wheel_tag.py dist/*.whl
{
echo "## 📦 Wheel: arcadedb-jvm (${{ matrix.platform }})"
echo
for w in dist/*.whl; do
echo "- \`$(basename "$w")\`, $(du -m "$w" | cut -f1) MB"
done
} >> "$GITHUB_STEP_SUMMARY"

# One wheel per platform: the test job of the platform installs this file on every Python, and the release workflow
# publishes it (verify-wheels downloads wheel-<platform>).
- name: Upload wheel artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}
path: dist/*.whl
if-no-files-found: error
retention-days: 7

# Third job: install the wheel of the platform into each Python and run the suite. Nothing is built here: the wheel is the file the
# build job made (py3-none-<platform tag>), so each Python is tested against the same file that is published.
test:
name: Test arcadedb (${{ matrix.platform }}, Python ${{ matrix.python-version }})
runs-on: ${{ matrix.runs-on }}
needs: [download-jars, build]
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
strategy:
# Don't cancel other matrix jobs if one fails
fail-fast: false
# Chosen by the "Select the test matrix" step of download-jars from the event that
# started the run (scripts/select_matrix.py holds the platform and Python lists):
# small on pull requests, the full 4 platforms x 4 Pythons on every other event.
# Every leg carries platform, runs-on and python-version, so the job names stay
# "Test arcadedb (<platform>, Python <version>)".
matrix: ${{ fromJSON(needs.download-jars.outputs.matrix) }}

steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up Python
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: ${{ matrix.python-version }}

- name: Setup UV package manager
shell: bash
run: |
# Retry transient network failures (a connection reset failed a whole examples job on #8897); the
# script is downloaded first so a retried transfer is never piped half-written into sh.
installer="$(mktemp)"
curl -LsSf --retry 5 --retry-all-errors --retry-delay 5 -o "$installer" https://astral.sh/uv/install.sh
sh "$installer"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
export PATH="$HOME/.local/bin:$PATH"
uv --version

# The wheel of this platform, built once by the build job and the same file for every Python.
- name: Download the wheel of ${{ matrix.platform }}
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}
path: test-install

- name: Set UTF-8 encoding (Windows only)
if: matrix.platform == 'windows/amd64' || matrix.platform == 'windows/arm64'
shell: bash
run: |
echo "PYTHONIOENCODING=utf-8" >> $GITHUB_ENV
echo "PYTHONUTF8=1" >> $GITHUB_ENV

# The tests below run on the runner (glibc 2.39). This step runs the wheel where its tag says it must
# work: a manylinux_2_28 container (glibc 2.28, AlmaLinux 8, the oldest system the Linux wheels are tagged for).
Expand Down Expand Up @@ -397,11 +474,6 @@ jobs:
PYSMOKE
CONTAINER

- name: Set up Python for host testing
uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: ${{ matrix.python-version }}

# Note: Java is NOT required for arcadedb (JRE is bundled)
# This package works without any external Java installation!

Expand All @@ -411,9 +483,11 @@ jobs:
# This list is hand-maintained and mirrors the `test` extra. It had
# already drifted: pyarrow and pandas were missing, so to_arrow() and
# to_dataframe() tests skipped themselves in this job while it
# reported green. The guard step below now fails the build if any
# reported green. setuptools is there for tests/test_wheel_packages.py and
# tests/test_wheel_tags.py, which run setup.py on a copy of the sources (this job
# no longer installs build tools: it builds nothing). The guard step below now fails the build if any
# test skips for a missing import, so the next drift is loud.
uv pip install --system test-install/*.whl pytest pytest-cov requests numpy "psycopg[binary]" redis neo4j pyarrow pandas polars "adbc-driver-postgresql>=1.12" "duckdb>=1.4.2" "ladybug>=0.21.2" "networkx>=3.4"
uv pip install --system test-install/*.whl "setuptools>=84.1.0" pytest pytest-cov requests numpy "psycopg[binary]" redis neo4j pyarrow pandas polars "adbc-driver-postgresql>=1.12" "duckdb>=1.4.2" "ladybug>=0.21.2" "networkx>=3.4"

- name: Run pytest on host
id: pytest
Expand Down Expand Up @@ -557,7 +631,7 @@ jobs:
f.write("\n")

# Get wheel size and analyze contents
wheel_files = glob.glob("dist/*.whl")
wheel_files = glob.glob("test-install/*.whl")
if wheel_files:
wheel_file = wheel_files[0]

Expand Down Expand Up @@ -640,25 +714,10 @@ jobs:
.coverage
retention-days: 7

- name: Upload wheel artifact
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}-py${{ matrix.python-version }}
path: dist/*.whl
retention-days: 7

- name: Upload wheel artifact for release
if: matrix.python-version == '3.12'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: wheel-${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || matrix.platform == 'linux/arm64' && 'linux-arm64' || matrix.platform == 'darwin/amd64' && 'darwin-amd64' || matrix.platform == 'darwin/arm64' && 'darwin-arm64' || matrix.platform == 'windows/amd64' && 'windows-amd64' || 'windows-arm64' }}-test
path: dist/*.whl
retention-days: 7

# Summary job that checks all platforms
test-summary:
name: Test Summary
needs: test
needs: [build, test]
runs-on: ubuntu-24.04
# Skipped (not red) when the matrix was skipped for a draft PR.
if: always() && (github.event_name != 'pull_request' || github.event.pull_request.draft == false)
Expand All @@ -668,10 +727,10 @@ jobs:
echo "## 🎯 Overall Test Summary" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY

if [ "${{ needs.test.result }}" = "success" ]; then
if [ "${{ needs.build.result }}" = "success" ] && [ "${{ needs.test.result }}" = "success" ]; then
echo "✅ **All platforms passed testing!**" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "The arcadedb package has been successfully built and tested." >> $GITHUB_STEP_SUMMARY
echo "The arcadedb package has been successfully built (one py3-none wheel for each platform) and tested." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Package**: arcadedb" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
Expand Down
Loading
Loading