[2010YOUY01]

Which issue does this PR close?

• Closes #.

Rationale for this change

Introduce a centralized struct CommandSpec for individual xtask commands.

This Keep each step’s environment, prerequisites, and documentation alongside its command, and it can make maintenance easier, and also easier to use: running with --explain shows docs, prerequisites, etc. for this command.

Demo

The default rust test in CI run this command

datafusion/.github/workflows/rust.yml

Line 467 in 350ca76

run: cargo xtask ci step test workspace

And we can run it locally, and also use --explain to show required dependencies, envvars, documentations. (now it only has required dependency, envar. Documentations for this specific commands can be add later)

yongting@Yongtings-MacBook-Pro-2 ~/C/t/datafusion-xtask-cleanup-refactor (xtask-cleanup-refac…)> cargo xtask ci step test workspace --explain
# Requires executable `cargo-llvm-cov` (install it with `cargo install cargo-llvm-cov --locked`)
cd /Users/yongting/Code/tmp/datafusion-xtask-cleanup-refactor && \
RUST_BACKTRACE=1 \
cargo llvm-cov \
--profile ci \
--exclude datafusion-examples \
--exclude ffi_example_table_provider \
--exclude datafusion-cli \
--workspace \
--lib \
--tests \
--bins \
--features serde,avro,json,backtrace,integration-tests,parquet_encryption,substrait \
--codecov \
--output-path target/codecov.json

What changes are included in this PR?

Key changes are:

1. Introduce CommandSpec
2. Move existing CI commands implementation to use this unified struct

What is the testing strategy for this PR?

UTs, also CI run itself

Are there any user-facing changes?

[alamb]

Looks good to me -- I tried it out locally

cargo xtask ci step test cli

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ cargo xtask ci step check --help
DataFusion CI command: check

Check workspace or package compilation

Usage:
  cargo xtask ci step check <workspace|package> [default|no-default|feature] [--explain]

Examples:
  cargo xtask ci step check workspace
  cargo xtask ci step check datafusion default

Use '--explain' to show the full command:
  cargo xtask ci step check workspace --explain

[alamb]

i wn't say I followed all this implementation, but I skimmed it all and it is pretty neat

[alamb]

I also took the liberty of merging up from main to resolve a merge conflict

[2010YOUY01]

@alamb Thanks for the review.

[2010YOUY01]

The failing fuzzing UT uses different seed in each run.

https://github.com/apache/datafusion/actions/runs/41196052207/job/124522323666

I'll capture the failing case and open an issue later. Now retry merge.