HDDS-16600. Reduce Develocity CI build failures - #11320
jojochuang wants to merge 5 commits into
Conversation
Reuse ozone-repo for integration (mvn test + OZONE_REPO_CACHED), harden RocksDB source download timeouts, and align repro.sh with install-based artifact:compare plus reproducible source-jar packaging. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I7622b48b92ffc48d08d6b7e29fb7b5ae1bf57caf
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Fix native-library bootstrap failure handling and preserve the downloaded reference artifacts during reproducibility checks.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Reduces Develocity CI failures through cached artifacts, native-library bootstrapping, longer download timeouts, and reproducibility updates.
Changes:
- Adds cached integration-test execution and native-library setup.
- Improves RocksDB download reliability.
- Updates reproducibility and source-JAR handling.
| File | Description |
|---|---|
pom.xml |
Configures source manifests and download timeouts. |
hadoop-ozone/dev-support/checks/repro.sh |
Updates the reproducibility build lifecycle. |
hadoop-ozone/dev-support/checks/junit.sh |
Adds cached tests and native-library bootstrapping. |
hadoop-hdds/rocks-native/pom.xml |
Adjusts RocksDB source download behavior. |
.github/workflows/ci.yml |
Enables the Ozone repository artifact for integration jobs. |
.github/workflows/check.yml |
Downloads and exposes the cached repository. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # (e.g. test-om in ozone-manager and ozone-integration-test). Use full-reactor mvn test. | ||
| # hadoop-native-lib is bound on ozone-main (inherited=false). | ||
| if [[ "$*" == *"-Phadoop-native-lib"* ]]; then | ||
| if ! mvn ${MAVEN_OPTIONS} -pl :ozone-main -Phadoop-native-lib generate-resources; then |
Use fail-fast Maven options for hadoop-native-lib generate-resources so bootstrap failures are not masked by --fail-never. Run repro as verify+artifact:compare when OZONE_REPO_CACHED preserves build-job refs. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I1a18c203f0d6a504c65333fbf4a4cf06a8551011
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @jojochuang for the patch.
| <configuration> | ||
| <archive> | ||
| <manifest> | ||
| <addDefaultImplementationEntries>false</addDefaultImplementationEntries> | ||
| <addDefaultSpecificationEntries>false</addDefaultSpecificationEntries> | ||
| </manifest> | ||
| </archive> | ||
| </configuration> |
There was a problem hiding this comment.
I don't think this is a problem, see doc:
Starting with version 2.1, Maven Archiver no longer creates the Implementation and Specification details in the manifest by default.
| # CI repro downloads build-job jars into ~/.m2 (OZONE_REPO_CACHED); compare against that reference. | ||
| # Local runs without a cached repo need install first to populate the reference. | ||
| if [[ "${OZONE_REPO_CACHED:-false}" == "true" ]]; then | ||
| source "${DIR}"/_build.sh verify artifact:compare "$@" | ||
| else | ||
| source "${DIR}"/_build.sh install artifact:compare "$@" | ||
| fi |
There was a problem hiding this comment.
This change is unnecessary. artifact:compare does not re-build the project, only compares. So installing first is useless, comparison can never fail afterwards.
For using repro.sh locally, you have to run build.sh first.
| mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" "$@" verify \ | ||
| # shellcheck disable=SC2068 | ||
| mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" \ | ||
| ${PL_ARGS[@]+"${PL_ARGS[@]}"} "$@" ${MAVEN_LIFECYCLE} \ |
There was a problem hiding this comment.
PL_ARGS is never populated, please don't add it.
| if [[ "${CHECK}" == "integration" ]]; then | ||
| # test-* profiles configure Surefire on the root POM; matching tests can live in any module | ||
| # (e.g. test-om in ozone-manager and ozone-integration-test). Use full-reactor mvn test. | ||
| # hadoop-native-lib is bound on ozone-main (inherited=false). | ||
| if [[ "$*" == *"-Phadoop-native-lib"* ]]; then | ||
| # Bootstrap must not inherit --fail-never from the main test invocation. | ||
| NATIVE_LIB_MVN_OPTIONS="${MAVEN_OPTIONS//--fail-never/}" | ||
| NATIVE_LIB_MVN_OPTIONS="${NATIVE_LIB_MVN_OPTIONS} --fail-fast" | ||
| if ! mvn ${NATIVE_LIB_MVN_OPTIONS} -pl :ozone-main -Phadoop-native-lib generate-resources; then | ||
| echo "Failed to populate target/native-lib (hadoop-native-lib profile)" >&2 | ||
| exit 1 | ||
| fi | ||
| if [[ ! -d target/native-lib ]] || ! ls target/native-lib/* >/dev/null 2>&1; then | ||
| echo "target/native-lib is missing after hadoop-native-lib generate-resources" >&2 | ||
| exit 1 | ||
| fi | ||
| fi | ||
| fi |
There was a problem hiding this comment.
Running generate-resources separately is unnecessary. It is already executed as part of the default lifecycle when running install (or verify, or test, or any similar phase).
| <outputFileName>rocksdb-v${rocksdb.source.version}.tar.gz</outputFileName> | ||
| <outputDirectory>${project.build.directory}/rocksdb</outputDirectory> | ||
| <readTimeOut>120000</readTimeOut> | ||
| <skipCache>false</skipCache> |
There was a problem hiding this comment.
… reactor Full-reactor mvn test with ozone-repo reuse fails compiling ozone-filesystem-hadoop2 main sources without a verify/shade build. Skip those adapter modules via -pl while keeping cross-module test-* coverage. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Id2ea315c59d92738d700934ee8552445dcd15038
Hadoop 2/3 FS modules are omitted from the reactor when integration runs with -DskipShade; -pl exclusions then fail project resolution. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ie59b71e01c8188664411b75e6fe5170b6f29ebb5
|
I had cancelled CI and converted to draft, because I believe changes are needed, so running checks on an "obsolete" version is wasting resources. Please keep as draft until implementation is stable, review comments are addressed, CI passes in fork. Please report distinct issues separately, instead of posting a "solve top N problems" issue / patch. Please validate both Cursor's problem analysis and its proposed solution. (See review comments for details where one or the other is wrong.) |
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I04707d5376178cd7ee7176f87bbcb1e61a11572c
Summary
ozone-repoartifact and run full-reactormvn testviaOZONE_REPO_CACHEDinjunit.sh, avoiding full-reactorverifyand the dominant MDEP-187 failure mode.target/native-libbefore integration when-Phadoop-native-libis set; fail fast if native-lib generation fails.readTimeOutonhdds-rocks-nativeand in rootdownload-maven-pluginpluginManagement.repro.shasinstall artifact:compareand stabilize-Pdistsource JAR manifests for reproducible builds (rootproject.build.outputTimestampunchanged).Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16600
Context
Develocity Failures for Apache Ozone Main (Jul 1–Sep 24, 2026 PDT): https://develocity.apache.org/scans/failures?search.rootProjectNames=Apache%20Ozone%20Main&search.startTimeMax=1790319599999&search.startTimeMin=1782889200000&search.timeZoneId=America%2FLos_Angeles
Top non-verification failures addressed: MDEP-187 (integration reactor packaging),
artifact:compare/ repro, and RocksDB sourcewgettimeouts onhdds-rocks-native.Test plan
./hadoop-ozone/dev-support/checks/checkstyle.shOZONE_REPO_CACHED=trueintegration smoke (-Ptest-om -Phadoop-native-lib, single test class) aftermvn install -DskipTests./hadoop-ozone/dev-support/checks/repro.shthrough:ozone-manager -amwith CI-like argsbuild-branchintegration + repro jobs on fork/ASFGenerated-by: Cursor (Auto)
Made with Cursor