Conversation
The execution-spec-test suites run as a single `evmone test` invocation over the whole fixture tree, so they execute serially on one core. Add EVMONE_SPEC_TESTS, a list of base directories scanned at configure time: the fixture directories three levels under state_tests / blockchain_tests become one ctest test each, labelled by type and named by the path relative to the base. A tree shallower than that is registered at its leaves, so every fixture runs exactly once. Empty by default, so normal builds register nothing. Registering each file instead is slower than the serial run it replaces, because ctest pays its per-test cost on one thread whatever --parallel says. Per directory it is up to ~4x faster on four cores.
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1705 +/- ##
=======================================
Coverage 98.02% 98.02%
=======================================
Files 179 179
Lines 16357 16357
Branches 3765 3765
=======================================
Hits 16034 16034
Misses 243 243
Partials 80 80
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The execution-spec-test suites run as a single
evmone testinvocation over the whole fixture tree, so they execute serially on one core.EVMONE_SPEC_TESTSis a list of base directories scanned at configure time: the fixture directories three levels understate_tests/blockchain_testsbecome one ctest test each, soctest --parallelspreads them over the cores. Empty by default, so normal builds register nothing.The granularity is the whole point. Registering each fixture file — the obvious approach, and what this branch did before — is slower than the serial run it replaces, 0.45x on four cores, because ctest pays its per-test cost on a single thread whatever
--parallelsays: 8172 trivial tests take 25s at-j1and 25s at-j32. Grouping by directory pays that cost 502 times instead of 8172 and gives roughly 4x on four cores, 55s down to 14s.A tree shallower than three levels is registered at its leaves instead, so every fixture runs exactly once. That case is real: the
developfixtures are two levels deep, and a fixed three-level glob silently skipped 261 state and 380 blockchain files. Verified against four fixture releases that the registered directories cover every file exactly once, with no duplicates.Tests are named by the path relative to the base directory, so fixtures from different sources stay distinct and are selectable with
ctest -R;ctest -L state_testsselects a type. CI is not switched over here — that needs the download step to run before configure, and is worth its own change.