Report benchmarks with no baseline counterpart in their own array

[jviotti]

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

[augmentcode]

🤖 Augment PR Summary

Summary: This PR separates relative benchmark output by measurement semantics.

• Refactors benchmark-record JSON construction into a reusable `to_entry` helper.
• Keeps baseline-matched records in `benchmarks` with proportional (`x`) timings.
• Writes records without a matching baseline to a new top-level `unmatched` array with raw nanosecond timings.
• Preserves the existing non-relative JSON report format and updates the partial-relative fixture.

_{🤖 Was this summary useful? React with 👍 or 👎}

[augmentcode]

Review completed. 1 suggestion posted.

Comment augment review to trigger a new review at any time.

[augmentcode]

src/lang/benchmark/benchmark.cc:408: When a relative run has no baseline matches, all of its raw ns durations are written only to unmatched, but read_baseline reads only benchmarks. Reusing that output with --relative-to therefore silently builds an empty baseline and reports every benchmark as unmatched instead of comparing against these recorded durations.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[github-actions]

Benchmark (linux/llvm)

Details

Benchmark suite	Current: 97de26f	Previous: 481d155	Ratio
Regex_Lower_S_Or_Upper_S_Asterisk	2.514438956691941 ns/iter	1.9139442061001783 ns/iter	1.31
Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar	2.493678426805379 ns/iter	1.9101969728297563 ns/iter	1.31
Regex_Period_Asterisk	2.805793219692698 ns/iter	1.9123239076586376 ns/iter	1.47
Regex_Group_Period_Asterisk_Group	2.8039364930204465 ns/iter	1.9191913643114247 ns/iter	1.46
Regex_Period_Plus	3.1137584489625287 ns/iter	1.9136148746996444 ns/iter	1.63
Regex_Period	2.913484854871095 ns/iter	1.9091126162443286 ns/iter	1.53
Regex_Caret_Period_Plus_Dollar	3.1173273059080455 ns/iter	1.9091676547649161 ns/iter	1.63
Regex_Caret_Group_Period_Plus_Group_Dollar	3.115015279058611 ns/iter	1.908898996475345 ns/iter	1.63
Regex_Caret_Period_Asterisk_Dollar	2.5296282781409127 ns/iter	2.7266649216967194 ns/iter	0.93
Regex_Caret_Group_Period_Asterisk_Group_Dollar	2.4913078885503728 ns/iter	2.726593272188447 ns/iter	0.91
Regex_Caret_X_Hyphen	6.384345063511081 ns/iter	5.452641925914444 ns/iter	1.17
Regex_Period_Md_Dollar	28.74001267652556 ns/iter	20.34681513266955 ns/iter	1.41
Regex_Caret_Slash_Period_Asterisk	6.5367535880884855 ns/iter	5.1804375903892605 ns/iter	1.26
Regex_Caret_Period_Range_Dollar	3.7487698606348787 ns/iter	3.2718656335601226 ns/iter	1.15
Regex_Nested_Backtrack	38.741890606305496 ns/iter	30.011024995553814 ns/iter	1.29
JSON_Array_Of_Objects_Unique	434.8827952947217 ns/iter	328.75641814776566 ns/iter	1.32
JSON_Parse_1	4706.575104117171 ns/iter	3428.611570897024 ns/iter	1.37
JSON_Parse_Real	5360.0641258882615 ns/iter	3729.2506217685273 ns/iter	1.44
JSON_Parse_Decimal	7905.102821188905 ns/iter	5868.3838095555075 ns/iter	1.35
JSON_Parse_Schema_ISO_Language	3372271.2523809522 ns/iter	2731449.876 ns/iter	1.23
JSON_Parse_Integer	3850.067563021503 ns/iter	3085.556721250646 ns/iter	1.25
JSON_Parse_String_NonSSO_Plain	3263.132949544738 ns/iter	2409.4726701534532 ns/iter	1.35
JSON_Parse_String_SSO_Plain	2791.183373181505 ns/iter	2272.7396027965506 ns/iter	1.23
JSON_Parse_String_Escape_Heavy	12735.608222710516 ns/iter	9605.989611817246 ns/iter	1.33
JSON_Parse_Object_Short_Keys	7942.614078665512 ns/iter	6230.260705255917 ns/iter	1.27
JSON_Parse_Object_Scalar_Properties	3956.8102142368975 ns/iter	3238.4159647997235 ns/iter	1.22
JSON_Parse_Object_Array_Properties	5806.859361705678 ns/iter	4520.532480410021 ns/iter	1.28
JSON_Parse_Object_Object_Properties	5621.382685733545 ns/iter	4466.215241490296 ns/iter	1.26
JSON_Parse_Nested_Containers	35866.00513795407 ns/iter	27036.505925954523 ns/iter	1.33
JSON_Fast_Hash_Helm_Chart_Lock	59.90184152877197 ns/iter	57.66671951266656 ns/iter	1.04
JSON_Equality_Helm_Chart_Lock	169.08566126907655 ns/iter	121.93667202948313 ns/iter	1.39
JSON_String_Equal_10	6.549118231069093 ns/iter	4.566082380035539 ns/iter	1.43
JSON_String_Equal_100	7.177403417475805 ns/iter	4.911568480550482 ns/iter	1.46
JSON_String_Equal_Small_By_Perfect_Hash_10	0.9359748730199842 ns/iter	0.8185388742190093 ns/iter	1.14
JSON_String_Equal_Small_By_Runtime_Perfect_Hash_10	14.970614212598308 ns/iter	9.619681530874368 ns/iter	1.56
JSON_String_Fast_Hash_10	2.491286245484838 ns/iter	1.9110428336403606 ns/iter	1.30
JSON_String_Fast_Hash_100	2.49313496995331 ns/iter	1.9107274713744875 ns/iter	1.30
JSON_String_Key_Hash_10	2.580144098381407 ns/iter	2.1832693797781606 ns/iter	1.18
JSON_String_Key_Hash_100	9.085917131459132 ns/iter	5.999478627404139 ns/iter	1.51
JSON_Object_Defines_Miss_Same_Length	2.8911537150758644 ns/iter	2.307684455508276 ns/iter	1.25
JSON_Object_Defines_Miss_Too_Small	2.6572969944320515 ns/iter	2.2997669606370135 ns/iter	1.16
JSON_Object_Defines_Miss_Too_Large	3.4267549011921883 ns/iter	3.2751061165110213 ns/iter	1.05
JSON_From_String_Copy	12.315058765495994 ns/iter	9.66579907832116 ns/iter	1.27
JSON_From_String_Temporary	10.930048223034252 ns/iter	7.700208885800761 ns/iter	1.42
JSON_Number_To_Double	23.084956191290154 ns/iter	17.46371655828596 ns/iter	1.32
JSON_Object_At_Last_Key_8	3.852992717640743 ns/iter	3.1190747709562094 ns/iter	1.24
JSON_Object_At_Last_Key_32	12.183985077835423 ns/iter	10.192938841385441 ns/iter	1.20
JSON_Object_At_Last_Key_128	49.14061431227171 ns/iter	36.3450140255977 ns/iter	1.35
JSON_Object_At_Last_Key_512	397.6475801344326 ns/iter	294.62849929791224 ns/iter	1.35
JSON_Divisible_By_Decimal	245.23928103027905 ns/iter	188.66264647783288 ns/iter	1.30
Pointer_Object_Traverse	29.204026562339347 ns/iter	24.10155972759 ns/iter	1.21
Pointer_Object_Try_Traverse	30.223237347109595 ns/iter	25.39619926450457 ns/iter	1.19
Pointer_Push_Back_Pointer_To_Weak_Pointer	169.26284776408545 ns/iter	122.7918375279984 ns/iter	1.38
Pointer_Walker_Schema_ISO_Language	1677705.42364532 ns/iter	1157982.1555183947 ns/iter	1.45
Pointer_Untracked_Deeply_Nested	1242584.0907504363 ns/iter	999353.4985673352 ns/iter	1.24
Pointer_Tracked_Deeply_Nested	1625255.211627907 ns/iter	1228980.074468085 ns/iter	1.32
Pointer_Position_Tracker_Get_Deeply_Nested	569.7717731277254 ns/iter	525.147671384846 ns/iter	1.08
Schema_Tracker_ISO_Language	4557897.552631579 ns/iter	3533433.223880597 ns/iter	1.29
Schema_Tracker_ISO_Language_To_JSON	17513499.075 ns/iter	12652263.545454545 ns/iter	1.38
Schema_Frame_WoT_References	2223308.933753943 ns/iter	1741979.0757575757 ns/iter	1.28
Schema_Frame_OMC_References	10010485.52112676 ns/iter	8055453.021978022 ns/iter	1.24
Schema_Frame_OMC_Pointers	19505067.666666668 ns/iter	15872450.108695652 ns/iter	1.23
Schema_Frame_OMC_Locations	8896578.023809524 ns/iter	6978516.978947368 ns/iter	1.27
Schema_Frame_ISO_Language_Locations	42817170.75 ns/iter	33240064.19047619 ns/iter	1.29
Schema_Frame_ISO_Language_Root	307.4571213018064 ns/iter	222.82100881666864 ns/iter	1.38
Schema_Frame_KrakenD_References	16782979.85365854 ns/iter	13275847.185185185 ns/iter	1.26
Schema_Frame_KrakenD_Reachable	156847782 ns/iter	103278045 ns/iter	1.52
Schema_Frame_ISO_Language_Locations_To_JSON	33216860.681818184 ns/iter	27651288.111111112 ns/iter	1.20
Schema_Frame_Many_Resources_References	143476300 ns/iter	116104750.66666667 ns/iter	1.24
Schema_Frame_Deeply_Nested_References	11958925.593220338 ns/iter	9284103.453333333 ns/iter	1.29
JSONPath_Descendant_Filter_Nested	1564.301546672113 ns/iter	1181.3253320282836 ns/iter	1.32
URITemplateRouter_Create	23631.06629330994 ns/iter	15880.02134639583 ns/iter	1.49
URITemplateRouter_Match	175.56177640293888 ns/iter	136.0381070655385 ns/iter	1.29
URITemplateRouterView_Restore	8701.69872791606 ns/iter	7707.42175457409 ns/iter	1.13
URITemplateRouterView_Match	143.49903878801996 ns/iter	126.74341524996827 ns/iter	1.13
URITemplateRouterView_Arguments	438.01785071032594 ns/iter	348.13430936563697 ns/iter	1.26
URITemplateRouter_Match_BasePath	198.49446562312647 ns/iter	151.05399783949025 ns/iter	1.31
URITemplateRouterView_Match_BasePath	165.7870368529856 ns/iter	147.50348361316773 ns/iter	1.12
JSONL_Parse_Large	9180604.342105264 ns/iter	6814151.86407767 ns/iter	1.35
JSONL_Parse_Large_GZIP	9345534.173333334 ns/iter	6927329.306930693 ns/iter	1.35
JSONLD_Catalog_Annotation_List_Populate	603067.0497854077 ns/iter	453918.43701298704 ns/iter	1.33
JSONLD_Catalog_Materialize	3814203.5572916665 ns/iter	2739758.513833992 ns/iter	1.39
HTML_Build_Table_100000	37598262.7368421 ns/iter	27201292.269230768 ns/iter	1.38
HTML_Render_Table_100000	2021273.0782608695 ns/iter	1944349.6516853932 ns/iter	1.04
Markdown_To_HTML_Realistic_Document	4105015.3076923075 ns/iter	3143088.9324324327 ns/iter	1.31
Markdown_To_HTML_Pathological	12982365.462962963 ns/iter	10653605.242424242 ns/iter	1.22
Markdown_To_HTML_Short_Descriptions	559.1403473718649 ns/iter	378.5144014960573 ns/iter	1.48
GZIP_Compress_ISO_Language_Set_3_Locations	37576004.368421055 ns/iter	31255176.045454547 ns/iter	1.20
GZIP_Decompress_ISO_Language_Set_3_Locations	5165359.05882353 ns/iter	4308724.12962963 ns/iter	1.20
GZIP_Decompress_Default_Level_ISO_Language_Set_3_Locations	7088862.132653061 ns/iter	5723866.4754098365 ns/iter	1.24
GZIP_Decompress_Stream_Default_Level_ISO_Language_Set_3_Locations	9897238.178082192 ns/iter	7426673.9130434785 ns/iter	1.33
GZIP_Compress_ISO_Language_Set_3_Schema	2193610.4402515725 ns/iter	1927981.123966942 ns/iter	1.14
GZIP_Decompress_ISO_Language_Set_3_Schema	427383.8317132442 ns/iter	261539.47583364556 ns/iter	1.63
JOSE_VerifySignature_RS256	47063.90043028103 ns/iter	40216.160751206065 ns/iter	1.17
JOSE_VerifySignature_ES512	1750308.3532338308 ns/iter	1471700.512605042 ns/iter	1.19
CRYPTO_CRC32_Large_JSONL	101795.12501821361 ns/iter	88755.85948002536 ns/iter	1.15

This comment was automatically generated by workflow using github-action-benchmark.

[github-actions]

Benchmark (linux/gcc)

Details

Benchmark suite	Current: 97de26f	Previous: 481d155	Ratio
CRYPTO_CRC32_Large_JSONL	101646.03648569761 ns/iter	101755.51498399767 ns/iter	1.00
JOSE_VerifySignature_RS256	22004.936786489292 ns/iter	22189.017073170733 ns/iter	0.99
JOSE_VerifySignature_ES512	582451.6104651163 ns/iter	581621.0872817956 ns/iter	1.00
GZIP_Compress_ISO_Language_Set_3_Locations	30146413.347826086 ns/iter	30656790.13043478 ns/iter	0.98
GZIP_Decompress_ISO_Language_Set_3_Locations	5194132.126865672 ns/iter	5150739.818181818 ns/iter	1.01
GZIP_Decompress_Default_Level_ISO_Language_Set_3_Locations	7270089.375 ns/iter	7241730.71875 ns/iter	1.00
GZIP_Decompress_Stream_Default_Level_ISO_Language_Set_3_Locations	9094196.753246753 ns/iter	9563581.561643835 ns/iter	0.95
GZIP_Compress_ISO_Language_Set_3_Schema	2060863.2064896754 ns/iter	2069251.5 ns/iter	1.00
GZIP_Decompress_ISO_Language_Set_3_Schema	439391.260951189 ns/iter	429134.76715686277 ns/iter	1.02
Markdown_To_HTML_Realistic_Document	5100467.97080292 ns/iter	5256062.992366413 ns/iter	0.97
Markdown_To_HTML_Pathological	13387321.830188679 ns/iter	13592812.76923077 ns/iter	0.98
Markdown_To_HTML_Short_Descriptions	677.69913674825 ns/iter	706.3100479863642 ns/iter	0.96
HTML_Build_Table_100000	38639395.05555555 ns/iter	38613019 ns/iter	1.00
HTML_Render_Table_100000	1824646.6069518717 ns/iter	1971312.3885714286 ns/iter	0.93
JSONLD_Catalog_Annotation_List_Populate	730945.1096033403 ns/iter	705897.6084762865 ns/iter	1.04
JSONLD_Catalog_Materialize	3815698.870786517 ns/iter	3586520.282051282 ns/iter	1.06
JSONL_Parse_Large	9796250.291666666 ns/iter	9863246.450704226 ns/iter	0.99
JSONL_Parse_Large_GZIP	9969758.628571428 ns/iter	10090943.666666666 ns/iter	0.99
URITemplateRouter_Create	21974.89197346496 ns/iter	22317.622944013285 ns/iter	0.98
URITemplateRouter_Match	159.04974831578494 ns/iter	157.11833673570135 ns/iter	1.01
URITemplateRouterView_Restore	8543.087875350006 ns/iter	8453.789193508728 ns/iter	1.01
URITemplateRouterView_Match	124.24507238287168 ns/iter	125.076840560213 ns/iter	0.99
URITemplateRouterView_Arguments	471.2205650773586 ns/iter	474.2237961672937 ns/iter	0.99
URITemplateRouter_Match_BasePath	191.27502663493883 ns/iter	183.3086114823231 ns/iter	1.04
URITemplateRouterView_Match_BasePath	143.16033195315157 ns/iter	146.41445230453638 ns/iter	0.98
JSONPath_Descendant_Filter_Nested	1650.444335868349 ns/iter	1727.4549971067406 ns/iter	0.96
Schema_Frame_WoT_References	2348847.9698996656 ns/iter	2368046.058608059 ns/iter	0.99
Schema_Frame_OMC_References	10752327.119402984 ns/iter	10104505.63235294 ns/iter	1.06
Schema_Frame_OMC_Pointers	19808044.055555556 ns/iter	19022137.81081081 ns/iter	1.04
Schema_Frame_OMC_Locations	9040932.256410256 ns/iter	9645454.172839506 ns/iter	0.94
Schema_Frame_ISO_Language_Locations	43099970.8125 ns/iter	40868286.1875 ns/iter	1.05
Schema_Frame_ISO_Language_Root	314.72465852357425 ns/iter	316.55444732517174 ns/iter	0.99
Schema_Frame_KrakenD_References	16634377.166666666 ns/iter	16379964.720930232 ns/iter	1.02
Schema_Frame_KrakenD_Reachable	144224675.6 ns/iter	146915612 ns/iter	0.98
Schema_Frame_ISO_Language_Locations_To_JSON	32592199.772727273 ns/iter	32474526.04761905 ns/iter	1.00
Schema_Frame_Many_Resources_References	146371703 ns/iter	147512792.4 ns/iter	0.99
Schema_Frame_Deeply_Nested_References	11223473.111111112 ns/iter	11261557.274193548 ns/iter	1.00
Pointer_Object_Traverse	23.49641653608405 ns/iter	24.97493386401393 ns/iter	0.94
Pointer_Object_Try_Traverse	24.635386796140093 ns/iter	24.51802468999132 ns/iter	1.00
Pointer_Push_Back_Pointer_To_Weak_Pointer	130.29759747340205 ns/iter	131.96029712502008 ns/iter	0.99
Pointer_Walker_Schema_ISO_Language	1525036.0869565217 ns/iter	1524818.0264900662 ns/iter	1.00
Pointer_Untracked_Deeply_Nested	1230569.0105263158 ns/iter	1237692.6812609457 ns/iter	0.99
Pointer_Tracked_Deeply_Nested	1954035.2348066298 ns/iter	1939376.0558659218 ns/iter	1.01
Pointer_Position_Tracker_Get_Deeply_Nested	620.2858526931481 ns/iter	602.677973074277 ns/iter	1.03
Schema_Tracker_ISO_Language	4977568.567375886 ns/iter	4960028.725352113 ns/iter	1.00
Schema_Tracker_ISO_Language_To_JSON	15794910.488372093 ns/iter	15795868.2 ns/iter	1.00
JSON_Array_Of_Objects_Unique	471.32734707347316 ns/iter	472.1122565722468 ns/iter	1.00
JSON_Parse_1	5393.2393307823195 ns/iter	5427.484124511754 ns/iter	0.99
JSON_Parse_Real	5691.068675753205 ns/iter	5614.204878516777 ns/iter	1.01
JSON_Parse_Decimal	10415.434966241337 ns/iter	10541.85753226608 ns/iter	0.99
JSON_Parse_Schema_ISO_Language	3728376.138297872 ns/iter	3725560.1170212766 ns/iter	1.00
JSON_Parse_Integer	3787.8883464575406 ns/iter	3781.2670236111785 ns/iter	1.00
JSON_Parse_String_NonSSO_Plain	3582.9620845035647 ns/iter	3587.688447250734 ns/iter	1.00
JSON_Parse_String_SSO_Plain	3043.098925281918 ns/iter	3088.364881622206 ns/iter	0.99
JSON_Parse_String_Escape_Heavy	16608.465605457648 ns/iter	16377.757609789196 ns/iter	1.01
JSON_Parse_Object_Short_Keys	8817.05648870352 ns/iter	8646.25860514518 ns/iter	1.02
JSON_Parse_Object_Scalar_Properties	4446.420066919531 ns/iter	4397.865969108121 ns/iter	1.01
JSON_Parse_Object_Array_Properties	6666.121055940858 ns/iter	6669.559743567728 ns/iter	1.00
JSON_Parse_Object_Object_Properties	6549.579659018603 ns/iter	6770.576228878648 ns/iter	0.97
JSON_Parse_Nested_Containers	40764.14808594429 ns/iter	40453.81792277274 ns/iter	1.01
JSON_Fast_Hash_Helm_Chart_Lock	75.12312168510445 ns/iter	78.3691316593568 ns/iter	0.96
JSON_Equality_Helm_Chart_Lock	155.802780723611 ns/iter	152.42938671910193 ns/iter	1.02
JSON_String_Equal_10	6.002355797002148 ns/iter	6.040627470236804 ns/iter	0.99
JSON_String_Equal_100	6.624823650270392 ns/iter	6.659576485157815 ns/iter	0.99
JSON_String_Equal_Small_By_Perfect_Hash_10	0.6265965203272945 ns/iter	0.6236349042365588 ns/iter	1.00
JSON_String_Equal_Small_By_Runtime_Perfect_Hash_10	13.73076503886918 ns/iter	13.080364914372794 ns/iter	1.05
JSON_String_Fast_Hash_10	3.151986096421649 ns/iter	3.1499407057876545 ns/iter	1.00
JSON_String_Fast_Hash_100	3.1602227106142933 ns/iter	3.1541317113945677 ns/iter	1.00
JSON_String_Key_Hash_10	1.558865051187689 ns/iter	1.2471110372876373 ns/iter	1.25
JSON_String_Key_Hash_100	12.75948723018792 ns/iter	12.442295458950232 ns/iter	1.03
JSON_Object_Defines_Miss_Same_Length	3.114967454378294 ns/iter	3.427026937548038 ns/iter	0.91
JSON_Object_Defines_Miss_Too_Small	3.1145001011420432 ns/iter	3.1176839963703524 ns/iter	1.00
JSON_Object_Defines_Miss_Too_Large	3.1136499569739646 ns/iter	3.4252561310985965 ns/iter	0.91
JSON_From_String_Copy	9.11531984635682 ns/iter	9.14623948956175 ns/iter	1.00
JSON_From_String_Temporary	8.622285819695982 ns/iter	8.237940199330863 ns/iter	1.05
JSON_Number_To_Double	19.143771281654335 ns/iter	19.66719496745814 ns/iter	0.97
JSON_Object_At_Last_Key_8	5.657430990766539 ns/iter	5.729889385652037 ns/iter	0.99
JSON_Object_At_Last_Key_32	21.82841087146266 ns/iter	23.326813777167775 ns/iter	0.94
JSON_Object_At_Last_Key_128	87.60738512260043 ns/iter	87.35908471443145 ns/iter	1.00
JSON_Object_At_Last_Key_512	389.99824635288195 ns/iter	400.3726351237158 ns/iter	0.97
JSON_Divisible_By_Decimal	278.4277909051895 ns/iter	283.10979729661346 ns/iter	0.98
Regex_Lower_S_Or_Upper_S_Asterisk	2.412554389641894 ns/iter	2.101266200135369 ns/iter	1.15
Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar	2.3347857405758123 ns/iter	2.02317834785274 ns/iter	1.15
Regex_Period_Asterisk	2.413780479383777 ns/iter	2.1016608471499634 ns/iter	1.15
Regex_Group_Period_Asterisk_Group	2.4119611764775817 ns/iter	2.1011246082854056 ns/iter	1.15
Regex_Period_Plus	2.412513032031254 ns/iter	2.102021758887454 ns/iter	1.15
Regex_Period	2.3576058329880922 ns/iter	2.025578961197259 ns/iter	1.16
Regex_Caret_Period_Plus_Dollar	2.412775546408031 ns/iter	2.104389279303436 ns/iter	1.15
Regex_Caret_Group_Period_Plus_Group_Dollar	2.412134154814597 ns/iter	2.105515008697042 ns/iter	1.15
Regex_Caret_Period_Asterisk_Dollar	2.422982261280363 ns/iter	2.1032760357496416 ns/iter	1.15
Regex_Caret_Group_Period_Asterisk_Group_Dollar	2.337075303281145 ns/iter	2.0249686522489823 ns/iter	1.15
Regex_Caret_X_Hyphen	8.016629790741975 ns/iter	7.400599733282675 ns/iter	1.08
Regex_Period_Md_Dollar	34.75231420443889 ns/iter	30.480980524334814 ns/iter	1.14
Regex_Caret_Slash_Period_Asterisk	8.950439906962487 ns/iter	8.95579748559763 ns/iter	1.00
Regex_Caret_Period_Range_Dollar	3.8905790366769692 ns/iter	3.8927568780458754 ns/iter	1.00
Regex_Nested_Backtrack	38.77830348969506 ns/iter	44.07115616186421 ns/iter	0.88

This comment was automatically generated by workflow using github-action-benchmark.

[github-actions]

Benchmark (windows/msvc)

Details

Benchmark suite	Current: 97de26f	Previous: 481d155	Ratio
Regex_Lower_S_Or_Upper_S_Asterisk	2.079598655620272 ns/iter	4.761672664901385 ns/iter	0.44
Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar	2.1009517717640716 ns/iter	4.803998613578055 ns/iter	0.44
Regex_Period_Asterisk	2.0673778487244197 ns/iter	4.741879717569577 ns/iter	0.44
Regex_Group_Period_Asterisk_Group	1.988705297294006 ns/iter	4.756595317973551 ns/iter	0.42
Regex_Period_Plus	2.0008811059793645 ns/iter	4.46667667183735 ns/iter	0.45
Regex_Period	2.4205424293167366 ns/iter	4.61320468152044 ns/iter	0.52
Regex_Caret_Period_Plus_Dollar	2.026416991360218 ns/iter	4.505615810232925 ns/iter	0.45
Regex_Caret_Group_Period_Plus_Group_Dollar	2.438641357363044 ns/iter	4.62614002853871 ns/iter	0.53
Regex_Caret_Period_Asterisk_Dollar	2.0992380423678862 ns/iter	4.738664442479198 ns/iter	0.44
Regex_Caret_Group_Period_Asterisk_Group_Dollar	2.0946391162847524 ns/iter	4.749830244127482 ns/iter	0.44
Regex_Caret_X_Hyphen	5.901625643891394 ns/iter	7.870659409719172 ns/iter	0.75
Regex_Period_Md_Dollar	26.11147527820178 ns/iter	48.39383086492583 ns/iter	0.54
Regex_Caret_Slash_Period_Asterisk	5.504775615331232 ns/iter	7.568944439424493 ns/iter	0.73
Regex_Caret_Period_Range_Dollar	3.0100330494294525 ns/iter	5.398915953534258 ns/iter	0.56
Regex_Nested_Backtrack	36.15985669370172 ns/iter	58.09812632214421 ns/iter	0.62
JSON_Array_Of_Objects_Unique	485.6011045590871 ns/iter	578.7917118816813 ns/iter	0.84
JSON_Parse_1	6460.3181809758535 ns/iter	8892.37113665716 ns/iter	0.73
JSON_Parse_Real	10834.782750628407 ns/iter	16171.990166345005 ns/iter	0.67
JSON_Parse_Decimal	8053.664912461734 ns/iter	12095.37985274431 ns/iter	0.67
JSON_Parse_Schema_ISO_Language	7256418.75 ns/iter	7725735.106382979 ns/iter	0.94
JSON_Parse_Integer	4067.4213738816757 ns/iter	6352.85697795221 ns/iter	0.64
JSON_Parse_String_NonSSO_Plain	5794.873746566503 ns/iter	7920.743383883438 ns/iter	0.73
JSON_Parse_String_SSO_Plain	2561.7990880603184 ns/iter	3774.899479057914 ns/iter	0.68
JSON_Parse_String_Escape_Heavy	16428.116446300162 ns/iter	22190.365089652194 ns/iter	0.74
JSON_Parse_Object_Short_Keys	9073.073677377475 ns/iter	13267.969474900112 ns/iter	0.68
JSON_Parse_Object_Scalar_Properties	4663.29151212458 ns/iter	6892.903181835884 ns/iter	0.68
JSON_Parse_Object_Array_Properties	8209.442776109443 ns/iter	11559.526469362234 ns/iter	0.71
JSON_Parse_Object_Object_Properties	8465.647736880337 ns/iter	11931.197780953182 ns/iter	0.71
JSON_Parse_Nested_Containers	59688.74322670862 ns/iter	83896.80437424059 ns/iter	0.71
JSON_Fast_Hash_Helm_Chart_Lock	78.81182659373654 ns/iter	102.99678188664026 ns/iter	0.77
JSON_Equality_Helm_Chart_Lock	200.54415417153194 ns/iter	215.58661722106453 ns/iter	0.93
JSON_String_Equal_10	8.321449263770797 ns/iter	10.445066169036194 ns/iter	0.80
JSON_String_Equal_100	11.605393463640283 ns/iter	11.709660223622775 ns/iter	0.99
JSON_String_Equal_Small_By_Perfect_Hash_10	1.2434287585535908 ns/iter	2.2279590187057265 ns/iter	0.56
JSON_String_Equal_Small_By_Runtime_Perfect_Hash_10	9.340336833013177 ns/iter	14.301667463667933 ns/iter	0.65
JSON_String_Fast_Hash_10	3.74325661008676 ns/iter	6.833282299183591 ns/iter	0.55
JSON_String_Fast_Hash_100	3.9038024368667985 ns/iter	6.685171789887977 ns/iter	0.58
JSON_String_Key_Hash_10	3.389313203334278 ns/iter	5.194490114806414 ns/iter	0.65
JSON_String_Key_Hash_100	7.623825918997592 ns/iter	11.972924224266132 ns/iter	0.64
JSON_Object_Defines_Miss_Same_Length	3.817773911722869 ns/iter	4.2801276555897365 ns/iter	0.89
JSON_Object_Defines_Miss_Too_Small	3.5233076989879204 ns/iter	3.841774590730692 ns/iter	0.92
JSON_Object_Defines_Miss_Too_Large	3.0152372263982397 ns/iter	3.540776480218318 ns/iter	0.85
JSON_From_String_Copy	45.13532486422485 ns/iter	64.31359721821552 ns/iter	0.70
JSON_From_String_Temporary	51.06616268719835 ns/iter	61.54212809392798 ns/iter	0.83
JSON_Number_To_Double	80.24044150852127 ns/iter	121.306299350662 ns/iter	0.66
JSON_Object_At_Last_Key_8	7.231234763507371 ns/iter	7.084179437259166 ns/iter	1.02
JSON_Object_At_Last_Key_32	16.893872806963262 ns/iter	23.894715190071913 ns/iter	0.71
JSON_Object_At_Last_Key_128	93.44968435526405 ns/iter	91.57536364331294 ns/iter	1.02
JSON_Object_At_Last_Key_512	289.71081090275754 ns/iter	428.1514875531269 ns/iter	0.68
JSON_Divisible_By_Decimal	240.44708330834115 ns/iter	320.1401398708023 ns/iter	0.75
Pointer_Object_Traverse	35.240346784002675 ns/iter	70.66027069833052 ns/iter	0.50
Pointer_Object_Try_Traverse	39.30702982272322 ns/iter	70.92403405953293 ns/iter	0.55
Pointer_Push_Back_Pointer_To_Weak_Pointer	136.52222413674994 ns/iter	187.86566513664093 ns/iter	0.73
Pointer_Walker_Schema_ISO_Language	9467108.108108109 ns/iter	11579948.4375 ns/iter	0.82
Pointer_Untracked_Deeply_Nested	1993761.4705882352 ns/iter	2511420.5776173286 ns/iter	0.79
Pointer_Tracked_Deeply_Nested	3559194.7916666665 ns/iter	3865024.4565217393 ns/iter	0.92
Pointer_Position_Tracker_Get_Deeply_Nested	500.539649153002 ns/iter	550.9773101735345 ns/iter	0.91
Schema_Tracker_ISO_Language	10108231.147540983 ns/iter	10757348.484848484 ns/iter	0.94
Schema_Tracker_ISO_Language_To_JSON	47447207.14285714 ns/iter	66424600 ns/iter	0.71
Schema_Frame_WoT_References	4427129.9363057325 ns/iter	5634760.1626016265 ns/iter	0.79
Schema_Frame_OMC_References	19098702.85714286 ns/iter	23174593.333333332 ns/iter	0.82
Schema_Frame_OMC_Pointers	33965242.85714286 ns/iter	43985366.666666664 ns/iter	0.77
Schema_Frame_OMC_Locations	16103288.095238095 ns/iter	20010515.151515152 ns/iter	0.80
Schema_Frame_ISO_Language_Locations	60556250 ns/iter	87310977.77777778 ns/iter	0.69
Schema_Frame_ISO_Language_Root	1578.3070342626656 ns/iter	2190.041461340271 ns/iter	0.72
Schema_Frame_KrakenD_References	31429560.86956522 ns/iter	42292811.76470588 ns/iter	0.74
Schema_Frame_KrakenD_Reachable	206359533.33333334 ns/iter	405768000 ns/iter	0.51
Schema_Frame_ISO_Language_Locations_To_JSON	135883360 ns/iter	165672725 ns/iter	0.82
Schema_Frame_Many_Resources_References	436086200 ns/iter	731750400 ns/iter	0.60
Schema_Frame_Deeply_Nested_References	15471141.860465117 ns/iter	25114108.695652176 ns/iter	0.62
JSONPath_Descendant_Filter_Nested	1565.4487581320036 ns/iter	2581.447055514358 ns/iter	0.61
URITemplateRouter_Create	32588.880082058935 ns/iter	42742.93249649463 ns/iter	0.76
URITemplateRouter_Match	222.78280313106353 ns/iter	246.26785187280612 ns/iter	0.90
URITemplateRouterView_Restore	16259.185681917012 ns/iter	36672.86616222469 ns/iter	0.44
URITemplateRouterView_Match	163.5535796927118 ns/iter	189.18350875203652 ns/iter	0.86
URITemplateRouterView_Arguments	399.76629254734814 ns/iter	544.6621130393055 ns/iter	0.73
URITemplateRouter_Match_BasePath	253.5707829980552 ns/iter	282.672969639176 ns/iter	0.90
URITemplateRouterView_Match_BasePath	183.79268918753405 ns/iter	216.24664424864505 ns/iter	0.85
JSONL_Parse_Large	24040680 ns/iter	36128847.368421055 ns/iter	0.67
JSONL_Parse_Large_GZIP	21836753.125 ns/iter	33718840 ns/iter	0.65
JSONLD_Catalog_Annotation_List_Populate	2548583.650190114 ns/iter	3389242.5 ns/iter	0.75
JSONLD_Catalog_Materialize	6011238.260869565 ns/iter	11983410.126582278 ns/iter	0.50
HTML_Build_Table_100000	75530966.66666667 ns/iter	103643485.71428572 ns/iter	0.73
HTML_Render_Table_100000	11550558.620689655 ns/iter	8279452.873563218 ns/iter	1.40
Markdown_To_HTML_Realistic_Document	4348150.909090909 ns/iter	5979582.258064516 ns/iter	0.73
Markdown_To_HTML_Pathological	14880542.553191489 ns/iter	18147590 ns/iter	0.82
Markdown_To_HTML_Short_Descriptions	526.9874014752214 ns/iter	737.6115687404392 ns/iter	0.71
GZIP_Compress_ISO_Language_Set_3_Locations	28652020 ns/iter	40623200 ns/iter	0.71
GZIP_Decompress_ISO_Language_Set_3_Locations	10525807.462686567 ns/iter	9883209.090909092 ns/iter	1.07
GZIP_Decompress_Default_Level_ISO_Language_Set_3_Locations	11841391.07142857 ns/iter	12443319.298245614 ns/iter	0.95
GZIP_Decompress_Stream_Default_Level_ISO_Language_Set_3_Locations	16494718.604651162 ns/iter	16166455.813953489 ns/iter	1.02
GZIP_Compress_ISO_Language_Set_3_Schema	1931095.518207283 ns/iter	2333047.972972973 ns/iter	0.83
GZIP_Decompress_ISO_Language_Set_3_Schema	644915.2573529412 ns/iter	626606.4833005894 ns/iter	1.03
JOSE_VerifySignature_RS256	19292.01435841959 ns/iter	21381.640374698683 ns/iter	0.90
JOSE_VerifySignature_ES512	1008158.3665338645 ns/iter	1379792.5049309665 ns/iter	0.73
CRYPTO_CRC32_Large_JSONL	46578.939333598835 ns/iter	103953.0070754717 ns/iter	0.45

This comment was automatically generated by workflow using github-action-benchmark.

[github-actions]

Benchmark (macos/llvm)

Details

Benchmark suite	Current: 97de26f	Previous: 481d155	Ratio
Regex_Lower_S_Or_Upper_S_Asterisk	1.8082287959516437 ns/iter	3.2531331892182513 ns/iter	0.56
Regex_Caret_Lower_S_Or_Upper_S_Asterisk_Dollar	1.8241281606254427 ns/iter	2.492205320480249 ns/iter	0.73
Regex_Period_Asterisk	1.8407620957085034 ns/iter	3.3447419679705286 ns/iter	0.55
Regex_Group_Period_Asterisk_Group	2.3370053316154085 ns/iter	2.6348660583328063 ns/iter	0.89
Regex_Period_Plus	2.367719440018216 ns/iter	3.7322110547291025 ns/iter	0.63
Regex_Period	2.1294641840369715 ns/iter	2.9969948427796522 ns/iter	0.71
Regex_Caret_Period_Plus_Dollar	2.387014854423238 ns/iter	3.023214282824546 ns/iter	0.79
Regex_Caret_Group_Period_Plus_Group_Dollar	2.25655704876843 ns/iter	3.4686405962865874 ns/iter	0.65
Regex_Caret_Period_Asterisk_Dollar	1.7935613315643635 ns/iter	2.7460822157457474 ns/iter	0.65
Regex_Caret_Group_Period_Asterisk_Group_Dollar	1.9761281626803966 ns/iter	3.7872772214530777 ns/iter	0.52
Regex_Caret_X_Hyphen	6.70345264218697 ns/iter	9.277758016621291 ns/iter	0.72
Regex_Period_Md_Dollar	20.05039652992525 ns/iter	28.93181367953272 ns/iter	0.69
Regex_Caret_Slash_Period_Asterisk	5.0714475 ns/iter	6.778455677889932 ns/iter	0.75
Regex_Caret_Period_Range_Dollar	2.3706665652229573 ns/iter	3.2286780599619638 ns/iter	0.73
Regex_Nested_Backtrack	29.917318082532187 ns/iter	60.06040879067403 ns/iter	0.50
JSON_Array_Of_Objects_Unique	381.61975836933414 ns/iter	558.64957573784 ns/iter	0.68
JSON_Parse_1	3462.2141369878573 ns/iter	11632.633830124576 ns/iter	0.30
JSON_Parse_Real	5997.373017003219 ns/iter	8745.283645802387 ns/iter	0.69
JSON_Parse_Decimal	5891.029782757656 ns/iter	8077.658958875228 ns/iter	0.73
JSON_Parse_Schema_ISO_Language	2369146.8949044584 ns/iter	3329658.3333333335 ns/iter	0.71
JSON_Parse_Integer	3649.684706350271 ns/iter	5034.888872099891 ns/iter	0.72
JSON_Parse_String_NonSSO_Plain	3232.494347705987 ns/iter	4611.385398937206 ns/iter	0.70
JSON_Parse_String_SSO_Plain	1812.6221641248349 ns/iter	2486.34870893964 ns/iter	0.73
JSON_Parse_String_Escape_Heavy	17513.967589466578 ns/iter	22869.187433964922 ns/iter	0.77
JSON_Parse_Object_Short_Keys	5556.027296175331 ns/iter	8232.725586618488 ns/iter	0.67
JSON_Parse_Object_Scalar_Properties	3416.43678407923 ns/iter	3769.264069165317 ns/iter	0.91
JSON_Parse_Object_Array_Properties	3970.1661755603636 ns/iter	4964.021633319143 ns/iter	0.80
JSON_Parse_Object_Object_Properties	3679.379322343067 ns/iter	4979.220888026 ns/iter	0.74
JSON_Parse_Nested_Containers	28414.9913061651 ns/iter	31913.457554254932 ns/iter	0.89
JSON_Fast_Hash_Helm_Chart_Lock	61.74748726326466 ns/iter	80.56824209441373 ns/iter	0.77
JSON_Equality_Helm_Chart_Lock	178.95225554158435 ns/iter	205.27963553828238 ns/iter	0.87
JSON_String_Equal_10	7.1340509138935415 ns/iter	9.369892468199474 ns/iter	0.76
JSON_String_Equal_100	6.8639291354285765 ns/iter	9.458785075477474 ns/iter	0.73
JSON_String_Equal_Small_By_Perfect_Hash_10	0.3558586876393651 ns/iter	0.4315899979059799 ns/iter	0.82
JSON_String_Equal_Small_By_Runtime_Perfect_Hash_10	3.450559207125818 ns/iter	3.8217864041284844 ns/iter	0.90
JSON_String_Fast_Hash_10	2.4946620016544325 ns/iter	3.128456705171279 ns/iter	0.80
JSON_String_Fast_Hash_100	2.1321637714794086 ns/iter	3.044680497233384 ns/iter	0.70
JSON_String_Key_Hash_10	1.8472037672236739 ns/iter	3.217416099035375 ns/iter	0.57
JSON_String_Key_Hash_100	2.3094661250392217 ns/iter	3.677706223609192 ns/iter	0.63
JSON_Object_Defines_Miss_Same_Length	3.168564839298447 ns/iter	3.8486645362557685 ns/iter	0.82
JSON_Object_Defines_Miss_Too_Small	2.8307500022526995 ns/iter	3.5835505142738677 ns/iter	0.79
JSON_Object_Defines_Miss_Too_Large	2.9318349781638533 ns/iter	4.0601896698373485 ns/iter	0.72
JSON_From_String_Copy	10.915610683936936 ns/iter	14.509659801783648 ns/iter	0.75
JSON_From_String_Temporary	6.638834390696722 ns/iter	10.00013957938204 ns/iter	0.66
JSON_Number_To_Double	33.98304139314243 ns/iter	49.702597276395544 ns/iter	0.68
JSON_Object_At_Last_Key_8	4.1718871094898065 ns/iter	6.208087596982013 ns/iter	0.67
JSON_Object_At_Last_Key_32	14.19980475641125 ns/iter	18.9423913691473 ns/iter	0.75
JSON_Object_At_Last_Key_128	57.575571597948745 ns/iter	91.43260675364495 ns/iter	0.63
JSON_Object_At_Last_Key_512	199.0435555252068 ns/iter	318.8067092461845 ns/iter	0.62
JSON_Divisible_By_Decimal	197.62430413022273 ns/iter	289.8292984803475 ns/iter	0.68
Pointer_Object_Traverse	22.634680724211076 ns/iter	23.943145820936376 ns/iter	0.95
Pointer_Object_Try_Traverse	42.024457915373425 ns/iter	43.773362476536185 ns/iter	0.96
Pointer_Push_Back_Pointer_To_Weak_Pointer	153.97169635777396 ns/iter	204.28651824591543 ns/iter	0.75
Pointer_Walker_Schema_ISO_Language	1635088.1126760563 ns/iter	2757797.0963172805 ns/iter	0.59
Pointer_Untracked_Deeply_Nested	901976.500621118 ns/iter	1275731.3674796748 ns/iter	0.71
Pointer_Tracked_Deeply_Nested	1122375.5821678322 ns/iter	1429794.3638392857 ns/iter	0.78
Pointer_Position_Tracker_Get_Deeply_Nested	383.16224964992387 ns/iter	634.1586417925326 ns/iter	0.60
Schema_Tracker_ISO_Language	3361024.0084745763 ns/iter	5270917.244755245 ns/iter	0.64
Schema_Tracker_ISO_Language_To_JSON	13562216.387096774 ns/iter	17966141.891891893 ns/iter	0.75
Schema_Frame_WoT_References	2415358.3669354836 ns/iter	4071442.3076923075 ns/iter	0.59
Schema_Frame_OMC_References	10462114.436619718 ns/iter	16636835.978723405 ns/iter	0.63
Schema_Frame_OMC_Pointers	22729630.03030303 ns/iter	36695836.538461536 ns/iter	0.62
Schema_Frame_OMC_Locations	10730033.216216216 ns/iter	13283751.482142856 ns/iter	0.81
Schema_Frame_ISO_Language_Locations	24931063.259259257 ns/iter	32346714.6 ns/iter	0.77
Schema_Frame_ISO_Language_Root	122.05937872434433 ns/iter	166.63352142026986 ns/iter	0.73
Schema_Frame_KrakenD_References	16622701.025 ns/iter	28997216.285714287 ns/iter	0.57
Schema_Frame_KrakenD_Reachable	189076347.33333334 ns/iter	328103375 ns/iter	0.58
Schema_Frame_ISO_Language_Locations_To_JSON	24410320.413793102 ns/iter	34925461.95652174 ns/iter	0.70
Schema_Frame_Many_Resources_References	133250133.4 ns/iter	195175833.25 ns/iter	0.68
Schema_Frame_Deeply_Nested_References	14336232.843137255 ns/iter	18815514.948717948 ns/iter	0.76
JSONPath_Descendant_Filter_Nested	1353.268556024445 ns/iter	1920.3176001145819 ns/iter	0.70
URITemplateRouter_Create	19800.104853205514 ns/iter	29116.70814429811 ns/iter	0.68
URITemplateRouter_Match	188.5507559752048 ns/iter	227.92447850155006 ns/iter	0.83
URITemplateRouterView_Restore	12202.89994106685 ns/iter	15955.219130906662 ns/iter	0.76
URITemplateRouterView_Match	152.59651707108057 ns/iter	166.766871934495 ns/iter	0.92
URITemplateRouterView_Arguments	588.6310983045984 ns/iter	695.0027865078205 ns/iter	0.85
URITemplateRouter_Match_BasePath	244.64994186482065 ns/iter	248.8315814111614 ns/iter	0.98
URITemplateRouterView_Match_BasePath	167.19451163326423 ns/iter	220.28860385859733 ns/iter	0.76
JSONL_Parse_Large	6711026.315789473 ns/iter	9921188.356164383 ns/iter	0.68
JSONL_Parse_Large_GZIP	7126354.944444444 ns/iter	8367412.162162162 ns/iter	0.85
JSONLD_Catalog_Annotation_List_Populate	568215.3079710145 ns/iter	756451.1306865178 ns/iter	0.75
JSONLD_Catalog_Materialize	2323672.1631944445 ns/iter	3634966.4512820514 ns/iter	0.64
HTML_Build_Table_100000	30258662.90909091 ns/iter	41541384.823529415 ns/iter	0.73
HTML_Render_Table_100000	2053608.534939759 ns/iter	2807372.230769231 ns/iter	0.73
Markdown_To_HTML_Realistic_Document	3383398.420118343 ns/iter	5216516.4522293 ns/iter	0.65
Markdown_To_HTML_Pathological	11881085.794117646 ns/iter	11170822.368421054 ns/iter	1.06
Markdown_To_HTML_Short_Descriptions	346.20690183325667 ns/iter	487.7352722396423 ns/iter	0.71
GZIP_Compress_ISO_Language_Set_3_Locations	31143840.608695652 ns/iter	45390950.941176474 ns/iter	0.69
GZIP_Decompress_ISO_Language_Set_3_Locations	3379355.7808219176 ns/iter	4293481.148809524 ns/iter	0.79
GZIP_Decompress_Default_Level_ISO_Language_Set_3_Locations	4725626.689189189 ns/iter	6129006.008474576 ns/iter	0.77
GZIP_Decompress_Stream_Default_Level_ISO_Language_Set_3_Locations	5953514.664 ns/iter	8289191.119565218 ns/iter	0.72
GZIP_Compress_ISO_Language_Set_3_Schema	1896781.2750642675 ns/iter	3054327.047413793 ns/iter	0.62
GZIP_Decompress_ISO_Language_Set_3_Schema	332004.2978723404 ns/iter	433383.47704081633 ns/iter	0.77
JOSE_VerifySignature_RS256	32517.98933375786 ns/iter	26944.96102841678 ns/iter	1.21
JOSE_VerifySignature_ES512	1131296.4794303798 ns/iter	1058406.5573529413 ns/iter	1.07
CRYPTO_CRC32_Large_JSONL	26420.942843436245 ns/iter	22521.377543035993 ns/iter	1.17

This comment was automatically generated by workflow using github-action-benchmark.

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="test/benchmark/benchmark_relative_unmatched_all.clitest">

<violation number="1" location="test/benchmark/benchmark_relative_unmatched_all.clitest:10">
P2: The test never inspects report.json, which is the actual output of the feature under test. The second RUN's error ("proportional results") is triggered solely by `context.relative_to` and would fire even if the "unmatched" array feature regressed entirely; and nothing asserts that `benchmarks` is empty or that the three unmatched entries land in `unmatched` with duration fields. benchmark_relative_partial.clitest shows the intended pattern: it normalizes and COMPAREs out.json against an expected document. Add a REPLACE/COMPARE block asserting `"benchmarks": []` and the three entries (addition, multiplication, subtraction) in `unmatched` with `time_unit` "ns", so a regression in the array split can't pass silently.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The test never inspects report.json, which is the actual output of the feature under test. The second RUN's error ("proportional results") is triggered solely by context.relative_to and would fire even if the "unmatched" array feature regressed entirely; and nothing asserts that benchmarks is empty or that the three unmatched entries land in unmatched with duration fields. benchmark_relative_partial.clitest shows the intended pattern: it normalizes and COMPAREs out.json against an expected document. Add a REPLACE/COMPARE block asserting "benchmarks": [] and the three entries (addition, multiplication, subtraction) in unmatched with time_unit "ns", so a regression in the array split can't pass silently.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At test/benchmark/benchmark_relative_unmatched_all.clitest, line 10:

<comment>The test never inspects report.json, which is the actual output of the feature under test. The second RUN's error ("proportional results") is triggered solely by `context.relative_to` and would fire even if the "unmatched" array feature regressed entirely; and nothing asserts that `benchmarks` is empty or that the three unmatched entries land in `unmatched` with duration fields. benchmark_relative_partial.clitest shows the intended pattern: it normalizes and COMPAREs out.json against an expected document. Add a REPLACE/COMPARE block asserting `"benchmarks": []` and the three entries (addition, multiplication, subtraction) in `unmatched` with `time_unit` "ns", so a regression in the array split can't pass silently.</comment>

<file context>
@@ -0,0 +1,26 @@
+  ]
+}
+BASELINE
+RUN --relative-to base.json --output report.json STDIN /dev/null IN . INTO first.txt EXPECTING 0
+REPLACE MATCHING "Running on [0-9]+ processors" WITH "Running on N processors" IN first.txt
+DROP LINES MATCHING "^1>   " IN first.txt
</file context>