pred.py:98-99 (and :105-106 for the CoT second call), with result.py:37:
# pred.py — query_llm returns '' after 5 failed tries ("Max tries. Failed.")
if output == '':
continue # row is never written to results/*.jsonl
# result.py
output.append(name+'\t'+str(round(100*(easy_acc+hard_acc)/len(pred_data), 1)) + ...)When a request fails 5 times (connection error, server-side context or length rejection, timeout, rate limit), the question is skipped and never written. result.py then divides by len(pred_data), which counts only the rows that were written. It does not compare that count against the 503 questions and prints no warning. A missing question therefore counts neither as right nor as wrong. When failures are correlated with difficulty or length, which is likely for server-side length or timeout errors on the longest contexts, the reported accuracy goes up.
What happens (reproduced on the real module)
v2 pred.py drops items whose API call fails 5 times (pred.py:98-99, and 105-106 for the CoT answer call), and result.py:37 divides by the number of rows actually written, never by 503. result.py does not check completeness. So a run where some calls failed produces an ordinary-looking results row scored on fewer items, and when the failures are not random the score goes up. The resume cache (pred.py:130-139) retries missing ids on a re-run, but a failure that repeats on the same items (for example a content-filter rejection) keeps them missing on every run. With the unmodified pred.get_pred and result.py (dependencies stubbed, v2 bucket sizes 192/311 and 180/215/108), 60 failed calls on long items the model would get wrong move Overall from 39.0 to 44.2 and Long from 21.3 to 47.9, with nothing in the output to show it. That holds for plain and CoT runs alike. If a whole bucket is missing, or every call fails as in #124, result.py raises ZeroDivisionError at line 37 and writes no result.txt at all, so the rows for every other model in results/ are lost too. A results file with exactly one row crashes differently: json.load at result.py:10 parses it as a dict, which raises TypeError. Suggested fix: in result.py, count rows per bucket against the 503 ids of THUDM/LongBench-v2 and treat missing ids as wrong, or at least print n/503 and refuse to report an incomplete run. Also parse .jsonl files line by line instead of trying json.load first. Optionally, pred.py can write failed items with response='', pred=None, judge=False, the same way pred.py:109-112 already records unparseable replies.
Happy to open the PR.
pred.py:98-99(and:105-106for the CoT second call), withresult.py:37:When a request fails 5 times (connection error, server-side context or length rejection, timeout, rate limit), the question is skipped and never written.
result.pythen divides bylen(pred_data), which counts only the rows that were written. It does not compare that count against the 503 questions and prints no warning. A missing question therefore counts neither as right nor as wrong. When failures are correlated with difficulty or length, which is likely for server-side length or timeout errors on the longest contexts, the reported accuracy goes up.What happens (reproduced on the real module)
v2 pred.py drops items whose API call fails 5 times (pred.py:98-99, and 105-106 for the CoT answer call), and result.py:37 divides by the number of rows actually written, never by 503. result.py does not check completeness. So a run where some calls failed produces an ordinary-looking results row scored on fewer items, and when the failures are not random the score goes up. The resume cache (pred.py:130-139) retries missing ids on a re-run, but a failure that repeats on the same items (for example a content-filter rejection) keeps them missing on every run. With the unmodified pred.get_pred and result.py (dependencies stubbed, v2 bucket sizes 192/311 and 180/215/108), 60 failed calls on long items the model would get wrong move Overall from 39.0 to 44.2 and Long from 21.3 to 47.9, with nothing in the output to show it. That holds for plain and CoT runs alike. If a whole bucket is missing, or every call fails as in #124, result.py raises ZeroDivisionError at line 37 and writes no result.txt at all, so the rows for every other model in results/ are lost too. A results file with exactly one row crashes differently: json.load at result.py:10 parses it as a dict, which raises TypeError. Suggested fix: in result.py, count rows per bucket against the 503 ids of THUDM/LongBench-v2 and treat missing ids as wrong, or at least print n/503 and refuse to report an incomplete run. Also parse .jsonl files line by line instead of trying json.load first. Optionally, pred.py can write failed items with response='', pred=None, judge=False, the same way pred.py:109-112 already records unparseable replies.
Happy to open the PR.