Specimen

Specimen complements the Plausible property-based testing library by automatically deriving generators, enumerators, and checkers for inductive relations.

Specimen's design is heavily inspired by Coq/Rocq's QuickChick library and the following papers:

• Testing Theorems, Fully Automatically (OOPSLA 2026)
• Computing Correctly with Inductive Relations (PLDI 2022)
• Generating Good Generators for Inductive Relations (POPL 2018)

Specimen is a testing and verification tool - it is designed to help find bugs during development, not to serve as a security guarantee or correctness proof for production or enterprise workloads. Intended use is development-time property-based testing, rapid prototyping of invariants, and pre-proof exploration of conjectures.

Overview

Like QuickChick, Specimen uses the following typeclasses:

• Arbitrary: unconstrained random generators for inhabitants of algebraic data types. This is imported from Plausible
• ArbitrarySuchThat: constrained generators which only produce random values that satisfy a user-supplied inductive relation
• ArbitraryFueled, ArbitrarySizedSuchThat: versions of the two typeclasses above where the generator's size parameter is made explicit (the former is imported from Plausible)
• Enum, EnumSuchThat, EnumSized, EnumSizedSuchThat: Like their Arbitrary counterparts but for deterministic enumerators instead
• DecOpt: Checkers (partial decision procedures that return Except GenError Bool) for inductive propositions

Specimen provides various top-level commands which automatically derive generators for Lean inductives (the file Specimen/README.md has more details):

1. Deriving unconstrained generators/enumerators
An unconstrained generator produces random inhabitants of an algebraic data type, while an unconstrained enumerator enumerates (deterministically) these inhabitants.

Users can write deriving Arbitrary and/or deriving Enum after an inductive type definition, e.g.

inductive Foo where
  ...
  deriving Arbitrary, Enum

Alternatively, users can also write deriving instance Arbitrary for T1, ..., Tn (or deriving instance Enum ...) as a top-level command to derive Arbitrary / Enum instances for types T1, ..., Tn simultaneously. This also works for mutually recursive types:

mutual
  inductive MutEven where
    | zero : MutEven
    | succOdd : MutOdd → MutEven
  inductive MutOdd where
    | succEven : MutEven → MutOdd
end

deriving instance Enum for MutEven, MutOdd

To sample from a derived unconstrained generator, users can simply call runArbitrary, specify the type for the desired generated values and provide some Nat to act as the generator's size parameter (10 in the example below):

#eval runArbitrary (α := Tree) 10

Similarly, to return the elements produced from a derived enumerator, users can call runEnum like so:

#eval runEnum (α := Tree) 10

2. derive_mutual — the recommended command for constrained derivation

derive_mutual is the primary command for deriving constrained generators, enumerators, and checkers. It supersedes the older derive_generator/derive_enumerator/derive_checker commands by providing:

• Automatic dependency discovery (derives instances for sub-relations)
• Multi-output generation (a single hypothesis step can produce multiple existential variables)
• True mutual recursion (multiple specs compiled into a shared mutual block)
• Quality scoring and schedule search with branch-and-bound optimization

Syntax (autoDeriveDeps and multiOutput are on by default; set them false to opt out):

-- Derive a constrained generator (default sort is `generator`)
derive_mutual
  (fun n => ∃ (t : BinaryTree), balancedTree n t)

-- Derive multiple specs at once (they can call each other)
derive_mutual
  (fun G t => ∃ (e : term), typing G e t)

-- Derive with explicit sort keywords
derive_mutual
  generator (fun lo hi => ∃ (t : BinaryTree), BST lo hi t),
  checker (fun lo hi t => BST lo hi t)

-- Derive an enumerator
derive_mutual enumerator
  (fun n => ∃ (t : BinaryTree), balancedTree n t)

-- Multi-output: generate all existentials at once
derive_mutual
  (∃ (Γ : List type) (e : term) (τ : type), typing Γ e τ)

Each entry can be prefixed with generator (default), enumerator, or checker. While specimen.autoDeriveDeps is true (the default), Specimen automatically discovers and derives instances for sub-relations referenced in the constructors. While specimen.multiOutput is true (the default), the scheduler can produce multiple existential outputs in a single hypothesis step; set it false for relations where no joint producer instance exists for the tupled outputs.

To sample from a generator derived via derive_mutual:

#eval runSizedGen (ArbitrarySizedSuchThat.arbitrarySizedST (fun t => balanced 5 t)) 10

3. derive_generator / derive_enumerator — single-spec constrained derivation

These commands derive a constrained generator or enumerator for a single specification. They are still supported and useful for quick one-off derivations:

derive_generator (fun n => ∃ t, balanced n t)
derive_enumerator (fun n => ∃ t, balanced n t)

In the command derive_generator (fun x1 ... xn => ∃ x, P x1 ... x ... xn):

• P must be an inductively defined relation
• x is the value to be generated (bound by ∃)
• x1 ... xn are input parameters (bound by fun)
• Multiple existential outputs are supported: derive_generator (fun n => ∃ a b, Split n a b)

To sample from the derived producer:

#eval runSizedGen (ArbitrarySizedSuchThat.arbitrarySizedST (fun t => balanced 5 t)) 10
#eval runSizedEnum (EnumSizedSuchThat.enumSizedST (fun t => balanced 3 t)) 3

4. derive_checker — partial decision procedures

A checker for an inductively-defined Prop is a Nat -> Except GenError Bool function, which takes a Nat argument as fuel and returns an error if it can't decide whether the Prop holds (e.g. it runs out of fuel), and otherwise returns ok true / ok false depending on whether the Prop holds.

derive_checker (fun n t => balanced n t)

5. Options

Option	Default	Description
specimen.autoDeriveDeps	true	Automatically derive dependency instances for sub-relations in derive_mutual
specimen.multiOutput	true	Allow multi-output production steps (multiple ∃ vars generated per hypothesis)
specimen.scoreType	"Scoring.BudgetAwareScore"	Scoring strategy for schedule quality evaluation (see below)
specimen.weightFn	"Scoring.qualityCtorWeight"	Weight function for constructor frequency in derived generators (see below)
specimen.weightModifier	""	Optional modifier layered on top of the weight function (see below)
specimen.precomputeWeights	true	Partially evaluate weight expressions under the size binder at elaboration time
specimen.fuel	10000	Fuel (termination budget) for derived generators/enumerators/checkers
specimen.richOutput	true	Emit rich HTML widget output in the Lean infoview
specimen.textOutput	0	Plain-text output verbosity (0=off, 1=summary, 2=problems, 3=full)
specimen.silent	false	Suppress all informational derivation output (Try this: suggestions and derive_mutual widgets/text). Instances are still installed
specimen.searchLimit	200000	Max hypothesis orderings to evaluate per constructor during schedule search
specimen.reportParallelCeiling	false	Log the parallelism ceiling (total vs critical-path self-time) after derive_mutual
specimen.shrink	true	Enable counterexample shrinking in specimen_test / specimen
specimen.shrinkBreadth	8	Max shrink candidates considered per variable/step
specimen.shrinkDepth	100	Max greedy shrink steps per counterexample

Scoring strategies control how Specimen evaluates and selects among candidate schedules during derivation. The specimen.scoreType option selects the active strategy:

Strategy	Option value	Description
Budget-aware	"Scoring.BudgetAwareScore"	Default. Input-aware grading plus call locality: self-recursion shares the size budget (cheap), while a call to a different mode of the same relation invokes a separate instance (expensive, may cascade).
Source-quality	"Scoring.SourceQualityScore"	Tracks per-variable provenance across steps — SuchThat-produced vars carry their dep's density, Unconstrained vars carry maximum penalty.
Input-aware graded	"Scoring.InputAwareGradedScore"	Graded uniform density plus a penalty for checks whose generated variables overlap the inputs (generate-then-verify patterns).
Bounded graded	"Scoring.BoundedGradedScore"	Refines graded uniform density with a bounded metric.
Graded uniform density	"Scoring.GradedUniformDensityScore"	Uniform density with two-axis check severity.
Uniform density	"Scoring.UniformDensityScore"	Density analysis without checker inversion.
Density	"Scoring.DensityScore"	Categorical density classification (Total/Partial/Backtracking/Checking) from Section 4 of Testing Theorems, Fully Automatically. Prefers schedules that avoid backtracking.
Worst-leaf	"Scoring.WorstLeafScore"	Takes the max (not sum) across coverage-trie leaves — penalizes worst-case input paths.
Default	"Scoring.DefaultScore"	Sum of (checks, length, unconstrained) — the original heuristic. Minimizes total checking work. Deliberately blind to sub-relation cost, so it is a structural baseline rather than a recommended production setting.
Dep-aware default	"Scoring.DepAwareDefaultScore"	DefaultScore plus transitive sub-relation cost folded in from the memo. Sensitive to dependency cost, but the metric grows geometrically with dependency depth and is therefore unbounded.

For example, to pin the density scoring strategy from the Testing Theorems paper:

set_option specimen.scoreType "Scoring.DensityScore"
derive_mutual
  (fun lo hi => ∃ (t : BinaryTree), BST lo hi t)

See ScheduleQualityRegressionTest.lean for a comparison of strategies on the same relation.

Weight functions control how often each constructor is chosen at runtime by the backtracking combinator. The specimen.weightFn option selects the active weight function. A weight function has the signature:

CtorWeightFn := Name → List Nat → DeriveSort → Nat → Bool → Nat → Nat → Nat → Nat → Nat

Arguments: (ctorName, outputIndices, deriveSort, scoreBadness, isRec, size, numBase, numRec, numRecCalls) → weight

• ctorName: the fully qualified name of the constructor (e.g. `List.cons).
• outputIndices: the output position indices for this derivation.
• deriveSort: whether we are deriving a Generator, Enumerator, Checker, or Theorem.
• scoreBadness: schedule quality for this constructor as a per-mille Nat in 0..1000 (0 = best, 1000 = worst). Computed at elaboration time and baked in as a Nat literal (rather than a Float) so the weight application reduces cleanly. Bucket boundaries that were 0.25 / 0.5 / 0.75 in float terms are 250 / 500 / 750 here.
• isRec: whether this constructor is recursive.
• size: the current generation size parameter (decreases as the generator recurses deeper).
• numBase / numRec: counts of base vs recursive constructors for this inductive.
• numRecCalls: the number of size-consuming (recursive / same-inductive) calls made in this constructor's schedule (e.g. a binary-tree node ctor has 2). Computed at elaboration time and baked in as a literal — useful for penalizing high-fan-out constructors.

The return value is a Nat weight — the backtracking combinator picks constructors proportionally to their weights. The ctorName, outputIndices, and deriveSort arguments enable per-constructor and per-mode weight overrides without needing to write a separate weight function for each type.

Weight function	Option value	Description
Quality-only	"Scoring.qualityCtorWeight"	Default. No structural bias — maps badness to 1–4, ignores size/recursion. Relies on budget splitting for termination.
Balanced	"Scoring.balancedCtorWeight"	Controls aggregate P(recursive) with quality bias. Base ctors get a 4x boost. Good for inductives with many recursive constructors.
Size-proportional	"Scoring.sizeProportionalCtorWeight"	base=1, rec=size+1. The strategy used by QuickChick.
Score-aware	"Scoring.scoreAwareCtorWeight"	Boosts good constructors (low badness 1–4) and applies size-based penalty to recursive ones.
Flat	"Scoring.flatCtorWeight"	Every constructor gets weight 1. Ignores everything.
Default	"Scoring.defaultCtorWeight"	base=1, rec=numBase*size/numRec. Ignores score.

For example, to use size-proportional weights (QuickChick-style):

set_option specimen.weightFn "Scoring.sizeProportionalCtorWeight"
derive_mutual
  (fun lo hi => ∃ (t : BinaryTree), BST lo hi t)

Defining a custom weight function. You can define and register your own weight function in your own file — no need to modify Specimen. This is useful when you've derived generators and found the distribution poorly tuned for your application:

import Specimen

open Scoring Schedules in
-- Custom weight function: heavily favor base cases
def myCtorWeight (ctorName : Name) (outputIndices : List Nat) (deriveSort : DeriveSort)
    (scoreBadness : Nat) (isRec : Bool) (size : Nat) (numBase numRec : Nat) (numRecCalls : Nat) : Nat :=
  if isRec then
    if size == 0 then 0
    else max 1 (size / (max 1 numRec * 4))
  else 10

-- Use it for a specific derivation
set_option specimen.weightFn "myCtorWeight" in
derive_mutual
  (fun n => ∃ (xs : List Nat), SortedList n xs)

Targeting specific constructors. The ctorName, outputIndices, and deriveSort arguments let you override weights for particular constructors or modes while falling back to a default for everything else:

open Scoring Schedules in
def myTargetedWeight (ctorName : Name) (outputIndices : List Nat) (deriveSort : DeriveSort)
    (scoreBadness : Nat) (isRec : Bool) (size : Nat) (numBase numRec : Nat) (numRecCalls : Nat) : Nat :=
  -- Give a specific constructor a constant low weight
  if ctorName == ``MyType.ExpensiveCtor then 1
  -- Boost another constructor
  else if ctorName == ``MyType.PreferredCtor then 8
  -- Fall back to the default balanced strategy for everything else
  else balancedCtorWeight ctorName outputIndices deriveSort scoreBadness isRec size numBase numRec numRecCalls

The set_option ... in scoping means different derivations in the same file can use different weight functions. After changing the weight function, rederive and recompile your file to produce a generator reflecting the new weights.

Weight modifiers. Instead of replacing the entire weight function, you can layer a modifier on top. A CtorWeightModifier receives the base weight (already computed by the active weight function) as its first argument and can transform it — multiply, cap, override, or pass through:

CtorWeightModifier := Nat → Name → List Nat → DeriveSort → Nat → Bool → Nat → Nat → Nat → Nat → Nat
                      (baseWeight, ctorName, outputIndices, deriveSort, scoreBadness, isRec, size, numBase, numRec, numRecCalls) → finalWeight

Example — triple the weight for a preferred constructor, halve an expensive one, leave everything else alone:

open Scoring Schedules in
def myModifier (baseWeight : Nat) (ctorName : Name) (_outputIndices : List Nat)
    (_deriveSort : DeriveSort) (_scoreBadness : Nat) (_isRec : Bool)
    (_size : Nat) (_numBase _numRec : Nat) (_numRecCalls : Nat) : Nat :=
  if ctorName == ``MyType.PreferredCtor then baseWeight * 3
  else if ctorName == ``MyType.ExpensiveCtor then baseWeight / 2
  else baseWeight

set_option specimen.weightModifier "myModifier" in
derive_mutual
  (fun n => ∃ (xs : List Nat), SortedList n xs)

The modifier composes with whatever specimen.weightFn is active — you don't need to know or reimplement the base weight logic. This is the lightest-weight way to nudge the distribution for specific constructors.

6. specimen_test / specimen — property testing a statement

Everything above derives producers for a relation. specimen_test instead takes a whole proposition and looks for a counterexample, the way QuickChick's quickchick does. The statement is treated as a virtual constructor whose variables are all outputs: the hypotheses become a schedule that generates witnesses satisfying them, and the conclusion is checked against each witness.

-- As a command, on a closed proposition:
specimen_test (∀ n : Nat, Even n → Even (n + 2))

-- As a tactic, on the current goal:
example : ∀ n : Nat, Even n → Even (n + 2) := by
  specimen

The tactic closes the goal over its local hypotheses, tests it, and leaves the goal open — it is a diagnostic, not a proof. Derived generators and checkers live in the environment only for the duration of the test.

A counterexample is reported as an error naming each variable and its value, after shrinking:

error: Found counter-example!
  n : Nat := 0
(0 tests passed, 0 discarded, 1 shrinks)

Both forms accept an optional size configuration, in any order:

specimen_test (min := 5, max := 200, tests := 500) (∀ n, P n → Q n)
example : ∀ n, P n → Q n := by specimen (max := 20)

Generation size ramps linearly from min to max across the run, so the defaults min := 1, max := 100, tests := 100 try each size once. Shrinking is on by default and controlled by specimen.shrink, specimen.shrinkBreadth and specimen.shrinkDepth.

A hypothesis whose generator dead-ends is counted as a discard rather than a failure, so the reported total distinguishes "no counterexample found" from "could not generate valid inputs" — worth checking when a heavily-constrained property passes suspiciously fast.

Repo overview

Building & compiling:

• To compile, run lake build from the top-level repository.
• To run snapshot tests, run lake test.
• To run linter checks, run lake lint.
  • This invokes the linter provided via the Batteries library.

Typeclass definitions:

• ArbitrarySizedSuchThat.lean: The ArbitrarySuchThat & ArbitrarySizedSuchThat typeclasses for constrained generators, adapted from QuickChick
• DecOpt.lean: The DecOpt typeclass for partially decidable propositions, adapted from QuickChick
• Enumerators.lean: The Enum, EnumSized, EnumSuchThat, EnumSizedSuchThat typeclasses for constrained & unconstrained enumeration

Combinators for generators & enumerators:

• GeneratorCombinators.lean: Extra combinators for Plausible generators (e.g. analogs of the sized and frequency combinators from Haskell QuickCheck)
• EnumeratorCombinators.lean: Combinators over enumerators

Algorithm for deriving constrained producers & checkers (adapted from the QuickChick papers):

• UnificationMonad.lean: The unification monad described in Generating Good Generators
• DeriveConstrainedProducer.lean: Algorithm for deriving constrained generators, including the derive_mutual command for multi-spec mutual derivation
• MExp.lean: An intermediate representation for monadic expressions (MExp), used when compiling schedules to Lean code
• MakeConstrainedProducerInstance.lean: Auxiliary functions for creating instances of typeclasses for constrained producers (ArbitrarySuchThat, EnumSuchThat)
• DeriveChecker.lean: Deriver for automatically deriving checkers (instances of the DecOpt typeclass)
• Schedules.lean: Type definitions for generator schedules
• DeriveSchedules.lean: Algorithm for deriving generator schedules
• SearchTree.lean: Dependency-aware hypothesis ordering via lazy search tree with branch-and-bound pruning

Property testing a statement:

• Tactic.lean: The specimen_test command and specimen tactic — schedules a proposition as a virtual constructor, compiles a test loop, and shrinks counterexamples
• TheoremChecker.lean: Runtime helpers referenced by the code specimen_test emits

Schedule scoring & quality analysis:

• Score.lean: Type-erased score values used by the modular scoring framework
• Scoring.lean: Modular scoring framework with pluggable strategies (BudgetAwareScore, SourceQualityScore, the graded/density family, DefaultScore) for evaluating schedule quality, plus the constructor weight functions and modifiers
• PatternCoverage.lean: Pattern coverage trie that partitions the input space of an inductive relation, identifies weak spots, and annotates leaves with constructor coverage

Derivers for unconstrained producers:

• DeriveArbitrary.lean: Deriver for unconstrained generators (instances of the Arbitrary / ArbitrarySized typeclasses), including support for mutually recursive and parameterized types
• DeriveEnum.lean: Deriver for unconstrained enumerators (instances of the Enum / EnumSized typeclasses), including nested and mutually recursive types

Miscellany:

• TSyntaxCombinators.lean: Combinators over TSyntax for creating monadic do-blocks & other Lean expressions via metaprogramming
• LazyList.lean: Implementation of lazy lists (used for enumerators)
• LazyRoseTree.lean: Lazy rose tree data structure
• Idents.lean: Utilities for dealing with identifiers / producing fresh names
• Utils.lean: Other miscellaneous utils
• Debug.lean: Debug tracing and option flags for Specimen

Tests

Overview of test corpus:

• The SpecimenTest subdirectory contains snapshot tests (aka expect tests) for the derivation commands.
• Run lake test to check that the derived generators in SpecimenTest typecheck, and that the code for the derived generators match the expected output.
• Key test directories:
  • DeriveArbitrarySuchThat/ — constrained generators (BST, balanced tree, STLC, regex, permutations, multi-output, mutual recursion)
  • DeriveEnumSuchThat/ — constrained enumerators
  • DeriveDecOpt/ — checkers
  • DeriveArbitrary/ — unconstrained generators (parameterized types, mutually recursive types, structures)
  • DeriveEnum/ — unconstrained enumerators (nested recursion, mutual recursion)
  • CedarExample/ — real-world application: well-typed Cedar policy expression generators
  • ArithCompiler/ — end-to-end example: compiler correctness testing

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.