property-based-testing

Property-Based Testing

RULES — Follow these with no exceptions

1. Define properties, not examples — state what should always be true, not specific inputs/outputs 2. Test invariants, not specific values — "output length equals input length" not "output is [1, 2, 3]" 3. Leverage shrinking — let StreamData find minimal failing cases; add constraints like min_length: 1 or integer(1..100) when shrunk cases expose invalid generator inputs

Setup

# mix.exs
defp deps do
  [
    {:stream_data, "~> 1.0", only: :test}
  ]
end

Basic Property Test

defmodule MyApp.StringUtilsTest do
  use ExUnit.Case, async: true
  use ExUnitProperties

  property "reversing a string twice returns the original" do
    check all string <- string(:ascii) do
      assert string |> String.reverse() |> String.reverse() == string
    end
  end

  property "length of reversed string equals original length" do
    check all string <- string(:ascii) do
      assert String.length(String.reverse(string)) == String.length(string)
    end
  end
end

Generators

Less Obvious Built-ins

# Combine multiple generators with one_of
one_of([constant(:a), constant(:b), constant(:c)])
one_of([integer(), string(:ascii), boolean()])

# Compose generators with gen all
gen all x <- integer(0..100),
        y <- integer(0..100),
        x != y do
  {x, y}
end

# Non-empty collections
list_of(integer(), min_length: 1)
map_of(string(:alphanumeric), integer())

Custom Generators

# Generate a valid email
def email_generator do
  gen all name <- string(:alphanumeric, min_length: 3),
          domain <- string(:alphanumeric, min_length: 3) do
    "#{name}@#{domain}.com"
  end
end

# Generate a user struct
def user_generator do
  gen all email <- email_generator(),
          age <- integer(18..120) do
    %User{email: email, age: age}
  end
end

# Use in tests
property "users have valid emails" do
  check all user <- user_generator() do
    assert user.email =~ ~r/@.*\.com$/
    assert user.age >= 18 and user.age <= 120
  end
end

Testing Invariants

Key invariants to test for collections: ordering, element preservation, and idempotence.

defmodule MyApp.SortingTest do
  use ExUnit.Case, async: true
  use ExUnitProperties

  # Invariant: sorted list is ordered
  property "sorted list is ordered" do
    check all list <- list_of(integer()) do
      sorted = Enum.sort(list)

      sorted
      |> Enum.chunk_every(2, 1, :discard)
      |> Enum.each(fn [a, b] -> assert a <= b end)
    end
  end

  # Invariant: sorting preserves all elements
  property "sorted list contains same elements" do
    check all list <- list_of(integer()) do
      assert Enum.sort(list) |> Enum.frequencies() == Enum.frequencies(list)
    end
  end
end

Shrinking

StreamData automatically shrinks failing inputs to the smallest example that still fails. The property below intentionally fails to illustrate shrinking output:

property "no list contains its own length as element" do
  check all list <- list_of(integer()) do
    refute list |> Enum.member?(length(list))
  end
end

# StreamData will find and report:
# Failed with generated values: [0]
# (shrunk from something like [1, 5, 0, 3, 2])

Workflow: Write → Run → Interpret → Refine

1. Write property — define an invariant using check all with appropriate generators
2. Run test — mix test test/my_test.exs
3. Interpret shrinking output — on failure, StreamData reports both the original failing input and the shrunk minimal example:

   ** (ExUnit.AssertionError)
   Failed with generated values (after 3 successful runs):
   
       * Clause:    list <- list_of(integer())
         Generated: [0]
         (shrunk from [42, -7, 0, 15])
   

4. Refine generator or property — if the shrunk case reveals a generator producing invalid inputs, add constraints (e.g. min_length: 1, integer(1..100)); if it reveals a real bug, fix the implementation
5. Re-run — confirm the fix holds across new generated cases

Common Pitfalls

❌ Don't	✅ Do
Assert exact outputs (== [1, 2, 3])	Assert invariants that always hold (ordering, length, membership)
Generate unconstrained inputs that break the code under test	Constrain generators (min_length: 1, integer(1..100)) to valid domains
Ignore the shrunk minimal case in failure output	Read the shrunk values first — they point straight at the bug
Use check all without use ExUnitProperties	Add use ExUnitProperties alongside use ExUnit.Case
Write one giant property covering many behaviours	Split into focused properties, one invariant each
Leave generators unbounded, causing slow/huge cases	Bound ranges and collection sizes to keep runs fast

---

Integration

Predecessor	This Skill	Successor
testing-essentials	property-based-testing	None (standalone)