Testing Algorithms

This section presents a standardized methodology for testing algorithms using the TypedMatrices package.

Algorithm Development

Consider, for example, a simple algorithm that computes the sum of all elements in a matrix. This algorithm is implemented in the function sum_elements as follows:

julia> function sum_elements(A::AbstractMatrix)
           return sum(A)
       endsum_elements (generic function with 1 method)

Algorithm Testing

To test the algorithm, use the test_algorithm function. This function accepts the algorithm as its first argument and a vector of matrix sizes as its second argument. Additionally, it supports optional arguments to specify matrix properties and error handling behavior.

For example, to test the sum_elements algorithm on symmetric and positive definite matrices of sizes 1 through 4, with known eigenvalues and inverse, one can invoke the following:

julia> test_algorithm(
           sum_elements,
           [1, 2, 3, 4],
           props=[
               Property(:symmetric), Property(:posdef),
               Property(:eigen), Property(:inverse)
           ],
           errors_as_warnings=true
       )┌ Warning: Error running sum_elements on Poisson with size 2: ArgumentError("2 is not a perfect square integer")
└ @ TypedMatrices ~/work/TypedMatrices.jl/TypedMatrices.jl/src/interfaces.jl:54
┌ Warning: Error running sum_elements on Poisson with size 3: ArgumentError("3 is not a perfect square integer")
└ @ TypedMatrices ~/work/TypedMatrices.jl/TypedMatrices.jl/src/interfaces.jl:54
10-element Vector{Any}:
 (Minij, 1, 1)
 (Minij, 2, 5)
 (Minij, 3, 14)
 (Minij, 4, 30)
 (Pascal, 1, 1)
 (Pascal, 2, 5)
 (Pascal, 3, 19)
 (Pascal, 4, 69)
 (Poisson, 1, 4)
 (Poisson, 4, 8)

In this example, the first two arguments specify the target algorithm and the set of matrix sizes to test, respectively. The props keyword argument defines the properties of matrices. The errors_as_warnings flag determines whether runtime errors should be reported as warnings instead of causing test failure.

The function returns a vector of tuples, each containing three elements: the matrix type, the matrix size, and the corresponding output from the algorithm under test.

Performance and Memory Profiling

In many cases, it is desirable to capture performance metrics such as execution time and memory allocation alongside the algorithm’s output. To facilitate this, the algorithm can be modified accordingly:

julia> function sum_elements(A::AbstractMatrix)
           result = @timed sum(A)
           return result.value, result.time, result.bytes
       endsum_elements (generic function with 1 method)
julia> test_algorithm(
           sum_elements,
           [1, 2, 3, 4],
           props=[
               Property(:symmetric), Property(:posdef),
               Property(:eigen), Property(:inverse)
           ],
           errors_as_warnings=true
       )┌ Warning: Error running sum_elements on Poisson with size 2: ArgumentError("2 is not a perfect square integer")
└ @ TypedMatrices ~/work/TypedMatrices.jl/TypedMatrices.jl/src/interfaces.jl:54
┌ Warning: Error running sum_elements on Poisson with size 3: ArgumentError("3 is not a perfect square integer")
└ @ TypedMatrices ~/work/TypedMatrices.jl/TypedMatrices.jl/src/interfaces.jl:54
10-element Vector{Any}:
 (Minij, 1, (1, 9.498e-6, 160))
 (Minij, 2, (5, 2.465e-6, 448))
 (Minij, 3, (14, 2.364e-6, 928))
 (Minij, 4, (30, 3.807e-6, 1600))
 (Pascal, 1, (1, 9.629e-6, 160))
 (Pascal, 2, (5, 2.404e-6, 448))
 (Pascal, 3, (19, 2.805e-6, 928))
 (Pascal, 4, (69, 4.018e-6, 1600))
 (Poisson, 1, (4, 1.0671e-5, 160))
 (Poisson, 4, (8, 7.144e-6, 1600))