Uses the Convex.jl Problem Depot to test various optimization solvers. See the sidebar for the solvers test results.

Some of the problems are fairly strange (e.g. only one feasible point), which can be difficult for solvers to handle.

Note: for some solvers, the problem constant_Issue_166 shows up as taking a long time. That is likely because it is the first problem tested and may include compilation time. To counteract that effect, a simple problem is solved prior to starting the tests, but it may not exercise all the compilation pathways.

A note on one of the problems

The problem sdp_Complex_Semidefinite_constraint is excluded from some of the solvers because they become very slow. I believe the difficulty is with solvers which do not natively support SOCP constraints and instead bridge via SDP; since it is larger than most of the other problems (with a 10 by 10 PSD variable), after the bridges it is just too large to solve quickly.

The problem is the following, copied from the Problem Depot source:

n = 10
A = rand(n,n) + im*rand(n,n)
A = A + A' # now A is hermitian
x = ComplexVariable(n,n)
objective = sumsquares(A - x)
c1 = x in :SDP
p = minimize(objective, c1; numeric_type = T)

# test that X is approximately equal to posA:
l,v = eigen(A)
posA = v*Diagonal(max.(l,0))*v'

if test
    real_diff = real.(x.value) - real.(posA)
    imag_diff = imag.(x.value) - imag.(posA)
    @test real_diff ≈ zeros(n, n) atol=atol rtol=rtol
    @test imag_diff ≈ zeros(n, n) atol=atol rtol=rtol

Running these tests directly

The problems tested here can easily be added to the test suite of an (MOI-compatible) optimization solver. For example, to run all the tests with the solver SCS, simply run

using Convex
using Convex.ProblemDepot: run_tests
@testset "Convex Problem Depot tests" begin
    run_tests(;  exclude=[r"mip"]) do problem
        solve!(problem, () -> SCS.Optimizer(verbose=0, eps = 1e-6))

The actual implementation used here is slightly more complicated in order to get timing information (via TimerOutputs.jl) as well as record the tests in a way that does not throw an error and can generate the HTML tables shown in the pages in the sidebar (via the unregistered package TableTestSets.jl). See the source code for details.