[ENHANCEMENT] Wrap runtests.jl in @testset (or @safetestset)

[mtfishman]

Right now, runtests.jl loops over a number of @safetestsets: https://github.com/ITensor/ITensorPkgSkeleton.jl/blob/07abef5da88b498a88cab1ffe2d48bb25f248304/templates/test/test/runtests.jl. It seems like that should all be wrapped in a call to @testset so all of the results get collated into a single test set, but maybe that doesn't play nice with @safetestset?

[mtfishman]

Replacing @time here:

ITensorPkgSkeleton.jl/templates/test/test/runtests.jl

Line 22 in 07abef5

@time begin

with @testset seems to work to collate all the results into a single test set, and also it reports the time so @time may not be needed (though I think you mentioned you wanted to use @time since it reports extra information like compilation time).

Additionally, there are some globals defined here:

ITensorPkgSkeleton.jl/templates/test/test/runtests.jl

Lines 6 to 20 in 07abef5

	const pat = r"(?:--group=)(\w+)"
	arg_id = findfirst(contains(pat), ARGS)
	const GROUP = uppercase(
	if isnothing(arg_id)
	get(ENV, "GROUP", "ALL")
	else
	only(match(pat, ARGS[arg_id]).captures)
	end,
	)
	function istestfile(filename)
	return isfile(filename) &&
	endswith(filename, ".jl") &&
	startswith(basename(filename), "test")
	end

which make me a little uncomfortable since I like to run include("runtests.jl") as a quick way to test things and I would prefer to not add things to the global scope unnecessarily, maybe we can just wrap all of that in @testset to minimize the number of global variables. I know I could do something like module M include("runtests.jl") end to be safe but it would be nice not to have to.

[mtfishman]

After discussing with @lkdvos we won't do this, I think basically each file should be agnostic about how you decide to run in, what environment it is run in, etc. You can always include the test files in a "safe" way (i.e. isolated from other code) by calling one of these:

julia> module M include("runtests.jl") end

julia> @eval module $(gensym()) include("runtests.jl") end

julia> @safetestset begin include("runtests.jl") end

julia> @testset begin include("runtests.jl") end

As an alternative, I'm defining:

macro safeinclude(file)
  return quote
    @eval module $(gensym())
    const ans = include($file)
    end
  end
end

in my ~/.julia/config/startup.jl, so you can include tests with:

julia> @safeinclude("runtests.jl")

and additionally you can access the test set with:

julia> testset = @safeinclude("runtests.jl").ans