RFC: remove invoke

[JeffBezanson]

I have come around to the view that invoke should be removed from the language. I wrote up this patch to explore what this change would look like in Base.

invoke can be seen as an abstraction violation: to use it, you need to know not just which function to call but what argument types it declares. Argument types are much more of an implementation detail, especially in julia. For example, one will often start with f(x::Any), then later realize this can be narrowed to f(x::Real). Doing this can break invoke call sites, even if all call sites actually pass Reals.

invoke can be seen as an overly elaborate form of renaming (much like static overloading). You want to call a particular method, but to do it you have to write down its signature. Arguably, an isolated method that people want to call directly should just have its own name. For example, I often changed call sites as:

-        invoke(write, Tuple{IO, Array}, s, a)
+        write_each(s, a)

write_each is not a great name, but despite that the new code is way shorter and easier to write.

In one case (BitArray cat) I was able to eliminate invoke just by adding another 1-line method, which I think is a better solution.

In still other cases the "abstraction" of invoke didn't seem too necessary at all, for example

-        invoke(println, Tuple{IO, map(typeof,xs)...}, io, xs...)
+        for x in xs print(io, x) end
+        print(io, '\n')

Here the idea of invoke was "acquire a lock, then do whatever else println was going to do". But it better not do anything except print all arguments followed by a newline! Ideally, yes you'd rather write a function call than duplicate this for loop, but on the other hand this clarifies why we're acquiring a lock (because there are multiple I/O operations inside).

So I think we're better off without this. Obviously the code deletion and simplification of the language are also significant benefits. invoke is used in packages, but from what I can tell not an overwhelming number of times. Hopefully authors will weigh in.

[johnmyleswhite]

+1

[StefanKarpinski]

I've never liked invoke, so +1, although I worry somewhat about the lack of escape hatch.

[simonbyrne]

This seems reasonable: invoke was never really sufficiently flexible for cases when I have tried to use it.

For future reference: we can then also close #7045.

[mauro3]

How would I do this after this PR:

function Base.show{T<:MyType}(io::IO, a::Type{T}) 
    println("This is one of my types:")
    invoke(show, Tuple{IO, DataType}, io, a)
end

[JeffBezanson]

@mauro3 I don't have a perfect answer, but there are a couple options:

1. Don't do that.
2. Renaming. A case like this already occurs in base/show.jl, and we have a separate show_default.
3. A better system for handling different levels of printing. In order of increasing "decoration" we have print, show, display. In some cases like yours it makes sense to define display and call show in the implementation, instead of defining show.

[mauro3]

It certainly is a common OO pattern to build upon the methods which are defined for the supertype which is what I do in that example. Maybe something like this could support that:

function Base.show{T<:MyType}(io::IO, a::Type{T}) 
    println(io, "This is one of my types:")
    show(io, superfy(a))
end

[JeffBezanson]

invoke is used suspiciously rarely, for something that is indeed such a common pattern in class-based languages.

I don't see how superfy can work --- what does it return?

[mauro3]

I guess superfy(a) would be the same as a but with the supertype as type-tag. I guess that can't be done, it was more thought to convey an idea.

[mauro3]

That was a bit a stupid comment of mine as my proposition really is the same thing as invoke but different syntax.

Anyway, this could be used instead of invoke:

julia> function Base.show(io::IO, b::B)
           which(show, Tuple{IO, super(B)}).func(io,b)
           println(io, "and B")
       end

and may could be @StefanKarpinski's escape hatch? Is there any difference to invoke?

[JeffBezanson]

That isn't type safe; nothing will check that (io,b) matches the given Tuple type. It also won't be optimized, and depends on internal data structures that will change.

[mauro3]

Thanks for the clarification!

I vote for keeping invoke, Julia doesn't usually forbid to do something, even though it may not be clever. Maybe just stop exporting it and document it accordingly.

[JeffBezanson]

If we're going to keep it, some interface like superfy would probably be better; basically something that expands to invoke(f, Tuple{typeof(x), super(typeof(y))}, x, y). That could be a macro. Or possibly call-next-method, which calls the method that would be picked if the current one didn't exist. I still find these features icky though.

[yuyichao]

The syntax I used for this was @invoke f(x, y::SuperType, z) (which expands to invoke(f, Tuple{typeof(x), SuperType, typeof(z)}, x, y::SuperType, z)). The nice thing about this syntax is that it doesn't change the meaning of ::SuperType by too much.

[andrewcooke]

i was pointed here after asking for the equivalent of calling a super-method. i would appreciated a call-next-method or similar.

i honestly don't know why this idiom is not common in julia. guesses include: julia being largely numerical computing (so many types are simple values); multiple dispatch helping decompose "gnarly" complex types into simpler constituents; lack of libraries that rely on inheritance because only leaf types are concrete.

[wbhart]

We just spent all day looking for a workaround for the obvious problem of how to call a more general implementation from within a more specific implementation of a function (in particular across packages/modules), and everything we tried had either been removed from Julia or doesn't work. We were finally happy when we discovered invoke, and now that too is being removed!

Why would you take something like this out of the language? Please, by all means, mark it "this is very slow and should not be used unless absolutely necessary" in the documentation. But please, please reconsider removing things from the language before an equally capable replacement has been developed. Without invoke, we don't have any way of doing something that should be trivial.

Please see the thread on julia-users that I started earlier today about this. We need this function.

[wbhart]

By the way, the argument Jeff makes about it being an abstraction violation doesn't persuade me. Changing the types for the arguments of a function could also introduce ambiguities that didn't otherwise exist (actually this happens in practice to us). But we don't then remove Julia's sophisticated abstract type system just because this can happen.

Besides, if I change the type of the function being called by invoke, the worst that will happen is that the test code (or user code) will break at precisely the point where invoke is used, telling me there is no such function. That seems perfectly reasonable to me.

[JeffBezanson]

Feedback noted! The way this is going, it does not look like we'll end up removing invoke any time soon at least. How do you feel about some of the other options discussed in this thread, e.g. call_next_method? Would that work in your case?

[wbhart]

Jeff, I am not sure about call_next_method. For what it is worth, the Gap computer algebra system has a mechanism for a function to silently fail and pass to the next most highly ranked method in the method table (users can specify their own personal ranking of methods in GAP). And people swear by it. However, there are also people wanting to move towards a kind of dispatch model based on computed "rules", where a method is called based on the precomputed cost of calling that method vs some alternative. So even that system hasn't worked out perfectly for them. A CAS called Polymake has such a dispatch mechanism. But now people want something that combines both capabilities and computes the best "strategy", based on what has already been computed and cached, so that the cost can change at runtime, etc. (E.g. there is no point in ranking a particular strategy lower because it relies on a high cost determinant computation if that determinant was already previously computed and cached). In other words, the system can never be too flexible for computer algebra.

On the other hand, computer algebra systems are unusual, in that they are populated with "types" that depend on runtime values. For example, Z/nZ is a field if and only if n is prime. And Z/nZ depends on the value of n anyway. In such cases we overload call (or () or whatever it is now called) in Julia so that certain objects can behave like types. e.g. for the "type" Z/nZ.

There are also other dispatch scenarios we are thinking about, for example calling a special implementation of an algorithm for GCD domains, which are basically integral domains for which a gcd function is available. For this we hope traits will eventually be the solution in Julia.

In short, it is not trivial to see what we gain/lose in the long run from any given decision in Julia. We basically see it as a serious research project to push Julia as far as we can given what is available. Mainly we are concerned about existing functionality being taken away after we have come to rely on it, when there is no alternative provided.

Believe it or not, there are people seriously trying to rewrite Gap in Julia. And they just might succeed. Performance of all the dynamic clunking around that they need to do for dispatch is not an issue if it is the difference between an algorithm that will see the universe die of heat death before it finishes and one that takes 10 minutes. But the advantage for them is to also benefit from fast Julia dispatch for mid-level algorithms that interpreted computer algebra systems currently can't ever make fast (plus integration with the growing Julia ecosystem).

But getting back to your question. I am happier about call_next_method than about removing invoke altogether. I can only comment on how that would affect us right now.

I think this would adequately solve the problem we have calling across modules. Basically Nemo is to become a kind of central module containing generic implementations and specialisations provided by various C libraries. On top of Nemo will be many packages, for number theory, group theory, computer algebra, etc., which "plug into" the abstract types provided by Nemo. For free they get the generic implementation in Nemo, if they implement some very basic functionality. But they can then overload this with more specific functionality for their own types. Chances are, the "next_method" after their own implementation fails is the one Nemo provides, and so this should be fine.

My concern that remains is actually what happens within a single module, e.g. Nemo itself. The main issue is when over some range a certain implementation is faster than the generic method, but the generic method wins elsewhere (this happens often, and prompted today's difficulties).

Some examples of this include classical multiplication of polynomials vs say kronecker segmentation. For sparse polynomials, the classical algorithm will be faster. Otherwise KS will be faster, usually.

We currently have the following implemented in our polynomial code:

   function *{T <: RingElem}(a::PolyElem{T}, b::PolyElem{T})
      return mul_classical(a, b)
   end

This is simply because for the types that define * for polynomials over certain rings for which kronecker segmentation is not faster everywhere, we sometimes want to call the classical implementation (which used to just be called *). The only solution was to rename the * function to mul_classical and introduce this rather silly function to call it in the more general case, all so that mul_ks can call mul_classical when it needs to. And then * for those types can first call mul_ks, etc.

The function invoke solves this problem for us. The * function for those types can fallback to the more general * method (which implements classical multiplication). But the proposed call_next_method might actually call the wrong version. Technically we could probably work around this by having the "wrong version" also call call_next_method, so that it passes through again to the next most general method, etc., until the right method is finally encountered.

This sort of thing actually happens in our matrix code right now (though we solved it with renaming, because we didn't know about invoke). Consider for example the function solve for solving matrix equations Ax = B. There can be implementations of this for rings where coefficient blowup can occur (such as over the integers), and those where it can't (e.g. over Z/nZ), methods based on interpolation when working over polynomial rings and special versions of all these functions for the case where you are working over a field, not a ring.

Currently all those cases can be distinguished by (abstract and/or parameterised) type alone in Nemo. But often we have to dispatch to a more general implementation if one method fails.

I would be worried that it would be difficult to know which is the next most general method once we get implementations of solve for many more parameterised/abstract types.

So sometimes we just need finer control. The less flexible Julia becomes in this regard, the more likely we end up having to introduce functions everywhere whose only job is to look at the types of the arguments at runtime and dispatch to the right version of the function based on their types. This is the situation already with python.

In summary, I don't know for sure. I think the proposal would be less flexible for us. But if you decide to do it that way, I think we might be able to live with it, through a combination of renaming and adding additional short functions such as the one I gave above.

Sorry for the longwinded answer. I just wanted to illustrate why it's not an easy question to answer definitively. I think we can probably survive if you do go down that route. I will ask my colleagues about it too over the next couple of days, and post a followup if they raise objections I haven't thought of.

[mauro3]

Some related discussion: #17168 (comment)

[ccoffrin]

In my experience, being able to specifically call a more abstract method is quite valuable. A key use case is when a more specific method is "extending" the functions of the more abstract one.

I agree that invoke is maybe too general, an would be in favor of something along the lines of call_next. My only caveat would be to consider making call_next nest-able, so that you can go several steps up the hierarchy. I personally would never do this, but it is possible to do in OO super calls, so someone else might have a compelling case.

[StefanKarpinski]

I think the current solution, namely, to have invoke but discourage it, is fine. The main observation to be made from @JeffBezanson's RFC is that we can generally avoid invoke, but there's really nothing so awful about it. I don't like the callnext idea since in the multiple dispatch setting, it's completely unclear which less specific method is the next one, e.g.:

f(x::Int, y::Number) = 1
f(x::Number, y::Int) = 2
f(x::Int, y::Int) = callnext(f, (x,y)) # does this call method 1 or 2?
f(x::Int, y::Int) = invoke(f, (Int,Number), x, y) # clearly calls method 1

We could have some syntactic sugar for this in the form of an @invoke macro, e.g.:

f(x::Int, y::Int) = @invoke f(x, y::Number)

The macro could desugar to something like this:

invoke(f, (typeof(x), Number), x, y::Number)

I.e. use the actual type of arguments that don't have type annotations and add type asserts to arguments that do have type annotations, as the syntax would suggest (which also would prevent invoking methods that don't apply in the first place).

[vtjnash]

it's completely unclear which less specific method is the next one, e.g.:

that's easy: callnext would throw a method ambiguity error

[StefanKarpinski]

Here's a potential implementation of @invoke:

macro invoke(ex)
    ex.head == :call || error("@invoke requires a call expression")
    args = ex.args[2:end]
    types = map(args) do arg
        isa(arg,Expr) && arg.head == :(::) ? esc(arg.args[end]) : :(typeof($(esc(arg))))
    end
    :(invoke($(esc(ex.args[1])), ($(types...),), $(map(esc,args)...)))
end

Used like so:

f(::Int, ::Number) = 1
f(::Number, ::Int) = 2
f(::Number, ::Number) = 3

julia> @invoke f(1,2::Number)
1

julia> @invoke f(1::Number,2)
2

julia> @invoke f(1::Number,2::Number)
3

@vtjnash:

it's completely unclear which less specific method is the next one, e.g.:

that's easy: callnext would throw a method ambiguity error

If we remove invoke and only have callnext then there's no way to call a specific next method in ambiguous cases, so sure, you can do this but it doesn't remove the need for invoke.

[StefanKarpinski]

If the @invoke macro above is popular, I'll make a PR for it (it seems much easier to use than the bare invoke function to me); I couldn't find any examples after a little searching of what kind of error to raise when the macro "argument" doesn't have the right form – any suggestions?

[yuyichao]

Note that you need to hoist the evaluation of the arguments and make sure they are evaluated in a intuitive order. The version of @invoke I had also handles kwargs with a hack but I guess that can way after invoke itself supports it (or it can possibly be done with more hack in the macro too).

[StefanKarpinski]

I'll see if there's interest before trying to fix it. @yuyichao: where's your version of @invoke?

[tkelman]

I don't see what the macro gets you over using the function.

[simonbyrne]

Well, the functionality is the same, it's just easier to use.

[tkelman]

It's a bit of a syntax pun on the type assert, to make something that isn't needed all that often slightly more concise.

[simonbyrne]

To be honest, I've often considered the same sort of thing for ccall as well, i.e.

@ccall libm.sin(x::Float64)::Float64

[tkelman]

I was also tempted to comment that I liked the correspondence between ccall syntax and invoke syntax. A @ccall macro might be harder to swing because of the special way/time ccall's library argument needs to be handled, but I would use that way more often than an @invoke macro.

[yuyichao]

The types in invoke is pretty different from the types in ccall. The types in ccall are convert where as the types in invoke are/include typeassert.

As far as syntax correspondence goes, ccall should correspond to the syntax of cfunction (which it almost does). If anything, invoke should correspond to the syntax of function definition, which is almost what the @invoke macro provide.

[StefanKarpinski]

Also, in a ccall/cfunction you must give the type of every argument. For invoke there is often only one argument that you need to "widen" the type of, while allowing the other arguments to be dispatched on as is.

[StefanKarpinski]

See this SO question: http://stackoverflow.com/questions/39102198/is-it-possible-to-call-an-overloaded-function-from-overwriting-function-in-julia. The fact that someone asking this question very nearly used the exact syntax that I proposed for the @invoke macro, indicates to me that it's a pretty intuitive syntax – much more so than the current interface to invoke. The name @invoke could probably be significantly improved, however – @callnext? @super?

[simonbyrne]

@callsuper?

[Sacha0]

@callspecific`@callmethod\@callsignature, @specifymeth[od]\@specifysig[nature]\@specifycall, @withmeth[od]\@withsig[nature], @specificmeth[od]\@specificsig[nature]`, or something similar reflecting the explicit (partial) specification of a signature / method?

[StefanKarpinski]

I like @callsuper; maybe@dispatch? Probably too broad.

[StefanKarpinski]

The person asking the question on SO likes @callsuper. I think that's the best so far.

[StefanKarpinski]

Here's a version with the callsuper name that preserves evaluation order (afaict):

macro callsuper(ex)
    ex.head == :call || error("@invoke requires a call expression")
    args = ex.args[2:end]
    types = Symbol[]
    vals = Symbol[]
    blk = quote end
    for arg in args
            val = gensym()
            typ = gensym()
            push!(vals, val)
            push!(types, typ)
            if isa(arg,Expr) && arg.head == :(::) && length(arg.args) == 2
                push!(blk.args, :($typ = $(esc(arg.args[2]))))
                push!(blk.args, :($val = $(esc(arg.args[1]))::$typ))
            else
                push!(blk.args, :($val = $(esc(arg))))
                push!(blk.args, :($typ = typeof($val)))
            end
    end
    push!(blk.args, :(invoke($(esc(ex.args[1])), ($(types...),), $(vals...))))
    return blk
end

This uses gensym a fair bit, maybe emitting a let block with locals would be better.

[simonbyrne]

Can we incorporate @yuyichao's keyword support: #7045 (comment)

[yuyichao]

The kw support was mainly an earlier version of #11165 (combined with some simple ask shuffling to make sure kwcall is correctly handled by the macro). If we want to make @invoke (or whatever we name it) as the primary interface, we can directly that support in the macro. This is very similar to #11165 in that it adds kw support by creating a wrapper around the same simple primitive.

This also doesn't have to be in the first version of the macro given that the invoke function doesn't support kwargs yet.

[StefanKarpinski]

Let's move the conversation about @callsuper to #18252.

[marius311]

Was pointed to this thread after asking https://discourse.julialang.org/t/how-to-invoke-next-most-specific-method/7664

If you read the question, you'll see the call-next-method that was discussed above is really exactly what I was looking for. Although in many cases you can get around needing this (and quite likely in many cases you'd be better off not using it), in my case it may be necessary. What I'm trying to do is to extend loop fusion so that x .+ y .+ z does some extra stuff then calls the generic broadcast that would have been called. In this case, I need the loop fusion magic to create the anonymous function for me, I can't just call my function something other than broadcast.