Add a performance disclaimer to the banner

[johnmyleswhite]

I sincerely believe that adding this disclaimer (or one like it) to the start of every interactive Julia session will substantially decrease the confusion that new Julia users experience. Obviously this is a last-resort kind of measure, but we've had the same performance problems being reported for years. I would argue that the lack of optimization in the global namespace is the single easiest thing newcomers can complain about it, since it makes Julia's fabled C-like performance seem like a lie.

@stevengj also noted on julia-users that we could add this behavior to toc, although I couldn't tell if he was being sarcastic or not.

[MikeInnes]

Have you considered adding a link to the performance tips instead?

I can see how this is sort of a special case, but even so I find the idea of seeing various bits of documentation / warnings every time I start Julia pretty unappealing, personally.

I'd much prefer to enable Lint.jl or TypeCheck.jl by default and receive warnings when they're appropriate (but maybe the banner is a necessary evil for the time being).

[johnmyleswhite]

I think no one will read anything that we link to from the REPL.

I agree this is pretty unappealing. I see this only as a stopgap and a very ugly one at that.

I like the idea of defaulting to linting and typechecking code, but that's going to make the entire REPL session much more verbose. This sort of one-off screen is likely to be ignored after the very first read. (One could even make this text only appear on the first REPL load per machine.)

[nalimilan]

Printing a warning from toc() may indeed be a good solution, as that's most often how people do benchmarks that are not inside functions (as opposed to @time).

[ivarne]

Can't say I love the new welcome message.

[eschnett]

Before learning Julia I didn't know what a REPL is. Reading this, I would have assumed there's a module called REPL that I shouldn't be using while benchmarking.

"Note that the interactive interface below is intended for exploration and prototyping.
To see the full benefit from Julia's optimizing JIT compiler, put your code into functions."

[StefanKarpinski]

@ivarne: The annoyingness of the new welcome message is a feature.

+1 to @eschnett's proposed wording.

[ViralBShah]

As much as I do not like this, it will certainly help improve the experience for a lot of newcomers and reduce mailing list traffic. A lot of people who get bit by this probably don't even report on the list and may put off trying out Julia.

[nalimilan]

There's still a problem with @eschnett's wording: it makes it sound like there's something inherently wrong with the interactive interface, i.e. that you need to run programs in non-interactive mode. Maybe say something like "Please note that the global namespace is only...".

[pao]

"Please note that the global namespace is only..."

What's this "global namespace" y'all keep talking about?

(We have to decide what jargon we can rely on, but given the target audience of this message, I don't know how much jargon we can realistically use here.)

[nalimilan]

Or just skip the first sentence.

[StefanKarpinski]

NOTE: if the performance sucks, wrap it in a function.

[ViralBShah]

I like the terse version, but rewording it:

NOTE: If you get poor performance with code at the prompt, wrap your code in a function.
Also see the Performance Tips section in the manual.

[stevengj]

I was serious about giving a warning for toc() called from the REPL. I think that this will be more effective than printing a warning at startup (which is susceptible to banner blindness), can be made to work in environments like IJulia as well, and will be less annoying to people who are not doing benchmarking. It won't affect people using @time or @elapsed (which call time_ns directly), but by the time people learn about those macros they usually know better than to benchmark REPL code (and this should be reflected in the documentation for the macros, too).

Better yet, print a warning (with depwarn) when toc() or toq() are called in any context, reminding the user not to benchmark code outside of a function or using globals, and suggesting @time or @elapsed; tic and toc are superfluous with our other timing functions, so they are more useful as a mechanism to catch naive benchmarks.

That being said, I have no objection to printing a warning in the REPL banner, but I think this should be in addition to a (permanent) depwarn in toq.

[ViralBShah]

I think it would be annoying if there was a warning from toc() every single time it was used in the REPL.

[MikeInnes]

Targeting messages specifically at people benchmarking Julia might give a negative impression – it comes off a bit PR conscious to me, even though it's a nicer solution in principle.

+1 for Stefan's wording in the banner. Talking about the REPL is misleading because just writing the same code in a file won't actually help (as one might expect coming from GHCi, for example).

[MikeInnes]

Or how about

NOTE: Using global variables in loops will be slow

Since that gets straight to the heart of the problem – global variables are actually OK, unless they'll be accessed multiple times.

[nalimilan]

global variables are actually OK, unless they'll be accessed multiple times.

They can also break type inference in subtle ways even if you don't access them directly in a loop.

[MikeInnes]

Hmm ok, fair point. Just Global variables may be slow? Stefan's wording may be a bit more intuitive after all.

[quinnj]

This also seems like a 0.3 commit to me. Those tracking master are probably aware of simple performance issues like this. Unless it makes it easier to maintain going forward, then I guess it doesn't matter that its on master.

[vtjnash]

perhaps this would be better suited to be added to the download page, and as a footnote on the speed comparison chart? (and linking to the performance tips page)

the proposed text just seems too off-putting for the casual user (too much jargon)

[ihnorton]

+1 for Viral's version: #9123 (comment)
Jargon-free and to-the-point.

[mlubin]

@StefanKarpinski, if the point is to annoy Jeff, maybe we can display this only after checking the current user ;)

@ihnorton, it's clear to us, but surely will lead someone to think that it's reasonable to do

N = 100000
function foo()
    for i in 1:N
       ...
    end
end

Also this is missing the point a bit, I think. I know we're all sensitive to perceptions about Julia's performance, but global scope isn't the only easily visible performance issue in Julia. If I were a windows user who just had to wait 10 seconds to load the REPL and another minute to include Gadfly, a note about performance in global scope would seem a bit out of touch to me.

[ihnorton]

PERFORMANCE HINT: avoid non-constant global variables, and wrap code in a function.
Please see the Performance Tips section in the manual for more information.

I guess the point is to get some basic info in front of the user and let them know that there is a Performance Tips section to look at.

[johnmyleswhite]

Honestly, my only goal is to make sure that everyone who tries Julia is told that Julia's performance has some gotchas, which we document in Performance Hints / FAQ.

[ViralBShah]

@isaiah 's version is the best so far. We could even have a bunch of hints and show a different one every time!

[mlubin]

Fortunes.jl?

[elextr]

Just print the whole manual, a paragraph at a time. :-)

[ViralBShah]

FWIW, I have added it to the banner on julia-users. I am trying to figure out how to add this to the welcome email too, but google groups is just too frustrating.

[StefanKarpinski]

I like @ihnorton's version (#9123 (comment)) except that I would change "HINT" to "NOTE":

PERFORMANCE NOTE: avoid non-constant global variables and wrap code in a function.
Please see the Performance Tips section in the manual for more information.

[ViralBShah]

And we can also provide the equivalent of Matlab's why. I loved it when it just said stuff like "Because Cleve told you so".

[ViralBShah]

Jokes aside, let's go with the latest modification of @isaiah 's version that @StefanKarpinski posted.

[stevengj]

@ViralBShah, we can easily set up the warning so that it is printed only the first time toc is used. That's what the once parameter of warn is for.

[ViralBShah]

We can do the warn once in toc when used interactively too, in addition to the banner and the mailing list.

[ghost]

This also seems like a 0.3 commit to me. Those tracking master are probably aware of simple performance issues like this. Unless it makes it easier to maintain going forward, then I guess it doesn't matter that its on master.

Actually, for master, considering the number of mails we get regarding package incompatibilities/breakages, we should most likely have something along the lines of:

NOTE: This version is in development, for everyday usage please use the latest release version.

[JeffBezanson]

I much prefer the idea of warning once when toc or time is used. The point about ijulia is good, and adding stuff like this to the banner is just ugly. I do think the warning is more likely to be read if it's part of output the user is looking for.

[Sisyphuss]

I have just discovered this interesting issue. I try to point out some neglected points here:

• Users who know the time macro may have known the gotcha of global variables. Most users having doubt on the speed of Julia actually count the time with a watch (ref. https://groups.google.com/forum/#!topic/julia-users/-bx9xIfsHHw).
• A user who cares about speed is most likely either to check the Performance Tip or to ask it in the mailing list or somewhere else. So let's suppose that most users will finally reach the Performance Tip.
• Performance Tip is not clear. It is full of jargon, without detailed explanation, and is much more poorly-written than most newbie Julia code. Even a new user has read it, he is not likely to write the correct code, since he doesn't get deep understanding.

The conclusion is: let's stop complaining about the complain in the mailing list and get to improve the quality of the Performance Tip in the manual.

[Sisyphuss]

To illustrate how poorly-written the Performance Tip is, I will argue it one paragraph by one paragraph.

A global variable might have its value, and therefore its type, change at any point. This makes it difficult for the compiler to optimize code using global variables. Variables should be local, or passed as arguments to functions, whenever possible.

It's OK. I just want to use global variables. I'll use it definitely. I am willing to sacrifice some speed. You guys are genius. Then show me the performance enhancement over Python. (You said "it makes it difficult for the compiler to ...", but you didn't say you will get a performance even worse than Python.)

Any code that is performance critical or being benchmarked should be inside a function.

Really, global variables degrade the performance enormously. It's horrible! It's cruel! Let's put everything into a block. (Now I am frightened, and go to the opposite extreme.)

We find that global names are frequently constants, and declaring them as such greatly improves performance:

You just told me to "avoid global variables", so I won't even try the const option. All codes go to some block (Actually not all blocks introduce a local scope)!

Uses of non-constant globals can be optimized by annotating their types at the point of use:

global x
y = f(x::Int + 1)

What the hell is x::Int + 1 here? I haven't mastered all usages of the :: operator!

Writing functions is better style. It leads to more reusable code and clarifies what steps are being done, and what their inputs and outputs are.

I don't need you to tell me that. I already know it. But what's the issue with the performance?

NOTE: All code in the REPL is evaluated in global scope, so a variable defined and assigned at toplevel will be a global variable.

What the hell is "REPL"? Not in "toplevel" right? Let's put all code in a module, now it's in the module level!

[johnmyleswhite]

@Sisyphuss: The rhetorical strategy you're employing does not have a high success rate.

[jiahao]

@Sisyphuss if you can find places where the documentation can be improved, please prepare a pull request with your suggested changes instead of taking a closed issue in a different direction.

[quinnj]

@Sisyphuss, I'm not quite sure if you're directing your comments at someone in particular or to the core developers specifically, but your tone is a little over-the-top. Much more productive, especially in open-source, is to funnel passion into actually making changes and submitting proposals to improve things.

For documentation, GitHub makes it particularly easy to submit a PR to improve a single file (i.e. for Performance Tips, you just need to edit here)

[tkelman]

Thank you @Sisyphuss for opening #11077, which is a much more constructive way to improve matters. Could use a little bit of copy-editing.