try new option enzyme_nocache

[tthsqe12]

Initial versions of Enzyme had an option for globally nuking the caching mechanism (for extra performance, I suppose). Since this is clearly not very useful, I propose adding an option to allow the user to control this on a per-argument basis.
What does "nocache" mean?
The definition is not precise, and any user would have to try their code with it and see if it still works. The basic idea is that it is a promise between the __enzyme_augmentfwd and __enzyme_reverse that the given argument will not be modified, hence much of the caching can be skipped.
What does it mean "argument will not be modified"?
This is also not precisely defined, but in general it means that all loads emanating from the given argument can skip caching. The implementation in compute_nocache is quite flexible and allows for easy tweaking of the handling of specific instruction. For example, I currently said that all of the argument of a phi must be 'nocache' in order for the phi itself to be 'nocache'. This allows the linked list examples to work (they still get some caching because this is the only way they can work).

@wsmoses Please let me know if you think this kind of performance-minded option is viable in enzyme.

[wsmoses]

So the definition of the promise you gave (that it is not modified between forward and reverse) is equivalent to that of overwritten_args.

The only caveat is that the overwritten_args caveat only applies to direct loads (aka loads from that pointer or offsets of it), but makes no promises about loading from a loaded pointer.

So I suppose I'm wondering what your intended semantics are, and how they differ from overwritten_args?

[wsmoses]

This wouldn't work for recursive phi nodes, if that's desired.

E.g. phi = [ nocachearg if entry] else [ phi+1 if not entry ].

We could set up a map of values to states, initialized to unknown and build something to try to support this.