Repl config improvements & cntl-J force-submit

[luketpeterson]

Separating repl configs from MeTTa environment configs.
Using Pragmas for repl configs
Loosening pragma value type checking
#425

Adding cntl-J "force-submit" key, and changing behavior of enter key when cursor is in the middle of a line

[vsbogd]

Luke I feel that I was not very specific in when I was talking about unification in #425. Or may be you have plan to modify it further. What I mean is that we have two kinds of configuration (1) execution config (type checking in interpreter and similar) and (2) REPL config (color scheme and similar). In 325b8c0 you splitted them on two files which is good. But also you converted (2) into set of (pragma! <setting> <value>) settings which is not I would expect. Now we have both (1) and (2) in the same hash map inside MeTTa runner and I actually don't think we need to mix them in a such way.

As was proposed in #425 I think it is better to represent (2) as a set of (= ...) equality statements, keep them in separate atomspace and query using interpreter. At the same time we could keep (1) as is for a while. But as Alexey said more MeTTa way is to represent it as an atomspace as well. Thus the only question about this PR is that what is your plan regarding this? Is this PR just a first step or you consider it more as a final state?

[vsbogd]

JIC except the comment above other changes looks good to me.

[luketpeterson]

Regarding the pragma vs. atom-in-space question, I had a change of heart when I was working with it mainly because a pragma assignment is a simple exclusive key-value binding, where a (= key value) expression atom is fundamentally not exclusive.

Specifically I was experimenting with setting the REPL prompt to the number of atoms in the space using MeTTa at runtime, and realizing I either need to clean up the old expression first or I might get the old value when I match (since I was just blindly taking the first match.)

So on the atom vs. pragma question: I feel like pragma might be a better fit with what the app wants / needs. Unless there is a more elegant expression in MeTTa with built-in exclusivity. Like the StateAtoms from OpenCogClassic. What do you think?

On the segregation between runner (init) and repl config question: do you think it's necessary? We already have a pretty sparse namespace given the REPL params begin with "Repl". I was thinking it makes a lot of sense to keep the MeTTa object's settings as a big bag of configs and use naming conventions to keep them separate. Also, the repl.metta file should never run unless the app enters interactive mode.

One slight addendum to the "bag of settings" might be a "tree of settings". This is how many complex apps handle things (Google Chrome, Visual Studio Code, etc.), where each setting has a path from a settings root, and different extensions and features are responsible for the settings under different nodes.

[vsbogd]

So on the atom vs. pragma question: I feel like pragma might be a better fit with what the app wants / needs. Unless there is a more elegant expression in MeTTa with built-in exclusivity. Like the StateAtoms from OpenCogClassic. What do you think?

You probably mean that

(= var A)
(= var B)

in fact means var can be A or B while

!(pragma! var A)
!(pragma! var B)

means that var is set to B finally? Actually we have State in MeTTa stdlib as well (see ...StateOp in stdlib.rs). I actually don't bother about non-determinism here. I would just leave assert in code that configuration should have the only value per parameter. On the other hand (= ...) format allows us using functions to infer values.

On the segregation between runner (init) and repl config question: do you think it's necessary?

To me it is a question of a clear border between MeTTa runner and MeTTa REPL. I think they should be decoupled but keeping MeTTa REPL configuration inside MeTTa runner makes them coupled. It is really close to the previous question with pragma!. pragma! was introduced as a way to tune way of code execution. From this perspective I would also add a check that only specific set of properties can be set by pragma!, reject all other keys and remove universal get/set functions from the runner. Btw maybe @Necr0x0Der would not agree with me.

REPL is a separate project from the engine it has its own settings like color scheme. It can keep it and modify it absolutely independently from MeTTa runner. To me it is like using MeTTa runner as a HashMap while it is not a HashMap. In other words I would keep in MeTTa runner only settings of MeTTa runner.

[luketpeterson]

You probably mean... that var is set to B finally?

Yes, exactly. The config in practice has only one state, so having the possibility for a nondeterministic value in this case is simply a bug.

Actually we have State in MeTTa stdlib as well

I made an attempt at implementing the configs with State atoms, but I found it to be unwieldy (not impossible, but ugly) on account of the fact that an ordinary atom ends up shadowing the StateAtom so it's hard to refer to the StateAtom in MeTTa code, ie to call get-state or change-state!, unless you first use a query or have it bound to a variable some other way.

For example, one way I kludged it together was like this:

!(bind! State1 (new-state "one"))
(= ReplKey State1)
!(match &self (= ReplKey $state) (get-state $state))
!(match &self (= ReplKey $state) (change-state! $state "two"))

But this seems like it gives the user a lot of opportunities to make a mistake. Maybe I'm missing the straightforward way to use the StateAtoms. What do you think?

On the other hand (= ...) format allows us using functions to infer values.

I don't exactly understand what flexibility is available in the declaration that isn't available in execution. For example, this works exactly like you would expect:

(= (SumNumber) (+ 50 40))
!(pragma! ReplCommentStyle (SumNumber))

Are you saying it postpones the resolution of the value from the setup time to the query time? The reason I feel this is a double-edged sword is that I end up caching many of the values to avoid a bunch of MeTTa evaluation in the background. And so I'd end up only resolving them once in some cases. On the other hand, with the execution approach (like pragma!) the user can "push" a new value into the state and thus it will become active immediately.

If we create a dependency push mechanism in the future then maybe it'll be workable. (ie. a way we can register for updates on changes in a space that affect the way a specific atom will evaluate) But that sounds like a Big job that might not be within the design vision for MeTTa at all.

pragma! was introduced as a way to tune way of code execution...

If we decide a "simple stateful setter" pattern is the best way forward (it's certainly the simplest), a system-wide cfg! op might be the right solution. I think it makes sense to register legal values for the cfgs somewhere in the implementation to catch typos, etc.

I think the cfg! mechanism makes a lot of sense, because other plugins will inevitably have configs of their own, and a unified mechanism for managing the configurations will be needed at some point. That said, it could be declarative (as opposed to imperative) if we can come up with a friendly way for it to work.

What do you think?

[vsbogd]

The example we have discussed is while we have some default global setting like ReplPrompt user may want to override it in some sub-config. In this sense (= ReplPrompt ...) is not a best way because it doesn't override but augment the definition of the ReplPrompt. Thus I agree that for configuration we need something like states that we can change or some equality we can override.

I made an attempt at implementing the configs with State atoms

bind! helps to get rid of referencing to a state via another symbol, for example the following code works:

!(bind! &ReplPrompt (new-state ">"))
!(get-state &ReplPrompt)
; [">"]
!(change-state! &ReplPrompt ">>")
!(get-state &ReplPrompt)
; [">>"]

But to query the state from the Rust code you will need to use tokenizer in order to parse &ReplPrompt token, which can be inconvenient.

On the other hand (= ...) format allows us using functions to infer values.

I don't exactly understand what flexibility is available in the declaration that isn't available in execution.

Yes, you are right here, it is not very conscious comment from my side.

If we create a dependency push mechanism in the future then maybe it'll be workable. (ie. a way we can register for updates on changes in a space that affect the way a specific atom will evaluate) But that sounds like a Big job that might not be within the design vision for MeTTa at all.

In fact we may need it as well for implementing other things like JIT compiler or memoization. I agree it is big and we cannot count on it in this task for now.

If we decide a "simple stateful setter" pattern is the best way forward (it's certainly the simplest), a system-wide cfg! op might be the right solution.

Yes, we can also implement separate library to manage configurations. For instance implement cfg! with necessary semantics.

[luketpeterson]

bind! helps to get rid of referencing to a state via another symbol...

This is an interesting conceptual question. In my mind, a Tokenizer was more of an "support" component for the parser & interpreter, and the Space was the container for the meaningful MeTTa state. But it sounds like you and Alexey may be thinking about Tokenizer being a more fundamental part of the MeTTa language.

If we choose to lean heavily on the tokenizer, then a clean-looking solution will be to introduce new tokens for every config when initializing the REPL. And the config file will just be:

!(change-state! &ReplPrompt "> ")
!(change-state! &ReplBracketStyles ("36" "35" "33" "32")) ; 36: cyan, 35: magenta, 33: yellow, 32: green
!(change-state! &ReplCommentStyle "90")                   ; 90: light gray
!(change-state! &ReplVariableStyle "33")                  ; 33: yellow
!(change-state! &ReplSymbolStyle "0")                     ; 0:  default text (as per terminal setting)
!(change-state! &ReplStringStyle "31")                    ; 31: dark red
!(change-state! &ReplErrorStyle "91")                     ; 91: bright red
!(change-state! &ReplBracketMatchStyle "1;7")             ; 1:  bold, 7: reverse (swap background and foreground colors)

What do you think?

[luketpeterson]

This change is implemented in 5306394 if you want to try it out.

[vsbogd]

Thanks Luke, lets keep it in a state form before we have better idea.