Add a Terminal/Console/Shell #101
Replies: 5 comments 4 replies
-
I assume you are elotero from Discord. Again, thank you for the discussions. Just stopping by to remember you that I'll check and answer to them properly at the beginning of the week, as I said on Discord. I just wanted to reassure you of that since I just replied to another discussion here ahead of yours. Only did that because I deemed it more urgent since that discussion was related to a bad experience a new user had and I wanted to guarantee every concern was addressed properly. Have a nice weekend! |
Beta Was this translation helpful? Give feedback.
-
Finally got time to take a look into your post! First of all, thank you for your code contribution! Addressing a few of your comments
There is no such thing as a dumb contribution. Of course, we all have shortcomings, so at the end of the day a code contribution may indeed be or not be suitable for integration into the project's source, but that doesn't change its intrinsic worth a bit. That is, even if in the end the code isn't used, there are several other elements in that process that may help us in the end. For instance, the ideas/expectations/beliefs/assumptions that originated that code may provide insight into other future improvements to the project. The code may even be used in other parts or for other purposes. Besides, even when the code is of high-quality and well-thought-out, it may still not be integrated into the repository simply because it doesn't fit the design/scope/purposes of the project.
Again, don't worry about that, it is not a requirement to contribute code or sharing ideas. However, it is indeed important to be upfront about one's skills and shortcomings, so that everyone can set their expectations accordingly and be able to cooperate more efficiently. Additionally, there is no minimum level of skill for contributing code/ideas here. Someone with a month of coding can help someone with a week of coding, who can help someone with a week of coding and so forth. Also, even if someone has more experience than others, doesn't mean that experience applies to the same things. Someone with 10 years of programming can still lack knowledge and practice in a specific area in which another programmer with only 01 years of experience may excel. A bit of context on non-sequential execution paradigmsNow, allow me to give you a bit of context regarding threads and other non-sequential execution paradigms in Nodezator. After that I'll comment on your code contribution. First of all, I have been studying threads, multiprocessing and other non-synchronous execution paradigms in the past couple of years. Not much though, but on and off when I get a bit of time. That's cause the changes I'm planning for Nodezator as well as several of its features to be implemented are dependent on my knowledge of these paradigms. Even so, I can't focus on that yet because Nodezator still lacks other higher-priority features like group nodes and the missing test cases for the system testing feature, for instance (just to name a few), which are things on which I've been focusing my time. Because of that, I was only ever able to experiment with more simple and small-scoped examples, which are typical in learning materials for these subjects. To make matters even more complicated, the kind of non-sequential execution that Nodezator requires is much, much more complex than that presented in books and other learning/research materials. For instance, nodes represent callables, but a callable may be the kind that involves I/O or network-bound operations, for which threads are suitable. However, if the callable involves CPU-bound operations, multiprocessing is better suited. There's also async/coroutines, right? It means in theory Nodezator must be able to handle multiple non-synchronous paradigms to be able properly and efficiently address efficient execution of its graphs. And the complexity doesn't stop there. Nodezator is a meta-application, that is, it doesn't have a specific purpose, but rather loads and runs code from others for their specific purposes. This means that on top of being able to handle multiple non-sequential execution paradigms, Nodezator needs a way to tell which kind of paradigm is better suited for each given callable. Should we parse/analyse the code? Should we simply allow users to tell us via a specific variable? Unfortunately, it looks like no book author was ever crazy enough to consider approaches that address the need to mix these and other paradigms, specially in a context where the callables received are arbitrary, that is, the users can provide any number of callables of any kind they want. Analyzing the source of other existing node editor projects, Python-based or not, may not even provide a direct solution to the problem (although, of course, may allow us to collect useful insights to help with that task), because many of them either provide a limited set of nodes and can thus optimize their execution to suit these operations or simply operate in a scope smaller than that of Nodezator. That is, on top of allowing the users to provide any kind of callable they want, Nodezator is meant to be a visual representation of Python, which is why one of our proud features is the possibility to export any graph as Python code. This means the Python code exported must also run with mixed non-sequential execution paradigms. Your code contributionNow that you know all of that I can finally comment on your code! Being completely honest, your code looks like great, simple and clever usage of threading. Small solutions like that which provide a lot of new functionality are invaluable. It might not even need any refactoring or other refinements. The only reason I don't intend to integrate it right now is simply because, as I said before, I must first consider how all the non-sequential execution paradigms will play together within Nodezator. In other words, I don't know if in the future I'll have to start Nodezator in another thread, or just execute the graph in another thread, or just create execution pools for threads and processes for different kinds of callables to use during execution. I know how to use many of these tools in isolation and there's a lot of learning materials for that freely available, but the problem lies in having to consider them all together within Nodezator. It is not even that your solution isn't appropriate as far as its purpose is concerned, it is just that I don't have the knowledge, skills and experience enough to properly assess such suitability. And in addition to that, as I already pointed out, finding this out will likely require much research on my part when it is time to implement non-sequential execution. For now, as I said before, I need to focus my time on other features of higher priority and that are also important. Another point to consider is whether, despite being undoubtedly useful, we would indeed want to include in Nodezator's design the ability to run a console at all. But to be honest, if we can get a console with such minimal code, why not, right? Since a big part of the target users of Nodezator are programmers anyway, then there must indeed be a lot of people that would benefit from it one way or another. We could even use part of your code for using the terminal running Nodezator as the console itself, or provide a console panel within Nodezator as well. Therefore, I'm just mentioning this point for the sake of completion. What indeed prevents me from integrating this solution is, as I said before, the need to give much more consideration to the different non-sequential execution paradigms Nodezator needs to deal with, coupled with my current inability to properly assess the solution in regard to that. Regardless of all that though, your solution is very interesting and other people may need it as well, so I'll leave this discussion open. In addition to that, your code may be used in the future, though, once I'm able to properly assess and integrate it. If it ends up being the case, I'll either ask you to submit a PR or I'll commit the changes myself with you (that is, your GitHub profile) as the author of the changes (something that is not only possible, but quite easy to do in git, and something that I did before, with proper permission from the author, of course). I believe this is all I wanted to say for now. Let me know if you need anything else. I intend to answer your other discussion post soon as well, and your other comment made on this discussion regarding your suggestion of making another video tutorial for Nodezator. It might take a few hours or a couple of days though, because of other commitments that I need to attend to in the meantime. Have a nice week! Peace |
Beta Was this translation helpful? Give feedback.
-
Hey Kennedy, good to read from you, as always well articulated and thoughtful. I can see you have very valid points in here, i agree with this idea to not be implemented without being carefully measured. With that said, i think that if anyone need's a console in the way i needed (to investigate internal structure of the software), this is an easy solution that does not affect the general functioning of Nodezator in any way, it's just a launcher. As you said, make the console running Nodezator to be the console, or to have a console within Nodezator, opens a lot of opportunities for debugging and integration. One way or another it widens the scope of the software and makes it more flexible. Now about this other approaches to concurrency, async, subprocess, etc i wanted to ask you: So my question is: how does Nodezator handle this, is every node in it's own subprocess? no, I checked and did not found more python instances running, other than the mine and the Nodezator main. Why is that? Obviously you have some sort of concurrency with all the nodes. This bugged me, i wasn't able to think around this, and i still can't. I had to run a small local server where all the nodes are the clients, to give the nodes some sort of a clue about which node is which. About this "questions" Is great to see that you give to ideas great consideration and value, i'm glad to know that humble contributions like this might make it to the codebase of Nodezator or just be keept on sight for others to improve and build on top. Well this is getting too loong already, i read your comments on Run Single Nodes, great to know it might be implemented, i'll comment on that later separately. Have a great week!!! |
Beta Was this translation helpful? Give feedback.
-
Yes, I'm considering all of that. I just need time to assess the possibilities. Even simple things that don't interfere with others must only be added after proper thought, with each feature comes usage and with usage comes familiarity, which makes it more hard or just annoying to user when it is changed. Mostly, I need to check and understand inner workings and implications of running the app or parts of it in another thread. I'll probably end up using 80 to 90% of your original solution, though. Also, for now, I don't have a timeline for this, as it is not an urgent task nor a much requested feature. As you said, I'll leave it here for others that may need a similar solution. However, since it is a very small change, I'll see if I can indeed include it in a patch in the future.
There is no concurrency at all presently. All execution is sequential. As I said before, I must still design a concurrency/parallelism model, and the main hurdles for that is that such design must: (1) take into account all the available paradigms to avoid problems during execution, like nodes which would otherwise run normally ending up raising errors due to various reasons like scope/race problems and (2) allow the entire graph to be exported as the exact equivalent Python code (that is, it must also execute concurrently/in parallel). If I had implemented my own arbitrary concurrency/parallelism model without properly understanding all implications and corner cases I'd risk limiting what kind of code Nodezator could properly run. Regarding how the nodes are loaded, each node is loaded as a separate module/package. They are completely (or mostly) independent from each other. By mostly, I mean that there is a way for nodes to share resources: by importing them. This is explained in the manual here: https://manual.nodezator.com/ch-defining-your-first-node.html#sharing-resources-among-nodes That's why each node sits on its own folder within the categories instead of all of them being part of the same file like in other node editor projects. I designed the node packs like that, with nodes as independent from each other as possible, for 02 (two) reasons. First, because the simpler the functions, the more easy it is to combine/compose them into powerful graphs. Using an analogy, the simpler/smaller the Lego® pieces, the more freedom of building one has. For graphs it is just like that as well. It becomes more easy to compose them into many different graphs. Although Python is a multi-paradigm programming language (though mostly object-oriented), one of the most powerful paradigms in terms of both power and flexibility is FP (functional programming), and it works particularly synergistically with node editing (although I don't mention this specifically in the manual, many FP tools are demonstrated/taught in it, like this recipe from one of its appendices and the usage of the callable mode for both branching and looping, in their respective chapters). Why am I mentioning FP? Because just like in Lego®, FP also encourages simpler/pure functions. I'm not talking about simplicity as the opposite of complexity here, but just in the sense that a node doesn't depend on other nodes. Its source can still be as big as needed (just like the more specialized, thematic, bigger Lego® pieces). The node source may even consist of multiple modules. As long as it doesn't depend on other nodes, it is much more easy to compose them. The second reason I made nodes as independent as possible from each other is for the sake of the very concurrency/parallelism that we've been discussing. The less state/side effects an operation has, the easier it is to run it concurrently or in parallel. Of course, I had only 02 years of (serious) Python experience when I started making Nodezator in 2019, so I intend to change/relax a few decisions I made back then when I didn't have enough experience nor as much insight and feedback as I have now after discussing my decisions with a lot of users and developers and testing my early ideas/decisions. Even so, the principles behind the decisions should remain, even though the form they take may change a bit. I'll stop for now since I don't have much time, but I'll come back to address the rest of your comment as soon as I can (probably next week). Have a nice weekend! Peace. |
Beta Was this translation helpful? Give feedback.
-
I mentioned parsing/analysing for the sake of completeness. In practice, it is not only a nightmarish, but fraught with edge-cases and guessing and, as you aptly pointed out, likely unfeasible. It is the very reason why I use variables like "main_callable", etc. instead of reading them from the node's source. For instance, other users argued before in favor of parsing the code and automatically identifying the main callable, but I explained the impracticality of that in detail to them. As for coroutines, it is true they are seldom mentioned in Python materials/content, and likely used even less. However, if my research at some point do indicate their usefulness over alternative approaches, I'm inclined to also allow some sort of concurrency using them, specially as I want to Nodezator to be as equivalent to Python as possible. With that, I think I replied to all the main points of your previous message. Now only need to answer to this comment. Of course, if you need to discuss anything further, don't hesitate to comment again and again. As always, I'll answer when I can. Peace |
Beta Was this translation helpful? Give feedback.
-
I come with a feature request, a suggestion, perhaps is already implemented and i'm not aware.
Feel free to move this ticket if is not in the right place.
So i come with a request for something i already solved somehow for myself but it's perhaps useful to other users.
And probably can be better implemented from developers side than what i did.
I modified the "main.py" of nodezator to launch an ipython console to debug and "directly" interact with the nodes.
It runs nodezator it makes nodezator to run in it's own thread normally but exposing it's runtime variables and functions to the main execution.
Here is what i did, hope it's useful to someone.
This first file is in the directory where you nodezator is installed, normally in the site packages of your python environment.
You have to rename it as .py , back up (just in case) and replace the original file.
you dont have to actually rename it just change the .txt to .py github does not recognize the underscore that's why in added the nodezator-to__main__.py to diffrenciate it from the other file
nodezator__main__.py"
this other file you just run your ide or anywhere where it can import from nodezator.
it will launch nodezator normally but without blocking your terminal and giving you an ipython shell.
You'll need to pip install ipython if not installed on your environment.
main.py
Once it runs, you can do dir(ndz) to see whats doing on. and dir(ndz.APP_REFS)
If you are using pycharm i recommend to run in debug mode to have autocompletion and access to your file system as well ;)
I'm not a very tec guy so, perhaps is a dumb thing, but i hope someone find it useful.
Beta Was this translation helpful? Give feedback.
All reactions