Controllable memory allocation

[JensOgorek]

Hello,
we use owl on our ESP32 in our firmware zauberzeug/lizard. Occasionally, the ESP32 devices would hang during usage, especially when running long startup scripts.

After some digging, we found the function owl_default_tokenizer_advance attempts to allocate 40 KB of RAM for every line processed. On an ESP32, this can quickly lead to problems due to limited memory availability.
In our version of owl, we reduced the allocation size from 4096 to 256, which now only consumes 2.5 KB of RAM. For example, in a longer call, we only process up to 24 tokens at a time, which works well with this adjustment.

Do you have an idea for making tokenizer_advance more scalable? One suggestion we had was to introduce a configurable parameter in owl_tree_options to allow users to set the token count dynamically, with a default of 4096.

Optional Enhancement:
It could be beneficial to introduce a dedicated error state for cases where allocation fails, as the current error provided by owl does not clearly indicate that the issue is related to memory allocation. This would make diagnosing such problems more straightforward and user-friendly.

[ianh]

Sounds good -- with dba6335, you can use #define OWL_TOKEN_RUN_LENGTH before including the parser implementation to override the default. ae33e64 should properly report when the relevant allocation fails, though there are other allocations which don't have proper error handling at the moment.

[JensOgorek]

That works great, besides the #define OWL_TOKEN_RUN_LENGTH getting ignored from main.cpp, if I add it before #include parser.h.
But, adding the #define to parser.c or add_compile_definitions(OWL_TOKEN_RUN_LENGTH=256) in CMakeLists works.