using hylang for configuration of an application

[wrobell]

I am bit tired of YAML, and I am considering to use Hylang for configuration of an application.

This is one of my first steps in Lisp, so I hope for some help and patience. I still have big gaps here.

Let's consider

(import
  funcparserlib.parser [maybe many]
  hy.model-patterns *)

(setv form '(
  (storage
    (role n23-role)
    (extensions (timescaledb postgis)))))

(setv parser (whole [
  (pexpr
    (keepsym "storage")
    (pexpr (keepsym "role") (>> SYM str))
    (pexpr (keepsym "extensions") (pexpr (many (>> SYM str)))))
]))

(setv config (.parse parser form))
(print config)

Now, I would like to put configuration into a file

(storage
  (role n23-role)
  (extensions (timescaledb postgis)))

and then read the file, and parse it with parser defined in the example above.

Any suggestions how to do that?

I suspect I should start with read-many, and finish with eval. What to do in between?

[Kodiologist]

Just use (.parse parser form), like you did above. Did that fail somehow? hy.eval is not required; that's for evaluating forms that represent Hy code.

[wrobell]

The Hylang example I have provided works without a problem.

I want to save (storage ...) as a file, for example app.cfg.

Now, I would like to read the file and parse it

(import
  funcparserlib.parser [maybe many]
  hy [read-many]
  hy.model-patterns *)

(setv parser (whole [
  (pexpr
    (keepsym "storage")
    (pexpr (keepsym "role") (>> SYM str))
    (pexpr (keepsym "extensions") (pexpr (many (>> SYM str)))))
]))

(setv f (open "app.cfg"))
(setv form (read-many f))

(setv config (.parse parser form))
(print config)

This fails with the following exception

Traceback (most recent call last):
  File "/home/wrobell/.local/bin/hy", line 8, in <module>
    sys.exit(hy_main())
             ^^^^^^^^^
[...]

  File "/home/wrobell/.local/lib/python3.11/site-packages/funcparserlib/parser.py", line 614, in _some
    if s.pos >= len(tokens):
                ^^^^^^^^^^^
TypeError: object of type 'Lazy' has no len()

Therefore, I think, I am missing something here.

[Kodiologist]

I think this might be a bug, whereby model-pattern parsers can't cope with Lazy. I'll have to look into it. (Edit: it's an upstream issue: vlasovskikh/funcparserlib#78 .) In the meantime, try (setv form (list (read-many f))).

[wrobell]

This works indeed, thanks!

One more problem. I am keeping the definition of the parser intact, but I modify app.cfg to incorrect version

(storage
  (role n23-role))  ; extensions are missing

Running the script gives me

Traceback (most recent call last):
  File "/home/wrobell/.local/bin/hy", line 8, in <module>
    sys.exit(hy_main())
             ^^^^^^^^^
[...]

  File "/home/wrobell/.local/lib/python3.11/site-packages/funcparserlib/parser.py", line 616, in _some
    raise NoParseError("got unexpected end of input", s2)
funcparserlib.parser.NoParseError: got unexpected end of input, expected: some(...)

The bit to improve here is the ... in expected: some(...). It would be nice to let an user know that extensions configuration item is expected here. Could this be improved somehow?

[Kodiologist]

Maybe. I haven't made a real attempt at that sort of thing yet. See my description of #2415.

[wrobell]

OK. Thanks.

This leads me to another question (actually two, but I need to sleep over the 2nd one ;)

Let's assume the following configuration file

(storage
  (version 1)
  (role n23-role)
  (extensions (timescaledb postgis)))

To parse the version, I have added to my parser (pexpr (keepsym "version") Integer), and this fails with exception

Traceback (most recent call last):
  File "/home/wrobell/.local/bin/hy", line 8, in <module>
    sys.exit(hy_main())
             ^^^^^^^^^
[...]

  File "/home/wrobell/.local/lib/python3.11/site-packages/funcparserlib/parser.py", line 315, in __add__
    name = "(%s, %s)" % (self.name, other.name)
                                    ^^^^^^^^^^
AttributeError: type object 'Integer' has no attribute 'name'

Is there a way to specify that a string or an integer is expected?

[Kodiologist]

You gotta use some, similarly to how SYM and KEYWORD are defined in hy/model_patterns.py.

[wrobell]

I extended my configuration file. It can look now like

(storage
  (version "1.0")
  (role n23-role)
  (extensions (timescaledb postgis)))

(entities
  (entity pressure
    (partition-by location)
    (columns
      (location text)
      (value double precision)))
  (entity temperature
    (partition-by location)
    (columns
      (location text)
      (value float4))))

Would it be possible to enable scripting in my configuration file? I would like to turn above to

(defmacro table [name value-type]
  `(entity ~name
     (partition-by location)
     (columns
       (location text)
       (value ~value-type))))
  
(storage
  (version "1.0")
  (role n23-role)
  (extensions (timescaledb postgis)))

(entities
  (table pressure double precision)
  (table temperature float4))

I wonder if I need to turn storage and entities into macros, or is there some better way?

[wrobell]

I will try to evaluate both approaches.

I do understand what @scauligi is saying, and I see how this might work.

@Kodiologist I think I do understand what you are saying. I am using read-many... and then I should use one of the official functions to process the whole structure, and that processing will evaluate macros? I tried few blindly, probably incorrectly. Or I have to iterate and apply manually like @scauligi suggests?

[Kodiologist]

You'd use hy.read-many to turn the text into models and then hy.eval to resolve the quasiquotes, which would substitute in the results of table.

[wrobell]

I tried eval as first try indeed

...
(setv f (open "appm.cfg"))
(setv form (list (read-many f))) ; i have tried without a list as well
(.close f)
(print (eval form))

I am getting

Traceback (most recent call last):
  File "/home/wrobell/.local/bin/hy", line 8, in <module>
    sys.exit(hy_main())
             ^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/home/wrobell/projects/hy-config/rconf.hy", line 27, in <module>
    (print (eval form))
            ^^^^^^^^^^
TypeError: eval() arg 1 must be a string, bytes or code object

where appm.cfg file is

(defmacro table [name value-type]
  `(entity ~name
     (partition-by location)
     (columns
       (location text)
       (value ~value-type))))
  
(storage
  (version "1.0")
  (role n23-role)
  (extensions (timescaledb postgis)))

`(entities
  ~(table pressure double precision)
  ~(table temperature float4))

[scauligi]

eval is Python's eval; for Hy's you need specifically hy.eval

[wrobell]

🤦

@scauligi thanks :)

[wrobell]

I tried to use macros without Hylang's parser, but finally it clicked - it is all about code generation. Back to using parser then! The prototype of a config reader in the script below. I hope someone will find it useful. Thanks to all for help.

I am quite pleased with the result. It is really easy to create your own DSL and

• DSL has its own schema thanks to the parser
• there is a data model - Config and related classes in my case; thanks to the parser it is easy to read data into my structure of classes
• the data model is ready to be consumed by an application with proper type annotations
• DSL has programming capabilities

IMHO, this has much more potential, for example comparing to

• YAML and Jinja templating (see Ansible)
• YAML with programming constructs defined with YAML entries (see ESPHome)
• StrictYAML, which allows to parse data with a schema, but the result is mix of list and dictionaries
• TOML, schema in its infancy (there seems to be no appetite to adopt it), and bit heavy on the syntax (i.e. strings need to be quoted)

I am afraid I will see a lot of nails with that hammer. ;)

Config file read by the script (quasiquote for storage and entities is added before evaluation)

(defmacro table [name #* value-type]
  `(quote (entity ~name
     (partition-by location)
     (columns
       (location text)
       (value ~@value-type)))))
  
(storage
  (version "1.0")
  (role n23-role)
  (extensions (timescaledb postgis)))

(entities
  ~(table pressure double precision)
  ~(table temperature float4))

The script outputs

Config(storage=Storage(version=hy.models.String('1.0'),
                       role='n23-role',
                       extensions=['timescaledb', 'postgis']),
       entities=[Entity(name='pressure',
                        partition_by='location',
                        columns=[Column(name='location', type='text'),
                                 Column(name='value', type='double precision')]),
                 Entity(name='temperature',
                        partition_by='location',
                        columns=[Column(name='location', type='text'),
                                 Column(name='value', type='float4')])])

The script

(import
  dataclasses [dataclass fields]
  functools [partial]
  inspect [get_annotations]
  pprint [pprint]
  funcparserlib.parser [maybe many]
  hy.model-patterns [Symbol Expression Tag keepsym pexpr sym tag whole
                     STR SYM])

(require hyrule.misc [of])

(defclass [dataclass] Storage []
  #^str version
  #^str role
  #^(of list str) extensions)

(defclass [dataclass] Column []
  #^str name
  #^str type)

(defclass [dataclass] Entity []
  #^str name
  #^str partition-by
  #^(of list Column) columns)

(defclass [dataclass] Config []
  #^Storage storage
  #^(of list Entity) entities)

(defn first [coll] (get coll 0))

(defn parse-attr-name [attr] (hy.mangle (str attr)))

(defn get-attr-type [cls attr]
  (get (get_annotations cls) (parse-attr-name attr)))

(defn parse-attr-value [attr-type attr-value]
  (setv attr-sub-type (when (hasattr attr-type "__args__")
                            (get (. attr-type __args__) 0)))
  (match attr-value
    (| (list value) (Expression value)) (lfor item attr-value
                                              (parse-attr-value attr-sub-type item))
    (Symbol value) (str attr-value)
    (str value) value
    _ attr-value))

(defn parse-attrs-as-dict [cls args]
  (dfor [key value] args
        (parse-attr-name key)
        (parse-attr-value (get-attr-type cls key) value)))

(defn parse-struct-kw [cls args]
  #((get args 0)
    (cls #**(parse-attrs-as-dict cls (cut args 1 None)))))

(defn parse-struct-kw-named [cls args]
  (setv name (str (get args 0)))
  (cls name
       #**(parse-attrs-as-dict cls (cut args 1 None))))

(defn parse-struct-pos [cls args]
  (cls #*(lfor a args (parse-attr-value cls a))))

(defn parse-struct-debug [cls args] (+ (str cls) "::" (str args)))

(defn parse-struct-kw-debug [cls args]
  (+ (str cls) "::" (str (parse-attrs-as-dict cls args))))

(defn create-parser [func cls] (partial func cls))

(setv parse-column (create-parser parse-struct-pos Column))
(setv parse-entity (create-parser parse-struct-kw-named Entity))
(setv parse-storage (create-parser parse-struct-kw Storage))

(setv parser (whole [
  (>>
    (pexpr
      (keepsym "storage")
      (pexpr (keepsym "version") STR)
      (pexpr (keepsym "role") SYM)
      (pexpr (keepsym "extensions") (>> (pexpr (many SYM)) first)))
    parse-storage)
  (pexpr
    (keepsym "entities")
    (many (>> (pexpr (sym "entity")
                 SYM
                 (pexpr (keepsym "partition-by") SYM)
                 (pexpr
                  (keepsym "columns")
                  (many (>> (pexpr
                              SYM
                              (>> (many (>> SYM str)) (. " " join)))
                            parse-column))))
              parse-entity)))
]))


(with [f (open "app.cfg")]
  (setv form (list (hy.read-many f))))

(defn has-config-attr [expr]
  (setv name (str (get expr 0)))
  (setv cfg-fields (lfor f (fields Config) f.name))
  (in name cfg-fields))

(setv qform (lfor s form (if (has-config-attr s)
                             (Expression [(Symbol "quasiquote") s])
                             s)))

(setv eform (lfor s (hy.eval qform) :if (not (is s None)) s))

(setv config (Config #** (parse-attrs-as-dict Config (.parse parser eform))))
(pprint config)

[Kodiologist]

IMHO, this has much more potential, for example comparing to…

Yeah, I've always thought that specialized template languages were a mistake. Ultimately they just have to painfully reproduce a subset of the features of a full-blown programming language. Greenspun's tenth rule and all that.