add clnrest object to plugin manifest

[gudnuf]

Allows a plugin to register custom REST method, path, and return content_type for any of its RPC methods along with a rune flag that signifies whether or not authentication should be applied to this route.

Motivation

We would like plugins to be able to customize how they are used via a REST interface such as clnrest. An example use case is a Cashu mint plugin that must follow the Cashu protocol for each of the endpoints, so each RPC method must be mapped to a specific REST route (ie GET /v1/keys).

Will also help with #7164

What changed?

• added plugin_rpcmethod_clnrest_add to lightningd/plugin which will parse out clnrest data from the get_manifest response. For each RPC method, the get_manifest response can now contain:

{
  "clnrest": {
    "method": "string",
    "path": "string",
    "content_type": "string",
    "rune": "boolean"
  }
}

Where:

• method: The HTTP method (e.g., "GET")
• path: The endpoint path (e.g., "/path/to/me")
• content_type: The content type for the return value (e.g., "application/json")
• rune: Whether or not this route requires a rune to be accessed

---

• modified jsonrpc_command_add so that we can check for name collisions and method/path collisions. Will now return false if there is an RPC method with the same name or an RPC method that registers the same route (ie. matching path + method)

---

• modified json_add_help_command so the output of lightning-cli help can return clnrest data:

{
   "command": "checkmymanifest [cmd_name]",
   "category": "plugin",
   "description": "Returns the manifest of this plugin.",
   "verbose": "Returns the manifest of this plugin.",
   "clnrest": {
      "method": "POST",
      "path": "/path/to/me",
      "content_type": "application/json",
      "rune": true
   }
}

---

• added option to specify clnrest_data to plugin.py so that pyln-client can add this to the get_manifest response

---

• add a plugin tests/plugins/get_manifest.py to test this functionality

Next steps

I will be submitting another PR that adds two new notifications: plugin_started and plugin_stopped.

This will allow other plugins such as clnrest to subscribe to these notifications and dynamically register and drop routes.

[niftynei]

few minor comments, overall looks great.

the checks found some extra whitespace.

lightningd/plugin.c:1388:static const char *plugin_rpcmethod_clnrest_add(struct plugin *plugin, 
lightningd/plugin.c:1389:                                                const char *buffer, 
lightningd/plugin.c:1390:                                                const jsmntok_t *tok, 
Extraneous whitespace found
lightningd/plugin.c:1391:                                                struct json_command *cmd) 

[niftynei]

if you add this are any of the fields required? maybe path and content_type?

[gudnuf]

Oh yea, this is the response for the help command. So if all of the fields have default values, then these would all be required, yea?

[gudnuf]

I set all these fields to required in 61c471a

That makes sense right?

[niftynei]

nit: i think technically the 'name' here is the name of the plugin command, not the plugin?

might be worthwhile to add a note somewhere that it defaults to the command name.

am i right in thinking this will make it such that clnrest will dynamically add a GET endpoint for every new plugin method that's added?? incredible.

[niftynei]

note that if that's the case we might have some default /v1/ etc prefix to add to the name?

[gudnuf]

'name' here is the name of the plugin command, not the plugin

oo yea you're right

am i right in thinking this will make it such that clnrest will dynamically add a GET endpoint for every new plugin method that's added??

Only if that method includes clnrest data in response to get_manifest. A plugin might want to specify the return content type, but nothing else...

[gudnuf]

some default /v1/ etc prefix

added this prefix in d52a311

[ShahanaFarooqui]

This manifest is not being reflected in clnrest yet. For example, below method will still require rune header and respond in json format:

@plugin.method("infotext", clnrest_data={"content_type": "text/html", "rune": False})
def info_text(plugin, **kwargs):
    """This methods sends text content!"""
    message = "--------------------------------\n"
    message += "Id: 0253afcb231426"
    message += "\n--------------------------------\n"
    message += "Alias: CLN"
    message += "\n--------------------------------\n"
    message += "Version: v24.05"
    message += "\n--------------------------------\n"
    return message

We should update file plugins/clnrest/utilities/rpc_routes.py from line 31 for above changes to reflect in clnrest also.

@rpcns.route("/<rpc_method>")
class RpcMethodResource(Resource):
    @rpcns.doc(security=[{"rune": []}])
    @rpcns.doc(params={"rpc_method": (f"Name of the RPC method to be called")})
    @rpcns.expect(payload_model, validate=False)
    @rpcns.response(201, "Success")
    @rpcns.response(500, "Server error")
    def post(self, rpc_method):
        """Call any valid core lightning method (check list-methods response)"""
        try:
            rune = request.headers.get("rune", None)
            rpc_method = request.view_args.get("rpc_method", None)
            rpc_params = request.form.to_dict() if not request.is_json else request.get_json() if len(request.data) != 0 else {}
            method_manifest = call_rpc_method(plugin, "help", [rpc_method]).get('help', [])[0].get('clnrest', {})
            try:
                if method_manifest.get('rune', True):
                    verify_rune(plugin, rune, rpc_method, rpc_params)
            except RpcError as rpc_err:
                plugin.log(f"RPC Error: {str(rpc_err.error)}", "info")
                return rpc_err.error, 401
            except RuneError as rune_err:
                plugin.log(f"Rune Error: {str(rune_err)}", "info")
                return rune_err.error, 403

            try:
                response = call_rpc_method(plugin, rpc_method, rpc_params)
                if method_manifest.get("content_type", "application/json") == "application/json":
                    return response, 201
                else:
                    response = make_response(response)
                    response.headers["Content-Type"] = method_manifest['content_type']
                    return response
            except RpcError as rpc_err:
                plugin.log(f"RPC Error: {str(rpc_err.error)}", "info")
                return (rpc_err.error, 404) if rpc_err.error["code"] == -32601 else (rpc_err.error, 500)

        except Exception as err:
            return f"Unable to parse request: {err}", 500

[ShahanaFarooqui]

Adding path and method seems useless as these cannot be changed for clnrest. Path will always be the name of the rpc command (@rpcns.route("/<rpc_method>")) and method is always POST (def post(self, rpc_method)). We can add GET, DELETE etc. methods types in clnrest but it seems unnecessary because POST can be used for all types.

[gudnuf]

I was thinking clnrest could be modified to register path and method dynamically. Already been talking to @daywalker90 about adding that feature to his rust version of clnrest (#7509)
This is why I also opened #7508, so that clnrest could dynamically add routes as new plugins start, and remove those routes when plugins stop.
True that POST can be used for all types, but a plugin may want to implement a specific protocol that defines what the method should be for some endpoint. Note "Motivations" above...

[gudnuf]

rebased to 7dc3fe3