Execute plugin outside Obsidian / from command line

[ThomasLudwig]

Hi,

I would like to publish my vault automatically.
Your plugin is great and the only one that really works for me and does not require a ton of third party tool installations.

Is there a way to launch your scripts from outside Obsidian ?
The way I would like to use it is:

1. Edit my vault on Windows ✔️
2. Automatically push to github each x minutes ✔️
3. On linux, automatically get the files from github ✔️
4. Convert MDs to HTML with your scripts, from a command line ❌
5. From a script, Delete HTML files I listed as "Private" ✔️
6. Copy the results in my webserver directory ✔️

At this time, I manually launch the conversion in windows from your plugin, then copy the HTML to linux, but I'd rather launch the conversion on linux from a cron job.

Thanks for any hint on how to do that.

[gnd]

+1

[KosmosisDire]

Thanks for the suggestion.

I don't think I will be making a commandline tool as this plugin realies heavily on obsidian's environment. There wouldn't be a viable way to make this work from the command line. I also cannot run the conversion in the background as, again, it relies on obsidian's renderer. I do think that there should be a solution for your use case, but I don't think command line features are the best solution, nor a viable one.

I will certainly keep this workflow in mind and look for ways to remedy this.

Thanks again.

[KosmosisDire]

I have gotten anumber of requests for something like this. I'd love to hear any ideas people have for ways to implement this. Here are the constraints I have:

• Obsidian has to be open with the plugin installed and enabled.
• The markdown files have to be part of the currently open vault (This maybe could change in the future, but not yet)
• The export will open new files in the foreground and make it so obsidian can't be used until the export is done. (I hope to change this along with the previous one, however it is harder than it seems because of some weird behavior)

This means that if you want to export the vault while you are editing it needs to happen in a seperate instance of obsidian. It also means that I can't run the plugin independently of obsidian.

A commandline tool would be possible but it would have to be a completely seperate program that the user would have to install alongside the plugin, since javascript is not made for console applications and there is no way to directly interface with them through the commandline as they are not executable (let me know if this is wrong).

In the end it might have to wait until I can get something to work in the background (although it would still require obsidian). But if you have any suggestions please drop them here. Thanks!

[ThomasLudwig]

For me specifically it would not be a problem to install a second obsidian on Linux (as the file that would be traited would be copies of my original vault anyway).

There would have to be a command like obsidian --export-html OUTPUT_DIR, that would

• open obsidian
• launch the export of the whole vault to the directory
• close obsidian

[KosmosisDire]

The main issue there is the "launch the export of the whole vault to the directory" part. I am sure there is a way to do it, but I am not sure how to run a command in obsidian from an external program. I will definetly consider it though. Thanks for the input

[KosmosisDire]

I now have background exporting available, this should make it abit easier to automate as it wouldn't have to take over the obsidian instance (even if that wouldn't matter in your case). I am looking into implementing this now.

[KosmosisDire]

@ThomasLudwig Would a simple "Export the vault every x minutes" work for your use case? It would be simple and fast to implement and since you are already pushing every x minutes from one end it seems that would work.

On the other hand I could just write two scripts one batch and one bash (so nothing extra has to be downloaded) that would trigger an export in an open instance of obsidian and wait until it was complete.

[ThomasLudwig]

Hi Might be a solution. But a found a workaround for my "problem". My need was to completely avoid human intervention and have cron jobs. Maybe a might keep an obsidian open all the time on my Linux server ? Le mar. 2 mai 2023, 10:03, Nathan George ***@***.***> a écrit :

…

@ThomasLudwig <https://github.com/ThomasLudwig> Would a simple "Export the vault every x minutes" work for your use case? It would be simple and fast to implement and since you are already pushing every x minutes from one end it seems that would work. On the other hand I could just write two script one batch and one bash (so nothing extra has to be downloaded) that would trigger an export in an open instance of obsidian and wait until it was complete. — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNFWEBLMUE6GBD3VS26MPLXEC5VJANCNFSM6AAAAAAVY5DZC4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[karabays]

Hi,
since you are at it, it would be also great to be able to choose the obsidian folder exported... 😄
instead of being able to export entire vault only.

[KosmosisDire]

Hi,
since you are at it, it would be also great to be able to choose the obsidian folder exported... 😄
instead of being able to export entire vault only.

You can already do this. Just right click any folder or file

[Regenhardt]

Made an obsidian feature request, although in the related links I added there you can see similar things have been requested for quite some time now:
https://forum.obsidian.md/t/uri-scheme-execute-command/62567

[Regenhardt]

How about copying the export-html-vault command to a export-html-vault-default one that uses sane defaults to export the vault (like multi file website + don't open after export)?
We could then use Obsidian Advanced URI to call this command from a CLI.

Of course I only found these after creating the obsidian topic. We'll see which proposal goes down first.

[KosmosisDire]

Made an obsidian feature request, although in the related links I added there you can see similar things have been requested for quite some time now: https://forum.obsidian.md/t/uri-scheme-execute-command/62567

Hey thanks, I took a look at this request and it looks like this is already built in to obsidian as well. I will look into implementing a commandline program to call the command using the URI actions.

(URI handler docs: registerObsidianProtocolHandler)

[KosmosisDire]

Note for future: Being able to automate this inside a docker container would be great because then the CLI would not rely on a seperate running version of obsidian, additionaly it should allow #140 to be implemented

[Mikle-Bond]

@KosmosisDire I think this is very possible.
This can be achieved with the https://github.com/Vinzent03/obsidian-advanced-uri plugin with something as simple as curl to trigger and obtain a fresh copy of the vault. There is a POST /commands/<command_id> method to trigger commands. The only lacking piece of the puzzle is the command to export vault into predefined export path.

Then, we could run obsidian in docker using something like https://hub.docker.com/r/metal3d/xvfb to mock an X server, and only communicate with it via HTTP requests. And in docker, user could mount a volume to the output path and use the output as web server's root.

It would be a bit cooler, if there was a way to expand the REST API somewhere around setupRouter routine here to install custom method and pass export directory and source directories (and possibly other options) on the URI. But a command that allows unattended export of the vault is enough to get started)

[KosmosisDire]

@Mikle-Bond

I think I may not even need the advanced uri plugin, because the obsidian API has a uri handler that can trigger commands. I would rather not require plugin dependencies because they are not automatic. Of course, more advanced things like passing information in the uri may not be supported with only that.

However everything that you have mentioned sounds amazing and summarizes very well the thoughts I was having surrounding this issue. Thank you for the resources and ideas!

[Mikle-Bond]

@KosmosisDire Sorry, i linked the wrong plugin, I meant to link to https://github.com/coddingtonbear/obsidian-local-rest-api (probably 'cos the amount of tabs open in chrome)
I'm not sure how obsidian URI custom schemes would work over network and in docker. So the REST API would be a bit more in the spirit of docker environments.

Sorry again for the confusion

[KosmosisDire]

@Mikle-Bond Ah yes, I think I have looked at that plugin too. That's definitely another option. I wonder how difficult it would be to just make my own REST API built into the plugin rather than relying on a separate plugin? I have never really done anything like that before.

[mc0e]

At this time, I manually launch the conversion in windows from your plugin, then copy the HTML to linux, but I'd rather launch the conversion on linux from a cron job.

I can imagine running this on a web server. Locate the git repository there, and use a local hook, or use a webhook from something like guthub to call the server. Obsidian could sit inside a docker container on the web server, and get activated for the publishing whenever the git repo gets a submission.

[mc0e]

I'm a bit uncomfortable with the idea of turning obsidian into a server listening to requests on the public web. Obsidian isn't written with this sort of thing in mind, and there's substantial security issues.. It could very easily turn into a way to run arbitrary code on the server, or even just to run arbitrary javascript in browsers.

From a security standpoint, I'd prefer to run a very simple web hook, that doesn't really take any input, just listens on a specified url, receives a request when the git repo has been updated, and runs a script in response. That script would pull in the repo, and turn it into web pages via a command line interface.

[KosmosisDire]

@mc0e It kinda sounds like you are describing github actions with extra steps. However, I don't plan on creating anything where obsidian is open to the public internet. If you were to setup something like you described the security would be your responsibility. Anything I create to address this issue would be an offline cli compatible tool. Your usage of that tool would be up to you.

[KosmosisDire]

Additionally, any REST API I create (which would be local not public) will not allow the modification of any of obsidian's files. It would only allow you to set the plugin's settings and then export the files to a given directory.

[SteinGaming]

Hi there, after modifying the main file and adding a bit of my own implementation, I was able to make it run headless (kinda? will mention it later) under a raspberry pi 4!
I'm adding this comment as a proof of concept, because this has to be implemented by everyone themselves.

THIS TEXT DOSEN'T REALLY HAVE A GOOD STRUCTURE, READ IT THOROUGLY IF YOU WISH TO REPLICATE!!!!!!

The only thing you have to add into the main file is this:

this.registerObsidianProtocolHandler("webpage-export", async (e) =>{
      await HTMLExporter.export(true)
})

This isn't really secure, but again, just a proof of concept + a private implementation of mine :).

It's a bit stupidly made, but I have the following setup:

• REQUIRED: X11 server, which is only started to satisfy obsidian (which is why I called it "kinda headless")
• RPI 4 with 64-bit Debian Buster (32-Bit is not possible, due to the Obsidian binaries being 64-Bit)
• A personal gitea server on my RPI 4
• A custom act runner connected to that gitea inside a docker container on the RPI 4
• The extracted AppImage from https://obsidian.md.
• A nginx server serving the files; has a symlink from /var/www/html/ to /home/stein/Exported/

As a preperation you'll need to modify/create your $XDG_CONFIG/obsidian/obsidian.json, in my case it looks like this:

{"vaults":{"4f7a2aefbf31a16d":{"path":"/opt/test","ts":1699390107453,"open":true}}}

You'll also need a json file according to that ID in the same directory, but I'm pretty sure that you can modify the parameters as you see fit, as these are irrelevant to us.
$XDG_CONFIG/obsidian/4f7a2aefbf31a16d.json:

{"x":128,"y":112,"width":1024,"height":800,"isMaximized":false,"devTools":false,"zoom":0}

ALSO: As a verfication process, if you run it headless, I would recommend you check what obsidian says at the first time you do this, because my plugins didn't enable right away, due to (literal) trusting issues from Obsidian.
The way I did it was to use SSH X11 forwarding with ssh rpi -X /home/stein/obsidian/obsidian.

IMPORTANT: The gitea act runner has a host volume connection to /opt/test/:/opt/test/.

These are the steps it takes:

1. Upon a push, the act runner is executed with the following code:

name: Gitea Actions Demo
run-name: ${{ gitea.actor }} is testing out Gitea Actions 🚀
on: [push]
jobs:
  Explore-Gitea-Actions:
    runs-on: ubuntu-latest
    container:
      image: node:16-bullseye
      volumes:
        - /opt/test/:/opt/test/
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: rm -rf /opt/test/*
      - run: cp -R . /opt/test/
      - run: chmod -R 777 /opt/test
      - run: curl IP_RPI:NOTIFY_PORT

This clones the repository, removes the old repository from the volume and copies the new one into it, assigns permissions and notfies my custom programm of the change using NOTIFY_PORT.

2. My custom programm, written in Kotlin, does the following things:

   1. It parses the config using GSON
   2. Sets exportPath to /home/stein/Exported
   3. Clears the filesToExport[0] array
   4. Collects files by a custom regex, adds the relative path to the filesToExport[0] array
   5. Saves the config
   6. Kills obsidian
   7. Runs the next script

3. After rewriting the config, my program executes /home/stein/export.sh, which contains:

#!/bin/bash
rm -rf ~/Exported/*
cd ~/obsidian
ELECTRON_ENABLE_LOGGING=1 DISPLAY=:0 ./obsidian&
sleep 20
ELECTRON_ENABLE_LOGGING=1 DISPLAY=:0 ./obsidian obsidian://webpage-export

(you could use xdg-open obsidian://webpage-export as an alternative instead to ELECTRON_ENABLE_LOGGING=1 DISPLAY=:0 ./obsidian obsidian://webpage-export, but this doesn't work on my machine due to the desktop file being non-existent)
This does:
1. Clear existing files from the Exported directory
2. cd's into the obsidian directory
3. Starts obsidian as a job
4. Waits 20 seconds for obsidian to completly prepare the extensions (depends on the speed of the machine)
5. Run obsidian with the protocol handler
This starts the exporting process, and after some time (due to the poor performance), it successfully serves it onto my path!

I hope this helps anyone to set up their own (and better lol) version of this monstrosity, questions are welcome!

[fachschaft-tga]

@KosmosisDire Thanks for your work and effort.
I'm interested in this feature as well :)

[KosmosisDire]

Just as a small update on this, I played around with getting obsidian set up in docker. I was able to create a docker image which installs obsidian, opens it, automatically creates a new vault, and disables restricted mode to allow plugins to be installed or enabled. All this headless (although running an x11 server) without user interaction. So I think I have a good path forward for getting this to work using docker.

[spartacus04]

Hello!
First of all I wanted to thank you for this project, I currently use it to share my university notes on gh pages and it works wonderfully.

About this feature, I'm willing to help finish this if you share your progress.
I would make a pr with my implementation but I'm stuck at disabling the restricted mode.

[KosmosisDire]

Awesome, I can definitely send you that. It's actually surprisingly simple. I'll post it here in a bit when I've got time

[KosmosisDire]

@spartacus04
Here is my docker setup, it has two scripts which are injected into obsidian on startup. One forces obsidian to create a new vault and open it, the other disables restricted mode. These scripts are injected using the electron-inject python library. Any other javascript can also be injected which can be pretty useful for automating any other tasks that need to happen to setup obsidian.

obsidian-docker.zip

[spartacus04]

Aight, I'll be working on it on my fork, I don't really have an ETA since I also have to study but I'll definitely put some work into it.

[sykdan]

Hi @spartacus04, thanks for your work. I'm trying to run your scripts without success, it doesn't work even after simply forking this repository and creating a tag:

https://github.com/sykdan/obsidian-webpage-export/actions/runs/10975846065/job/30475944687
Looking into the actions history of this repo, it seems to be working over here; any clue as to what could the problem be on my side? I toyed around with apt install ... -f and similar options but to no avail.
Many thanks.

[spartacus04]

Hello, sorry for the late response, I don't really know why this happens, as theoretically the environment should be the same inside the docker container.

I'll try investigate the issue in the next few days