Unicode in file path

[Erimus-Koo]

I have tried copying the full path of the current file. use

echo "{{file_path:absolute}}" | pbcopy

My file path is:

/Users/erimus/OneDrive/site/notebook/00_读书/03_日译中/西文字体设计方法.md

But the shell command output is:

\/Users\/erimus\/OneDrive\/site\/notebook\/00_\读\书\/03_\日\译\中\/\西\文\字\体\设\计\方\法\.md

It seems like there is a Unicode conversion issue. Do you have any advice on how to resolve it?

[Taitava]

Hi,

the backslashes \ are there because the plugin escapes all "special characters" in {{variable}} values (that is all charaacters other than a - z, A - Z, 0 - 9 and _ (an underscore)). So, also those unicode characters are escaped, i.e. prefixed with backslashes \. More about the escaping system.

You can solve this by removing the double quotes " around the variable, so:

echo "{{file_path:absolute}}" | pbcopy

becomes:

echo {{file_path:absolute}} | pbcopy

Now your Bash (or similar) shell will discard the backslashes.

Did this help to solve your problem? 🙂

[Taitava]

Oh but the backslashes are not the only problem. The characters seem to change:

• /Users/erimus/OneDrive/site/notebook/00_读书/03_日译中/西文字体设计方法.md (your original file path)
• /Users/erimus/OneDrive/site/notebook/00_读书/03_日译中/西文字体设计方法.md (your shell command output after I manually removed backslashes from it)

I'll try to inspect this.

[Taitava]

I tried to reproduce the problem on these platforms:

• Windows 10 with PowerShell
• Windows 10 with Subsystem for Linux using Bash
• Linux with Bash
• I do not own a Mac, so I can't test with that. I assume you are using macOS as you are using the pbcopy command?

My test on each platform involved the following steps:

1. Create an empty note file in Obsidian named 西文字体设计方法.md.
2. Create a shell command: echo "{{file_name}}" >> TestResults.txt
3. Have 西文字体设计方法.txt open and focused.
4. Execute the shell command.
5. Open up TestResults.txt to see the result. Results:
   • Windows PowerShell: 西文字体设计方法.md ✔️ (No backslashes because PowerShell handles escaping characters differently)
   • Windows WSL: \西\文\字\体\设\计\方\法\.md ✔️
   • Linux Bash: \西\文\字\体\设\计\方\法\.md ✔️

WSL and Linux Bash have the backslashes \ there because I kept the double quotes " in place in this test command. Same test with double quotes " removed from the command: echo {{file_name}} >> TestResults.txt ➡️ 西文字体设计方法.md ✔️ (on both Windows WSL and Linux Bash).

Conclusion

Maybe the unicode characters corruption only occurs on macOS? I mean it can be a bug in the SC plugin, and I'm willing to fix it if possible. But if it's a bug, then I'll need to borrow a Mac for being able to hunt it down.

To proceed one step at a time, I think it would be good if you can test what happens if you leave out the double quotes " like I suggested in my first message 👍 . Then let's see how the problem looks like after that. If the problem won't get solved by that, we can continue inspecting. In this case, please let me know what version of the SC plugin are you using? 0.21.0, I guess?

[Erimus-Koo]

1. I'm using a Mac

2. Version is 0.21.0

3. If use echo {{file_path:absolute}} | pbcopy, the output is:

/Users/erimus/OneDrive/site/notebook/00_读书/03_日译中/西文字体设计方法.md

4. Sometimes it works, e.g.

open -a typora {{file_path:absolute}}

5. I tried to print the output type, and that looks correct.

echo {{file_path:absolute}} | file -
> /dev/stdin: Unicode text, UTF-8 text

And I also tried to write the path to a file, that looks correct.
The issue may have occurred with pbcopy.

6. Write the full path to a file, then read it, then copy.

echo {{file_path:absolute}} > ~/path.txt & cat ~/path.txt | pbcopy

Something wired happens, if I run this in SC, the path convert to some strange letters.
But if I directly read the file and pbcopy in Terminal, like:

cat ~/path.txt | pbcopy

The result is correct.

Does that mean the pbcopy in SC and zsh is different?
I also tried which pbcopy, and both output /usr/bin/pbcopy.

7. I can directly use your output setting to satisfy my needs

echo {{file_path:absolute}}

output channel for stdout: Clipboard

But this is still a mystery, right?

[Taitava]

Sorry that it took so long for me to reply.

Does that mean the pbcopy in SC and zsh is different?
I also tried which pbcopy, and both output /usr/bin/pbcopy.

Good that you checked which pbcopy. Now we know that the same pbcopy is used both when executing via SC and via a normal terminal. I suspect that the problem is not in pbcopy, it's not different.

echo {{file_path:absolute}} | file -
> /dev/stdin: Unicode text, UTF-8 text

This looks like SC used a correct input encoding (UTF-8). Can you verify by trying these commands via SC:

echo $LC_CTYPE
echo $_LANG

One of those environment variables should contain the shell's current encoding (I'm not certain if it's input encoding or output encoding, or both). if the output is anything else than UTF-8, then it might indicate that SC does not communicate the encoding correctly to the shell (Zsh). Then again, there's always the theoretical possibility that SC says that the input is UTF-8, but actually submits UTF-16 content as input - I'm not certain about this.

I just happened to find a StackOverflow question asking about a similar problem you are having:

• [How can I input the correct encoding to pbcopy in node.js?](https://stackoverflow.com/questions/68788516/how-can-i-input-the-correct-encoding-to-pbcopy-in-node-js)
• It uses the same pbcopy command as you do.
• The question is related to Node.js which is used by SC to execute shell commands under the hood.
• There's one answer, and it says that LC_CTYPE environment variable should be set to UTF-8.

So if it's not UTF-8 already, please put this at the very beginning of your shell command: export LC_CTYPE="UTF-8". Does it work now? If LC_CTYPE is not UTF-8 when executing commands via SC, then I think it's something that I need to fix in the SC plugin.

Zsh as a "login" shell

SC's "built-in" configuration for Zsh executes /bin/zsh -c without an -l switch. The -l switch (if it was present) would tell Zsh to load profile files (i.e. ~/.zprofile and ~/.zlogin) before executing a command.

These profile files may set the LC_CTYPE / LANG environment variables to UTF-8. Can you please check if one of those files contains LC_CTYPE or LANG assignments?

To include the -l switch when invoking Zsh with SC, can you please try to:

1. Create a custom shell configuration in SC.
2. Shell name: Customized Zsh (or whatever).
3. Description: Not needed.
4. Executable binary path: /bin/zsh
5. Shell arguments:

   -l
   -c
   {{!shell_command_content}}
   

6. Host operating system: macOS
7. Shell's operating system: macOS
8. Special characters escaping: Unix shell style with \ as escape character
9. Path translator: Not needed.
10. Wrapper for shell command: Not needed.

Now please make your shell command use this Customized Zsh as its shell and try if echo "{{file_path:absolute}}" | pbcopy works correctly?

[Erimus-Koo]

I can not direct output _LANG or LC_CTYPE, but echo $LANG output en_US.UTF-8.
And if I use locale in both zsh and bash, output is same:

> locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

But while I use this in SC, there's a different output.

LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Maybe you notice that I've used zsh in SC, but the output is a C. I think that is the problem.

And I tried to set the LC_CTYPE, finally the CORRECT name "pbcopied" to the clipboard. 🎉🎉🎉

export LC_CTYPE="en_US.UTF-8" && locale && echo {{file_path:absolute}} | pbcopy

• the step "locale" could be removed.

[Taitava]

Good that export LC_CTYPE="en_US.UTF-8" fixes the problem! 😎

---

LANG=""
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

I found an explanation to the letter "C":

C means "use whatever locale is hard coded" (and since most *NIX programs are written in C, it's called C). However, it is usually not an UTF-8 locale.

I still need to learn what those different LC_ locale environment variables do. The rest of this post contains questions for myself that I need to do some research on.

Even though export LC_CTYPE="en_US.UTF-8" fixes the problem for you, I think that the SC plugin needs further development so that it can provide an easier solution to set the correct encoding. I need to think about different solutions:

• Could the plugin set LC_CTYPE to en_US.UTF-8 for all users by default? How to determine the language part en_US, should it be Obsidian's language or customizable by user?
• Could the plugin just offer a way to set environment variables? It could then have LC_CTYPE set to en_US.UTF-8 by default, and user could alter it if needed.

Or maybe use LC_ALL instead of LC_CTYPE, as it should override all those LC_ environment variables.

Do these LC_ variables work for Windows too, or only for macOS and Linux? Does CMD.EXE support UTF-8?

---

P.S. I'll move this discussion from Q & A to Debugging and testing to reflect the fact that the ultimate goal now is to improve the plugin.

[Erimus-Koo]

Normally, if you can read the LC_ from the shell, and with the same value, that should be the best.

In my case, I use en_US zh_CN ja_JP, so forcing the value to en_US.utf-8 is probably not a good solution. But the two parts around the dot, look like the left half is lang and location, and the right part is encoding. So that could be UTF-8 GBK or something else.

So I think keeping the LC in the environment should be better.
And I still do not understand why it becomes C.

[Erimus-Koo]

And I thought it again,

1. get {{file_path}}
2. copy
   2.1. Use SC output to clipboard, that is correct. So the first step should be OK.
   2.2. Using pbcopy got an error, probably because it used the wrong LC_.

And tried this

# use zsh and load .zshrc (output correct)
# I've set LANG and LC_ALL in .zshrc, that's the reason.
source ~/.zshrc && echo {{file_path:absolute}} | pbcopy

[Taitava]

Normally, if you can read the LC_ from the shell, and with the same value, that should be the best.

In my case, I use en_US zh_CN ja_JP, so forcing the value to en_US.utf-8 is probably not a good solution.

Oh, that's true. I should read the language part from the environment variables.

Obsidian gets passed at least some environment variables when it's launched. This means I can read some environment variables without needing to execute a shell command that reads them. In JavaScript/NodeJS, they can be read from process.env.

However, it looks like I can't read all the LC_* variables from process.env:

• On Linux, I was able to read LANG, LC_MONETARY, LC_NUMERIC, and LC_TIME. For example LC_CTYPE and LC_ALL seem to be unavailable. (Maybe on macOS Obsidian gets the same set of environment variables as on Linux?)
• On Windows, none of LC_* or LANG were available. But anyway it's always a step forward if these encoding problems can be improved on some operating systems (Linux & macOS in this case), even if one operating system needs to wait its fix a bit longer.

That said, I think I could read values of LANG, LC_MONETARY, LC_NUMERIC, and LC_TIME individually, and:

1. Retrieve the language part from each one.
2. Add .UTF-8 to the end of each language part, replacing whatever encoding there were originally.

I'm not sure if that's enought to fix the problem, or does it still require setting LC_CTYPE or LC_ALL.

And I still do not understand why it becomes C.

When it's C, it uses some default locale settings originating from the programming language C. But I can't either say why it becomes C. And I don't know what the C locale actually is in practise.

[Taitava]

2. 2.2. Using pbcopy got an error, probably because it used the wrong LC_.

What was the error message? 🙂

# use zsh and load .zshrc (output correct)
# I've set LANG and LC_ALL in .zshrc, that's the reason.
source ~/.zshrc && echo {{file_path:absolute}} | pbcopy

Good that you are including ~/.zshrc, I guess it should improve the compatibility between executing shell commands by using this plugin vs. executing normally via a terminal.

(Also, if you ever have some PATH problems (i.e. programs not found when trying to execute), you could try source ~/.zprofile in addition to source ~/.zshrc. But that's another story. 🙂 )

[Erimus-Koo]

source ~/.zshrc doesn't help because if I remove these two lines(not everyone added them),

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

then the locale output is

LANG=""
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The above is in "terminal.app"

And if I run the same code in "kitty" kitty, that's always "utf-8", it's too complex for the environment. Many steps changed the LC_.

[Taitava]

Ok.

So, to conclude, I still need to think this through to come up with a setting that is easy to use (to fix the encoding problem as automatically as possible while still taking different locales and different shells & operating systems into account) and to also easily turn off if it doesn't work for someone.

I'll post to this discussion when I come up with a solution (or at least some progress), but that might take time.