Obsidian Converter #1

LuccaHellriegel · 2022-09-13T22:20:24Z

Progress tracker: #3

Test the converter here: https://luccahellriegel.github.io/tana-import-tools/

Added a Obsidian/Markdown converter.

Happy to hear your feedback or any misunderstandings of the format I might have had! :)
Be sure to check out the result file or the test-vault that I included: https://github.com/LuccaHellriegel/tana-import-tools/blob/main/src/converters/obsidian/tests/fixtures/full.tif.json

Bunch of fixes

vicrdguez · 2022-10-04T12:31:58Z

Is there any list with tasks that need to be completed to get this converter to the ready state? I might be able to give a hand

houshuang · 2022-10-04T12:38:22Z

@vicrdguez Creating a list would be useful. Anecdotally, I know that we should match date pages to Tana calendar nodes - however that might have to be configurable since I think at least Obsidian allows custom date formats. Also, parsing YAML frontmatter/Dataview attributes/Logseq attributes (page/block) into Tana fields would be amazing.

vicrdguez · 2022-10-04T13:00:38Z

@houshuang thanks, that is a starting point. A couple of questions:

Is all one needs to know from Tana Intermediate Format here ?
I assume Tana calendar nodes are nodes with NodeType=date. Is this correct? I don't see any other flag that could mark a node as calendar

I don't want to flood this channel with questions but those will help a lot.

Thanks!

houshuang · 2022-10-04T13:09:31Z

That should be it - feel free to ask questions. @eirikhm Can answer the question about how we set up calendar nodes.

Otherwise, the Roam importer which is in this repo is production-quality, it's exactly the code that we run when you import Roam workspaces into Tana, and it has been polished a lot - it should cover all the functionality you are after. So have a look at how it works/what it outputs.

Here's an example output from Roam with a date node:

{
  "version": "TanaIntermediateFile V0.1",
  "summary": {
    "leafNodes": 20,
    "topLevelNodes": 16,
    "totalNodes": 36,
    "calendarNodes": 1,
    "fields": 5,
    "brokenRefs": 0
  },
  "nodes": [
    {
      "uid": "U6b_lAcTE",
      "name": "Stian Håklev",
      "children": [],
      "createdAt": 1662544329796,
      "editedAt": 1662544329796,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "09-07-2022",
      "name": "09-07-2022",
      "children": [
        {
          "uid": "ZWWrfuNpM",
          "name": "Today I will write about [[59QkLn2qT]]",
          "children": [],
          "createdAt": 1662544333148,
          "editedAt": 1662544339648,
          "refs": [
            "59QkLn2qT"
          ],
          "type": "node"
        }
      ],
      "createdAt": 1662544329798,
      "editedAt": 1662544329798,
      "refs": [],
      "type": "date"
    },
    {
      "uid": "59QkLn2qT",
      "name": "Sapiens",
      "children": [
        {
          "uid": "o19tvK1_F",
          "name": "author",
          "children": [
            {
              "uid": "znf5aop3l4r5ovpwv7lw5x",
              "name": "[[QE4P4RfSx]]",
              "createdAt": 1662544348847,
              "editedAt": 1662544354458,
              "type": "node",
              "refs": [
                "QE4P4RfSx"
              ]
            }
          ],
          "createdAt": 1662544348847,
          "editedAt": 1662544354458,
          "refs": [
            "QE4P4RfSx",
            "JW7KNY-Iv"
          ],
          "type": "field"
        },
        {
          "uid": "hnK14GY4f",
          "name": "url",
          "children": [
            {
              "uid": "v0ralrepr8gsas0vrxt2f",
              "name": "<a href=\"https://www.ynharari.com/book/sapiens-2/\">https://www.ynharari.com/book/sapiens-2/</a>",
              "createdAt": 1662544356428,
              "editedAt": 1662544368984,
              "type": "node",
              "refs": []
            }
          ],
          "createdAt": 1662544356428,
          "editedAt": 1662544368984,
          "refs": [
            "JWwi9GUJz"
          ],
          "type": "field"
        },
        {
          "uid": "3QlWJUbGK",
          "name": "abstract",
          "children": [
            {
              "uid": "ghw7himh99lvm1shvvf53i",
              "name": "Homo sapiens rules the world because it is the only animal that can believe in things that exist purely in its own imagination, such as gods, states, money, and human rights.",
              "createdAt": 1662544401360,
              "editedAt": 1662544402761,
              "type": "node",
              "refs": []
            }
          ],
          "createdAt": 1662544401360,
          "editedAt": 1662544402761,
          "refs": [
            "4QwWHZUaK"
          ],
          "type": "field"
        },
        {
          "uid": "v1twphbGc",
          "name": "This is a book about the history of [[lo9mb4HNa]]",
          "children": [],
          "createdAt": 1662544341268,
          "editedAt": 1662544346607,
          "refs": [
            "lo9mb4HNa"
          ],
          "type": "node"
        },
        {
          "uid": "iTrkXOIvm",
          "name": "Reading plan:",
          "children": [
            {
              "uid": "bH982GcdX",
              "name": "Introduction ([[date:2022-09-07]])",
              "children": [],
              "createdAt": 1662544448829,
              "editedAt": 1662544488603,
              "refs": [
                "09-07-2022",
                "IrRKbsZ9I"
              ],
              "type": "node",
              "todoState": "done"
            },
            {
              "uid": "2D7vECGu0",
              "name": "Chapter 1 ([[date:2022-09-07]])",
              "children": [],
              "createdAt": 1662544459090,
              "editedAt": 1662544493778,
              "refs": [
                "09-07-2022",
                "IrRKbsZ9I"
              ],
              "type": "node",
              "todoState": "done"
            },
            {
              "uid": "Hz3m5GbwC",
              "name": "Chapter 2",
              "children": [],
              "createdAt": 1662544462818,
              "editedAt": 1662544464710,
              "refs": [
                "MmDDRH8fG"
              ],
              "type": "node",
              "todoState": "todo"
            },
            {
              "uid": "4RjIo17bX",
              "name": "Chapter 3",
              "children": [],
              "createdAt": 1662544464705,
              "editedAt": 1662544466639,
              "refs": [
                "MmDDRH8fG"
              ],
              "type": "node",
              "todoState": "todo"
            }
          ],
          "createdAt": 1662544447163,
          "editedAt": 1662544448831,
          "refs": [],
          "type": "node"
        },
        {
          "uid": "2dMYBkOSh",
          "name": "Notes:",
          "children": [
            {
              "uid": "Rjjqv92pa",
              "name": "The fact that [[3QlWJUbGK]] is debatable.",
              "children": [
                {
                  "uid": "Tlu8uBTS7",
                  "name": "(some people believe that this is completely true [*]([[Rjjqv92pa]])) [#controversy]([[-ticiabxm]]) [#debatable fact]([[hlhLLXdw6]])",
                  "children": [],
                  "createdAt": 1662544443319,
                  "editedAt": 1662544615068,
                  "refs": [
                    "Rjjqv92pa",
                    "-ticiabxm",
                    "hlhLLXdw6"
                  ],
                  "type": "node"
                }
              ],
              "createdAt": 1662544419609,
              "editedAt": 1662544443324,
              "refs": [
                "3QlWJUbGK"
              ],
              "type": "node"
            },
            {
              "uid": "wfAWWoTUY",
              "name": "<b>Was life really better before the introduction of farming?</b>",
              "children": [
                {
                  "uid": "e3jlFrFnX",
                  "name": "<mark>Fewer people to compete for the food</mark>",
                  "children": [],
                  "createdAt": 1662544636491,
                  "editedAt": 1662544652447,
                  "refs": [],
                  "type": "node"
                },
                {
                  "uid": "G-vB-1yqP",
                  "name": "<i>But also shorter human lifespans?</i>",
                  "children": [],
                  "createdAt": 1662544652442,
                  "editedAt": 1662544659179,
                  "refs": [],
                  "type": "node"
                }
              ],
              "createdAt": 1662544622155,
              "editedAt": 1662544636497,
              "refs": [],
              "type": "node"
            }
          ],
          "createdAt": 1662544417690,
          "editedAt": 1662544419612,
          "refs": [],
          "type": "node"
        },
        {
          "uid": "aEqgGjpox",
          "name": "Questions:",
          "children": [
            {
              "uid": "sU9_lL-9s",
              "name": "Does the fact that [Harari]([[QE4P4RfSx]]) is [Israeli]([[EfPG1VRQF]]) change his perspective on world history? (<a href=\"https://en.wikipedia.org/wiki/History_of_Israel\">More info about Israeli history</a>)",
              "children": [],
              "createdAt": 1662544690524,
              "editedAt": 1662544731944,
              "refs": [
                "QE4P4RfSx",
                "EfPG1VRQF"
              ],
              "type": "node"
            }
          ],
          "createdAt": 1662544686598,
          "editedAt": 1662544690526,
          "refs": [],
          "type": "node"
        }
      ],
      "createdAt": 1662544339651,
      "editedAt": 1662544339653,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "lo9mb4HNa",
      "name": "humanity",
      "children": [],
      "createdAt": 1662544346611,
      "editedAt": 1662544346612,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "QE4P4RfSx",
      "name": "Yuval Noah Harari",
      "children": [
        {
          "uid": "hdl4VUGSd",
          "name": "nationality",
          "children": [
            {
              "uid": "1psw9bcxzabtvnk5ps8hb",
              "name": "[[EfPG1VRQF]]",
              "createdAt": 1662544408417,
              "editedAt": 1662544413016,
              "type": "node",
              "refs": [
                "EfPG1VRQF"
              ]
            }
          ],
          "createdAt": 1662544408417,
          "editedAt": 1662544413016,
          "refs": [
            "EfPG1VRQF",
            "H2uDM4W1M"
          ],
          "type": "field"
        },
        {
          "uid": "9z0iCdaSd",
          "name": "image",
          "children": [
            {
              "uid": "kciee0j9h0lcg458t7eyks",
              "name": "image",
              "createdAt": 1662544546195,
              "editedAt": 1662544584349,
              "type": "image",
              "mediaUrl": "https://upload.wikimedia.org/wikipedia/commons/f/f9/Yuval_Noah_Harari_cropped.jpg",
              "refs": []
            },
            {
              "uid": "dj0t0xcpjra9p2e4y4n5zi",
              "name": "[[06xect7gchoitco3ohnmixd]]",
              "createdAt": 1662544546195,
              "editedAt": 1662544584349,
              "type": "node",
              "refs": [
                "06xect7gchoitco3ohnmixd"
              ]
            }
          ],
          "createdAt": 1662544546195,
          "editedAt": 1662544584349,
          "refs": [
            "kciee0j9h0lcg458t7eyks",
            "nikQwAgkZ"
          ],
          "type": "field"
        }
      ],
      "createdAt": 1662544354458,
      "editedAt": 1662544354459,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "JW7KNY-Iv",
      "name": "author",
      "children": [],
      "createdAt": 1662544354458,
      "editedAt": 1662544354459,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "qGKp1Ufdr",
      "name": "price",
      "children": [],
      "createdAt": 1662544358099,
      "editedAt": 1662544358099,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "JWwi9GUJz",
      "name": "url",
      "children": [],
      "createdAt": 1662544368987,
      "editedAt": 1662544368988,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "4QwWHZUaK",
      "name": "abstract",
      "children": [],
      "createdAt": 1662544381238,
      "editedAt": 1662544381238,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "EfPG1VRQF",
      "name": "Israel",
      "children": [],
      "createdAt": 1662544413018,
      "editedAt": 1662544413018,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "H2uDM4W1M",
      "name": "nationality",
      "children": [],
      "createdAt": 1662544413018,
      "editedAt": 1662544413018,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "IrRKbsZ9I",
      "name": "DONE",
      "children": [],
      "createdAt": 1662544458944,
      "editedAt": 1662544458945,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "MmDDRH8fG",
      "name": "TODO",
      "children": [],
      "createdAt": 1662544464005,
      "editedAt": 1662544464006,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "nikQwAgkZ",
      "name": "image",
      "children": [],
      "createdAt": 1662544548803,
      "editedAt": 1662544548804,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "-ticiabxm",
      "name": "controversy",
      "children": [],
      "createdAt": 1662544608280,
      "editedAt": 1662544608280,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "hlhLLXdw6",
      "name": "debatable fact",
      "children": [],
      "createdAt": 1662544615070,
      "editedAt": 1662544615071,
      "refs": [],
      "type": "node"
    },
    {
      "uid": "06xect7gchoitco3ohnmixd",
      "name": "kciee0j9h0lcg458t7eyks",
      "createdAt": 1662544546195,
      "editedAt": 1662544584349,
      "type": "node",
      "refs": [
        "06xect7gchoitco3ohnmixd"
      ]
    }
  ],
  "attributes": [
    {
      "name": "author",
      "values": [
        "[[Yuval Noah Harari]]"
      ],
      "count": 1
    },
    {
      "name": "url",
      "values": [
        "https://www.ynharari.com/book/sapiens-2/"
      ],
      "count": 1
    },
    {
      "name": "abstract",
      "values": [
        "Homo sapiens rules the world because it is the only animal that can believe in things that exist purely in its own imagination, such as gods, states, money, and human rights."
      ],
      "count": 1
    },
    {
      "name": "nationality",
      "values": [
        "[[Israel]]"
      ],
      "count": 1
    },
    {
      "name": "image",
      "values": [
        "image",
        "[[kciee0j9h0lcg458t7eyks]]"
      ],
      "count": 1
    }
  ]
}

vicrdguez · 2022-10-04T13:22:31Z

@houshuang that will do it thanks! I see calendar nodes are date nodes whose uid=name=date of the day.

vicrdguez · 2022-10-04T17:00:54Z

@houshuang @LuccaHellriegel I'm trying to push some changes but I get a 403. Looks like a need access to the branch

houshuang · 2022-10-04T17:52:40Z

Fork it and make a pr

LuccaHellriegel · 2022-10-04T19:03:26Z

FYI I have opened an issue to collect TODOs: #3

eirikhm · 2022-10-04T19:09:45Z

I've added some sample files in #4. I have a PR incoming that makes uid, createdAt and modifiedAt optional

livekth · 2022-11-02T15:24:58Z

Hello @LuccaHellriegel ! Thank you for taking the time to build this.

I tried to use your converter, but I ended up having most of my notes missed. Instead of them, I got some notes with broken titles - I guess it is because my note titles' are in Korean. Does your converter handle UTF-8 encoded notes? If not, do you have any plan to add the feature to the converter?

LuccaHellriegel · 2022-11-02T15:44:34Z

@livekth could you send me / paste an example file so that I can debug the issue? I read all files as UTF-8.

livekth · 2022-11-02T23:57:10Z

For example, the above two are the same notes: first one is from Obsidian, and the another one is from Tana.

So the original title in Korean is:
📔 노트 정리 2022년 1월 16일

But converted to:
≡ƒôö ßäéßà⌐ßäÉßà│ ßäîßàÑßå╝ßäàßà╡ 2022ßäéßàºßå½ 1ßäïßà»ßå» 16ßäïßà╡ßå»

If I compare them word wise:

📔 → ≡ƒôö
노트 → ßäéßà⌐ßäÉßà│
정리 → ßäîßàÑßå╝ßäàßà╡
2022년 → 2022ßäéßàºßå½
1월 → 1ßäïßà»ßå»
16일 → 16ßäïßà╡ßå»

The characters on the right looks quite different from what I saw the broken Koreans in other applications because of encoding/decoding problem.

I am also attaching the original MD file for your reference.

📔 노트 정리 2022년 1월 16일.md

daviddelven · 2022-11-05T15:03:01Z

Hello! Do you know what am I doing wrong? After importing the *.tif.json (from Obsidian, once I've converted it using your converter) as a new workspace and cloning the nodes into my own personal workspace, the predefined tags within the imported data seem not to be mapped and recognized as the Schema ones.

LuccaHellriegel · 2022-11-05T19:04:08Z

Hello! Do you know what am I doing wrong? After importing the *.tif.json (from Obsidian, once I've converted it using your converter) as a new workspace and cloning the nodes into my own personal workspace, the predefined tags within the imported data seem not to be mapped and recognized as the Schema ones.

For other readers: Tana will add a way to merge tags on import as discussed in the slack.

LuccaHellriegel · 2022-11-05T19:10:03Z

@livekth you are using the online zip converter right? the webpage? Can you check that the filenames are correct inside the zip?

livekth · 2022-11-05T22:31:20Z

@LuccaHellriegel Yes! I used the online converter. And I also checked the file names inside the zip by unzipping it. I have successfully got the files with the correct names.

And I found one interesting thing while I tested the zip file: even a file with the English name, the emoji in the name was also broken:

The emoji 📚 is broken and converted to "≡ƒôÜ"

The original Obsidian file

By the way, can you check out your online unzipper if it supports Koreans and emoji?

livekth · 2022-11-06T09:01:49Z

I solved the problem!

The problem was on my side: when I used Bandizip (a Korean archive manager), everything worked. Thank you so much for the converter, Lucca!

livekth · 2022-11-14T09:53:53Z

Lucca, I have one question - can I import all Obsidian daily notes to the Tana's Calendar node?

LuccaHellriegel and others added 10 commits September 10, 2022 00:36

added functions for splitting up markdown files

da01abb

added conversion of obsidan nodes to tana nodes

6751816

added integration test for obsidian vault converter

bc5d19c

added obsidian converters to runner

39ad1d5

added obsidian converter to scripts

398a40c

Merge branch 'tanainc:main' into main

fd0062f

remove basic markdown / empty space in obsidian converter

883434c

WIP

baa4b1a

Fixes suggested by Lucca

f7632c1

Merge pull request #1 from tanainc/sh/StianImprovements

f033959

Bunch of fixes

houshuang mentioned this pull request Oct 1, 2022

Add Logseq to import tools #2

Closed

LuccaHellriegel added 3 commits October 3, 2022 20:17

fix sub-outline node detection

06613bb

remove potentially wrong post processing from markdown node extraction

3752622

improved hierarchy detection for numbered lists and paragraphs

8f9e506

LuccaHellriegel and others added 7 commits October 5, 2022 09:26

introduce VaultContext

785ca45

reworked the obsidian converter to include the file structure

a77d27c

added a README for the obsidian converter

fbd98db

shorten file names

eb12434

Merge branch 'tanainc:main' into main

fc64be9

helper functions to support handling of heading links in Obsidian

7f87f79

added support for block refs in Obsidian

b41db85

Lucca Hellriegel and others added 11 commits October 17, 2022 15:18

Obsidian: replace recursive functions with loop to avoid overflow (2)

d2e5695

Obsidian: added basic dataview attribute support

c21fd7c

Obsidian: support links in Dataview attributes

66665c2

increase CI node version

a4863ff

Obsidian: add refs to image nodes

0f0ca63

Merge branch 'tanainc:main' into main

ae108d9

Obsidian: convert daily notes link to the proper format

5505258

Obsidian: convert daily notes link to the proper format (2)

332b314

Obsidian: added log for empty data view attribute bug

c89444e

Obsidian: support block ids in multiline content

73579f8

Obsidian: support block ids in multiline content (2)

671f854

LuccaHellriegel added 2 commits November 5, 2022 19:52

Obsidian: added brute force filtering of invalid obsidian tags

8085ee1

Obsidian: added more invalid symbols to tag detection filter

ae890ad

Obsidian: be more lenient in multi-line bullet content

b616690

LuccaHellriegel added 5 commits November 29, 2022 19:26

Obsidian: add logging to path finding

70d8bff

Obsidian: added "Inofficial" to web page

989846f

Obsidian: added logging

97bc486

Obsidian: added logging

fe72309

Obsidian: filter out more invalid links

d686343

eirikhm pushed a commit that referenced this pull request Aug 11, 2023

Merge pull request #1 from cheslip/main

76a81a8

LuccaHellriegel closed this Sep 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Obsidian Converter #1

Obsidian Converter #1

LuccaHellriegel commented Sep 13, 2022 •

edited

Loading

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

LuccaHellriegel commented Oct 4, 2022

eirikhm commented Oct 4, 2022

livekth commented Nov 2, 2022

LuccaHellriegel commented Nov 2, 2022 •

edited

Loading

livekth commented Nov 2, 2022

daviddelven commented Nov 5, 2022

LuccaHellriegel commented Nov 5, 2022

LuccaHellriegel commented Nov 5, 2022

livekth commented Nov 5, 2022 •

edited

Loading

livekth commented Nov 6, 2022

livekth commented Nov 14, 2022

Obsidian Converter #1

Obsidian Converter #1

Conversation

LuccaHellriegel commented Sep 13, 2022 • edited Loading

Progress tracker: #3

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

vicrdguez commented Oct 4, 2022

houshuang commented Oct 4, 2022

LuccaHellriegel commented Oct 4, 2022

eirikhm commented Oct 4, 2022

livekth commented Nov 2, 2022

LuccaHellriegel commented Nov 2, 2022 • edited Loading

livekth commented Nov 2, 2022

daviddelven commented Nov 5, 2022

LuccaHellriegel commented Nov 5, 2022

LuccaHellriegel commented Nov 5, 2022

livekth commented Nov 5, 2022 • edited Loading

livekth commented Nov 6, 2022

livekth commented Nov 14, 2022

LuccaHellriegel commented Sep 13, 2022 •

edited

Loading

LuccaHellriegel commented Nov 2, 2022 •

edited

Loading

livekth commented Nov 5, 2022 •

edited

Loading