zotero.lua is not producing scannable cite documents with pandoc < 2.9.2

[kjambunathan]

filter broken

**pandocc is installed**

~$ pandoc --version
pandoc 2.9.1.1
Compiled with pandoc-types 1.20, texmath 0.12.0.2, skylighting 0.8.4
Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc
Copyright (C) 2006-2019 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

pandoc datadir has pandoc-zotxt.lua and zotero.lua filters

~$ ls /home/kjambunathan/.local/share/pandoc/filters
pandoc-zotxt.lua  pandoc-zotxt.lua-0.3.17  zotero.lua

A very simple md file

~$ cat ww.md
@wrigstad2017mastery

Conversion to using pandoc-zotxt works fine => citekey is present in Zotero

~$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain


Wrigstad and Castegren (2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

Try using zotero.lua to produce 'scannable-cite` ODT file

(cf. https://retorque.re/zotero-better-bibtex/exporting/pandoc/)

~$ pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=zotero.lua -o ww.odt ww.md
~$ rm *.xml mimetype

content.xml is not OK. I see no scannable cite. The citekey is literally carried over with no translations whatsoever.

~$ unzip ww.odt
Archive:  ww.odt
 extracting: mimetype                
  inflating: meta.xml                
  inflating: META-INF/manifest.xml   
  inflating: content.xml             
  inflating: manifest.rdf            
  inflating: styles.xml              
  inflating: settings.xml            
 extracting: Configurations2/accelerator/current.xml  
  inflating: Thumbnails/thumbnail.png  
~$ cat content.xml | tail
<office:body>
<office:text>
<text:p text:style-name="Text_20_body">@wrigstad2017mastery</text:p>
</office:text>
</office:body>
</office:document-content>

Desired output

In the Zotero UI, a quick copy of the entry using scannable cite format produces this

{ | Wrigstad, & Castegren, 2017 | | |zu:0:TH5CXWRG}

[kjambunathan]

KMFGJF27-apse

[kjambunathan]

Additional context

I have checked out your repo, and looked at the pandoc files in there.

In the Makefile, I see invocation for pandoc-zotero-live-citemarkers.lua (and NOT zotero.lua). I am wondering if you are working on improvements to zotero.lua. This paper document doesn't produce a scannable-cite version.

In the transcript of the bug, I have used one of the keys from biblio.bib in your report, and manually imported it in to zotero client.

paper:
	@rm -f *.docx *.odt *.json
	@pandoc -s --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP).docx main.md
	@pandoc -s --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP).odt main.md
	@pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=pandoc-zotero-live-citemarkers.lua -o paper$(TIMESTAMP)-scannable-cite.odt main.md

When I tried to programmatically import the biblio.bib to Zotero, I realized that your repo needs python 3. I am little bit hesitant to move to python3.

start:
	ZOTERO_IMPORT=$(PWD)/biblio.bib npm start

[retorquere]

Scannable cite was indeed broken; try the version at https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua

In the Makefile, I see invocation for pandoc-zotero-live-citemarkers.lua (and NOT zotero.lua). I am wondering if you are working on improvements to zotero.lua. This paper document doesn't produce a scannable-cite version.

zotero.lua is a bundled version that includes the json encoder/decoder, utils.lua and zotero.lua to get to a standalone filter, so you don't have to install the dependencies. pandoc-zotero-live-citemarkers.lua is the "main" code that loads and calls the other parts.

In the Zotero UI, a quick copy of the entry using scannable cite format produces this

The lua filter won't produce exactly that; the human-readable part ("Wrigstad, & Castegren, 2017") will just be the citekey. Scannable cite doesn't need that part, it's just a user convenience.

When I tried to programmatically import the biblio.bib to Zotero, I realized that your repo needs python 3. I am little bit hesitant to move to python3.

That's not changing. My test framework has a lot of assumptions expecting Python 3.8. You might want to try pyenv or conda if you really want to use the test runner.

You don't really need all that to test the pandoc stuff though. Just create a new profile (using zotero -datadir profile -P), install BBT in that profile, and import the bib file. After that, whenever you start Zotero with that profile (with -datadir profile), everything you need for the pandoc tests is present. The automated import is mainly because I do all my testing in a dedicated testing profile, and that gets erased very often.

If you don't include -datadir profile on each start of Zotero for these tests, Zotero will point to your main datadir, and the import will land in your regular Zotero library. I'm presuming you don't want that. The test runner automates that, but it's a very heavyweight process; it also rebuilds & installs BBT in the testing profile it creates.

[kjambunathan]

still doesn't work

> Scannable cite was indeed broken; try the version at https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua

Doesn't work. Here is an annotated transcript:

See what is in pandoc filters

~/.local/share/pandoc/filters$ ls

pandoc-zotxt.lua  pandoc-zotxt.lua-0.3.17  v0.3.17.zip  ww.md

pandoc-zotxt works as expected => zotero is running and the citekey indeed exists

~/.local/share/pandoc/filters$ cat ww.md

@wrigstad2017mastery

~/.local/share/pandoc/filters$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain

Wrigstad and Castegren (2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

Download the new zotero.lua

~/.local/share/pandoc/filters$ wget https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua

--2020-07-23 08:32:48--  https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/gh-1581/site/content/exporting/zotero.lua
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.152.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.152.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48420 (47K) [text/plain]
Saving to: ‘zotero.lua’

zotero.lua          100%[===================>]  47.29K  --.-KB/s    in 0.1s

2020-07-23 08:32:49 (406 KB/s) - ‘zotero.lua’ saved [48420/48420]

Use it

~/.local/share/pandoc/filters$ pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=zotero.lua -o ww.odt ww.md

Unzip odt output

~/.local/share/pandoc/filters$ unzip ww.odt

Archive:  ww.odt
 extracting: mimetype
  inflating: meta.xml
  inflating: META-INF/manifest.xml
  inflating: content.xml
  inflating: manifest.rdf
  inflating: styles.xml
  inflating: settings.xml
 extracting: Configurations2/accelerator/current.xml
  inflating: Thumbnails/thumbnail.png

Examine content.xml. Not OK -- The key comes out verbatim. No translation whatsoever to scannable cite format

~/.local/share/pandoc/filters$ cat content.xml | tail

  <office:automatic-styles>
    <style:style style:name="fr2" style:family="graphic" style:parent-style-name="Formula"><style:graphic-properties style:vertical-pos="middle" style:vertical-rel="paragraph-content" style:horizontal-pos="center" style:horizontal-rel="paragraph-content" style:wrap="none" /></style:style>
    <style:style style:name="fr1" style:family="graphic" style:parent-style-name="Formula"><style:graphic-properties style:vertical-pos="middle" style:vertical-rel="text" /></style:style>
  </office:automatic-styles>
<office:body>
<office:text>
<text:p text:style-name="Text_20_body">@wrigstad2017mastery</text:p>
</office:text>
</office:body>
</office:document-content>

[retorquere]

Can you download this, place it on the pandoc directory of the clone you have (replacing the existing Makefile), and run make 1581 (assuming you have imported biblio.bib)? The pandoc call should only output the following:

@sec:theory: not found
@sec:achievement-driven: not found
@sec:the-game: not found
@fig:matrix: not found
@sec:implementation: not found

This downloads the bundled lua filter so that we'll be sure that we're using the same filter, and applies it to the etst file. This reliably gets me a scannable cite odt.

[retorquere]

Wait, let me retool that to use your citekey so you don't have to import biblio.bib never mind, you were already using a key from biblio.bib. Yeah, so test that Makefile please.

[kjambunathan]

still doesn't work

> Can you download [this](https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/pandoc/Makefile), place it on the `pandoc` directory of the clone you have (replacing the existing `Makefile`), and run `make 1581` (assuming you have imported biblio.bib)? The pandoc call should _only_ output the following: > > ``` > @sec:theory: not found > @sec:achievement-driven: not found > @sec:the-game: not found > @fig:matrix: not found > @sec:implementation: not found > ```

I am getting a different output, and I am NOT getting a scannable cite odt.

Reset hard to origin/master

~/src/zotero-better-bibtex/pandoc$ git log --format=oneline | head

6cba3f799647710c018ce112cb0538b0d3a4158d Makefile for #1581
abbade7b6b6d89d2a187fcacbff29045a364be0e Merge branch 'gh-1581'
8cf2a21be39e0c94078f1e9b83c5d9a60c2f4ee7 site update
5be8d55edf98b188185d1410ca1cbc60040c05cf ncu
de9953283e7436c12503d3d4d8e49e4479f64de3 ncu
c1111b0d5adbed41744d58368a9aafee8cb32eb0 Merge pull request #1580 from theworldisnotflat/patch-3
762e25e2713fbefca5fc9a4d4bce3d7847461ae1 main.md #norelease
fb5b19cc71b6153346ec1cb185a3784aea8f113f scannable cite was broken #norelease
ea9a88edbd54d9c0e555f1aa94c98e25830fbcf6 Update auto.md
93eccea835a73349432eaa3f7b95ef51ba71288a another day, another juris-m

make 1581

~/src/zotero-better-bibtex/pandoc$ make 1581
curl -L https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/site/content/exporting/zotero.lua -o zotero-bundled.lua
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48420  100 48420    0     0   8005      0  0:00:06  0:00:06 --:--:--  9969
pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T12:30:46-scannable-cite.odt main.md
unzip paper2020-07-23T12:30:46-scannable-cite.odt content.xml
Archive:  paper2020-07-23T12:30:46-scannable-cite.odt
  inflating: content.xml             
grep @wrigstad2017mastery content.xml
<text:p text:style-name="First_20_paragraph">The course is built around the achievement-driven learning methodology [@wrigstad2017mastery] (more details in [@sec:theory]). This methodology gives students more control of their learning with enough flexibility to engage them in deeper learning [@biggsTang2011]. Yet, year after year, although the course has increased in the number of students, we observed that students do not pursue higher grades. In this course, a higher grade implies the possibility of deeper learning, as students need to justify the design choices of the software-under-construction and they cannot just memorise solutions (Section explains more details).</text:p>
<text:p text:style-name="Text_20_body">To make these learning outcomes more achievable, comprehensible and accessible, Tobias Wrigstad and Elias Castegren [@wrigstad2017mastery] developed a new assessment technique, called , based on a more fine-grained collection of achievements, which taken together encompass the course’s learning outcomes, but are more achievable when taken individually. The list of achievements makes more explicit what a student needs to master in order to pass the course, but also to achieve higher grades. In a sense, achievement-driven learning refines constructive alignment to the micro level and provides a means for linking learning outcomes with assessment activities—it aims to hit a sweet spot in the constructive alignment design space.</text:p>
<text:p text:style-name="First_20_paragraph">Advanced Software Design is a masters-level course taught in the Department of Information Technology at Uppsala University. The course covers topics such as object-oriented analysis and design, domain modelling, software architecture, class and object modelling, behavioural modelling, design patterns, GRASP principles, design evaluation, and design improvement/refactoring. The course does not involve any programming, and instead uses UML, text, and oral presentation as the means for recording and communicating designs. Students taking the course are expected to have a solid background in programming, in particular in object-oriented programming. The course offers 8–10 interactive lectures describing the material and a course-long project. Students form teams on their own (of 4 members and they are encouraged to diversify the team’s skill set) and work together on the achievements. Teams get a fixed teacher assistant (TA) for the duration of the course. Each team has a weekly, thirty-minute-long meeting and feedback sessions with a TA. During the meetings team members are evaluated individually, i.e. some team members may pass an achievement while others will have to try again in the next meeting. The assessment scheme for the project is based on achievement-driven learning [@wrigstad2017mastery], as described in Section [@sec:achievement-driven]. There is a total of 21 achievements: 11 achievements of level 3, 9 achievements of level 4 and a single individual achievement of level 5. There is no exam.</text:p>
<text:p text:style-name="Text_20_body">The achievement-driven learning [@wrigstad2017mastery] uses gamification to force students to solve a puzzle in order to optimise their time. This work extends achievement-driven learning, introducing explicit gaming elements to foster competition, engagement and the possibility of deeper learning.</text:p>
<text:p text:style-name="First_20_paragraph">We have added gaming elements to the course, based on the achievement-driven learning methodology [@wrigstad2017mastery], to motivate students to get higher grades. The gaming elements provide enjoyment and motivational factors. We have built a regression model where enjoyment and motivation explain 55% of the variation in grades, whereby motivation drives the overall effect of the model, meaning that students who were motivated by the game also got higher grades. The link between enjoyment and grades, however, was insignificant. A future research direction could be to further explore the motivational factors that drive higher grades. Are students motivated intrinsically or extrinsically by opt-in gamification elements? What is it that drives motivation when it comes to gamification? If enjoyment, which was insignificant in our model, is indeed an intrinsic motivational factor, as suggested by [@RYAN200054], our results suggest that intrinsic motivational factors may be less likely to influence students’ grades after all. Our results provide a hint that extrinsic motivational factors in form of achievements may be more important in students’ motivation. However, future research is needed to corroborate those suggestions.</text:p>

Create a simple ww.md and ensure that citekey wrigstad2017masteryexists

~/src/zotero-better-bibtex/pandoc$ cat ww.md
cat: ww.md: No such file or directory

~/src/zotero-better-bibtex/pandoc$ echo "[@wrigstad2017mastery]" > ww.md

~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain


(Wrigstad and Castegren 2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

git status

~/src/zotero-better-bibtex/pandoc$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	content.xml
	ww.md
	zotero-bundled.lua

nothing added to commit but untracked files present (use "git add" to track)

[kjambunathan]

No luck even after installing 5.2.45.6838.xpi.

This is what I did.

1. installed https://github.com/retorquere/zotero-better-bibtex/releases/download/builds/zotero-better-bibtex-5.2.45.6838.xpi
2. Re-started zotero UI
3. Waited until the initialization is over
4. Re-ran make 1581.

[retorquere]

I have updated the Makefile to only build your minimal ww.md. I still reliably get scannable cite. What version of pandoc are you using?

I appreciate you are trying to include as much relevant info as possible, but your posts are very overwhelming to read for me. Could you use bold instead of ## headings, and maybe a little more selective output? And no all-caps please. I'm an oldschool internet habitant from the usenet days, and allcaps sounds like shouting in my mind, I can't help it.

No need to install 6838, sorry about that post, that was automated. Nothing has changed in BBT, so the normal release will suffice.

[kjambunathan]

I skipped the lua script, and hit the URL directly

I see

TypeError: citationKeys is undefined[1]

The above error is with /zotero-better-bibtex-5.2.45.6838.xpi

~/src/zotero-better-bibtex/pandoc$ curl -s http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true&translator=jzon&citationKeys=wrigstad2017mastery

[1] 16130
[2] 16131
TypeError: citationKeys is undefined[1]-  Done                    curl -s http://127.0.0.1:23119/better-bibtex/export/item?pandocFilterData=true
[2]+  Done                    translator=jzon

~/src/zotero-better-bibtex/pandoc$ pandoc -s --lua-filter=pandoc-zotxt.lua -F pandoc-citeproc ww.md -t plain


(Wrigstad and Castegren 2017)

Wrigstad, Tobias, and Elias Castegren. 2017. “Mastery Learning-Like
Teaching with Achievements.” In SPLASH-E.

[kjambunathan]

Quick upate. Scannable cite does NOT work with test build 5.2.45.6841 as well.

[retorquere]

Sorry about those test builds. You can ignore them; nothing has changed in BBT for this. I'm just checking in changes to the Makefile.

[retorquere]

TypeError: citationKeys is undefined[1]

You have to quote the URL. The & in the URL means "execute in background" to the shell when not quoted.

[retorquere]

Yeah I know but you get the same identifiers, and inside the Lua filters they look differently -- citation nodes in the AST as a Lue filter sees it have an id, not an citationId, and these nodes have a .t property that you find their type name on, and that's not present in the json/native AST dump.

[kjambunathan]

This is a pandoc issue.

zotero.lua doesn't work with this version of pandoc-2.9.1.1, specifically the version that comes with Debian sid

~$ pandoc --version
pandoc 2.9.1.1
Compiled with pandoc-types 1.20, texmath 0.12.0.2, skylighting 0.8.4
Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc
Copyright (C) 2006-2019 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

~$ dpkg -l | grep pandoc
ii  libghc-pandoc-citeproc-data           0.16.4.1-1                         all          Pandoc support for Citation Style Language - data files
ii  pandoc                                2.9.1.1-2                          amd64        general markup converter
ii  pandoc-citeproc                       0.16.4.1-1+b1                      amd64        Pandoc support for Citation Style Language - tools
ii  pandoc-data                           2.9.1.1-2                          all          general markup converter - data files

zotero.lua works with pandoc 2.10, specifically latest binary release from https://github.com/jgm/pandoc/releases/tag/2.10

~$ dpkg -l | grep pandoc
ii  libghc-pandoc-citeproc-data           0.16.4.1-1                         all          Pandoc support for Citation Style Language - data files
ii  pandoc                                2.10-1                             amd64        general markup converter
ii  pandoc-citeproc                       0.16.4.1-1+b1                      amd64        Pandoc support for Citation Style Language - tools
ii  pandoc-data                           2.9.1.1-2                          all          general markup converter - data files


~$ pandoc --version
pandoc 2.10
Compiled with pandoc-types 1.21, texmath 0.12.0.2, skylighting 0.8.5
Default user data directory: /home/kjambunathan/.local/share/pandoc or /home/kjambunathan/.pandoc
Copyright (C) 2006-2020 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

[kjambunathan]

@retorquere As part of the resolution, you may want to check for pandoc version in zotero.lua. Thanks for your support and patience.

I will leave one more comment, for the benefit of people who could be hitting this bug through search engines.

[kjambunathan]

This comment captures the results of a successful run.

~/src/zotero-better-bibtex/pandoc$ make 1581
curl -L https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/site/content/exporting/zotero.lua -o zotero-bundled.lua
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48420  100 48420    0     0  12785      0  0:00:03  0:00:03 --:--:-- 12785
echo "[@wrigstad2017mastery]" > ww.md
pandoc -s --metadata=zotero_scannable_cite:true --lua-filter=./zotero-bundled.lua -o paper2020-07-23T18:38:31-scannable-cite.odt ww.md
unzip paper2020-07-23T18:38:31-scannable-cite.odt content.xml
Archive:  paper2020-07-23T18:38:31-scannable-cite.odt
  inflating: content.xml             
grep @wrigstad2017mastery content.xml
<text:p text:style-name="Text_20_body">{ <text:s text:c="1" />| [@wrigstad2017mastery] | <text:s text:c="1" />| <text:s text:c="1" />| zu:0:TH5CXWRG }</text:p>

These are screenshots taken on LibreOffice

Output as produced by zotero.lua

Output after running it through Tools->ODF Scan

Output after after changing Document Preferences to use Chicago Author Date

Output after after inserting Bibliography

[retorquere]

That's going to be hard to test for, since it's not particular to the pandoc version. Pandoc 2.9 worked for me on MacOS. I'm not sure even how to test for the pandoc version in the filter.

[kjambunathan]

I am happy that I could make good progress today.

[kjambunathan]

Closing this issue.

[kjambunathan]

I'm not sure even how to test for the pandoc version in the filter.

From https://pandoc.org/lua-filters.html

PANDOC_VERSION

	Contains the pandoc version as a Version object which behaves like a
	numerically indexed table, most significant number first. E.g., for
	pandoc 2.7.3, the value of the variable is equivalent to a table {2,
	7, 3}. Use tostring(PANDOC_VERSION) to produce a version string. This
	variable is also set in custom writers.

PANDOC_API_VERSION

	Contains the version of the pandoc-types API against which pandoc was
	compiled. It is given as a numerically indexed table, most significant
	number first. E.g., if pandoc was compiled against pandoc-types
	1.17.3, then the value of the variable will behave like the table {1,
	17, 3}. Use tostring(PANDOC_API_VERSION) to produce a version
	string. This variable is also set in custom writers.
	

[retorquere]

I don't see how to ask for platform though.

[kjambunathan]

Details on a working version

~$ pandoc -s --lua-filter=./test.lua ww.md -t plain
PANDOC_VERSION2.10
PANDOC_API_VERSION1.21

[kjambunathan]

Details on a non-working version

~$ pandoc -s --lua-filter=./test.lua ww.md -t plain
PANDOC_VERSION2.9.1.1
PANDOC_API_VERSION1.20

[retorquere]

Non-working on Debian. It works just fine on MacOS. And I don't know how to test for platform.

[retorquere]

Turns out it's 2.9.2 or later that's needed. I've added a check for that.

[retorquere]

Turns out in fact the Inlines wasn't needed. I've put up a new filter, the site rebuild generally takes 10 minutes to show up.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.