reference file is not updated when new citations are added

[newhallroad]

I have a markdown file test.md like this:

---
title: zotxt test
zotero-bibliography: bib.json
---

[@soper_LegalTheoryObligation_1977]

If I run pandoc like this:

pandoc test.md -o test.docx --standalone --lua-filter=pandoc-zotxt.lua  --citeproc

then everything works fine. pandoc-zotxt creates a references file called bib.json and adds passes it as a bibliography to citeproc. The outcome is a properly formatted document with all citations in place as expected.

However, if I add another citation to test.md, things go wrong. bib.json is not updated with the new reference.

Additionally, pandoc-zotxt no longer passes the reference file to citeproc, because even the references that were properly processed before are now missing, and replaced with the citation keys appended by question marks, implying that citeproc cannot find the references. If I add bibliography=bib.json to the metadata, then citeproc can find the file again, but still doesn't process the citation(s) added after the original creation of the reference file, which is never updated.

I have tested this with docx and html output with the same result. I have also tested passing citeproc as a lua filter rather than a regular switch. That doesn't make a difference either.

Thanks for a wonderful project. I'd be grateful for your thoughts.

[odkr]

Thank you for taking the time to open the issue! This is rather odd. What version of Pandoc are you using? Could you post a minimal (non-)working example, so that I can re-produce the behaviour?

[newhallroad]

Thank you for your response. I am sorry you have been sick and I hope
you are feeling well. Here's a full example.

I am using pandoc 3.1.6.1 with Lua 5.4

I have a directory with one file, test.md. (I suppress the
bibliography to make the example simpler):

---
title: zotxt test
zotero-bibliography: bib.json
suppress-bibliography: true
---

[@mailund_IntroducingMarkdownPandoc_2019]

I run this command from the Windows 10 cmd cli, which returns the
following output:

$> pandoc test.md -o test.html --lua-filter=pandoc-zotxt.lua  --citeproc
pandoc-zotxt.lua: made temporary directory pdz-f596cad43e80b7c3.
pandoc-zotxt.lua: removed pdz-f596cad43e80b7c3.

At this point everything is running as expected. I now have a new
file, bib.json, which has the single refernce in it. I also have
test.html which lists that single reference in the default Chicago
author-date format:

(Mailund 2019)

Now I add a new reference to test.md, which now looks like this:

---
title: zotxt test
zotero-bibliography: bib.json
suppress-bibliography: true
---

[@mailund_IntroducingMarkdownPandoc_2019]

[@ierusalimschy_ProgrammingLua_2003]

If I re-run exactly the same command again, I get the following:

$> pandoc test.md -o test.html --lua-filter=pandoc-zotxt.lua  --citeproc
pandoc-zotxt.lua: made temporary directory pdz-9e6f9bf561015ee4.
pandoc-zotxt.lua: removed pdz-9e6f9bf561015ee4.
pandoc-zotxt.lua: File exists
[WARNING] Citeproc: citation ierusalimschy_ProgrammingLua_2003 not found
[WARNING] Citeproc: citation mailund_IntroducingMarkdownPandoc_2019 not found

Now, the output file test.html suggests that citeproc was unable to
process either citation, even the one that was previously correct:

(mailund_IntroducingMarkdownPandoc_2019?)

(ierusalimschy_ProgrammingLua_2003?)

bib.json is still in the same location but it has not been updated; it still only includes the first reference. So it seems that the second time I run the command, pandoc-zotxt does not add the second reference to bib.json and also that it does not specify bib.json as a bibliography file in the metadata.

I assume this behavior is unwanted as it means the filter can only work once on a given document. If the document is edited by adding additional references, it seems to break.

Please let me know if you need any other information. Thank you again for your kind attention and for your generous work for the community.

[odkr]

Yes, this is definitely a bug.

Here's what I think is happening: The bibliography file is not updated directly (What if the file is read by another process while it's being changed? What if updating the file fails mid-way?); instead, the new bibliography is written to a temporary file and then moved into place. However, it appears that os.rename fails on Windows if the target file already exists (the target file is overwritten silently on POSIX systems). I don't have access to a Windows system, so I never noticed.

I'm a bit swamped at the moment, so I don't know when I'll get around to fixing this.

However, here's something you can try for the time being: Open the file filters\pandoc-zotxt.lua\share\lua\5.4\pancake.lua in your Pandoc data directory with a text editor and search for file_write =.

There should be a code block that reads:

    if not pandoc.types or PANDOC_VERSION < {2, 8}
        then file_write = __file_write_legacy
        else file_write = __file_write_modern
    end

Change that code block to:

    -- if not pandoc.types or PANDOC_VERSION < {2, 8}
    --    then file_write = __file_write_legacy
    --    else file_write = __file_write_modern
    -- end
    file_write = write

This way, the bibliography should be updated directly, by-passing the need to call os.rename.

Let me know if that works (but don't close the issue, I'll do that once this is actually fixed)!

[newhallroad]

Thank you for responding so quickly. Your diagnosis makes a lot of sense and I bet you are right.

Oddly, I don't have filters\pandoc-zotxt.lua\share\lua\5.4\pancake.lua. In the latest release, v1.2.0, (both .zip and .tgz versions,) that directory only holds the directory lunajson/ and a file called lunajson.lua.

I do see pancake.lua in that directory in the code on github, but not in the latest release. What would you advise?

[odkr]

You're welcome!

What would you advise?

That I should work on my time management, so that I get around to finishing the next release ;-).

Okay, so the file you need to edit is filters\pandoc-zotxt.lua\pandoc-zotxt.lua. The code you need to change is also different, if subtly.

Again, search for file_write = . That should take you to line 1981. There should be a code block that reads:

if not pandoc.types or PANDOC_VERSION < {2, 8}
    then file_write = file_write_legacy
    else file_write = file_write_modern
end

Change that to:

-- if not pandoc.types or PANDOC_VERSION < {2, 8}
--    then file_write = file_write_legacy
--    else file_write = file_write_modern
-- end
file_write = write

Let me know whether this fixes the problem.

[newhallroad]

Unfortunately, that edit didn't solve the problem. The outcome was exactly the same as before.
Incidentally, if I don't include zotero-bibliography in the metadata, the filter works as expected, and I can add references with no problem. This supports your idea that this has something to do with writing to bib.json. Unfortunately, that solution is untenable because without caching references, the filter is very slow even with incremental changes.

Thanks for persevering with this. This is a very useful filter and I'd like to do what I can to help you get it working on windows. Let me know if there's anything else you'd like me to try.

[odkr]

The outcome was exactly the same as before.

Did it even give you the same error messages? Including "pandoc-zotxt.lua: File exists"?

[newhallroad]

Yes, the same error messages:

pandoc-zotxt.lua: made temporary directory pdz-37745e654c3fe9bd.
pandoc-zotxt.lua: removed pdz-37745e654c3fe9bd.
pandoc-zotxt.lua: File exists
[WARNING] Citeproc: citation ierusalimschyProgrammingLua2003 not found
[WARNING] Citeproc: citation mailundIntroducingMarkdownPandoc2019 not found

[odkr]

Hmm, odd. Say, I just noticed that the path I gave you, filters\pandoc-zotxt.lua\pandoc-zotxt.lua, is wrong. I assume you noticed and edited filters\pandoc-zotxt.lua instead, yes? Depending on how you installed the filter, there may be two copies of pandoc-zotxt.lua, and only one of them is used. Could you check?

I’m grasping at straws here, because I have no idea which other function could complain about the file already existing; and the two instances of os.rename are in file_write_modern and file_write_legacy -- which should no longer be called.

[newhallroad]

You're a genius! That was the problem - I had only edited the file in the filters/ directory, not the pandoc-zotxt.lua that I had copied up a level during install. This now produces the expected behaviour.

Thank you for your troubleshooting and for the project as a whole. This will save a lot of bother in my workflow. Thank you!

[odkr]

Happy to help! :-) I'll release a proper fix at some point in the future. It'll be a while though.