Vite chokes generated *.bib files in production build?

[tombreit]

Hi,

I'm generating some *.bib-files from global data js via pagination. When building via npx eleventy --serve everything works as expected: These *.bib-files end up in the output directory (here: _site/publications/bibtex/*bib).

These bib-files are plaintext files with bibliographic data and are not meant to be further processed by vite:

@article{lorenzetto_inovacao_2022,
        author = {Lorenzetto, Andrei Meneses and Brasil, B{\' a}rbara Dayana},
        journal = {International Journal of Digital Law},
        number = {1},
        year = {2022},
        month = {7},
        note = {Number: 1},
        title = {A inova{\c c}{\~ a}o digital aplicada na formula{\c c}{\~ a}o das pol{\' i}ticas p{\' u}blicas: mecanismo de participa{\c c}{\~ a}o popular e concretiza{\c c}{\~ a}o da cidadania: Digital innovation applied to public policy formulation: popular participation mechanism and citizenship achievement},
        howpublished = {https://journal.nuped.com.br/index.php/revista/article/view/lorenzetto2022},
        volume = {3},
}

But in a production build via npx eleventy, it seems these *.bib-files get rendered by eleventy, but somehow got choked/missed/deleted by vite afterwards...:

via npx eleventy --serve

[11ty] Writing _site/index.html from ./src/index.md (njk)
[11ty] Writing _site/publications/index.html from ./src/publications/index.md (njk)
...
[11ty] Writing _site/publications/bibtex/aponte_crowdfunding_2020.bib from ./src/publications/bibtex.njk
[11ty] Writing _site/publications/bibtex/nagarathna_cybercrime_2020.bib from ./src/publications/bibtex.njk
[11ty] Benchmark     83ms  22%     1× (Data) `./src/_data/bibtex.js`
[11ty] Benchmark    164ms  44%     1× (Data) `./src/_data/publications.js`
[11ty] Copied 10 files / Wrote 21 files in 0.33 seconds (15.7ms each, v2.0.0-canary.35)
[11ty] Watching…
[11ty] Server at http://localhost:8080/

==>

_site/
├── assets
│   ├── app.js
│   ├── app.scss
├── index.html
└── publications
    ├── bibtex
    │   ├── aponte_crowdfunding_2020.bib
    │   ├── arafa_arqueologia_2020.bib
    │   ├── ...
    │   ├── stringhini_asistencia_2020.bib
    │   └── violada_colisao_2020.bib
    └── index.html

via npx eleventy

...
[11ty] Writing _site/publications/index.html from ./src/publications/index.md (njk)
[11ty] Writing _site/index.html from ./src/index.md (njk)
[11ty] Writing _site/publications/bibtex/kobus_educacao_2021.bib from ./src/publications/bibtex.njk
[11ty] Writing _site/publications/bibtex/lorenzetto_inovacao_2022.bib from ./src/publications/bibtex.njk
...
[11ty] Writing _site/publications/bibtex/nagarathna_cybercrime_2020.bib from ./src/publications/bibtex.njk
vite v4.1.1 building for production...
✓ 11 modules transformed.
../_site/index.html                                                    8.10 kB
../_site/publications/index.html                                      40.46 kB
../_site/assets/app-1033ec98.css                                      24.60 kB │ gzip: 11.06 kB
../_site/assets/app-1c6be869.js                                        1.41 kB │ gzip:  0.74 kB
[11ty] Benchmark     85ms  10%     1× (Data) `./src/_data/bibtex.js`
[11ty] Copied 10 files / Wrote 21 files in 0.76 seconds (36.2ms each, v2.0.0-beta.3)

==>

_site/
├── assets
│   ├── app-1033ec98.css
│   └── app-1c6be869.js
├── index.html
└── publications
    └── index.html

If I disable eleventy-plugin-vite or generate these files as *.html (instead of *.bib) everything works as expected.

What am I doing wrong or how do I keep my generated .bib files in a production build?

Versions

npm list
├── @11ty/eleventy-navigation@0.3.5
├── @11ty/eleventy-plugin-vite@4.0.0
├── @11ty/eleventy@2.0.0-beta.3
├── @citation-js/plugin-bibtex@0.6.6
└──  citation-js@0.6.6

[zachleat]

I believe this is a vite thing. Can you generate them inside of public/** instead? https://vitejs.dev/guide/assets.html#the-public-directory

[tombreit]

Sorry, I do not know how to to that... How should/would I reference this vite public dir?

Currently I build these files via

---
pagination:
  data: bibtex
  size: 1
  alias: bibtex
permalink: "publications/bibtex/{{ bibtex.bibtexLabel }}.bib"
layout: "layouts/bibtex.njk"
---

{{ bibtex.bibtexItem | safe }}

I've found a promising vite config option: https://vitejs.dev/config/shared-options.html#assetsinclude

  eleventyConfig.addPlugin(EleventyVitePlugin, {
    viteOptions: {
      assetsInclude: ['**/*.bib'],
    }
  });

...but this did not resolve my issue....

[tombreit]

Perhaps related:
eleventy-plugin-vite seems to choke every static asset which is mentioned in a addPassthroughCopy statement and not handled by vite, like a PDF file:

├── assets
│   ├── app.js
│   └── app.scss
├── downloads
│   └── myfile.pdf
├── eleventy.config.js
└── src
    ├── _includes
    │   └── layout.njk
    └── index.md

// eleventy.config.js
const EleventyVitePlugin = require('@11ty/eleventy-plugin-vite')

module.exports = function (eleventyConfig) {
  eleventyConfig.addPlugin(EleventyVitePlugin);
  eleventyConfig.addPassthroughCopy('assets');
  eleventyConfig.addPassthroughCopy('downloads');

  return {
    dir: {
      input: "src",
      output: "_site"
    },
  }
}

❌ Build via npx eleventy or npx eleventy --watch, got

#  _site generated via `npx eleventy`
_site/
├── assets
│   ├── index.74d94aed.css
│   └── index.b2fff5ad.js
└── index.html

✅ ...but expected (and strangely I get the desired result with the --serve option):

# _site generated via `npx eleventy --serve`
_site/
├── assets
│   ├── app.js
│   ├── app.scss
│   ├── index.74d94aed.css
│   └── index.b2fff5ad.js
├── downloads
│   └── myfile.pdf
└── index.html

Am I doing something completely wrong? Thanks in advance for any hints.

I've put together a minimal eleventy project for this setup: https://github.com/tombreit/eleventy-vite-minimal/tree/vite-and-pdfs

[yellowled]

@tombreit So if you have a public folder on the same level in your repository as your src, vite will serve files in that folder as if they were in your site's root in development and copy them to the root level of your _site in production. For instance:

public/
|-- test.txt
src/
|-- [usual 11ty stuff here

and running eleventy --serve (using vite) should serve http://localhost:8080/test.txt while running eleventy should copy public/test.txt to _site/test.txt. (I'm trying to figure out the same thing as you, this is as far as I got.)

Can you generate them inside of public/** instead?

@zachleat Do mean generate them from src/ to public/, where vite would “pick them up” and copy them to _site/? [insert short break to read the fantastic 11ty docs] Ooooooh, I think I got you – I was not aware that we can “override” the output directory, i.e. set permalink to a different directory than _site.

So for instance I have a src/robots.txt.njk with permalink: robots.txt which currently generates _site/robots.txt. With

---
permalink: "public/robots.txt"
permalinkBypassOutputDir: true
---

that files is generated at public/robots.txt, it is served at http://localhost:8080/robots.txt in development and … sadly not copied to _site/robots.txt in a production build, which seems to be expected in hindsight because it is created during said build. 😞

I seem to be missing something here …?

[freddyheppell]

I had a similar problem with PDF files. I've managed to get it to work by having this folder structure

public/
    myfile.pdf
    favicon.ico
    ...
src/
    index.njk
    ...

and then passthrough-copying public:

eleventyConfig.addPassthroughCopy("public");

This then gives the output of:

_site/
    index.html
    myfile.pdf
    favicon.ico

The passthrough copying is necessary, it doesn't work if you just put public inside src.

For things such as sitemaps and feeds, I was able to get them to work by leaving them in src with a permalink like /public/sitemap.xml.

[groenroos]

Ran into the same problem, too; and while moving static assets into the public folder does work, it doesn't help when you're hoping to generate these files with Eleventy:

---
permalink: generated.json
value: true
---
{
	"generated": {{ value }}
}

While the generated.json does get built by Eleventy, and served as normal during watch, it does not get included in the final production build.

I've tried addPassthroughCopy, adding them to Vite's assetsInclude, or to Eleventy's templateFormats, and none seem to fix production builds to match the dev build's expected behaviour. Naturally, there's no way to move this file to the public folder either, since I need Eleventy to process it.

This essentially means that Eleventy-generated pages of certain file extensions are mysteriously excluded/deleted from the final build by Vite; and it means that Vite's behaviour is significantly different between dev and prod. Neither seems acceptable default behaviour to me.

Is there a workaround for this?

[KiwiKilian]

The solution is mostly adding public/ to your permalink, did you try this approach? I can't reproduce reports here which state otherwise, works fine here. See this frontmatter:

---
permalink: public/generated.json
---

During the prod build, vite only includes assets of specific types and tags referenced from your HTML. I'm working on extensive docs around this topic, which should clear this up in the future hopefully.

[groenroos]

Thanks! 👍 Adding public/ to the permalink does indeed work on both prod and watch, but eh... it feels like a really odd gotcha.

Now we'd have to remember to add this segment to certain permalinks but not others. It also feels like a trap that it works either way on watch, but not on prod; in theory, the behaviour between the two should be identical, no?

Is there any documentation (from Vite's side) as to which files Vite will nix and which it won't? Is there truly no configuration option in Vite to customise that behaviour?

[KiwiKilian]

AFAIK public is the way to go, if you want something just copied over to your prod build. The behavior during watch is indeed not optimal, will have to check if this is eleventy specific or just how Vite does it.

With Vite 6 (stable plugin release is on v5!), this is the list of supported attributes: https://vite.dev/guide/features.html#html

After all, I think if you chose to use Vite, you will want to apply their way of doing stuff. Otherwise a different asset pipeline might be better suited.

[groenroos]

Agreed that putting static assets to be copied into the public/ folder is the ideal solution.

But for Eleventy-generated files (such as the generated.json example above), I think needing to remember to add a prefix to the permalink attribute isn't ideal - especially given the pitfalls already mentioned.

Thanks for the link, it's quite insightful! However, worth noting that not all generated files like this would or even could necessarily be loaded directly in HTML or JS (examples from our project: robots.txt, _redirects, _headers, any RSS .xml feed).

Let me know what you find in your research! Maybe it's a bug in Vite itself; or maybe it's something the eleventy-plugin-vite ought to bridge (i.e. by silently prepending public/ to the permalink behind the scenes; automatically generating a list of these files to be used as entrypoints in Vite; or similar)?

[Aankhen]

I just ran into this bug too. The explanations in this issue are much appreciated.