Support nvim-treesitter's parser_install_dir configuration

[dslinger-a2]

Does this feature exist in Emacs orgmode core?

N/A

Orgmode link

No response

Feature value

Currently, the orgmode plugin always installs the org parser to its own package directory ([package_root]/parser/org.so). It would be beneficial to respect nvim-treesitter's parser_install_dir configuration when available, which would allow users to keep all parsers in a single consistent location. This is especially useful in setups where the plugin is being installed by nix.

Additional context

No response

[rynplynch]

I'm trying to use this plugin but it fails to install the treesitter org parser. The nix store is read only so nvim-orgmode fails at the install. Allowing us to handle this dependency on our own would be great!

Also I looked at what I think is the org parser but it is archived? Is this the current parser that is used by orgmode?

[kristijanhusak]

This is the official parser being used: https://github.com/nvim-orgmode/tree-sitter-org/tree/next

I'm not sure how nix works, so I'm open for suggestions, as long as it does not involve adding a dependency on nvim-treesitter.

How are you installing other parsers?

[rynplynch]

I've been installing tree-sitters parsers via nixpkgs;
available parsers found here

We don't have to use the nixpkgs repo though, we could build the parser as a flake output. Then anyone who wants to use the orgmode parser could bring it in as a flake input. They would handle the parser dependency on their own.

Then we would set an environment variable for orgmode to look for. If it finds that environment variable it would use the parser at that location instead of attempting to install on its own. Or we could allow for a global variable to be set. I'm not good with Lua, working on learning it.

I can try writing a flake.nix file for the tree-sitter-org repo. That would be the first step.

[kristijanhusak]

I'm ok with using nixpkg. We probably need a github action to publish to nixpkg. Since I have no idea how it works, can you create a PR on https://github.com/nvim-orgmode/tree-sitter-org with the setup? You can follow luarocks release to figure out how to get necessary data for publishing, like version, etc.

Once we do that, I'll figure out a way to handle this separately installed parser, since now I expect it to be installed in the orgmode/parser/ folder.

Does that sound ok?

[rynplynch]

That sounds great! I'll see if I can get that done today. I'm looking into norg's implementation and using it to see what a proper implementation looks like.

[llakala]

Any updates on this?

[rynplynch]

There is now a pull request to update the tree-sitter grammar to the correct version here

I also plan on creating a pull request that introduces an overlay to the parser. None of this will resolve the issue with orgmode attempting to install the parser on it's own though. Will need to point the plugin to the already compiled parser somehow

[kristijanhusak]

Once the version is updated I'll update the code to not require a parser to be located in the orgmode folder. That should solve the issue.
With nixos, are you able to load the parser before orgmode? Because orgmode will attempt to install the parser if it's missing while loading the setup.

[rynplynch]

In my configuration I am able to build tree-sitter with the grammars I would like to use. This means that by the time the setup function for orgmode gets called there is already a compiled org.so library.

This path on my system contains all the parsers installed on my system.

/nix/store/rh6sba98r48dkzpxwh8lj34cqb18zhhg-vim-pack-dir/pack/myNeovimGrammars/start/vimplugin-treesitter-grammar-ALL-INCLUDED/parser/

If I could point orgmode to that directory it would find the org.so file.

Perhaps a new configuration option in the setup function for orgmode could achieve this?

[kristijanhusak]

This path on my system contains all the parsers installed on my system

Is this path part of your rtp ? If yes, then there's no need the configuration option. It should already read the parser from there. The problem is that there's a check to expect the parser to in orgmode/parser folder, which I plan to remove. As long as there is an org parser, we should be fine.

[rynplynch]

That's a good question. I'm new to Lua and neovim so take what I say with a grain of salt. All I know is that when I perform a healthcheck on tree-sitter it shows the org parser as OK.

I'm guessing that if tree-sitter can find it all right then it should work fine with orgmode?

[kristijanhusak]

You can check by doing :echo &rtp and finding the path in the comma separated list. Most likely it's there since other languages are working fine.

I'll be updating the logic around this in the following days to not require the parser to be in the orgmode folder. It will still notify if there is more than 1 org parser found, but that should not happen if you load the parser before the orgmode is loaded.

[farcaller]

Another nix user here 😄 there are several issues I stumbled on while making orgmode usable.

I patched the TS grammar in my flake:

              (nvim-treesitter.withPlugins (_:
                 nvim-treesitter.allGrammars ++
                [(pkgs.tree-sitter.buildGrammar {
                  language = "org";
                  version = "2.0.0";
                  src = pkgs.fetchFromGitHub {
                    owner = "nvim-orgmode";
                    repo = "tree-sitter-org";
                    rev = "2.0.0";
                    hash = "sha256-+eixAPvEq7LPylaovrSeT18h6FKnhVoYS68B1B5xhPs=";
                  };
                  meta.homepage = "https://github.com/nvim-orgmode/tree-sitter-org";
                })]))

and made nvim-treesitter aware of it:

      local parser_config = require "nvim-treesitter.parsers".get_parser_configs()
      parser_config.org = {
        filetype = "org",
        install_info = {
        },
      }

This, however, doesn't stop orgmode from trying to install the ts plugin again. I had to monkeypatch orgmode's installer to not do anything:

    config = function()
      -- Setup orgmode
      local ts_install = require('orgmode.utils.treesitter.install')
      ts_install.install = function() return true end

      require('orgmode').setup({
        ...

The problem is that despite all this and TS clearly working (as seen in :InspectTree below), the highlighting does not. Actually, no "nicer" features work, e.g. the links don't hide the extra formatting symbols by default as seen in the demo gifs.

I'm not sure if that's because how I shoveled the TS grammar in or there's some obscure integration bug.

[farcaller]

Oddly enough, if I symlink the orgmode's queries into my neovim config folder, then it seems to work. 🤔

[kristijanhusak]

I'll update the installation during the weekend to allow for any location for the parser. That should solve most of these issues

[kristijanhusak]

I updated the treesitter installation logic to not require the parser to be installed in orgmode folder. Released in 0.6.0 version. Let me know if that fixes it for you.

[farcaller]

@kristijanhusak still doesn't work:

...s/start/orgmode/lua/orgmode/utils/treesitter/install.lua:117: EACCES: permission denied: /nix/store/3297n9ymij2xh2123x3s269mr0433j28-vim-pack
-dir/pack/myNeovimPackages/start/orgmode/.org-ts-lock.json

It seems some codepath still tries to write a lockfile?

And just for the record, that's with org grammar installed as per :checkhealth vim.treesitter:

• ✅ OK Parser: org ABI: 14, path: /nix/store/kjzz6565l7ij4zmj3ya3v0w1n3qmiqhk-vimplugin-treesitter-grammar-ALL-INCLUDED/parser/org.so

[kristijanhusak]

@farcaller what about :checkhealth orgmode ?

[farcaller]

Orgmode ~

• ❌ ERROR Failed to run healthcheck for "orgmode" plugin. Exception:
  ...s/start/orgmode/lua/orgmode/utils/treesitter/install.lua:117: EACCES: permission denied: /nix/store/3297n9ymij2xh2123x3s269mr0433j28-vim-pack-dir/pack/myNeovimPackages/start/orgmode/.org-ts-lock.json

[farcaller]

The problem here is because you expect the orgmode's plugin root directory is in user's $HOME somewhere, but in nix it's in the global and very read-only store.

[kristijanhusak]

It seems some codepath still tries to write a lockfile?

Yeah it did. Just pushed fixed to master. Can you give it another test?

[farcaller]

It loads without issues now! But now we're back to this problem: #967 (comment). There's no syntax highlighting whatsoever.

[kristijanhusak]

Do you get anything from this command when you run it from the opened org file?

:lua= vim.treesitter.query.get_files('org', 'highlights')

Also, do you have to do something around queries for nvim-treesitter?

[farcaller]

I do:

{ "/nix/store/kjzz6565l7ij4zmj3ya3v0w1n3qmiqhk-vimplugin-treesitter-grammar-ALL-INCLUDED/queries/org/highlights.scm" }

Which seemingly comes from nvim-orgmode/tree-sitter-org.

[kristijanhusak]

I released 2.0.1 version of tree-sitter-org that removes those queries. Those are just the examples. Can you try using version 2.0.1 and see if it works?

[farcaller]

That seems to do the trick, thanks! And thank you so much for all the debugging effort on this!

[kristijanhusak]

Awesome! So there's no need for any monkey patching?
I'm still open to having a custom nix packages for both orgmode and tree-sitter-org, but I'm not a nix user so no idea how that works.

[farcaller]

Yep, it all works as expected now and the highlights are pulled from the correct place:

{ "/nix/store/76wnqzh7lviz0kczqaljacnhgysqx3sx-vim-pack-dir/pack/myNeovimPackages/start/orgmode/queries/org/highlights.scm" }

It's still somewhat strange they both aren't being pulled in, but whatever, it all works as intended now.

I am currently not in a position to submit new packages towards nixpkgs (it's complicated), but I expect that to change soon-ish, so if no one else beats me to it I'll work on adding both.