Separation of completion directories

[akinomyoga]

We currently instruct upstream packages to place completion files in the directory $prefix/share/bash-completion/completions/, while we also provide completions in the same directory.

When an upstream project decides to host its completion setting in the upstream project, we have renamed our completion setting from completions/<cmd> to completions/_<cmd> to mark it as a fallback. However, this only works when the bash-completion package of the system has newer versions. Otherwise, it would end up with a situation where two packages (of bash-completion and the upstream project) in distribution would want to install files in a conflicting path.

Technically, the package repository maintainer could choose the proper combination of the versions of bash-completion and the upstream project, but bash-completion may have another restriction. For example, macOS bash-completion is a very old one that needs to work with Bash 3.2. Also, this requirement on the version of the bash-completion packages seems inconsistent with the arguments for the new API. In the discussion of the new API introduced in 2.12 (such as in #1328), we expect a new version of bash-completion to be available everywhere only after about a decade.

I'm going to start threads for each subtopic.

[akinomyoga]

Separation of external/bundlded completion directories

I think it would be cleaner and easier to separate the directories for the external completions and the completions that we provide by default.

Since the external completions are already installed to $prefix/share/bash-completion/completions/, we may move our bundlded completions to another directory, such as $prefix/share/bash-completion/completions/default/ and $prefix/share/bash-completion/completions-dist/.

[yedayak]

I think this is a good idea, will be better than prefixing everything with _, although it might hurt discoverability of where completions are coming from a bit. Maybe adding a README file that points to the default completions will be nice.
Do you have a proposal for what to do with the already deprecated completions starting with _? We can remove the prefix when moving them, but maybe a distinction between 3rd party completion fallbacks (like for cobra) and stuff that are implemented here should stay? This does introduce another check for file, but it seems we already do a lot of those.
I guess #1329 (comment) is also related

[scop]

I also kind of like the idea. Leaning towards a separate completions-dist or the like rather than completions/default, because the latter could cause problems with completion for command named default.

I suppose the same consideration applies to the helpers dir, i.e. we'd have helpers-dist?

BTW I prefer the name dist over default, because default makes it sound as if we'd load something from there by default (with high precedence) while the reality is kind of the other way around (we load from there ~last).

[akinomyoga]

I also kind of like the idea. Leaning towards a separate completions-dist or the like rather than completions/default, because the latter could cause problems with completion for command named default.

That seems to make sense. Let's think about completions-*.

I suggested completions-dist above, but maybe it's confusing in the present context. I again checked how *-dist or dist-* is used. For example, texmf-dist in the TeX ecosystem is used for different packages to put their files in it. It's not used for separating the files from the main package from other packages in the same distribution, but it's used to separate distribution files from host-local files (added by system administrators or users).

I asked ChatGPT about other cases of the separation of directories. According to ChatGPT,

Ecosystem	Main package	Other packages	System administrators or users
Perl	/usr/lib/perl5/core_perl	/usr/lib/perl5/vendor_perl	/usr/lib/perl5/site_perl
Emacs	/usr/share/emacs/<ver>/lisp	/usr/share/emacs/site-lisp/<package>	/usr/share/emacs/site-lisp
Emacs (Debian)	/usr/share/emacs/<ver>/lisp	/usr/share/emacs/vendor-lisp	/usr/share/emacs/site-lisp
Ruby	/usr/lib/ruby/stdlib	/usr/lib/ruby/vendor_ruby	/usr/lib/ruby/site_ruby
Python	/usr/lib/python<ver>/stdlib	/usr/lib/python<ver>/dist-packages	/usr/lib/python<ver>/site-packages

but this seems hallucination. I checked the directory structures in Fedora 41, where the directory structures appear to be the following:

Ecosystem	Main package	Other packages	System administrators or users
Perl	/usr/lib64/perl5/CORE	/usr/lib64/perl5/vendor_perl	(Same as "Other packages"? or /usr/local?)
Emacs	/usr/share/emacs/<ver>/lisp	/usr/share/emacs/site-lisp	(Same as "Other packages"? or /usr/local?)
Ruby	/usr/lib/ruby	(Same as "Main package"?)	/usr/lib/ruby/vendor_ruby
Python	/usr/lib/python<ver>/stdlib	/usr/lib/python<ver>/site-packages	/usr/local/lib{,64}/python<ver>/site-packages

I asked ChatGPT about a suffix alternative to -dist. It suggested -core, -base, -main, -official, -standard, etc. I guess -core would be a good choice. It reminds me of the homebrew-core repository.

I suppose the same consideration applies to the helpers dir, i.e. we'd have helpers-dist?

Has helpers been suggested as a place for external packages to put files? I didn't know that, but I've now realized that we've been exposing helpersdir in bash-completion.pc.in, but it's not mentioned elsewhere.

bash-completion/bash-completion.pc.in

Line 7 in 594de4e

helpersdir=${datadir}/@PACKAGE@/helpers

I generally agree to introduce a similar structure for helpers, and I think we can also mention it in README in the same way as completionsdir.

---

My current suggestions for this topic are as follows:

• Move completions provided by this repository to completions-core
• Move helper scripts provided by this repository to helpers-core.
  • These files are called by ${BASH_SOURCE[0]%/*}/../helpers/<file>, so we need to rename both the directories in this repository and the installation target, so that completion files can find the helper script regardless of whether it is located in the repository tree or the installation tree.
• Mention the pkg-config variable helpersdir in README.
  • We also mention that the completion file can locate the file as ${BASH_SOURCE[0]%/*}/../helpers/<file>.

[akinomyoga]

Directory for "eagerly loaded completions"

There are some completion settings that people want to load at the shell startup rather than when completion of the corresponding command is attempted. Such a setting includes the one that wants to replace/modify the behavior of the default completion complete -D.

Our README calls such a setting "eagerly loaded completions" and recommends the user to put it in /etc/bash_completion.d. However, at the same time, /etc/bash_completion.d is kind of a deprecated mechanism for bash-completion v1 (which covered Bash <= 4.0 lacking complete -D). I raised a discussion about this subtlety in #1220 (comment). A partial answer has been given at #1220 (comment), but I think this subtlety itself hasn't been discussed.

I personally think the directory for the "eagerly loaded completions" and "v1 completions (in /etc/bash_completion.d)" can also be separated to make the intention clear to the upstream package maintainers. While the latter with API v1 (as defined in #1328) is deprecated, the former is still useful with API v2/v3 loading. The directory for eagerly loaded completions can be, for example, completions/startup/. This was essentially proposed by @kislyuk at #1220 (comment) originally. Here, I quote my reply at that time, which summarized the situation:

and if not, I'm looking for opinions from @scop on whether this functionality could be added.

@scop I think the current situation of BASH_COMPLETION_COMPAT_DIR is unclear. Although its name looks like it will be deprecated and should not be used, we added 000_bash_completion_compat.bash which I guess we need to maintain perpetually. The description in README is also not so clear about whether its use is encouraged or discouraged: "Q. How can I override a completion shipped by bash-completion?" seems to suggest using compatdir while "Q. I author/maintain package X and would like to maintain ..." (which is a question for the completion for a specific command) seems to deprecate compatdir. I've been thinking we would continue to offer compatdir for eagerly loaded settings while deprecating it for the completion settings for specific commands, but it is not so obvious. Maybe another option would be, as suggested, to prepare another directory where the eagerly loaded files can be put, though it is functionally equivalent to the current compatdir.

[akinomyoga]

@scop This subtopic is actually the one I think is related to #1399. My (current) conclusion is that the root cause of the problem with the UAPI spec for image-based Linux is /etc/bash_completion.d/000_bash_completion_compat.bash installed by us.

Background: First of all, we actually haven't officially deprecated API v2 (which is provided through 000_bash_completion_compat.bash) because external completions should use API v2 for now to work with different versions of bash-completion in the market. Even after declaring the switch to API v3 for external completions, we don't have a plan to remove API v2 as long as it doesn't affect maintainability. In this sense, 000_bash_completion_compat.bash isn't deprecated but is simply a file that should be loaded at startup. Other packages also want some settings to be loaded at the startup of bash-completion. For example, fzf wants to rewrite the default completion complete -D after bash-completion is loaded, so it puts a file in /etc/bash_completion.d/fzf utilizing the fact that /etc/bash_completion.d/* are loaded at startup, even though they are not API-v1 completions. So, I'd like to suggest introducing a new directory for (non-deprecated) eagerly-loaded files, such as /etc/bash-completion.d/startup{,-core} and move 000_bash_completion_compat.bash there.

PR 1399: The problem that #1399 tries to solve is actually the conflict of /etc/bash_completion.d/000_bash_completion_compat.bash within the image-based Linux (where /etc is externally mounted). However, the file will be moved to another directory under this subtopic (if accepted). Otherwise, we don't put any files in /etc/bash_completion.d, so there shouldn't be a problem with the image-based Linux.

External packages using /etc/bash_completion.d: Existing external packages that put files in /etc/bash_completion.d need to be updated regardless of #1399, by changing their installation location to /usr/share/bash-completion/completions with API v2 (or, if they want their file to be loaded at startup like fzf and _python-argcomplete, putting the file in /usr/share/bash-completion/startup, i.e., the directory suggested in this subtopic). I doubt the necessity of a new directory that essentially has the same purpose as /usr/share/bash-completion/completions, such as /usr/share/bash-completion/compat for API v1 introduced in #1339. Rather, it is strange to provide a new location for API-v1 completions when API-v1 completions are already broken (as we've already removed some functions of API v1, see #1220).

[akinomyoga]

Hmm, I've been forgetting about the benefit of updating compatdir I mentioned in #1399. The location of external completion settings in the upstream may be changed to outside /etc altogether through compatdir, if the upstream uses the pkg-config variable compatdir.

Quote from #1399 (review):

• In this context, if the project author sets up the installation of the completion script using autoconf or cmake following our README.md, if we change the compatdir in pkg-config, the installation location should automatically be adjusted even if the project is not actively maintained. In this sense, changing the default location of compatdir now still has some effects. A little concern is how widely our pkg-config settings are installed in the system in distributions.

After finding the UAPI spec, I'm not sure if the alternative location should be /usr/share/bash-completion/compat (as in the current #1399), /usr/share/bash_completion.d (as in the initial version of #1399), or /usr/etc/bash_completion.d (as suggested in the UAPI spec).

• If we had originally put files in /etc/bash_completion.d because we considered the files a part of the configuration at that time, it may be minimal and natural to change the compatdir as { => /usr}/etc/bash_completion.d to allow overriding the "default" completions. Overriding under the UAPI rule is actually useful to allow users to disable the distribution-provided setting (such as /usr/etc/bash_completion.d/fzf by putting an empty file in /etc/bash_completion.d (such as /etc/bash_completion.d/fzf).
  • However, the same mechanism for overriding may also be implemented for /usr/share/bash-completion/compat in the current #1399, so it's not a benefit only possible with /usr/etc/bash_completion.d. In addition, I think it is also useful for /usr/share/bash-completion/startup to have the same mechanism.

[akinomyoga]

Recommending filename extension .bash

This may be a separate discussion, but another thing is that the filename extension of a completion setting. Currently, we accept both completions/<cmd> and completions/<cmd>.bash, but this is ambiguous when there are two commands task and task.bash.

From a packaging point of view, this may also introduce conflicts between packages when the package for task wants to install its completion to <cmd>/task.bash (with the filename extension) while the package for task.bash wants to install its completion to <cmd>/task.bash (without the filename extension). The executable names shared by people do not usually have the filename extension .bash, so this wouldn't become a big problem, but still, personal scripts installed to a home directory (~/.local) may have the filename extension, where a conflict may happen on the filename path ~/.local/share/bash-completion/completions/task.bash.

Even without such a conflict, a problem arises in loading the completions. When a user wants to complete the arguments to task.bash [TAB], the completion loader first attempts reading task.bash, but it might be a completion setting for task. Or, when a user wants to complete the arguments to task [TAB], the completion loader first checks task and, if not found, task.bash. However, task.bash might not be the completion setting for task.bash.

I feel we should prefer completions/<cmd>.bash to completion/<cmd> because it confuses text editors less (see #1323). We have -*- shell-script -*- and ex: filetype=sh in every completion file, but not all the text editors respect this. For example, if a completion setting for a python script mytool.py would install the completion setting at completions/mytool.py, text editors may easily mis-detects the language to be Python even though the completion setting is written in Bash.

[scop]

Makes sense to me to prefer cmdname.bash over cmdname for the completion file, and also recommend doing so in the docs for 3rd party completions. As long as we try to load both, possibility for a mixup still exists though (possibly loading task.bash that is intended for task when completing task.bash if task.bash.bash does not exist). Dropping loading non-.bash would be too big a change though, so we get to pick our poison here. Anyway we could make the preference stronger by documenting completion filenames without added .bash as deprecated.

[akinomyoga]

Thank you for the comment. OK, let's consider this direction.

Dropping loading non-.bash would be too big a change though, so we get to pick our poison here.

I think we can drop the support for non-.bash for completions-dist or completions-core, where we have full control. For compatibility, I think we can continue to check non-.bash files (at least for a sufficient transition period).

Anyway we could make the preference stronger by documenting completion filenames without added .bash as deprecated.

OK. I agree with this.

---

Current suggestions are

• Rename our completions/* to completions-core/*.bash
• Check both <cmd>.bash and <cmd> in completions but only <cmd>.bash in completions
• Recommend <cmd>.bash and deprecate <cmd> in README

[akinomyoga]

xfunc compatibility

This is even another topic but somewhat related. In #1321 (comment), it was suggested to move some of nslookup completions to the upstream bind9 project so that they include all the related functions in a single completion file and reuse the functions in completions/nslookup from their completion.

An alternative solution would be to keep those functions in completions/nslookup (but rename them to _comp_xfunc_nslookup_*) and call them through _comp_xfunc. However, this is non-trivial when an older version of completions/nslookup is loaded, where the xfunc interface is not defined.

[akinomyoga]

Deprecated completions in completions/

I think #894 is also related, where moving "the deprecated completions" to a separate place was suggested. However, the details haven't yet been discussed.

For example, I think these could be moved to completions/deprecated/.

[akinomyoga]

I prefer merging sub-directries (or filter out) at ./configure and make install instead of implementing something to bash-completion. I think user is not interested about deprecated or not.

I don't understand. We discuss the separation of the directory structure of the installation target but not the one in the source tree. The reason that I suggest introducing a separate directory is to avoid conflicts happening in the installation target. In that sense, the directory structures within the source tree don't need to be modified, but the directory tree in the target tree needs to be modified to solve the problem.

[oech3]

User does not need both of completions: for fallbacking and deprecated. So they does not conflict at file level. No?

[akinomyoga]

Right. Fallback and deprecated completions wouldn't conflict with each other because both are maintained by us, but the original discussion should have been the conflict between the deprecated/fallback completions vs external completions. If files completions/<subdir>/* are merged (or filtered out) into the parent directory completions, the conflicts appear.

If you talk about merging the completion files into a single sub-directory completions/<single-subdir>, the conflict problem doesn't appear, but bash-completion doesn't support it currently, so we still need to implement it after discussion.

[scop]

If we implement the completions-dist idea from above, would it not make this unnecessary? I.e. everything we ship is in completions-dist, and everything in it can be overridden by placing a suitably named file in completions, so there may not be any need to deprecate any of our files in a sense. (We may want to continue documenting in our files's comments where we know that a preferred upstream completion exists though.)

[akinomyoga]

I think this is technically unnecessary, though it could be useful for maintenance purposes.

• Even though we keep old files, I think it could be useful to separate the deprecated files (whose behaviors shouldn't be updated) from the non-deprecated files (which should be updated following the upstream change). For example, when the upstream of a command started to offer a completion script by themselves in version 1.2.3, and when we deprecated the completion script in our repository for version 1.2.2, I think we should freeze the completion candidates generated by the completion setting for 1.2.2, rather than updating them for version >= 1.2.3 because the load of the deprecated file implies that the system has an old version (<= 1.2.2) of the command.
• These files are kept for compatibility with old versions of external commands that didn't ship their completions, so if we separate the deprecated completion files, the directory name could be completions-compat.
• Maybe we can separate the directories for deprecated completions in the repository tree, but merge them in the installation target.

though they are not strong opinions. How do you feel?

[oech3]

Thankyou.

If you mean automatic loading of the file (without explicitly calling source ...), no versions of Bash support loading completions from anywhere.

This is something what I wanted to know.

We need to implement it after this discussion.

I prefer merging sub-directries (or filter out) at ./configure and make install instead of implementing something to bash-completion. I think user is not interested about deprecated or not.