Benchmark is unfair. find(1) should not be used for grepping.

[alejandro-colomar]

Hi,

I believe the benchmarks you provide compared to find(1) are unfair. find(1) should not be used for grepping; following the Unix principles, find(1) should just find, and grep(1) should be responsible for filtering the output of find(1).

If we pipe find(1) to grep(1), the performance is significantly faster than just using find(1):

$ time find ~ -iregex '.*[0-9]\.c$' >/dev/null

real	0m1.103s
user	0m0.881s
sys	0m0.220s
$ time find ~ | grep -i '/[^/]*[0-9]\.c$' >/dev/null

real	0m0.346s
user	0m0.103s
sys	0m0.260s

find | grep seems to be faster than fdfind in my own simple test:

$ time fdfind -HI '.*[0-9]\.c$' ~ >/dev/null

real	0m0.383s
user	0m1.920s
sys	0m6.126s

Can you please provide benchmarks against this pipeline in your readme?

What version of fd are you using?
[paste the output of fd --version here]

$ dpkg -l | grep fd-find
ii  fd-find                               8.7.0-3+b1                               amd64        Simple, fast and user-friendly alternative to find

$ fdfind --version
fdfind 8.7.0

Just for completeness, here's my CPU:

$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               GenuineIntel
  Model name:            13th Gen Intel(R) Core(TM) i9-13900T
    CPU family:          6
    Model:               183
    Thread(s) per core:  1
    Core(s) per socket:  24
...
Caches (sum of all):     
  L1d:                   896 KiB (24 instances)
  L1i:                   1.3 MiB (24 instances)
  L2:                    32 MiB (12 instances)
  L3:                    36 MiB (1 instance)

Thanks!

[tavianator]

I believe the benchmarks you provide compared to find(1) are unfair. find(1) should not be used for grepping; following the Unix principles, find(1) should just find, and grep(1) should be responsible for filtering the output of find(1).

find has supported tests like -name since antiquity, so I think it's an exaggeration to say they "should not" be used. And in the limit, find | grep foo has to send arbitrarily more data through a pipe than find -name foo outputs, so it has the potential to be much faster. (There are other reasons to prefer find -name too, such as the fact that find | grep is broken for filenames that contain newlines.)

Unfortunately, the regex implementation in GNU find (which comes from Gnulib) is quite slow, so you can easily beat it by using the optimized one from grep. Even just -iname is faster than -iregex, though not quite as fast as grep:

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
find -iregex '.*[0-9]\.c$'	671.9 ± 12.0	660.0	686.9	2.14 ± 0.04
find -iname '*[0-9].c'	475.2 ± 13.4	463.2	492.9	1.51 ± 0.05
find | grep -i '[0-9]\.c$'	314.0 ± 3.2	309.9	318.7	1.00
fd -u '[0-9]\.c$'	333.2 ± 3.5	329.3	341.0	1.06 ± 0.02
fd -u | grep -i '[0-9]\.c$'	337.2 ± 3.2	332.9	342.9	1.07 ± 0.02

This is on a checkout of rust-lang/rust (see the bfs benchmarks). A build of fd that includes BurntSushi/ripgrep@d938e95 does much better:

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
find -iregex '.*[0-9]\.c$'	663.9 ± 13.0	654.1	688.6	4.55 ± 0.54
find -iname '*[0-9].c'	471.1 ± 13.9	459.8	489.6	3.23 ± 0.39
find | grep -i '[0-9]\.c$'	306.4 ± 11.8	289.5	320.0	2.10 ± 0.26
fd -u '[0-9]\.c$'	145.8 ± 17.2	123.7	174.2	1.00
fd -u | grep -i '[0-9]\.c$'	210.7 ± 11.6	198.5	238.9	1.45 ± 0.19

I agree that the benchmarks should probably be updated, and we should make the comparisons as fair as possible. #893 is relevant.

[alejandro-colomar]

I believe the benchmarks you provide compared to find(1) are unfair. find(1) should not be used for grepping; following the Unix principles, find(1) should just find, and grep(1) should be responsible for filtering the output of find(1).

find has supported tests like -name since antiquity, so I think it's an exaggeration to say they "should not" be used.

Relevant stuff:
http://doc.cat-v.org/unix/find-history

I understand that people use them out of habit, but the design of the program is rather dubious. But yeah, I could change the "should not be used" to maybe "are suboptimal, both in performance terms, and in simplicity".

Piping to grep(1), you get faster results, and you don't even need to check the find(1) manual page for all the similar but different options that it has for grepping files.

And in the limit, find | grep foo has to send arbitrarily more data through a pipe than find -name foo outputs, so it has the potential to be much faster.

The pipe shouldn't be a bottleneck. The bottleneck is usually I/O.

$ time find >/dev/null

real	0m0.327s
user	0m0.091s
sys	0m0.233s
$ time find | grep -i '[0-9]\.c$' >/dev/null

real	0m0.335s
user	0m0.098s
sys	0m0.250s

You can see that piping all filenames only adds a little bit of time to a simple find(1).

Also, not only the real time is important. fdfind(1), with the appropriate patches and optimizations may beat find | grep by a slight advantage in real time, as you show, but it still uses heavy CPU time; it's just that it runs in parallel.

alx@debian:~$ time find | grep -i '[0-9]\.c$' >/dev/null

real	0m0.358s
user	0m0.098s
sys	0m0.276s
alx@debian:~$ time fdfind -u | grep -i '[0-9]\.c$' >/dev/null

real	0m0.390s
user	0m1.831s
sys	0m6.067s
alx@debian:~$ time fdfind -u '[0-9]\.c$' >/dev/null

real	0m0.385s
user	0m1.846s
sys	0m6.222s

That's fine if the bottleneck is in find(1), but if I pipe this command to a more consuming pipeline where find(1) is not the bottleneck, I fear that it may actually be slower, and will occupy the CPU that could be running other tasks.

If I have other heavy tasks at the same time, like compiling some software, it will also probably affect the performance of fdfind(1) significantly, while find | grep wouldn't be affected as much, probably.

(There are other reasons to prefer find -name too, such as the fact that find | grep is broken for filenames that contain newlines.)

Those filenames are already broken (they are unportable, according to POSIX https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_06). Please don't use them. :)

If you need them, though, you can use find -print0 | grep -z.

Unfortunately, the regex implementation in GNU find (which comes from Gnulib) is quite slow, so you can easily beat it by using the optimized one from grep. Even just -iname is faster than -iregex, though not quite as fast as grep:
Command Mean [ms] Min [ms] Max [ms] Relative
find -iregex '.*[0-9]\.c$' 671.9 ± 12.0 660.0 686.9 2.14 ± 0.04
find -iname '*[0-9].c' 475.2 ± 13.4 463.2 492.9 1.51 ± 0.05
find | grep -i '[0-9]\.c$' 314.0 ± 3.2 309.9 318.7 1.00
fd -u '[0-9]\.c$' 333.2 ± 3.5 329.3 341.0 1.06 ± 0.02
fd -u | grep -i '[0-9]\.c$' 337.2 ± 3.2 332.9 342.9 1.07 ± 0.02

This is on a checkout of rust-lang/rust (see the bfs benchmarks). A build of fd that includes BurntSushi/ripgrep@d938e95 does much better:
Command Mean [ms] Min [ms] Max [ms] Relative
find -iregex '.*[0-9]\.c$' 663.9 ± 13.0 654.1 688.6 4.55 ± 0.54
find -iname '*[0-9].c' 471.1 ± 13.9 459.8 489.6 3.23 ± 0.39
find | grep -i '[0-9]\.c$' 306.4 ± 11.8 289.5 320.0 2.10 ± 0.26
fd -u '[0-9]\.c$' 145.8 ± 17.2 123.7 174.2 1.00
fd -u | grep -i '[0-9]\.c$' 210.7 ± 11.6 198.5 238.9 1.45 ± 0.19

I agree that the benchmarks should probably be updated, and we should make the comparisons as fair as possible. #893 is relevant.

Thanks.

[tavianator]

Relevant stuff: http://doc.cat-v.org/unix/find-history

Thanks for this link, very cool!

The pipe shouldn't be a bottleneck. The bottleneck is usually I/O.

This is often true, but in my personal uses the whole tree is usually in cache (dcache, or at least buffer cache). Most of the overhead then comes from kernel and syscall overhead.

You can see that piping all filenames only adds a little bit of time to a simple find(1).

That benchmark still has find writing all the paths to stdout. This one shows the difference:

Command	Mean [s]	Min [s]	Max [s]	Relative
find -false	2.928 ± 0.023	2.892	2.953	1.00
find >/dev/null	3.076 ± 0.010	3.059	3.095	1.05 ± 0.01
find | cat >/dev/null	3.129 ± 0.020	3.099	3.169	1.07 ± 0.01

Anyway, more important than things like -name are things like -prune, -xdev, etc. that reduce the search space in a way that post-processing cannot.

Also, not only the real time is important. fdfind(1), with the appropriate patches and optimizations may beat find | grep by a slight advantage in real time, as you show, but it still uses heavy CPU time; it's just that it runs in parallel.

That's fine if the bottleneck is in find(1), but if I pipe this command to a more consuming pipeline where find(1) is not the bottleneck, I fear that it may actually be slower, and will occupy the CPU that could be running other tasks.

Agreed. fd tends to use a bit too much parallelism by default (see #1203). But fd -j1 performance is also pretty good:

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
find -iregex '.*[0-9]\.c$'	667.6 ± 12.7	655.4	685.7	2.42 ± 0.11
find -iname '*[0-9].c'	472.0 ± 12.3	460.5	488.1	1.71 ± 0.08
find | grep -i '[0-9]\.c$'	312.6 ± 7.8	291.0	317.2	1.13 ± 0.05
fd -j1 -u '[0-9]\.c$'	276.1 ± 10.9	261.8	287.1	1.00
fd -j1 -u | grep -i '[0-9]\.c$'	340.7 ± 4.5	335.3	347.1	1.23 ± 0.05

[alejandro-colomar]

Relevant stuff: http://doc.cat-v.org/unix/find-history

Thanks for this link, very cool!

:-)

The pipe shouldn't be a bottleneck. The bottleneck is usually I/O.

This is often true, but in my personal uses the whole tree is usually in cache (dcache, or at least buffer cache). Most of the overhead then comes from kernel and syscall overhead.

You can see that piping all filenames only adds a little bit of time to a simple find(1).

That benchmark still has find writing all the paths to stdout. This one shows the difference:
Command Mean [s] Min [s] Max [s] Relative
find -false 2.928 ± 0.023 2.892 2.953 1.00
find >/dev/null 3.076 ± 0.010 3.059 3.095 1.05 ± 0.01
find | cat >/dev/null 3.129 ± 0.020 3.099 3.169 1.07 ± 0.01

A 7% lost on the pipe seems quite reasonable. How much does fdfind(1) take on, say, fdfind vs fdfind .? That is, how much % of the time does fdfind take in filtering the output by name (but not in the regex itself)? (My own tests seem to say it's negligible.)

Anyway, more important than things like -name are things like -prune, -xdev, etc. that reduce the search space in a way that post-processing cannot.

Yup, those things must run under find(1), but they are actually fast, aren't they? The problem with find(1)'s performance, AFAIK, is just with name filtering. Do you have benchmarks for those things comparing to find(1)?

Also, not only the real time is important. fdfind(1), with the appropriate patches and optimizations may beat find | grep by a slight advantage in real time, as you show, but it still uses heavy CPU time; it's just that it runs in parallel.
That's fine if the bottleneck is in find(1), but if I pipe this command to a more consuming pipeline where find(1) is not the bottleneck, I fear that it may actually be slower, and will occupy the CPU that could be running other tasks.

Agreed. fd tends to use a bit too much parallelism by default (see #1203). But fd -j1 performance is also pretty good:
Command Mean [ms] Min [ms] Max [ms] Relative
find -iregex '.*[0-9]\.c$' 667.6 ± 12.7 655.4 685.7 2.42 ± 0.11
find -iname '*[0-9].c' 472.0 ± 12.3 460.5 488.1 1.71 ± 0.08
find | grep -i '[0-9]\.c$' 312.6 ± 7.8 291.0 317.2 1.13 ± 0.05
fd -j1 -u '[0-9]\.c$' 276.1 ± 10.9 261.8 287.1 1.00
fd -j1 -u | grep -i '[0-9]\.c$' 340.7 ± 4.5 335.3 347.1 1.23 ± 0.05

Interesting; -j1 is good.

[jgardona]

find is running faster than fd just now in the example of hyperfind

[tavianator]

@jgardona find completing in 2.9 ms means that's probably a very small directory tree. I'd imagine that most of the time taken by fd is startup overhead, maybe due to #1203

[jgardona]

@jgardona find completing in 2.9 ms means that's probably a very small directory tree. I'd imagine that most of the time taken by fd is startup overhead, maybe due to #1203

tested it on /usr/bin directory

[tmccombs]

Unless you have an incredibly lare number of executables installed, /usr/bin won't be very large, relatively speaking. It is also pretty flat, which I think hinders the parallelizability.

[jgardona]

Well, its near 3.000 executables.

[alejandro-colomar]

@jgardona

That's a small thing, actually. Compare to this:

alx@debian:~$ time find ~ | wc -l
660487

real	0m0.338s
user	0m0.080s
sys	0m0.273s
alx@debian:~$ time find ~ -type f | wc -l
603577

real	0m0.344s
user	0m0.096s
sys	0m0.262s
alx@debian:~$ time find ~ -type d | wc -l
53914

real	0m0.314s
user	0m0.091s
sys	0m0.223s

:)

[vegerot]

find(1) should not be used for grepping

Then why does find(1) support grepping?

[alejandro-colomar]

find(1) should not be used for grepping

Then why does find(1) support grepping?

Because its inventor was told to do it that way, it seems. Here's the (funny) history of the design of find(1):
https://doc.cat-v.org/unix/find-history

[vegerot]

@alejandro-colomar gotcha. So in that case we shouldn't benchmark fd's grepping too

[alejandro-colomar]

@alejandro-colomar gotcha. So in that case we shouldn't benchmark fd's grepping too

The differencce is that fd(1) is intended for grepping. So we need to benchmark what this project is selling.

But I agree that it would be interesting to also test fd | grep.

I'll benchmark it when I arrive home.

[alejandro-colomar]

Cc: @vegerot

New benchmark below in this message.

TL;DR:

find | grep is by far the most efficient command in terms of CPU load, and performance per CPU time. It is also decently fast. However, it doesn't parallelize as much as fdfind.

fdfind | grep is useless. If you want excellent efficiency with great performance, use find | grep; and if you want absolute performance, and have lots of iddle CPUs and you want to use them all for finding your files, use fdfind.

find alone is not useful. I think everybody agrees on this. Let's not use it.

fdfind alone is the fastest if you have many iddle cores, and want to use them all at once for finding your files. It is not efficient at all, but it is fast. If you care about latency of fractions of a second, go ahead.

---

For some reason, this time fdfind(1) wins by far in ellapsed time. However, this is because it parallelizes the load. That's unfair, because in non-trivial tasks where the other CPUs are loaded, fdfind(1) won't have such an advantage.

Another result of my new benchmark is that piping fdfind | grep is slower than just fdfind in ellapsed time, but generates less load than fdfind.

However, the command that generates less load is find | grep. It is only slightly slower than fdfind | grep in ellapsed time, while generating significantly less load. It is slightly faster than just find, and generates significantly less load.

Here's the new benchmark. (I run a few runs of each command before the ones I show, to cache stuff to be fair.)

alx@devuan:~$ /bin/time find ~/src/ -iregex '.*[0-9]\.c$' | wc -l
1.08user 0.26system 0:01.34elapsed 100%CPU (0avgtext+0avgdata 6236maxresident)k
0inputs+0outputs (0major+2487minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time find ~/src/ -iregex '.*[0-9]\.c$' | wc -l
1.09user 0.23system 0:01.33elapsed 99%CPU (0avgtext+0avgdata 6244maxresident)k
0inputs+0outputs (0major+2484minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time find ~/src/ -iregex '.*[0-9]\.c$' | wc -l
1.11user 0.22system 0:01.33elapsed 99%CPU (0avgtext+0avgdata 6272maxresident)k
0inputs+0outputs (0major+2487minor)pagefaults 0swaps
95023

alx@devuan:~$ /bin/time find ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.10user 0.28system 0:00.53elapsed 71%CPU (0avgtext+0avgdata 5784maxresident)k
0inputs+0outputs (0major+1031minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time find ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.14user 0.24system 0:00.54elapsed 71%CPU (0avgtext+0avgdata 5892maxresident)k
0inputs+0outputs (0major+1032minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time find ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.10user 0.28system 0:00.54elapsed 71%CPU (0avgtext+0avgdata 6020maxresident)k
0inputs+0outputs (0major+1033minor)pagefaults 0swaps
95023

alx@devuan:~$ /bin/time fdfind -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.77user 0.48system 0:00.06elapsed 1870%CPU (0avgtext+0avgdata 35400maxresident)k
0inputs+0outputs (0major+9712minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.69user 0.54system 0:00.06elapsed 1868%CPU (0avgtext+0avgdata 42100maxresident)k
0inputs+0outputs (0major+10872minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.68user 0.45system 0:00.05elapsed 2063%CPU (0avgtext+0avgdata 32700maxresident)k
0inputs+0outputs (0major+8965minor)pagefaults 0swaps
95023

alx@devuan:~$ /bin/time fdfind -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.49user 0.47system 0:00.38elapsed 251%CPU (0avgtext+0avgdata 39260maxresident)k
0inputs+0outputs (0major+10958minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.48user 0.50system 0:00.39elapsed 250%CPU (0avgtext+0avgdata 39620maxresident)k
0inputs+0outputs (0major+10665minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.48user 0.51system 0:00.39elapsed 256%CPU (0avgtext+0avgdata 40668maxresident)k
0inputs+0outputs (0major+11167minor)pagefaults 0swaps
95023

Here's a similar benchmark run in several directories across different file systems, which shows no significant differences.

alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ -iregex '.*[0-9]\.c$' | wc -l
1.33user 0.29system 0:01.63elapsed 99%CPU (0avgtext+0avgdata 10324maxresident)k
0inputs+0outputs (0major+2081minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ -iregex '.*[0-9]\.c$' | wc -l
1.32user 0.28system 0:01.61elapsed 99%CPU (0avgtext+0avgdata 10268maxresident)k
0inputs+0outputs (0major+2081minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ -iregex '.*[0-9]\.c$' | wc -l
1.30user 0.29system 0:01.60elapsed 99%CPU (0avgtext+0avgdata 10204maxresident)k
0inputs+0outputs (0major+2082minor)pagefaults 0swaps
95866

alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.09user 0.34system 0:00.61elapsed 72%CPU (0avgtext+0avgdata 9904maxresident)k
0inputs+0outputs (0major+2016minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.11user 0.32system 0:00.61elapsed 72%CPU (0avgtext+0avgdata 9784maxresident)k
0inputs+0outputs (0major+2013minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time find ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.12user 0.32system 0:00.61elapsed 72%CPU (0avgtext+0avgdata 9860maxresident)k
0inputs+0outputs (0major+2015minor)pagefaults 0swaps
95866

alx@devuan:~$ sudo /bin/time fdfind -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.83user 0.67system 0:00.07elapsed 1975%CPU (0avgtext+0avgdata 43120maxresident)k
0inputs+0outputs (0major+13598minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time fdfind -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.86user 0.62system 0:00.07elapsed 1932%CPU (0avgtext+0avgdata 48424maxresident)k
0inputs+0outputs (0major+12847minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time fdfind -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.81user 0.65system 0:00.07elapsed 1871%CPU (0avgtext+0avgdata 50764maxresident)k
0inputs+0outputs (0major+13807minor)pagefaults 0swaps
95866

alx@devuan:~$ sudo /bin/time fdfind -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.58user 0.62system 0:00.42elapsed 282%CPU (0avgtext+0avgdata 61820maxresident)k
0inputs+0outputs (0major+16526minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time fdfind -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.59user 0.68system 0:00.42elapsed 300%CPU (0avgtext+0avgdata 57180maxresident)k
0inputs+0outputs (0major+17473minor)pagefaults 0swaps
95866
alx@devuan:~$ sudo /bin/time fdfind -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.55user 0.57system 0:00.41elapsed 274%CPU (0avgtext+0avgdata 62628maxresident)k
0inputs+0outputs (0major+16952minor)pagefaults 0swaps
95866

[vegerot]

find alone is not useful. I think everybody agrees on this. Let's not use it. @alejandro-colomar

Sounds good.👍 Let's open a patch to GNU Coreutils to delete that functionality from GNU find. Once that lands we can update these benchmarks.

[alejandro-colomar]

find alone is not useful. I think everybody agrees on this. Let's not use it. @alejandro-colomar

Sounds good.👍 Let's open a patch to GNU Coreutils to delete that functionality from GNU find. Once that lands we can update these benchmarks.

Heh! I think that's going to find a wall of concrete. That would break scripts, which means will never be removed. BTW, find(1) is from GNU findutils, not coreutils.

Blame the people at Bell Labs who wrote find(1), or its specification. We now have it. But let's not use it. We have better tools, like grep(1).

[alejandro-colomar]

On the other hand, it might be interesting to fork find(1) into a new program f(1), which would get rid of all the crap from find(1). Just find stuff. It looks like a fun project for me.

[tavianator]

New benchmark below in this message.

Thanks for doing those. Here's my attempt to replicate them, with hyperfine. The working directory is a checkout of the Chromium source tree (2,119,292 files):

Benchmark 1: find . -iregex '.*[0-9]\.c$' | wc -l
  Time (mean ± σ):      8.736 s ±  0.057 s    [User: 6.468 s, System: 2.222 s]
  Range (min … max):    8.614 s …  8.791 s    10 runs
 
Benchmark 2: find . | grep -i '/[^/]*[0-9]\.c$' | wc -l
  Time (mean ± σ):      2.970 s ±  0.017 s    [User: 1.718 s, System: 2.387 s]
  Range (min … max):    2.946 s …  3.003 s    10 runs
 
Benchmark 3: fd -u '.*[0-9]\.c$' | wc -l
  Time (mean ± σ):     221.8 ms ±   2.1 ms    [User: 4561.5 ms, System: 5194.0 ms]
  Range (min … max):   216.4 ms … 224.5 ms    13 runs
 
Benchmark 4: fd -u | grep -i '/[^/]*[0-9]\.c$' | wc -l
  Time (mean ± σ):     681.0 ms ±  15.2 ms    [User: 4200.0 ms, System: 4313.2 ms]
  Range (min … max):   656.1 ms … 701.3 ms    10 runs
 
Benchmark 5: fd -u -j1 '.*[0-9]\.c$' | wc -l
  Time (mean ± σ):      2.543 s ±  0.021 s    [User: 0.671 s, System: 1.905 s]
  Range (min … max):    2.504 s …  2.562 s    10 runs
 
Benchmark 6: fd -u -j1 | grep -i '/[^/]*[0-9]\.c$' | wc -l
  Time (mean ± σ):      3.732 s ±  0.081 s    [User: 4.850 s, System: 3.626 s]
  Range (min … max):    3.600 s …  3.833 s    10 runs

TL;DR:

find | grep is by far the most efficient command in terms of CPU load, and performance per CPU time. It is also decently fast. However, it doesn't parallelize as much as fdfind.

True, but for me fd -j1 is even better than find | grep.

fdfind | grep is useless.

That's not surprising to me. With fd you get multi-threaded filtering, but with fd | grep you get at most one core of grep.

What does surprise me is that fd -j1 | grep is slower than find | grep. Not sure why that is, I'll look into it.

If you want excellent efficiency with great performance, use find | grep; and if you want absolute performance, and have lots of iddle CPUs and you want to use them all for finding your files, use fdfind.

Don't forget about fd -j1! But to be fair, there's no reason a find implementation couldn't be just as fast.

find alone is not useful. I think everybody agrees on this. Let's not use it.

find -iregex is pretty slow because grep uses a much faster regex implementation. Theoretically, GNU find could use the same implementation and become more useful. (Though find | grep does allow for more parallelism, assuming single-threaded find.)

fdfind alone is the fastest if you have many iddle cores, and want to use them all at once for finding your files. It is not efficient at all, but it is fast. If you care about latency of fractions of a second, go ahead.

Personally I use fd mainly interactively, and I'm happy to burn CPU cores for lower latency. In other contexts, like for example a background task on a loaded machine, efficiency is more important.

But fd -j1 is pretty efficient too! It has the lowest user+system CPU consumption of anything in the benchmark. The reason fd appears CPU-inefficient is that crossbeam (used to communicate between threads) tends to prefer spinning over blocking. Spinning burns CPU time but has much lower latency than blocking.

[alejandro-colomar]

Hmmm. Thanks!

For completeness, here are the results of fdfind(1) with -j1 in my system.

In my system, fdfind -j1 is slightly faster than find | grep, although it takes slightly more CPU time.

And fdfind -j1 | grep is slightly slower and uses more CPU than both fdfind -j1 and find | grep, which is weird. It seems something is weird in the synchronization. Which means piping fdfind(1) to a usual pipeline that runs a dozen of processes would be inefficient compared to find(1). You should check this; I suspect it will be a bug that you can fix.

alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.25user 0.22system 0:00.43elapsed 109%CPU (0avgtext+0avgdata 9860maxresident)k
0inputs+0outputs (0major+2695minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.26user 0.23system 0:00.44elapsed 110%CPU (0avgtext+0avgdata 10072maxresident)k
0inputs+0outputs (0major+2799minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~/src/ | wc -l
0.27user 0.22system 0:00.45elapsed 109%CPU (0avgtext+0avgdata 10488maxresident)k
0inputs+0outputs (0major+2813minor)pagefaults 0swaps
95023

alx@devuan:~$ /bin/time fdfind -j1 -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.70user 0.38system 0:00.77elapsed 140%CPU (0avgtext+0avgdata 10400maxresident)k
0inputs+0outputs (0major+2628minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -j1 -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.68user 0.37system 0:00.75elapsed 139%CPU (0avgtext+0avgdata 11720maxresident)k
0inputs+0outputs (0major+3005minor)pagefaults 0swaps
95023
alx@devuan:~$ /bin/time fdfind -j1 -HI . ~/src/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.67user 0.41system 0:00.77elapsed 140%CPU (0avgtext+0avgdata 11748maxresident)k
0inputs+0outputs (0major+3143minor)pagefaults 0swaps
95023

And with several dirs:

alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.27user 0.31system 0:00.63elapsed 93%CPU (0avgtext+0avgdata 15572maxresident)k
2080inputs+0outputs (0major+5396minor)pagefaults 0swaps
95866
alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.25user 0.31system 0:00.52elapsed 108%CPU (0avgtext+0avgdata 15672maxresident)k
0inputs+0outputs (0major+5398minor)pagefaults 0swaps
95866
alx@devuan:~$ /bin/time fdfind -j1 -HI '.*[0-9]\.c$' ~ ~/src/ ~/mail/ /boot/ | wc -l
0.28user 0.27system 0:00.51elapsed 109%CPU (0avgtext+0avgdata 15972maxresident)k
0inputs+0outputs (0major+8012minor)pagefaults 0swaps
95866

alx@devuan:~$ /bin/time fdfind -j1 -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.79user 0.46system 0:00.89elapsed 140%CPU (0avgtext+0avgdata 15824maxresident)k
0inputs+0outputs (0major+5679minor)pagefaults 0swaps
95866
alx@devuan:~$ /bin/time fdfind -j1 -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.80user 0.40system 0:00.86elapsed 139%CPU (0avgtext+0avgdata 14976maxresident)k
0inputs+0outputs (0major+5552minor)pagefaults 0swaps
95866
alx@devuan:~$ /bin/time fdfind -j1 -HI . ~ ~/src/ ~/mail/ /boot/ | grep -i '/[^/]*[0-9]\.c$' | wc -l
0.80user 0.44system 0:00.89elapsed 139%CPU (0avgtext+0avgdata 15204maxresident)k
0inputs+0outputs (0major+8036minor)pagefaults 0swaps
95866

[tavianator]

What does surprise me is that fd -j1 | grep is slower than find | grep. Not sure why that is, I'll look into it.

This may be a resurgence of / incomplete fix for #1313:

$ strace -cf -e 'write' -- fd -j1 -u | grep -i '/[^/]*[0-9]\.c$' | wc -l
 % time     seconds  usecs/call     calls    errors syscall
 ------ ----------- ----------- --------- --------- ----------------
 100.00    0.392255          11     32957           write
$ strace -cf -e 'write' -- find | grep -i '/[^/]*[0-9]\.c$' | wc -l
 % time     seconds  usecs/call     calls    errors syscall
 ------ ----------- ----------- --------- --------- ----------------
 100.00    0.023571           7      3104           write

I think my reasoning in #1452 doesn't really apply to the -j1 case. We probably need additional buffering there. I'll split this out as a separate issue.

[alejandro-colomar]

What does surprise me is that fd -j1 | grep is slower than find | grep. Not sure why that is, I'll look into it.

This may be a resurgence of / incomplete fix for #1313:

$ strace -cf -e 'write' -- fd -j1 -u | grep -i '/[^/]*[0-9].c$' | wc -l
% time seconds usecs/call calls errors syscall

---

100.00 0.392255 11 32957 write
$ strace -cf -e 'write' -- find | grep -i '/[^/]*[0-9].c$' | wc -l
% time seconds usecs/call calls errors syscall

---

100.00 0.023571 7 3104 write

I think my reasoning in #1452 doesn't really apply to the -j1 case. We probably need additional buffering there. I'll split this out as a separate issue.

BTW, maybe you should default to -j1 when writing to a pipe? Usually, the pipe will be slower consuming the path names than find(1) producing them.

[tavianator]

BTW, maybe you should default to -j1 when writing to a pipe? Usually, the pipe will be slower consuming the path names than find(1) producing them.

There are two important cases where this isn't true:

• When I/O is slow, multi threaded I/O can be a huge improvement
• When filtering, the output may be very small but we want to parallelize the filtering as much as possible