Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault when spawned task uses regex during exit #57817

Open
IanButterworth opened this issue Mar 18, 2025 · 4 comments
Open

Segfault when spawned task uses regex during exit #57817

IanButterworth opened this issue Mar 18, 2025 · 4 comments
Labels
bug Indicates an unexpected problem or unintended behavior multithreading Base.Threads and related functionality

Comments

@IanButterworth
Copy link
Member

IanButterworth commented Mar 18, 2025 •
edited
Loading

Update: this appears to be just the regex finalizer being called at exit before the task has finished. i.e. one fix is #51466

I have a test suite that uses a custom logger, and a task that continues beyond the end of a test run that sometimes hits this segfault.

It seems pcre is in a bad state

julia/base/regex.jl

Line 689 in 213feae

mat = match(itr.regex, itr.string, offset,

On 1.10.8

[16993] signal (11.2): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x31645430)
iterate at ./regex.jl:689 [inlined]
iterate at ./regex.jl:687 [inlined]
DateFormat at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:412
#format#32 at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:701
format at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:700 [inlined]
#37 at [private code]
#handle_message#13 at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/transformer.jl:22 [inlined]
handle_message at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/transformer.jl:20
jfptr_handle_message_144283 at [private sysimage]
_jl_invoke at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:3077
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/julia.h:1982 [inlined]
do_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/builtins.c:768
#handle_message#4 at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/earlyfiltered.jl:26
handle_message at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/earlyfiltered.jl:24
jfptr_handle_message_144280 at [private sysimage]
_jl_invoke at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:3077
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/julia.h:1982 [inlined]
do_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/builtins.c:768
#handle_message#6 at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/tee.jl:24
handle_message at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/tee.jl:21 [inlined]
#handle_message#4 at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/earlyfiltered.jl:26 [inlined]
handle_message at /home/runner/.julia/packages/LoggingExtras/cFgEq/src/CompositionalLoggers/earlyfiltered.jl:24
jfptr_handle_message_144275 at [private sysimage]
_jl_invoke at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:3077
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/julia.h:1982 [inlined]
jl_f__call_latest at /cache/build/tester-amdci4-11/julialang/julia-master/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
macro expansion at ./logging.jl:342 [inlined]
#1463 at [private code]
unknown function (ip: 0x71a30e6eb4d2)
_jl_invoke at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:3077
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/julia.h:1982 [inlined]
start_task at /cache/build/tester-amdci4-11/julialang/julia-master/src/task.c:1256
Allocations: 25116319 (Pool: 25082353; Big: 33966); GC: 40
@IanButterworth IanButterworth added bug Indicates an unexpected problem or unintended behavior logging The logging framework multithreading Base.Threads and related functionality labels Mar 18, 2025
@IanButterworth
Copy link
Member Author

On a quick look it seems like uses of DATEFORMAT_REGEX_CACHE should be inside this lock?

julia/stdlib/Dates/src/io.jl

Lines 401 to 429 in 5595d20

lock(DATEFORMAT_REGEX_LOCK)
try
dateformat_regex_hash = hash(keys(CONVERSION_SPECIFIERS))
if dateformat_regex_hash != DATEFORMAT_REGEX_HASH[]
DATEFORMAT_REGEX_HASH[] = dateformat_regex_hash
DATEFORMAT_REGEX_CACHE[] = compute_dateformat_regex(CONVERSION_SPECIFIERS)
end
finally
unlock(DATEFORMAT_REGEX_LOCK)
end
for m in eachmatch(DATEFORMAT_REGEX_CACHE[], f)
tran = replace(f[prev_offset:prevind(f, m.offset)], r"\\(.)" => s"\1")
if !isempty(prev)
letter, width = prev
push!(tokens, DatePart{letter}(width, isempty(tran)))
end
if !isempty(tran)
push!(tokens, Delim(length(tran) == 1 ? first(tran) : tran))
end
letter = f[m.offset]
width = length(m.match)
prev = (letter, width)
prev_offset = m.offset + width
end

@IanButterworth IanButterworth mentioned this issue Mar 18, 2025
Closed
@Octogonapus
Copy link
Contributor

I got a core dump with the segfault.

#0  0x0000000031e351f0 in ?? ()
#1  0x000070df8fa11d9f in _pcre2_memctl_malloc_8 () from actions-runner/_work/_tool/julia/1.10.8/x64/bin/../lib/julia/libpcre2-8.so
#2  0x000070df8fa6a6ae in pcre2_match_data_create_8 () from actions-runner/_work/_tool/julia/1.10.8/x64/bin/../lib/julia/libpcre2-8.so
#3  0x000070dfcab7aeb8 in create_match_data () at pcre.jl:225
#4  exec_r_data () at pcre.jl:219
#5  julia_match_139569 () at regex.jl:409
#6  0x000070dfca91ffa7 in iterate () at regex.jl:689
#7  iterate () at regex.jl:687
#8  julia_DateFormat_52033 () at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:412
#9  0x000070dfcb0bd295 in julia_#format#32_162419 () at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:701
#10 0x000070dfcb0caf00 in format () at /cache/build/tester-amdci4-11/julialang/julia-master/usr/share/julia/stdlib/v1.10/Dates/src/io.jl:700

Dump of assembler code for function _pcre2_memctl_malloc_8:
   0x000070df8fa11d90 <+0>:     test   %rsi,%rsi
   0x000070df8fa11d93 <+3>:     push   %rbx
   0x000070df8fa11d94 <+4>:     mov    %rsi,%rbx
   0x000070df8fa11d97 <+7>:     je     0x70df8fa11dc0 <_pcre2_memctl_malloc_8+48>
   0x000070df8fa11d99 <+9>:     mov    0x10(%rsi),%rsi
   0x000070df8fa11d9d <+13>:    call   *(%rbx)
=> 0x000070df8fa11d9f <+15>:    test   %rax,%rax
   0x000070df8fa11da2 <+18>:    je     0x70df8fa11df0 <_pcre2_memctl_malloc_8+96>
   0x000070df8fa11da4 <+20>:    mov    (%rbx),%rdx
   0x000070df8fa11da7 <+23>:    mov    %rdx,(%rax)
   0x000070df8fa11daa <+26>:    mov    0x8(%rbx),%rdx
   0x000070df8fa11dae <+30>:    mov    %rdx,0x8(%rax)
   0x000070df8fa11db2 <+34>:    mov    0x10(%rbx),%rdx
   0x000070df8fa11db6 <+38>:    mov    %rdx,0x10(%rax)
   0x000070df8fa11dba <+42>:    pop    %rbx
   0x000070df8fa11dbb <+43>:    ret

@Octogonapus
Copy link
Contributor

Octogonapus commented Mar 19, 2025 •
edited
Loading

Here is an MWE. This illustrates that PCRE is not safe to use from another thread while Julia is exiting.

re = Regex("")
Threads.@spawn match(re, "", 1, UInt32(0))

segfaults with

[1350612] signal (11.2): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x10cfa9b0)
#1 at /home/salmon/git/julia/dateformat_crash.jl:2
unknown function (ip: 0x7fc7614f6912)
_jl_invoke at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/tester-amdci4-11/julialang/julia-master/src/gf.c:3077
jl_apply at /cache/build/tester-amdci4-11/julialang/julia-master/src/julia.h:1982 [inlined]
start_task at /cache/build/tester-amdci4-11/julialang/julia-master/src/task.c:1256
Allocations: 2909 (Pool: 2899; Big: 10); GC: 0

from gdb:

#0  0x0000000010cfa9b0 in ?? ()
#1  0x00007fc72d811d9f in _pcre2_memctl_malloc_8 () from /home/salmon/.julia/juliaup/julia-1.10.8+0.x64.linux.gnu/bin/../lib/julia/libpcre2-8.so
#2  0x00007fc72d86a6ae in pcre2_match_data_create_8 () from /home/salmon/.julia/juliaup/julia-1.10.8+0.x64.linux.gnu/bin/../lib/julia/libpcre2-8.so
#3  0x00007fc762adad3f in create_match_data () at pcre.jl:225
#4  exec_r_data () at pcre.jl:219
#5  julia_match_71632 () at regex.jl:409

Dump of assembler code for function _pcre2_memctl_malloc_8:
   0x00007fc72d811d90 <+0>:     test   %rsi,%rsi
   0x00007fc72d811d93 <+3>:     push   %rbx
   0x00007fc72d811d94 <+4>:     mov    %rsi,%rbx
   0x00007fc72d811d97 <+7>:     je     0x7fc72d811dc0 <_pcre2_memctl_malloc_8+48>
   0x00007fc72d811d99 <+9>:     mov    0x10(%rsi),%rsi
   0x00007fc72d811d9d <+13>:    call   *(%rbx)
=> 0x00007fc72d811d9f <+15>:    test   %rax,%rax
   0x00007fc72d811da2 <+18>:    je     0x7fc72d811df0 <_pcre2_memctl_malloc_8+96>
   0x00007fc72d811da4 <+20>:    mov    (%rbx),%rdx
   0x00007fc72d811da7 <+23>:    mov    %rdx,(%rax)
   0x00007fc72d811daa <+26>:    mov    0x8(%rbx),%rdx
   0x00007fc72d811dae <+30>:    mov    %rdx,0x8(%rax)
   0x00007fc72d811db2 <+34>:    mov    0x10(%rbx),%rdx
   0x00007fc72d811db6 <+38>:    mov    %rdx,0x10(%rax)
   0x00007fc72d811dba <+42>:    pop    %rbx
   0x00007fc72d811dbb <+43>:    ret

@IanButterworth IanButterworth removed the logging The logging framework label Mar 19, 2025
@IanButterworth IanButterworth changed the title Regex segfault when logging during test exit Regex segfault when spawned task uses regex during exit Mar 19, 2025
@IanButterworth IanButterworth mentioned this issue Mar 20, 2025
Open
@IanButterworth
Copy link
Member Author

I can't get this to reproduce on master, but I have a proposed fix for 1.11 & 1.10 #57834

@IanButterworth IanButterworth changed the title Regex segfault when spawned task uses regex during exit Segfault when spawned task uses regex during exit Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior multithreading Base.Threads and related functionality
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@IanButterworth @Octogonapus