Apache 2.4.48 + php 5.6.40 crash Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c

[ecomstation]

Describe the bug
Apache crash/exit (no exceptQ dmp) and don't restart again, i need to do a setboot /b

To Reproduce
Quite often about once per day

Expected behavior
i'd expect to be able to restart apache, instead it don't restart anymore
and i've to do a setboot /b to have it running again

server (please complete the following information):
VM (2 cores, 3GB ram) under Vbox

Additional context

Server Version: Apache/2.4.46 (OS/2) OpenSSL/1.1.1j PHP/5.6.40Server Built: Apr 1 2021 14:58:19

Apache error log:

[Fri Jun 04 10:28:17.649000 2021] [mpm_mpmt_os2:notice] [pid 493:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
[Fri Jun 04 10:30:47.086000 2021] [mpm_mpmt_os2:notice] [pid 493:tid 1] (OS 10035)Resource temporarily unavailable: apr_socket_accept
Assertion info: 105
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c, line 564

Killed by SIGABRT
pid=0x01f2 ppid=0x01b4 tid=0x0001 slot=0x007d Assertion info: 105
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c, line 564

Killed by SIGABRT
pid=0x01f4 ppid=0x01b4 tid=0x0012 slot=0x00b9 pri=0x0200 mc=0x0001 ps=0x0010
D:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Fri Jun 04 10:52:07.031000 2021] [mpm_mpmt_os2:error] [pid 502:tid 1] (OS 105)The previous ownership of this semaphore has ended. : Sem owner died pidSemaphoreOwner = 501, tidSemaphoreOwner = 1, ulRequestCount = 1 rc = 105 at ap_mpm_child_main #0
[Fri Jun 04 10:52:07.037000 2021] [mpm_mpmt_os2:error] [pid 502:tid 1] (OS 105)The previous ownership of this semaphore has ended. : AH00194: apr_socket_accept
[Fri Jun 04 10:52:07.028000 2021] [mpm_mpmt_os2:error] [pid 503:tid 1] (OS 105)The previous ownership of this semaphore has ended. : Sem owner died pidSemaphoreOwner = 501, tidSemaphoreOwner = 1, ulRequestCount = 1 rc = 105 at ap_mpm_child_main #0
[Fri Jun 04 10:52:07.038000 2021] [mpm_mpmt_os2:error] [pid 503:tid 1] (OS 105)The previous ownership of this semaphore has ended. : AH00194: apr_socket_accept

after reboot:

[Fri Jun 04 11:33:38.336000 2021] [mpm_mpmt_os2:error] [pid 67:tid 1] AH00200: DosGetNamedSharedMem returned 0
[Fri Jun 04 11:33:38.365000 2021] [mpm_mpmt_os2:notice] [pid 67:tid 1] AH00206: Apache/2.4.46 (OS/2) OpenSSL/1.1.1j PHP/5.6.40 configured -- resuming normal operations

[ecomstation]

i've seen that i've an old version of libcx0.dll (0.6.6)
I've now updated it to version 0.7.0

even if yum update libcx fail to update it automatically
i've updated it manually
let's see if it improves the situation

[ecomstation]

even after upgrading to libcx 0.7.0 i had one of these:

HTTPD.EXE
PHP5 0:0078a90a
cs:eip=005b:1dbca90a ss:esp=0053:0022fbc0 ebp=0022fc08
ds=0053 es=0053 fs=150b gs=0000 efl=00010206
eax=00000000 ebx=0003fff0 ecx=00000000 edx=22738300 edi=22738308 esi=20111600
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
0022fbc8
ds=0053 es=Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 7910 bytes
ProcDesc structs used now: 8
FileDesc structs used now: 17
SharedFileDesc structs used now: 10
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x17ba ppid=0x10f1 tid=0x0001 slot=0x009b pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Sun Jun 13 11:01:21.532000 2021] [mpm_mpmt_os2:error] [pid 6334:tid 1] (OS 10038)Socket operation on non-socket: AH00194: apr_socket_accept
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 3492 bytes
ProcDesc structs used now: 5
FileDesc structs used now: 4
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x18bc ppid=0x0010 tid=0x0003 slot=0x00b2 pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 2856 bytes
ProcDesc structs used now: 4
FileDesc structs used now: 3
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

[dmik]

There is not enough information. Do you have anything in /var/log/libcx? I need to see all LIBCx logs (from all involved processes). So far it looks like one process crashes while holding an internal LIBCx mutex and this harms all other LIBCx processes. I need to see where exactly that first LIBCx process crashed (and what it was doing inside LIBCx).

[ecomstation]

Hi,

yes, i've a lot of logs in that directory, but they are from LIBCX 0.6.6., i've upgraded to 0version .7.0 29 days ago
and after i don't have anymore these logs, but i still have the issue on apache

Opened log at 2021-05-07 13:27:11.73
Process ID: 0x265 (613) Parent PID: 0x10 (16) Type: 2
Exe hmte : 0xcad (X:\APACHE\BIN\HTTPD.EXE)
First arg : X:\apache\bin\httpd.exe
Second arg: -d x:\apache
Cur dir : x:\apache
CRT Module: LIBCN0 hmod=0x667 (C:\USR\LIB\LIBCN0.DLL)
__libc_logInit: addr 0x1ec1f259 iObj=0 offObj=0x4f259
Millsecond Timestamp.
| Thread ID.
| | Call Nesting Level.
| | | Log Group.
| | | | Message Type.
| | | | | errno in hex (0xface if not available).
| | | | | | Function Name.
| | | | | | | Millisconds In function (Optional).
v v v v v v v v
xxxxxxxx tt nn gggg dddd eeee function [(ms)]: message
LIBCx version : 0.6.6
LIBCx module : C:\USR\LIB\LIBCX0.DLL (hmod=075e)
02ec36e8 01 ff 0000 Asrt: Assertion Failed!!!
02ec36e8 01 ff 0000 Asrt: Function:
02ec36e8 01 ff 0000 Asrt: File: D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c
02ec36e8 01 ff 0000 Asrt: Line: 189
02ec36e8 01 ff 0000 Asrt: Expr: arc == NO_ERROR
02ec36e8 01 ff 0000 Asrt: 105

Opened log at 2021-06-04 11:30:42.65
Process ID: 0x20b (523) Parent PID: 0x10 (16) Type: 2
Exe hmte : 0x9bc (X:\APACHE\BIN\HTTPD.EXE)
First arg : X:\apache\bin\httpd.exe
Second arg: -d x:\apache
Cur dir : X:\apache
CRT Module: LIBCN0 hmod=0x667 (C:\USR\LIB\LIBCN0.DLL)
__libc_logInit: addr 0x1ec0f259 iObj=0 offObj=0x4f259
Millsecond Timestamp.
| Thread ID.
| | Call Nesting Level.
| | | Log Group.
| | | | Message Type.
| | | | | errno in hex (0xface if not available).
| | | | | | Function Name.
| | | | | | | Millisconds In function (Optional).
v v v v v v v v
xxxxxxxx tt nn gggg dddd eeee function [(ms)]: message
LIBCx version : 0.6.6
LIBCx module : C:\USR\LIB\LIBCX0.DLL (hmod=075f)
02816271 01 ff 0000 Asrt: Assertion Failed!!!
02816271 01 ff 0000 Asrt: Function:
02816271 01 ff 0000 Asrt: File: D:/Users/dmik/rpmbuild/BUILD/libcx-0.6.6/src/shared.c
02816271 01 ff 0000 Asrt: Line: 189
02816271 01 ff 0000 Asrt: Expr: arc == NO_ERROR
02816271 01 ff 0000 Asrt: 105

[psmedley]

That log suggests you're running libcx 0.6.6:
LIBCx version : 0.6.6

I'd suggest you have a more recent example that is from 0.7.0 based on the log at https://mantis.smedley.id.au/view.php?id=736

[ecomstation]

I repeat:

all the logs in that directory are from 0.6.6, on 4th jun i upgrded to 0.7.0

after upgrading to 0.7.0 i don't have anymore logs in /var/log/libcx

but apache has the same crashes

HTTPD.EXE
PHP5 0:0078a90a
cs:eip=005b:1dbca90a ss:esp=0053:0022fbc0 ebp=0022fc08
ds=0053 es=0053 fs=150b gs=0000 efl=00010206
eax=00000000 ebx=0003fff0 ecx=00000000 edx=22738300 edi=22738308 esi=20111600
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
0022fbc8
ds=0053 es=Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 7910 bytes
ProcDesc structs used now: 8
FileDesc structs used now: 17
SharedFileDesc structs used now: 10
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x17ba ppid=0x10f1 tid=0x0001 slot=0x009b pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
[Sun Jun 13 11:01:21.532000 2021] [mpm_mpmt_os2:error] [pid 6334:tid 1] (OS 10038)Socket operation on non-socket: AH00194: apr_socket_accept
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 3492 bytes
ProcDesc structs used now: 5
FileDesc structs used now: 4
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

Killed by SIGABRT
pid=0x18bc ppid=0x0010 tid=0x0003 slot=0x00b2 pri=0x0200 mc=0x0001 ps=0x0010
X:\APACHE\BIN\HTTPD.EXE
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.
Assertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size tAssertion info: 105

===== LIBCx resource usage =====
Reserved memory size: 2097152 bytes
Committed memory size: 65536 bytes
Heap size total: 65120 bytes
Heap size used now: 2856 bytes
ProcDesc structs used now: 4
FileDesc structs used now: 3
SharedFileDesc structs used now: 1
===== LIBCx stats end =====
Assertion failed: arc == NO_ERROR, file D:/Users/dmik/rpmbuild/BUILD/libcx-0.7.0/src/shared.c, line 564

[dmik]

This is still quite useless, sorry. What you need to do is: reboot, clean the /var/log/libcx dir (and all NNNN.TRP files from EXCEPTQ you will find around) then reproduce the problem, pack all libcx logs (and .TRP files you will find — they may be anywhere on your disk because EXCEPTQ writes them to the process current directory), zip them up and attach here.

The thing is that I need only that libcx log file where the original problem happens, not all the others which are just consequences.

[ecomstation]

cleaned everything

but the problem is not easy to reproduce, it happens about randomly

and after upgrading trom libcx 0.6.6 to 0.7.0 exceptq environment do not work anymore

[ecomstation]

this can help?

from popuplog.os2

07-03-2021 19:39:12 SYS3175 PID 0290 TID 0001 Slot 00f7
X\APACHE\BIN\HTTPD.EXE
c0000005
1eb5f5f9
P1=00000001 P2=00000004 P3=XXXXXXXX P4=XXXXXXXX
EAX=00000000 EBX=000004fc ECX=00000004 EDX=000004fc
ESI=00000004 EDI=00000000
DS=0053 DSACC=d0f3 DSLIM=5fffffff
ES=0053 ESACC=d0f3 ESLIM=5fffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********
CS:EIP=005b:1eb5f5f9 CSACC=d0df CSLIM=5fffffff
SS:ESP=0053:0022f768 SSACC=d0f3 SSLIM=5fffffff
EBP=0022f820 FLG=00010212

LIBCX0.DLL 0001:0000f5f9

---

07-03-2021 19:39:12 SYS3175 PID 0291 TID 0001 Slot 00dc
X:\APACHE\BIN\HTTPD.EXE
c0000005
1eb5f5f9
P1=00000001 P2=00000004 P3=XXXXXXXX P4=XXXXXXXX
EAX=00000000 EBX=000004fc ECX=00000004 EDX=000004fc
ESI=00000004 EDI=00000000
DS=0053 DSACC=d0f3 DSLIM=5fffffff
ES=0053 ESACC=d0f3 ESLIM=5fffffff
FS=150b FSACC=00f3 FSLIM=00000030
GS=0000 GSACC=**** GSLIM=********
CS:EIP=005b:1eb5f5f9 CSACC=d0df CSLIM=5fffffff
SS:ESP=0053:0022f768 SSACC=d0f3 SSLIM=5fffffff
EBP=0022f820 FLG=00010212

LIBCX0.DLL 0001:0000f5f9

[StevenLevine]

Massimo, the trap address implies you are not running a current version of libcx0.dll.

[StevenLevine]

Massimo, you said:

and after upgrading trom libcx 0.6.6 to 0.7.0 exceptq environment do not work anymore

Recall that Paul told you that httpd-2.4.48-os2-20210622-debug.zip prevents exceptq from working. Why are you blaming libcx? You need to use httpd-2.4.48-os2-20210619-debug.zip

[psmedley]

Pretty sure Max isn't running the non libcx build if Apache 2 Cheers, Paul

…

On Mon, 5 July 2021, 09:46 StevenLevine, ***@***.***> wrote: Massimo, you said: >and after upgrading trom libcx 0.6.6 to 0.7.0 exceptq environment do not >work anymore Recall that Paul told you that httpd-2.4.48-os2-20210622-debug.zip prevents exceptq from working. Why are you blaming libcx? You need to use httpd-2.4.48-os2-20210619-debug.zip — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHZY4TUHHRCJS3EHBNCY2DTWD2UVANCNFSM46CSN5JQ> .

[ecomstation]

exactly, i'm running standard apache 2.4.48 build

Server Version: Apache/2.4.48 (OS/2) OpenSSL/1.1.1k PHP/5.6.40
Server MPM: mpmt_os2
Server Built: Jun 6 2021 15:51:55

[ecomstation]

Massimo, the trap address implies you are not running a current version of libcx0.dll.

C:\usr\lib]bldlevel libcx0.dll
Build Level Display Facility Version 6.12.675 Sep 25 2001
(C) Copyright IBM Corporation 1993-2001
Signature: @#bww bitwise works GmbH:0.7#@##1## 2021-02-26 23:57:55 noVendor: bww bitwise works GmbH
Revision: 0.07
Date/Time: 2021-02-26 23:57:55
Build Machine: novator
File Version: 0.7
Description: kLIBC Extension Library

23/07/20 1:15 1.216.805 124 a--- libcn0.dll
26/02/21 21:58 56.436 124 a--- libcx0.dll

libcn0.cll has no bldlevel information

yum libcx0 check-update
Comando sconosciuto: libcx0. Eseguire C:\USR\BIN\YUM --help

[<web.quasarbbs.net>-C:]yum check-update libcx
arcanoae-rel | 1.9 kB 00:00
arcanoae-rel/primary | 19 kB 00:00 ==] 0.0 B/s | 19 kB --:-- ETA
netlabs-rel | 2.9 kB 00:00
netlabs-rel/primary_ | 2.5 MB 00:11 =-] 211 kB/s | 2.4 MB 00:00 ETA

libcx.i686 0.7.0-1.oc00 netlabs-rel

[StevenLevine]

@dmik. I recommend you hold off on this ticket until Paul and I can verify that Massimo's system has been updated to a current set of netlabs-rel DLLs. We will be back after this happens. I apologize for the noise.