2020.0.0: test suite is failing #12
Comments
|
Closing. Looks like that projet has nothing to do with work which has been done recently on ksh. |
|
@kloczek A bit frustrating to see issues opened and closed within a couple of minutes, and questions answered within the same time lapse by the originating author. I would have been tempted to delete this issue altogether, but then we would have lost the prime motivation on you initial visit to this project. We are all committed to have a working KornShell in all environments. So could you please elaborate on your requirements? The 2020 version you mention is a solo initiative by some which unfortunately weared the AT&T hat while not being supported in any way by AT&T. Nor are we for that matter. Our difference is in our name; we are a community edition where the community is important. So, as a KornShell user, if you have important feedback relative to the existence and use of a |
|
It is a bit frustrating to see abandonedo meson and other things which are now now allow reproduce the same effect. Is it any plan to reintroduce meson files here? |
|
@kloczek I fully understand your frustration as a user depending on a package that fails. No, meson is not on our roadmap. Our initial steps as a new community will be to ensure that all environments have a working copy of In this respect, changing the build system, is all but a priority. The importance is that the product gets built. How it is built is of secondary or even tertiary importance. I have already given my opinion on the build system. This opinion is not necessarily shared. Time will come to discuss this. But not now. |
Why? So it will be necessary again sit upside down on chair to have ksh with LTO/PGO? :/ To be honest I don't care about Self Adoration Society aka "community". I care only about the code. PS. I'm really disappointed now that some programmers do not understand how important is to have proper build framework. |
|
@kloczek If your topic is to theorise on a build system, by my guest, elsewhere, a this is not among the immediate tasks we have to solve here. But please share if you have KornShell specific comments, suggestions, or even if you want to share the workload this community is putting in after hours. Otherwise I suggest we do indeed close this issue. |
|
Issue is that As long as I've been using completely different method on testing that ticket is a bit useless and to be honest it is a bit discouraging to even try to reproduce that as long as there is no here proper build framework. Do you see my point? It was very good reason to use meson. As long as that part is already done pushing that on the bottom of the ToDo list is rally hard to understand. Proper build framework like meson based will allow easy build and test it on many OS types with %100. reuse of the build procedure and testing it with different compiler, optimisation or with coverage or other quality control code filters/tools. Do you see that now? Without that quite quickly will be wasted tenths times more time on trying retest ksh in different contexts. |
|
@kloczek I don't see your point. But happy to discuss it elsewhere. I have created this issue if you want to discuss this further. And obviously anybody interested is welcome. In the meantime I will close this issue here. In due time we can reopen it :-) |
|
as @marcastel already pointed out, reinstating meson as build system is currently not under consideration. we are considering to also support cmake in the future. and I do not agree that the build system in itself is really of much relevance. what +is+ relevant is that ksh93 builds everywhere (again). there also are/were several people objecting against meson. personally, I simply do not care too much as long as it builds easily. please also note that we were not involved in and have not stopped ksh2020. the one to two guys behind that effort have stopped it (for the time being or forever: I have no idea) when ATT for good reasons did no longer grant them the opportunity to pretend they were maintaining the "real" ksh93 "officially". I am very happy about ATT's decision. as you can find out yourself by reading through the issues in the ATT ksh repo (if you care...) ksh2020 has had lots of trouble and was doing harm to stability and compatibility and performance relative to ksh93u+ (at least that is my definitive assessment). so the rewind to ksh93u+ has been a "good thing" (™) in my view. but if you do not really care for any of those issues and put more importance on the build system than on the shortcomings of ksh2020 relative to ksh93u+, this is fine with me. but it might then be better to try to get in touch with the guy(s) behind ksh2020... |
Which linker would you like to use?
How would you like to profile? Using the testsuite or some custom code? |
`read -S` now correctly handles nested double quotes
I didn't trust this back in e3d7bf1 (which disabled it for interactive shells) and I trust it less now. In af6a32d/6b380572, this was also disabled for virtual subshells as it caused program flow corruption there. Now, on macOS 10.14.6, a crash occurs when repeatedly running a command with this optimisation: $ ksh -c 'for((i=0;i<100;i++));do print -n "$i ";(sleep 1&);done' 0 1 2 3 4 5 6 7 Illegal instruction Oddly enough it seems that I can only reproduce this crash on macOS -- not on Linux, OpenBSD, or Solaris. It could be a macOS bug, particularly given the odd message in the stack trace below. I've had enough, though. Out it comes. Things now work fine, the reproducer is fixed on macOS, and it didn't optimise much anyway. The double-fork issue discussed in e3d7bf1 remains. ________ For future reference, here's an lldb debugger session with a stack trace. It crashes on calling calloc() (via sh_calloc(), via sh_newof()) in jobsave_create(). This is not an invalid pointer problem as we're allocating new memory, so it does look like an OS bug. The "BUG IN CLIENT OF LIBPLATFORM" message is interesting. $ lldb -- arch/*/bin/ksh -c 'for((i=0;i<100;i++));do print -n "$i ";(sleep 1&);done' (lldb) target create "arch/darwin.i386-64/bin/ksh" Current executable set to 'arch/darwin.i386-64/bin/ksh' (x86_64). (lldb) settings set -- target.run-args "-c" "for((i=0;i<100;i++));do print -n \"$i \";(sleep 1&);done" (lldb) run error: shell expansion failed (reason: lldb-argdumper exited with error 2). consider launching with 'process launch'. (lldb) process launch Process 35038 launched: '/usr/local/src/ksh93/ksh/arch/darwin.i386-64/bin/ksh' (x86_64) 0 1 2 3 4 5 6 7 8 9 Process 35038 stopped * thread ksh-community#1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) frame #0: 0x00007fff70deb1c2 libsystem_platform.dylib`_os_unfair_lock_recursive_abort + 23 libsystem_platform.dylib`_os_unfair_lock_recursive_abort: -> 0x7fff70deb1c2 <+23>: ud2 libsystem_platform.dylib`_os_unfair_lock_unowned_abort: 0x7fff70deb1c4 <+0>: movl %edi, %eax 0x7fff70deb1c6 <+2>: leaq 0x1a8a(%rip), %rcx ; "BUG IN CLIENT OF LIBPLATFORM: Unlock of an os_unfair_lock not owned by current thread" 0x7fff70deb1cd <+9>: movq %rcx, 0x361cb16c(%rip) ; gCRAnnotations + 8 Target 0: (ksh) stopped. (lldb) bt * thread ksh-community#1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) * frame #0: 0x00007fff70deb1c2 libsystem_platform.dylib`_os_unfair_lock_recursive_abort + 23 frame ksh-community#1: 0x00007fff70de7c9a libsystem_platform.dylib`_os_unfair_lock_lock_slow + 239 frame ksh-community#2: 0x00007fff70daa3bd libsystem_malloc.dylib`tiny_malloc_should_clear + 188 frame ksh-community#3: 0x00007fff70daa20f libsystem_malloc.dylib`szone_malloc_should_clear + 66 frame ksh-community#4: 0x00007fff70dab444 libsystem_malloc.dylib`malloc_zone_calloc + 99 frame ksh-community#5: 0x00007fff70dab3c4 libsystem_malloc.dylib`calloc + 30 frame ksh-community#6: 0x000000010003fa5d ksh`sh_calloc(nmemb=1, size=16) at init.c:264:13 frame ksh-community#7: 0x000000010004f8a6 ksh`jobsave_create(pid=35055) at jobs.c:272:8 frame ksh-community#8: 0x000000010004ed42 ksh`job_reap(sig=20) at jobs.c:363:9 frame ksh-community#9: 0x000000010004ff6f ksh`job_waitsafe(sig=20) at jobs.c:511:3 frame ksh-community#10: 0x00007fff70de9b5d libsystem_platform.dylib`_sigtramp + 29 frame ksh-community#11: 0x00007fff70d39ac4 libsystem_kernel.dylib`__fork + 12 frame ksh-community#12: 0x00007fff70c57d80 libsystem_c.dylib`fork + 17 frame ksh-community#13: 0x000000010009590d ksh`sh_exec(t=0x0000000101005d30, flags=4) at xec.c:1883:16 frame ksh-community#14: 0x0000000100096013 ksh`sh_exec(t=0x0000000101005d30, flags=4) at xec.c:2019:4 frame ksh-community#15: 0x0000000100096c4f ksh`sh_exec(t=0x0000000101005a40, flags=5) at xec.c:2213:9 frame ksh-community#16: 0x0000000100096013 ksh`sh_exec(t=0x0000000101005a40, flags=5) at xec.c:2019:4 frame ksh-community#17: 0x000000010001c23f ksh`exfile(iop=0x0000000100405750, fno=-1) at main.c:603:4 frame ksh-community#18: 0x000000010001b23c ksh`sh_main(ac=3, av=0x00007ffeefbff4f0, userinit=0x0000000000000000) at main.c:365:2 frame ksh-community#19: 0x0000000100000776 ksh`main(argc=3, argv=0x00007ffeefbff4f0) at pmain.c:45:9 frame ksh-community#20: 0x00007fff70bfe3d5 libdyld.dylib`start + 1
The ASan crash in basic.sh when sourcing multiple files is caused by a bug that is similar to the crash fixed in 59a5672. This is the trace for the regression test crash (note that in order to see the trace, the 2>/dev/null redirect must be disabled): ==1899388==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150000005b0 at pc 0x55a5e3f9432a bp 0x7ffeb91ea110 sp 0x7ffeb91ea100 WRITE of size 8 at 0x6150000005b0 thread T0 #0 0x55a5e3f94329 in funct /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:967 ksh-community#1 0x55a5e3f96f77 in item /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:1349 ksh-community#2 0x55a5e3f90c9f in term /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:642 ksh-community#3 0x55a5e3f90ac1 in list /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:613 ksh-community#4 0x55a5e3f90845 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:561 ksh-community#5 0x55a5e3f909e0 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:586 ksh-community#6 0x55a5e3f8fd5e in sh_parse /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:438 ksh-community#7 0x55a5e3fc43c1 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:635 ksh-community#8 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318 ksh-community#9 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254 ksh-community#10 0x55a5e3fd01d4 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1932 ksh-community#11 0x55a5e3fc4544 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:651 ksh-community#12 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318 ksh-community#13 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254 ksh-community#14 0x55a5e3ecc1cd in exfile /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:604 ksh-community#15 0x55a5e3ec9e7f in sh_main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:369 ksh-community#16 0x55a5e3ec801d in main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/pmain.c:41 ksh-community#17 0x7f637b4db2cf (/usr/lib/libc.so.6+0x232cf) ksh-community#18 0x7f637b4db389 in __libc_start_main (/usr/lib/libc.so.6+0x23389) ksh-community#19 0x55a5e3ec7f24 in _start ../sysdeps/x86_64/start.S:115 Code in question: https://github.com/ksh93/ksh/blob/8d57369b0cb39074437dd82924b604155e30e1e0/src/cmd/ksh93/sh/parse.c#L963-L968 To avoid any more similar crashes, all of the fixes introduced in 7e317c5 that set slp->slptr to null have been improved with the fix in 59a5672.
I didn't trust this back in e3d7bf1 (which disabled it for interactive shells) and I trust it less now. In af6a32d/6b380572, this was also disabled for virtual subshells as it caused program flow corruption there. Now, on macOS 10.14.6, a crash occurs when repeatedly running a command with this optimisation: $ ksh -c 'for((i=0;i<100;i++));do print -n "$i ";(sleep 1&);done' 0 1 2 3 4 5 6 7 Illegal instruction Oddly enough it seems that I can only reproduce this crash on macOS -- not on Linux, OpenBSD, or Solaris. It could be a macOS bug, particularly given the odd message in the stack trace below. I've had enough, though. Out it comes. Things now work fine, the reproducer is fixed on macOS, and it didn't optimise much anyway. The double-fork issue discussed in e3d7bf1 remains. ________ For future reference, here's an lldb debugger session with a stack trace. It crashes on calling calloc() (via sh_calloc(), via sh_newof()) in jobsave_create(). This is not an invalid pointer problem as we're allocating new memory, so it does look like an OS bug. The "BUG IN CLIENT OF LIBPLATFORM" message is interesting. $ lldb -- arch/*/bin/ksh -c 'for((i=0;i<100;i++));do print -n "$i ";(sleep 1&);done' (lldb) target create "arch/darwin.i386-64/bin/ksh" Current executable set to 'arch/darwin.i386-64/bin/ksh' (x86_64). (lldb) settings set -- target.run-args "-c" "for((i=0;i<100;i++));do print -n \"$i \";(sleep 1&);done" (lldb) run error: shell expansion failed (reason: lldb-argdumper exited with error 2). consider launching with 'process launch'. (lldb) process launch Process 35038 launched: '/usr/local/src/ksh93/ksh/arch/darwin.i386-64/bin/ksh' (x86_64) 0 1 2 3 4 5 6 7 8 9 Process 35038 stopped * thread ksh-community#1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) frame #0: 0x00007fff70deb1c2 libsystem_platform.dylib`_os_unfair_lock_recursive_abort + 23 libsystem_platform.dylib`_os_unfair_lock_recursive_abort: -> 0x7fff70deb1c2 <+23>: ud2 libsystem_platform.dylib`_os_unfair_lock_unowned_abort: 0x7fff70deb1c4 <+0>: movl %edi, %eax 0x7fff70deb1c6 <+2>: leaq 0x1a8a(%rip), %rcx ; "BUG IN CLIENT OF LIBPLATFORM: Unlock of an os_unfair_lock not owned by current thread" 0x7fff70deb1cd <+9>: movq %rcx, 0x361cb16c(%rip) ; gCRAnnotations + 8 Target 0: (ksh) stopped. (lldb) bt * thread ksh-community#1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) * frame #0: 0x00007fff70deb1c2 libsystem_platform.dylib`_os_unfair_lock_recursive_abort + 23 frame ksh-community#1: 0x00007fff70de7c9a libsystem_platform.dylib`_os_unfair_lock_lock_slow + 239 frame ksh-community#2: 0x00007fff70daa3bd libsystem_malloc.dylib`tiny_malloc_should_clear + 188 frame ksh-community#3: 0x00007fff70daa20f libsystem_malloc.dylib`szone_malloc_should_clear + 66 frame ksh-community#4: 0x00007fff70dab444 libsystem_malloc.dylib`malloc_zone_calloc + 99 frame ksh-community#5: 0x00007fff70dab3c4 libsystem_malloc.dylib`calloc + 30 frame ksh-community#6: 0x000000010003fa5d ksh`sh_calloc(nmemb=1, size=16) at init.c:264:13 frame ksh-community#7: 0x000000010004f8a6 ksh`jobsave_create(pid=35055) at jobs.c:272:8 frame ksh-community#8: 0x000000010004ed42 ksh`job_reap(sig=20) at jobs.c:363:9 frame ksh-community#9: 0x000000010004ff6f ksh`job_waitsafe(sig=20) at jobs.c:511:3 frame ksh-community#10: 0x00007fff70de9b5d libsystem_platform.dylib`_sigtramp + 29 frame ksh-community#11: 0x00007fff70d39ac4 libsystem_kernel.dylib`__fork + 12 frame ksh-community#12: 0x00007fff70c57d80 libsystem_c.dylib`fork + 17 frame ksh-community#13: 0x000000010009590d ksh`sh_exec(t=0x0000000101005d30, flags=4) at xec.c:1883:16 frame ksh-community#14: 0x0000000100096013 ksh`sh_exec(t=0x0000000101005d30, flags=4) at xec.c:2019:4 frame ksh-community#15: 0x0000000100096c4f ksh`sh_exec(t=0x0000000101005a40, flags=5) at xec.c:2213:9 frame ksh-community#16: 0x0000000100096013 ksh`sh_exec(t=0x0000000101005a40, flags=5) at xec.c:2019:4 frame ksh-community#17: 0x000000010001c23f ksh`exfile(iop=0x0000000100405750, fno=-1) at main.c:603:4 frame ksh-community#18: 0x000000010001b23c ksh`sh_main(ac=3, av=0x00007ffeefbff4f0, userinit=0x0000000000000000) at main.c:365:2 frame ksh-community#19: 0x0000000100000776 ksh`main(argc=3, argv=0x00007ffeefbff4f0) at pmain.c:45:9 frame ksh-community#20: 0x00007fff70bfe3d5 libdyld.dylib`start + 1
The ASan crash in basic.sh when sourcing multiple files is caused by a bug that is similar to the crash fixed in f24040e. This is the trace for the regression test crash (note that in order to see the trace, the 2>/dev/null redirect must be disabled): ==1899388==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150000005b0 at pc 0x55a5e3f9432a bp 0x7ffeb91ea110 sp 0x7ffeb91ea100 WRITE of size 8 at 0x6150000005b0 thread T0 #0 0x55a5e3f94329 in funct /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:967 ksh-community#1 0x55a5e3f96f77 in item /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:1349 ksh-community#2 0x55a5e3f90c9f in term /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:642 ksh-community#3 0x55a5e3f90ac1 in list /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:613 ksh-community#4 0x55a5e3f90845 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:561 ksh-community#5 0x55a5e3f909e0 in sh_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:586 ksh-community#6 0x55a5e3f8fd5e in sh_parse /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/parse.c:438 ksh-community#7 0x55a5e3fc43c1 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:635 ksh-community#8 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318 ksh-community#9 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254 ksh-community#10 0x55a5e3fd01d4 in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1932 ksh-community#11 0x55a5e3fc4544 in sh_eval /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:651 ksh-community#12 0x55a5e4012172 in b_dot_cmd /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/bltins/misc.c:318 ksh-community#13 0x55a5e3fca3cb in sh_exec /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/xec.c:1254 ksh-community#14 0x55a5e3ecc1cd in exfile /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:604 ksh-community#15 0x55a5e3ec9e7f in sh_main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/main.c:369 ksh-community#16 0x55a5e3ec801d in main /home/johno/GitRepos/KornShell/ksh/src/cmd/ksh93/sh/pmain.c:41 ksh-community#17 0x7f637b4db2cf (/usr/lib/libc.so.6+0x232cf) ksh-community#18 0x7f637b4db389 in __libc_start_main (/usr/lib/libc.so.6+0x23389) ksh-community#19 0x55a5e3ec7f24 in _start ../sysdeps/x86_64/start.S:115 Code in question: https://github.com/ksh93/ksh/blob/8d57369b0cb39074437dd82924b604155e30e1e0/src/cmd/ksh93/sh/parse.c#L963-L968 To avoid any more similar crashes, all of the fixes introduced in 69d37d5 that set slp->slptr to null have been improved with the fix in f24040e.
All is on att/ast#1478
PS. Any plans to make forst release?
The text was updated successfully, but these errors were encountered: