Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on debug-status and any client interaction #1264

Open
swaits opened this issue Dec 27, 2024 · 6 comments
Open

Crash on debug-status and any client interaction #1264

swaits opened this issue Dec 27, 2024 · 6 comments

Comments

@swaits
Copy link

swaits commented Dec 27, 2024

Initially noticed problem because jj status on a fresh EndeavourOS install wasn't working with watchman:

❯ jj
watchman client task failed: EOF on Watchman socket
The working copy is clean
Working copy : nyxt 4591 (empty) (no description set)

I noticed in /tmp/<user>/log that it was segfaulting. I found I could reproduce the segfault with just debug-status, too.

❯ watchman debug-status
ROOTS
-----
CLIENTS
-------
225644: 'watchman' 'debug-status'
*** Aborted at 1735272704 (Unix time, try 'date -d @1735272704') ***
*** Signal 11 (SIGSEGV) (0x8) received by PID 225644 (pthread TID 0x7ce784d724c0) (linux TID 225644) (code: address not mapped to object), stack trace: ***
(error retrieving stack trace)
fish: Job 1, 'watchman debug-status' terminated by signal SIGSEGV (Address boundary error

And here's the output from the server:

❯ watchman --persistent --server-encoding=json log-level debug
{
    "version": "2024.12.02.00",
    "log_level": "debug"
}
{
    "log": "2024-12-26T21:11:41,198: [client=8:stm=130659213380832:pid=225633] dispatch_command: log-level (completed)\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:09:04,100: [sanitychecks] starting sanityCheckThread\n",
    "unilateral": true,
    "level": "error"
}
{
    "log": "2024-12-26T21:09:04,107: [client=1:stm=130659213380832:pid=223747] send_error_response: [\"get-sock-name\"], failed: watchman::CommandValidationError: failed to validate command: unknown command get-sock-name\n",
    "unilateral": true,
    "level": "error"
}
{
    "log": "2024-12-26T21:11:44,774: [unix-listener] accepted client:stm=0x76d578203650\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:11:44,774: [client=9:stm=130659215488592:pid=225644] dispatch_command: debug-status\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:11:44,774: [client=9:stm=130659215488592:pid=225644] dispatch_command: debug-status (completed)\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:11:44,921: [NOT_CONN:client=9:stm=130659215488592:pid=225644] client_delete 9\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [sanitychecks] running sanity checks\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [unix-listener] accepted client:stm=0x76d578403ec0\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [client=10:stm=130659217587904:pid=223749] dispatch_command: get-pid\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [client=10:stm=130659217587904:pid=223749] dispatch_command: get-pid (completed)\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [client=10:stm=130659217587904:pid=223749] dispatch_command: watch-list\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [client=10:stm=130659217587904:pid=223749] dispatch_command: watch-list (completed)\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:04,160: [NOT_CONN:client=10:stm=130659217587904:pid=223749] client_delete 10\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,954: [unix-listener] accepted client:stm=0x76d578404240\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,955: [client=11:stm=130659217588800:pid=225949] dispatch_command: get-sockname\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,955: [client=11:stm=130659217588800:pid=225949] dispatch_command: get-sockname (completed)\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,955: [NOT_CONN:client=11:stm=130659217588800:pid=225949] client_delete 11\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,956: [unix-listener] accepted client:stm=0x76d578404240\n",
    "unilateral": true,
    "level": "debug"
}
{
    "log": "2024-12-26T21:12:11,956: [client=12:stm=130659217588800:pid=225947] dispatch_command: watch-project\n",
    "unilateral": true,
    "level": "debug"
}
2024-12-26T21:12:12,352: [cli] failed to identify PDU: fill_buffer: EOF
2024-12-26T21:12:12,352: [cli] unable to talk to your watchman on /tmp/swaits-state/sock! (Success)

The contents of my state dir:

❯ eza --git --icons auto --long /tmp/swaits-state/
.rw------- 2.0k swaits 26 Dec 21:12  log
.rw-r--r--    6 swaits 26 Dec 21:09  pid
srw-------    - swaits 26 Dec 21:09  sock

The version:

❯ watchman --version
2024.12.02.00

Installed via Linuxbrew:

❯ brew list watchman
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-diag
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-make
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-replicate-subscription
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-wait
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchmanctl
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/lib/python3.13/ (15 files)
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/sbom.spdx.json

❯ brew info watchman
brew==> watchman: stable 2024.12.02.00 (bottled), HEAD
Watch files and take action when they change
https://github.com/facebook/watchman
Installed
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1 (26 files, 14.3MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-12-26 at 21:08:26
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/w/watchman.rb
License: MIT
==> Dependencies
Build: cmake ✘, cpptoml ✘, googletest ✘, mvfst ✘, pkgconf ✘, python-setuptools ✘, rust ✘
Required: edencommon ✔, fb303 ✔, fbthrift ✔, fmt ✔, folly ✔, gflags ✔, glog ✔, libevent ✔, openssl@3 ✔, pcre2 ✔, [email protected] ✔, boos
t ✔, libunwind ✔
==> Options
--HEAD
        Install HEAD version
 ⏎

❯ brew list watchman
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-diag
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-make
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-replicate-subscription
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchman-wait
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/bin/watchmanctl
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/lib/python3.13/ (15 files)
/home/linuxbrew/.linuxbrew/Cellar/watchman/2024.12.02.00_1/sbom.spdx.json

❯ which watchman
/home/linuxbrew/.linuxbrew/bin/watchman

I gave brew install watchman --HEAD a shot and got this (and I'm not complaining or reporting it as an issue!) :)

READ THIS: https://docs.brew.sh/Troubleshooting

These open issues may also help:
Watchman socket path too long https://github.com/Homebrew/homebrew-core/issues/181152
watchman's formula was built from an unstable upstream --HEAD.
This build failure is expected behaviour.
Do not create issues about this on Homebrew's GitHub repositories.
Any opened issues will be immediately closed without response.
Do not ask for help from Homebrew or its maintainers on social media.
You may ask for help in Homebrew's discussions but are unlikely to receive a response.
Try to figure out the problem yourself and submit a fix as a pull request.
We will review it but may or may not accept it.
@maschwenk
Copy link

watchman --foreground --logfile=/dev/stdout

same issue on mac with install from homebrew

2024-12-27T12:44:50,961: [listener] Watchman 2024.12.02.00 <no build info set> starting up on MSchwenk by command /bin/zsh-il
2024-12-27T12:44:50,962: [listener] file limit is 9223372036854775807 kern.maxfilesperproc=16777216
2024-12-27T12:44:50,962: [sanitychecks] starting sanityCheckThread
Terminating due to signal 11 Segmentation fault generated by pid=0 uid=0 invalid permissions for mapped object (0)
0   watchman                            0x00000001049c11dc _ZL13crash_handleriP9__siginfoPv + 556
1   libsystem_platform.dylib            0x000000018ab28184 _sigtramp + 56
2   libfmt.11.1.0.dylib                 0x00000001054ba790 _ZN3fmt3v116detail13copy_noinlineIcPKcNS0_14basic_appenderIcEEEET1_T0_S8_S7_ + 32
3   libfmt.11.1.0.dylib                 0x00000001054ba790 _ZN3fmt3v116detail13copy_noinlineIcPKcNS0_14basic_appenderIcEEEET1_T0_S8_S7_ + 32
4   libfmt.11.1.0.dylib                 0x00000001054b6fac _ZN3fmt3v116detail19parse_format_stringIcNS1_14format_handlerIcEEEEvNS0_17basic_string_viewIT_EEOT0_ + 256
5   libfmt.11.1.0.dylib                 0x00000001054b8280 _ZN3fmt3v116detail10vformat_toIcEEvRNS1_6bufferIT_EENS0_17basic_string_viewIS4_EENS1_12vformat_argsIS4_E4typeENS1_10locale_refE + 48
6   watchman                            0x0000000104977998 _ZNK3fmt3v119formatterI8w_stringcvE6formatINS0_7contextEEEDaRKS2_RT_ + 80
7   watchman                            0x000000010497793c _ZN3fmt3v116detail5valueINS0_7contextEE17format_custom_argI8w_stringNS0_9formatterIS6_cvEEEEvPvRNS0_26basic_format_parse_contextIcEERS3_ + 28
8   libfmt.11.1.0.dylib                 0x00000001054b7250 _ZN3fmt3v116detail19parse_format_stringIcNS1_14format_handlerIcEEEEvNS0_17basic_string_viewIT_EEOT0_ + 932
9   libfmt.11.1.0.dylib                 0x00000001054b6c64 _ZN3fmt3v116detail10vformat_toERNS1_6bufferIcEENS0_17basic_string_viewIcEENS0_17basic_format_argsINS0_7contextEEENS1_10locale_refE + 76
10  libfmt.11.1.0.dylib                 0x00000001054b68bc _ZN3fmt3v117vformatENS0_17basic_string_viewIcEENS0_17basic_format_argsINS0_7contextEEE + 104
11  watchman                            0x0000000104a1f424 _ZN8watchman3Log4logfIJRPKcRK8w_stringEEEvNS_8LogLevelEN3fmt3v1117basic_string_viewIcEEDpOT_ + 136
12  watchman                            0x0000000104a1dc54 _Z12root_resolvePKcbPb + 436
13  watchman                            0x0000000104a1ef54 _Z14w_root_resolvePKcb + 36
14  watchman                            0x00000001049d2e54 _Z17resolveRootByNamePN8watchman6ClientEPKcb + 88
15  watchman                            0x0000000104a0fcf8 _ZL17cmd_watch_projectPN8watchman6ClientERK8json_ref + 608
16  watchman                            0x00000001049713d0 _ZN8watchman6Client15dispatchCommandERKNS_7CommandENS_12CommandFlagsE + 796
17  watchman                            0x0000000104972f4c _ZN8watchman10UserClient12clientThreadEv + 496
18  watchman                            0x0000000104976924 _ZNSt3__114__thread_proxyB8ne180100INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN8watchman10UserClient6createENS2_INS7_6StreamENS4_IS9_EEEEE3$_0EEEEEPvSE_ + 56
19  libsystem_pthread.dylib             0x000000018aaf32e4 _pthread_start + 136
20  libsystem_pthread.dylib             0x000000018aaee0fc thread_start + 8
[1]    22922 segmentation fault  /opt/homebrew/bin/watchman --foreground --logfile=/dev/stdout

@maschwenk
Copy link

maschwenk commented Dec 27, 2024

I think will be fixed by this

@WhiteDove666
Copy link

❯ eza --git --icons auto --long /tmp/swaits-state/
.rw------- 2.0k swaits 26 Dec 21:12  log
.rw-r--r-- 6 swaits 26 Dec 21:09  pid
srw------- - swaits 26 Dec 21:09  sock

@OllieJennings
Copy link

OllieJennings commented Jan 5, 2025

@swaits I'm now seeing this issue after upgrading today.

Annoyingly can't downgrade and so now I'm struggling to run relay.

Tagging a few people who might be able to help: @MichaelCuevas, @jdelliot, @ahornby ?

@maschwenk
Copy link

@OllieJennings if you are using homebrew it should be fixed if you uninstall/reinstall. the actual issue was in fmt not in watchman. See above

@OllieJennings
Copy link

@maschwenk so l tried doing a complete uninstall of watchman, but still not working for me.

Errors l get when running relay-compiler is:

 > relay-compiler --watch
│ 
│ watchman client task failed: Received a unilateral PDU from the server
│ [INFO] Querying files to compile...
│ [ERROR] Watchman error: Lost connection to watchman
│ [ERROR] Compilation failed.
│ [ERROR] Unable to run relay compiler. Error details: 
│ Watchman { source: ConnectionLost(ClientTaskExited) }

And if l look at the debug log-level error output l can see:

{
    "log": "2025-01-06T19:16:24,295: [client=30:stm=140301216799760:pid=69030] root /home/*****/Code/*****/******-platform using watcher mechanism inotify (auto was requested)\n",
    "unilateral": true,
    "level": "error"
}
{
    "log": "2025-01-06T19:16:26,120: [io 140300867499168 /home/*****/Code/*****/******-platform] PERF: {\"user_time\": 0.77868400000000004, \"description\": \"full-crawl\", \"system_time\": 1.0452349999999999, \"start_time\": 1736190984.295996, \"elapsed_time\": 1.824343, \"pid\": 61339, \"meta\": {\"root\": {\"watcher\": \"inotify\", \"path\": \"/home/*****/Code/*****/******-platform\", \"recrawl_count\": 0, \"case_sensitive\": true}}, \"version\": \"2024.12.02.00\"}\n",
    "unilateral": true,
    "level": "error"
}
{
    "log": "2025-01-06T19:16:26,145: [io 140300867499168 /home/*****/Code/*****/******-platform] Exception:  cancel root\n",
    "unilateral": true,
    "level": "error"
}
{
    "log": "2025-01-06T19:16:26,171: [client=30:stm=140301216799760:pid=69030] send_error_response: [\"watch\", \"/home/*****/Code/*****/******-platform\"], failed: std::domain_error: \n",
    "unilateral": true,
    "level": "error"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants