Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

504 Gateway time-out errors #188

Open
rachel3834 opened this issue Aug 22, 2024 · 2 comments
Open

504 Gateway time-out errors #188

rachel3834 opened this issue Aug 22, 2024 · 2 comments
Assignees
@phycodurus @rachel3834 @jchate6 @jashan-lco
Labels
bug Something isn't working

Comments

@rachel3834
Copy link
Contributor

Describe the bug
I'm updating MOP to the latest (v2.19.5) release of the TOM toolkit and deploying it in the AWS Cloud. The following issues affect mop-dev most acutely since it is allowed fewer resources that mop-prod, but the same issues are also occuring with mop-prod to a degree where using the system is hampered.

The updated system works fine in my localhost testing version, and target detail pages load normally without errors.
When I push this to the dev and prod servers, it appears to build and deploy cleanly.

Navigating to mop-dev in the browser, the whole site shows serious performance issues, with even the home page taking minutes to load, but often produces a 504 gateway error. Attempting to load a target detail page gives the same result.

The stern logs for mop-dev record the following:

mop-dev-7ffc48f95f-cx7ll mop 127.0.0.1 - - [22/Aug/2024:18:09:36 +0000] "GET /targets/ HTTP/1.1" 200 27856 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
mop-dev-7ffc48f95f-cx7ll nginx 10.100.28.102 - - [22/Aug/2024:18:09:36 +0000] "GET /targets/ HTTP/1.1" 200 27856 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0" "172.16.4.138"
mop-dev-7ffc48f95f-cx7ll mop Not Found: /targets/P/DSS2/color/properties
mop-dev-7ffc48f95f-cx7ll mop Not Found: /targets/P/DSS2/color/properties
mop-dev-7ffc48f95f-cx7ll nginx 10.100.28.102 - - [22/Aug/2024:18:09:38 +0000] "GET /targets/P/DSS2/color/properties HTTP/1.1" 404 10832 "http://mop-dev.lco.gtn/targets/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0" "172.16.4.138"
mop-dev-7ffc48f95f-cx7ll mop 127.0.0.1 - - [22/Aug/2024:18:09:38 +0000] "GET /targets/P/DSS2/color/properties HTTP/1.1" 404 10832 "http://mop-dev.lco.gtn/targets/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:38 +0000] [1] [ERROR] Worker (pid:1586) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:38 +0000] [1589] [INFO] Booting worker with pid: 1589
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:43 +0000] [1] [ERROR] Worker (pid:1589) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:43 +0000] [1592] [INFO] Booting worker with pid: 1592
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:48 +0000] [1] [ERROR] Worker (pid:1592) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:48 +0000] [1595] [INFO] Booting worker with pid: 1595
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:53 +0000] [1] [ERROR] Worker (pid:1595) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:53 +0000] [1598] [INFO] Booting worker with pid: 1598
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:58 +0000] [1] [ERROR] Worker (pid:1598) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:09:58 +0000] [1601] [INFO] Booting worker with pid: 1601
^[[B^[[Bmop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:03 +0000] [1] [ERROR] Worker (pid:1601) was sent SIGKILL! Perhaps out of memory?

Clicking on the link for the target detail page for Gaia24cgl, stern has the following output:

mop-dev-7ffc48f95f-cx7ll mop STARTING TargetDetail page load: 2024-08-22 18:10:20.822130
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 7, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop TARGETDETAIL: chk 1, time taken 0:00:00.012881
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 9, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop TARGETDETAIL: chk 2, time taken 0:00:00.002419
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 9, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop TARGETDETAIL: chk 3, time taken 0:00:00.001606
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 9, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop TARGETDETAIL: get took 0:00:00.016906
mop-dev-7ffc48f95f-cx7ll mop STARTED GET_CONTEXT 2024-08-22 18:10:20.852837
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 11, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop queryTools: Retrieved associated data for 1 Targets
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 15, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop queryTools: Time taken: 0:00:00.000474
mop-dev-7ffc48f95f-cx7ll mop queryTools: collating data on microlensing event set
mop-dev-7ffc48f95f-cx7ll mop queryTools: collated data for target Gaia24cgl, 0 out of 1
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 19, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop queryTools: Collated data for 1 targets in 0:00:00.076568
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 19, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop GET_CONTEXT took 0:00:00.095039
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 19, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop FINISHED GET_CONTEXT 2024-08-22 18:10:20.958146
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 19, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 25, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 25, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop MULENS TARGET DATA time taken: 0:00:00.003443
mop-dev-7ffc48f95f-cx7ll mop CLASS FORM started at 2024-08-22 18:10:20.990928
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 27, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop Target post save hook: Gaia24cgl created: False
mop-dev-7ffc48f95f-cx7ll mop CLASS FORM took 0:00:00.018837
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 33, memory: 661.63MiB
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:22 +0000] [1] [ERROR] Worker (pid:1613) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:22 +0000] [1616] [INFO] Booting worker with pid: 1616
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:27 +0000] [1] [ERROR] Worker (pid:1616) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:27 +0000] [1619] [INFO] Booting worker with pid: 1619
mop-dev-7ffc48f95f-cx7ll mop MOP PHOTOMETRY: Started at 2024-08-22 18:10:29.541332, got 113 datasets for target Gaia24cgl
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 43, memory: 659.66MiB
mop-dev-7ffc48f95f-cx7ll mop MOP PHOTOMETRY took 0:00:00.009959
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 43, memory: 659.66MiB
mop-dev-7ffc48f95f-cx7ll mop PYLIMA MODEL EXTRACT started 2024-08-22 18:10:29.663708
mop-dev-7ffc48f95f-cx7ll mop PyLIMA model load took 0:00:00.001585
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 47, memory: 661.71MiB
mop-dev-7ffc48f95f-cx7ll mop MOP INTERFEROMETRY started at: 2024-08-22 18:10:29.671338
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 49, memory: 661.71MiB
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 49, memory: 661.71MiB
mop-dev-7ffc48f95f-cx7ll mop MOP INTERFEROMETRY took 0:00:00.000899
mop-dev-7ffc48f95f-cx7ll mop GAIA NEIGHBOURS started 2024-08-22 18:10:29.673136
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 49, memory: 661.71MiB
mop-dev-7ffc48f95f-cx7ll mop GAIA NEIGHBOURS took 0:00:00.000540
mop-dev-7ffc48f95f-cx7ll mop CHECKPOINT: N DB connections: 49, memory: 661.71MiB
mop-dev-7ffc48f95f-cx7ll mop END Targetpage get2024-08-22 18:10:29.673676
mop-dev-7ffc48f95f-cx7ll nginx 2024/08/22 18:10:30 [warn] 8#8: *56 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/1/00/0000000001 while reading upstream, client: 10.100.28.102, server: _, request: "GET /targets/6869/ HTTP/1.1", upstream: "http://127.0.0.1:8080/targets/6869/", host: "mop-dev.lco.gtn", referrer: "http://mop-dev.lco.gtn/targets/"
mop-dev-7ffc48f95f-cx7ll mop 127.0.0.1 - - [22/Aug/2024:18:10:30 +0000] "GET /targets/6869/ HTTP/1.1" 200 11078143 "http://mop-dev.lco.gtn/targets/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
mop-dev-7ffc48f95f-cx7ll nginx 10.100.28.102 - - [22/Aug/2024:18:10:31 +0000] "GET /targets/6869/ HTTP/1.1" 200 11078143 "http://mop-dev.lco.gtn/targets/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0" "172.16.4.138"
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:32 +0000] [1] [ERROR] Worker (pid:1619) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:32 +0000] [1622] [INFO] Booting worker with pid: 1622
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:36 +0000] [1] [ERROR] Worker (pid:1622) was sent SIGKILL! Perhaps out of memory?
mop-dev-7ffc48f95f-cx7ll mop [2024-08-22 18:10:37 +0000] [1625] [INFO] Booting worker with pid: 1625

On this occasion, the page did load - slowly. But it appears that the gunicorn workers (?) are very unhappy.
@rachel3834 rachel3834 added the bug Something isn't working label Aug 22, 2024
@jchate6
Copy link
Contributor

jchate6 commented Aug 22, 2024

Can you give me a pip freeze from your deployment environment?

@jchate6
Copy link
Contributor

jchate6 commented Aug 22, 2024

And, could you try re-deploying with 2.19.4 and see if the problem goes away?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phycodurus phycodurus

@rachel3834 rachel3834

@jchate6 jchate6

@jashan-lco jashan-lco

Labels
bug Something isn't working
Projects
MOP Development
Status: No status
Milestone
No milestone
Development

No branches or pull requests
4 participants
@phycodurus @rachel3834 @jchate6 @jashan-lco