Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Platform] Getting "too many clients already" issue while connecting to ysqlsh #8416

Open
VijiYB opened this issue May 12, 2021 · 5 comments · May be fixed by yugabyte/yb-sample-apps#65
Open
Assignees
Labels
area/platform Yugabyte Platform QA QA filed bugs
Milestone

Comments

@VijiYB
Copy link

VijiYB commented May 12, 2021

Sometimes getting "too many clients already" health alert while connecting to ysqlsh for some nodes. Was running postgresqlkeyvalue sample apps on that universe.

Error executing command ['ssh', '[email protected]', '-p',
'54422', '-o', 'StrictHostKeyChecking no', '-o', 'ConnectTimeout=10',
'-o', 'UserKnownHostsFile /dev/null', '-o', 'LogLevel ERROR', '-i',
u'/opt/yugabyte/yugaware/data/keys/109e95b5-bf08-4a8f-a7fb-2d2866865e15/yb-gcp-config-key.pem',
'set -o pipefail; /home/yugabyte/tserver/bin/ysqlsh -h 10.150.1.107 -p
5433 -U yugabyte -c "\conninfo"']: ysqlsh: FATAL: sorry, too many
clients already

@VijiYB VijiYB added the area/platform Yugabyte Platform label May 12, 2021
@VijiYB
Copy link
Author

VijiYB commented May 12, 2021

Observed this problem in 2.7.1 (dev portal)
@iSignal Please add your comments as you were debugging on this issue.

@iSignal
Copy link
Contributor

iSignal commented May 12, 2021

Sample apps was running from 10.9.6.228 with

[centos@ip-10-9-6-228 ~]$ ps -ef | egrep sample
centos    38455  17588  0 18:34 pts/13   00:00:00 grep -E --color=auto sample
root      88370  88353  0 00:07 ?        00:11:01 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload SqlInserts --nodes 10.150.1.107:5433,172.151.27.97:5433,172.151.19.105:5433,172.151.17.99:5433
root      88537  88520 99 00:07 ?        1-17:46:05 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload CassandraKeyValue --nodes 10.150.1.107:9042,172.151.27.97:9042,172.151.19.105:9042,172.151.17.99:9042

causing many postgres processes (295) in "idle" status as seen at https://gist.github.com/iSignal/c6cce1239d6f64ca3c17005922d80aa8

Killing the sample apps processes reduced the number of postgres processes to <5.

There were repeated postgres errors of the form in the db postgres logs - https://gist.github.com/iSignal/2003f756551edf5ed7cd38c41f05799e

2021-05-12 12:29:11.518 UTC [3265] ERROR:  duplicate key value violates unique constraint "postgresqlkeyvalue_pkey"
2021-05-12 12:29:11.518 UTC [3265] STATEMENT:  INSERT INTO postgresqlkeyvalue (k, v) VALUES ($1, $2)

/mnt/d0/pg_data/ysql_pg.conf showed postgres max_connections setting at 300

@gurushant-yb : I saw that you made some changes to yb-sample-apps recently, is this something you have run into or have some idea about?

@gurushant-yb
Copy link
Contributor

Sample apps was running from 10.9.6.228 with

[centos@ip-10-9-6-228 ~]$ ps -ef | egrep sample
centos    38455  17588  0 18:34 pts/13   00:00:00 grep -E --color=auto sample
root      88370  88353  0 00:07 ?        00:11:01 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload SqlInserts --nodes 10.150.1.107:5433,172.151.27.97:5433,172.151.19.105:5433,172.151.17.99:5433
root      88537  88520 99 00:07 ?        1-17:46:05 /usr/bin/java -jar /opt/yugabyte/yb-sample-apps.jar --workload CassandraKeyValue --nodes 10.150.1.107:9042,172.151.27.97:9042,172.151.19.105:9042,172.151.17.99:9042

causing many postgres processes (295) in "idle" status as seen at https://gist.github.com/iSignal/c6cce1239d6f64ca3c17005922d80aa8

Killing the sample apps processes reduced the number of postgres processes to <5.

There were repeated postgres errors of the form in the db postgres logs - https://gist.github.com/iSignal/2003f756551edf5ed7cd38c41f05799e

2021-05-12 12:29:11.518 UTC [3265] ERROR:  duplicate key value violates unique constraint "postgresqlkeyvalue_pkey"
2021-05-12 12:29:11.518 UTC [3265] STATEMENT:  INSERT INTO postgresqlkeyvalue (k, v) VALUES ($1, $2)

/mnt/d0/pg_data/ysql_pg.conf showed postgres max_connections setting at 300

@gurushant-yb : I saw that you made some changes to yb-sample-apps recently, is this something you have run into or have some idea about?

@iSignal : I have observed 'unique constraint violation' , this happens when you have a --num_unique_keys < --num_writes

@streddy-yb streddy-yb added this to the 2.7.x milestone May 23, 2021
@streddy-yb
Copy link
Contributor

hi @gurushant-yb - Can you add input validation & throw an error to handle --num_unique_keys < --num_writes case?

@gurushant-yb
Copy link
Contributor

hi @gurushant-yb - Can you add input validation & throw an error to handle --num_unique_keys < --num_writes case?

Hi @streddy-yb @iSignal : Shall I set num_unique_keys = num_writes , by adding following condition instead of throwing an IllegalArgumentException and aborting the execution:
cc: @kkg-yb

 if(AppBase.appConfig.numKeysToWrite != AppBase.appConfig.numUniqueKeysToWrite){
      AppBase.appConfig.numUniqueKeysToWrite = AppBase.appConfig.numKeysToWrite;
    }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform Yugabyte Platform QA QA filed bugs
Projects
None yet
4 participants