Getting Connection reset by peer) while proxying upgraded connection & Connection refused while connecting to upstream

Question

I have been getting these errors in my nginx logs for quite some time now. I dont see any issues with the website performance lately but yes we have issues when we have more then 1000 users connected.

Here are some error logs

2018/03/23 20:14:18 [error] 1356#0: *540195 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 10.253.220.93, server: playtest2.ea.com, request: "GET /socket.io/?email=email&name=name&clientId=35e9c1be-f1fc-4e34-ba68-ce72c6e905c8&version=2.0.0.1704&fullName=test&EIO=3&transport=websocket&t=636573072574620756-17 HTTP/1.1", upstream: "http://127.0.0.1:3000/socket.io/?email=email&name=DICELA-RSTAR&clientId=35e9c1be-f1fc-4e34-ba68-ce72c6e905c8&version=2.0.0.1704&fullName=Dtest&EIO=3&transport=websocket&t=636573072574620756-17", host: "playtest2.xx.com"

2018/03/23 20:14:18 [error] 1356#0: *867401 connect() failed (111: Connection refused) while connecting to upstream, client: 10.xx.xxx.211, server: playtest2.xx.com, request: "GET /bin/PlaytestMateSetup.exe.json HTTP/1.1", upstream: "http://127.0.0.1:3000/bin/PlaytestMateSetup.exe.json", host: "playtest2.xx.com"

Here is my nginx conf, any help will be greatly appreciated.

The server module is listed under another directory.

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
worker_rlimit_nofile 30000;
debug_points abort;
worker_rlimit_core 500M;
events {
    worker_connections 8024; ## Default: 1024;
#    use epoll;
#    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 128;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout  180m;

    error_page 404 /offline_error.html;
        location = /offline_error.html {
                root /opt/ptm/server-src/views;
                internal;
        }

#    open_file_cache max=1000 inactive=1200s;
#    open_file_cache_valid 1800s;
#    open_file_cache_min_uses 5;
#    open_file_cache_errors off;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

}

Answer 1

Sea to this https://www.nginx.com/blog/websocket-nginx/

   map $http_upgrade $connection_upgrade {
     default     upgrade;
     ''          close;
   }
   
   server xxx {
     ...
     location /socket.io/ {
       proxy_pass http://127.0.0.1:3000;
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "Upgrade";
       proxy_set_header Host $host;
       proxy_connect_timeout 15s;
       proxy_send_timeout 3600s; # ws will open for 1 hour
       proxy_read_timeout 3600s; # ws will open for 1 hour
       proxy_buffers 512 256M;
       proxy_buffer_size 256M;
     }
   }