Varnish/Nginx cached SSL Certificate mystery

Question

I have Varnish load balancing three front end Rails servers with Nginx acting as a reverse proxy for FastCGI workers. Yesterday, our certificate expired, and I got a new certificate from GoDaddy, and installed it. When accessing static resources directly, I see the updated certificate, but when accessing them from a "virtual subdomain" I'm seeing the old certificate. My nginx config only cites my new chained certificate, so I'm wondering how the old certificate is being displayed. I've even removed it from the directory.

example: https://www212.doostang.com/javascripts/base_packaged.js?1331831461 (no certificate problem with SSL) https://asset5.doostang.com/javascripts/base_packaged.js?1331831461 (the old certificate is being used!) (maps to www212.doostang.com)

• I've reloaded and even stopped-and-restarted nginx, tested nginx to make sure that it's reading from the right config, and restarted varnish with a new cache file.

• When I curl the file at asset5.doostang.com I get a certificate error:

  curl: (60) SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed More details here: http://curl.haxx.se/docs/sslcerts.html

  curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

• When I add the -k option, I get the file requested, and I can see it in my nginx access log. I don't get an nginx error when I don't provide the -k; nginx is silent about the certificate error.

10.99.110.27 - - [20/Apr/2012:18:02:52 -0700] "GET /javascripts/base_packaged.js?1331831461 HTTP/1.0" 200 5740 "-" "curl/7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18"

I've put what I think is the relevant part of the nginx config, below:

server {
  # port to listen on. Can also be set to an IP:PORT
  listen 443;
  server_name www.doostang.com, *.doostang.com;

  passenger_enabled on;
  rails_env production;

  ssl on;
  ssl_certificate /.../doostang_combined.crt;
  ssl_certificate_key /.../doostang.com.key;
  ssl_protocols SSLv3;

  # doc root
  root /.../public/files;

  if ($host = 'doostang.com' ) {
        rewrite  ^/(.*)$  https://www.doostang.com/$1  permanent;
   }
}


# Catchall redirect
server {
  # port to listen on. Can also be set to an IP:PORT
  listen 443;

  ssl on;
  ssl_certificate /.../doostang_combined.crt;
  ssl_certificate_key /.../doostang.com.key;

  rewrite ^(.*)$ https://www.doostang.com$1;
} 

Answer 1

Ba dum ching. My non-standardized load balancer actually had nginx running for SSL termination. I failed to notice this, but I think I did everything else correctly. Point being, when you take over operations upon acquisition, standardize and document! There are some really odd engineers out there :)