Why is Firefox not trusting my self-signed certificate?

Question

I have a local domain (markfisher.local) for which I have made a self-signed certificate that I have added to the Certificates in Keychain Access (I am using Mac). This results in the certificate being trusted in Chrome and Safari. If I edit the certificates Trust in KeyChain Access to "Never Trust" then Chrome and Safari no longer accept it, then if I change it back to "Always Trust" then I can access markfisher.local OK again.

But when accessing the site in Firefox I get the following:

markfisher.local uses an invalid security certificate.
The certificate does not come from a trusted source.
Error code: MOZILLA_PKIX_ERROR_CA_CERT_USED_AS_END_ENTITY

I have set security.enterprise_roots.enabled to true as suggested in this answer. Also, going to Preferences > Privacy & Security > View Certificates, I can see my certificate is in the Authorities tab and I have checked the "This certificate can identify websites" in the "Edit Trust" dialog. This was not checked originally

However Firefox refuses to accept the certificate. How can I fix this?

Answer 1

This answer is expanding on Patrick Mevzek's answer.

CAVEAT
These steps were tested on MacOS. They may work on Linux out of the box, but not in Windows 10 or 11.

NOTE: The commands below were all used in Bash, so they may need adjustments in other shells. ^{Thank you, TAbdiukov!}

In short, to use self-signed SSL certificates to work in Firefox (e.g., for testing HTTPS), you'll have to go the long route of creating and using your own public key infrastructure (PKI):

Step 1. Establish your private certificate authority (CA)

By creating a private key and self-signed certificate:

openssl req -x509 -nodes \
  -newkey RSA:2048       \
  -keyout root-ca.key    \
  -days 365              \
  -out root-ca.crt       \
  -subj '/C=US/ST=Denial/L=Earth/O=Atest/CN=root_CA_for_firefox'

Generated files:

• root-ca.key: The private key of your CA
• root-ca.crt: The x509 certificate of your CA (signed with root-ca.key).

Step 2. Create a private key and a certificate signing request (CSR) for your server

openssl req -nodes   \
  -newkey rsa:2048   \
  -keyout server.key \
  -out server.csr    \
  -subj '/C=US/ST=Denial/L=Earth/O=Dis/CN=anything_but_whitespace'

Generated files:

• server.key: Private key
• server.csr: Certificate signing request; does not need any domain names to be specified.

Step 3. Generate a certificate for your server

NOTE
Make sure to use the right domain name at the subjectAltName = DNS:<domain> part below!

openssl x509 -req    \
  -CA root-ca.crt    \
  -CAkey root-ca.key \
  -in server.csr     \
  -out server.crt    \
  -days 365          \
  -CAcreateserial    \
  -extfile <(printf "subjectAltName = DNS:localhost\nauthorityKeyIdentifier = keyid,issuer\nbasicConstraints = CA:FALSE\nkeyUsage = digitalSignature, keyEncipherment\nextendedKeyUsage=serverAuth")

^{The <(..) part is called process substitution in Bash. If you use another shell without this feature, then simply put the text inside the double quotes in a file, replace "\n" with newlines, and specify the file instead for -extfile.}

Generated files:

• root-ca.srl: I usually ignore it
• server.crt: Certificate to be used on your server

Step 4. Add root-ca.crt to Firefox's trust store

By following these steps.

NOTE
As @sstchur pointed out in the comments, this step may not be necessary if the options

• security.certerrors.mitm.auto_enable_enterprise_roots and
• security.enterprise_roots.enabled

are set to true. (See the Mozilla support article How to disable the Enterprise Roots preference for details.)
(subnote: I didn't have the time to test these yet in practice!)

Step 5. Configure your server

You'll need server.key and server.crt. The specific steps depend on the technology.

---

Why are these steps needed for Firefox?

While the following one-liner below works both in Chrome and Safari,

(That is, where one would use the created cert both as a root CA cert (to install in a system's trust store) and a cert for a web server (e.g., in NGINX with ssl_certificate).)

openssl req -x509 -new -nodes                                            \
  -newkey RSA:2048                                                       \
  -days 365                                                              \
  -subj '/C=US/ST=Denial/L=Springfield/O=Dis/CN=anything_but_whitespace' \
  -addext 'subjectAltName = DNS:doma.in,DNS:anoth.er'                    \
  -addext 'authorityKeyIdentifier = keyid,issuer'                        \
  -addext 'basicConstraints = CA:FALSE'                                  \
  -addext 'keyUsage = digitalSignature, keyEncipherment'                 \
  -addext 'extendedKeyUsage=serverAuth'                                  \
  -out self-signed.crt                                                   \
  -keyout private.key

it won't in Firefox, because:

1. You can't install self-signed.crt in Firefox's trust store, because it explicitly states that the certificate is not a certificate authority (CA:FALSE).

2. What if you change CA:FALSE to CA:TRUE in the command above?

   Chrome and Safari will still accept it, you can install self-signed.crt into Firefox's trust store, but it won't work from the server's side because of the quoted reasons at the top of this answer.

3. What if you simply delete the -addext 'basicConstraints = CA:FALSE option?

   Firefox will complain when trying to import self-signed.crt in its trust store:

   

   (Probably because of the x509 v3 extensions added with -addext.)

4. What if you omit all the x509 v3 extensions (i.e., -addext options)?

   openssl req -x509 -new -nodes                                            \
     -newkey RSA:2048                                                       \
     -days 365                                                              \
     -subj '/C=US/ST=Denial/L=Springfield/O=Dis/CN=anything_but_whitespace' \
     -out self-signed.crt                                                   \
     -keyout private.key
   

   Now you can import self-signed.crt into Firefox's trust store, but you will get SSL_ERROR_BAD_CERT_DOMAIN (and you'll get a similar error in Chrome and Safari as they need the subjectAltName extension to be present on the certificate from the server's side).

5. What about changing the command to contain CN=<your domain>?

   You'll probably still get the same error as "Firefox from 101.0 onward no longer uses certificate CN (Common Name) for matching domain name to certificate and have migrated to only using SAN (Subject Alternate Name)".

   Which means you have to add at least the subjectAltName x509 v3 extension, but then you are back at items 1, 2, or 3 above.