Install4J v8 PKCS#11 Code Signing with Google Cloud KMS (libkmsp11.so) - Certificate Not Found

Question

As a long-time user of Install4J, I appreciate its robust, direct implementation of code signing since version 5.1, which leverages open standards and eliminates the need for external tools like signtool.exe (as noted in ej-technologies blog and Stack Overflow discussions). With the introduction of PKCS#11 support for Windows code signing in Install4J v8 (as highlighted in the What's New in Install4J 8.0 page), I'm now attempting to integrate our build process with Google Cloud KMS for secure private key management.

I'm trying to set up code signing for my Install4J v8 project on Linux, using a Sectigo code signing certificate with a private key stored in Google Cloud KMS (backed by Cloud HSM). (We also have a license for V10, but I don't see anything in the v9 or v10 release notes that indicate improved code signing.) I'm utilizing v1.7 of the libkmsp11.so library to expose the Cloud KMS key via PKCS#11.

I've followed the setup for libkmsp11.so and have confirmed that the library correctly identifies and accesses my private key in Cloud KMS.

1. Google Cloud HSM Setup (Key Management without Certificate)

Our Sectigo code signing key has been uploaded into Google Cloud KMS. I can see the keyring and the key in the GCP console. The private key itself resides in Cloud KMS; the X.509 certificate issued by Sectigo is a separate file.

2. Cloud KMS Library for PKCS #11 Configuration & Verification

I have libkmsp11.so (v1.7) installed locally (in /opt/gcp-kms-pkcs11/libkmsp11.so). My environment variables are set as follows:

• KMS_PKCS11_CONFIG=/path/to/pkcs11.yaml

• GOOGLE_APPLICATION_CREDENTIALS=/path/to/json-key.json

I've successfully verified that libkmsp11.so can access the key using pkcs11-tool:

pkcs11-tool --module /opt/gcp-kms-pkcs11/libkmsp11.so -O

Output confirms the public and private key objects are visible:

Using slot 0 with a present token (0x0)
Public Key Object; RSA 4096 bits
label:       code-sign-cert-key
ID:          <redacted>
Usage:       verify
Access:      local
uri:         pkcs11:model=Cloud%20KMS%20Token;manufacturer=Google;serial=0000000000000000;token=;id=<redacted>;object=code-sign-cert-key;type=public
Private Key Object; RSA
label:       code-sign-cert-key
ID:          <redacted>
Usage:       sign
Access:      sensitive, always sensitive, never extractable, local
Allowed mechanisms: RSA-PKCS,SHA256-RSA-PKCS
uri:         pkcs11:model=Cloud%20KMS%20Token;manufacturer=Google;serial=0000000000000000;token=;id=<redacted>;object=code-sign-cert-key;type=private

I also have the public certificate file (codesigning_certificate.pem) from Sectigo, which I extracted from a PKCS#7 bundle using OpenSSL:

openssl pkcs7 -print_certs -in PKCS7.p7b -out codesigning_certificate.pem

This .pem file contains the full certificate chain, including the leaf certificate.

3. Install4J v8 Code Signing Configuration (from .install4j file)

My current configuration in the .install4j project file is as follows: (The windowsCertificatePath attribute was added as insisted by AI, even though I'm skeptical whether Install4J v8 or v10 supports it)

<codeSigning windowsEnabled="true" windowsKeySource="pkcs11"
             windowsPkcs11Library="${compiler:sys.ext.pkcs11LibPath}"
             windowsCertificatePath="${compiler:sys.ext.certPath}">
    <windowsPkcs11Identifier issuer="(issuer of leaf cert)" serial="(serial of leaf cert)" subject="(subject of leaf cert)" />
</codeSigning>

(I've defined compiler:sys.ext.certPath in Install4J's Compiler Variables to point to /path/to/codesigning_certificate.pem).

4. Persistent Error Message

Despite providing the windowsCertificatePath, the build fails with the following error:

Build failed.

Cause: com.install4j.d.j
       did not find (serial of leaf cert) issued by (issuer of leaf cert) from /opt/gcp-kms-pkcs11/libkmsp11.so
Stack trace:

com.exe4j.a.d: com.install4j.d.j: did not find (serial of leaf cert) issued by (issuer of leaf cert) from /opt/gcp-kms-pkcs11/libkmsp11.so
    at com.install4j.b.j.a(ejt:846)
    at com.install4j.b.j.a(ejt:338)
    at com.install4j.b.j.b(ejt:118)
    at com.install4j.gui.b.run(ejt:100)
Caused by: com.install4j.d.j: did not find (serial of leaf cert) issued by (issuer of leaf cert) from /opt/gcp-kms-pkcs11/libkmsp11.so
    at com.install4j.d.f.l(ejt:70)
    at java.base/java.util.Optional.orElseThrow(Optional.java:408)
    at com.install4j.d.f.b(ejt:70)
    at com.install4j.d.f.a(ejt:30)
    at com.install4j.d.h.a(ejt:78)
    at com.install4j.b.j.a(ejt:887)
    at com.install4j.b.j.a(ejt:838)
    ... 3 more

Troubleshooting already attempted:

• Removed the <windowsPkcs11Identifier> block. This resulted in Install4J saying "Please choose a certificate first" (in the GUI), indicating it still needs a certificate reference for selection.

• Confirmed codesigning_certificate.pem contains the correct public certificate and chain.

• Enabled verbose debugging in Install4J GUI (output remains as above).

• My understanding is that libkmsp11.so only exposes key objects (public/private) and not certificate objects, so Install4J v8 is failing when it tries to search for the certificate within the PKCS#11 module using the provided identifier.

Questions:

1. Given this scenario, will upgrading to Install4J v11 resolve this issue? The release notes mention "PKCS #11 code signing improvements" and a rewritten implementation, as well as the introduction of selecting certificates by label and auto-selection. This suggests v11 might handle the separation of key (from PKCS#11) and certificate (from file) more gracefully.

2. If upgrading to v11, will I still need to explicitly provide the windowsCertificatePath (with the chained certificates in the .pem file), or will the new PKCS#11 integration be able to automatically fetch or combine this information?

3. Are there any other configuration attributes or approaches in Install4J v8 (or v10) that I might be missing to make this work without an upgrade?

Any insights from the Install4J community would be greatly appreciated.

Update

After reading Sakshi Mali's answer I decided to try the 90-day evaluation of Install4J v11. I specified a directory as the "Chained certificates" path and placed the codesigning_certificate.pem in it.

Select from keystore

When trying "Select from keystore" the dialog doesn't show any certificates. This failed with an error on v8 before.

Without specifying certificate manually

When not specifying the certificate manually, the default/fallback is then:

Code signing certificate with maximum validity

but it still fails:

      Signing executable

Build failed.

Cause: com.install4j.d.j
       did not find a valid code signing certificate in key store
Stack trace:

com.exe4j.a.d: com.install4j.d.j: com.install4j.d.j: did not find a valid code signing certificate in key store
    at com.install4j.b.g.c.a(ejt:220)
    at com.install4j.b.g.c.a(ejt:95)
    at com.install4j.b.g.f.a(ejt:75)
    at com.install4j.b.c.b(ejt:417)
    at com.install4j.b.c.b(ejt:321)
    at com.install4j.b.b.a(ejt:147)
    at com.install4j.b.g.f.a(ejt:102)
    at com.install4j.b.j.a(ejt:592)
    at com.install4j.b.j.c(ejt:227)
    at com.install4j.gui.b.run(ejt:101)
Caused by: com.install4j.d.j: com.install4j.d.j: did not find a valid code signing certificate in key store
    at com.install4j.d.c.f.a(ejt:115)
    at com.install4j.d.c.g.a(ejt:57)
    at com.install4j.b.g.c.a(ejt:218)
    ... 9 more
Caused by: com.install4j.d.j: did not find a valid code signing certificate in key store
    at com.install4j.d.b.d.j(ejt:99)
    at java.base/java.util.Optional.orElseThrow(Optional.java:403)
    at com.install4j.d.b.d.c(ejt:99)
    at com.install4j.d.b.d.a(ejt:80)
    at com.install4j.d.b.d.b(ejt:63)
    at com.install4j.d.b.d.a(ejt:165)
    at com.install4j.d.c.f.a(ejt:74)
    ... 11 more

Specifying certificate manually by label

When specifying the label as it is reported by pkcs11-tool --module /opt/gcp-kms-pkcs11/libkmsp11.so -O, e.g. code-sign-cert-key, it also fails with this output:

      Signing executable

Build failed.

Cause: com.install4j.d.j
       did not find certificate with unique label 'code-sign-cert-key' from /opt/gcp-kms-pkcs11/libkmsp11.so
Stack trace:

com.exe4j.a.d: com.install4j.d.j: com.install4j.d.j: did not find certificate with unique label 'code-sign-cert-key' from /opt/gcp-kms-pkcs11/libkmsp11.so
    at com.install4j.b.g.c.a(ejt:220)
    at com.install4j.b.g.c.a(ejt:95)
    at com.install4j.b.g.f.a(ejt:75)
    at com.install4j.b.c.b(ejt:417)
    at com.install4j.b.c.b(ejt:321)
    at com.install4j.b.b.a(ejt:147)
    at com.install4j.b.g.f.a(ejt:102)
    at com.install4j.b.j.a(ejt:592)
    at com.install4j.b.j.c(ejt:227)
    at com.install4j.gui.b.run(ejt:101)
Caused by: com.install4j.d.j: com.install4j.d.j: did not find certificate with unique label 'code-sign-cert-key' from /opt/gcp-kms-pkcs11/libkmsp11.so
    at com.install4j.d.c.f.a(ejt:115)
    at com.install4j.d.c.g.a(ejt:57)
    at com.install4j.b.g.c.a(ejt:218)
    ... 9 more
Caused by: com.install4j.d.j: did not find certificate with unique label 'code-sign-cert-key' from /opt/gcp-kms-pkcs11/libkmsp11.so
    at com.install4j.d.b.d.l(ejt:73)
    at java.base/java.util.Optional.orElseThrow(Optional.java:403)
    at com.install4j.d.b.d.b(ejt:73)
    at com.install4j.d.b.d.a(ejt:165)
    at com.install4j.d.c.f.a(ejt:74)
    ... 11 more

Answer 1

This will be supported in install4j 12 with a separate certificate file selection for PKCS#11 in the IDE .

Please contact [email protected], we will backport this via compiler variables to install4j 11, but you would need to try a beta build.

---

Update:

According to

https://github.com/GoogleCloudPlatform/kms-integrations/blob/master/kmsp11/docs/user_guide.md#configuration

you can use "certs" to configure certificates that will be presented together with the key. Setting this option to your certificate should work. Intermediate certificates should be placed into a a directory and configured with the "Intermediate certificates" setting in install4j.