How to implement RSA Encryption in C?

Question

I'm trying to use asymmetric encryption in a C program, to encrypt strings.

I chose to go with RSA, but if there's a less complicated yet secure way, do tell me.

OpenSSL is one library I've looked at and found no documentation on implementing it in C code. (I may just have been unlucky, I've been looking for many days)

No luck on YouTube/Google either...

Please point me to a detailed source of information on how how to do this...

I understand both C and the fundamental concepts of RSA pretty well, but I have no idea how to:

1. Generate the large primes needed to generate the keys.
2. Fix e = 65537 .
3. Generate the public / private keys in an alphanumeric format (they are actually numbers, arent they ?) .
4. Seamlessly combine (e,n) in the public key and (d,n) in the private key in the way tools like OpenSSL seem to do (in an alphanumeric string) .

Answer 1

Here's an example of what you're looking to do. First a utility function to print OpenSSL error messages:

void log_ssl_err(const char *mes)
{
    unsigned long err, found;
    char errstr[1000];

    found = 0;
    while ((err = ERR_get_error())) {
        ERR_error_string(err, errstr);
        printf("%s: %s", mes, errstr);
        found = 1;
    }
    if (!found) {
        printf("%s", mes);
    }
}

Generating a key with a given exponent:

RSA *rsa;
BIGNUM *e;
uint32_t exponent_bin, exponent_num;

exponent_num = 65537;
exponent_bin = htonl(exponent);
e = BN_bin2bn((const unsigned char *)&exponent_bin, 4, NULL);
if (e == NULL) {
    log_ssl_err("BN_bin2bn failed for e");
    exit(1);
}

if ((rsa = RSA_new()) == NULL) {
    log_ssl_err("RSA_new failed");
    BN_free(e);
    exit(1);
}
if (!RSA_generate_key_ex(rsa, 2048, e, NULL)) {
    log_ssl_err("couldn't generate rsa key");
    BN_free(e);
    exit(1);
}

Encrypting and decrypting:

unsigned char plaintext[] = "this is the plaintext";
unsigned char *ciphertext, *decrypted;
int cipher_len, decrypted_len;

ciphertext = malloc(RSA_size(rsa));
if ((cipher_len = RSA_public_encrypt(strlen(plaintext), plaintext, ciphertext, 
                                    rsa, RSA_PKCS1_OAEP_PADDING)) == -1) {
    log_ssl_err("RSA_public_encrypt failed");
    exit(1);
}

decrypted = malloc(RSA_size(rsa));
if ((decrypted_len = RSA_private_decrypt(cipher_len, ciphertext, decrypted, 
                                         rsa, RSA_PKCS1_OAEP_PADDING)) == -1) {
    log_ssl_err("RSA_private_decrypt failed");
    return 0;
}

The documentation for OpenSSL can be difficult to navigate, but the information you need can be found in the man pages. If you run man 3 rsa you'll see a list of all the RSA related functions. From there you can look at the man page for each function.