Implementing security protocols in a robust manner
After the recent vulnerabilities in OpenSSL and seeing the fixes and poor coding practices, I thought it would make sense to describe how modern SSL/TLS libraries implement much more robust programming practices in C/C++ (but mostly C).
The following projects are used as examples:
- Google BoringSSL (https://github.com/google/boringssl)
- OpenBSD LibreSSL (https://www.libressl.org/releases.html)
- Amazon s2n-tls (https://github.com/aws/s2n-tls)
Environment
Most libraries are developed to support multiple architectures. This means the following design items must be considered early:
- The smallest CPU register bit size (e.g. 16, 31, 32, 64, etc.). This makes it easier to use fixed width integer types such as uint64_t.
- The CPU endianness options to support. Endianness defines the order or sequence of bytes of a register size integer. This is expressed as big-endian (BE) or little-endian (LE).
- The operating systems to support (some features like threading and sockets are not portable and requires operating system specific implementations). Moreover, character sets handling is also defined by the operating systems (ASCII, Unicode, EBCDIC).
- The compilers to support (e.g. GCC, ICC, MSVS) and which language level (K&R, C89, C99). The compilers may also support intrinsic functions, implement va_args differently, and may also implementing optimizations that remove certain function which are assessed to be superfluous (e.g. removing calls to memset before memory is being freed; such as sensitive key material).
Error handling
Dealing with errors is clearly an important topic. Typically, functions return an integer type (0 ≥ is good, 0 < indicates an error). Checking for errors are often implemented using macros for consistency. The following examples is from s2n.
This function validates input and return values using macros for consistency:
int s2n_stuffer_init_written(struct s2n_stuffer *stuffer, struct s2n_blob *in)
{
POSIX_ENSURE_REF(in);
POSIX_GUARD(s2n_stuffer_init(stuffer, in));
POSIX_GUARD(s2n_stuffer_skip_write(stuffer, in->size));
return S2N_SUCCESS;
}
Callers again check for errors and a number of other defensive error conditions:
S2N_RESULT s2n_client_hello_parse_raw(struct s2n_client_hello *client_hello,
uint8_t client_protocol_version[S2N_TLS_PROTOCOL_VERSION_LEN],
uint8_t client_random[S2N_TLS_RANDOM_DATA_LEN])
{
RESULT_ENSURE_REF(client_hello);
struct s2n_stuffer in_stuffer = { 0 };
RESULT_GUARD_POSIX(s2n_stuffer_init_written(&in_stuffer, &client_hello->raw_message));
struct s2n_stuffer *in = &in_stuffer;
/* legacy_version */
RESULT_GUARD_POSIX(s2n_stuffer_read_bytes(in, client_protocol_version, S2N_TLS_PROTOCOL_VERSION_LEN));
/* random */
RESULT_GUARD_POSIX(s2n_stuffer_erase_and_read_bytes(in, client_random, S2N_TLS_RANDOM_DATA_LEN));
/* legacy_session_id */
uint8_t session_id_len = 0;
RESULT_GUARD_POSIX(s2n_stuffer_read_uint8(in, &session_id_len));
RESULT_ENSURE(session_id_len <= S2N_TLS_SESSION_ID_MAX_LEN, S2N_ERR_BAD_MESSAGE);
uint8_t *session_id = s2n_stuffer_raw_read(in, session_id_len);
RESULT_ENSURE(session_id != NULL, S2N_ERR_BAD_MESSAGE);
RESULT_GUARD_POSIX(s2n_blob_init(&client_hello->session_id, session_id, session_id_len));
/* cipher suites */
uint16_t cipher_suites_length = 0;
RESULT_GUARD_POSIX(s2n_stuffer_read_uint16(in, &cipher_suites_length));
RESULT_ENSURE(cipher_suites_length > 0, S2N_ERR_BAD_MESSAGE);
RESULT_ENSURE(cipher_suites_length % S2N_TLS_CIPHER_SUITE_LEN == 0, S2N_ERR_BAD_MESSAGE);
uint8_t *cipher_suites = s2n_stuffer_raw_read(in, cipher_suites_length);
RESULT_ENSURE(cipher_suites != NULL, S2N_ERR_BAD_MESSAGE);
RESULT_GUARD_POSIX(s2n_blob_init(&client_hello->cipher_suites, cipher_suites, cipher_suites_length));
/* legacy_compression_methods (ignored) */
uint8_t num_compression_methods = 0;
RESULT_GUARD_POSIX(s2n_stuffer_read_uint8(in, &num_compression_methods));
RESULT_GUARD_POSIX(s2n_stuffer_skip_read(in, num_compression_methods));
/* extensions */
RESULT_GUARD_POSIX(s2n_extension_list_parse(in, &client_hello->extensions));
return S2N_RESULT_OK;
}
Some checks are only enabled during development and test (i.e. the macros evaluate to nothing during production builds):
S2N_RESULT s2n_blob_validate(const struct s2n_blob *b)
{
RESULT_ENSURE_REF(b);
RESULT_DEBUG_ENSURE(S2N_IMPLIES(b->data == NULL, b->size == 0), S2N_ERR_SAFETY);
RESULT_DEBUG_ENSURE(S2N_IMPLIES(b->data == NULL, b->allocated == 0), S2N_ERR_SAFETY);
RESULT_DEBUG_ENSURE(S2N_IMPLIES(b->growable == 0, b->allocated == 0), S2N_ERR_SAFETY);
RESULT_DEBUG_ENSURE(S2N_IMPLIES(b->growable != 0, b->size <= b->allocated), S2N_ERR_SAFETY);
RESULT_DEBUG_ENSURE(S2N_MEM_IS_READABLE(b->data, b->allocated), S2N_ERR_SAFETY);
RESULT_DEBUG_ENSURE(S2N_MEM_IS_READABLE(b->data, b->size), S2N_ERR_SAFETY);
return S2N_RESULT_OK;
}
Memory management
Memory management, i.e. heap based memory, is typically managed directly though wrappers for malloc/free. Some libraries choose to implement stack canaries (builtin or custom) directly for defense-in-depth and also to better catch overflow conditions.
Data parsing
One of the big differences between OpenSSL and modern libraries is the use of structured data parsing. OpenSSL seems to prefer ad-hoc parsing of byte arrays which resulted in e.g. CVE-2022–3602:

Modern libraries parse data via structured functions. For instance, BoringSSL implements parsing functions using the following data structure (also known as Crypto Byte String (CBS)):
struct cbs_st {
const uint8_t *data;
size_t len;
};
Functions manipulating with CBS structure are then used to implement whatever parsing is needed:
static int SSL_SESSION_to_bytes_full(const SSL_SESSION *in, CBB *cbb,
int for_ticket) {
if (in == NULL || in->cipher == NULL) {
return 0;
}
CBB session, child, child2;
if (!CBB_add_asn1(cbb, &session, CBS_ASN1_SEQUENCE) ||
!CBB_add_asn1_uint64(&session, kVersion) ||
!CBB_add_asn1_uint64(&session, in->ssl_version) ||
!CBB_add_asn1(&session, &child, CBS_ASN1_OCTETSTRING) ||
!CBB_add_u16(&child, (uint16_t)(in->cipher->id & 0xffff)) ||
// The session ID is irrelevant for a session ticket.
!CBB_add_asn1_octet_string(&session, in->session_id,
for_ticket ? 0 : in->session_id_length) ||
!CBB_add_asn1_octet_string(&session, in->secret, in->secret_length) ||
!CBB_add_asn1(&session, &child, kTimeTag) ||
!CBB_add_asn1_uint64(&child, in->time) ||
!CBB_add_asn1(&session, &child, kTimeoutTag) ||
!CBB_add_asn1_uint64(&child, in->timeout)) {
return 0;
}
// SNIP
}
This is clearly a lot easier to understand and also way more robust than the example from OpenSSL… Both LibreSSL and s2n implemeting similar data structures and functions.
Note: A huge difference between the SSL/TLS and SSH protocols is that SSL/TLS use binary structures (for defining e.g. the cipher suites) while SSH relies on UTF-8 strings. Interestingly, SSH libraries have not been riddled with the same vulnerabilities as the SSL/TLS libraries…
Cryptography engineering
A really challenging part, is whats known as cryptography engineering. This covers topics like:
- Not properly seeding the random number generator (such as the “Debian OpenSSH” vulnerability).
- Not implementing the algorithms correctly (such as not using a random number during DSA signature generation).
- Not validating an RSA signature before returning it.
- Not being aware of side-channel attacks (such as not doing critical operations in constant time making key recovery possible over the network).
- Not being aware of previous attacks (such as Bleichenbacher attack).
- And so on…
Test and validation
Structured testing and validation is critical. The following techniques are very useful:
- Enable compiler warnings and fix findings; when using different compilers, this is very helpful as all compilers generate different warnings.
- Unit tests using known test data from e.g. standards. This also includes the great test suite from Project Wycheproof (https://github.com/google/wycheproof).
- Validation using other good implementations.
- Fuzzing is especially good when testing ASN.1 parsing…
- Formal verification (see e.g. https://www.wireguard.com/formal-verification/).
- And also, use this process to remove unused / legacy features.