Random Thoughts

.NET's Cryptographic One-Shots

Thu, 30 Dec 2021 20:00:00 +0000

Over the past few releases in .NET, formerly .NET Core, there has been progress on making cryptographic primitives like AES, SHA, etc. better for developers to use.

“Better” is an interesting point of conversation with cryptographic API design. To a developer, better may mean more throughput, less allocations, or a simply less cumbersome API. To a framework or library author, it means thinking about how developers will use, or mis-use, an API.

Let’s look at AES encryption as it was in the .NET Framework days:

using System.Security.Cryptography;

byte[] data = default;

using (Aes aes = Aes.Create())
{
    byte[] key = default; // Get a key from somewhere
    byte[] iv = default; // Get a unique IV from somewhere

    using ICryptoTransform transform = aes.CreateEncryptor(key, iv);
    byte[] encrypted = transform.TransformFinalBlock(data, 0, data.Length);
}

This may only be a dozen or so lines of a code, but not all of it is straight forward. What is a “transform”? What is a “block”? What does “final” mean?

The APIs expose quite a bit of functionality, and are confusingly named. TransformFinalBlock for example, despite having “Block” in it’s name, is almost always going to be capable of encrypting more than one block. It also means the data doesn’t need to be block aligned, so it handles padding appropriately. Since none of that may be understood, developers often times work around perceived problems, like handling individual blocks. While this API design offers the most flexibility for developers, it also offers the most complexity.

This complexity exists for a small group of people. Most developers have some small amount of data they want to encrypt, and when they don’t, they have a Stream.

Complex APIs that are prone to misuse are problematic in a security context. A misused cryptographic API almost always harms intended goal of the cryptographic API, whether that be secrecy, integrity, etc.

For .NET, this became a concern when AES GCM and CCM were exposed in .NET Core 3.1. The AesGcm and AesCcm classes do not follow the design of AES prior. In addition to not inheriting from Aes or SymmetricAlgorithm, they also do not expose any block functionality. This was discussed at length, and instead these types offer simple Encrypt or Decrypt APIs that takes data and return data, or write to an existing buffer.

This, while less flexible, resolves many concerns about misusing the AES GCM cipher mode. Primarily among those concerns was releasing unauthenticated data. Streaming decryption is, put simply, difficult to do safely.

“Streaming” decryption doesn’t necessarily mean the use of System.IO.Stream. Rather, it means processing a block of plaintext or ciphertext a block at a time and doing something with it in the middle of encrypting or decrypting. This is often perceived as desirable when handling large amounts of data. After all, if I have a 12 gigabyte file, I can’t just put that in a byte array and encrypt it. Rather, processing it in chunks lets me handle it in memory.

In pseudo code, let’s say I wanted to decrypt a file and send it over the network:

# NOTE: This is an example of doing things improperly
encryptedFileStream = getFile()
stream = getStream()

loop {
    data = encryptedFileStream.read(128 / 8) # One AES block size

    if (data.length == 128 / 8) {
        stream.write(decrypt(data))
    }
    else {
        stream.write(decryptFinal(data))
        stream.close()
        break
    }
}

Recall though that AES GCM is authenticated. That is, AES GCM can tell if your ciphertext has been modified while in storage or in transit. An important detail of this though is that GCM cannot authenticate until it has processed the entire ciphertext (when decryptFinal is called.)

This is breaks down because as we are decrypting, we’re sending (releasing) the plaintext before AES GCM has been able to authenticate the entire cipher text. If the person, tool, whatever on the other end of the network is processing that decrypted data in real time, then they have processed unauthentic data and it’s too late to go back and tell them “Never mind, that data I send you a few seconds ago might have been tampered with.”

There are correct ways to do this, but are also still difficult to do correctly. You could break the file up in to small chunks and treat them as individual ciphertexts. However then you need to worry about many nonces, ensuring chunks are processed in the right order, a chunk isn’t missing, or replayed, etc.

Before long you’ve invented a cryptographic protocol. This is largely why many folks will recommend using something that is well understood and robust rather than trying to build it yourself. Though not a primitive cipher, this kind of problem falls in the “roll your own cryptography” bucket.

It’s rather easy to accidentally roll your own cryptography, especially so when working with “streaming” data.

In .NET then, AES GCM and CCM do not support encrypting individual blocks. This still does not solve the issue of ensuring chunks are handled appropriately when handling large amounts of data. For that, higher level tools are still recommended. However it removes the temptation for streaming AES GCM, for which any attempt to use is almost always incorrect. Since it is very difficult to use correctly with no practical use cases, it isn’t offered.

Toward Better APIs

Simple APIs are important to making them misuse resistent, and .NET has gotten better at that over the past few releases.

Like AesGcm, the SymmetricAlgorithm and its derivatives like Aes, TripleDES, etc. all offer similar one-shot APIs starting in .NET 6 in the form of EncryptCbc, EncryptEcb or DecryptCbc and DecryptEcb.

using System.Security.Cryptography;

byte[] data = default;

using (Aes aes = Aes.Create())
{
    byte[] key = default; // Get a key from somewhere
    byte[] iv = default; // Get a unique IV from somewhere

    aes.Key = key;

    // Encrypt all the data at once
    byte[] encrypted = aes.EncryptCbc(data, iv);
}

There is no ICryptoTransform that needs to be reasoned about or disposed, and there is no need to worry about blocks, padding, etc. Where possible, one shots have been added in most places, and made static where possible.

For hashing, prior to .NET 5 it would look something like this:

// Prior to .NET 5
using System.Security.Cryptography;

byte[] data = default; // Some data

using (SHA256 hash = SHA256.Create())
{
    byte[] digest = hash.ComputeHash(data);
}

Rather than creating an instance of a hash algorithm, HashData now exists:

// Starting in .NET 5
using System.Security.Cryptography;

byte[] data = default; // Some data
byte[] digest = SHA256.HashData(data);

This is much easier to reason about. The method is static, there is no stateful hash object that needs to be instantiated, no need to remember to dispose of it, and no need to worry about thread safety. Not only are the one shots easier to use, they almost always offer better performance, either in throughput or reduced allocations. These one shots are not simple wrappers around HashAlgorithm.Create() and then hashing something. They internally do not allocate on the managed heap at all. Everyone benefits here: the APIs are simpler, and developers get better performance.

For .NET 6, the one shot hashing APIs were brought to the HMAC classes as well, offering the same improved APIs and better performance.

Also for .NET 6 PBKDF2 got the same treatment with Rfc2898DeriveBytes.Pbkdf2.

using System.Security.Cryptography;

byte[] salt = RandomNumberGenerator.GetBytes(32);
byte[] prk = Rfc2898DeriveBytes.Pbkdf2(
    userPassword,
    salt,
    iterations: 200_000,
    HashAlgorithmName.SHA256,
    outputLength: 32);

All of these APIs also offer modern amenities, like working ReadOnlySpan<byte> for input data and being able to write to a Span<byte> for output data.

I’m happy with .NETs move toward easier to use APIs for cryptographic primitives. I still largely believe many developers should use higher level concepts rather than these basic building blocks. However, for those that need the building blocks, they are getting better.

Dos and Don'ts of stackalloc

Mon, 24 Feb 2020 05:24:00 +0000

In .NET Core 2.1 a small but well-received feature was the ability to “safely” allocate a segment of data on the stack, using stackalloc, when used with Span<T>.

Before Span<T>, stackalloc required being in an unsafe context:

unsafe {
    byte* data = stackalloc byte[256];
}

The use of unsafe, along with the little number of APIs in .NET that could work with pointers, was enough to deter a lot of people from using it. As a result, it remained a relatively niche feature. The introduction of Span<T> now means this can be done without being in an unsafe context now:

Span<byte> data = stackalloc byte[256];

The .NET team has also been working diligently to add Span<T> APIs where it makes sense. There is now more appeal and possibility to use stackalloc.

stackalloc is desirable in some performance sensitive areas. It can be used in places where small arrays were used, with the advantage that it does not allocate on the heap - and thus does not apply pressure to the garbage collector. stackalloc is not a general purpose drop-in for arrays. They are limited in a number of ways that other posts explain well enough, and require the use of Span<T>.

Recently I vented a bit about stackalloc on Twitter, as one does on Twitter, specifically the community’s fast embrace of it, without discussing or well-documenting some of stackalloc’s sharp edges. I’m going to expand on that here, and make an argument for stackalloc still being unsafe and requiring some thought about being used.

DON’T: Use variable allocation lengths

A large risk with using stackalloc is running out of stack space. If you’ve ever written a method that is recursive and went too deep, you’ll eventually receive a StackOverflowException. The StackOverflowException is a bit special in that it is one of the exceptions that cannot be caught. When a stack overflow occurs, the process immediately exits. Allocating too much with stackalloc has the same effect - it causes a stack overflow and the process immediately terminates.

This is particularly worrisome when the allocation’s length is determined by user input:

Span<char> buffer = stackalloc char[userInput.Length]; //DON'T

This allows users to take down your process, an effective denial-of-service.

Using a constant also reduces the risk of arithmetic or overflow mistakes. Currently, .NET Core’s CLR interprets that amount to be allocated as an unsigned integer. This means that an arithmetic over or under flow may result in a stack overflow.

Span<byte> b = stackalloc byte[(userInput % 64) - 1]; //DON'T

DO: Use a constant for allocation size

Instead, it’s better to use a constant value for stackalloc, always. It immediately resolves any ambiguities about how much is allocated on the stack.

Span<char> buffer = stackalloc char[256]; //better

Once you have an allocated buffer, you can use Span’s Slice funtionality to adjust it to the correct size:

Span<char> buffer = stackalloc char[256];
Span<char> input = buffer.Slice(0, userInput.Length);

DON’T: Use stackalloc in non-constant loops

Even if you allocate a fixed length amount of data on the stack, doing so in a loop can be dangerous as well, especially if the number of the iterations the loop makes is driven by user input:

for (int i = 0; i < userInput; i++) { // DON'T
    Span<char> buffer = stackalloc char[256];
}

This also can cause a denial of service, since this allows someone to control the number of stack allocations, though not the length of the allocation.

DO: Allocate outside of loops

Span<char> buffer = stackalloc char[256]; //better
for (int i = 0; i < userInput; i++) {
    //Do something with buffer
}

Allocating outside of the loop is the best solution. This is not only safer, but also better for performance.

DON’T: Allocate a lot on the stack

It’s tempting to allocate as much as nearly possible on the stack:

Span<byte> data = stackalloc byte[8000 * 1024]; // DON'T

You may find that this runs fine on Linux, but fails on Windows with a stack overflow. Different operating systems, architectures, and environments, have different stacks limits. Linux typically allows for a larger stack than Windows by default, and other hosting scenarios such as in an IIS worker process come with even lower limits. An embedded environment may have a stack of only a few kilobytes.

DO: Conservatively use the stack

The stack should be used for small allocations only. How much depends on the size of each element being allocated. It’s also desirable to not allocate many large structs, either.

I won’t prescribe anything specific, but anything larger than a kilobyte is a point of concern. You can allocate on the heap depending on how much you need. A typical pattern might be:

const int MaxStackSize = 256;
Span<byte> buffer =
    userInput > MaxStackSize
      ? new byte[userInput]
      : stackalloc byte[MaxStackSize];

Span<byte> data = buffer.Slice(0, userInput);

This will allocate on the stack for small amounts, still in a constant amount, or if too large, will use a heap-allocated array. This pattern may also make it easier to use ArrayPool, if you choose, which also does not guarantee that the returned array is exactly the requested size:

const int MaxStackSize = 256;
byte[]? rentedFromPool = null;
Span<byte> buffer =
    userInput > MaxStackSize
    ? (rentedFromPool = ArrayPool<byte>.Shared.Rent(userInput))
    : stackalloc byte[MaxStackSize];

// Use data
Span<byte> data = buffer.Slice(0, userInput);

// Return from pool, if we rented
if (rentedFromPool is object) {
    // DO: if using ArrayPool, think carefully about clearing
    // or not clearing the array.
    ArrayPool<byte>.Shared.Return(rentedFromPool, clearArray: true);
}

DON’T: Assume stack allocations are zero initialized

Most normal uses of stackalloc result in zero-initialized data. This behavior is however not guaranteed, and can change depending if the application is built for Debug or Release, and other contents of the method. Therefore, don’t assume that any of the elements in a stackalloced Span<T> are initialized to something by default. For example:

Span<byte> buffer = stackalloc byte[sizeof(int)];
byte lo = 1;
byte hi = 1;
buffer[0] = lo;
buffer[1] = hi;
// DONT: depend on elements at 2 and 3 being zero-initialized
int result = BinaryPrimitives.ReadInt32LittleEndian(buffer);

In this case, we might expect the result to be 257, every time. However if the stackalloc does not zero initialize the buffer, then the contents of the upper-half of the integer will not be as expected.

This behavior will not always be observed. In Debug builds, it’s likely that you will see that stackalloc zero-initializes its contents every time, whereas in Release builds, you may find that the contents of a stackalloc are uninitialized.

Starting in .NET 5, developers can opt to explicitly skip zero-initializing stackalloc contents with the SkipLocalsInit attribute. Without it, whether or not stackalloc is default initialized is up to Roslyn.

DO: Initialize if required

Any item read from a stackalloced buffer should be explicitly assigned, or use Clear to explicitly clear the entire Span<T> and initialize it to defaults.

Span<byte> buffer = stackalloc byte[sizeof(int)];
buffer.Clear(); //explicit zero initialize
byte lo = 1;
byte hi = 1;
buffer[0] = lo;
buffer[1] = hi;
int result = BinaryPrimitives.ReadInt32LittleEndian(buffer);

Though not explicitly covered in this post, the same advice applies to arrays rented from the ArrayPool.

Summary

In summary, stackalloc needs to be used with care. Failing to do so can result in process termination, which is a denial-of-service: your program or web server aren’t running any more.

Import and Export RSA Key Formats in .NET Core 3

Mon, 07 Oct 2019 20:24:00 +0000

.NET Core 3.0 introduced over a dozen new APIs for importing and exporting RSA keys in different formats. Many of them are a variant of another with a slightly different API, but they are extremely useful for working with private and public keys from other systems that work with encoding keys.

RSA keys can be encoded in a variety of different ways, depending on if the key is public or private or protected with a password. Different programs will import or export RSA keys in a different format, etc.

Often times RSA keys can be described as “PEM” encoded, but that is already ambiguous as to how the key is actually encoded. PEM takes the form of:

-----BEGIN LABEL-----
content
-----END LABEL-----

The content between the labels is base64 encoded. The one that is probably the most often seen is BEGIN RSA PRIVATE KEY, which is frequently used in web servers like nginx, apache, etc:

-----BEGIN RSA PRIVATE KEY-----
MII...
-----END RSA PRIVATE KEY-----

The base64-encoded text is an RSAPrivateKey from the PKCS#1 spec, which is just an ASN.1 SEQUENCE of integers that make up the RSA key. The corresponding .NET Core 3 API for this is ImportRSAPrivateKey, or one of its overloads. If your key is “PEM” encoded, you need to find the base64 text between the label BEGIN and END headers, base64 decode it, and pass to ImportRSAPrivateKey. There is currently an API proposal to make reading PEM files easier. If your private key is DER encoded, then that just means you can read the content directly as bytes in to ImportRSAPrivateKey.

Here is an example:

var privateKey = "MII..."; //Get just the base64 content.
var privateKeyBytes = Convert.FromBase64String(privateKey);
using var rsa = RSA.Create();
rsa.ImportRSAPrivateKey(privateKeyBytes, out _);

When using openssl, the openssl rsa commands typically output RSAPrivateKey PKCS#1 private keys, for example openssl genrsa.

A different format for a private key is PKCS#8. Unlike the RSAPrivateKey from PKCS#1, a PKCS#8 encoded key can represent other kinds of keys than RSA. As such, the PEM label for a PKCS#8 key is “BEGIN PRIVATE KEY” (note the lack of “RSA” there). The key itself contains an AlgorithmIdentifer of what kind of key it is.

PKCS#8 keys can also be encrypted protected, too. In that case, the PEM label will be “BEGIN ENCRYPTED PRIVATE KEY”.

.NET Core 3 has APIs for both of these. Unencrypted PKCS#8 keys can be imported with ImportPkcs8PrivateKey, and encrypted PKCS#8 keys can be imported with ImportEncryptedPkcs8PrivateKey. Their usage is similar to ImportRSAPrivateKey.

Public keys have similar behavior. A PEM encoded key that has the label “BEGIN RSA PUBLIC KEY” should use ImportRSAPublicKey. Also like private keys, the public key has a format that self-describes the algorithm of the key called a Subject Public Key Info (SPKI) which is used heavily in X509 and many other standards. The PEM header for this is “BEGIN PUBLIC KEY”, and ImportSubjectPublicKeyInfo is the correct way to import these.

All of these APIs have export versions of themselves as well, so if you are trying to export a key from .NET Core 3 to a particular format, you’ll need to use the correct export API.

To summarize each PEM label and API pairing:

“BEGIN RSA PRIVATE KEY” => RSA.ImportRSAPrivateKey
“BEGIN PRIVATE KEY” => RSA.ImportPkcs8PrivateKey
“BEGIN ENCRYPTED PRIVATE KEY” => RSA.ImportEncryptedPkcs8PrivateKey
“BEGIN RSA PUBLIC KEY” => RSA.ImportRSAPublicKey
“BEGIN PUBLIC KEY” => RSA.ImportSubjectPublicKeyInfo

One gotcha with openssl is to pay attention to the output of the key format. A common enough task from openssl is “Given this PEM-encoded RSA private key, give me a PEM encoded public-key” and is often enough done like this:

openssl rsa -in key.pem -pubout

Even if key.pem is a PKCS#1 RSAPrivateKey (“BEGIN RSA PRIVATE KEY”), the -pubout option will output a SPKI (“BEGIN PUBLIC KEY”), not an RSAPublicKey (“BEGIN RSA PUBLIC KEY”). For that, you would need to use -RSAPublicKey_out instead of -pubout. The openssl pkey commands will also typically give you PKCS#8 or SPKI formatted keys.

Sometimes valid RSA signatures in .NET

Thu, 18 Jul 2019 17:27:00 +0000

One of the nice things about .NET Core being open source is following along with some of the issues that people report. I tend to keep an eye on System.Security tagged issues, since those tend to be at the intersection of things that interest me and things I can maybe help with.

A user filed an issue where .NET Framework considered a CMS valid, and .NET Core did not. This didn’t entirely surprise me. In the .NET Framework, the SignedCms class is heavily backed by Windows’ handling of CMS/PKCS#7. In .NET Core, the implementation is managed (sans the cryptography). The managed implementation adheres somewhat strictly to the CMS specification. As other issues have noticed, Windows’, thus .NET Framework’s, implementation was a little more relaxed in some ways.

This turned out not to be one of those cases. The CMS part was actually working just fine. What was failing was RSA itself. The core of the issue was that different implementations of RSA disagreed on the RSA signature’s validity.

That seems pretty strange!

When I talk about different implementations on Windows, I am usually referring to CAPI vs CNG, or RSACryptoServiceProvider and RSACng, respectively. For now, I’m keeping this post to the .NET Framework. We’ll bring .NET Core in to the discussion later.

There are two implementations because, well, Windows has two of them. CNG, or “Cryptography API: Next Generation” is the newer of the two and is intended to be future of cryptographic primitives on Windows. It shipped in Windows Vista, and offers functionality that CAPI cannot do. An example of that is PSS RSA signatures.

.NET Framework exposes these implementations as RSACryptoServiceProvider and RSACng. They should be interchangable, and CNG implementations should be used going forward. However, there is one corner case where the old, CAPI implementation considers a signature valid while the CNG one does not.

The issue can be demonstrated like so:

byte[] n = new byte[] { ... };
byte[] e = new byte[] { ... };
byte[] signature = new byte[] { ... };
var digest = new byte[] {
    0x68, 0xB4, 0xF9, 0x26, 0x34, 0x31, 0x25, 0xDD,
    0x26, 0x50, 0x13, 0x68, 0xC1, 0x99, 0x26, 0x71,
    0x19, 0xA2, 0xDE, 0x81, 
};
using (var rsa = new RSACng())
{
    rsa.ImportParameters(new RSAParameters {
        Modulus = n,
        Exponent = e
    });
    var valid = rsa.VerifyHash(digest, signature, HashAlgorithmName.SHA1,
                               RSASignaturePadding.Pkcs1);
    Console.WriteLine(valid);
}
using (var rsa = new RSACryptoServiceProvider())
{
    rsa.ImportParameters(new RSAParameters {
        Modulus = n,
        Exponent = e
    });
    var valid = rsa.VerifyHash(digest, signature, HashAlgorithmName.SHA1,
                               RSASignaturePadding.Pkcs1);
    Console.WriteLine(valid);
}

When used with one of the curious signatures that exhibits this behavior, such as the one in the GitHub link, the first result will be false, and the second will be true.

Nothing jumped out at me as being problematic. The signature padding is PKCS, the public exponent is the very typical 67,537, and the RSA key is sensible in size.

To make it stranger, this signature came off the timestamp of Firefox’s own signed installer. So why are the results different?

Jeremy Barton from Microsoft on .NET Core made the observation that the padding in the RSA signature itself is incorrect, but in a way that CAPI tollerates and CNG does not, at least by default. Let’s look at the raw signature. To do that, we need the public key and signature on disk, and we can poke at them with OpenSSL.

Using the command:

openssl rsautl -verify -in sig.bin -inkey key.der \
    -pubin -hexdump -raw -keyform der

We get the following output:

0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00d0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00e0 - ff ff ff ff ff ff ff ff-ff ff ff 00 68 b4 f9 26
00f0 - 34 31 25 dd 26 50 13 68-c1 99 26 71 19 a2 de 81

This is a PKCS#1 v1.5 padded signature, as indicated by by starting with 00 01. The digest at the end can be seen, 68 b4 f9 26 ... 19 a2 de 81 which matches the digest above, so we know that the signature is for the right digest.

What is not correct in this signature is how the digest is encoded. The signature contains the bare digest. It should be encoded as an ASN.1 sequence along with the AlgorithmIdentifer of the digest:

DigestInfo ::= SEQUENCE {
	digestAlgorithm AlgorithmIdentifier,
	digest OCTET STRING
}

This goes back all the way to a document (warning: link is to an ftp:// site) written in 1993 by RSA labratories explaining how PKCS#1 v1.5 works,and was standardized in to an RFC in 1998.

The RSA signature we have only contains the raw digest. It is not part of a DigestInfo. If the digest were properly encoded, it would look something like this:

0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
00d0 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 21 30
00e0 - 09 06 05 2b 0e 03 02 1a-05 00 04 14 68 b4 f9 26
00f0 - 34 31 25 dd 26 50 13 68-c1 99 26 71 19 a2 de 81

The signature now includes DigestInfo along with the OID 1.3.14.3.2.26 to indicate that the digest is SHA1.

At this point we know what the difference is, and the original specification in part 10.1.2 makes it fairly clear that the “data” should be a digest and should be encoded as DigestInfo, not a bare digest.

The source of this signature is from Verisign's timestamp authority at http://timestamp.verisign.com/scripts/timstamp.dll. After checking with someone at DigiCert (now running this TSA), it was launched in May 1995.

I suspect that the TSA is old enough that the implementation was made before the specification was complete or simply got the specification wrong and no one noticed. Bringing this back to CNG and CAPI, CNG can validate this signatures, but you must explicitly tell CNG that the signature does not have an object identifier. BCRYPT_PKCS1_PADDING_INFO’s documentation has the detail there, but gist of it is

If there is no OID in the signature, then verification fails unless this member is NULL.

This would be used with {B,N}CryptVerifySignature. To bring this back around to the .NET Framework, how do we use RSACng and give null in for the padding algorithm? The short answer is: you cannot. If you try, you will get an explicit ArgumentException saying that the hash algorithm name cannot be null.

For .NET Framework, this solution “keep using RSACryptoServiceProvider”. If you need to validate these signatures, chances are you do not need to use CNG’s newer capabilities like PSS since these malformed signatures appear to be coming from old systems. Higher level things like SignedCms and SignedXml use RSACryptoServiceProvider by default, so they will continue to work.

To bring in .NET Core, the situation is a little more difficult. If you are using SignedCms like so:

var signedCms = new SignedCms();
signedCms.Decode(File.ReadAllBytes("cms-with-sig.bin"));
signedCms.CheckSignature(true);

This will start throwing when you migrate to .NET Core. .NET Core will use CNG when run on Windows to validate RSA signatures for SignedCms and SignedXml. This is currently not configurable, either. When used with SignedCms, it ultimately calls the X509Certificate2.GetRSAPublicKey() extension method, and that will always return an implementation based on CNG.

If you are using SignedCms on .NET Core and need to validate a CMS that is signed with these problematic signatures, you are currently out of luck using in-the-box components. As far as other platforms go, both macOS and Linux environments for .NET Core will agree with CNG - that the signature is invalid.

The good news is, these signatures are not easy to come by. So far, only the old Verisign timestamp authority is known to have produced signatures like this.

C# ReadOnlySpan and static data

Fri, 01 Feb 2019 14:20:00 +0000

Since C# 7 there have been a lot of point releases that contain all kinds of goodies. Many of them are performance focused, such as safe stack allocations using Span<T>, or interoperability with improvements to fixed.

One that I love, but is not documented well, is some special treatment that ReadOnlySpan<byte> gets when its contents are known at compile time.

Here’s an example of a lookup table I used to aide with hex encoding that uses a byte[]:

private static byte[] LookupTable => new byte[] {
    (byte)'0', (byte)'1', (byte)'2', (byte)'3', (byte)'4',
    (byte)'5', (byte)'6', (byte)'7', (byte)'8', (byte)'9',
    (byte)'A', (byte)'B', (byte)'C', (byte)'D', (byte)'E',
    (byte)'F',
};

This binary data has to get stored somewhere in our produced library. If we use dumpbin we can see it in the .text section of the binary.

dumpbin /RAWDATA /SECTION:.text mylib.dll

Right at the bottom, we see:

00402A40: 30 31 32 33 34 35 36 37 38 39 41 42 43 44 45 46  0123456789ABCDEF

I won’t go into the a lot of the details on how this data is compiled into the .text section, but at this point we need to get that data into the array somehow.

If we look at the jit assembly of LookupTable, we see:

sub rsp, 0x28
vzeroupper
mov rcx, 0x7ffc4638746a
mov edx, 0x10
call 0x7ffc49b52630
mov rdx, 0x1b51450099c
lea rcx, [rax+0x10]
vmovdqu xmm0, [rdx]
vmovdqu [rcx], xmm0
add rsp, 0x28
ret

Where 0x7ffc49b52630 is InitializeArray.

With an array, our property leans on InitializeArray, the source of which is in the CoreCLR. For little-endian platforms, it boils down to a memcpy from a runtime field handle.

Indeed, with a debugger we finally see:

00007ffd`b18b701a e831a40e00       call    coreclr!memcpy (00007ffd`b19a1450)

Dumping @rdx L10 yields:

000001f0`4c552a90  30 31 32 33 34 35 36 37-38 39 41 42 43 44 45 46  0123456789ABCDEF

So that was a very long-winded way of saying that when using arrays, initializing a field or variable with bytes results in memcpy from the image into the array, which results in more data on the heap.

Now, starting in 7.3, we can avoid that memcpy when using ReadOnlySpan<byte>.

private static ReadOnlySpan<byte> LookupTable => new byte[] {
    (byte)'0', (byte)'1', (byte)'2', (byte)'3', (byte)'4',
    (byte)'5', (byte)'6', (byte)'7', (byte)'8', (byte)'9',
    (byte)'A', (byte)'B', (byte)'C', (byte)'D', (byte)'E',
    (byte)'F',
};

Looking at the jit assembly:

mov eax, 0x10
xor edx, edx
mov r8, 0x1b5144c0968
mov [rcx], rdx
mov [rcx+0x8], r8
mov [rcx+0x10], eax
mov rax, rcx
ret

We see that there is mov r8, 0x1b5144c0968. The contents of 0x1b5144c0968 are:

000001b5`144c0968  30 31 32 33 34 35 36 37-38 39 41 42 43 44 45 46  0123456789ABCDEF

So we see that the method is now returning the data directly and omitting the memcpy entirely, so our ReadOnlySpan<byte> is pointing directly to the .text section.

This works for property getters as shown above, but also as the return of a method:

ReadOnlySpan<byte> GetBytes() {
    return new byte[] { ... };
}

Which works similar to the getter of the property. In addition, this also works for locals in a method body as well:

void Write200Ok(Stream s) {
    ReadOnlySpan<byte> data = new byte[] {
        (byte)'H', (byte)'T', (byte)'T', (byte)'P',
        (byte)'/', (byte)'1', (byte)'.', (byte)'1',
        (byte)' ', (byte)'2', (byte)'0', (byte)'0',
        (byte)' ', (byte)'O', (byte)'K'
    };
    s.Write(data);
}

Which also produces a reasonable JIT disassembly:

sub     rsp, 0x38
xor     eax, eax
mov     qword ptr [rsp+0x28], rax
mov     qword ptr [rsp+0x30], rax
mov     rcx, 0x1e595b42ade
mov     eax, 0x0F
lea     r8, [rsp+0x28]
mov     qword ptr [r8], rcx
mov     dword ptr [r8+8], eax
mov     rcx, rdx
lea     rdx, [rsp+0x28]
cmp     dword ptr [rcx], ecx
call    0x7ff89ede10c8 (Stream.Write(System.ReadOnlySpan`1<Byte>), mdToken: 0000000006000001)
add     rsp, 0x38
ret

Here we see mov rcx, 0x1e595b42ade which moves the address of the static data directly in to the register with no additional work to create a byte array.

These optimizations currently only works with ReadOnlySpan<byte> right now. Other types will continue to use InitializeArray due to needing to handle different platforms and how they handle endianness.

C# 8 using declarations

Wed, 30 Jan 2019 23:20:00 +0000

Visual Studio 2019 preview 2 was released a few days ago and I took the time to install it. Visual Studio itself is actually rather uninteresting to me, however the inclusion of the next C# 8 preview got my attention. I glanced at the feature highlights and posted “looks nice” on Twitter.

Predictably, I got a few responses like “I’m not sure I like that”, and there is always a guarantee that if F# has a similar feature, an F# developer will appear and tell you F# has had this feature for 600 years.

The one I like a lot is using declarations. This allows a local to automatically be disposed at the end of the block. Essentially, it hides the try/finally or the using() {...}. The .NET team’s blog kind of gave a bad example of this, so I’ll use one from Open OPC SignTool. Here is the original snippet:

private static X509Certificate2 GetCertificateFromCertificateStore(string sha1)
{
    using (var store = new X509Store(StoreName.My, StoreLocation.LocalMachine))
    {
        store.Open(OpenFlags.OpenExistingOnly | OpenFlags.ReadOnly);
        var certificates = store.Certificates.Find(X509FindType.FindByThumbprint, sha1, false);
        return certificates.Count > 0 ? certificates[0] : null;
    }
}

A using var can make this:

private static X509Certificate2 GetCertificateFromCertificateStore(string sha1)
{
    using var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
    store.Open(OpenFlags.OpenExistingOnly | OpenFlags.ReadOnly);
    var certificates = store.Certificates.Find(X509FindType.FindByThumbprint, sha1, false);
    return certificates.Count > 0 ? certificates[0] : null;
}

This has the same effect of store having Dispose called on it at the end of the method. The benefit here being that there is less indentation and braces. This keeps me focused on the code that matters. I don’t care when store is disposed in the method, I can just observe that it has a using modifier on the local and be assured that Dispose will be called.

This isn’t the same as garbage collection or finalizers. Both of those are non- deterministic, and can lead to unexpected program behavior. That’s less so in the case of X509Store, so let’s look at another example:

using Stream stream = entry.Open();
var xmlDocument = XDocument.Load(stream, LoadOptions.PreserveWhitespace);
return new OpcRelationships(location, xmlDocument, readOnlyMode);

Not disposing a stream that is backed by a file can cause access errors later in software that might try to open that file again - it is already open, so not only is it a bad idea it leave streams to the GC, it is just simply incorrect.

However again using on the local ensures it is deterministically closed.

When it gets disposed I can see being slightly unclear to the developer. The quick explanation is when the local is no longer reachable, not when it is last used. The C# 8 above gets compiled roughly to:

var stream = entry.Open();
try
{
    var xmlDocument = XDocument.Load(stream, LoadOptions.PreserveWhitespace);
    return new OpcRelationships(location, xmlDocument, readOnlyMode);
}
finally
{
    if (stream != null)
    {
        ((IDisposable)stream).Dispose();
    }
}

The disposal is done after the return, when the local is no longer reachable, not after XDocument is created.

I find this very helpful to keep code readable. This doesn’t work when you need fine control over when Dispose is called. A place where this does not work well is when the Dispose pattern is used for scopes, such as logging. The AzureSignTool project has code similar to this in SignCommand:

var logger = loggerFactory.CreateLogger<SignCommand>();
Parallel.ForEach(AllFiles, options, () => (succeeded: 0, failed: 0), (filePath, pls, state) =>
{
    using (var loopScope = logger.BeginScope("File: {Id}", filePath))
    {
        logger.LogInformation("Signing file.");
        //Sign the file. omit a bunch of other code.
        logger.LogInformation("Done signing the file.");
    }
    logger.LogDebug("Incrementing success count.");
    return (state.succeeded + 1, state.failed);
}

Here, we cannot change this to a using var because then the LogDebug would be inside of that logging scope, which it wasn’t before. This is a place where we continue to want Dispose to be called at a different time from the when loopScope would no longer be in scope.

My impression from C# developers is that they do not tend to call Dispose on resources as soon as it can be disposed, just at a reasonable point in the same method. Most developers do not write this code:

public bool MightBeExe(string filePath)
{
    var firstBytes = new byte[2];
    int bytesRead;
    using (var file = File.Open(filePath, FileMode.Open))
    {
        bytesRead = file.Read(firstBytes, 0, 2);
    }
    return bytesRead == 2 && firstBytes[0] == (byte)'M' && firstBytes[1] == (byte)'Z';
}

They will instead write something like:

public bool MightBeExe(string filePath)
{
    using (var file = File.Open(filePath, FileMode.Open))
    {
        var firstBytes = new byte[2];
        var bytesRead = file.Read(firstBytes, 0, 2);
        return bytesRead == 2 && firstBytes[0] == (byte)'M' && firstBytes[1] == (byte)'Z';
    }
}

Which is a perfect candidate for using var:

public bool MightBeExe(string filePath)
{
    using var file = File.Open(filePath, FileMode.Open);
    var firstBytes = new byte[2];
    var bytesRead = file.Read(firstBytes, 0, 2);
    return bytesRead == 2 && firstBytes[0] == (byte)'M' && firstBytes[1] == (byte)'Z';
}

There are of course some reasonable limitations to this feature. For example, it cannot be combined with out-variables.

if (Crypto32.CryptEncodeObjectEx(
    // other stuff
    out var handle,
    ref size)
)
{
    using (handle)
    {
        // Do stuff
    }
}

This does not work:

if (Crypto32.CryptEncodeObjectEx(
    // other stuff
    out using var handle,
    ref size)
)
{
    // Do stuff
}

Jared Parsons said on Twitter that C# folks thought of this, and decided that it had “Too much confusion about ownership.” Thinking about it myself, I agree, so it’s nice that the feature is limited in that regard.

Another limitation is that the variable cannot be reassigned. For example:

using var stream = entry.Open();
stream = entry2.Open();

This will produce error CS1656, “Cannot assign to ‘stream’ because it is a ‘using variable’”.

All in all, I very much like this small feature in C# 8. It has reasonable guard rails on it from doing something too weird like re-assigning to it, while giving the benefit of less blocks, braces, indentation.

Secure Random Integers in .NET Core 3

Tue, 23 Oct 2018 22:35:00 +0000

.NET Core 3.0 is tentatively set to include a new API for securely generating a random integer bound to a specific range.

I won’t be shy in admitting that it was something I pushed for and made the initial attempt at implementing, though it’s unfair to say that I implemented it by myself given all of the outstanding feedback I got on the initial pull request (thanks Levi and Jeremy!)

It’s been known for a while that System.Random shouldn’t be used when cryptographic randomness is required. Despite that, there wasn’t anything built in to .NET that made creating bounded random integers easy. You could either use System.Random and hope for the best, or use a CSPRNG like RandomNumberGenerator that gave back raw bytes, which requires some thought on how to to properly convert it to a random integer without introducing any kind of bias.

Starting in .NET Core 3.0, you’ll be able to do:

var min = 1;
var max = 1_000;
var randomNumber = RandomNumberGenerator.GetInt32(min, max);

If you need this before .NET Core 3, well, the source is right there. It can be adapted with a bit of effort to work on the .NET Framework as well as other environments that don’t have Span<T>.

FixedTimeEquals in .NET Core

Thu, 04 Oct 2018 22:35:00 +0000

.NET Core introduced FixedTimeEquals which I have personally found to be very helpful. It’s a small method, but given the kind of code I tend to write, I was writing it a lot from project to project, and am happy to see it in the box.

This API is meant to prevent a timing side-channel when comparing two sequences of bytes. The goal is, the comparison should take the same amount of time regardless of the contents of the bytes, assuming they are the same length. This is often required when doing comparisons of MACs or simple possession demonstrations for APIs.

Consider you had a web API that required an API key and was implemented in such a way that it just expected a magic string to act as a password to interact with the API.

It required an HTTP header, like

X-Api-Key: SecretVooDooWord

and TLS was used to ensure it was kept confidential in transit. The API did a naive implementation like this:

[HttpGet]
public IActionResult SuperSensitiveApi() {
    var key = Request.Headers["X-Api-Key"].Single();
    if (key != "SecretVooDooWord") {
        return Unauthorized();
    }
    return new JsonResult(new { AllTheSecretData = "tada" });
}

The issue here is that neither != or == are fixed-time, and will return false as soon as there is an immediate difference. In pseudo code, it might look something like this:

compare(word1, word2) {
    if word1.length != word2.length
        return false
    i = 0
    while (i < word1.length) {
        letter1 = word1[i]
        letter2 = word2[i]
        if letter1 != letter2
            return false
        else
            i = i + 1
    }
    return true
}

This function will compare the letters one by one. As soon as it encounters a difference, it returns false and doesn’t bother checking the rest. Why should it? The answer is going to be false every time.

To an attacker that is trying to figure out the contents of the string, carefully measuring the time can leak the contents of the string. For argument’s sake, let’s say that checking each letter takes 2ns and the attacker has a very accurate way of measuring time over a network and can account for jitter and latency.

GET /SuperSensitiveApi HTTP/1.1
X-Api-Key: Raaaaaaaaaaaaaaa

Checking the API key will take 2ns because the first letters do not match. The attacker moves on to the next letter.

GET /SuperSensitiveApi HTTP/1.1
X-Api-Key: Saaaaaaaaaaaaaaa

This time, checking the API key takes 4ns because the first letter matched (2ns) and the second failed (another 2ns)

GET /SuperSensitiveApi HTTP/1.1
X-Api-Key: Sbaaaaaaaaaaaaaa

Fails again taking 4ns to check the API key. After a few more tries…

GET /SuperSensitiveApi HTTP/1.1
X-Api-Key: Seaaaaaaaaaaaaaa

This time it took 6ns to check the API key because the first and second letter were checked, and the third failed.

The attacker can keep doing this by observing the amount of time each call takes. The longer it takes, the attacker can assume they have guessed the next letter. In practice, this is extremely difficult to do over a network and to get accurate enough timing information, but it is believed that it might be possible given enough persistence and an adversary that has the means to be in a network position that is very stable.

These kinds of attacks do exist, an example timing side-channel is Lucky 13, which affected many library’s approach to handling CBC padding in TLS.

So == in C# is a bad way to check strings for equality where timing side channels need to be mitigated. So you say, perhaps you’ll do something like this:

private static bool CheckStringsFixedTime(string str1, string str2) {
    if (str1.Length != str2.Length) {
        return false;
    }
    var allTheSame = true;
    for (var i = 0; i < str1.Length; i++) {
        if (str1[i] != str2[i]) {
            allTheSame = false;
        }
    }
    return allTheSame;
}

This looks like it’s constant time, but on modern CPUs and .NET, it’s not. Branch statements, like if, have timing implications. We can’t take a branch.

OK, what about this?

private static bool CheckStringsFixedTime(string str1, string str2) {
    if (str1.Length != str2.Length) {
        return false;
    }
    var allTheSame = true;
    for (var i = 0; i < str1.Length; i++) {
        allTheSame &= str1[i] == str2[i];
    }
    return allTheSame;
}

This looks somewhat promising for C#. We know from the language specification that &= does not short circuit, so str1[i] == str2[i] will always be evaluated.

We still have a few problems though, and this is where the JIT and x86 can make things more complicated for us.

The first issue is that allTheSame is a simple true/false flag. That still leaves the .NET JIT and x86 instruction execution to make a few optimizations that introduce timing attacks. Timing side channel mitigation should avoid all decisions until the very end. & is a decision. However, we can improve that some:

private static bool CheckStringsFixedTime(string str1, string str2) {
    if (str1.Length != str2.Length) {
        return false;
    }
    var result = 0;
    for (var i = 0; i < str1.Length; i++) {
        result |= str1[i] ^ str2[i];
    }
    return result == 0;
}

This is fairly close to being time-fixed. We update an integer using | to combine the bits from the result of an XOR. XOR anything with itself, and the the result is zero. Anything else, and you get a non-zero value. We use a binary OR to combine any bits. At the end, if we have a non-zero value, we know one of the comparison checks failed.

There is some debate as to what arithmetic operators are better for fixed time operations between str1[x] and str2[x]. Some implementations use XOR, like above, other may use SUB for subtraction. Unfortunately, most CPU architectures make no guarantee if any operations are constant-time.

We still have a problem to address. The JIT could be making unintended optimizations. The C# compiler itself makes very little optimizations. The JIT on the other hand may do all sorts of things, like unroll a loop or make a variety of optimizations so the code executes faster.

We can tell the JIT to leave it alone, like so.

[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)]
private static bool CheckStringsFixedTime(string str1, string str2) {
    if (str1.Length != str2.Length) {
        return false;
    }
    var result = 0;
    for (var i = 0; i < str1.Length; i++) {
        result |= str1[i] ^ str2[i];
    }
    return result == 0;
}

So now the JIT will cease to optimize this method at all and it will be closer to as-written.

This is close to time-independent as possible with the String type. We can improve things a bit by making the parameters ReadOnlySpan<char> which will generate very similar x86 as string parameters, however the null checks will be eliminated since a ROS cannot be null.

This is what .NET Core 2.1’s FixedTimeEquals does. It just does it over a ReadOnlySpan<byte> instead of characters. The x86 produced by the current JIT will be identical for ReadOnlySpan<char> and ReadOnlySpan<byte> with the exception of the size of the MOVZX.

It’s handy that this is just in the box for .NET Core now.

Playing with RISC-V

Fri, 31 Aug 2018 00:45:00 +0000

Over the past few years I’ve started to sour on x86 architecture for just about everything. From servers, desktop, and mobile, I’ve long wished we had architectures competing with x86 at the high end.

Fortunately, there are plenty of architectures out there. From ARM and ARM64, to MIPS, there are choices. There is one recent one that has been getting my attention lately, and I’ve thought it’s time to start diving in to it.

RISC-V

RISC-V (pronounced “Risk Five”) is a fairly new architecture that has some interesting ideas behind it that make it worth looking it. Originally designed at UC Berkeley, it is an open architecture with the goal of being applicable to a wide range of devices.

An open ISA that is available for commercial use is not unique, but it is rare. Contrast to ARM, if you want to make your own ARM CPU, you need to license the ISA from ARM Holdings Group, to the tune of millions of dollars. While “open” is not a direct benefit to developers, it does mean that a variety of companies that fabricate their own silicon are interested.

RISC-V is also designed for multiple device profiles. It supports 32-bit and 64-bit, and is broken down in to a set of extension instruction sets, plus the always available integer base set. The ISA also documents and reserves a 128-bit architecture.

This “base” integer instruction set is present on any RISC-V implementation. Often called “RV32I” for 32-bit, or “RV64I”, this instruction set consists of 47 instructions required to move memory, perform computations, and arithmetic.

As of writing, some of the designs of the RISC-V architecture are still under active development, while others are finalized. The RV64I and RV32I are complete, while RV128I is still open for changes. The RV64I and RV32I set guarantees 32 registers. For smaller implementations, there is RV32E which limits the register count to 16.

The extensions are are follows

“M” extension instructions offer multiplication and division.
“A” extension instructions offer atomic instructions.
“C” extension instructions are a few compressed for smaller encoding.
“F” extension instructions offer single-precision floating point.
“D” extension instructions offer double-precision floating point.
“Q” extension instructions offer quad-precision floating point.
“L” extension instructions offer decimal floating point.

There are more than that, but these are the basic and complete extensions. A vendor may choose to implement any, or none, of these extensions. Most of these extensions can be emulated with the base Integer instructions using compiler replacements, save for the atomic instruction set.

A key aspect though is that the ISA leaves open guidance and design for allowing a vendor to implement their own extensions and describing how they would be encoded.

The extension set is not entirely meant to be pick and choose whatever. The RV32E base set was designed with only supporting the M, A, C extensions in mind, plus any additional extensions sets implemented on top of it.

Hardware

That all sounds swell. But what about actual hardware? Currently, there is not a lot of hardware out there. SiFive though, makes a board called the HiFive 1 which is a small board with an RV32IMAC capable processor.

Notably, it works with the Arduino Studio, so there is a plenty easy way to get started with that.

I however didn’t have a lot of interest in using it as a board to run code that I already knew, I wanted to learn the instruction set, which meant I wanted to write assembly and throw it at the board and see what it did.

I initially struggled doing this in WSL for some reason, the details of which I haven’t quite fully comprehended yet. After switching to MacOS, things went a lot smoother, since much of the documentation was for Linux and Unix-like systems.

I decided to start with a very simple program. Add two numbers.

.section .text
.globl _start
_start:
    li t1, 42
    li t2, 48
    add t3, t1, t2
    nop

li is “load immediate” which allows putting numbers in to registers immediately. The t prefixed registers are temporary registers. A simple compile might look something like this:

riscv32-unknown-elf-gcc example.S -nostdlib -o example.elf

This will produce a binary that is close to working but has a problem that took me a while to track down. The disassembly looks like this:

example.elf:     file format elf32-littleriscv


Disassembly of section .text:

00010054 <_start>:
   10054:	02a00313          	li	t1,42
   10058:	03000393          	li	t2,48
   1005c:	00730e33          	add	t3,t1,t2
   10060:	0001                	nop

Loading this on the HiFive 1 doesn’t work because it doesn’t load at the correct address. Reading through the HiFive 1 ISA and their documentation, the entry point must be at the address 0x20400000. Fortunately, the HiFive 1 comes with a GCC linker script that does all the right things. It handles the start address as well as other memory layout concerns. After re-compiling with their linker script:

riscv32-unknown-elf-gcc example.S -nostdlib -o example.elf -T link.lds

Re-dumping the assembly shows that _start appears in the correct location. Loading the program can be done with openocd, which is included in the HiFive software kit.

Starting it with the correct configuration:

openocd -f ~/.openocd/config/hifive1.cfg

I moved some files around to make things easier, but they should be easy enough to track down in their software kit.

Once openocd, I can use the included RISC-V GDB debugger to remotely debug it. openocd will start a GDB server on localhost:3333. Firing up the RISC-V GDB, like riscv32-unknown-elf-gdb ~/Projects/riscv/example.elf, I can run the following commands to load the software and start debugging it.

target extended-remote localhost:3333
monitor reset halt
monitor flash protect 0 64 last off
load
layout asm

and if all goes well, we get something like this:

Which is fairly exciting! Debugging my first RISC-V process, even if it does something simple like addition. Performing a few stepi to step a few instructions up to the nop, I can then use info register t3 (or just i r t3 for short) we see we have 90 in the register, which the result of adding 42 and 48.

I’m hopeful that RISC-V will be competitive in the embedded and mobile CPU ISA space. It will take a very long time to get there, and even longer for less homogeneous environment like desktops and laptops, but I believe everyone will be better off if we simply had more to choose from. Even those that will firmly continue to use x86 would benefit from additional innovation and research into architecture design.

I’ll continue to play with RISC-V. If something interesting pops up, I’ll do my best to write about it.

Authenticated Encryption

Sun, 24 Jun 2018 14:42:00 +0000

There’s been some buzz lately about authenticated encryption. The buzz comes from some interesting issues in OpenGPG, and more recently the folks at Microsoft put out an advisory stating that unauthenticated encryption is simply not advisable anymore.

I thought it would be fun to write about cryptography, despite rarely doing it, even though I find it part of what defines me as an engineer and developer.

When we think of “encryption”, we usually think of an entity that has sensitive information they want to protect with a “key” of some kind.

ciphertext = encrypt(plaintext, key)

Something like that. “Decryption” is usually visualized as the process in reverse. The entity has encrypted data that they want to decrypt into its plaintext form because they know the key.

The truth is, well designed systems that rely on cryptography aren’t this simple for a variety of reasons. Further on top of that, software developers struggle to encrypt and decrypt information correctly because many frameworks or libraries that developers depend on offer primitives.

A primitive is a cryptographic function that does very little, but it does its job and it does its job well. That doesn’t mean though that the primitive is enough to fully complete the task. “AES” is a widely known primitive encryption function.

Its most common mode of operation is Cipher Block Chaining, or CBC, which is not authenticated. To put another way, it is malleable. Let’s demonstrate with some ruby code.

require 'openssl'

encrypt_me = "what a fine day for coding" # Data to encrypt
aes_key = (1..16).to_a.pack("C*") # Dummy bad key
aes_iv = (17..32).to_a.pack("C*") # Dummy bad initialization vector
cipher = OpenSSL::Cipher::AES.new(128, :CBC)
cipher.encrypt # Put it in "encrypt" mode, doesn't actually encrypt
cipher.key = aes_key
cipher.iv = aes_iv
ciphertext = cipher.update(encrypt_me) + cipher.final
puts ciphertext.bytes.inspect

Which produces

[15, 90, 144, 183, 105, 160, 17, 219, 160, 166, 20, 201, 53, 30, 2, 29,
217, 115, 3, 249, 2, 170, 203, 32, 37, 234, 147, 188, 167, 254, 254, 192]

There are some bad things in the code example above - it uses a hard-coded, easy to guess key and initialization vector. If you borrow this code, please be wary that it is to demonstrate.

Decryption is a similar process.

aes_key = (1..16).to_a.pack("C*") # Dummy bad key
aes_iv = (17..32).to_a.pack("C*") # Dummy bad initialization vector
cipher = OpenSSL::Cipher::AES.new(128, :CBC)
cipher.decrypt # Put it in "decrypt" mode, doesn't actually decrypt
cipher.key = aes_key
cipher.iv = aes_iv
plaintext = cipher.update(ciphertext) + cipher.final
puts plaintext

Which produces the original string, “what a fine day for coding”.

What if we just… changed the first byte of the cipher text though?

ciphertext[0] = "\1"
# same decryption code as above

That decrypts to “,=m1aH-q8hor coding”. The decryption process didn’t fail in any clear way, it just produced some garbage. We’ve broken the entire “block” of data that we changed a byte in, plus the Nth byte of the next block which was changed. Since we changed the first (0th) byte, the first byte in the second block is “h”, not “f”.

If we slice the data:

plaintext[0..15]  # produces ,=m1aH-q8
plaintext[16..-1] # produces hor coding

If we change the second value of the cipher text:

ciphertext[1] = "\1"

# Decrypt...
plaintext[0..15]  # produces non-ASCII characters
plaintext[16..-1] # produces f4r coding

We see that the “f” is correct for the first byte, but wrong for the second byte.

This is malleable encryption. It allows an attacker to change bits or whole bytes. However the decryption process doesn’t “know” that it is producing invalid data. It’s obvious when you are processing something like text, but it can be less obvious if the data being encrypted is binary data, like a JPEG image.

More interestingly, let’s try changing the last byte of the cipher text:

ciphertext[-1] = "\1"

# Decrypt...

Boom! We get an error, in 'final': bad decrypt (OpenSSL::Cipher::CipherError). Why is that? What we just changed was a padding byte. You might have noticed that when we encrypted our string, the resulting cipher text was bigger than then plain text.

That’s because AES-CBC is a block cipher. It has to operate on chunks of data that are 16 bytes in size. Many implementations will pad the data for you. So when we encrypted “what a fine day for coding”, what actually got encrypted was “what a fine day for coding\6\6\6\6\6\6”.

Where did those \6 bytes come from? That how many bytes it took to reach a multiple of 16. During decryption, it looks at the last byte to determine how much padding to remove.

We can demonstrate this by telling the AES cipher during decryption that there is no padding.

cipher.padding = 0
# Continue decryption...

puts plaintext.bytes.inspect

Which produces:

[119, 104, 97, 116, 32, 97, 32, 102, 105, 110, 101, 32, 100, 97, 121,
32, 102, 111, 114, 32, 99, 111, 100, 105, 110, 103, 6, 6, 6, 6, 6, 6]

So we can see the padding is left there. Six bytes of padding.

Most implementations of AES will automatically apply and remove padding for you.

A poor implementation of AES padding removal will look at the last byte and blindly remove that many bytes:

# Bad padding removal
last_octet = plaintext[-1].ord
unpadded = plaintext[0...-last_octet]

A better implementation of AES padding removal will check that all of the padding bytes are equal to the amount of padding removed.

# Better
last_octet = plaintext[-1].ord
padding = plaintext[-last_octet..-1]
is_padding_valid = padding.bytes.all? { |b| b == last_octet }

# Handle "false" for is_padding_valid
unpadded = plaintext[0...-last_octet]

An even better implementation of AES padding removal would validate the padding in constant time, which is out of my hands with ruby.

It turns out that “false” case for is_padding_valid has caused a lot of problems with AES-CBC, resulting in a padding oracle.

For fun, let’s change the last byte of the first block, and look at the decrypted result and leave the padding on. Remember as we saw previously, changing the Nth byte of the previous block affects the Nth byte of the current block. That’s true for padding as well, it get encrypted like any other data.

ciphertext[15] = "\1"
cipher.padding = 0

#Decrypt...

We get:

[... 110, 103, 6, 6, 6, 6, 6, 26]

Clearly this padding is invalid, because the padding bytes are not all equal to 26. In our home-grown padding removal, is_padding_valid would be false and an error would be returned.

There is one other value that is valid padding, which is 1. If we can change the last byte of the first block so that the last padding byte is one, the padding will appear valid and no error is raised. Let’s use our bad padding removal code and throw an exception if the padding is bad.

def decrypt(data)
    aes_key = (1..16).to_a.pack("C*") # Dummy bad key
    aes_iv = (17..32).to_a.pack("C*") # Dummy bad initialization vector
    cipher = OpenSSL::Cipher::AES.new(128, :CBC)
    cipher.padding = 0
    cipher.decrypt # Put it in "decrypt" mode, doesn't actually decrypt
    cipher.key = aes_key
    cipher.iv = aes_iv
    plaintext = cipher.update(data)
    last_octet = plaintext[-1].ord
    padding = plaintext[-last_octet..-1]
    is_padding_valid = padding.bytes.all? { |b| b == last_octet }
    raise "BAD PADDING" unless is_padding_valid
    return plaintext[0...-last_octet]
end

Our test string is made up of two blocks. We happen to know the padding length is 6, but let’s pretend we don’t. Here is the cipher text. Let’s try and break the last block.

Block 1: [15, 90, 144, 183, 105, 160, 17, 219, 160, 166, 20, 201, 53, 30, 2, 29]
Block 2: [217, 115, 3, 249, 2, 170, 203, 32, 37, 234, 147, 188, 167, 254, 254, 192]

Let’s assume we have a server that we can submit a cipher text to and we have some encrypted data we want to decrypt. The server can accept encrypted data, but it never actually reveals what the plaintext is. The server will either give a padding error back, or say “OK, I was able to decrypt and process the data”.

Remember earlier we said:

We’ve broken the entire “block” of data that we changed a byte in, plus the Nth byte of the next block which was changed.

It goes to reason then, that if we change the last value of the first block, it will affect the last byte of the second block. We are assuming that this encrypted data has padding, but we don’t know how much. Perhaps we can fiddle with the last byte of the penultimate block.

Fortunately, our decryption process makes a nice big fat error when the padding is wrong. The byte of the padding has one or two possible values. The value it is supposed to be (remember, it’s six because we are cheating for now) and one. If it’s one, the padding remover will just remove the last byte. If we just try all combinations for the last byte of the first block, perhaps we can figure out what value makes a padding value of one.

                                                                   Change this
                                                                             ↓
Block 1: [15, 90, 144, 183, 105, 160, 17, 219, 160, 166, 20, 201, 53, 30, 2, 29]
Block 2: [217, 115, 3, 249, 2, 170, 203, 32, 37, 234, 147, 188, 167, 254, 254, 192]
                                                                               ↑
                                                                  To affect this

Let’s loop over it.

(0..255).each do |b|
    # Set last byte of first block
    ciphertext[15] = b.chr
    begin
        decrypt(ciphertext)
        puts b
    rescue
        # The decryption process an error. Skip it and move on.
        next
    end
end

We get two values back. We get 29 and 26. We already know that the value 29 is valid because that’s the actual value from the ciphertext. So 26 forces the padding to one.

Let’s cheat for a moment and verify our findings so that we are comfortable. Given:

ciphertext[15] = 26.chr
decrypt(ciphertext)
return

and if we peek inside the decrypted result with the padding left on, we get:

[95, 9, 61, 149, 138, 173, 150, 56, 255, 200, 46, 73, 45, 145, 185,
77, 102, 111, 114, 32, 99, 111, 100, 105, 110, 103, 6, 6, 6, 6, 6, 1]

The first block is garbage, but crucially we see that we indeed coerce the padding byte to 1.

OK, no more pretending. We know that if we tweak the cipher text’s 15th byte to 26, we end up with a padding of one. What can we learn from that? Let’s take the original ciphertext value, 29, and xor it with our guessed value 26, and then with 1 which is the plaintext value we know it happens to be because the padding was removed successfully.

29 ^ 26 ^ 1 => 6

We were able to figure out the plaintext last byte of the second block. This isn’t “sensitive”, yet, this is the only a padding value. However, we did successfully decrypt a byte without explicit knowledge of the key or the decryption process telling it was 6.

Let’s see if we can figure out how to get the padding to (2, 2).

We can control the last byte of the plaintext now. We can force it to 2 using

ciphertext_value xor known_plaintext_value xor 2

original_ct = ciphertext.dup
ciphertext[15] = (original_ct[15].ord ^ 6 ^ 2).chr
(0..255).each do |b|
    ciphertext[14] = b.chr
    begin
        decrypt(ciphertext)
        puts b
    rescue
        next
    end
end

The result we get back is 6 for this one. If we follow the same formula of

ciphertext_byte xor guess xor result

We get 2 ^ 6 ^ 2, which is 6. So we we have decrypted another byte. Let’s use that to force our padding to three.

ciphertext[15] = (original_ct[15].ord ^ 6 ^ 3).chr
ciphertext[14] = (original_ct[14].ord ^ 6 ^ 3).chr
(0..255).each do |b|
    ciphertext[13] = b.chr
    begin
        decrypt(ciphertext)
        puts b
    rescue
        next
    end
end

The result is 27. 30 ^ 27 ^ 3 is yet again, 6. Which is what we expect since we expect 6 padding bytes with a value of 6.

For the sake of brevity, let’s skip ahead a bit. Let’s see what happens if we trick it in to thinking there are 7 padding bytes.

ciphertext[15] = (original_ct[15].ord ^ 6 ^ 7).chr
ciphertext[14] = (original_ct[14].ord ^ 6 ^ 7).chr
ciphertext[13] = (original_ct[13].ord ^ 6 ^ 7).chr
ciphertext[12] = (original_ct[12].ord ^ 6 ^ 7).chr
ciphertext[11] = (original_ct[11].ord ^ 6 ^ 7).chr
ciphertext[10] = (original_ct[10].ord ^ 6 ^ 7).chr
(0..255).each do |b|
    ciphertext[9] = b.chr
    begin
        decrypt(ciphertext)
        puts b
    rescue
        next
    end
end

The result is 198. 166 ^ 198 ^ 7 is 103. 103 on the ASCII table is “g”. Our plaintext string ends in a “g”. Let’s keep advancing.

ciphertext[10] = (original_ct[10].ord ^ 6 ^ 8).chr
ciphertext[9] = (original_ct[9].ord ^ 103 ^ 8).chr
# etc...

We get 198, and 160 ^ 198 ^ 8 is “n”. If we repeat this pattern, we can fully decrypt the block. Let’s automate this a bit now.

ciphertext.freeze
decrypt_data = ciphertext.dup
recovered = {}
(1..16).each do |i|
    position = 16-i
    (0..255).each do |guess|
        decrypt_data[position] = guess.chr
        begin
            decrypt(decrypt_data)
        rescue
            next # Bad padding.
        end
        recovered[position] = ciphertext[position].ord ^ guess ^ i
        (1..i).each do |j|
            z = 16 - j
            decrypt_data[z] = (ciphertext[z].ord ^ recovered[z] ^ (i+1)).chr
        end
        break
    end
end
pp recovered.sort.map { |k, v| v }.pack("c*")

The final output is:

for coding\x06\x06\x06\x06\x06\x06

The everything put together example of attacking padding is available on GitHub. You can run that in most online ruby REPLs, like repl.it.

The crucial thing about this is that the “decrypt” process never returns the decrypted data, it either raises an exception or returns “Data processed”. Yet we were still able to determine the contents.

A common example of this is encrypted data over insecure transports and home-grown transport protocols. Such an example might be two servers that need to communicate with each other securely. The owners of the servers are able to pre-share an AES key for data transmission, say with a configuration file of sorts, so they assume they don’t need to use HTTPS because they can use AES-CBC to communicate data with their preshared keys. If that encrypted data is intercepted, and the intercepter is able to re-submit that data to one of the servers with padding changed, they are now able to decrypt this data.

Aside: You really should just use TLS with a modern configuration of protocol and cipher suites.

The solution for this is to authenticate our cipher text, which is to say, make it tamper-proof. This is easier said than done.

Let’s clean up symmetric encryption a bit to make it less of an example.

def process_data(iv, data)
    cipher = OpenSSL::Cipher::AES.new(128, :CBC)
    cipher.padding = 0
    cipher.decrypt
    cipher.key = @aes_key
    cipher.iv = iv
    plaintext = cipher.update(data)
    last_octet = plaintext[-1].ord
    raise PaddingError if plaintext.length - last_octet < 0
    padding = plaintext[-last_octet..-1]
    is_padding_valid = padding.bytes.inject(0) { |a, b|
        a | (b ^ last_octet)
    }.zero?
    raise PaddingError unless is_padding_valid

    # Process data
    return true
end

There. We have a function that takes an independent initialization vector and the data, and does a little bit more error handling. It still has flaws mind you, regarding padding removal. There are also some short comings of our attack, such has handling the case where the amount of padding on the ciphertext really is one or decrypting multiple blocks. Both of those are currently left as an exercise.

The usual means of authenticating AES-CBC is with an HMAC using Encrypt-then-MAC (EtM) to ensure the integrity of the ciphertext. Let’s cleanup our code a bit and reduce the amount of assumptions we make. No more hard coded keys and IVs. We’ll go with in-memory values, for now.

The final cleaned up version of this is also available on GitHub. For brevity, here are the function definitions:

def encrypt_data(plaintext); # return is [ciphertext, random_iv]
def process_data(iv, data); #return is "true", or an exception is raised

HMAC is a symmetric signature, or a “keyed hash”. If we sign the contents of the cipher text, then validate the HMAC before attempting to decrypt the data, then we have reasonable assurance that the cipher text has not been changed.

Let’s update the encrypt_data function to include a MAC.

HASH_ALGORITHM = 'sha256'.freeze

def encrypt_data(plaintext)
    new_iv = OpenSSL::Random.random_bytes(16)
    cipher = OpenSSL::Cipher::AES.new(128, :CBC)
    cipher.encrypt
    cipher.key = @aes_key
    cipher.iv = new_iv
    ciphertext = cipher.update(plaintext) + cipher.final
    digest = OpenSSL::Digest.new(HASH_ALGORITHM)
    mac = OpenSSL::HMAC.digest(digest, @hmac_key, ciphertext)
    [mac.freeze, ciphertext.freeze, new_iv.freeze]
end

Our encrypt function now returns the MAC as well. Many uses of EtM simply prepend the MAC to the front of the cipher text. The MAC’s size will be consistent with the underlying algorithm. For example, HMAC-SHA256 will always produce a 256-bit length digest, or 32-bytes. When it comes time to decrypt the data, the HMAC is split off from the cipher text since the length is known.

The process_data can be updated like this:

HASH_ALGORITHM = 'sha256'.freeze

def process_data(iv, mac, data)
    raise MacError if mac.length != 32
    digest = OpenSSL::Digest.new(HASH_ALGORITHM)
    recomputed_mac = OpenSSL::HMAC.digest(digest, @hmac_key, data)
    is_mac_valid = 0
    (0...32).each do |index|
        is_mac_valid |= recomputed_mac[index].ord ^ mac[index].ord
    end
    raise MacError if is_mac_valid != 0

    # Continue normal decryption

Our attack on the padding no longer works. When we attempt to tweak the first block, the HMAC no longer matches the supplied value. As an attacker, we can’t feasibly re-compute the HMAC without the HMAC key, and trying to guess one that works is also not doable.

We have some simple authenticated data applied to AES-CBC now, and defeated our padding oracle attack.

Authenticated encryption is a bare minimum for properly encrypting data, it isn’t “bonus” security. This attack is not new, is have been known at least since Vaudenay described it in 2002. Yet application developers continue to use unauthenticated encryption. Part of that is because people have stated that this oracle attack can be mitigated by not revealing that there was a padding error during decryption and instead being generic about the failure. This was likely incorrect advice - the correct solution is to authenticate the data which mitigates the padding oracle attack quite well.

We’re not quite done yet. In a later post, we’ll look at some of the problems with authenticating data using HMAC or using a mode of AES that is already authenticated like AES-GCM.