How To

Key Rotation Best Practices in Hadoop

As discussed in our Encryption Key Management Best Practices post, implementing an effective encryption regime requires diligence and thoughtfulness as it relates to proper key lifecycle management. A key aspect of this is key rotation, which can help keep intruders out and improve the protection of sensitive business information.

Key rotation is simply the process of decrypting data with an old key and re-encrypting it with a new key. There are many reasons for performing regular key rotation, including the departure of people with key access from an organization, the compromise of a master key, internal security policy requirements, or regulatory and compliance mandates.

Key rotation can impose considerable costs in time and effort on IT managers and key management tools are frequently used to facilitate and simplify the process. At least, that's the case for legacy data storage infrastructure. When it comes to modern, unstructured data repositories the ecosystem is much less developed. As a result, IT managers are forced to fallback to manual, homegrown key rotation processes.

Key Rotation in Hadoop

For Hadoop, which lacks an in-built key rotation mechanism or adequate third-party tools, key rotation means shutting down an encryption zone, copying the data over and shuttling it through a hardware security module (HSM) where it is decrypted and then re-encrypted with a new key.

This process can take hours, resulting in downtimes of up to a day and imposing a significant cost to the business. With existing tools, businesses simply have to bear this cost and, in practice, many enterprises many enterprises have opted out of storing sensitive data in their data lakes to avoid this downtime altogether. Either way, this imposes a significant cost to the business.

Solved, with NuCypher Hadoop

Among other things, NuCypher Hadoop performs cryptographically secure key rotation in seconds, as opposed to hours. By employing the entire cluster to rotate keys, it solves the downtime issue and enables IT managers to perform key rotation at will for their Hadoop data lakes. NuCypher Hadoop doesn't require shuttling data in and out of an HSM yet it is still secure and doesn't expose the encryption key to the cluster.

DashR vs. Manual Key Rotation Performance

As the graph above shows, this is a massive performance improvement and, because NuCypher Hadoop employs the cluster to perform key rotation the performance increases with the size of the cluster.

Moreover, because key rotation in NuCypher Hadoop doesn’t require shuttling data in and out of an HSM, this means the performance is much less sensitive to latency between the HSM and the cluster. It doesn’t matter how far apart the two are. For example, with NuCypher Hadoop, enterprise can employ an on-premise HSM with a cloud-based Hadoop cluster and perform key rotation easily, as shown in the graph below.

DashR vs. Manual Key Rotation Performance