Zero Knowledge System Using Libsodium For Client Side Encryption
Well, Recently while I was testing my cloud file storage platform, I found out that all the files uploaded, no matter who the user is all the files was stored in the database open (which means the admin can view all the photos and its metadata) and actually it is a security problem since files includes public and personal ones.
while I looked onto what can be a solution for this problem I found out some platforms (actually competitors for my file storage system 😅) used the client side encryption strategy and that opened the way into Zero Knowledge System. This was the first time I got introduced into this kind of architecture where only clients can view and access there original content where as in the database only some useless blogs are stored.
Libsodium is the industrial standerd for implementing this solution since it is powerfull, easy to use with providing various implementation techniques for cryptography, including encryption, decryption, digital signatures, and password hashing.
FileBox is build upon the NextJS for the frontend with typescript, so I had to implement libsodium in JS/TS. Thanks to Javascript ecosystem that we have npm packages for libsodium since the original libsodium is completely written in C. libsodium-wrappers-sumo is the javascript wrapper over the libsodium which provides all the functions and opertaions of native libsodium, where us we also have libsodium-wrappers which is the standerd package with light weight bundle containg only high frequency and commonly used cryptographic functions.
Since the package utilizes WebAssembly (wasm) for running libsodium, we have to start it asynchroniously. We all know that libsodium is written in highly optimized C, wasm provides the fastest, most secure way to run that exact same C code in a JavaScript environment because if wasm is not used then, js engine will struggle hard for executing low-level bitwise operations and affects the performance.
// frontend/lib/sodium.ts
import _sodium from 'libsodium-wrappers-sumo';
// this promise is used to provide the globally initialized sodium instance to any part where the program is called.
// as a result each time creating new promize when each pairt of the function calls the getSodium is solved
let sodiumReadyPromise: Promise<typeof _sodium> | null = null;
export async function getSodium() {
// loading the sodium asynchronously to ensure that the Wasm is ready before using it
if (!sodiumReadyPromise) {
sodiumReadyPromise = (async () => {
await _sodium.ready;
return _sodium;
})();
}
return sodiumReadyPromise;
}in the lib folder we are initializing the sodium.ts file, which starts the libsodium engine powered by webAssembly asynchroniously. For ensuring that the engine is ready and we can use the crypto funtionalies we have to await sodium.ready method, thorugh which we can ensure that libsodium is ready before calling or executing required fucntions. sodiumReadyPromise is used as a cache here to save the sodium which gets started on the inital boot up of the website, so that functions which we will call later can use this sodium and avoid re initializing the sodium each time when each fucntions on each part of the system calls it.
Once the core libsodium engine is setup, next comes the phase of integrating the sodium with our workflow, as mentioned in our case we wanted to encrypt all the data uploaded by the user in client side itself and will be decrypted only using the client keys. The first step comes here is of defining the key hirarcy.
In this ZKS, all the user will have a master key that will be generated once during there registration or sign-up. As the name suggests this key stays on the top of every thing such that, this key has influence on all the ecryptions and decryptions that will happen directly or indirectly. But the problem is that how will be keep this key persist, such that if a user logs in another device we need to provide the same key to user and that makes conclusion that master key should be stored but not in plain format that which breaks our entire Zero Knowledge System, inplace we encrypt this master key and store the encrypted key in database. So the next question that comes will be like with what the master key gets encrypted and login comes as a solution.
When the user registers with his email and password using the password, we will encrypt the generated master key and store that encrypted one in the database, so whenever the user tries to login through any devices, as they enter the password using the password the encrypted master key will be decrypted and stored in the client for that active session, another fact is that we if the password wrong decryption fails and user can log in since no master key is decryptes which gives the security to the system that only verified people (those who knows the right password) gets in.
Generating the Master Key
As of the first step, for generating the unique master key for each user, libsodum provides as with crypto_secretbox_keygen that which generates 256 bits (32 bits) encryption key for symmertic authentication. The 256 bit key is the most secure form as of the current crypto technologies.
export async function generateMasterKey(): Promise<Uint8Array | null> {
try {
const sodium = await getSodium();
return sodium.crypto_secretbox_keygen(); // Returns a Uint8Array (32 bytes)
} catch (error) {
console.error("Failed to generate master key:", error);
return null;
}
}
Key Derivation Key (used to encrypt the master key)
Get Ziyaudheenms’s stories in your inbox
Join Medium for free to get updates from this writer.
For the provided implementation of using the password for encrypting master key, we can implement that using the crypto_pwhash that which using a salt generated, creates the derivation key which will further used to encrypt the master key, note that password directky is not and should not be used for the encryption process. This function depends on 3 parameters which that helps with the performance and the security of the derivation key that which generates. crypto_pwhash_OPSLIMIT_ crypto_pwhash_MEMLIMIT_ crypto_pwhash_ALG_
export async function deriveKeyEncryptionKey(password: string, salt: Uint8Array): Promise<Uint8Array | null> {
try {
const sodium = await getSodium()
return sodium.crypto_pwhash(
32, // the length of the derived key in bytes
sodium.from_string(password), // the password to be hashed is converted to a Uint8Array using the from_string function from the libsodium library.
salt, // the salt to be used for hashing
// _INTERACTVE IS USED FOR THE SMOOTH AND SNAPPY PERFORMANCE ON THE CLIENT BROWSER SIDE
sodium.crypto_pwhash_OPSLIMIT_INTERACTIVE, // the number of operations to perform during hashing
sodium.crypto_pwhash_MEMLIMIT_INTERACTIVE, // the amount of memory to use during hashing
sodium.crypto_pwhash_ALG_DEFAULT // the algorithm to use for hashing
);
} catch (error) {
console.error("Failed to derive key encryption key:", error);
return null;
}
}by using these KDK, we can encrypt the master key and stores the encrypted one in the database. As of now the core worklow is like login -> user password -> KDK -> decrypting the encrypted master key. For the encryption and decrytion of keys libsodium provides as crypto_secretbox_easy and crypto_secretbox_open_easy respectively.For the encryption we have to create a salt, using const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES); we can generate the salt required for encryption and we have to provide the same during the decryption face too. (salts are public and they are meant to be stored in the db in the same form).
export async function encryptMasterKey(masterKey: Uint8Array, kek: Uint8Array): Promise<{ nonce: Uint8Array; encryptedMasterKey: Uint8Array } | null> {
try {
const sodium = await getSodium();
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES); // generating a random nonce for encryption using the libsodium library. .crypto_secretbox_NONCEBYTES is a constant that defines the length of the nonce in bytes.
const encryptedMasterKey = sodium.crypto_secretbox_easy(masterKey, nonce, kek);
return { nonce, encryptedMasterKey };
} catch (error) {
console.error("Failed to encrypt master key:", error);
return null;
}
}
// During the login we have to decrypt the master key using the KEK generated from the password and the nonce stored in the DB
export async function decryptMasterKey(encryptedMasterKey: Uint8Array, nonce: Uint8Array, kek: Uint8Array): Promise<Uint8Array | null> {
try {
const sodium = await getSodium();
return sodium.crypto_secretbox_open_easy(encryptedMasterKey, nonce, kek);
} catch (error) {
console.error("Failed to decrypt master key:", error);
return null;
}
}Once we have the master key, next comes directly to the core login of the application, that we want to implement the encryption with files, so for encrypting the files we need to create a hierarchy or sequential order of keys that comes in play to encrypt the files and see how the master key connects indirectly with the file encryption.
for each files that which we want to encrypt it should have a collection key (like having a parent folder), if the file doesn't have a collection key, it has to be generated. Once that gets sorted, ultimately for encrypting each file we have a file key using which the stream bytes (visual) and the file metadata are encrypted. And looking the chain :-
- Load the master key (it will be decrypted once during the login and it should be persisted across the active session)
- From the backend response, the collection key is decrypted using the master key
- Using the collection key the file key is decrypted
- With the file key ultimate stream content and the metadata is decrypted and loaded in the UI!
All the mentioned keys are generated with the libsodium function
crypto_secretbox_keygen
Encryption of file content
Now comes the main part encryption of the actual file content and it has many challenges too. When I was off encrypting the file content thought just like others (encryptions done before) we can just pass the complete content into the function, but that introduced the new problem memory. what if we have a huge file and we load it all once for encryption, it can lead to crashing of the browser with out of memory, there stream encryption came into action and libsodium provides as with a powerful function that encrypts the streams of the file crypto_secretstream and also handles:-
- the sequence in which the file was broken and passed as stream
- the very first stream and the very last stream
handling this two is very important such that, if any tampering has happened to the no of streams (length of total streams) or like if the initial stream is missing or very last final stream is missing, libsodium will flag red at that time itself.
export async function encryptFileContent(
file: File,
encryptionKey: Uint8Array
): Promise<{ encryptedStream: ReadableStream<Uint8Array>, header: Uint8Array } | null> {
// here for the file content encryption we are using the stream encryption techniqu where we encrypt the stream of data instead of treating it in the goooo.
try {
const sodium = await getSodium()
// first we have to check the length of the key that is used for the encryption if it supports the stream Encryption or not
if (encryptionKey.length != sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES) {
console.log(`we require a size of ${sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES} bytes but got a size of ${encryptionKey.length}bytes and its invalid!!`)
return null
}
// next we have to generate the core thing for the entire encryption and decryption and that is the header and states
const stateAndHeader = sodium.crypto_secretstream_xchacha20poly1305_init_push(encryptionKey) //state and header based on the given encryptionKey will be generated.
// header --> header is used for intializing the nonce and the salt for the entire file encryption through streams and it is reuired for the decryption engine too, header contains the initialization metadata and salt for the stream.
// state --> mean while state is used for the security purpose of the encrypion , state is used to identify the order in which the streams are arragned, which means it prevents the attacks like changing the order oof streams or manuplating the total no of streams
const { header, state } = stateAndHeader
const CHUNK_SIZE = 3 * 1024 * 1024; // 3 MB chunks
let offset = 0 // this variable is used to track the no of bytes / size that which is being encrypted.
let emptyFileChunkSent = false
const encryptedStream = new ReadableStream<Uint8Array>({
async pull(controller) {
// controller is the JS object that is used to controll the process of enqueing , catching the stream error or closing the stream
try {
// since we are migrated to the pull method, no need for while loop and for ending the stream we are checking if the offset is greater than filesize or not
if (file.size === 0 && !emptyFileChunkSent) {
const encryptedBlock = sodium.crypto_secretstream_xchacha20poly1305_push(
state,
new Uint8Array(),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL
);
emptyFileChunkSent = true;
controller.enqueue(encryptedBlock);
return;
}
if (offset >= file.size) { // if satisfies means the code completed the encryption of all the chunks of the file.
controller.close();
return;
}
const isLastChunk = (offset + CHUNK_SIZE) >= file.size // tracking whether its the last.
//now we have to cut the chunks into the parts as we need
const blobsize = file.slice(offset, offset + CHUNK_SIZE)
// Next we have to load this file into binary and then load it into array buffer of Uint8
const arrayBuffer = await blobsize.arrayBuffer()
const Uint8chunk = new Uint8Array(arrayBuffer)
// Now for true utilization of state, we have to assign tags with each stream with a special tag for the last stream so that the encryption can understand where does the stream ends and can prevent ffrom the truncation attacks.
const tag = isLastChunk
? sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL // TAG_FINAL hepls is identifying the last chunk so to determine how many chunks are there.
: sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE;
// Now using all these data we have to encrypt each streams
const encryptedBlock = sodium.crypto_secretstream_xchacha20poly1305_push(
state,
Uint8chunk,
null, // Additional optional unencrypted data (AD)
tag
);
// instead of stroing this in the encrypted array, we will stream this to the responce or the request whre the data is to be send
controller.enqueue(encryptedBlock)
offset += CHUNK_SIZE
}
catch (streamError) {
console.log('some error occured with the stream')
controller.error(streamError)
}
}
})
return {
encryptedStream,
header // Returning the raw Uint8Array header as well
};
}
catch (e) {
console.log('some kind of error occureed')
console.error(e)
return null
}
}since we have to handle the stream by stream upload of the data , we here using the ReadableStream of JS which helps in allowing us to process large or data chunks incrementally without loading the entire payload into memory. As a result we can reserve only a fixed memory spot and accepts the stream of chunks contineuosly and perform our operations. const stateAndHeader = sodium.crypto_secretstream_xchacha20poly1305_init_push(encryptionKey) state and header are cruitial initail steps for this stream encryption which sets the order of the streams being encrypted and the salt or nonce for entire streams respectively. sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL is assigned with the last stream to identify that the stream has end, if the stream of chunk is not the final one it is embedded with the sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE which is used to identify wheter the stream belongs to that particular entire object or not.
the encrypted stream of chunks also goes thorugh a readable stream so that it doesnt takes up the extra space
the encrypted chunks coming out as a stream is mapped to a backend from where it gets merged together and stored in the storage bucket.
Decryption of file content
just like the encryption decryption also is a complex procedure and it too have the issues that we would face in the encryption phase. libsodium provides a powerfull function called crypto_secretstream_ . It checks the state, header initially as that are the key initial content to be passed with the decryption function.
export async function decryptFileContentStream(
encryptedStream: ReadableStream<Uint8Array>,
header: Uint8Array,
encryptionKey: Uint8Array
): Promise<ReadableStream<Uint8Array> | null> {
try {
const sodium = await getSodium();
// 1. Validate key and header lengths
if (encryptionKey.length !== sodium.crypto_secretstream_xchacha20poly1305_KEYBYTES) {
console.error(`Invalid decryption key size.`);
return null;
}
if (header.length !== sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES) {
console.error(`Invalid header size.`);
return null;
}
// 2. Initialize the pull state
const state = sodium.crypto_secretstream_xchacha20poly1305_init_pull(header, encryptionKey);
const reader = encryptedStream.getReader();
const CHUNK_SIZE = 3 * 1024 * 1024;
const CIPHER_CHUNK_SIZE =
CHUNK_SIZE + sodium.crypto_secretstream_xchacha20poly1305_ABYTES;
const cipherBuffer = new Uint8Array(CIPHER_CHUNK_SIZE); // pre-defing the buffer which that is used with the size of the chunk
let bufferedBytes = 0; //is used to track how many data bytes are read into the buffer.
let sourceChunk: Uint8Array | undefined; // is used to store the remaining bytes for the next cycle
let sourceOffset = 0;
let streamEnded = false;
let completed = false;
return new ReadableStream<Uint8Array>({
async pull(controller) {
try {
if (completed) {
return;
}
while (bufferedBytes < CIPHER_CHUNK_SIZE && !streamEnded) {
if (sourceChunk && sourceOffset < sourceChunk.length) {
const bytesToCopy = Math.min(
CIPHER_CHUNK_SIZE - bufferedBytes,
sourceChunk.length - sourceOffset
);
cipherBuffer.set(
sourceChunk.subarray(sourceOffset, sourceOffset + bytesToCopy),
bufferedBytes
);
bufferedBytes += bytesToCopy;
sourceOffset += bytesToCopy;
continue;
}
// making the sourceChunk and its tracker clean for next processess
sourceChunk = undefined;
sourceOffset = 0;
const next = await reader.read();
if (next.done) { //done gives true if the .read() completes reading the entire file
streamEnded = true;
} else if (next.value.length > 0) {
sourceChunk = next.value;
}
}
if (bufferedBytes === 0) {
console.error("Decryption failed: Stream ended before TAG_FINAL.");
return
}
const blockLength = bufferedBytes;
if (blockLength < sodium.crypto_secretstream_xchacha20poly1305_ABYTES) {
console.error("Decryption failed: Incomplete ciphertext record.");
}
const result = sodium.crypto_secretstream_xchacha20poly1305_pull(
state,
cipherBuffer.subarray(0, blockLength),
null
);
const { message, tag } = result;
const isFinal =
tag === sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL;
if (
tag !== sodium.crypto_secretstream_xchacha20poly1305_TAG_MESSAGE &&
!isFinal
) {
console.error("Decryption failed: Unexpected secretstream tag.");
return
}
if (isFinal) {
if (sourceChunk && sourceOffset < sourceChunk.length) {
console.error("Security violation: Trailing data after final file block.");
return
}
while (!streamEnded) {
const next = await reader.read();
if (next.done) {
streamEnded = true;
} else if (next.value.length > 0) {
console.error("Security violation: Trailing data after final file block.");
return
}
}
} else if (streamEnded) {
console.error("Decryption failed: Stream ended without TAG_FINAL.");
return
}
bufferedBytes = 0;
if (isFinal) {
completed = true;
controller.enqueue(message);
controller.close();
reader.releaseLock();
return;
}
controller.enqueue(message);
} catch (err) {
controller.error(err);
try {
await reader.cancel(err);
} catch (cancelError) {
console.error("Failed to cancel encrypted input stream:", cancelError);
}
}
},
cancel(reason) {
return reader.cancel(reason);
}
});
} catch (e) {
console.error("Failed to initialize decryption stream:", e);
return null;
}
}just like encryption function, decryption too accepts the encrypted as streams, using the api it validates them, here with the streams it can come of any size so checking it with like if it is off the required size takes them, if it contains the extra it is reserved for the next stream slot and it goes on till the last byte so that we can ensure that all the chunks will only be of designed chunk size. Also the encrypted chunks is checked with whether it contains the required and accurate tags to ensure nothing wrong has happened. Once all this checks gets completed the file contents get decrypted and streamed back in the form of chunks through Readable stream.
And when comes to the case of the metadata all except the size is encrypted inlcuding the file name, extension etc. All the required metadata are packed into an json file and is encrypted with the file key using the libsodium funtion crypto_secretbox_easy .
export async function encryptFileMetadata(
encryptionKey: Uint8Array,
metadata: {
name: string,
fileType: string,
description: string,
}
): Promise<{ encryptedFileMetadata: Uint8Array; nonce: Uint8Array } | null> {
try {
const sodium = await getSodium();
// we have to convert the object into the JSON string
const jsonString = JSON.stringify(metadata);
// the JSON converted string is converted into Uint8 array.
const uint8MetaData = new TextEncoder().encode(jsonString);
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encryptedFileMetadata = sodium.crypto_secretbox_easy(uint8MetaData, nonce, encryptionKey);
return { encryptedFileMetadata, nonce };
} catch (error) {
console.error("Failed to encrypt file metadata:", error);
return null;
}
}the json metadata is converted into a stringified one and is converted into uint8 array format. that uint8 array is uploaded into the given function along with the nonce (salt) and the encryption key which is the file key. In the case of decryption the encrypted metadata is passed along with the nonce and encryption key which then gets decrypted with the help of crypto_secretbox_open_easy function and converted into json format and is returned back
// DECRYPTION OF THE FILE METADATA
export async function decryptFileMetadata(
encryptedMetaData: string,
nonce: string,
encryptionKey: Uint8Array
): Promise<{ name: string; fileType: string; description: string } | null> {
try {
const sodium = await getSodium();
// the encrypted metadata is converted into the Uint8 array and then it is encrypted with the file encryption key
const encryptedMetadataBytes = sodium.from_base64(encryptedMetaData);
const nonceBytes = sodium.from_base64(nonce);
const decryptedMetadata = sodium.crypto_secretbox_open_easy(
encryptedMetadataBytes,
nonceBytes,
encryptionKey
);
const metadata: unknown = JSON.parse(new TextDecoder().decode(decryptedMetadata));
if (
typeof metadata !== "object" ||
metadata === null ||
!("name" in metadata) ||
!("fileType" in metadata) ||
!("description" in metadata) ||
typeof metadata.name !== "string" ||
typeof metadata.fileType !== "string" ||
typeof metadata.description !== "string"
) {
console.error("Decrypted file metadata has an invalid format");
return null;
}
return {
name: metadata.name,
fileType: metadata.fileType,
description: metadata.description
};
} catch (error) {
console.error("Failed to decrypt file metadata:", error);
return null;
}
}The thing to note is that all the direct key interaction with the libsodium should be in the uint8 Array format . libsodium provides as with the required helper functions to convert the uint8 into bs64 and also into hex .
The next bigger problem with this client side encryption is persistence of the keys all over the active session, like for each operation we cant prompt for password again and again for decrypting the master key it affects the UX. Also for that we cant store the raw master key in the local storage and session storage blankly also this is a big engineering concept which we have to discuss.
