AWS Lambdas, Secrets and Asynchronous Updates
Dealing with confidential data in Lambdas is a common theme. Instead of hard-coding data in code, or passing it in as environment variables (that might be visible in the AWS console), confidential data can be stored in AWS SecretsManager and Lambdas can load secrets at runtime, which then leads to the next question: What if the confidential data gets updated? Common approaches are to use explicit time to live, or to handle failures when said confidential data is used (e.g. when it represents credentials that are being used to authenticate at a service and when a failing authentication is then used as a signal to refresh the credentials once. But what if the credentials are genuinely wrong and not just outdated, how to distinguish those use cases then?).
In the following a series of scenarios where Lambdas deal with secrets are presented that culminate in an approach that makes use of a new AWS feature to reload secrets when being notified.
The Setup
All following examples will be using the same basic setup: A Secret that contains a “Prefix” text that will be used to greet the user who invokes the Lambdas. To set up the infrastructure CDK with TypeScript will be used and the Lambdas are written in Rust.
const secret = new Secret(this, 'Secret', {
secretName: 'Prefix',
secretStringValue: SecretValue.unsafePlainText('Hello, ')
});Read
The simplest and naive way to read secrets is once per invocation:
First the Lambda is created
use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::Deserialize;
use std::env;
struct Handler<'a> {
secret_id: String,
client: &'a Client,
}
#[derive(Deserialize, Debug)]
struct ReadRequest {
name: String,
}
impl<'a> Handler<'a> {
async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
let pattern = self
.client
.get_secret_value()
.secret_id(self.secret_id.as_str())
.send()
.await?
.secret_string
.unwrap();
Ok(format!("{}{}", pattern, lambda_event.payload.name))
}
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let secret_id = env::var("SECRET_ID")?;
let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&sdk_config);
let handler = Handler {
secret_id,
client: &client,
};
run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
handler.handle(lambda_event)
}))
.await
}And then the infrastructure is set up
const readSecretLambda = new RustFunction(this, 'ReadSecret', {
entry: 'read_secret',
environment: {
'SECRET_ID': secret.secretArn,
},
functionName: 'ReadSecret',
memorySize: 1024,
logGroup: new LogGroup(this, 'ReadSecretLogGroup', {
logGroupName: '/aws/lambda/ReadSecret',
removalPolicy: RemovalPolicy.DESTROY,
}),
});
secret.grantRead(readSecretLambda);Performance wise this approach isn’t ideal: ~350ms for cold starts and ~30ms for warm starts. Can’t the secret be stored in between invocations?
Lazy Read
Following the idea of the previous step the secret is lazy loaded and stored in between invocations.
Get Johannes Geiger’s stories in your inbox
Join Medium for free to get updates from this writer.
First the Lambda is created
use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use serde::Deserialize;
use std::{env, sync::Mutex};
struct Handler<'a> {
secret_id: String,
client: &'a Client,
secret_value: Mutex<Option<String>>,
}
#[derive(Deserialize, Debug)]
struct ReadRequest {
name: String,
}
impl<'a> Handler<'a> {
async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
match self.secret_value.lock() {
Ok(mut guard) => {
let pattern = match guard.as_ref() {
None => {
let pattern = self
.client
.get_secret_value()
.secret_id(self.secret_id.as_str())
.send()
.await?
.secret_string
.unwrap();
guard.replace(pattern.clone());
println!("Lazy-Loaded the Secret {}", self.secret_id);
pattern
}
Some(pattern) => {
println!("Secret {} already loaded", self.secret_id);
pattern.to_string()
}
};
Ok(format!("{}{}", pattern, lambda_event.payload.name))
}
Err(err) => Err(Error::from(err.to_string())),
}
}
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let secret_id = env::var("SECRET_ID")?;
let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&sdk_config);
let secret_value = Mutex::new(None);
let handler = Handler {
secret_id,
client: &client,
secret_value,
};
run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
handler.handle(lambda_event)
}))
.await
}And then the infrastructure is set up
const readSecretLambdaLazy = new RustFunction(this, 'ReadSecretLazy', {
entry: 'read_secret_lazy',
environment: {
'SECRET_ID': secret.secretArn,
},
functionName: 'ReadSecretLazy',
memorySize: 1024,
logGroup: new LogGroup(this, 'ReadSecretLazyLogGroup', {
logGroupName: '/aws/lambda/ReadSecretLazy',
removalPolicy: RemovalPolicy.DESTROY,
}),
});
secret.grantRead(readSecretLambdaLazy);Performance wise this approach is similar to the previous one for cold starts but manages to achieve 1ms for warm starts.
Preload
Another idea is to load the secret on startup, considering it shouldn’t matter if the secret is lazy loaded or preloaded.
First the Lambda is created
use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::Deserialize;
use std::env;
struct Handler {
secret_value: String,
}
#[derive(Deserialize, Debug)]
struct ReadRequest {
name: String,
}
impl Handler {
async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
Ok(format!(
"{}{}",
self.secret_value, lambda_event.payload.name
))
}
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&sdk_config);
let secret_id = env::var("SECRET_ID")?;
let secret_value = client
.get_secret_value()
.secret_id(secret_id)
.send()
.await?
.secret_string
.unwrap();
let handler = Handler { secret_value };
run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
handler.handle(lambda_event)
}))
.await
}And then the infrastructure is set up
const preloadSecretLambda = new RustFunction(this, 'PreloadSecret', {
entry: 'preload_secret',
environment: {
'SECRET_ID': secret.secretArn,
},
functionName: 'PreloadSecret',
memorySize: 1024,
logGroup: new LogGroup(this, 'PreloadSecretLogGroup', {
logGroupName: '/aws/lambda/PreloadSecret',
removalPolicy: RemovalPolicy.DESTROY,
}),
});
secret.grantRead(preloadSecretLambda);Interestingly the cold starts get reduced to ~300ms but the warm starts are similarly performant as the last solutions.
Preload with Updates
Unfortunately all three previous approaches assume the secret will either never change or that if it changes in between invocations that this coincides with the Lambda environment being shutdown so that the next cold start will pick up the updated secret. For an approach that makes use of the AWS Workload Credentials Provider see https://medium.com/@johannesfloriangeiger/aws-workload-credentials-provider-for-aws-lambdas-hands-on-ae436604b573 for example.
But what if a Lambda could react to a changing secret to reload it independently? Recently AWS introduced a mechanism that so that AWS Secrets Manager now publishes secret update notifications to Amazon EventBridge and so the idea of the last scenario is to use those events to invoke a Lambda, triggering a secret reload.
First the Lambda is created
use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use serde::Deserialize;
use serde_json::Value;
use std::{env, sync::Mutex};
struct Handler<'a> {
secret_id: String,
client: &'a Client,
secret_value: Mutex<Option<String>>,
}
#[derive(Deserialize, Debug)]
struct ReadRequest {
name: String,
}
impl<'a> Handler<'a> {
async fn handle(&self, lambda_event: LambdaEvent<Value>) -> Result<String, Error> {
match lambda_event.payload.get("name") {
None => match lambda_event.payload.get("id") {
None => {
println!("Unknown event");
Ok("".into())
}
Some(_) => match self.secret_value.lock() {
Ok(mut guard) => {
let pattern = self
.client
.get_secret_value()
.secret_id(self.secret_id.as_str())
.send()
.await?
.secret_string
.unwrap();
guard.replace(pattern.clone());
println!("Reloaded the Secret: {}", pattern);
Ok("".into())
}
Err(err) => Err(Error::from(err.to_string())),
},
},
Some(_) => match self.secret_value.lock() {
Ok(mut guard) => {
let pattern = guard.as_mut().unwrap();
let read_request: ReadRequest =
serde_json::from_value(lambda_event.payload.clone())?;
Ok(format!("{}{}", pattern, read_request.name))
}
Err(err) => Err(Error::from(err.to_string())),
},
}
}
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let client = Client::new(&sdk_config);
let secret_id = env::var("SECRET_ID")?;
let secret_value = client
.get_secret_value()
.secret_id(secret_id.clone())
.send()
.await?
.secret_string
.unwrap();
let handler = Handler {
secret_id,
client: &client,
secret_value: Mutex::new(Some(secret_value)),
};
run(service_fn(|lambda_event: LambdaEvent<Value>| {
handler.handle(lambda_event)
}))
.await
}And then the infrastructure is set up
const secretEventBus = new EventBus(this, 'SecretEventBus', {
eventBusName: 'SecretEventBus'
});
const eventBusTarget = new EventBusTarget(secretEventBus);
const defaultEventBus = EventBus.fromEventBusName(this, 'DefaultEventBus', 'default');
new Rule(this, 'DefaultEventBusSecretRule', {
ruleName: 'DefaultEventBusSecretRule',
eventPattern: {
detailType: ['Secret Label Updated'],
source: ['aws.secretsmanager'],
account: [this.account],
detail: {
name: [secret.secretName],
}
},
targets: [eventBusTarget],
eventBus: defaultEventBus,
});
const preloadSecretUpdatesLambda = new RustFunction(this, 'PreloadSecretUpdatesLambda', {
entry: 'preload_secret_updates',
environment: {
'SECRET_ID': secret.secretArn,
},
functionName: 'PreloadSecretUpdatesLambda',
memorySize: 1024,
logGroup: new LogGroup(this, 'PreloadSecretUpdatesLambdaLogGroup', {
logGroupName: '/aws/lambda/PreloadSecretUpdatesLambda',
removalPolicy: RemovalPolicy.DESTROY,
}),
});
secret.grantRead(preloadSecretUpdatesLambda);
const preloadSecretUpdatesLambdaTarget = new LambdaTarget(preloadSecretUpdatesLambda);
new Rule(this, 'PreloadSecretUpdatesLambdaRule', {
ruleName: 'PreloadSecretUpdatesLambdaRule',
eventPattern: {
account: [this.account],
},
targets: [preloadSecretUpdatesLambdaTarget],
eventBus: secretEventBus,
});The setup is a bit more complex than the last one because the Lambda now needs to be able to handle both EventBridge events and manual invocations (for this the Lambda checks if the event contains a name field for a manual invocation or an id field for an EventBridge invocation) and when receiving the EventBridge event it reloads the secret. Infrastructure wise the secret update events are being routed to a new “secret” event bus via an EventBridge rule from the default event bus (for clean separation) and a second EventBridge rule then invokes the Lambda for all events appearing in that “secret” event bus.
The performance is similar to the last scenario, the only difference is that changing the secret value will lead to an invocation that’s slightly higher (~50ms) but results in subsequent invocations that make use of the new secret value and thus avoid stale information.
Summary
Whilst the setup works relatively well and is relatively performant, dealing with multiple types of events for the Lambda in the last case is not very elegant and mixes different concerns in the same code. Ideally Lambdas would have a separate channel for notifications (perhaps also a potential improvement for the AWS Workload Credentials Provider?) that would allow for the secrets to be reloaded in the background.
