Ever since AWS announced AWS Lambda MicroVMs I was thinking of use cases where Serverless stateful Docker based virtual machines can demonstrate a use case that goes past a very basic and artificial “Hello World” where we run an application that proves how subsequent invocations keep local state. Inspired by performance problems with build pipelines from current and previous real world projects, a recurring issue that requires resources on demand and benefits from keeping state in between invocations is a Build Server that both allows users to spin up instances on demand (and to scale those instances down when not needed, e.g. over night) and that the same instances will keep build artefacts such as dependencies between runs to optimise runtime whilst at the same time don’t require users to handle the case of a brand new instance spun up as the build tool will take care of transparently downloading dependencies if they aren’t available at the time of build.

The setup will be that we will be storing two “Hello World” applications (one written in Golang to test cross-compilation and one written in Java built with Maven to test the caching of dependencies on the MicroVM between runs) in S3, exposing it via S3 Files to a AWS Lambda MicroVM that on invoke will build either application on the MicroVM and writing the resulting artefacts back onto S3.

Setup

Bucket and Filesystem

For this I am reusing the Stack defined in my article on S3 Files and AWS Lambda. So that afterwards we will have a new, empty Bucket with a Filesystem, Mount Targets and Security Group.

If we would now deploy an EC2 instance into one of the new Subnets we could mount the Filesystem and access it to prove the setup works so far. In case you do that, don’t forget to allow your EC2 instance’s Security Group access to the Mount Target’s Security Group.

MicroVM Image

I am generally using default values for the new MicroVM image, only to ensure the “Enable all operating system capabilities” flag is active. The ZIP archive we are using as the basis for the MicroVM image consists of five files: Dockerfile, three shell scripts (to mount S3 Files, build Go and build Java) and the application itself that will be running on the MicroVM. I’ve added all files in an appendix below, create a new archive containing those five files, upload it to an S3 Bucket and point the MicroVM’s “S3 destination” to it.

Create the new MicroVM image and check the logs for potential errors, the build might take several minutes. Afterwards, the state should be “Successful”.

Demo

Before we can run the MicroVM itself we both need a new IAM role and a File Connector allow the MicroVM to access S3 Files.

The IAM role needs to be manually created in the IAM section and have the relevant permissions to access S3 (I gave the role temporarily AdministratorAccess to avoid too much time spent on this) and to trust the Lambda service

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

The File Connector can be created from the MicroVM creation screen, sitting in the same Subnets as we used for the File System above and its Security Group needs to be able to access the File System’s Security Group. The remaining settings can be left at defaults.

Now we can run the MicroVM with the image we built in the last step, using the role we just created and making sure the “Egress network interface” is set to the File Connector we just created. The MicroVM should come up almost immediately.

To demo, we need to copy the example projects into the Bucket, I chose to put both into separate ‘folders’, so that the Bucket has a ‘go’ and a ‘java’ root folder now, containing the respective example. I’ve added the example files I used in the appendix below again.

Finally, we can now invoke the MicroVM, in my case I did that manually from my local machine with JavaScript (replace the endpoint with the correct one and obtain a valid token via the AWS Console for the MicroVM, replace the File System ID with the correct one from your setup):

const endpoint = '<ENDPOINT>';
const token = '<TOKEN>';
const type = 'java'
const projectPath = './java';
const mountPath = '/mnt/app';
const fileSystemId = '<FILE_SYSTEM_ID>';
const response = await fetch(`https://${endpoint}`, {
    method: 'POST',
    headers: {
        'X-aws-proxy-auth': token,
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        type,
        projectPath,
        mountPath,
        fileSystemId,
    })
});
console.log(await response.json());

This builds the example Java application under the /java ‘folder’ and will place the resulting Application-1.0-SNAPSHOT.jar in the Bucket under the /target ‘folder’. Download it and run it with java -jar Application-1.0-SNAPSHOT.jar and you should see [main] INFO org.example.Main — Hello World! . Note that the response from the MicroVM contains a lot of ‘Maven dependency download’ noise that on subsequent runs is absent as the Maven repository on the MicroVM is kept between runs. The same build can be invoked for the Go application as well, replacing the type with go and specifying the right location in the Bucket.

Summary

Giving a MicroVM access to S3 works well but can be fiddly at times and more convenience from AWS side’ would be very helpful. The MicroVM performance is great and the ability to re-use MicroVM to optimise builds by re-using cached information I could imagine opens up a lot of possibilities for use cases where intermediate artefacts are heavily relied upon, such as building Java application with Maven where a lot of dependencies are needed.

Appendix

MicroVM

Dockerfile

FROM public.ecr.aws/amazonlinux/amazonlinux:2023

RUN dnf -y install amazon-efs-utils nfs-utils nodejs golang java-21-amazon-corretto maven \
    && dnf clean all \
    && rm -rf /var/cache/dnf

RUN mkdir /mnt/app

WORKDIR /app
COPY --chmod=+x mount-s3.sh .
COPY --chmod=+x build-go.sh .
COPY --chmod=+x build-java.sh .
COPY app.js .

EXPOSE 8080

CMD ["node", "app.js"]

mount-s3.sh

MOUNT_PATH=$1
FILE_SYSTEM_ID=$2
echo "Mounting file system '$FILE_SYSTEM_ID' under '$MOUNT_PATH'..."
mount -t s3files "$FILE_SYSTEM_ID" "$MOUNT_PATH"
echo "File system mounted."

build-go.sh

As I am writing this on a Mac but we’re building on Linux I have added the cross-compile flags GOOS=darwin GOARCH=amd64 so that I can execute the resulting application locally.

MOUNT_PATH=$1
PROJECT_PATH=$2
echo "Building application under mount path '$MOUNT_PATH' and project path '$PROJECT_PATH' with Go..."
cd "$MOUNT_PATH" || exit
cd "$PROJECT_PATH" || exit
env GOOS=darwin GOARCH=amd64 go build
echo "Application built."

build-java.sh

MOUNT_PATH=$1
PROJECT_PATH=$2
echo "Building application under mount path '$MOUNT_PATH' and project path '$PROJECT_PATH' with Maven..."
cd "$MOUNT_PATH" || exit
cd "$PROJECT_PATH" || exit
mvn clean install
echo "Application built."

app.js

const http = require('http');
const {exec} = require('child_process');

const TYPE_GO = 'go';
const TYPE_JAVA = 'java';

const populateResponse = (response, status, content, stdout, stderr) => {
    response.writeHead(status, {
        'Content-Type': 'application/json'
    });

    response.end(JSON.stringify({
        content,
        stdout,
        stderr,
    }));
}

const server = http.createServer((req, res) => {
    console.log('Request received');

    let body = '';
    req.on('data', (chunk) => {
        body += chunk;
    });
    req.on('end', () => {
        const content = JSON.parse(body);
        const {type, mountPath, fileSystemId, projectPath} = content;

        if (type === TYPE_GO) {
            exec(`./mount-s3.sh ${mountPath} ${fileSystemId} && ./build-go.sh ${mountPath} ${projectPath}`, (err, stdout, stderr) => {
                if (err) {
                    populateResponse(res, 500, content, stdout, stderr);
                } else {
                    populateResponse(res, 200, content, stdout, stderr);
                }
            });
        } else if (type === TYPE_JAVA) {
            exec(`./mount-s3.sh ${mountPath} ${fileSystemId} && ./build-java.sh ${mountPath} ${projectPath}`, (err, stdout, stderr) => {
                if (err) {
                    populateResponse(res, 500, content, stdout, stderr);
                } else {
                    populateResponse(res, 200, content, stdout, stderr);
                }
            });
        }
    });
});

server.listen(8080, () => {
    console.log('Listening on port 8080');
});

Example Project

main.go

package main

import "fmt"

func main() {
    fmt.Println("Hello", "World!")
}

go.mod

module github.com/johannesfloriangeiger/go

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>Application</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>2.0.20</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>2.0.20</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.6.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>org.example.Main</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

src/main/java/org/example/Main.java

package org.example;

import org.slf4j.LoggerFactory;

public class Main {

    public static void main(final String[] args) {
        LoggerFactory.getLogger(Main.class)
                .info("Hello World!");
    }
}

In addition to ‘classic’ Lambdas which AWS have now rebranded Event Functions, AWS now also allows to host Web Functions which (for now) run Serverless Node.js web servers that are now also supported as Origins of CloudFront which makes it possible to run websites consisting of static HTML and dynamic API purely based on Web Functions, distributed by CloudFront and using a custom domain name, with a minimal infrastrucutre setup. In the following we will be looking at a step by step process to deploy an example application, using a custom domain name managed in Route53, with Web Functions and CloudFront only.

Stage 1: Lambda

The application supports two routes: A static HTML page that allows invoking an API, and said “dynamic” API that returns a text via GET (that otherwise would probably be read from a data source such as DynamoDB but to keep the example simple returns a static text for now). There are three files necessary for this:

First the package.json:

{
  "name": "lambda-web-function",
  "version": "1.0.0",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "dev": "node index.js"
  }
}

then the index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Lambda Web Function</title>
</head>
<body>
<div>
    <button id="greet"
            onclick="fetch('/greet').then(response => response.text()).then(greeting => document.getElementById('greeting').innerText = greeting)">
        Press to Greet
    </button>
    <p id="greeting"></p>
</div>
</body>
</html>

and finally the index.js:

import {createServer} from 'http';
import {readFileSync} from 'fs';

const server = createServer((req, res) => {
    if (req.url === '/greet') {
        res.writeHead(200, {'Content-Type': 'text/plain'});
        res.end('Hello Lambda Web Function!');
    } else {
        try {
            const html = readFileSync('index.html', 'utf8');
            res.writeHead(200, {'Content-Type': 'text/html'});
            res.end(html);
        } catch (error) {
            res.writeHead(404);
            res.end('Not Found');
        }
    }
});

const [host, port] = process.env.AWS_LAMBDA_HTTP_ENDPOINT.split(':');

server.listen(port, host, () => {
    console.log(`Server running on ${host}:${port}`);
});

to test this locally, simply run AWS_LAMBDA_HTTP_ENDPOINT=localhost:3000 npm run dev, open http://localhost:3000 in a Browser and see that the page loads and the request to load the greeting works as well.

Creating a ZIP file containing those three files is then needed for the next step.

Stage 2: The Infrastructure

The Lambda

In the Lambda section of the AWS Console we create a new Web Function based off the previously created ZIP archive (I am doing that in eu-west-1 ), using the “Function name”: “LambdaWebFunction”, “Endpoint name”: “LambdaWebFunction” and the “Authentication type”: “IAM authentication”.

If we now try to invoke the Endpoint after creation we see the expected “403 Forbidden” as we have configured the Lambda to require IAM/SigV4 as part of a request.

The CloudFront Distribution

In the CloudFront section of the AWS Console we create a new CF Distribution, I was using “Pay as you go”, the “Distribution name”: “LambdaWebFunction” and the aforementioned custom domain. Select “Origin type”: “Lambda function” and select the previously created Lambda in the dialog which should populate CF with the Lambda Endpoint URL. Enable “Allow private Lambda function access to CloudFront” and leave the other options on their defaults. I skipped security for now, created a TLS certificate in ACM for the domain including the Route53 record needed to validate the certificate and after the CF distribution was created successfully also created the necessary Route53 records to route traffic to the domain to the CF Distribution.

After the CF Distribution was successfully deployed (which can take several minutes), invoking either the autogenerated CF domain or the aforementioned domain leads to an expected “403 Forbidden” again as the Lambda currently doesn’t allow CF to invoke itself.

Access Control

Finally back at the Lambda we create a Resource-based policy similar to the following

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCloudFrontOACAccess",
            "Effect": "Allow",
            "Principal": {
                "Service": "cloudfront.amazonaws.com"
            },
            "Action": "lambda:InvokeWebFunctionEndpoint",
            "Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:web-function/LambdaWebFunction/*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceArn": "arn:aws:cloudfront::<ACCOUNT_ID>:distribution/<DISTRIBUTION_ID>"
                }
            }
        }
    ]
}

that allows our CF Distribution (identified by Account ID and Distribution ID) to invoke all endpoints of our Lambda.

Stage 3: Demo and Summary

Invoking either the autogenerated CF domain or the aforementioned domain (which might require you to flush your DNS cache) now work and we can see both the static HTML page and also the request to the API working.

Setting up a Node.js based application serving both static content and also API worked relatively well, infrastructure support would obviously be nice and the ability to run the Web Function locally without significant overhead is a nice addition. Whether Web Functions have a large enough unique selling point over hosting static websites in S3 and API via ‘classic’ Lambdas, either behind an API Gateway or Application Load Balancer, remains to be seen.

Unfortunately not all applications running within AWS are Serverless native and use IAM for Authorisation and Authentication, so the ability of AWS STS to issue short lived JWT that can be verified against JWKS can be a solution. As of recently, the service to issue and verify tokens can be reached via VPC Endpoints, without the traffic leaving the AWS network.

To prove this out, the scenario will be a simple Lambda that first creates a short-lived JWT and subsequently validates it using the JWKS of the Issuer, all privately within a VPC. The infrastructure will be set up with CDK in TypeScript, the Lambda in Rust.

The Foundation

The Stack containing the data needed to set up the private Lambda in a VPC:

const vpcId = '...';
const vpcCidrBlock = '...';

const availabilityZones = ['...', '...', '...']

const firstSubnetId = '...'
const secondSubnetId = '...'
const thirdSubnetId = '...'

const vpc = Vpc.fromVpcAttributes(this, 'VPC', {
    vpcId,
    vpcCidrBlock,
    availabilityZones,
});

const firstSubnet = Subnet.fromSubnetId(this, 'FirstSubnet', firstSubnetId);
const secondSubnet = Subnet.fromSubnetId(this, 'SecondSubnet', secondSubnetId);
const thirdSubnet = Subnet.fromSubnetId(this, 'ThirdSubnet', thirdSubnetId);
const subnets = [firstSubnet, secondSubnet, thirdSubnet];
const vpcSubnets = {
    availabilityZones,
    subnets,
} as SubnetSelection;

const lambda = new RustFunction(this, 'Lambda', {
    entry: 'lambda',
    functionName: 'Lambda',
    memorySize: 1024,
    vpc,
    vpcSubnets,
    logGroup: new LogGroup(this, 'LambdaLogGroup', {
        logGroupName: '/aws/lambda/Lambda',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
lambda.role?.attachInlinePolicy(new Policy(this, 'Policy', {
    statements: [
        new PolicyStatement({
            actions: ['sts:GetWebIdentityToken'],
            resources: [`arn:aws:sts::${this.account}:self`],
        })
    ]
}))

and the Lambda:

struct Handler<'a> {
    client: &'a Client,
}

const AUDIENCE: &'static str = "MyExternalApplication";

const SIGNING_ALGORITHM: &'static str = "RS256";

#[derive(Serialize)]
struct Response {
    token: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, _: LambdaEvent<Value>) -> Result<Response, Error> {
        let token = self
            .client
            .get_web_identity_token()
            .audience(AUDIENCE)
            .signing_algorithm(SIGNING_ALGORITHM)
            .send()
            .await?
            .web_identity_token
            .unwrap();

        Ok(Response { token })
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);
    let handler = Handler {
        client: &client,
    };
    run(service_fn(|lambda_event: LambdaEvent<Value>| {
        handler.handle(lambda_event)
    }))
    .await
}

are both pretty straightforward. The Lambda resides in private Subnets in a VPC, requires IAM permissions to create the WebIdentityToken and uses the STS Client to create said token on invoke, using a fixed audience and algorithm. Invoking the Lambda leads to a timeout though: The STS client tries to reach the respective service via its public endpoint which isn’t reachable from the private Subnets that contain no route to the internet.

Private Networking: Issuing the Token

The solution to this is to use a VPC Endpoint, so expanding the Stack with the right VPC Endpoint that will allow the Lambda to reach STS:

new InterfaceVpcEndpoint(this, 'VPCEndpointSTS', {
    vpc,
    service: InterfaceVpcEndpointAwsService.STS,
    subnets: vpcSubnets
});

And on deployment and invoking the Lambda the token is now returned. Decoding the token leads to the confirmation that aud is being set to MyExternalApplication , sub is the role of our Lambda and iss is the Token Issuer URL that can be found in the Outbound Identity Federation section of the IAM Account Settings.

Verifying the Token

Now for the second step, expand the setup to verify the token that’s just been created.

The Lambda is slightly extended:

struct Handler<'a> {
    client: &'a Client,
    issuer: String,
}

const AUDIENCE: &'static str = "MyExternalApplication";

const SIGNING_ALGORITHM: &'static str = "RS256";

#[derive(Deserialize)]
struct Claims {
    pub sub: String,
}

#[derive(Serialize)]
struct Response {
    token: String,
    subject: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, _: LambdaEvent<Value>) -> Result<Response, Error> {
        let token = self
            .client
            .get_web_identity_token()
            .audience(AUDIENCE)
            .signing_algorithm(SIGNING_ALGORITHM)
            .send()
            .await?
            .web_identity_token
            .unwrap();

        let header = decode_header(&token)?;

        let jwks = Jwks::from_oidc_url(format!("{}/.well-known/openid-configuration", self.issuer))
            .await?;
        let kid = header.kid.unwrap();
        let jwk = jwks.keys.get(&kid).unwrap();

        let mut validation = Validation::new(header.alg);
        validation.validate_exp = true;
        validation.validate_nbf = true;
        validation.set_issuer(&[self.issuer.clone()]);
        validation.set_audience(&[AUDIENCE]);

        let token_data = decode::<Claims>(&token, &jwk.decoding_key, &validation)?;
        let subject = token_data.claims.sub;

        Ok(Response { token, subject })
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);
    let issuer = env::var("ISSUER")?;
    let handler = Handler {
        client: &client,
        issuer,
    };
    run(service_fn(|lambda_event: LambdaEvent<Value>| {
        handler.handle(lambda_event)
    }))
    .await
}

now reading the Issuer from an environment variable, using it to obtain JWKS and verifying the token whilst the infrastructure also gets slightly extended:

const uuid = '...';
const zoneName = 'tokens.sts.global.api.aws';

const lambda = new RustFunction(this, 'Lambda', {
    entry: 'lambda',
    environment: {
        ISSUER: `https://${uuid}.${zoneName}`,
    },
    functionName: 'Lambda',
    memorySize: 1024,
    vpc,
    vpcSubnets,
    logGroup: new LogGroup(this, 'LambdaLogGroup', {
        logGroupName: '/aws/lambda/Lambda',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});

by passing the Issuer to the Lambda.

Invoking the Lambda now leads to a timeout again as the Lambda tries to read the JWKS from the OpenID-configuration via the Issuer which again is not a routable URL for a Lambda in private Subnets.

Private Networking: Verifying the Token

So, lastly the solution is similar to above: A VPC Endpoint, just this time for STS-OIDC and not STS. And because the Issuer URL (see above) does not use the com.amazonaws.<REGION>.sts-oidc URL and instead the <UUID>.tokens.sts.global.api.aws URL we also need to create a Private Hosted Zone and a Route53 Alias that ensures that the Lambda when using the Issuer will instead resolve the VPC Endpoint:

const privateHostedZone = new PrivateHostedZone(this, 'PrivateHostedZone', {
    vpc,
    zoneName,
});
const interfaceVpcEndpoint = new InterfaceVpcEndpoint(this, 'VPCEndpointSTSOIDC', {
    vpc,
    service: new InterfaceVpcEndpointAwsService('sts-oidc'),
    subnets: vpcSubnets
});
const interfaceVpcEndpointTarget = new InterfaceVpcEndpointTarget(interfaceVpcEndpoint);
new ARecord(this, 'ARecord', {
    zone: privateHostedZone,
    recordName: uuid,
    target: RecordTarget.fromAlias(interfaceVpcEndpointTarget),
})

and thus, on deploying the last change, the Lambda can now also verify the JWT and extracting the subject from it successfully.

Summary

Whilst I still only see limited use of Outbound Identity Federation the possibility for non-Serverless-native applications to trust JWT including a completely VPC-private verification is a nice addition to the existing feature set.

In the past AWS API Gateways were only able to integrate with plain HTTP or HTTPS URL, now they also support integrating with backends that require mutual TLS. I found the SAM example a bit overly complex so I reproduced the same: Feature an Application Load Balancer (ALB), in CDK with TypeScript, almost completely self contained.

The Setup

Preparation

In preparation for this proof of concept we assume to manage a Route53 domain in AWS already. We also have an S3 Bucket at hand and a couple of public Subnets. Finally, we have created a Certificate for our domain as well.

The first part of our Stack will be the imports of resources that already exist:

const vpcId = '<VPC-ID>';
const availabilityZones = ['eu-central-1', 'eu-central-2'];
const vpc = Vpc.fromVpcAttributes(this, 'VPC', {
    vpcId,
    availabilityZones,
});

const bucketArn = '<BUCKET-ARN>';
const bucket = Bucket.fromBucketArn(this, 'Bucket', bucketArn);

const firstSubnetId = '<FIRST-SUBNET-ID>';
const firstSubnet = Subnet.fromSubnetId(this, 'FirstSubnet', firstSubnetId);
const secondSubnetId = '<SECOND-SUBNET-ID>';
const secondSubnet = Subnet.fromSubnetId(this, 'SecondSubnet', secondSubnetId);
const subnets = [firstSubnet, secondSubnet];
const vpcSubnets = {
    subnets,
} as SubnetSelection;

const certificateArn = '<CERTIFICATE-ARN>';
const certificate = Certificate.fromCertificateArn(this, 'Certificate', certificateArn);
const listenerCertificate = ListenerCertificate.fromCertificateManager(certificate);
const certificates = [listenerCertificate];

const hostedZoneId = '<HOSTED-ZONE-ID>';
const zoneName = '<ZONE-NAME>';
const zone = HostedZone.fromHostedZoneAttributes(this, 'HostedZone', {
    hostedZoneId,
    zoneName,
});

We are importing a VPC, a Bucket (that will hold the TrustStore later), the Subnets our ALB will be deployed into, the ALB’s public certificate that ensures the communication works with TLS in the first place and finally the Hosted Zone that holds the DNS information for our domain.

The Application Load Balancer

Next up, we set up the ALB with CDK:

const securityGroup = new SecurityGroup(this, 'SecurityGroup', {
    vpc,
})
securityGroup.addIngressRule(Peer.anyIpv4(), Port.HTTPS)

const applicationLoadBalancer = new ApplicationLoadBalancer(this, 'ALB', {
    vpc,
    internetFacing: true,
    vpcSubnets,
    securityGroup,
});

const trustStore = new TrustStore(this, 'TrustStore', {
    bucket,
    key: 'truststore.pem',
});

applicationLoadBalancer.addListener('Listener', {
    protocol: ApplicationProtocol.HTTPS,
    certificates,
    defaultAction: ListenerAction.fixedResponse(200, {
        contentType: 'text/plain',
        messageBody: 'Hello, mutual TLS authenticated World!'
    }),
    mutualAuthentication: {
        mutualAuthenticationMode: MutualAuthenticationMode.VERIFY,
        trustStore,
    }
})

const loadBalancerTarget = new LoadBalancerTarget(applicationLoadBalancer);
const target = RecordTarget.fromAlias(loadBalancerTarget);
new ARecord(this, 'ARecord', {
    target,
    zone,
})

For this, we create a Security Group (SG) that allows Ingress for any IPv4 address and the internet-facing ALB in the VPC/Subnets with said SG. We then define a TrustStore based off the Bucket we imported and add an HTTPS listener to the ALB that runs with the imported certificates and returns a fixed plain text message plus requires a valid client certificate that will be verified against the TrustStore. We finally create an A record in the Hosted Zone to allow clients to resolve the domain against our ALB’s DNS name. Before we continue we need to have created a Private Certificate Authority, stored its certificate in the Bucket and Key mentioned in the TrustStore above and have created a client certificate for that Private CA as well. Please note that a Private CA is a quite expensive resource and shouldn’t be kept longer than absolutely needed.

The API Gateway

Finally we can now create the API Gateway:

const clientCertificateId = '<CLIENT-CERTIFICATE-ARN>';

const api = new RestApi(this, 'API', {
    deploy: false,
});
api.root.addMethod('GET', new Integration({
    type: IntegrationType.HTTP_PROXY,
    uri: `https://${zoneName}`,
    integrationHttpMethod: 'GET',
}))
const deployment = new Deployment(this, 'Deployment', {
    api,
});
const plainStage = new Stage(this, 'PlainStage', {
    deployment: deployment,
    stageName: 'plain'
});
const mtlsStage = new Stage(this, 'MtlsStage', {
    deployment: deployment,
    stageName: 'mtls',
    clientCertificateId,
});

new CfnOutput(this, 'PlainURL', {
    key: 'PlainURL',
    value: plainStage.urlForPath()
})
new CfnOutput(this, 'MTLSURL', {
    key: 'MTLSURL',
    value: mtlsStage.urlForPath()
})

Here we define our API with a single GET method that integrates with our ALB (via the domain name) and create 2 stages: One “plain” stage without a client certificate and one “mtls” stage with a client certificate (the only difference is specifying the clientCertificateId field). Note that the client certificate is not set on the resource but on the stage which would e.g. allow us to have a backend without mTLS for development purposes and the backend with mTLS in production.

Demo

To demo the setup, simply invoke either URL: The PlainURL request fails with a connection error (because the API GW tries to proxy to a backend that expects a client certificate without providing a client certificate) whilst the MTLSURL works as expected.

Summary

The setup is straightforward and the integration is seamless, the fact that ALB’s require the mTLS CA to be uploaded to an S3 Bucket is a bit clunky though.

Ever since the first time I have researched best practices, performance tweaks and tooling for the use of synchronous Java Lambdas in my day job I have kept an eye on new developments that might make developers life easier and improve the experience of using Java in Lambda context. AWS recently published that AWS Lambda now supports SnapStart for container image functions so I wanted to take a look how containerized Java Lambdas with SnapStart perform against Plain Java Lambdas and Quarkus and how the developer experience is looking like.

The Setup

To be able to make informed decisions we first need a way to reproduce comparable runtime behaviour. The following setup doesn't claim to be scientific but it tries to mimic a common use case for Lambdas, reading a small file from an S3 Bucket. The scenarios are going to be:

1. Plain Java Lambda
2. Plain Java Lambda with SnapStart enabled
3. Containerized Java Lambda
4. Containerized Java Lambda with SnapStart enabled
5. Native Quarkus Lambda

and to provision the infrastructure and to stay on topic we are going to use CDK with Java.

The Lambdas

First, the Plain Java Lambda that is going to be used for the first four scenarios:

package com.example.myapp;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;

import java.io.IOException;

public class App implements RequestHandler<App.Request, App.Response> {

    public record Request(String bucket, String key) {
    }

    public record Response(String content) {
    }

    private final S3Client s3Client;

    public App() {
        this.s3Client = S3Client.builder()
                .build();
    }

    @Override
    public Response handleRequest(final Request request, final Context context) {
        final var getObjectRequest = GetObjectRequest.builder()
                .bucket(request.bucket)
                .key(request.key)
                .build();

        try (final var responseInputStream = this.s3Client.getObject(getObjectRequest)) {
            final var bytes = responseInputStream.readAllBytes();
            final var content = new String(bytes);

            return new Response(content);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }
}

and the Dockerfile used to create the image for scenarios 3 and 4:

FROM public.ecr.aws/lambda/java:21

COPY target/classes ${LAMBDA_TASK_ROOT}
COPY target/dependency/* ${LAMBDA_TASK_ROOT}/lib/

CMD [ "com.example.myapp.App::handleRequest" ]

Secondly, the Quarkus Java Lambda used in the fifth scenario:

package com.example.myapp;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;

import java.io.IOException;

public class App implements RequestHandler<App.Request, App.Response> {

    public record Request(String bucket, String key) {
    }

    public record Response(String content) {
    }

    private final S3Client client;

    public App() {
        this.client = S3Client.builder().build();
    }

    @Override
    public Response handleRequest(final Request request, final Context context) {
        final var getObjectRequest = GetObjectRequest.builder()
                .bucket(request.bucket)
                .key(request.key)
                .build();

        try (final var responseInputStream = this.client.getObject(getObjectRequest)) {
            final var bytes = responseInputStream.readAllBytes();
            final var content = new String(bytes);

            return new Response(content);
        } catch (final IOException e) {
            throw new RuntimeException(e);
        }
    }
}

When building on a Mac (like I did), when creating Docker images or native executables as usual cross-compile for Linux:

docker buildx build --platform linux/amd64 --provenance=false -t myapp:latest .

explicitly stating the platform as linux/amd64, respectively

quarkus build --native --no-tests -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi9-quarkus-mandrel-builder-image:25.0.4.1-Final-java25-amd64

explicitly stating the Quarkus native builder image as amd64.

The Stack

Now, having built the JAR, the Docker image and the Quarkus native executable, we can create the infrastructure:

final var bucket = Bucket.fromBucketName(this, "Bucket", "<BUCKET-NAME>");

final var lambda = Function.Builder.create(this, "MyAppLambda")
        .functionName("MyApp")
        .logGroup(LogGroup.Builder.create(this, "MyAppLogGroup")
                .removalPolicy(RemovalPolicy.DESTROY)
                .build())
        .memorySize(1024)
        .timeout(Duration.seconds(30))
        .code(Code.fromAsset("../myapp/target/myapp.jar"))
        .handler("com.example.myapp.App::handleRequest")
        .runtime(Runtime.JAVA_21)
        .build();
bucket.grantRead(lambda);

final var snapStartLambda = Function.Builder.create(this, "MyAppSnapStartLambda")
        .functionName("MyAppSnapStart")
        .logGroup(LogGroup.Builder.create(this, "MyAppSnapStartLogGroup")
                .removalPolicy(RemovalPolicy.DESTROY)
                .build())
        .memorySize(1024)
        .timeout(Duration.seconds(30))
        .snapStart(SnapStartConf.ON_PUBLISHED_VERSIONS)
        .code(Code.fromAsset("../myapp/target/myapp.jar"))
        .handler("com.example.myapp.App::handleRequest")
        .runtime(Runtime.JAVA_21)
        .build();
bucket.grantRead(snapStartLambda);
Version.Builder.create(this, "MyAppSnapStartLambdaVersion")
        .lambda(snapStartLambda)
        .build();

final var repository = Repository.fromRepositoryArn(this, "MyAppRepository", "arn:aws:ecr:<REGION>:<ACCOUNT>:repository/myapp");

final var containerizedLambda = Function.Builder.create(this, "MyContainerizedAppLambda")
        .functionName("MyContainerizedApp")
        .logGroup(LogGroup.Builder.create(this, "MyContainerizedAppLogGroup")
                .removalPolicy(RemovalPolicy.DESTROY)
                .build())
        .memorySize(1024)
        .timeout(Duration.seconds(30))
        .code(EcrImageCode.fromEcrImage(repository))
        .handler(Handler.FROM_IMAGE)
        .runtime(Runtime.FROM_IMAGE)
        .build();
bucket.grantRead(containerizedLambda);

final var containerizedSnapStartLambda = Function.Builder.create(this, "MyContainerizedAppSnapStartLambda")
        .functionName("MyContainerizedAppSnapStart")
        .logGroup(LogGroup.Builder.create(this, "MyContainerizedAppSnapStartLogGroup")
                .removalPolicy(RemovalPolicy.DESTROY)
                .build())
        .memorySize(1024)
        .timeout(Duration.seconds(30))
        .snapStart(SnapStartConf.ON_PUBLISHED_VERSIONS)
        .code(EcrImageCode.fromEcrImage(repository))
        .handler(Handler.FROM_IMAGE)
        .runtime(Runtime.FROM_IMAGE)
        .build();
bucket.grantRead(containerizedSnapStartLambda);
Version.Builder.create(this, "MyContainerizedAppSnapStartLambdaVersion")
        .lambda(containerizedSnapStartLambda)
        .build();

final var quarkusLambda = Function.Builder.create(this, "MyAppQuarkusLambda")
        .functionName("MyAppQuarkus")
        .logGroup(LogGroup.Builder.create(this, "MyAppQuarkusLogGroup")
                .removalPolicy(RemovalPolicy.DESTROY)
                .build())
        .memorySize(1024)
        .timeout(Duration.seconds(30))
        .code(Code.fromAsset("../myapp-quarkus/target/function.zip"))
        .handler("not.used.in.provided.runtime.")
        .runtime(Runtime.PROVIDED_AL2023)
        .build();
bucket.grantRead(quarkusLambda);

Note that all Lambdas run with the same memory size.

The Statistics

I invoked all five Lambdas repeatedly over the course of a day, gathering a series of data points that varied in detail but usually ended up in a similar region per type of Lambda. One detail I learnt about later is that the AWS SDK often requires to be “warmed up” manually as it’s lazily loading resources but as this “workaround” hasn’t been used anywhere I would assume it would only affect the absolute number and not the relative size of the cold start invocation compared between the different types of scenarios. The runtime for warm starts doesn't differ significantly between the Lambdas so the statistics below concentrates on cold starts.

Memory

Memory usage across the first four scenarios averaged at around 160 MB, Quarkus only used 70 MB of memory. As comparison, the same functionality written in Rust uses around 35 MB of memory.

Runtime

Runtime behaviour between the five scenarios is roughly that the Plain Java Lambda (~2.9s) runs twice as long as the Plain Java Lambda with SnapStart enabled (~1.5s), the Containerized Lambda (~2.3s) is around 25% faster than the Plain Java Lambda but significantly slower than the Plain Java Lambda with SnapStart enabled and the Containerized Java Lambda with SnapStart enabled (~1.1s) is roughly 2.5 times as fast as the Plain Java Lambda. The Quarkus Lambda finally returns in 0.5s and is thus again the fastest Lambda version between the five scenarios. Again as comparison, the same functionality written in Rust takes around 300ms.

In addition to that, for the SnapStart enabled scenarios the Lambdas need to publish a version which adds minor complexity to the infrastructure which might affect the decision to use SnapStart or to simply switch to Quarkus instead.

Summary

The differences between the scenarios are significant, for synchronous Lambdas I don’t see any reason not to use Quarkus if using the Java ecosystem is a mandatory criterium or to move to a compiled language altogether. For asynchronous invocations where cold starts are of less relevance staying with Plain Java Lambdas is the least amount of effort although I personally really like the Quarkus developer experience and tooling so at least personally I would most likely even use Quarkus for asynchronous Lambdas.

Apart from that, if for governance or other reasons the use of containerized Java Lambdas has to be considered, SnapStart looks to be a very promising addition here, although I would like the AWS SDK to support that directly via a dedicated warming up mechanism without the need for users to perform dummy API requests during initialisation as that’s quite clunky and not very transparent at the moment.

AWS Cognito has been supporting machine-to-machine communication before, for users with a Client ID and Secret, requiring users to get tokens via OAuth which meant performing an HTTP request to the Cognito User Pool Domain. Now with AWS Cognito supporting machine-to-machine authorization without a user pool domain it is possible to get a token with Client ID and Secret via AWS SDK and CLI alone so the following describes the minimal setup to obtain and use the token as part of a simple REST API.

The Setup

Using CDK with TypeScript, setting up a Lambda behind an API Gateway that will use the AWS provided Cognito User Pools Authorizer to minimize the amount of custom code needed. The Lambda will extract the sub (i.e. in this case the Client ID) and respond with it. As CDK currently doesn’t support the machine-to-machine authorization without a user pool domain with their Cognito constructs a few tweaks are needed and as the GetClientToken API results in an Access Token which by default isn’t trusted by the Cognito User Pools Authorizer on its own, a OAuth scope also needs to be defined on the client and the API to enable using Access Tokens to successfully communicate with the API.

The Lambda

The Lambda is a pretty straightforward TypeScript-Lambda that extracts the sub and returns it as part of a message:

exports.handler = async (event: any) => {
    return {
        isBase64Encoded: false,
        statusCode: 200,
        headers: {},
        body: `Hello ${event.requestContext.authorizer.claims.sub}!`,
    }
}

The Stack

The Stack defines a Lambda, a Cognito User Pool with a Client that allows machine-to-machine using a scope and a REST API that makes use of the pool in its Authorizer:

const lambda = new NodejsFunction(this, 'Lambda', {
    functionName: 'Lambda',
    logGroup: new LogGroup(this, 'LambdaLogGroup', {
        logGroupName: 'LambdaLogGroup',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
    entry: './lambda/index.ts',
    handler: 'handler'
});

const userPool = new UserPool(this, 'UserPool', {
    userPoolName: 'users',
    removalPolicy: RemovalPolicy.DESTROY,
});

const resourceServerScope = new ResourceServerScope({
    scopeName: 'read',
    scopeDescription: 'API Read Access',
});
const userPoolResourceServer = userPool.addResourceServer('ResourceServer', {
    identifier: 'resource-server',
    scopes: [
        resourceServerScope,
    ],
});
const authorizationScope = `${userPoolResourceServer.userPoolResourceServerId}/${resourceServerScope.scopeName}`;

const userPoolClient = userPool.addClient('Client', {
    userPoolClientName: 'headless-client',
    generateSecret: true,
    disableOAuth: true,
});
const cfnUserPoolClient = userPoolClient.node.defaultChild as CfnUserPoolClient;
cfnUserPoolClient.addPropertyOverride('ExplicitAuthFlows', ['ALLOW_CLIENT_TOKEN_AUTH']);
cfnUserPoolClient.addPropertyOverride('AllowedOAuthScopes', [authorizationScope]);

const cognitoUserPoolsAuthorizer = new CognitoUserPoolsAuthorizer(this, 'CognitoUserPoolsAuthorizer', {
    authorizerName: 'CognitoUserPoolsAuthorizer',
    cognitoUserPools: [userPool]
});

const restApi = new RestApi(this, 'APIGateway');
restApi.root.addMethod('GET', new LambdaIntegration(lambda), {
    authorizationType: AuthorizationType.COGNITO,
    authorizer: cognitoUserPoolsAuthorizer,
    authorizationScopes: [authorizationScope]
})

new Deployment(this, 'Deployment', {
    api: restApi,
})

new CfnOutput(this, 'ClientId', {
    key: 'ClientId',
    value: userPoolClient.userPoolClientId
});

new CfnOutput(this, 'ClientSecret', {
    key: 'ClientSecret',
    value: userPoolClient.userPoolClientSecret.unsafeUnwrap()
})

As mentioned before as CDK currently doesn’t support setting up the client directly the approach is to define a client that generates a secret but explicitly disables OAuth and to override the properties using the L1 construct afterwards:

const userPoolClient = userPool.addClient('Client', {
    userPoolClientName: 'headless-client',
    generateSecret: true,
    disableOAuth: true,
});
const cfnUserPoolClient = userPoolClient.node.defaultChild as CfnUserPoolClient;
cfnUserPoolClient.addPropertyOverride('ExplicitAuthFlows', ['ALLOW_CLIENT_TOKEN_AUTH']);
cfnUserPoolClient.addPropertyOverride('AllowedOAuthScopes', [authorizationScope]);

The Demo

After deploying the Stack and extracting the URL, the Client ID and the Client Secret, first an Access Token is requested using ID, Secret and the Scope defined above:

TOKEN=$(aws cognito-idp get-client-token \
    --client-id $CLIENT_ID \
    --secret $CLIENT_SECRET \
    --scopes resource-server/read \
    | jq -r '.ClientAuthenticationResult.AccessToken')

Now just for the sake of completeness and to demo that the API is set up right, an unauthenticated request expecting an HTTP 401 is performed:

curl -v $URL

And finally using the Token that’s just been generated, an authenticated request expecting an HTTP 200 is performed:

curl -v -H "Authorization: Bearer $TOKEN" $URL

Which greets us with the Client ID that has been used to create the Access Token.

Summary

Whilst the setup in CDK is a bit clunky due to the missing L2 construct support the functionality works straightforward and removing the need to get a token indirect via the OAuth URL reduces effort and streamlines machine-to-machine communication.

Dealing with confidential data in Lambdas is a common theme. Instead of hard-coding data in code, or passing it in as environment variables (that might be visible in the AWS console), confidential data can be stored in AWS SecretsManager and Lambdas can load secrets at runtime, which then leads to the next question: What if the confidential data gets updated? Common approaches are to use explicit time to live, or to handle failures when said confidential data is used (e.g. when it represents credentials that are being used to authenticate at a service and when a failing authentication is then used as a signal to refresh the credentials once. But what if the credentials are genuinely wrong and not just outdated, how to distinguish those use cases then?).

In the following a series of scenarios where Lambdas deal with secrets are presented that culminate in an approach that makes use of a new AWS feature to reload secrets when being notified.

The Setup

All following examples will be using the same basic setup: A Secret that contains a “Prefix” text that will be used to greet the user who invokes the Lambdas. To set up the infrastructure CDK with TypeScript will be used and the Lambdas are written in Rust.

const secret = new Secret(this, 'Secret', {
    secretName: 'Prefix',
    secretStringValue: SecretValue.unsafePlainText('Hello, ')
});

Read

The simplest and naive way to read secrets is once per invocation:

First the Lambda is created

use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::Deserialize;
use std::env;

struct Handler<'a> {
    secret_id: String,
    client: &'a Client,
}

#[derive(Deserialize, Debug)]
struct ReadRequest {
    name: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
        let pattern = self
            .client
            .get_secret_value()
            .secret_id(self.secret_id.as_str())
            .send()
            .await?
            .secret_string
            .unwrap();
        Ok(format!("{}{}", pattern, lambda_event.payload.name))
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let secret_id = env::var("SECRET_ID")?;

    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);

    let handler = Handler {
        secret_id,
        client: &client,
    };

    run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
        handler.handle(lambda_event)
    }))
    .await
}

And then the infrastructure is set up

const readSecretLambda = new RustFunction(this, 'ReadSecret', {
    entry: 'read_secret',
    environment: {
        'SECRET_ID': secret.secretArn,
    },
    functionName: 'ReadSecret',
    memorySize: 1024,
    logGroup: new LogGroup(this, 'ReadSecretLogGroup', {
        logGroupName: '/aws/lambda/ReadSecret',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
secret.grantRead(readSecretLambda);

Performance wise this approach isn’t ideal: ~350ms for cold starts and ~30ms for warm starts. Can’t the secret be stored in between invocations?

Lazy Read

Following the idea of the previous step the secret is lazy loaded and stored in between invocations.

First the Lambda is created

use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use serde::Deserialize;
use std::{env, sync::Mutex};

struct Handler<'a> {
    secret_id: String,
    client: &'a Client,
    secret_value: Mutex<Option<String>>,
}

#[derive(Deserialize, Debug)]
struct ReadRequest {
    name: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
        match self.secret_value.lock() {
            Ok(mut guard) => {
                let pattern = match guard.as_ref() {
                    None => {
                        let pattern = self
                            .client
                            .get_secret_value()
                            .secret_id(self.secret_id.as_str())
                            .send()
                            .await?
                            .secret_string
                            .unwrap();
                        guard.replace(pattern.clone());
                        println!("Lazy-Loaded the Secret {}", self.secret_id);

                        pattern
                    }
                    Some(pattern) => {
                        println!("Secret {} already loaded", self.secret_id);

                        pattern.to_string()
                    }
                };

                Ok(format!("{}{}", pattern, lambda_event.payload.name))
            }
            Err(err) => Err(Error::from(err.to_string())),
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let secret_id = env::var("SECRET_ID")?;

    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);

    let secret_value = Mutex::new(None);

    let handler = Handler {
        secret_id,
        client: &client,
        secret_value,
    };

    run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
        handler.handle(lambda_event)
    }))
    .await
}

And then the infrastructure is set up

const readSecretLambdaLazy = new RustFunction(this, 'ReadSecretLazy', {
    entry: 'read_secret_lazy',
    environment: {
        'SECRET_ID': secret.secretArn,
    },
    functionName: 'ReadSecretLazy',
    memorySize: 1024,
    logGroup: new LogGroup(this, 'ReadSecretLazyLogGroup', {
        logGroupName: '/aws/lambda/ReadSecretLazy',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
secret.grantRead(readSecretLambdaLazy);

Performance wise this approach is similar to the previous one for cold starts but manages to achieve 1ms for warm starts.

Preload

Another idea is to load the secret on startup, considering it shouldn’t matter if the secret is lazy loaded or preloaded.

First the Lambda is created

use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::Deserialize;
use std::env;

struct Handler {
    secret_value: String,
}

#[derive(Deserialize, Debug)]
struct ReadRequest {
    name: String,
}

impl Handler {
    async fn handle(&self, lambda_event: LambdaEvent<ReadRequest>) -> Result<String, Error> {
        Ok(format!(
            "{}{}",
            self.secret_value, lambda_event.payload.name
        ))
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);
    let secret_id = env::var("SECRET_ID")?;
    let secret_value = client
        .get_secret_value()
        .secret_id(secret_id)
        .send()
        .await?
        .secret_string
        .unwrap();
    let handler = Handler { secret_value };
    run(service_fn(|lambda_event: LambdaEvent<ReadRequest>| {
        handler.handle(lambda_event)
    }))
    .await
}

And then the infrastructure is set up

const preloadSecretLambda = new RustFunction(this, 'PreloadSecret', {
    entry: 'preload_secret',
    environment: {
        'SECRET_ID': secret.secretArn,
    },
    functionName: 'PreloadSecret',
    memorySize: 1024,
    logGroup: new LogGroup(this, 'PreloadSecretLogGroup', {
        logGroupName: '/aws/lambda/PreloadSecret',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
secret.grantRead(preloadSecretLambda);

Interestingly the cold starts get reduced to ~300ms but the warm starts are similarly performant as the last solutions.

Preload with Updates

Unfortunately all three previous approaches assume the secret will either never change or that if it changes in between invocations that this coincides with the Lambda environment being shutdown so that the next cold start will pick up the updated secret. For an approach that makes use of the AWS Workload Credentials Provider see https://medium.com/@johannesfloriangeiger/aws-workload-credentials-provider-for-aws-lambdas-hands-on-ae436604b573 for example.

But what if a Lambda could react to a changing secret to reload it independently? Recently AWS introduced a mechanism that so that AWS Secrets Manager now publishes secret update notifications to Amazon EventBridge and so the idea of the last scenario is to use those events to invoke a Lambda, triggering a secret reload.

First the Lambda is created

use aws_config::BehaviorVersion;
use aws_sdk_secretsmanager::Client;
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use serde::Deserialize;
use serde_json::Value;
use std::{env, sync::Mutex};

struct Handler<'a> {
    secret_id: String,
    client: &'a Client,
    secret_value: Mutex<Option<String>>,
}

#[derive(Deserialize, Debug)]
struct ReadRequest {
    name: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<Value>) -> Result<String, Error> {
        match lambda_event.payload.get("name") {
            None => match lambda_event.payload.get("id") {
                None => {
                    println!("Unknown event");

                    Ok("".into())
                }
                Some(_) => match self.secret_value.lock() {
                    Ok(mut guard) => {
                        let pattern = self
                            .client
                            .get_secret_value()
                            .secret_id(self.secret_id.as_str())
                            .send()
                            .await?
                            .secret_string
                            .unwrap();
                        guard.replace(pattern.clone());
                        println!("Reloaded the Secret: {}", pattern);

                        Ok("".into())
                    }
                    Err(err) => Err(Error::from(err.to_string())),
                },
            },
            Some(_) => match self.secret_value.lock() {
                Ok(mut guard) => {
                    let pattern = guard.as_mut().unwrap();
                    let read_request: ReadRequest =
                        serde_json::from_value(lambda_event.payload.clone())?;

                    Ok(format!("{}{}", pattern, read_request.name))
                }
                Err(err) => Err(Error::from(err.to_string())),
            },
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);
    let secret_id = env::var("SECRET_ID")?;
    let secret_value = client
        .get_secret_value()
        .secret_id(secret_id.clone())
        .send()
        .await?
        .secret_string
        .unwrap();
    let handler = Handler {
        secret_id,
        client: &client,
        secret_value: Mutex::new(Some(secret_value)),
    };
    run(service_fn(|lambda_event: LambdaEvent<Value>| {
        handler.handle(lambda_event)
    }))
    .await
}

And then the infrastructure is set up

const secretEventBus = new EventBus(this, 'SecretEventBus', {
    eventBusName: 'SecretEventBus'
});
const eventBusTarget = new EventBusTarget(secretEventBus);
const defaultEventBus = EventBus.fromEventBusName(this, 'DefaultEventBus', 'default');
new Rule(this, 'DefaultEventBusSecretRule', {
    ruleName: 'DefaultEventBusSecretRule',
    eventPattern: {
        detailType: ['Secret Label Updated'],
        source: ['aws.secretsmanager'],
        account: [this.account],
        detail: {
            name: [secret.secretName],
        }
    },
    targets: [eventBusTarget],
    eventBus: defaultEventBus,
});

const preloadSecretUpdatesLambda = new RustFunction(this, 'PreloadSecretUpdatesLambda', {
    entry: 'preload_secret_updates',
    environment: {
        'SECRET_ID': secret.secretArn,
    },
    functionName: 'PreloadSecretUpdatesLambda',
    memorySize: 1024,
    logGroup: new LogGroup(this, 'PreloadSecretUpdatesLambdaLogGroup', {
        logGroupName: '/aws/lambda/PreloadSecretUpdatesLambda',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
secret.grantRead(preloadSecretUpdatesLambda);
const preloadSecretUpdatesLambdaTarget = new LambdaTarget(preloadSecretUpdatesLambda);
new Rule(this, 'PreloadSecretUpdatesLambdaRule', {
    ruleName: 'PreloadSecretUpdatesLambdaRule',
    eventPattern: {
        account: [this.account],
    },
    targets: [preloadSecretUpdatesLambdaTarget],
    eventBus: secretEventBus,
});

The setup is a bit more complex than the last one because the Lambda now needs to be able to handle both EventBridge events and manual invocations (for this the Lambda checks if the event contains a name field for a manual invocation or an id field for an EventBridge invocation) and when receiving the EventBridge event it reloads the secret. Infrastructure wise the secret update events are being routed to a new “secret” event bus via an EventBridge rule from the default event bus (for clean separation) and a second EventBridge rule then invokes the Lambda for all events appearing in that “secret” event bus.

The performance is similar to the last scenario, the only difference is that changing the secret value will lead to an invocation that’s slightly higher (~50ms) but results in subsequent invocations that make use of the new secret value and thus avoid stale information.

Summary

Whilst the setup works relatively well and is relatively performant, dealing with multiple types of events for the Lambda in the last case is not very elegant and mixes different concerns in the same code. Ideally Lambdas would have a separate channel for notifications (perhaps also a potential improvement for the AWS Workload Credentials Provider?) that would allow for the secrets to be reloaded in the background.

Inspired by the DynamoDB feature of streaming change events to consumers via DynamoDB Streams the news of Aurora DSQL now generally supporting Change Data Capture (CDC) via Kinesis is an exciting one as implementing Event-Driven Architectures using a native feature of an SQL database reduces complexity on the application side and streamlines consumers by adopting a standardised pattern, in this case a Kinesis Data Stream.

The Scenario

As a proof of concept we are going to look at a relatively simple scenario of a Request Processor that accepts messages containing a URL and stores the status code of the HTTP response when calling that URL in a table. The process is initiated by a Create Requests Lambda that writes “Request Created” events into a database table, then picked up by the Dispatcher Lambda via the CDC Kinesis Stream, dispatched to a SQS Queue and finally picked up by the Process Requests Lambda that processes the requests (by calling the URL) and stores the status code as a “Request Processed” event in the table again. As the structure of the table is relatively simple we are going to use a Single Table design instead of one table per event type which would require some JOIN’s for reading the actual status of an event.

The Infrastructure

Now we need to create some infrastructure for the above, starting with the Aurora DSQL cluster and the Kinesis Data Stream that will be used for CDC (all using TypeScript CDK in the following):

const cluster = new Cluster(this, 'Cluster', {
    removalPolicy: RemovalPolicy.DESTROY,
});

const stream = new Stream(this, 'Stream', {
    streamName: 'Stream',
    removalPolicy: RemovalPolicy.DESTROY,
});

next we need an IAM role that Aurora DSQL is going to use to write to Kinesis:

const cdcRole = new Role(this, 'CDCRole', {
    assumedBy: new ServicePrincipal('dsql'),
    inlinePolicies: {
        Default: new PolicyDocument({
            statements: [
                new PolicyStatement({
                    actions: [
                        'kinesis:DescribeStreamSummary',
                        'kinesis:ListShards',
                        'kinesis:PutRecord',
                        'kinesis:PutRecords',
                    ],
                    resources: [
                        stream.streamArn
                    ]
                })
            ]
        })
    },
    roleName: 'CDCRole'
});

and to create the CDC Stream we currently need to use an AWS Custom Resource as CloudFormation is currently not supporting it, and for this we are referencing the cluster, the stream and the role created above:

new AwsCustomResource(this, 'CreateCDCStream', {
    onCreate: {
        service: 'dsql',
        action: 'createStream',
        parameters: {
            clusterIdentifier: cluster.clusterId,
            targetDefinition: {
                kinesis: {
                    streamArn: stream.streamArn,
                    roleArn: cdcRole.roleArn,
                }
            },
            ordering: 'UNORDERED',
            format: 'JSON',
        },
        physicalResourceId: PhysicalResourceId.of('create-cdc-stream')
    },
    policy: AwsCustomResourcePolicy.fromStatements([
        new PolicyStatement({
            actions: ['dsql:CreateStream'],
            resources: [cluster.clusterArn]
        }),
        new PolicyStatement({
            actions: ['iam:PassRole'],
            resources: [cdcRole.roleArn]
        })
    ])
});

Finally we need to create the three Lambdas, together with Kinesis and SQS Event Sources to invoke the Lambdas:

const queue = new Queue(this, 'RequestQueue', {
    queueName: 'RequestQueue',
});

const createRequestLambda = new RustFunction(this, 'CreateRequest', {
    entry: 'create-request',
    environment: {
        'CLUSTER_ENDPOINT': `${cluster.clusterId}.dsql.${this.region}.on.aws`,
    },
    functionName: 'CreateRequest',
    logGroup: new LogGroup(this, 'CreateRequestLogGroup', {
        logGroupName: '/aws/lambda/CreateRequest',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
cluster.grantConnectAdmin(createRequestLambda);

const processRequestLambda = new RustFunction(this, 'ProcessRequest', {
    entry: 'process-request',
    environment: {
        'CLUSTER_ENDPOINT': `${cluster.clusterId}.dsql.${this.region}.on.aws`,
    },
    functionName: 'ProcessRequest',
    logGroup: new LogGroup(this, 'ProcessRequestLogGroup', {
        logGroupName: '/aws/lambda/ProcessRequest',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
cluster.grantConnectAdmin(processRequestLambda);
const sqsEventSource = new SqsEventSource(queue, {
    batchSize: 1,
});
processRequestLambda.addEventSource(sqsEventSource)

const dispatcherLambda = new RustFunction(this, 'Dispatcher', {
    entry: 'dispatcher',
    environment: {
        'QUEUE_URL': queue.queueUrl,
    },
    functionName: 'Dispatcher',
    logGroup: new LogGroup(this, 'DispatcherLogGroup', {
        logGroupName: '/aws/lambda/Dispatcher',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
});
queue.grantSendMessages(dispatcherLambda);
const kinesisEventSource = new KinesisEventSource(stream, {
    startingPosition: StartingPosition.LATEST,
});
dispatcherLambda.addEventSource(kinesisEventSource);

We are using the Aurora DSQL Admin user for this for simplicities sake, for a more fine grained setup of Aurora DSQL see https://medium.com/@johannesfloriangeiger/amazon-aurora-dsql-with-lambdas-revisited-and-this-time-with-rust-ece95306b902.

After having deployed everything we are logging into Aurora DSQL via the AWS Console and create the table structure:

CREATE TABLE public.requests (id UUID, request_type VARCHAR(255), url VARCHAR(255), status_code VARCHAR(255), PRIMARY KEY(id, request_type))

And when invoking the Create Request Lambda with a URL we can query said table and see the “Request Created” and shortly after the “Request Processed” events:

SELECT * FROM public.requests;

Summary

The integration between Aurora DSQL and Kinesis works seamless, and consuming Kinesis with Lambdas is similar. If Aurora DSQL would now also support setting up the CDC Stream via CloudFormation and we wouldn't have to use the AWS SDK via an AWS Custom Resource for this it would be even better.

Appendix

create-request/src/main.rs

use aurora_dsql_sqlx_connector::{retry_on_occ, OCCRetryConfig};
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::Deserialize;
use sqlx::PgPool;
use std::env;
use uuid::Uuid;

struct Handler<'a> {
    pool: &'a PgPool,
}

#[derive(Deserialize, Debug)]
struct CreateRequest {
    url: String,
}

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<CreateRequest>) -> Result<(), Error> {
        let occ_config = OCCRetryConfig::default();
        retry_on_occ(&occ_config, || async {
            let mut transaction = self.pool.begin().await?;
            sqlx::query("INSERT INTO public.requests(id, request_type, url) VALUES ($1, $2, $3);")
                .bind(Uuid::new_v4())
                .bind("REQUEST_CREATED")
                .bind(lambda_event.payload.url.clone())
                .execute(&mut *transaction)
                .await?;
            transaction.commit().await?;

            Ok(())
        })
        .await?;

        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let cluster_endpoint = env::var("CLUSTER_ENDPOINT")?;
    let connection_string = format!("postgres://admin@{}/postgres", cluster_endpoint);
    let pool = aurora_dsql_sqlx_connector::pool::connect(connection_string.as_str()).await?;
    let handler = Handler { pool: &pool };
    run(service_fn(|lambda_event: LambdaEvent<CreateRequest>| {
        handler.handle(lambda_event)
    }))
    .await?;
    pool.close().await;

    Ok(())
}

dispatcher/src/main.rs

use aws_config::{BehaviorVersion, load_defaults};
use aws_lambda_events::kinesis::KinesisEvent;
use aws_sdk_sqs::{Client, types::builders::MessageAttributeValueBuilder};
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use serde::Deserialize;
use std::env;

struct Handler<'a> {
    queue_url: String,
    client: &'a Client,
}

#[derive(Deserialize, Debug)]
struct After {
    id: String,
    request_type: String,
    url: String,
}

#[derive(Deserialize, Debug)]
struct CDCMessage {
    op: String,
    after: Option<After>,
}

const OPERATION_CREATE: &'static str = "c";

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<KinesisEvent>) -> Result<(), Error> {
        for kinesis_event_record in lambda_event.payload.records {
            let cdc_message: CDCMessage =
                serde_json::from_slice(kinesis_event_record.kinesis.data.0.as_slice())?;
            let operation = cdc_message.op;
            let create = operation == OPERATION_CREATE;

            if create {
                let after = cdc_message.after.unwrap();
                let id = uuid::Uuid::parse_str(after.id.as_str()).unwrap();
                let request_type = after.request_type;
                println!("Request received: {} ({}).", request_type, id);

                match request_type.as_str() {
                    "REQUEST_CREATED" => {
                        println!("Request created.");
                        self.client
                            .send_message()
                            .queue_url(self.queue_url.clone())
                            .message_attributes(
                                "id",
                                MessageAttributeValueBuilder::default()
                                    .string_value(id)
                                    .data_type("String".to_string())
                                    .build()?,
                            )
                            .message_body(after.url.as_str())
                            .send()
                            .await?;
                    }
                    "REQUEST_PROCESSED" => {
                        println!("Request processed.");
                    }
                    _ => {
                        println!("Unknown request type: {}", request_type);
                    }
                }

                println!("Request handled ({}).", id);
            } else {
                println!("Unknown operation: {}", operation);
            }
        }

        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let queue_url = env::var("QUEUE_URL")?;
    let sdk_config = load_defaults(BehaviorVersion::latest()).await;
    let client = Client::new(&sdk_config);
    let handler = Handler {
        queue_url,
        client: &client,
    };
    run(service_fn(|lambda_event: LambdaEvent<KinesisEvent>| {
        handler.handle(lambda_event)
    }))
    .await?;

    Ok(())
}

process-request/src/main.rs

use aurora_dsql_sqlx_connector::{OCCRetryConfig, retry_on_occ};
use aws_lambda_events::sqs::SqsEvent;
use lambda_runtime::{Error, LambdaEvent, run, service_fn};
use sqlx::PgPool;
use std::env;

struct Handler<'a> {
    pool: &'a PgPool,
}

impl<'a> Handler<'a> {
    async fn handle(&self, lambda_event: LambdaEvent<SqsEvent>) -> Result<(), Error> {
        for sqs_message in lambda_event.payload.records {
            let url = sqs_message.body.unwrap();
            let id = sqlx::types::Uuid::parse_str(
                sqs_message
                    .message_attributes
                    .get("id")
                    .unwrap()
                    .string_value
                    .clone()
                    .unwrap()
                    .as_str(),
            )?;
            let status_code = ureq::get(url.as_str()).call()?.status();
            let occ_config = OCCRetryConfig::default();
            retry_on_occ(&occ_config, || async {
                let mut transaction = self.pool.begin().await?;
                sqlx::query("INSERT INTO public.requests(id, request_type, url, status_code) VALUES ($1, $2, $3, $4);")
                    .bind(id)
                    .bind("REQUEST_PROCESSED")
                    .bind(url.clone())
                    .bind(status_code.as_str())
                    .execute(&mut *transaction)
                    .await?;
                transaction.commit().await?;

                Ok(())
            }).await?;
        }

        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let cluster_endpoint = env::var("CLUSTER_ENDPOINT")?;
    let connection_string = format!("postgres://admin@{}/postgres", cluster_endpoint);
    let pool = aurora_dsql_sqlx_connector::pool::connect(connection_string.as_str()).await?;
    let handler = Handler { pool: &pool };
    run(service_fn(|lambda_event: LambdaEvent<SqsEvent>| {
        handler.handle(lambda_event)
    }))
    .await?;
    pool.close().await;

    Ok(())
}

Access control across AWS accounts is something I’m encountering every now and then and it’s an interesting topic that adds some intricacies to the usual AWS setup. Not always we can rely on clients being able to use IAM auth, especially when operating in abstracted environments such as Kubernetes so the need to share explicit credentials can arise. Not considering a solution where a Third party store like Vault is being used we are going to concentrate on as much AWS native services as possible to facilitate access control. As an example for a service that can be easily used for a proof of concept we will use AWS API Gateway with API Keys, and store those API Keys encrypted with a custom KMS Key in AWS Secrets Manager and we will be looking at scenarios where a Producer account provides an API that a Consumer account wants to access properly.

Solution 1: Pull

The first solution we are going to look at is the Pull model. In this case, the Producer is responsible for creating the credential store, sharing it with the Consumer and keeping the credentials up to date.

Producer

We start by creating an API Gateway with a Mock integration that will always return a 200 and require an API key. For this we will use CDK written in Golang:

restApi := awsapigateway.NewRestApi(stack, jsii.String("API"), &awsapigateway.RestApiProps{})
restApi.Root().AddMethod(jsii.String("GET"), awsapigateway.NewMockIntegration(&awsapigateway.IntegrationOptions{
 IntegrationResponses: &[]*awsapigateway.IntegrationResponse{{
  StatusCode: jsii.String("200"),
 },
},
 PassthroughBehavior: awsapigateway.PassthroughBehavior_NEVER,
 RequestTemplates: &map[string]*string{
  "application/json": jsii.String("{ \"statusCode\": 200 }"),
 },
}), &awsapigateway.MethodOptions{
 ApiKeyRequired: jsii.Bool(true),
 MethodResponses: &[]*awsapigateway.MethodResponse{
  {
   StatusCode: jsii.String("200"),
  },
 },
})

awsapigateway.NewDeployment(stack, jsii.String("Deployment"), &awsapigateway.DeploymentProps{
 Api: restApi,
})

even though this API gets deployed, calling it will fail because we’re missing an API Key at the moment which we are adding next:

apiKey := awsapigateway.NewApiKey(stack, jsii.String("APIKey"), &awsapigateway.ApiKeyProps{})

usagePlan := restApi.AddUsagePlan(jsii.String("UsagePlan"), &awsapigateway.UsagePlanProps{
 ApiStages: &[]*awsapigateway.UsagePlanPerApiStage{
  {Api: restApi, Stage: restApi.DeploymentStage()},
 },
})
usagePlan.AddApiKey(apiKey, &awsapigateway.AddApiKeyOptions{})

this will create an API Key and attach it to a Usage Plan which is attached to the API Gateway stage. Calling the API with this API Key should now return the mock result above.

Next, we will create a KMS Key and a Secret that will hold the API Key in the future:

consumerAccount := "ACCOUNT_ID"
principalArnWildcard := fmt.Sprintf("arn:aws:iam::%s:role/ConsumerStack-*", consumerAccount)

key := awskms.NewKey(stack, jsii.String("Key"), &awskms.KeyProps{
 RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
})
key.Policy().AddStatements(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
 Actions: &[]*string{
  jsii.String("kms:Decrypt"),
 },
 Conditions: &map[string]interface{}{
  "ArnLike": &map[string]interface{}{
   "aws:PrincipalArn": jsii.String(principalArnWildcard),
  },
 },
 Principals: &[]awsiam.IPrincipal{
  awsiam.NewStarPrincipal(),
 },
 Resources: &[]*string{
  jsii.String("*"),
 },
}))

secret := awssecretsmanager.NewSecret(stack, jsii.String("Secret"), &awssecretsmanager.SecretProps{
 EncryptionKey: key,
})
secret.AddToResourcePolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
 Actions: &[]*string{
  jsii.String("secretsmanager:GetSecretValue"),
 },
 Conditions: &map[string]interface{}{
  "ArnLike": &map[string]interface{}{
   "aws:PrincipalArn": jsii.String(principalArnWildcard),
  },
 },
 Principals: &[]awsiam.IPrincipal{
  awsiam.NewStarPrincipal(),
 },
 Resources: &[]*string{
  jsii.String("*"),
 },
}))

Here we see the first speciality of cross-account Secret access: The need to explicitly specify the Consumer accounts’ principal that will get access to the Key and the Secret. If the Consumer setup would exist already (in a “dummy” form waiting for the Producer setup to be created) we could use the ARN of the Consumer directly but a more flexible alternative is to use a wildcard Principal and restrict via an IAM Condition which allows the Producer to be created without a Consumer when both have agreed on a pattern of ARN to be used.

Finally we want to ensure that the API Key is automatically updating the Secret and for this we first create a Synchronize Lambda:

synchronizeFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("Synchronize"), &awscdklambdagoalpha.GoFunctionProps{
 FunctionName: jsii.String("Synchronize"),
 LogGroup: awslogs.NewLogGroup(stack, jsii.String("SynchronizeLogGroup"), &awslogs.LogGroupProps{
 LogGroupName:  jsii.String("SynchronizeLogGroup"),
  RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
 }),
 Entry: jsii.String("synchronize"),
})

ensure it has access to all relevant resources:

apiKey.GrantRead(synchronizeFunction)
//...
key.GrantEncrypt(synchronizeFunction)
// ...
secret.GrantWrite(synchronizeFunction)

and we finally call the Lambda as part of the CDK Stack:

synchronizeApiKeyValue := customresources.NewAwsCustomResource(stack, jsii.String("SynchronizeAPIKeyValue"), &customresources.AwsCustomResourceProps{
 OnCreate: &customresources.AwsSdkCall{
  Service: jsii.String("Lambda"),
  Action:  jsii.String("invoke"),
  Parameters: &map[string]interface{}{
   "FunctionName": jsii.String("Synchronize"),
   "Payload":      jsii.Sprintf("{\"apiKeyId\":\"%s\",\"secretId\":\"%s\"}", *apiKey.KeyId(), *secret.SecretArn()),
  },
  PhysicalResourceId: customresources.PhysicalResourceId_Of(apiKey.KeyId()),
 },
 Policy: customresources.AwsCustomResourcePolicy_FromStatements(&[]awsiam.PolicyStatement{
  awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
   Actions: &[]*string{
    jsii.String("lambda:InvokeFunction"),
   },
   Resources: &[]*string{
    synchronizeFunction.FunctionArn(),
   },
  }),
 }),
})
synchronizeApiKeyValue.Node().AddDependency(synchronizeFunction, apiKey, secret)

and afterwards we should be able to verify that the Secret contains indeed the API Key value.

Consumer

The Consumer then just needs to make sure that the Identity policy of the Lambda has access to the KMS Key and Secret above:

keyArn := "KEY_ARN"
key := awskms.Key_FromKeyArn(stack, jsii.String("Key"), jsii.String(keyArn))

secretArn := "SECRET_ARN"
secret := awssecretsmanager.Secret_FromSecretCompleteArn(stack, jsii.String("Secret"), jsii.String(secretArn))

url := "API_GATEWAY_URL"
function := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("Consumer"), &awscdklambdagoalpha.GoFunctionProps{
 Environment: &map[string]*string{
  "SECRET_ID": jsii.String(secretArn),
  "URL":       jsii.String(url),
 },
 FunctionName: jsii.String("Consumer"),
 LogGroup: awslogs.NewLogGroup(stack, jsii.String("ConsumerLogGroup"), &awslogs.LogGroupProps{
  LogGroupName:  jsii.String("ConsumerLogGroup"),
  RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
 }),
 Entry: jsii.String("consumer"),
})
key.GrantDecrypt(function)
secret.GrantRead(function, nil)

and invoking the Consumer then should lead to a 200 OK being returned to us.

Solution 2: Push

The second solution, the Push model, will invert the roles a bit. In this case, the Consumer is responsible for creating the credential store, sharing it with the Producer and the Producer will keep the credentials up to date.

Consumer

We first start by setting up the Consumer:

producerAccount := "ACCOUNT_ID"
principalArnWildcard := fmt.Sprintf("arn:aws:iam::%s:role/ProducerStack-*", producerAccount)

key := awskms.NewKey(stack, jsii.String("Key"), &awskms.KeyProps{
 RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
})
key.Policy().AddStatements(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
 Actions: &[]*string{
  jsii.String("kms:GenerateDataKey"),
  jsii.String("kms:Encrypt"),
 },
 Conditions: &map[string]interface{}{
  "ArnLike": &map[string]interface{}{
   "aws:PrincipalArn": jsii.String(principalArnWildcard),
  },
 },
 Principals: &[]awsiam.IPrincipal{
  awsiam.NewStarPrincipal(),
 },
 Resources: &[]*string{
  jsii.String("*"),
 },
}))

secret := awssecretsmanager.NewSecret(stack, jsii.String("Secret"), &awssecretsmanager.SecretProps{
 EncryptionKey: key,
})
secret.AddToResourcePolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
 Actions: &[]*string{
  jsii.String("secretsmanager:PutSecretValue"),
 },
 Conditions: &map[string]interface{}{
  "ArnLike": &map[string]interface{}{
   "aws:PrincipalArn": jsii.String(principalArnWildcard),
  },
 },
 Principals: &[]awsiam.IPrincipal{
  awsiam.NewStarPrincipal(),
 },
 Resources: &[]*string{
  jsii.String("*"),
 },
}))

url := ""
consumer := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("Consumer"), &awscdklambdagoalpha.GoFunctionProps{
 Environment: &map[string]*string{
  "SECRET_ID": secret.SecretArn(),
  "URL":       jsii.String(url),
 },
 FunctionName: jsii.String("Consumer"),
 LogGroup: awslogs.NewLogGroup(stack, jsii.String("ConsumerLogGroup"), &awslogs.LogGroupProps{
  LogGroupName:  jsii.String("ConsumerLogGroup"),
  RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
 }),
 Entry: jsii.String("consumer"),
})
key.GrantDecrypt(consumer)
secret.GrantRead(consumer, nil)

The overall logic with regards to Wildcard Principals is similar to above, the KMS Key and the Secret are set up with an agreed pattern of how the IAM Roles in the Producer stack will be called and the Consumer Lambda does not yet refer to the “real” URL as it doesn’t exist yet.

Producer

The Producer looks similar to the last scenario:

restApi := awsapigateway.NewRestApi(stack, jsii.String("API"), &awsapigateway.RestApiProps{})
restApi.Root().AddMethod(jsii.String("GET"), awsapigateway.NewMockIntegration(&awsapigateway.IntegrationOptions{
 IntegrationResponses: &[]*awsapigateway.IntegrationResponse{{
  StatusCode: jsii.String("200"),
 },
 },
 PassthroughBehavior: awsapigateway.PassthroughBehavior_NEVER,
 RequestTemplates: &map[string]*string{
  "application/json": jsii.String("{ \"statusCode\": 200 }"),
 },
}), &awsapigateway.MethodOptions{
 ApiKeyRequired: jsii.Bool(true),
 MethodResponses: &[]*awsapigateway.MethodResponse{
  {
   StatusCode: jsii.String("200"),
  },
 },
})

awsapigateway.NewDeployment(stack, jsii.String("Deployment"), &awsapigateway.DeploymentProps{
 Api: restApi,
})

keyArn := "KEY_ARN"
secretArn := "SECRET_ARN"
synchronizeFunction := awscdklambdagoalpha.NewGoFunction(stack, jsii.String("Synchronize"), &awscdklambdagoalpha.GoFunctionProps{
 FunctionName: jsii.String("Synchronize"),
 LogGroup: awslogs.NewLogGroup(stack, jsii.String("SynchronizeLogGroup"), &awslogs.LogGroupProps{
  LogGroupName:  jsii.String("SynchronizeLogGroup"),
  RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
 }),
 Entry: jsii.String("synchronize"),
})
synchronizeFunction.Role().AttachInlinePolicy(awsiam.NewPolicy(stack, jsii.String("SynchronizeRolePolicy"), &awsiam.PolicyProps{
 Statements: &[]awsiam.PolicyStatement{
  awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
   Actions: &[]*string{
    jsii.String("kms:GenerateDataKey"),
    jsii.String("kms:Encrypt"),
   },
   Resources: &[]*string{
    jsii.String(keyArn),
   },
  }),
  awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
   Actions: &[]*string{
    jsii.String("secretsmanager:PutSecretValue"),
   },
   Resources: &[]*string{
    jsii.String(secretArn),
   },
  }),
 },
}))

apiKey := awsapigateway.NewApiKey(stack, jsii.String("APIKey"), &awsapigateway.ApiKeyProps{})
apiKey.GrantRead(synchronizeFunction)

usagePlan := restApi.AddUsagePlan(jsii.String("UsagePlan"), &awsapigateway.UsagePlanProps{
 ApiStages: &[]*awsapigateway.UsagePlanPerApiStage{
  {Api: restApi, Stage: restApi.DeploymentStage()},
 },
})
usagePlan.AddApiKey(apiKey, &awsapigateway.AddApiKeyOptions{})

synchronizeApiKeyValue := customresources.NewAwsCustomResource(stack, jsii.String("SynchronizeAPIKeyValue"), &customresources.AwsCustomResourceProps{
 OnCreate: &customresources.AwsSdkCall{
  Service: jsii.String("Lambda"),
  Action:  jsii.String("invoke"),
  Parameters: &map[string]interface{}{
   "FunctionName": jsii.String("Synchronize"),
   "Payload":      jsii.Sprintf("{\"apiKeyId\":\"%s\",\"secretId\":\"%s\"}", *apiKey.KeyId(), secretArn),
  },
  PhysicalResourceId: customresources.PhysicalResourceId_Of(apiKey.KeyId()),
 },
 Policy: customresources.AwsCustomResourcePolicy_FromStatements(&[]awsiam.PolicyStatement{
  awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
   Actions: &[]*string{
    jsii.String("lambda:InvokeFunction"),
   },
   Resources: &[]*string{
    synchronizeFunction.FunctionArn(),
   },
  }),
 }),
})
synchronizeApiKeyValue.Node().AddDependency(synchronizeFunction, apiKey)

with the main difference that the KMS Key and Secret ARN are now coming from the Consumer stack. Other than that, the Synchronize Lambda again takes care of writing the API Key value to the Secret and after a deployment the Consumer Secret should contain the correct value.

Consumer

Finally, changing the Consumer stack by adding the newly created URL of the API Gateway allows us to invoke the Consumer and get the same 200 OK as above.

Summary

Both Pull and Push models work similar, accessing Secrets across account boundaries works fine. The decision where to eventually place the Secret is not clear cut as in both scenarios the stacks require knowledge of the respective other account and of course, Pull and Push could also even be combined by having a dedicated Credential account that is being written to by the Producer and read from by the Consumer. We haven’t covered the topic of notifying the Consumer on Secret changes but that could be easily done as part of the Synchronize Lambda, e.g. by sending a message to an SNS topic or similar.

Appendix

Synchronize

package main

import (
 "context"

 "github.com/aws/aws-lambda-go/lambda"
 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/service/apigateway"
 "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
 "github.com/aws/jsii-runtime-go"
)

type Request struct {
 ApiKeyId string `json:"apiKeyId"`
 SecretId string `json:"secretId"`
}

type Handler struct {
 ApiGatewayClient     *apigateway.Client
 SecretsManagerClient *secretsmanager.Client
}

func (handler *Handler) handleRequest(ctx context.Context, event Request) error {
 getApiKeyOutput, err := handler.ApiGatewayClient.GetApiKey(ctx, &apigateway.GetApiKeyInput{
  ApiKey:       aws.String(event.ApiKeyId),
  IncludeValue: jsii.Bool(true),
 })
 if err != nil {
  return err
 }

 _, err = handler.SecretsManagerClient.PutSecretValue(ctx, &secretsmanager.PutSecretValueInput{
  SecretId:     aws.String(event.SecretId),
  SecretString: aws.String(*getApiKeyOutput.Value),
 })
 if err != nil {
  return err
 }

 return nil
}

func main() {
 cfg, err := config.LoadDefaultConfig(context.TODO())
 if err != nil {
  panic(err)
 }

 apiGatewayClient := apigateway.NewFromConfig(cfg)
 secretsManagerClient := secretsmanager.NewFromConfig(cfg)

 handler := Handler{
  ApiGatewayClient:     apiGatewayClient,
  SecretsManagerClient: secretsManagerClient,
 }

 lambda.Start(handler.handleRequest)
}

Consumer

package main

import (
 "context"
 "io"
 "net/http"
 "net/url"
 "os"

 "github.com/aws/aws-lambda-go/lambda"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
 "github.com/aws/jsii-runtime-go"
)

type Handler struct {
 SecretId string
 Client   *secretsmanager.Client
 Url      string
}

func (handler *Handler) handleRequest(ctx context.Context) (string, error) {
 getSecretValueOutput, err := handler.Client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
  SecretId: jsii.String(handler.SecretId),
 })
 if err != nil {
  return "", err
 }

 client := http.Client{}

 parsedUrl, err := url.Parse(handler.Url)
 if err != nil {
  return "", err
 }

 request := http.Request{
  Method: "GET",
  URL:    parsedUrl,
  Header: map[string][]string{
   "x-api-key": {*getSecretValueOutput.SecretString},
  },
 }
 response, err := client.Do(&request)
 if err != nil {
  return "", err
 }
 defer func(Body io.ReadCloser) {
  err := Body.Close()
  if err != nil {
  }
 }(response.Body)

 return response.Status, nil
}

func main() {
 cfg, err := config.LoadDefaultConfig(context.TODO())
 if err != nil {
  panic(err)
 }

 client := secretsmanager.NewFromConfig(cfg)

 handler := Handler{
  SecretId: os.Getenv("SECRET_ID"),
  Client:   client,
  Url:      os.Getenv("URL"),
 }

 lambda.Start(handler.handleRequest)
}

On June 11, 2026 AWS announced the AWS Workload Credentials Provider (AWS WCP), a client-side application that provides unified access to AWS services such as SecretsManager and ACM for various Serverless deployment models (EC2, ECS, Lambda) so why not try that with AWS Lambdas? In addition to being a facade for SecretsManager, the AWS WCP caches Secrets and allows users to pre-fetch Secrets and also to use a specific IAM role that differs from the Lambdas’ role to read those Secrets.

Introduction

In the following, we will be looking at Setup, Programming Model and Runtime Characteristics of four different examples: Direct (loading secrets with the AWS SDK in a Lambda the usual way), Indirect (loading secrets via the workload credentials provider), IndirectPreFetch (loading secrets via the workload credentials provider with pre-fetching them on startup) and IndirectPreFetchAssumeRole (loading secrets via the workload credentials provider with pre-fetching them on startup using an IAM role separate from the Lambda IAM role).

All four Lambdas invoke a REST API that requires an API Key stored in a Secret, obtained with lazy loading and stored within the Lambdas’ memory for subsequent invocations. The Lambdas further handle the error case of receiving a HTTP 403 Forbidden from the API when the API Key has been “rotated” by re-loading the secret on demand before trying the request one more time to compare whether that behaviour has an effect on the Programming Model.

Setup

All four Lambdas are setup with CDK, the actual Lambda code is added further below.

Direct

The setup for the Direct Lambda is straightforward:

const directLambda = new NodejsFunction(this, 'DirectLambda', {
    environment: {
        'SECRET_ARN': secret.secretArn,
    },
    functionName: 'Direct',
    logGroup: new LogGroup(this, 'DirectLogGroup', {
        logGroupName: 'DirectLogGroup',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
    entry: './direct/index.ts',
    handler: 'handler'
});
secret.grantRead(directLambda);

No surprises here. A Lambda, the ARN of the Secret is passed in, we’re using a CDK Construct to build the Lambda from TypeScript and grant read access to the Secret.

Indirect

To setup the Indirect Lambda we first need to build the AWS Workload Credentials Provider, see the instructions at https://github.com/aws/aws-workload-credentials-provider/tree/main. As I am on a MacBook I had been using cross to cross-compile for Linux, and needed the following as Cross.toml due to a gcc issue:

[target.x86_64-unknown-linux-gnu]
image = "ghcr.io/cross-rs/x86_64-unknown-linux-gnu:edge"
pre-build = [
    'apt-get update && apt-get -y upgrade',
    # Need to upgrade gcc to a version that does have a bug (because aws-lc-sys's custom build script errors with this:
    #  ### COMPILER BUG DETECTED ###
    #  Your compiler (cc) is not supported due to a memcmp related bug reported in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95189. We strongly recommend against using this compiler.
    'apt-get -y install gcc-10 && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 60',
]

afterwards, build it with

cross build --release --target=x86_64-unknown-linux-gnu

and we can follow the instructions for AWS Lambda including creating a Lambda Layer containing the extension.

The setup itself is then relatively straightforward again:

const layerVersion = LayerVersion.fromLayerVersionArn(this, 'LayerVersion', '<LAYER-ARN>');

const indirectLambda = new NodejsFunction(this, 'IndirectLambda', {
    environment: {
        'SECRET_ID': secret.secretName,
    },
    functionName: 'Indirect',
    memorySize: 1024,
    layers: [
        layerVersion
    ],
    logGroup: new LogGroup(this, 'IndirectLogGroup', {
        logGroupName: 'IndirectLogGroup',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
    entry: './indirect/index.ts',
    handler: 'handler'
});
secret.grantRead(indirectLambda);

IndirectPreFetch

If we want to make use of the pre-fetching functionality we need to adjust the Extension configuration:

Putting the following into a config.toml file

[[capabilities.secrets_manager.prefetch.secrets]]
secret_id = "<SECRET-ARN>"

adjusting the Extensions’ shell script to use it

/opt/bin/aws-workload-credentials-provider sm start --config /opt/bin/config.toml &

and making sure the config file ends up in the Lambda Layer should do it.

The main difference to the Indirect scenario is that the Lambda role needs the permission secretsmanager:BatchGetSecretValue :

const indirectLambdaPreFetch = new NodejsFunction(this, 'IndirectLambdaPreFetch', {
    environment: {
        'SECRET_ID': '<SECRET-ID>',
    },
    functionName: 'IndirectPreFetch',
    memorySize: 1024,
    layers: [
        LayerVersion.fromLayerVersionArn(this, 'LayerVersionPreFetch', '<LAYER-ARN>')
    ],
    logGroup: new LogGroup(this, 'IndirectPreFetchLogGroup', {
        logGroupName: 'IndirectPreFetchLogGroup',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
    entry: './indirect/index.ts',
    handler: 'handler'
});
indirectLambdaPreFetch.role!.attachInlinePolicy(new Policy(this, 'IndirectLambdaPreFetchRole', {
    statements: [new PolicyStatement({
        actions: ['secretsmanager:DescribeSecret', 'secretsmanager:GetSecretValue', 'secretsmanager:BatchGetSecretValue'],
        resources: ['<SECRET-ARN>']
    })]
}))

The rest is identical with the Indirect scenario. Note that the Lambda role still needs to be able to read the Secret as before, as the Extension runs with the role of the Lambda.

IndirectPreFetchAssumeRole

Similar to the previous scenario, we need a config file:

[[capabilities.secrets_manager.prefetch.secrets]]
secret_id = "<SECRET-ARN>"
role_arn = "<ROLE-ARN>"

This time with a Secret and a Role.

Again as the previous scenario, the base setup is similar, the permissions change though cause the Lambda no longer needs to be able to read the Secret but to assume the role:

const secretRoleArn = '<SECRET-ROLE-ARN>';
const indirectLambdaPreFetchAssumeRole = new NodejsFunction(this, 'IndirectLambdaPreFetchAssumeRole', {
    environment: {
        'SECRET_ID': '<SECRET-ID>',
        'ROLE_ARN': secretRoleArn
    },
    functionName: 'IndirectPreFetchAssumeRole',
    memorySize: 1024,
    layers: [
        LayerVersion.fromLayerVersionArn(this, 'LayerVersionPreFetchAssumeRole', '<LAYER-ARN>')
    ],
    logGroup: new LogGroup(this, 'IndirectPreFetchAssumeRoleLogGroup', {
        logGroupName: 'IndirectPreFetchAssumeRoleLogGroup',
        removalPolicy: RemovalPolicy.DESTROY,
    }),
    entry: './indirect/index.ts',
    handler: 'handler'
});
indirectLambda1024PreFetchAssumeRole.role!.attachInlinePolicy(new Policy(this, 'IndirectLambdaPreFetchAssumeRoleRole', {
    statements: [
        new PolicyStatement({
            actions: ['sts:AssumeRole'],
            resources: [secretRoleArn]
        })
    ]
}))

Programming Model

To look at the differences in the Programming Models lets look at the Direct scenario first:

async function ensureSecretLoaded() {
    if (secretsManagerClient === undefined) {
        secretsManagerClient = new SecretsManagerClient();
    }

    if (secretValue === undefined) {
        const getSecretValueCommand = new GetSecretValueCommand({
            SecretId: secretArn
        });
        const getSecretValueCommandOutput = await secretsManagerClient.send(getSecretValueCommand);
        secretValue = getSecretValueCommandOutput.SecretString!
    }

    return secretValue;
}

Here we initialize the Secrets Manager on demand and the Secret afterwards. If the request fails and we need to enforce a reload of the Secret it’s sufficient to set the secretValue to undefined and invoke ensureSecretLoaded again to reload it.

In the Indirect scenarios reloading the Secret doesn’t require invalidation and instead we force a reload via the refreshNow parameter:

async function ensureSecretLoaded(refreshNow: boolean) {
    if (refreshNow || secretValue === undefined) {
        const response = await fetch(`http://localhost:2773/secretsmanager/get?secretId=${secretId}${roleArn === undefined ? '' : `&roleArn=${roleArn}`}&refreshNow=${refreshNow}`, {
            headers: {
                'X-Aws-Parameters-Secrets-Token': awsSessionToken
            }
        });

        secretValue = (await response.json() as GetSecretValueResponse).SecretString;
    }

    return secretValue;
}

Here we initialize secretValue either if forced via the flag or if it’s undefined on demand, if forced we ask the Extension to reload it to make sure it’s no longer stale in the Extensions’ cache.

Runtime Characteristics

The first big difference between Direct and the Indirect scenarios is that Direct can run with minimal settings (128MB, 3s timeout) whilst the Indirect scenarios won’t even succeed in orderly fashion with that, and instead error. Using the learnings from https://medium.com/@jconning/aws-lambda-faster-is-cheaper-6bf32f58d741 and https://dev.to/neverendingqs/aws-lambda-always-allocate-at-least-1024-mb-349k I eventually used 1024MB memory as the default for all scenarios (see the CDK setup above).

When it comes to the comparison of the Cold and Warm starts between Direct, Indirect, IndirectPreFetch and IndirectPreFetchAssumeRole the tests have shown that these four scenarios are roughly splitting into two groups: The Direct scenario requires ~100MB memory, ~500ms for Cold starts and ~30–40ms for Warm starts. The Indirect scenarios require ~120–140MB memory, ~770–810ms for Cold starts and ~60–80ms for Warm starts. I would assume the higher memory usage and start times for the Indirect Scenarios originate in the Lambda Layer extension running alongside the Lambda although the differences in absolute numbers, especially compared to the Direct scenario, aren’t gigantic.

Summary

Whilst the setup of the AWS Workload Credentials Provider is not very complex, the fact that for the scenarios PreFetch and PreFetchAssumeRole the configuration needs to be baked into the Lambda Layer makes it not very flexible, the Programming Model is very similar and comes without any surprises and the Runtime Characteristics are also acceptable.

Personally the overhead of having to include the config for pre-fetching and assuming of roles wouldn’t outweigh the benefits of a unified interface for loading Secrets but if that will be supported via e.g. environment variables in the future I would be more than happy.

Appendix

direct/index.ts

import {GetSecretValueCommand, SecretsManagerClient} from '@aws-sdk/client-secrets-manager'

const secretArn = process.env.SECRET_ARN!;

let secretsManagerClient: SecretsManagerClient | undefined = undefined;
let secretValue: string | undefined;

function invalidateSecret() {
    secretValue = undefined;
}

async function ensureSecretLoaded() {
    if (secretsManagerClient === undefined) {
        secretsManagerClient = new SecretsManagerClient();
    }

    if (secretValue === undefined) {
        const getSecretValueCommand = new GetSecretValueCommand({
            SecretId: secretArn
        });
        const getSecretValueCommandOutput = await secretsManagerClient.send(getSecretValueCommand);
        secretValue = getSecretValueCommandOutput.SecretString!
    }

    return secretValue;
}

async function performRequest(url: string) {
    return await fetch(url, {
        headers: {
            'x-api-key': await ensureSecretLoaded(),
        }
    })
}

exports.handler = async (event: { url: string }) => {
    const response = await performRequest(event.url);

    if (response.status === 403) {
        invalidateSecret();
        const response = await performRequest(event.url);

        return response.status;
    } else {
        return response.status;
    }
}

indirect/index.ts

const awsSessionToken = process.env.AWS_SESSION_TOKEN!
const secretId = process.env.SECRET_ID!;
const roleArn = process.env.ROLE_ARN;

interface GetSecretValueResponse {
    SecretString: string;
}

let secretValue: string | undefined;

async function ensureSecretLoaded(refreshNow: boolean) {
    if (refreshNow || secretValue === undefined) {
        const response = await fetch(`http://localhost:2773/secretsmanager/get?secretId=${secretId}${roleArn === undefined ? '' : `&roleArn=${roleArn}`}&refreshNow=${refreshNow}`, {
            headers: {
                'X-Aws-Parameters-Secrets-Token': awsSessionToken
            }
        });

        secretValue = (await response.json() as GetSecretValueResponse).SecretString;
    }

    return secretValue;
}

async function performRequest(url: string, refreshNow: boolean) {
    return await fetch(url, {
        headers: {
            'x-api-key': await ensureSecretLoaded(refreshNow),
        }
    })
}

exports.handler = async (event: { url: string }) => {
    const response = await performRequest(event.url, false);

    if (response.status === 403) {
        const response = await performRequest(event.url, true);

        return response.status;
    } else {
        return response.status;
    }
}