Mounting S3 Files into AWS Lambda
In the past, whilst the integration of AWS Lambdas and S3 was standardised through the AWS SDK, reading objects (or, files) sometimes felt a bit overcomplicated, especially when comparing with file systems but as of two days ago, AWS now supports mounting S3 Buckets via S3 Files for Lambdas as file systems. In the following we will be looking at a simple end to end example of a Lambda incorporating mounted S3 Files.
Lambda
The Lambda will perform a simple greeting by reading the text to greet the invoker with from S3:
Get Johannes Geiger’s stories in your inbox
Join Medium for free to get updates from this writer.
greeting/main.go
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-lambda-go/lambda"
)
const GreetingFile = "greeting.txt"
type Handler struct {
MountPath string
}
func (handler *Handler) handleRequest(_ context.Context) (string, error) {
bytes, err := os.ReadFile(fmt.Sprintf("%s/%s", handler.MountPath, GreetingFile))
if err != nil {
return "", err
}
return string(bytes), nil
}
func main() {
handler := Handler{
MountPath: os.Getenv("MOUNT_PATH"),
}
lambda.Start(handler.handleRequest)
}It expects the path to be passed in via environment variable MOUNT_PATH and reads the file greeting.txt. Executing the Lambda on its own would lead to an error that the file doesn't exist.
Stack
To deploy the Lambda I have been using CDK again as it allows me to define the infrastructure including the Lambda easily:
vpc := awsec2.NewVpc(stack, jsii.String("VPC"), &awsec2.VpcProps{})
bucket := awss3.NewBucket(stack, jsii.String("Bucket"), &awss3.BucketProps{
Versioned: jsii.Bool(true),
})
role := awsiam.NewRole(stack, jsii.String("FileSystemRole"), &awsiam.RoleProps{
AssumedBy: awsiam.NewServicePrincipal(jsii.String("elasticfilesystem.amazonaws.com"), &awsiam.ServicePrincipalOpts{}),
})
role.AddToPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{
jsii.String("s3:ListBucket*"),
},
Resources: &[]*string{
bucket.BucketArn(),
},
}))
role.AddToPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{
jsii.String("s3:AbortMultipartUpload"),
jsii.String("s3:DeleteObject"),
jsii.String("s3:GetObject*"),
jsii.String("s3:List*"),
jsii.String("s3:PutObject*"),
},
Resources: &[]*string{
bucket.ArnForObjects(jsii.String("*")),
},
}))
role.AddToPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{
jsii.String("events:DeleteRule"),
jsii.String("events:DisableRule"),
jsii.String("events:EnableRule"),
jsii.String("events:PutRule"),
jsii.String("events:PutTargets"),
jsii.String("events:RemoveTargets"),
},
Resources: &[]*string{
jsii.String(fmt.Sprintf("arn:%s:events:*:*:rule/DO-NOT-DELETE-S3-Files*", *awscdk.Aws_PARTITION())),
},
Conditions: &map[string]interface{}{
"StringEquals": map[string]*string{
"events:ManagedBy": jsii.String("elasticfilesystem.amazonaws.com"),
},
},
}))
role.AddToPolicy(awsiam.NewPolicyStatement(&awsiam.PolicyStatementProps{
Actions: &[]*string{
jsii.String("events:DescribeRule"),
jsii.String("events:ListRuleNamesByTarget"),
jsii.String("events:ListRules"),
jsii.String("events:ListTargetsByRule"),
},
Resources: &[]*string{
jsii.String(fmt.Sprintf("arn:%s:events:*:*:rule/*", *awscdk.Aws_PARTITION())),
},
}))
fileSystem := awss3files.NewCfnFileSystem(stack, jsii.String("FileSystem"), &awss3files.CfnFileSystemProps{
Bucket: bucket.BucketArn(),
RoleArn: role.RoleArn(),
})
securityGroup := awsec2.NewSecurityGroup(stack, jsii.String("MountTargetSecurityGroup"), &awsec2.SecurityGroupProps{
Vpc: vpc,
})
for index, subnet := range *vpc.PrivateSubnets() {
awss3files.NewCfnMountTarget(stack, jsii.String(fmt.Sprintf("MountTarget-%d", index)), &awss3files.CfnMountTargetProps{
FileSystemId: fileSystem.AttrFileSystemId(),
SubnetId: subnet.SubnetId(),
SecurityGroups: &[]*string{
securityGroup.SecurityGroupId(),
},
})
}
accessPoint := awss3files.NewCfnAccessPoint(stack, jsii.String("AccessPoint"), &awss3files.CfnAccessPointProps{
FileSystemId: fileSystem.AttrFileSystemId(),
RootDirectory: &awss3files.CfnAccessPoint_RootDirectoryProperty{
Path: jsii.String("/greeting-lambda"),
CreationPermissions: &awss3files.CfnAccessPoint_CreationPermissionsProperty{
OwnerGid: jsii.String("1001"),
OwnerUid: jsii.String("1001"),
Permissions: jsii.String("750"),
},
},
PosixUser: &awss3files.CfnAccessPoint_PosixUserProperty{
Gid: jsii.String("1001"),
Uid: jsii.String("1001"),
},
})
mountPath := jsii.String("/mnt/s3files")
awscdklambdagoalpha.NewGoFunction(stack, jsii.String("Greeting"), &awscdklambdagoalpha.GoFunctionProps{
Environment: &map[string]*string{
"MOUNT_PATH": mountPath,
},
Filesystem: awslambda.FileSystem_FromS3FilesAccessPoint(accessPoint, mountPath),
FunctionName: jsii.String("Greeting"),
LogGroup: awslogs.NewLogGroup(stack, jsii.String("GreetingLogGroup"), &awslogs.LogGroupProps{
LogGroupName: jsii.String("GreetingLogGroup"),
RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
}),
Vpc: vpc,
Entry: jsii.String("greeting"),
})
awscdk.NewCfnOutput(stack, jsii.String("BucketName"), &awscdk.CfnOutputProps{
Value: bucket.BucketName(),
Key: jsii.String("BucketName"),
})Unfortunately defining the file system, the access point and the mount targets are only possible with L1 constructs in CDK which makes the code quite noisy but apart from that it’s relatively straightforward.
Greeting
Now we just need to define a greeting.txt with the right content and copy it into the right place:
echo -n "Hello World\!" > greeting.txt \
&& aws s3 cp greeting.txt s3://$BUCKET_NAME/greeting-lambda/greeting.txtand on invoking the Lambda we will receive the expected response.
Summary
Mounting S3 Files into Lambdas is relatively straightforward and the latency is acceptable (between 15 and 90ms in my tests), handling files should be an established pattern in all of the programming languages used for Lambdas so this looks to be a decent alternative to the S3 SDK if flexible file access is necessary.









