Implement Kinesis Firehose S3 delivery preprocessed by Lambda in AWS CloudFormation
Recently I have experimented a little with Kinesis Firehose. The result is this concise undocumented template which setup an Kinesis Firehose S3 Delivery Stream preprocessed by Lambda in AWS CloudFormation.
Template:
AWSTemplateFormatVersion: 2010-09-09
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
VersioningConfiguration:
Status: Enabled
DeliveryRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: 'sts:AssumeRole'
Condition:
StringEquals:
'sts:ExternalId': !Ref 'AWS::AccountId'
DeliveryPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: firehose_delivery_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:AbortMultipartUpload'
- 's3:GetBucketLocation'
- 's3:GetObject'
- 's3:ListBucket'
- 's3:ListBucketMultipartUploads'
- 's3:PutObject'
Resource:
- !Sub 'arn:aws:s3:::${S3Bucket}'
- !Sub 'arn:aws:s3:::${S3Bucket}*'
Roles:
- !Ref DeliveryRole
InvokeLambdaPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: firehose_lambda_policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'lambda:InvokeFunction'
Resource:
- !GetAtt ProcessLambdaFunction.Arn
Roles:
- !Ref DeliveryRole
DeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
ExtendedS3DestinationConfiguration:
BucketARN: !Sub 'arn:aws:s3:::${S3Bucket}'
BufferingHints:
IntervalInSeconds: 60
SizeInMBs: 50
CompressionFormat: UNCOMPRESSED
Prefix: firehose/
RoleARN: !GetAtt DeliveryRole.Arn
ProcessingConfiguration:
Enabled: true
Processors:
- Parameters:
- ParameterName: LambdaArn
ParameterValue: !GetAtt ProcessLambdaFunction.Arn
Type: Lambda
ProcessLambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: allowLambdaLogs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
ProcessLambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: >
console.log('Loading function');
exports.handler = function(event, context) {
console.log(JSON.stringify(event));
let records = [];
for(let i = 0; i<event.records.length; i++) {
let payload = new Buffer(event.records[i].data, 'base64').toString('ascii');
payload = JSON.parse(payload);
payload.decoded = true;
records.push({
recordId: event.records[i].recordId,
result: 'Ok',
data: Buffer.from(JSON.stringify(payload)).toString('base64')});
}
console.log(`Return: ${ JSON.stringify({records}) }`)
return Promise.resolve({records});
};
Handler: index.handler
Role: !GetAtt ProcessLambdaExecutionRole.Arn
Runtime: nodejs8.10
Timeout: 60
MemorySize: 128
Quelle: Gist auf GitHub
Create a Record on the Delivery Stream using a simple Node.js application:
const AWS = require("aws-sdk");
AWS.config.apiVersions = {
kinesis: '2013-12-02'
};
const firehose = new AWS.Firehose({
region: "us-east-1"
});
const main = async() => {
const message = {"message":"abc" };
var params = {
Record: {
Data: JSON.stringify(message)
},
DeliveryStreamName: 'test-DeliveryStream-1PMIBGYAJIE1'
};
let data = await firehose.putRecord(params).promise();
console.log("Done: "+JSON.stringify(data));
}
main();
Quelle: Gist auf GitHub
Output on S3:
