← Blog 06. Februar 2019 · Amazon Web Services · Snippets

Implement Kinesis Firehose S3 delivery preprocessed by Lambda in AWS CloudFormation

Recently I have experimented a little with Kinesis Firehose. The result is this concise undocumented template which setup an Kinesis Firehose S3 Delivery Stream preprocessed by Lambda in AWS CloudFormation.

Template:

AWSTemplateFormatVersion: 2010-09-09
Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: Enabled

  DeliveryRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: ''
            Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: 'sts:AssumeRole'
            Condition:
              StringEquals:
                'sts:ExternalId': !Ref 'AWS::AccountId'
             
  DeliveryPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: firehose_delivery_policy
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 's3:AbortMultipartUpload'
              - 's3:GetBucketLocation'
              - 's3:GetObject'
              - 's3:ListBucket'
              - 's3:ListBucketMultipartUploads'
              - 's3:PutObject'
            Resource:
              - !Sub 'arn:aws:s3:::${S3Bucket}'
              - !Sub 'arn:aws:s3:::${S3Bucket}*'
      Roles:
        - !Ref DeliveryRole

  InvokeLambdaPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: firehose_lambda_policy
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 'lambda:InvokeFunction'
            Resource: 
              - !GetAtt ProcessLambdaFunction.Arn
      Roles:
        - !Ref DeliveryRole

  DeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      ExtendedS3DestinationConfiguration:
        BucketARN: !Sub 'arn:aws:s3:::${S3Bucket}'
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 50
        CompressionFormat: UNCOMPRESSED
        Prefix: firehose/
        RoleARN: !GetAtt DeliveryRole.Arn
        ProcessingConfiguration:
          Enabled: true
          Processors:
            - Parameters:
                - ParameterName: LambdaArn
                  ParameterValue: !GetAtt ProcessLambdaFunction.Arn 
              Type: Lambda 

  ProcessLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
        - PolicyName: allowLambdaLogs
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - logs:*
              Resource: arn:aws:logs:*:*:*

  ProcessLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: >
          console.log('Loading function');

          exports.handler = function(event, context) {
              console.log(JSON.stringify(event));
              let records = [];
              for(let i = 0; i<event.records.length; i++) {
                 let payload = new Buffer(event.records[i].data, 'base64').toString('ascii');
                 payload = JSON.parse(payload);
                 payload.decoded = true;
                 records.push({
                   recordId: event.records[i].recordId,
                   result: 'Ok',
                   data: Buffer.from(JSON.stringify(payload)).toString('base64')});
              }
              console.log(`Return: ${ JSON.stringify({records}) }`)
              return Promise.resolve({records});
          };
      Handler: index.handler
      Role: !GetAtt ProcessLambdaExecutionRole.Arn
      Runtime: nodejs8.10
      Timeout: 60
      MemorySize: 128

Quelle: Gist auf GitHub

Create a Record on the Delivery Stream using a simple Node.js application:

const AWS = require("aws-sdk");

AWS.config.apiVersions = {
    kinesis: '2013-12-02'
};
  
const firehose = new AWS.Firehose({
    region: "us-east-1"
});


const main = async() => {

    const message = {"message":"abc" };

    var params = {
        Record: {
            Data: JSON.stringify(message)
        },
        DeliveryStreamName: 'test-DeliveryStream-1PMIBGYAJIE1'
      };

     let data = await firehose.putRecord(params).promise();
     console.log("Done: "+JSON.stringify(data));

}

main();

Quelle: Gist auf GitHub

Output on S3: Implement Kinesis Firehose S3 delivery preprocessed by Lambda in AWS CloudFormationn