Cutting Costs with AWS Lambda for Highly Scalable Image Processing

This blog is based on a POC and cost analysis that was done for a project for Raksul Hackweek 2022. https://recruit.raksul.com/story/hack-week-2022/ There are three of us who took part in this initiative. We are Nguyen Luong Hoang, Anh Tran Tuan and Afshad Eddy Dinshaw.

Summary:

Currently, some of the departments at Raksul are using ECS in AWS cloud to host image processing services such as background removal and resizing. The existing ECS setup in the cloud is resulting in higher costs even though the services are not in constant use. We did a cost analysis of our existing infrastructure in ECS versus potentially moving to AWS Lambda. We discovered that the move would significantly reduce infrastructure cost while improving performance. We did a POC as well as part of the Hackweek challenge to demonstrate hosting a simple image resizing service in AWS Lambda.

1. Goal: Reduce infrastructure costs associated with AWS ECS:

ECS requires a pre-estimated fixed number of containers be available even if they are not being used.
AWS Lambda offer:
- pay as you go
- highly scalable
- performance improvement.

All the above lead to reduced costs.

2. Cost Analysis and Evaluation:

Following is a breakdown of the costs associated with the existing infrastructure vs moving to AWS Lambda. We have 1 cluster with 30 tasks

Total cost: 1674.59 USD per month

Application in ECS

Estimated Cost Breakdown - AWS Lambda

AWS Lambda will serve based on number of requests. We estimate that one of our apps could have 700 submissions per day. Each submission can required up to 4 jobs/requests to generate images. That amounts to 3000 requests/day. So in total we could have around 8000 requests per day required to generate/convert images at current. Each request takes ~ 30seconds

So it costs 489.22 USD per month for all products in all environments using AWS Lambda.

Cost Effective - Amazon Lambda

To consider: If number of requests increasing, it will required more cost for amazon lambda (base on above calculation)

For ECS, the cost will not change. But if number of concurrency requests increasing, we may need to add more tasks/containers to the cluster. At this time, the cost of ECS will increase.

3. Performance Upgrade with increased scalability - AWS Lambda

We have seen a significant increase in performance switching to AWS Lambda. The throughput has increased due to the scalability of AWS Lambda.

Current ECS Fargate system → 90 concurrent requests. (30 containers * 3 jobs)

AWS Lambda → 1000 concurrent requests.

4. POC - Image Resize Workflow - AWS Lambda

How it works: We use ImageMagick for image processing. AWS Lambda Now Supports Up to 10 GB Ephemeral Storage. So it's enough space for ImageMagick works even with AI or PDF files. Thanks to the open source project https://github.com/serverlesspub/imagemagick-aws-lambda-2 . It's really helpful and we have the base to build, custom for the lambda layer, especially in the short duration of the HackWeek.

The lambda function: We created a simple lambda function that accepts two params: the object key (s3 key of the input image) and maxDimension (size that image want to convert to). After converting the image, the lambda will put the final image to the output bucket and publish an SNS notification.

var AWS = require("aws-sdk");

const s3Util = require('./s3-util'),
    childProcessPromise = require('./child-process-promise'),
    queryString = require('querystring'),
    path = require('path'),
    os = require('os'),
    EXTENSION = process.env.EXTENSION,
    OUTPUT_BUCKET = process.env.OUTPUT_BUCKET,
    INPUT_BUCKET = process.env.INPUT_BUCKET,

exports.handler = function (eventObject, context) {
    const key = eventObject.key,
        maxDimension = eventObject.maxDimension,
        inputBucket = INPUT_BUCKET,
        id = context.awsRequestId,
        resultKey = key.replace(/\.[^.]+$/, EXTENSION),
        workdir = os.tmpdir(),
        inputFile = path.join(workdir,  id  + path.extname(key)),
        outputFile = path.join(workdir, 'converted-' + id + EXTENSION);

    var sns = new AWS.SNS();
    var params = {
        Message: resultKey, 
        Subject: "image-resize-done",
        TopicArn: "arn:aws:sns:ap-northeast-1:************:image-resize-done"
    };
    
    console.log('converting', `${INPUT_BUCKET}`, key, 'using', inputFile);
    return s3Util.downloadFileFromS3(inputBucket, key, inputFile)
        .then(() => childProcessPromise.spawn(
            '/opt/bin/convert',
            [inputFile, '-resize', `${maxDimension}x${maxDimension}>`, outputFile],
            {env: process.env, cwd: workdir}
        ))
        .then(() => s3Util.uploadFileToS3(OUTPUT_BUCKET, resultKey, outputFile, MIME_TYPE))
        .then(() => {    
                      sns.publish(params, context.done);
                      return resultKey});


};

Invoking the lambda function from client code:

  RESIZE_FUNCTION = 'imagemagick-image-generator-la-ResizeImageFunction-******'

  def resize_image_function
    lambda_client.invoke(function_name: RESIZE_FUNCTION, payload: payload)
  end

  def payload
    JSON.generate(key: key, maxDimension: max_dimension)
  end

  def lambda_client
    @lambda_client ||= Aws::Lambda::Client.new(credentials: credentials)
  end

  def credentials
    @credentials ||= Aws::Credentials.new('*****', '*****')
  end

Post processing: Send an SNS notification and we can retrieve the image base on the key from the output bucket to continue our processing.

Conclusion

During the HackWeek, we created a POC to integrate the simple lambda above with our application for image resizing. TODOs: Migration/Apply for all kinds of image processing in our system.

We successfully demonstrated that AWS lambda works well for image processing. It not only helps to save cost but also provides better scalability for the project. Our cost analysis has shown that there is a significant reduction in cost when moving from ECS to Lambda.