As part of the Skyscanner ambassador group in Barcelona, we support the local software community by helping organize different events. On May 9th the Barcelona AWS meetup was held at the office.
There were two talks in the meetup, both dealing with serverless computing:
1st talk
The first talk by Serhat Can dealt with serverless architectures on AWS Lambda. During the talk it was presented the experience of OpsGenie with serverless architectures for their alerting and monitoring service.
OpsGenie is an alerting service somehow similar to the one that VictorOps offers. At Skyscanner we make use of VictorOps for alerting, on-call rotation and 24 hour support.
One of the use cases shown that caught my attention was the DynamoDB autoscaling system they have in place. Some of the cool features along it include a configurable DynamoDB table alarm in Cloudwatch that is updated with the increase/decrease of the table capacity.
At Skyscanner hotels we faced similar issues in the past as we make an intensive use of DynamoDB for the service that manages the sorting of hotel partners in the search results in case of price parity. The algorithm behind this computation, that we call Partner Sorting Algorithm serves the precomputed positions for each partner from a DynamoDB table.
Another familiar example was the use of Lambdas for replicating data across regions for DynamoDB and ElasticSearch aiming to decouple the replication logic from the application logic. This solution did not work for us due to the large size of our tables and the long time it took to replicate the data between regions.
Finally it was mentioned the challenges that cloud-first services pose such as proper monitoring, high availability, replication, cold start time and blue-green deployment (that we manage through Slingshot).
Overall it was really interesting listening to their experience as I can closely relate to many of the problems that surfaced to them while developing our own cloud-native services.
2nd talk
The second talk focused on serverless data processing with Data Pipeline, Docker, Lambda, API Gateway, ECR and SNS by one of my colleagues at Skycanner, Radek Miazio.
Radek talked about the different AWS services that could be used for the processing they needed to do in order to create an index for Skyscanner’s internal representation of the world. After discarding many of the AWS services due to their limitations, they settled up with Data Pipeline and explained some of the problems they faced such as:
- Complex setup due to the massive JSON
- The long initialization that caused API Gateway to timeout (service cold start again)
- Long delay in logs (up to 20 minutes)
- Limits to the infinite resources that the cloud provides
Final thoughts
From both talks it could be distilled that when designing a serverless service the solution has to take into account the constraints of the still immature services offered which sometimes are not clearly laid out in the docs.
Finally these experiences also surface that the vast computing resources that the cloud provides outgrow the intricacies of developing ad-hoc services coupled to the available proprietary services.
The meetup was recorded and is available in Youtube: