Caching Api Response At Edge
The title idea kicked when I saw our APM platform a static/config api getting hit 1200req/sec. So I thought let’s move this api to the edge so the server will be free of TCP connection, memory, and computing power. Started exploring how to serve this api from the edge.
The Advantage of Caching API at Edge
Improve latency.
Run the code closer to your customer.
Keep the server free.
DOS attack free.
The disadvantage of Caching API at Edge
We don’t have control over data.
To monitor CDN we need to pay extra costs to AWS.
Solution
After a lot of front and back. Decided to serve the static/config api response through s3.
Why s3?
- Let’s assume 1k-2k req/sec hit is coming to that api. we invalidate the distribution suddenly cache miss-hit come to the server. These can overwhelm our server having cascading effect to increase in the response time which leads to a bad customer experience. By using s3 we can avoid this issue.
- Our existing api is in graphQL we need to write a rest api / Lamba to support the CDN distribution part. It will add up the extra cost
Hence, Decided to use S3 for distribution.
Architecture
Invalidation approach
- Upload updated JSON data in s3.
- Trigger the invalidation using api.
Problem we faced
I’m ready with the CDN endpoint then my manager told let give a load test and see the stability. We saw the response is too high than we expect after doing all this we are disappointed. So decided to send mail to the AWS team as we are using CloudFront CDN to understand the problem.
Aws team comments in their own words
Traditional load testing methods don’t work well with Cloudfront. The reason is that CloudFront uses DNS to balance loads across geographically dispersed edge locations and within each edge location. When a request for content is sent to CloudFront from a client, the client receives a DNS response that includes a set of IP addresses. If you test by sending requests to just one of the IP addressed that DNS returns, you are testing only a small subset of the resources in one CloudFront edge location, which doesn’t accurately represent actual traffic patterns. The performance of the small subset of CloudFront servers may be degraded depending on the volume of data requested.
To conclude, CloudFront is designed to scale for viewers that have different client IP addresses and different DNS resolvers across multiple geographic regions.
To perform load testing that accurately assesses CloudFront performance, we recommend that you do all of the following:
- Send client requests from multiple geographic regions.
- Configure your test so each client makes an independent DNS request; each client will then receive a different set of IP addresses from DNS.
- For each client that is making requests, spread your client requests across the set of IP addresses that are returned by DNS, which ensures that the load is distributed across multiple servers in a CloudFront edge location.
Aws team helped us with the load test and we are satisfied with the response time.
Key takeaway
- Learned caching the api response at the edge
- Load testing for CloudFront.
Summary
To summarise, we have created CDN distribution with s3. S3 will have our response JSON. whenever we need to invalidate we upload new JSON data and invalidate CDN distribution.
Using this approach the server will be free of TCP/HTTP connection, memory, and computing power.
