While HTTP3 does provide some improvements, it's clear that further optimizations are needed, particularly for users in Europe and Asia.
One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month.
As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center.
It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency.
Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further.
---
Cost estimates
Assumptions:
1. Dropbox's peak traffic is 1,500 queries per second (QPS).
2. Average data transfer per query is 100 KB.
3. 50% of the traffic comes from North America, 25% from Europe, and 15% from Asia (remaining 10% from other regions, for pricing purposes put it into Asia's calculations).
---
1) AWS CloudFront cost estimation:
Data transfer:
- North America: 1,500 QPS * 0.5 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 194.4 TB
- Europe: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB
- Asia: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB
Data transfer cost:
- North America: 194.4 TB * $0.085/GB = $16,524
- Europe: 97.2 TB * $0.085/GB = $8,262
- Asia: 97.2 TB * $0.120/GB = $11,664
Total data transfer cost: $16,524 + $8,262 + $7,006 = $36,200
HTTP requests:
Total requests: 1,500 QPS * 60 seconds * 60 minutes * 24 hours * 30 days = 3,888,000,000
Using the updated AWS CloudFront pricing (as of May 22, 2023):
- HTTP requests cost: 3,888,000,000 * $0.0075/10,000 = $2,916
Total estimated monthly cost for AWS CloudFront: $36,200 (data transfer) + $2,916 (HTTP requests) = $40k
---
2) AWS Global Accelerator cost estimation:
Data transfer:
- Total data transfer: 194.4 TB (NA) + 97.2 TB (EU) + 97.2 TB (Asia) = 388.8 TB
Using the updated AWS Global Accelerator pricing (as of May 22, 2023):
- Data transfer cost (averaged across regions): 388.8 TB * $0.035/GB = $13,608
- (Also need to add EC2 egress cost, 388.8 TB * $0.02/GB = $7,776
- Total data transfer cost = $21,384
Accelerator:
- Assuming 1 accelerator with 2 endpoints (1 for HTTP/2 and 1 for HTTP/3)
- Accelerator cost: 1 * $18/accelerator/day * 30 days = $540
Total estimated monthly cost for AWS Global Accelerator: $21,384 (data transfer) + $540 (accelerator) = $22k
Former Dropbox employee, just correcting one assumption:
> Dropbox's peak traffic is 1,500 queries per second (QPS).
I can't speak to search QPS directly, but most individual serving hosts for file sync/retrieval were receiving tens of thousands of QPS. The overall edge QPS peaked at several hundreds of thousands QPS every day across all the hosts. So I'd guess that even just search is an order of magnitude higher than 1,500 :)
One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month.
As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center.
It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency.
Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further.
---
Cost estimates
Assumptions:
---1) AWS CloudFront cost estimation:
Total estimated monthly cost for AWS CloudFront: $36,200 (data transfer) + $2,916 (HTTP requests) = $40k---
2) AWS Global Accelerator cost estimation:
Total estimated monthly cost for AWS Global Accelerator: $21,384 (data transfer) + $540 (accelerator) = $22k