AfterShip charged us 26K USD and refused to refund
Qwintry
Qwintry is a mail forwarding company that helps shoppers from all over the world to buy goods from online stores. We have fulfilment centres in the USA, Turkey, UK and Germany. You sign up at Qwintry.com and immediately get your own personal mailbox at each of our warehouses. Then you simply send your online orders to your Qwintry address and we'll do the rest. We check, pack and send all your items in one box straight to your door - and you save on shipping costs.
AfterShip (Automizely)
AfterShip is a company that provides an API for tracking shipment status. They also offer email and SMS marketing services as Automizely.
The Bill
One day we (Qwintry) were faced with a staggering bill from AfterShip:
26,824.84 USD monthly bill. We never spent more than 650 USD with AfterShip, so this was about 40 times bigger than we expected.
After checking our web server logs, we realized that some malicious user had used our tracker.qwintry.com website to send 300k tracking requests in 3-4 days:
Some context: Qwintry has been utilizing AfterShip API to track shipments delivery status, for years. Over time we have implemented direct integration with major carriers (USPS, DHL, UPS) on our side to save costs, and we still used AfterShip for more exotic shipping carriers, and one of our developers have also built a landing page tracker.qwintry.com with a minimalistic tracking form so that anyone (who knows about this website) can track the shipment. In fact, this site was built for internal purposes and never got any real design (other than default Bootstrap HTML) or marketing, but it hung out there, accessible to the wider internet, for over 5 years.
Back to the story. It turned out the attacker used distributed proxy network and some legit tracking numbers along with completely random ones:
This is how it looked like from our MySQL database:
On discovering this, and seeing that this workload was obviously the result of a sophisticated attack where the malicious user was using real tracking from exotic carriers, we immediately shut down the tracker.qwintry.com web service and contacted AfterShip, informing them that this was a malicious attack on our infrastructure and asking them to waive the bill, which we clearly couldn't afford.
I'd like to point out that there was absolutely no way to set a monthly budget or prevent spikes in usage in AfterShip dashboard. Unlike AWS or Google Cloud Engine, where you can set a maximum budget and email alerts, there are no such things in AfterShip's settings. All they can do is reassure you that the attacker can max out your account at 10 requests per second (not 100 requests per second!). What a relief!
After providing the AfterShip team with all the necessary information that this was a malicious attack on our website, we waited a few days for them to investigate on their end.
Then we had a Zoom call with their representative, here is the generous offer we got:
Also, I have checked with our finance team further on this. Since we wish to continue growing our business & partnership with Qwintry, and the issue is not from AfterShip's end, I regret to say that we can only provide a 50% Store Credit of the amount, which can be used in your upcoming bills from us. - Aftership
The sad part is that during Zoom call (Qwintry has a Zoom recording copy) AfterShip admitted that they understand this was not our fault in this incident, and that charges are disproportionate to the services provided, and thus unreasonable. When we asked them why they don't have any anomaly detection and automated alerts about spikes, they explained that "this was not a spike" and this 40 times bigger usage was not something they think was really weird and strange. They also blamed us having the Essential (legacy) plan which meant we overaged every month (this is true, the Essential (legacy) plan only includes 100 shipments tracking, for 9.99 USD, and we usually tracked around 3000-5000 shipments every month) and this prevented them from seeing "any red flags" on their side. We had this old plan because we were AfterShip early adopter.
Budget control tooling reduces profits for vendors?
API service providers often face a conflict of interest regarding the installation of safeguards against unusual usage spikes. This conflict arises from a vendor's profit motives, which could discourage them from implementing mechanisms that protect clients from unexpected billing surges due to anomalous volumes. As a vendor, there is minimal tangible incentive, beyond ethical considerations, to invest in the development of protective measures against accidental usage. By choosing not to incur any development costs, they may face occasional disputes with customers who have been overcharged, yet still realize substantial profits at the end of the day.
Many vendors resist providing tools that allow clients to set maximum usage limits or manage their budgets, despite the clear advantages these offer for financial control of the customer. Such tools would allow clients to cap their service costs and avoid unpredictable cost escalations resulting from increased usage or system anomalies.
Additionally, lots of vendors commonly lack alert systems to inform clients of nearing or breached usage limits. Implementing such alerts would provide clients with greater management over service usage and budgeting. Yet, vendors might view this as a potential earnings decrease, perpetuating the conflict of interest."
Conclusion
We have about 20 APIs integrated into Qwintry.com. Many of them carry risks like this. It is obvious that this incident was a result of our reckless behavior, in first place – this is a hard and expensive lesson for us, and it teaches us to spend even more time and budgets on audits of all front-facing interfaces. What surprised us the most in this situation is a partner we have worked with for years who charged us twenty six thousand dollars for a shipment tracking API we didn't get any value from, and is still holding this money hostage.
For those unfamiliar with the industry, most shipment carrier APIs, such as USPS, DHL, and UPS, offer free integrations. These allow you to track thousands of packages at no cost. Thus, the financial burden placed on AfterShip due to this incident would be relatively insignificant.
In contrast, for a small enterprise like Qwintry, such a substantial invoice imposes a heavy financial strain. I must admit, in my 15-plus years of development experience, this is an unprecedented example of vendor ethics.
It's worth noting that major companies like AWS and Oracle have often waived high fees that were accidentally incurred on their services. Yet, it seems AfterShip does not show the same understanding and flexibility when their customers face similar problems.
External APIs with overage billing are a ticking bomb
Could we have prevented this? In the case of this AfterShip incident, we could (and should!)
- Implement some sort of captcha to make it harder for the attacker to abuse our web service. But this was not our primary web service – and it turned out it was more like an old ticking bomb sitting in a dusty corner. It is easy to miss things. Think about it when doing a regular security audit.
- Implement proper outgoing API request monitoring with good visibility and alerts. This can probably be done using some APM solution and/or API gateway / reverse proxy.
- Use virtual cards with capped max usage. This does not guarantee the protection, but at least it prevents huge unexpected charges going unnoticed.
We are using all these methods to protect our primary customer-facing web services which have high risks of being abused. This form never got any of these protections, as it was not customer-facing, but again, it's our responsibility that we were so reckless.
My advice to you is to carefully inventory the APIs you use, and make sure you set reasonable limits and maximum usage at vendor dashboards. Avoid using API providers that do not provide tools to limit your liability.
A list of Cloud Providers which have adequate budgeting and spike protection:
- Amazon Web Services (no kill-switch but budget alerts are there)
- Google Cloud Engine (with some exceptions like Firebase, read below)
- Sentry (spike protection, and you can't spend a fortune)
A blacklist, a list of Cloud Providers with shady and unpredictable billing:
Do you have any other providers to add to this list? Tell me about your huge bill stories: contact@apiroad.net