Workflow Changes & Fighting Spam
Since a last couple of days, we've faced multiple downtimes due to which, our user experienced heavy inconvenience and we truly apologize for this.
Since 14th April, 2019 we started receiving a flood of requests, which were polling the records unnecessarily
which further caused load on the systems and disabled the services further.
To quickly overcome the issue, we upgraded our machines to handle the traffic, but quick vertical scaling couldn't hold long, and started throwing exceptions again. Later, we tried debugging the issue in-depth, and it turns out the IO operations were too high as the requests were flooding in. We've attached the CPU Utilization chart for the last 20 days which shows the spam activities triggered since last couple of days.
On debugging further, we also fetched some graphs from other services which we use to monitor the network.
How did you fix it?
We do not want to provide bad experience to our genuine users, especially the ones who support us by buying the Pro plans. That's what helps
somewhat to pay half of our server bills. Hence, we initially blocked all the requests from the users who did not pass the
secret-key in the header.
Later, we have placed some rate limiting in place. Thanks to @amar_sharma3 for helping out on this one. From now, the bins will be blocked if accessed a few times in any way. This includes Create, Read, Update or Delete. We've also started to blacklist the user accounts (we will not unblock these on request).
We want to provide the best user experience possible, keeping Simplicity & Accessibility in mind. We've tried to change the workflows in a best minimal way possible.
We want to provide the service to the genuine users. When we say genuine users means the users who find the service useful and do not play around and abuse. The $ we earn from the service doesn't even cover 25% of the total expenses but we keep it running with a hope to atleast break-even some day. Not only that, but we want to keep the service up & running as it is useful to thousands of our users & it does help developers across the globe in some or the other way.
That said, we might put more stringent checks & workflow changes in place going ahead if the abuse still continues.
Please expect some bumpy ride for the next few days as we make some changes to the existing API. We will try not to make any major changes which might break the existing app of yours (preventing users to create records was a major change though). We will keep you updated on Twitter for these changes.