Yesterday wasn’t an ordinary day for us the Jotform team. I woke up with a “Jotform Down!” SMS message. Jotform load balancer was gone. At first I thought this would be a load balancer hardware or software issue. It turned out that we were under attack by thousands of zombie computers in China. To our amazement it was a massive DDoS attack.
We kept fighting to get Jotform back alive and we were brought down back many times throughout the day. It wasn’t working and our service provider was not happy with us. Their network and routers were getting saturated and other customers were getting affected. In fact, they told us that they cannot sustain these attacks any further and we might need to think about other options.
We kept asking ourselves why would anyone do something like this to Jotform. We are the good guys. We provide a free service to hundreds of thousands of people. Our users are very happy with us. We don’t receive any hate emails. In fact, we receive lots of emails every day from people who are very happy with the service Jotform provides. One possible exception is the phishers. They keep creating phishing forms even though we suspend their accounts. So, our conclusion was that the attack was probably initiated by a phisher who was suspended.
Unlike regular denial of service attacks, Distributed DoS attacks are very difficult beasts. They are usually initiated by thousands of compromised computers. So, they look like legitimate traffic coming from different IP addresses on different locations. Since they start their attack in sync, it takes them very short amount of time to saturate computers, load balancers and even routers on their way. A great tool that has helped us a lot to slow down attacks was DDoS Deflate.
Upon further investigation we found out that most of these attacks were coming from compromised DSL modems in China and some other Asian countries. What happens is people leave their DSL modems with factory settings and these settings may have default passwords, or they might contain firmware with security holes. Worms target these DSL models and once compromised they can be used for DDoS attacks.
The attacks are still continuing but we are now able to continue our service without any interruptions to our users. Our final solution was to move part of our infrastructure to Amazon Elastic Compute Cloud. EC2 was able to handle high levels of traffic and sustain us throughout the attacks. We are still getting sudden surges of attacks but our architecture is much more ready to handle these, thanks in part to Amazon’s services.
Jotform users have been very patient with us. We appreciate their support. We tried to keep our users updated on twitter as much as we can. Using our official twitter account we were able to broadcast real-time updates to our users.
We are very sorry about downtime these attacks may have caused for our users. We will keep improving our infrastructure to prevent future incidents like this.
Send Comment:
18 Comments:
More than a year ago
It appears the changes made to mitigate the DDOS attack have left my most profitable form dysfunctional. I'm hoping tech support can fix it.
More than a year ago
Having trouble accessing the website jotform.com just acts as its loading forever
More than a year ago
Wow. I didn't even notice any blips in the radar. I've been using Jotform since just before the new version release. It continues to be a life saver; and your customer service is the best. I'm recommending everyone. Thank you for keeping us secure and informed.
More than a year ago
hola, tienen ustedes un blog en español? I dont speak english...
More than a year ago
bagaimana cara biar bisa baca sms di nomer orang?
More than a year ago
Agree, Gr8 work.
Thanks for answering my email so promptly. And I very much appreciate that you post what happened.Keep up the good work, cheers.
More than a year ago
Great work on getting it back up, I've been using jotform through various companies for about 3 years now and this is pretty much the first time anything like this has happened. You can't complain at those odds and the fact you kept us all up to date and remained responsive throughout the day was fantastic, keep up the great work
More than a year ago
I lost a days business over it but I appreciate your responses to my emails and I love you guys! Jotform rocks, keep up the great work!
More than a year ago
Hi. Thank you for your work to fix the problem with DDoS attack and for keeping us informed via Twitter. It looks like our conference registrations are coming through fine again today and complaints have stopped. I also appreciate your explanation about what happened which I can pass along to other concerned members of our organization, Thanks.
More than a year ago
Hi patriccia,
If the problem with your form continues, please send us an email with a link to your page/form and we will look into it.
It could also be a DNS cache issue. Since we changed IP address for our site it might be cached on your computer or network. To see if that's the problem, try seeing the pages using a proxy such as hidemyass.com.
More than a year ago
Knew it had to be something bad when you were down for so long. How awful! Hope you can stop it soon and thank you for the updates. Good luck and good job!
More than a year ago
Have the problem been resolve, because my form is still down.
More than a year ago
A big thanks for the complete explanation. Helps us communicate with our clients. Appreciate all your efforts.
More than a year ago
Yeah - sounds a mare. keep up the good work - get that security sussed as well as you can :)
More than a year ago
Thanks so much! Glad to have you up and running again. I love jotform...makes my life much easier. :)
More than a year ago
Great work guys! Keep up the good work
More than a year ago
Thanks for taking the time to post what happened and why, and for the timely updates on Twitter to keep us posted. You must have had a rubbish day yesterday. You should all chill out with a few beers today :-)
More than a year ago
We are supporting you forever.