About DDoS Attack on Friday

What Happened

On September 19th, Friday, 3:30 EST. We were hit by a massive DDoS attack. The DDoS attack (probably caused by our recent crackdown on phishers) was so massive and sudden that it took down our data center’s firewall even before reaching us. In fact, our servers did not even see an increase in the load. We were down for several hours while we made changes on our system to make us more resilient to these kinds of attacks. We were back online fully before US woke up, but our European users felt the downtime worst.

Unfortunately, the changes we made required our IP addresses to be changed. This caused many users to not be able to access our site because of their DNS cache. In fact, many users kept reporting not being able to access our site in Saturday and even Sunday. We kept suggesting using jotform.net domain name temporarily on twitter.

We also had to move our SSL service from www.jotform.com to secure.jotform.com. The service we currently use for protection against such attacks needs to keep the security certificate on their servers. However, they cannot issue new certificates at this time because of an internal audit performed on their certificate provider. If you are using secure SSL forms, you must change the links on your site from  https://www.jotform.com to https://secure.jotform.com.

UPDATE: (9/20/2011) Our certificate for www.jotform.com is now working.

What We Learned From The Downtime

We have to run Jotform on multiple data centers. We actually knew this before and we were working on it. If the attack came a month later, the down time would have been much smaller. We are now running on multiple data centers and in case we lose one, we are ready to switch instantly.

We need to use a service that protects against DDoS attacks. This is the second time we had experienced such a DDoS attack. Since we fight against the phishers, we have to expect such attacks all the time in the future and be ready for them. That’s why we switched to a service that can protect against DDoS attacks.

We need to increase our fight against phishers. Jotform is so incredibly easy to use that even phishers prefer us for their forms. We think our recent crackdown on phishing has caused this attack. We will increase our efforts on preventing phishing.

We need to get you on Twitter. Our only way for communication was our Twitter feed and Facebook page when we lost connectivity, and we still had a lot of customers that don’t know what the heck happened to their forms.  Please, if you have a Twitter account, follow us on Twitter; or subscribe to our Facebook page.

What We Have Done About it

Since Friday we have been busy. We worked over the weekend to make sure we put everything in place before a new week starts. We are now more confident about our system.

We are now running a security proxy service in front of our firewall. They are specialized on preventing such attacks.

We are now running on multiple data centers. We are ready to instantly go live on another data center in case we lose  connectivity in our first data center.

We have a more robust DNS management system. We are now running on proxy IP addresses which means we can now change IP addresses for our servers and our users will not have similar DNS cache problems.

We have improved our Firewall and Load Balancer. We have a new firewall and load balancer for our app servers. This is actually costing us a lot, but our data center required it since losing shared firewall was not something they wanted to  experience again. On the positive side, having a dedicated hardware firewall and load balancers will probably improve the speed of our service.

We are notifying all users who use secure forms. Since we are not sure when our security proxy service will be able to add security certificate for jotform.com, we are sending an email today to all users who use SSL forms, and asking them to change the URL of their forms to secure.jotform.com until we are certificated again.

We are providing one month service refunds to all users who request it. We have always been trigger happy to provide refunds when one of our users is unhappy about a problem. So, we’d happy to provide full month’s refund to any user who sends an email to billing@interlogy.com.

We are aware that you are putting a lot of trust in us by placing Jotform code in your site. That’s actually a lot of pressure for us. 🙂 Any problem on our site can create problems for your site and for your customers as well. That’s why we worry a lot of about availability and we are continuously improving our systems.

We hope we will continue to deserve your trust.