e-conomic has completed a performance upgrade of its system on Sunday (August 21) from 00:01 to 06:00 CET. This upgrade required the system to be shut down in this period, which means that e-conomic was unavailable to all customers.

Please note that if your integrations to e-conomic do not support post-synchronization of data, any data exchanged with e-conomic in the upgrade period might be lost.

All systems are now back to normal operation and fully available to all customers.

e-conomic is performing an upgrade of its system during a 6-hour time span on Sunday (August 21) from 00:01 to 06:00 CET. Therefore you cannot log on to e-conomic and use the system in this period.

Please note that if your integrations to e-conomic do not support post-synchronization of data, any data exchanged with e-conomic in this period might be lost.

After the 6-hour upgrade, all systems will return to normal operation at 06:00 CET on August 21.

e-conomic has scheduled a performance upgrade involving six hours of downtime this coming Sunday (August 21) from 00:01 to 06:00 CET. This upgrade will further enhance system failover options and minimize system downtime in the future.

Please be aware of the following:
- You will not be able to use e-conomic at all during this 6-hour period
- If your integrations to e-conomic do not support post-synchronization of data, any data exchanged with e-conomic in this period might be lost

After the 6-hour upgrade, all systems will return to normal operation and be available for all customers.

We ran some stats for product planning purposes. We were looking at Invoicing and found that 5 million invoices were booked in the past year. This means that our customers will book roughly 7.9million invoices in e-conomic this year. A nice tangible way to see e-conomic bring value to our customers.

Sunday Dec 19, 2010 between 00:00 and 06:00 we will be conduction some business continuity test.

Doing this period we will among others conduct a series of tests on the datacenters failover capability.

This could result in some small period where the application appears to be slow or off line and you could have to re-do some of your data entry, however anything that has been confirmed as saved are saved and you would not need to be re-do these

As you may already know we at e-conomic value our customers security and access to our systems very high. We strive to ensure that your data is protected and that you can get access to it 24/7.

With regards to the latter we have recently successfully completed a transition, that ensure even higher availability and reliability. During the scheduled downtime on November 27/28 ,we gave our system an overhaul. In that weekend we split our datacenter into two physical locations ensuring redundancy even in the case of severe problems at a given datacenter ( fire, power outages etc.). We can now serve our customers without interruption in case of problems at a given location while at the same time having benefit of fewer and shorter scheduled maintenance periods.

Read more about the details of the upgrade on the TechTalk blog

17:05 – It is my privilege to announce that e-conomic is now online again!

In the grand scheme of things, everything went really well – so we’re of course happy to be able to open up for business roughly 19 hours earlier than previously announced.

15:30 – We are now in the process of running test on the new system.  It’s still cold out there

12:00 – Even that the weather isn’t the best today, we are struggling to get done before time. We are a bit ahead of schedule.

06:00 – Everything is running according to plan.

As previously announced here on the TechTalk blog and communicated to all end-users, we are shutting down e-conomic’s core servers from Nov 26, 2010 at 22:00 CET until Nov 28, 2010 at 12:00 CET.

Why do we need 38 hours of downtime?
Because we are making a number of major changes to the core system that will create a more stable, efficient and reliable application. The changes include.

• Replacing the physical hardware – servers, storage and network components
• Upgrading the operation system
• Upgrading the database system
• Migrate the application to a dual-site model

These are just some of the operations we will be performing during the maintenance period.

Note that if everything goes according to our main plan we do not expect the application to be down for the whole 38 hours. However, rather than speeding the changes through we prefer to do some extra tests to ensure that everything is working as planned before opening up for business again – after all this is your data that we are handling.

Today nov. 1 2010 between the hours 10:46 – 11:06 the e-conomic application was unavailable for most of our customers.

The root cause for this incident was that the core internet provider for our service became unstable and we had to switch to a backup provider. The switch unfortunately did require some manual work before it worked.

The need for manual work is of cause not acceptable nor was the time for the switch. We will work together with our provider to improve it.

We have been asked this question many times over the past 12 months – whenever we have experienced slowdowns with the application. The question has often been accompanied by the remark – why don’t you just get some bigger boxes (scale up)?

Well, it would be great if the purchase of new and bigger hardware could solve all performance challenges – in fact, however, this would only solve a few of the challenges that we face in creating a stable and high performance application for our customers.

Picture the following scenario

A big new store with many nice items on sale and with large discounts has just opened up near where you live. The store is the size of a dozen football fields – but the architect has been told to add only one entrance for security reasons. This entrance is only 1 meter wide.

As you can imagine, when people see the ads for the new store, they all head for it – within a few minutes there will be a big queue outside waiting to get in. So even though the store is the size of many footballs fields, the customers won’t be able to benefit from the size of the store before they are inside. Once inside, they can quickly locate the items that they like and proceed to checkout, but they then have to get out through the one door in the store with all the items while the people outside are trying to get inside.
In short, new and bigger boxes don’t do it alone!

Another question we also hear – if social media sites can handle millions of users why don’t you do what they do?

We could in fact do that, but none of our customers or regulators would most likely be happy if invoices entered into e-conomic for some reason disappeared. In other words we work under rules & regulations where data integrity is as important as speed. This results in some challenges…

Then what are we doing – nothing?

We are adding more and bigger boxes to our setup. Some of the boxes we can add as we go along without disturbing our customers. For other boxes, we have to create very detailed plans for how to implement them in our infrastructure. These other boxes often relate to core services in our infrastructure. Planning takes time.

Why?

Because when we add new core servers to our service application, we would like to ensure that our customers are inconvenienced as little as possible in terms of downtime.
Over the past couple of months, we have been working on improving our application infrastructure – from gathering everything in one data center to distributing across two data centers and having both data centers online at the same time – all efforts to scale out.

Why?

If one data center for some reason loses power, or its internet connection is interrupted, we could continue to deliver our service to you with as little downtime as possible – we are talking minutes instead of hours.
What remains for us to do now is to move our core servers. Typically, moving core servers into a distributed setup would mean up to 48 hours of downtime before everything is in sync, considering the amount of data we would have to sync up. As this is a very long period for our service/application to be offline, we have decided to replace and upgrade these core servers with both faster and more powerful CPU and more memory to also optimize the operative system for the new hardware.

In conclusion

Before the end of this year, we want to be up and running in a dual data center setup. The setup’s infrastructure is designed to support our service by allowing us to scale both out and up.