To the people that knows me well, this blog entry is just up my alley. I spent a lot of my time thinking about this and in fact one could argue too much. But hey, if you don’t have extreme field experts, to whom would you address a specific customer question from a Help-Desk?

It’s of course hot keys, or shortcuts if you will, on any software suite or operating system out there – its my mission to know all shortcuts ever implemented and tell the world about it.

However; for now I will direct my focus to our new, but rapidly growing, baby of e-conomic: The Datagrid of book keeping.

Read the rest of this entry »

Since around 23:00 CET Sunday, e-conomic has not been available. We are working hard to solve the issue. Sorry for the trouble, more info to come.

Update 00:50 CET Monday: Issue found, e-conomic is up and running again.
Error was a due an overfilled transaction log and the notification system for this situation failed, unfortunatly. We are taking steps to make sure this will not happen again. Sorry for the trouble.

Sunday Dec 19, 2010 between 00:00 and 06:00 we will be conduction some business continuity test.

Doing this period we will among others conduct a series of tests on the datacenters failover capability.

This could result in some small period where the application appears to be slow or off line and you could have to re-do some of your data entry, however anything that has been confirmed as saved are saved and you would not need to be re-do these

As you may already know we at e-conomic value our customers security and access to our systems very high. We strive to ensure that your data is protected and that you can get access to it 24/7.

With regards to the latter we have recently successfully completed a transition, that ensure even higher availability and reliability. During the scheduled downtime on November 27/28 ,we gave our system an overhaul. In that weekend we split our datacenter into two physical locations ensuring redundancy even in the case of severe problems at a given datacenter ( fire, power outages etc.). We can now serve our customers without interruption in case of problems at a given location while at the same time having benefit of fewer and shorter scheduled maintenance periods.

Read more about the details of the upgrade on the TechTalk blog

17:05 – It is my privilege to announce that e-conomic is now online again!

In the grand scheme of things, everything went really well – so we’re of course happy to be able to open up for business roughly 19 hours earlier than previously announced.

15:30 – We are now in the process of running test on the new system.  It’s still cold out there

12:00 – Even that the weather isn’t the best today, we are struggling to get done before time. We are a bit ahead of schedule.

06:00 – Everything is running according to plan.

As previously announced here on the TechTalk blog and communicated to all end-users, we are shutting down e-conomic’s core servers from Nov 26, 2010 at 22:00 CET until Nov 28, 2010 at 12:00 CET.

Why do we need 38 hours of downtime?
Because we are making a number of major changes to the core system that will create a more stable, efficient and reliable application. The changes include.

• Replacing the physical hardware – servers, storage and network components
• Upgrading the operation system
• Upgrading the database system
• Migrate the application to a dual-site model

These are just some of the operations we will be performing during the maintenance period.

Note that if everything goes according to our main plan we do not expect the application to be down for the whole 38 hours. However, rather than speeding the changes through we prefer to do some extra tests to ensure that everything is working as planned before opening up for business again – after all this is your data that we are handling.

Today nov. 1 2010 between the hours 10:46 – 11:06 the e-conomic application was unavailable for most of our customers.

The root cause for this incident was that the core internet provider for our service became unstable and we had to switch to a backup provider. The switch unfortunately did require some manual work before it worked.

The need for manual work is of cause not acceptable nor was the time for the switch. We will work together with our provider to improve it.

We have been asked this question many times over the past 12 months – whenever we have experienced slowdowns with the application. The question has often been accompanied by the remark – why don’t you just get some bigger boxes (scale up)?

Well, it would be great if the purchase of new and bigger hardware could solve all performance challenges – in fact, however, this would only solve a few of the challenges that we face in creating a stable and high performance application for our customers.

Picture the following scenario

A big new store with many nice items on sale and with large discounts has just opened up near where you live. The store is the size of a dozen football fields – but the architect has been told to add only one entrance for security reasons. This entrance is only 1 meter wide.

As you can imagine, when people see the ads for the new store, they all head for it – within a few minutes there will be a big queue outside waiting to get in. So even though the store is the size of many footballs fields, the customers won’t be able to benefit from the size of the store before they are inside. Once inside, they can quickly locate the items that they like and proceed to checkout, but they then have to get out through the one door in the store with all the items while the people outside are trying to get inside.
In short, new and bigger boxes don’t do it alone!

Another question we also hear – if social media sites can handle millions of users why don’t you do what they do?

We could in fact do that, but none of our customers or regulators would most likely be happy if invoices entered into e-conomic for some reason disappeared. In other words we work under rules & regulations where data integrity is as important as speed. This results in some challenges…

Then what are we doing – nothing?

We are adding more and bigger boxes to our setup. Some of the boxes we can add as we go along without disturbing our customers. For other boxes, we have to create very detailed plans for how to implement them in our infrastructure. These other boxes often relate to core services in our infrastructure. Planning takes time.

Why?

Because when we add new core servers to our service application, we would like to ensure that our customers are inconvenienced as little as possible in terms of downtime.
Over the past couple of months, we have been working on improving our application infrastructure – from gathering everything in one data center to distributing across two data centers and having both data centers online at the same time – all efforts to scale out.

Why?

If one data center for some reason loses power, or its internet connection is interrupted, we could continue to deliver our service to you with as little downtime as possible – we are talking minutes instead of hours.
What remains for us to do now is to move our core servers. Typically, moving core servers into a distributed setup would mean up to 48 hours of downtime before everything is in sync, considering the amount of data we would have to sync up. As this is a very long period for our service/application to be offline, we have decided to replace and upgrade these core servers with both faster and more powerful CPU and more memory to also optimize the operative system for the new hardware.

In conclusion

Before the end of this year, we want to be up and running in a dual data center setup. The setup’s infrastructure is designed to support our service by allowing us to scale both out and up.

As part of our ongoing commitment to make the best possible ERP integration platform available, we’ve added two new often-requested functions to the API this week:

OrderLine.GetAll() will allow you to retrieve all order lines across all orders.

Debtor.FindByPartialName() enables wildcard searches for customers by name.

Finally, bulk downloading of invoices and products have been sped up tremendously. Internal testing has shown speedups of factor 50 for invoices, and 7-8 for products.

If you’re using .NET, to use any of the new features, you’ll have to use the latest version of our .NET assembly – which is available here. The optimizations for bulk downloading of invoices and products are purely server-side, and thus, you’ll benefit from them automagically.

The second – and currently last – phase of our move to faster storage has now been successfully completed, and e-conomic is back online again.

Preliminary benchmarks show quite some performance improvements – although we expect this to be most noticeable as ‘much smaller peaks at high load’.

In addition to our storage upgrade, we’ve also been working on some ‘softer’ optimizations, particularly for the e-conomic API. More on this shortly…