We have experienced severe performance issues in e-conomic during the past 3-4 weeks, and we have now identified what we believe is the root cause.
The e-conomic application serves many users simultaneously, which means that we have a lot of servers in place to handle this load. In order to keep track of the actions of each user, we use session state (as do most other web applications).
As part of this, the system tries to lock the data of each user before performing simultaneous updates to ensure that session data are not overwritten. When there are high levels of activity on the system, the underlying session state database is unable to keep up, which leads to users experiencing slowdowns and continual spinners.
Why do we use locks?
The short answer: It is the easiest way to ensure that session data are updated correctly. It is also the standard procedure when you are working in .NET like we do.
But writing to session state can also be handled in other ways. As detailed in the June performance review, we have been working on implementing our own session state service, which now allows us to start removing locks. This should have a major positive impact on performance.
In June, when we first started looking into the session state service, removing locks was one of multiple areas to improve, but it did not have the highest priority. However, now that we suspect that locks have a large impact on stability, this is now our main priority.
What actions are we taking?
Our first action will be to remove the locks in the most affected parts of the application. This includes the document handling functionality, e.g., transferring documents using the Scanning module, which we have identified as a source of a high number of locks.
We will remove locks from document handling and also move everything related to Scanning to separate servers so any remaining document handling locks will not affect other users.
The next step is to ensure that we can update session state without needing to lock data. Additionally, we will replace the underlying session state database with a faster version. This update will require a service window where we shut down access to the e-conomic application.
For now, this update is scheduled for September 26 – we will confirm this date at a later time. Update: This date has now been confirmed.
Finally, in a longer perspective, we will also look to change our architecture and infrastructure to make different groups of customers and different functionality more autonomous from each other in terms of performance.