• Home
  • About TechTalk
  •  

    Performance update

    30 June 2010

    Following up on our last performance update, we have made a number of changes to both our hardware and software setup – working on multiple levels, such as optimizing code, tweaking our network infrastructure, updating storage, etc..

    Regarding code optimization, a number of tweaks have been made. This should be particularly obvious when bulk-synchronizing data via the e-conomic API, or in general use of the Stock Management module (although said Stock Management module does still reduce the speed of a lot of operations, due to the much more complex housekeeping required).

    A major change is a move of our database from traditional mechanical disks to flash (memory-based) disks. In addition to a general performance increase, this also results in much more consistent performance, thus resulting in fewer extreme peaks. This isn’t the end of our storage performance improvements, however – during the summer, we will be moving our entire setup to even faster storage.

    On the database software side, we are currently looking into an update of both the database engine and operating system. However, such updates always involve a certain risk – both in terms of performance and compatibility – a risk we’re currently working hard to resolve.


    How to speed up your daily workflows

    29 April 2010

    Hello.My name is Christian and I have an addiction; an addiction to constantly look for ways to do things easier in front of a computer. In more technical terms and from a usability point of view: how to minimize the number of steps on frequently performed work flows or to put it in laymen terms:

    How to work without ever reaching for the mouse.

    Read the rest of this entry »


    Service disruption 1-2 April

    26 March 2010

    Please note that due to system maintenance, e-conomic will be unavailable from
     

    10 pm CET on 1 April to 6 pm CET on 2 April

    We will keep you updated here on the blog during the application downtime and will of course notify you as soon as the system is up and running again.

    We apologise for any inconvenience related to this downtime.

     

    Update 10.00 pm CET 1 April: The scheduled service disruption has now started and e-conomic is not available. We will keep you posted.

     

    Update 12.00 am CET 2 April: The operation has so far taken longer than we expected. We still expect to be online again no later than 6 pm today (2 April), but part of the planned operation might need to be carried out at a later stage.

     

    Update 4.15 pm CET 2 April: The planned maintenance has been completed earlier than announced and e-conomic is now available again. We did not complete all of the planned maintenance tasks and will schedule the rest of the work to a later date.

    Thanks for your patience and sorry for any inconvenience related to this downtime.


    Nighttime downtime

    18 March 2010

    Improving the underlying storages of our database environment

    Due to a major database restructuring, e-conomic will be unavailable for shorter periods at nighttime over the next week. In addition, we are planning a longer downtime of up to 20 hours on 1-2 April (the Easter holidays Holy Thursday/Good Friday).

    In our continuous effort to improve the overall performance of the e-conomic application, we will be initiating a major restructuring of data over the next week. In other words, we are moving customer data files to faster SAN volumes.

    Read the rest of this entry »


    The challenges of creating a web application

    25 November 2009

    We tend to move around a little at the office mainly to achieve what is summarized in the popular buzz word knowledge sharing. A few weeks ago my turn was up and I was asked if I wanted to transfer to another team working on our future frontend interface, centralized around the Journal. The main objective is speed, performance and usability experience. And the Journal input experience will be very similar to what you see in most spreadsheet applications. This was too exciting, as a backend developer I had to give it a go and this is now my primary concern for the time to come.

    Read the rest of this entry »


    Performance update

    30 October 2009

    Over the last couple of days, we have experienced overall performance degradation on our application.
    We have now located the source of the problem to be an internal logging function. The problem was resolved yesterday.
    Read the rest of this entry »


    Scheduled downtime 5-6 September

    28 August 2009

    Update 6 September 05:48 CET/04:48 BST:  

    The planned system upgrade has been completed earlier than announced and e-conomic is now available again.

    Read the rest of this entry »


    Planned downtime 18th of May 00:15-06:00 (CET)

    14 May 2009

    Our hosting center are upgrading their high-voltage power supply on May 18th between 00:15 and 06:00 (CET). In this period e-conomic will be unavailable, this means that neither the web site nor the API will be available. So if you have any automatic jobs running within this time interval, they will not be run on May 18th.

    As the system comes online again, we will be monitoring it to make sure that everything is running correctly.


    API: Something new, something faster…

    16 April 2009

    Last night, our latest Market Package with enhancements, fixes and improvements was released. It contains one new API feature, as well as a vast improvement of the speed of another.

    As for the new feature, it is now possible to login (‘Connect‘ in API terms) as an administrator. This facilitates integration with external accounting applications, e.g. reporting tools.

    To utilize this feature, instead of ‘logging in’ by using the Connect() method, you use the ConnectAsAdministrator() method, which takes the administrator user’s credentials (administrator agreement number, user ID and password), as well as the agreement number of a client to which the administrator user has access, as parameters. Note that the client in question of course still needs to have the API add-on module enabled.

    As for optimizations, we were recently made aware of some serious bottlenecks in our GetDataArray() methods when large registers are downloaded. Somewhat embarrassing, since we’re constantly pushing the data array methods as the recommended way of bulk reading/writing of data :-(

    After digging deeper into this, we discovered that the culprit was our ORM tool’s extremely conservative approach to checking for ‘dirty objects’ in its cache. While you’d expect the execution time for a GetDataArray() call to be proportional to the number of objects retrieved, in practice, it turned out to be polynomial instead! Not exactly what you’d want when downloading product registers in the thousands…

    While we could conceivably implement a generic workaround directly in our ORM tool, for the sake of risk minimizing, we have instead opted to implement bespoke optimizations on a per-object level – starting with the objects our logs show us to be the most frequently used.

    Last night’s release includes this optimization for the IProduct object – in practice, the Product_GetDataArray() method, which now handles upwards of a couple of thousand objects in a few seconds. The relative speed improvement increases with the number of objects – we’ve measured the speed increase to a factor of 10+ with 2,000 products.

    Our next Market Package release, due in mid-May, will include similar optimizations for a lot of other objects (IDebtor, ICreditor, IAccount, IInvoice, IInvoiceLine, etc.). As for all these registers, we still recommend ‘paging’ to approximately 500 objects for each call to its GetDataArray() method.

    For more detailed technical API documentation, take a look here.


    Improving API performance beyond data classes

    6 April 2009

    In the e-conomic API, using data classes is a powerful way of optimizing the retrieval, creation or updating of multiple entity properties – or even multiple entities – to use as few round-trips as possible.

    However, many developers are put off by the fact that when using data classes, only simple-type properties are returned – thus still requiring round-trips to retrieve properties of referenced objects.

    Suppose, for example, that we wish to list the product number, product name and product group number of all our products. The following would seem to be the obvious way of achieving this:

    IProduct[] arrP = session.Product.GetAll();
    IProductData[] arrPD = session.ProductData.GetDataArray(arrP);

    foreach (IProductData pd in arrPD)
    {
    Console.WriteLine("Number: " + pd.Number.ToString());
    Console.WriteLine("Name: " + pd.Name);
    Console.WriteLine("Group: " + pd.ProductGroup.Number.ToString());
    Console.WriteLine("");
    }

    This should result in only two round-trips – one for Product.GetAll() and one for ProductData.GetDataArray() – right? Wrong! For each product group, the above code will also generate a round-trip to retrieve the product group number (effectively a call to the SOAP function ProductGroup_GetNumber). The final result is a number of round-trips equalling n+2, with n denoting the number of products. This seems very unnecessary – especially since the number of products is usually several orders of magnitude higher than the number of product groups.

    First of all – why are the data classes designed this way? Why does the IProductData class return IProductGroup object references instead of IProductGroupData objects? While that would seem to be more consistent with the point of data classes, consider the ramifications if all data objects were to include referred objects as other data objects. In the above example:

    IProductData would include IProductGroupData
    • which would include IAccountData
    • which might include IVatAccountData (IAccount.VatAccount is NULLable)
    • which would include IAccountData
    • etc.

    In other words, this could quickly result in massively ‘recursive’ retrievals of data one would in most cases not have any use for.

    Luckily, there is another way of reducing the number of round-trips to an absolute minimum: Retrieve all product groups in one go, and store them in a dictionary with IProductGroup as key:

    IProductGroup[] arrPG = session.ProductGroup.GetAll();
    IProductGroupData[] arrPGD = session.ProductGroupData.GetDataArray(arrPG);

    Dictionary<IProductGroup, IProductGroupData> pgd = new Dictionary<IProductGroup, IProductGroupData>();

    for (int x = 0; x < arrPG.Length; x++)
    {
    pgd[arrPG[x]] = arrPGD[x];
    }

    IProduct[] arrP = session.Product.GetAll();
    IProductData[] arrPD = session.ProductData.GetDataArray(arrP);

    foreach (IProductData pd in arrPD)
    {
    Console.WriteLine("Number: " + pd.Number.ToString());
    Console.WriteLine("Name: " + pd.Name);
    Console.WriteLine("Group: " + pgd[pd.ProductGroup].Number.ToString());
    Console.WriteLine("");
    }

    This generates only four round-trips: ProductGroup.GetAll(), ProductGroupData.GetDataArray(), Product.GetAll() and ProductData.GetDataArray() – irrespective of the number of products or product groups. A similar trick can be used with any data class that includes object references.

    A final, and somewhat related performance tip: Throwing the result of GetAll() at GetDataArray() may of course still lead to massive amounts of data being downloaded. As a general rule, we recommend that you split the GetAll() result into batches of 500 entities for the subsequent data retrieval. While this will of course result in slightly more round-trips, it is much less prone to timeouts in either end.

    Take a look at our API documentation.