Ticket S36497
Visible to All Users

XAF ASP.NET WebForms / Blazor UI deployment, scalability and load testing considerations

created 14 years ago (modified 2 years ago)

Stateful Architecture

XAF Web apps are stateful meaning that ASP.NET stores user state in memory to combine fast responses to user actions with server-side data processing. Technically, an XAF WebForms application uses the ASP.NET WebForms session to store non-serializable service information (XafApplication, WebWindow and many other entities by means of the IValueManager interface, etc. That means that in ASP.NET WebForms you cannot use a session state mode other than InProc. ASP.NET Core Blazor Server is also a stateful app framework (Microsoft Docs) - most of the time, the app maintains a connection to the server and the user's state is held in the server's memory in a circuit.

Pros

Maintaining the session state allowed us to deliver a greater functionality and an easier development model to XAF developers. For instance, you can implement a Controller and access and store a reference to the current View and its controls, ObjectSpace, other Controllers, etc. without caring a lot about ASP.NET WebForms or ASP.NET Core Blazor Server page life cycle specifics - just like in desktop clients (WinForms or WPF).

Cons

However, the chosen stateful architecture has its own drawbacks that may require a bit more effort from you while addressing certain business requirements such as good scalability. This stateful architecture means that memory consumption increases with each user. After a certain number of simultaneous users, the application can reach its hosting machine’s memory limit. Even if you have powerful web servers, thousands of concurrent requests may stack in a lock somewhere (race conditions or concurrency issues may occur). That is simply because XAF's core (Application Model, Types Info System, etc.) is not designed and is not tested against such high-load web scenarios.

Scalability Considerations

Unsupported & Typical Scenarios

XAF Web apps (both ASP.NET WebForms or ASP.NET Core Blazor Server) were not designed or tested for high-load scenarios where a single web server handles hundreds or thousands of concurrent web clients (for instance, in SaaS apps). Thus, we cannot guarantee that you will not meet unsolvable problems if you go this route with XAF (even behind the maximum user load). It is like the same car cannot be expected to be a comfortable sedan, SUV and F1 racer at the same time.

Like many software, XAF was born as an in-house project serving our Support and Client Centers (WinForms UI with about 200 concurrent users at peak times and WebForms UI with 5-10 concurrent users typically). XAF Web apps are mostly used in a local Intranet with a limited set of users (often deployed internally). It is important to understand that our tool is designed for a set of finite scenarios and using it in an edge case may not work well.

Most XAF WebForms or Blazor Server UI customers do not have more than 50 users according to their feedback (typically it is even less than 20-30 users). We heard of 1-2 edge cases with 300 and even 600 concurrent users, but we cannot imagine how much application and web server software customizations it took to achieve acceptable performance under such a load. We also ran internal tests with up to 30-50 concurrent web sessions for a single web server.

Web Server Hardware & Software Requirements

Initially, when choosing appropriate web server hardware or hosting environment, you need to assume that about 10-20 MBof memory will be required per each user session (this number may vary depending on the application complexity). We cannot provide a universal calculator for web server hardware/software requirements, nor can we comment on overall performance without tests. The complexity of your business model and implemented behaviors are significant factors in throughput/performance. Ultimately, performance will depend on development decisions, application type, environment, and even tested use-case scenarios. A few examples of factors that affect application performance are the number of persistent classes and their fields, Controller design, Application Model customizations, availability of memory intensive operations to end-users (frequent import/export of large data amounts, or complex report generation).

Load-Test First

First of all, make sure that scalability is really a practically confirmed "problem" for you, and you need to handle it. In many cases, this scaling over multiple web servers is not required, and your XAF website needs can already be handled by a single VM instance somewhere in the Cloud.

That said, the first thing is always to do before worrying about this is to practically test how your application behaves under the planned load in an environment close to the production one. For this real testing, you may be interested in this community video by Joche Ojeda - XAF Blazor load test (connecting 100 clients), and our GitHub example - XAF Blazor load testing on Linux and MySql using Puppeteer and GitHub Actions.

When planning an automated load (stress) test, consider that at peak time a regular XAF Web application user will perform no more than 1-2 operations per second or so, depending on your real use cases. It does not matter which application action you are going to perform (be it CRUD, navigation or refresh) as they are all comparable, unless you are loading ListView with thousands of records without the server mode. Here it is important to note that XAF Web apps are not optimized for load-testing tools that query the web site too often and their results may vary or even be inaccurate, because such tools often do not wait for full page initialization. To avoid such inaccuracies, consider waiting a few seconds for the previous operation to complete using the specialized 'wait' commands provided by your testing tools (refer to their docs) -  we have a similar delay between commands in our internal EasyTest functional tests.

Vertical Scaling (scaling up)

To a certain moment in the application life cycle as the business demand and the number of concurrent users increase, you can serve more users by providing more powerful hardware (vertical scaling), whose specs can only be determined empirically based on the request handling speed requirements for your specific application and environment. In general, the better hardware you provide, the better results you will likely get. Bear in mind, however, as written in docs, XAF apps are not designed to handle thousands of concurrent users at a time. To meet such business requirements, you would better consider using other tools to build your app.

Finally, prior to thinking about providing more powerful hardware, I recommend you check out the tips from the How to improve the application's performance article to see precisely where the application response time is spent and if something in your custom code can be improved. If you find anything related to the DevExpress code, we will be more than happy to look at your debuggable sample and repro instructions.

Horizontal Scaling (scaling out)

ASP.NET Core Blazor Server

You can minimize the aforementioned effects and improve performance if you use horizontal scaling (scaling out) instead of the vertical one (scaling up) - the fewer concurrent requests for a web server instance there are, the faster it should work. For instance, each pod/web server handles say 10-20 clients instead of 100 = decreases the chances to wait in the lock.

We published a GitHub example that shows how to deploy an XAF Blazor application to a Kubernetes cluster with horizontal autoscaling. We tested the application in two types of clusters: locally-run K3s and Azure Kubernetes Service (AKS). The maximum pod replica number (20) allowed around 300 concurrent users. An AKS cluster needs two nodes (B4ms machines: 4 Cores, 16 GB RAM) to operate with such a number of pod replicas and the same load.
We hope that our working example along with Deployment Recommendations for XAF Blazor UI Applications will address all related questions on Docker, K8s, Linux, nginx, and load-balancing in the future.

Will your own XAF Blazor app work with 100, 200, 300, or more concurrent users with similar performance, provided the hardware/software are the same? It depends - our example is not an official guarantee by any means. In brief, every application is unique. Even with horizontal scaling, we recommend that you carefully:

ASP.NET WebForms

In cases when you really need to scale out your XAF websites among multiple servers,  you can load balance web servers and route requests from clients to the same content server (aka "sticky" sessions): ASP.NET WebForms: Deploy XAF ASP.NET Applications to Amazon Web Services - Part 5: Load balancing . We do not provide special guidelines or documentation for this particular scenario as an XAF WebForms app is a regular ASP.NET WebForms app that maintains its state within the ASP.NET session.  That said, you can look for public community resources devoted to ASP.NET apps with a similar configuration. For instance, you may find other links under the eXpressApp Framework > Deployment > Deployment Tutorial > Deployment Troubleshooting Guide article helpful as well.

Important Notes

DevExpress does not assist in administering web servers or hosting environments for customers. We do not consult on various server and operating system configurations as part of our support services. For more information, please review the "Prerequisites" and "Technical Support Scope" sections. If you experience issues, we recommend that you first make sure that your deployment scenario works without XAF. Try a pure Blazor Server app (with the same database and XPO or EF Core for data access). Once you resolve issues with that application, an XAF Blazor app should work as expected.

Show previous comments (13)
Dennis Garavsky (DevExpress) 6 years ago

    @Markus: Thank you for your message. The old link became invalid after all the LinkedIn platform updates over the years. I cannot quickly locate this post anymore. Probably, you will be interested to get in touch with Robert Anderson (find his contacts at https://www.devexpress.com/aboutus/mvp/) who also built a complex Web app using XAF & XPO.

    RA RA
    Robert Anderson 6 years ago

      The LinkedIn group seems is here: https://www.linkedin.com/groups/2367509/ but I can't see anything older than 3 years ago.

      The info in the my old post about deploying XAF to Amazon is still pretty valid: Deploy XAF ASP.NET Applications to Amazon Web Services: Part 5

      I also have written a different old series of posts on load testing XAF with some further details of our setup. See here: Load Testing XAF: Part 5 - Analysis

      BF BF
      Bernd Freigang 6 years ago

        Hi Dennis, Hi Robert!

        Thank you very much for your hints, I will check this!

        Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

        Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.