Stateful Architecture
XAF Web apps are stateful meaning that ASP.NET stores user state in memory to combine fast responses to user actions with server-side data processing. Technically, an XAF WebForms application uses the ASP.NET WebForms session to store non-serializable service information (XafApplication, WebWindow and many other entities by means of the IValueManager interface, etc. That means that in ASP.NET WebForms you cannot use a session state mode other than InProc. ASP.NET Core Blazor Server is also a stateful app framework (Microsoft Docs) - most of the time, the app maintains a connection to the server and the user's state is held in the server's memory in a circuit.
Pros
Maintaining the session state allowed us to deliver a greater functionality and an easier development model to XAF developers. For instance, you can implement a Controller and access and store a reference to the current View and its controls, ObjectSpace, other Controllers, etc. without caring a lot about ASP.NET WebForms or ASP.NET Core Blazor Server page life cycle specifics - just like in desktop clients (WinForms or WPF).
Cons
However, the chosen stateful architecture has its own drawbacks that may require a bit more effort from you while addressing certain business requirements such as good scalability. This stateful architecture means that memory consumption increases with each user. After a certain number of simultaneous users, the application can reach its hosting machine’s memory limit. Even if you have powerful web servers, thousands of concurrent requests may stack in a lock somewhere (race conditions or concurrency issues may occur). That is simply because XAF's core (Application Model, Types Info System, etc.) is not designed and is not tested against such high-load web scenarios.
Scalability Considerations
Unsupported & Typical Scenarios
XAF Web apps (both ASP.NET WebForms or ASP.NET Core Blazor Server) were not designed or tested for high-load scenarios where a single web server handles hundreds or thousands of concurrent web clients (for instance, in SaaS apps). Thus, we cannot guarantee that you will not meet unsolvable problems if you go this route with XAF (even behind the maximum user load). It is like the same car cannot be expected to be a comfortable sedan, SUV and F1 racer at the same time.
Like many software, XAF was born as an in-house project serving our Support and Client Centers (WinForms UI with about 200 concurrent users at peak times and WebForms UI with 5-10 concurrent users typically). XAF Web apps are mostly used in a local Intranet with a limited set of users (often deployed internally). It is important to understand that our tool is designed for a set of finite scenarios and using it in an edge case may not work well.
Most XAF WebForms or Blazor Server UI customers do not have more than 50 users according to their feedback (typically it is even less than 20-30 users). We heard of 1-2 edge cases with 300 and even 600 concurrent users, but we cannot imagine how much application and web server software customizations it took to achieve acceptable performance under such a load. We also ran internal tests with up to 30-50 concurrent web sessions for a single web server.
Web Server Hardware & Software Requirements
Initially, when choosing appropriate web server hardware or hosting environment, you need to assume that about 10-20 MBof memory will be required per each user session (this number may vary depending on the application complexity). We cannot provide a universal calculator for web server hardware/software requirements, nor can we comment on overall performance without tests. The complexity of your business model and implemented behaviors are significant factors in throughput/performance. Ultimately, performance will depend on development decisions, application type, environment, and even tested use-case scenarios. A few examples of factors that affect application performance are the number of persistent classes and their fields, Controller design, Application Model customizations, availability of memory intensive operations to end-users (frequent import/export of large data amounts, or complex report generation).
Load-Test First
First of all, make sure that scalability is really a practically confirmed "problem" for you, and you need to handle it. In many cases, this scaling over multiple web servers is not required, and your XAF website needs can already be handled by a single VM instance somewhere in the Cloud.
That said, the first thing is always to do before worrying about this is to practically test how your application behaves under the planned load in an environment close to the production one. For this real testing, you may be interested in this community video by Joche Ojeda - XAF Blazor load test (connecting 100 clients), and our GitHub example - XAF Blazor load testing on Linux and MySql using Puppeteer and GitHub Actions.
When planning an automated load (stress) test, consider that at peak time a regular XAF Web application user will perform no more than 1-2 operations per second or so, depending on your real use cases. It does not matter which application action you are going to perform (be it CRUD, navigation or refresh) as they are all comparable, unless you are loading ListView with thousands of records without the server mode. Here it is important to note that XAF Web apps are not optimized for load-testing tools that query the web site too often and their results may vary or even be inaccurate, because such tools often do not wait for full page initialization. To avoid such inaccuracies, consider waiting a few seconds for the previous operation to complete using the specialized 'wait' commands provided by your testing tools (refer to their docs) - we have a similar delay between commands in our internal EasyTest functional tests.
Vertical Scaling (scaling up)
To a certain moment in the application life cycle as the business demand and the number of concurrent users increase, you can serve more users by providing more powerful hardware (vertical scaling), whose specs can only be determined empirically based on the request handling speed requirements for your specific application and environment. In general, the better hardware you provide, the better results you will likely get. Bear in mind, however, as written in docs, XAF apps are not designed to handle thousands of concurrent users at a time. To meet such business requirements, you would better consider using other tools to build your app.
Finally, prior to thinking about providing more powerful hardware, I recommend you check out the tips from the How to improve the application's performance article to see precisely where the application response time is spent and if something in your custom code can be improved. If you find anything related to the DevExpress code, we will be more than happy to look at your debuggable sample and repro instructions.
Horizontal Scaling (scaling out)
ASP.NET Core Blazor Server
You can minimize the aforementioned effects and improve performance if you use horizontal scaling (scaling out) instead of the vertical one (scaling up) - the fewer concurrent requests for a web server instance there are, the faster it should work. For instance, each pod/web server handles say 10-20 clients instead of 100 = decreases the chances to wait in the lock.
We published a GitHub example that shows how to deploy an XAF Blazor application to a Kubernetes cluster with horizontal autoscaling. We tested the application in two types of clusters: locally-run K3s and Azure Kubernetes Service (AKS). The maximum pod replica number (20) allowed around 300 concurrent users. An AKS cluster needs two nodes (B4ms machines: 4 Cores, 16 GB RAM) to operate with such a number of pod replicas and the same load.
We hope that our working example along with Deployment Recommendations for XAF Blazor UI Applications will address all related questions on Docker, K8s, Linux, nginx, and load-balancing in the future.
Will your own XAF Blazor app work with 100, 200, 300, or more concurrent users with similar performance, provided the hardware/software are the same? It depends - our example is not an official guarantee by any means. In brief, every application is unique. Even with horizontal scaling, we recommend that you carefully:
- Test your XAF Web apps under conditions close to your production environment;
- Emulate the user load. The following GitHub example may prove useful - XAF Blazor load testing on Linux and MySql using Puppeteer and GitHub Actions;
- Measure performance over time and decide on XAF's suitability after that.
ASP.NET WebForms
In cases when you really need to scale out your XAF websites among multiple servers, you can load balance web servers and route requests from clients to the same content server (aka "sticky" sessions): ASP.NET WebForms: Deploy XAF ASP.NET Applications to Amazon Web Services - Part 5: Load balancing . We do not provide special guidelines or documentation for this particular scenario as an XAF WebForms app is a regular ASP.NET WebForms app that maintains its state within the ASP.NET session. That said, you can look for public community resources devoted to ASP.NET apps with a similar configuration. For instance, you may find other links under the eXpressApp Framework > Deployment > Deployment Tutorial > Deployment Troubleshooting Guide article helpful as well.
Important Notes
DevExpress does not assist in administering web servers or hosting environments for customers. We do not consult on various server and operating system configurations as part of our support services. For more information, please review the "Prerequisites" and "Technical Support Scope" sections. If you experience issues, we recommend that you first make sure that your deployment scenario works without XAF. Try a pure Blazor Server app (with the same database and XPO or EF Core for data access). Once you resolve issues with that application, an XAF Blazor app should work as expected.
We have over 1500 user across 60 customers now and we really need to ability to take servers in and out of the load balancer. Right now if a server dies, the user is forced to log on again. This has been here for 4 years. Are there any plans to make XAF Web scalable in this scenario?
Thanks.
@Robert: I am afraid I cannot give any promises regarding making XAF Web apps stateless and suggest you consider alternatives for this particular scenario. While this scenario looks helpful overall, XAF Web UI was not originally designed to be used by many concurrent users at a time. Technically, supporting this means rewriting a lot of things not to store anything within the ASP.NET session + huge breaking changes for existing apps as there will be a different application life cycle and flow. In any event, we will take your feedback into account for the future.
'Scaling out' is easy. It is only 'scaling in' which results in dropped sessions. Ideally DevExpress will come up with some wizardry to permit XAF statelessness, but meanwhile here are some suggestions.
Short term, the best solution is to get the load balancer to handle the graceful shutdown of a surplus server. When a server needs to be removed, the load balancer should be configured to stop sending NEW sessions to the server, but continue to redirect EXISTING sessions to it. Then when the surplus server eventually has zero sessions (because of logoffs and session timeouts) then you can remove it without consequences. (You could either monitor the number of existing sessions, or just wait for a suitable long time, e.g., an hour before assuming all sessions are over).
Unfortunately I've never got around to trying to set this up, but I believe it's possible with many hardware load balancers or a reverse proxy such as nginx. Also, it might not be too hard to write your own C# load balancer (see for instance http://www.codeproject.com/Articles/318290/How-to-Implement-Load-Balancing-to-Distribute-Work) which you could extend to handle the graceful shutdown redirection and monitoring.
On a related note, the Elastic Load Balancer used by the Amazon Cloud does not have any such capabilities out of the box, but there is this forum post https://forums.aws.amazon.com/thread.jspa?threadID=61278 which you can follow in case they support it one day. (I believe the same problems exist with Azure). You might also like to read my blog post on the issue http://blog.zerosharp.com/deploy-xaf-asp-dot-net-applications-to-amazon-web-services-part-5-load-balancing/
Thanks for your additional feedback and links, Robert. I think it will be helpful for other users with the same requirement until an ultimate solution for such scenarios is provided.
Proposed solutions/Future considerations:
3. Make it possible to serialize/deserialize session data of XAF ASP.NET applications (support the StateServer and SQLServer session state modes, AppFabric caching, etc.).
1 vote.
Vote 2
I am waiting desperately for this solution. Any ETA?
Hello Mohammed,
To process your recent post more efficiently, I created a separate ticket on your behalf: T258086: About scaling XAF ASP.NET applications. This ticket is currently in our processing queue. Our team will address it as soon as we have any updates.
Me too:
3. Make it possible to serialize/deserialize session data of XAF ASP.NET applications (support the StateServer and SQLServer session state modes, AppFabric caching, etc.).
1 vote.
Hello Randy,
To process your recent post more efficiently, I created a separate ticket on your behalf: T291798: About scaling XAF ASP.NET applications. This ticket is currently in our processing queue. Our team will address it as soon as we have any updates.
Any development yet?
Since we are dependant on InProc sessions which rely on App Pool NOT Recycling , how can we certify our users won't get diconnected from our web application built with Xaf?
SessionState seems to be THE solution in our case, but since DX can't be serialized, what are we supposed to do?
3 years since last comment, none initiated by DevExpress Team, is this Support Ticket real?
@Programmeur Surplus: I've answered you in the InProc sessions in Web UI ticket.
Hi Dennis,
the link to the LinkedIn XAF group you mentioned in your answer above seems to be invalid. Is there a current link where I could see this information?
Kind regards,
MArkus
@Markus: Thank you for your message. The old link became invalid after all the LinkedIn platform updates over the years. I cannot quickly locate this post anymore. Probably, you will be interested to get in touch with Robert Anderson (find his contacts at https://www.devexpress.com/aboutus/mvp/) who also built a complex Web app using XAF & XPO.
The LinkedIn group seems is here: https://www.linkedin.com/groups/2367509/ but I can't see anything older than 3 years ago.
The info in the my old post about deploying XAF to Amazon is still pretty valid: Deploy XAF ASP.NET Applications to Amazon Web Services: Part 5
I also have written a different old series of posts on load testing XAF with some further details of our setup. See here: Load Testing XAF: Part 5 - Analysis
Hi Dennis, Hi Robert!
Thank you very much for your hints, I will check this!