Blazor Server WebApp - High memory usage under Linux

Question

I am seeing extremely high memory usage from my dotnet application when running under Linux. The memory is also never released. When users use my web app they do require large amounts of memory (1-2 GB to run a report) but when they close the tab resources are released and I would expect dotnet to do the same to some extent.

What I'm experiencing is near 90-95% memory usage from my dotnet app.

Problems

1. I'm unable to determine if my server is at peak memory usage or dotnet is holding onto memory as there is little memory pressure from other processes.
2. dotnet memory dumps are huge and I don't think they should be. (~5-6 GB download vs ~30-300 MB in used objects when analysed)

Mem Diagnostics

root@a9f25bad2d02:~/site/wwwroot# dotnet-counters monitor

[System.Runtime]
    % Time in GC since last GC (%)                                         0    
    Allocation Rate (B / 10 sec)                                     209,648    
    CPU Usage (%)                                                          0    
    Exception Count (Count / 10 sec)                                       0    
    GC Committed Bytes (MB)                                              105    
    GC Fragmentation (%)                                                  43.752
    GC Heap Size (MB)                                                     68    
    Gen 0 GC Count (Count / 10 sec)                                        0    
    Gen 0 Size (B)                                                19,577,728    
    Gen 1 GC Count (Count / 10 sec)                                        0    
    Gen 1 Size (B)                                                 1,608,856    
    Gen 2 GC Count (Count / 10 sec)                                        0    
    Gen 2 Size (B)                                                49,073,776    
    IL Bytes Jitted (B)                                            4,471,180    
    LOH Size (B)                                                  26,161,384    
    Monitor Lock Contention Count (Count / 10 sec)                         0    
    Number of Active Timers                                               15    
    Number of Assemblies Loaded                                        5,110    
    Number of Methods Jitted                                          64,067    
    POH (Pinned Object Heap) Size (B)                                706,440    
    ThreadPool Completed Work Item Count (Count / 10 sec)                 17    
    ThreadPool Queue Length                                                0    
    ThreadPool Thread Count                                                4    
    Time spent in JIT (ms / 10 sec)                                        0 
    Working Set (MB)                                                   5,773 
    
    
root@a9f25bad2d02:~/site/wwwroot# free -m
              total        used        free      shared  buff/cache   available
Mem:           7817        6858         121          19         837         740
Swap:          4095        2156        1939

root@a9f25bad2d02:~/site/wwwroot# top
Tasks:  11 total,   2 running,   8 sleeping,   1 stopped,   0 zombie
%Cpu(s):  1.5 us,  0.7 sy, 50.4 ni, 45.3 id,  2.2 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7817.4 total,     89.3 free,   7262.5 used,    465.6 buff/cache
MiB Swap:   4096.0 total,   1676.2 free,   2419.8 used.    341.3 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                      
                                                                                                                           
   74 root      30  10 9812340   5.4g  19812 t   0.0  70.7  21:22.97 dotnet                                                                                                                                       
 1908 root      30  10   13948    708    540 S   0.0   0.0   0:00.18 sshd                                                                                                                                         
 1910 root      30  10    5752   2040   1808 S   0.0   0.0   0:00.03 bash                                                                                                                                         
 2526 root      30  10    9780   3448   2972 R   0.0   0.0   0:00.00 top

As you see from the dotnet-counters monitor the working set is around 5.7 GB whereas the total of bytes used from all generations, as well as the Large and Pinned Object Heaps is only around 120 Mb.

Which is corroborated with an analysis of a memory dump which was 5 Gb in size.

What have I done already

1. I've switched between Server and Workstation GC - no real difference.
2. I've performed a dotnet dump dotnet-dump collect --type Full and searched for any memory pressure from within my app.
3. I've added dotnet counters into my app to report on Gen 0, 1, 2, LOH and Heap size - all values do go up but come down again after the user leaves the site.
4. I've switched my app to Windows and see that Memory usage is more 'normal' and memory is freed after a while.
5. I've run the app with dotnet-counters collect and simulated typical user load. I have graphed the result of GC Committed Bytes by time below.

Background

• The App is running as Blazor Server
• The App is being hosted in Azure under a Linux App Service Plan (P1v3 to be precise)

Answer 1

The root cause for my issue was not to do with my app or event dotnet at all - it was to do with glib malloc. There is some dynamic sizing of memory allocation blocks going on in malloc and when an app uses lots of memory, fragmentation occurs and the memory is reluctantly released. A better write-up can be found here: https://github.com/dotnet/runtime/issues/13301#issuecomment-535641506

This is not just affecting me but other programs are affected outside of dotnet. E.g.,

• https://github.com/dotnet/runtime/issues/49317
• https://github.com/dotnet/runtime/issues/13702
• https://github.com/dotnet/runtime/issues/13301
• https://github.com/prestodb/presto/issues/8993
• https://github.com/ErythroGuild/irene/issues/315
• https://bugs.squid-cache.org/show_bug.cgi?id=4117#c39
• https://thehftguy.com/2020/05/21/major-bug-in-glibc-is-killing-applications-with-a-memory-limit/

There appears to be 2 workarounds either fix the size of glibc.malloc.trim_threshold to stop it being dynamic or reduce the number of arenas via glibc.malloc.arena_max as the default number is 8 * NumOfCores. More information can be found here: https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html

Ultimately I chose to set the environment variable MALLOC_TRIM_THRESHOLD_=131072 which fixed the problem for me.

In Azure you can conveniently use the Configuration to do this,

Which I can confirm is set via bash,

kudu_ssh_user@ff3d088f400d:/$ env | grep MALLOC
APPSETTING_MALLOC_TRIM_THRESHOLD_=131072
MALLOC_TRIM_THRESHOLD_=131072

And now I can see memory being released in my Web App 😅,