Centos 8 insufficient memory for the Java Runtime Environment to continue

On Linux Centos 8, with Apache Tomcat that would start but almost immediately fail. There is 4 Gb of Ram assigned, and without any load, the service kept crashing. It was really strange.

The Java installed was Openjdk version "1.8.0_312", and Apache Tomcat version 9.0.46.

What was the issue?

Each attempt to start Tomcat, it fails then from the logs:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f32cd0a7000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 262144 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid49973.log

How can this be? There is 4Gb assigned to the server, with lots of free memory. As can be seen in the configuration, the maximum memory of 1024Mb is assigned to Tomcat. The configuration file is  <TOMCAT>/bin/setenv.sh

export CATALINA_OPTS="-server -Xms1024M -Xmx1024M"

JAVA_OPTS="$JAVA_OPTS -XX:+UnlockExperimentalVMOptions -XX:+UseTLAB -XX:+UseG1GC -XX:G1MaxNewSizePercent=30 -XX:InitiatingHeapOccupancyPercent=40 -XX:G1NewSizePercent=25 -XX:ConcGCThreads=2 -XX:ParallelGCThreads=2 -Dfile.encoding=UTF8 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:tomcat_gc.log"

export JAVA_OPTS

Here is the result of command free, hardly any memory is in use (less than 80% for sure) when tomcat is NOT running

Why?

Some how Linux was assigning memory usage not in the way I was expecting. Lets check values for memory overcommit with command

sysctl -a |grep overcommit

Or only display how memory is assigned

cat /proc/sys/vm/overcommit_memory

An explanation of this setting

vm.overcommit_memory=0 setting means that the allocated virtual memory is unlimited. Having unlimited virtual memory is important for database servers, which retain extra unused virtual memory allocations for dynamic memory management. Unreferenced allocated memory is not backed by RAM or paging space on Linux systems. 

vm.overcommit_memory=1 setting where the kernel performs no memory overcommit handling. This increases the possibility of memory overload, but improves performance for memory-intensive tasks. 

vm.overcommit_memory=2, setting limits the total amount of virtual memory that can be allocated, which can result in unexpected errors.

How was this resolved?

What worked for me is to set vm.overcommit_memory=1 and ensure Tomcat starts with memory size to use the free space. This is done by updating the variable vm.overcommit_memory to 1, then in tomcat configuration as mentioned earlier, set the CATALINA_OPTS="-server -Xms1024M -Xmx1024M". Once it was working, I gradually increase the memory allocation.

Start with the following command (without reboot). Suggest before applying commands below, to stop any HTTP, PHP-FPM, NGIX and Tomcat services if running.

echo 1 | sudo tee /proc/sys/vm/overcommit_memory

sysctl -p

Restart the Tomcat application and watch Tomcat running smoothly. If all works, then this can be applied permanently by editing the file in /etc/sysctl.d

Now, 

When Tomcat is running with 1024Mb memory assigned.

When Tomcat is running with 2048Mb memory assigned.

These are just overly simplified notes, which might help. Best to still refer to the experts that is managing the server, as each case might be different.