ELIMINATE CONSECUTIVE FULL GCs

Full GC is an important event in the garbage collection process. During this full GC phase, garbage is collected from all the regions in the JVM heap (Young generation, Old generation, and Metaspace). Full GC tends to evict more objects from memory, as it runs across all regions. A Full GC event pauses all the application threads that are running on the JVM. During this period, no customer transactions will be processed. JVM will use all the CPU cycles to perform garbage collection. Due to that CPU consumption will be quite high. Thus in general full GCs aren't desired. Needless to ask the desirability of consecutive full GCs. Consecutive full GCs will cause following problems:

1. CPU consumption will spike up.

2. Because JVM is paused, the response time of your application transactions will degrade. Thus it will impact your SLAs and cause poor customer experience.

In this article, let's learn about this consecutive full GCs - What is it? What causes it? How to fix it?

What is Consecutive full GCs?

It's always easier to explain through an example. So let's examine the GC log of a real world application, which suffered from this consecutive full GC problem. Notice the highlighted portion of the first graph. You can see the full GCs to be consecutively running (red triangles in the graph indicates full GC). If full GC runs consecutively, then it's indicative of a problem.

Even though full GCs were consecutively running, it wouldn't able to reclaim enough memory. You can observe it in the second graph (which shows the reclaimed bytes). In this graph you can see the memory reclaimed from these full GCs to be very less. It is because most of the objects in memory are in active use.

What causes Consecutive full GCs?

Consecutive Full GCs are caused because of one single reason: Under allocation of JVM heap size. It's indicative of the fact that application needs more memory than what you have allocated. In other words, you are trying to fit in a truck load of objects in a small compact car. In order to fit truck load of objects, you need a truck and not a small compact car.

Now you might have a question, my application was running fine all along, why all of a sudden I see this consecutive Full GC problem? That's a valid question. Answer to this question could be one of the following:

1. Your application's traffic volume has started to grow since the last time you have tuned the JVM heap size. May be your business is improving, more users have started to use your application.

2. During your peak volume time period, more objects would get created than normal time. May be you didn't tune your JVM for peak traffic volume or your peak traffic volume has increased since the last time you have tuned the JVM heap size.

How to solve consecutive Full GCs?

Consecutive Full GCs can be solved through one of the following solution:

1. Increase JVM Heap Size

Since consecutive Full GCs runs due to lack of memory, increasing JVM heap size should solve the problem. Say suppose you have allocated JVM heap size to be 2.5GB, then try increasing it to 3 GB and see whether it resolves the problem. JVM heap size can be increased by passing the argument: '-Xmx.' Example:

 -Xmx3G

This argument will set the JVM heap size to be 3 GB. If it still doesn't resolve the problem then try increasing the JVM heap size step by step. Over-allocation of JVM heap is also not good either, it might increase the GC pause time as well.

2. Add more JVM instances

Adding more JVM instances is an another solution to this problem.When you add more JVM instance, then traffic volume will get distributed. The amount of traffic volume handled by one single JVM instance will go down. If less traffic volume is sent to a JVM instance, then less objects will be created. If less objects are created, then you will not run into the consecutive Full GC problems.

3. Batch Jobs

When batch jobs run, they can trigger a significant amount of traffic to the application. This surge in traffic generates a large number of objects within the JVM, potentially leading to consecutive GC problems. To mitigate this, consider triggering batch jobs with a limited number of threads. This approach can help prevent batch traffic from overwhelming your JVMs.

4. Regulate Traffic

Heavyweight transactions can sometimes overwhelm your application. For instance, if your application allows users to create reports, this process may involve querying the database and loading a substantial number of objects into memory. If multiple such report creation requests gets executed on the same JVM, it can overwhelm the JVM. Explore ways to distribute these heavyweight transactions across multiple JVMs to avoid overloading any single one.