Eclipse Internal Error - Out of Memory

[chrislake]

We currently have issues using Eclipse Oxygen 3 and PMD Eclipse plugin (I've just updated to 4.0.16.v20180412-0833 because I saw similar logs to #29 with ~1900 PHP errors, but they still appear).

Essentially, after creating a workspace, the code is checked out (just one project), and a build script runs, and finally the workspace is refreshed to load all the new files from that build script. At this point, Eclipse is using ~500Mb of memory. Now a Review Code process kicks in (sometimes a great number of them), and quickly Eclipse uses up ~4Gb of memory and eventually spits out an "Internal Error", "An out of memory error has occurred. ... Do you want to exit?" (in this instance it's a small VM).

On a bigger machine, Eclipse can be started at ~4Gb, then run up 12Gb reviewing code before throwing the same error.

I've cranked the PMD logging right the way up to ALL but it seems that there no obvious error being produced. It does seem that it tests every single file in the workspace, regardless of the "Filter Files" settings (I tried exclude ".", include ".java" and similar (.java, **/.java, .*/.*java, etc)), or filters in the ruleset file itself.

Surely this excessive memory usage isn't right. Is there anything that can be done? Is there any information I can provide you with to help diagnose the problem?

[chrislake]

Follow up:

So when I import the project, PMD runs using ResourceVisitor, and appears to only consider files which are part of a source path for the project. The memory stays stable as the "Review Code" process runs without problems, completing checks on 2223 JAVA files, and 60 non-JAVA files.

After running the the build script, I refresh the workspace and now PMD runs with DeltaVisitor, adding all new files found to the workspace (including 1357 JAVA files, and 4736 non-JAVA files), even though they are not part of a source path, with logging now including:

DeltaVisitor                     Visiting added resource Clazz.class
BaseVisitor                      discovered language: LanguageModule:PHP: Hypertext Preprocessor(PhpLanguageModule)+version:
net.sourceforge.pmd.PMDException: There were processing errors!
Clazz.class: Error while processing Clazz.class net.sourceforge.pmd.PMDException: Error while processing Clazz.class

Eventually the 4Gb (-Xmx4096m) is used up and the OOM error occurs.

Interestingly, after killing Eclipse and relaunching, we're back to ResourceVisitor, and the PMD log shows it going over the same files as the first launch, with a few extras added by the build script, but now memory usage is unstable, maxing out at 4Gb before the "Review Code" process finishes. Comparing the log files, it actually looks as though it almost makes it, falling short by only a couple of files (first attempt), or only finishing up (second attempt, only "Processing marker directives", "End of processing marker directives" and "ReviewCode command has ended." was missing form the log).

So, I increased Eclipse's memory to 8Gb (-Xmx8192m), and the "Review Code" process topped out at around ~6.5Gb, sank back down to ~4Gb and left eclipse using ~60-100% CPU for a long while and never recovering. After sometime, the PMD log does look to say it has completed.

Eclipse Error log:

!ENTRY org.eclipse.core.jobs 4 2 2018-07-19 11:46:14.903
!MESSAGE An internal error occurred during: "ReviewCode".
!STACK 0
java.lang.OutOfMemoryError: Metaspace
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1007)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:545)
	at java.base/java.net.URLClassLoader.access$100(URLClassLoader.java:83)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:453)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:447)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:446)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:563)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:550)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
	at net.sourceforge.pmd.lang.java.typeresolution.PMDASMClassLoader.loadClass(PMDASMClassLoader.java:75)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.populateType(ClassTypeResolver.java:1213)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.populateType(ClassTypeResolver.java:1193)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.searchNodeNameForClass(ClassTypeResolver.java:290)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.visit(ClassTypeResolver.java:331)
	at net.sourceforge.pmd.lang.java.ast.ASTName.jjtAccept(ASTName.java:35)
	at net.sourceforge.pmd.lang.java.ast.AbstractJavaNode.childrenAccept(AbstractJavaNode.java:56)
	at net.sourceforge.pmd.lang.java.ast.JavaParserVisitorAdapter.visit(JavaParserVisitorAdapter.java:10)
	at net.sourceforge.pmd.lang.java.ast.JavaParserVisitorAdapter.visit(JavaParserVisitorAdapter.java:314)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.visit(ClassTypeResolver.java:939)
	at net.sourceforge.pmd.lang.java.ast.ASTPrimaryPrefix.jjtAccept(ASTPrimaryPrefix.java:42)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.visit(ClassTypeResolver.java:775)
	at net.sourceforge.pmd.lang.java.ast.ASTPrimaryExpression.jjtAccept(ASTPrimaryExpression.java:21)
	at net.sourceforge.pmd.lang.java.ast.AbstractJavaNode.childrenAccept(AbstractJavaNode.java:56)
	at net.sourceforge.pmd.lang.java.ast.JavaParserVisitorAdapter.visit(JavaParserVisitorAdapter.java:10)
	at net.sourceforge.pmd.lang.java.ast.JavaParserVisitorAdapter.visit(JavaParserVisitorAdapter.java:374)
	at net.sourceforge.pmd.lang.java.typeresolution.ClassTypeResolver.visit(ClassTypeResolver.java:1075)
	at net.sourceforge.pmd.lang.java.ast.ASTStatementExpression.jjtAccept(ASTStatementExpression.java:21)
	at net.sourceforge.pmd.lang.java.ast.AbstractJavaNode.childrenAccept(AbstractJavaNode.java:56)
	at net.sourceforge.pmd.lang.java.ast.JavaParserVisitorAdapter.visit(JavaParserVisitorAdapter.java:10)

With this error, I see that my -XX:MaxMetaspaceSize is at 512m, so I changed that to 2048m to see what would happen and now Eclipse successfully runs the "Review Code" process. Memory does increase somewhat, but Eclipse is useable once finished and a restart now doesn't restart PMD from scratch and development can continue :)

OK, so back to the start:

1. Import clean project:
   a. Loaded classes ~12500 -> ~21500
   b. Heap ~500Mb -> ~500 Mb
   c. Metaspace ~75Mb -> ~185Mb
   

2. Build and refresh:
   a. Loaded classes ~21500 -> ~110000
   b. Heap ~500Mb -> ~2500 Mb
   c. Metaspace ~75Mb -> ~950Mb
   

3. Import project where the build script has occurred
   a. Loaded classes ~12500 -> ~60000
   b. Heap ~500Mb -> ~750 Mb
   c. Metaspace ~75Mb -> ~950Mb
   

So I guess the question is, should PMD (or the Eclipse Plugin), release the metadata it is using to allow garbage collection on the Metaspace?
Is there even a problem?
Perhaps I should not be setting the MaxMetaspaceSize at all, as a few things I've read suggest?

For now, I have a workaround by increasing the MaxMetaspaceSize setting.

[adangel]

Thanks for the detailed analysis!
The memory problem sounds like that one described in #49 - the metadata is where the loaded classes are stored. I guess, the reason why the memory usage is higher after you have relaunched eclipse is: now the project is compiled and the classes exist, so the classloader of PMD can load the project's classes. I assume, the first time you imported the project, it has not been compiled yet?

The difference between ResourceVisitor and DeltaVisitor looks like an additional bug, though.

I thought about, whether the memory of claimed by the classloader can be garbage collected. There is one reason, why this might not happen: If PMD found violations, we might keep them inside the eclipse plugin for displaying. These violations contain a reference to the offending AST node, which in turn might keep a reference to the resolved class, loaded by the classloader. So, that might be a reason, why the memory is not freed - but still to be confirmed.

[adangel]

@chrislake I've released a new version. Could you verify, whether the out of memory problem is gone now? Thanks!
Update site: https://dl.bintray.com/pmd/pmd-eclipse-plugin/updates/

[chrislake]

@adangel,

I can confirm that with the new version, the memory usage remains minimal and stable and I no longer receive an OOM error.

Great job, thanks for the quick turnaround :)

[adangel]

Awesome, thanks for testing!

[farnetto]

Brilliant, thank you!