Testing Eclipse Plug-ins: Lessons from the Field

by Elliotte Rusty Harold

Testing Eclipse Plug-ins is (unnecessarily?) painful. Don’t believe me? Consider this bug.

A user reports a bug: I check it out. Yep, it’s a bug all right It seems we’re not adding a tab group to our launch configuration type and we should. OK, the fix is easy enough:

public class AppToolsTabGroup extends AbstractLaunchConfigurationTabGroup {
 	@Override
 	public void createTabs(ILaunchConfigurationDialog dialog, String mode) {
   		ILaunchConfigurationTab[] tabs = new ILaunchConfigurationTab[0];
   		setTabs(tabs);
 	}
}

(Actually not that easy. I went through a couple of variations of this code before hitting on this one that actually works. The call to setTabs was particularly surprising.)

A quick manual test shows that the problem is fixed, but I want to make sure it doesn’t occur. So I whip up a unit test:

@Test
public void testCreateTabs() {
	AppToolsTabGroup group = new AppToolsTabGroup();
  	group.createTabs(null, "");
  	Assert.assertEquals(0, group.getTabs().length);
}

(Yes, I really should have written the test first, but that’s a subject for another article. I should also write a test for the XML config used here, and in fact I did, but again: another article.)

It’s a basic test, but it will make sure nothing regresses too badly. Furthermore it’s a nice framework if someone wants to adds tabs in the future. The test is already here and will break, reminding whoever does that work that they need to write a new test.

I run the test in Eclipse and, to my surprise, it passes.

I run it at the command line. Maven spits out about a thousand screens of irrelevant chart junk, but again the test passes.

All looks good so I commit and push the branch to Github. Travis picks up the pull request, runs the test, and not so promptly fails:

!ENTRY org.eclipse.equinox.preferences 4 2 2016-07-22 22:02:05.217
	!MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.equinox.preferences".
	!STACK 0
		java.lang.IllegalStateException: Workbench has not been created yet.
			at org.eclipse.ui.PlatformUI.getWorkbench(PlatformUI.java:93)

WTF? Why is this trivial little class that returns an empty array loading the preference framework at all? This code is poorly factored. Unfortunately the problem is in the superclass and seems endemic to Eclipse. There’s not a lot I can do about this right now.

Some googling suggests the test has a race condition. I need to make sure my platform is initialized first. OK, let’s do that:

@Before
public void createWorkbench() {
	if (!PlatformUI.isWorkbenchRunning()) {
    	WorkbenchAdvisor workbenchAdvisor = new WorkbenchAdvisor() {
      		@Override
      	 	public String getInitialWindowPerspectiveId() {
        		return null;
      	 }
    	};
    	PlatformUI.createAndRunWorkbench(PlatformUI.createDisplay(), workbenchAdvisor);
  	}
}

Everything passes locally, so I push to Github and Travis fails again but in a different way:

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.474 sec 
<<< FAILURE! - in 	com.google.cloud.tools.eclipse.appengine.localserver.ui.AppToolsTabGroupTest
	testCreateTabs(com.google.cloud.tools.eclipse.appengine.localserver.ui.AppToolsTabGroupTest) Time elapsed: 0.466 sec 
<<< ERROR!
	org.eclipse.e4.core.di.InjectionException: java.lang.NoClassDefFoundError: javax/annotation/PostConstruct
			at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:389)

I have no idea what to do now, so I start working on this article instead in the hopes it will spark some inspiration. One thing I do is look at the code for the superclass AbstractLaunchConfigurationTabGroup and then things get really strange because I can see no way that code invokes anything that I’m seeing in the failure stack trace.

And it works! Because while I’m writing this my teammate Brian (who knows way more about Eclipse internals than I do) suggests that I “configure the tycho-surefire-plugin to useUIHarness and product to org.eclipse.sdk.ide” instead of setting up the workbench.

It seems that the Eclipse is loading junk it shouldn’t need, at least not for this test. Anyway that’s my working hypothesis but since the tests now pass in Travis I don’t over-think and move on to the next bug.

And that’s for one trivial test to make sure an array is empty and non-null. This is perhaps the deepest problem for testing Eclipse code, and one we struggle with daily. Much of the Eclipse codebase assumes there’s a huge monolithic framework loaded and running in memory, full of statics and globals, and with complete access to a single display. Furthermore there’s only one instance of Eclipse running. Honestly, this isn’t a great design for any application, and it’s an absolute disaster for testability.

There’s no one answer to this problem, no silver bullet to make Eclipse plug-in code testable. In fact, even if we were to throw out Eclipse and go back to the drawing board with testability first in mind, we’d rapidly discover that underlying native GUI frameworks such as X-Windows, Cocoa, and .Net are themselves far from perfectly testable and full of the same sort of static, global, singleton state that makes tests so hard to write.

But testing is too important to just throw up our hands and say we won’t do it. Comprehensive tests are a prerequisite for shipping robust software. No one said software development was easy. Even if the tests are challenging, we still need to write them. In my talk we’re going to explore various imperfect but still useful techniques to manage this complexity including:Elliotte Rusty Harold

  • Separate Maven libraries for non-Eclipse dependent code
  • Mocking what can be mocked
  • SWTBot
  • Testing plugin.xml files directly

I hope to see you there!

 

Our Sponsors

For information about becoming a sponsor, please visit the EclipseCon Europe 2016 sponsor prospectus page.

Elite Dual ECE/OSGi CE

Premium

Basic

Project Quality Day

IoT Theme Day

Media

EclipseCon Support Other Events

Our Other Events

Eclipse events are hosted all over the world!

  • Foss4G NA 2018
  • Eclipsecon France 2018