I am waiting for a new version of VLDocking to be released – and hence some delay in releasing next version of Marathon. Meanwhile, I worked on few more features (some small and some big) for easing recording of the automated tests.

1. When recording a test case marathon now minimizes the main window and shows a small controller window. This should reduce the clutter of the screen to a large extent. The main window is restored when either you close the controller window or stop recording the test case.
2. A new preferences dialog allows you to change options like hot key and menu click for popping up the Marathon assertion menu. You can even change the font used for the editor – that is the test case files that are shown.
3. Along with #2 comes the ability of using Marathon for recording test cases even when your application pushes its own event queue. If the application under test makes use of event queue, just use the keyboard short cut for popping up the Marathon assertion menu. The mouse click does not work.
4. Our Java testing tool’s UI has been spruced up. We are using JGoodies looks and VLSolution’s vldocking framework for most of the UI enhancements. Now you have the ability to move the panels like test tree panel or Navigator panel (that shows the files) whereever you want in the desktop. Marathon even remembers the preferences and restores the same state when you restart it.
5. Project directory structure for the testing projects in Marathon is changed. Now all paths (like classpath etc.) are saved relative to project directory. The MPF file is no more needed and the project settings are saved in the Project directory in a file named ‘.project’. Now, the project settings can also be modified from within Marathon. The Edit menu sports an option called ‘Project settings…’ (what else?) using which you can modify the settings for the current project. The CaptureScripts directory is renamed to Modules – this reflects the ability of Marathon to use any jython functions and not just captured scripts as functions.
6. A new Project Settings option ‘Ignore Components’ has been added to the test project. You can use this to inform Marathon that it need to ignore clicks on the components. By default when a new project is created this is populated with JScrollBar, JToolBar etc. You can update it if you find Marathon recording extraneous clicks while recording your test cases.
7. A new scripting element ‘select_menu’ makes its appearance in Marathon now. This is used to record menu activations either using keyboard or mouse.
8. Recording on collection components like JTree, JTable improved. Now Marathon records multiple selections on these components – so you can stop asking me to record CTRL and SHIFT keys while recording . Two more controls also make debut in this release. JSlider and JProgress bar. Those applications that make use of these two controls need not have work-arounds in the test cases.
9. Finally, the keystroke recording for test cases is here. Marathon not only records the keystrokes but it tries to be intelligent about not recording keystrokes that do not make sense. If a keystroke is typed that is a menu accelerator key, marathon records it differently.

There are lot more changes comming in next version – which I am toying with the idea of calling Version 1.0. The full list of changes will ofcourse be available in the ChangeLog.

There are few more things I am working on currently. Most important of those are:

1. Mouse clicks record with position and modifiers. These will be recorded only when Marathon does not support a particular component.
2. Drag and Drop. John submitted a patch for this. I am working on making it generic enough to work with all Marathon components.
3. A new naming strategy that gives more importance on what is shown on the screen like button texts and labels, rather than components name given by setName().

Comments?

Updated. Thanks to Arun for giving me a pointer to pstools.

This will be part of Marathon User Guide when it comes out. I am writing these snippets so that you can benefit from them now.

Now that you wrote a lot tests using marathon how do you run them? The problem with GUI testing is that your desktop is tied up for the duration of the tests. The monitor blinks like a tube-light with a starter problem with Windows popping up all over with crazy titles – ‘testStartAndStopRecordingWithSetupAndTearDown’ or ‘testDeleteFilesWithSomeFailures’. Even if you can live with this – all these popping up windows will be stealing your application’s focus (beside stealing yours) and some of your keystrokes and mouse clicks will go there. In the end you end up not knowing whether a test case failed because of you using the keyboard or a bug in the application under test.

Using java.awt.headless

Java since 1.4.2 supports a headless mode that is half-baked and not useful for the purposes we are talking about. Googling for ‘java headless’ turns up this page, so I am pointing this out. Java support for headless is really for applications that need to run without a display and keyboard. Anyhow if you do not need a display then the whole of this article does not make sense.

Running headless on Linux

An elegant solution I found to run GUI test suites on Linux is to use Xvbf – a X server that simulates a display, mouse and keyboard. On ubuntu, the platform of choice for me, the Xvfb package comes with a script xvfb-run. Executing an application under Xvfb is just typing:

xvfb-run marathon <path-to-your-mpf>

The standard error and output will be on your terminal, so you can see the progress of your test case execution and any errors thrown up by the test run.

Running headless on Windows

Using psexec

psexec is a small utility from Sysinternals from the pstools package. You can Download it here.

psexec allows you to run applications on remote machines. We will use it to run marathon headless on Windows. Download the pstools archive and install on the machine. Add the installed directory to the path. There are two ways you can execute Marathon – with a console and without a console.

For using it with a console (so that you can see the progress):

psexec -s cmd /c marathon -batch <path-to-your-mpf>

If you are not interested in a console:

psexec -s -d cmd /c marathon -batch <path-to-your-mpf>

Note that -s uses the system account (i.e the administrator) to execute the command. I found it easier to create a batch file that runs Marathon with all options and execute the batch file using psexec:

psexec -s cmd /c <path-to-batch-file>

Using telnet

You can run the tests headless on Windows using the telnet service that is available on Windows XP Professional. You can enable telnet service from Control Panel -> Services. Once you enable the service log-in using the command:

telnet localhost

Give your user name and password. You will land up at C:> prompt. Run your command:

marathon -batch <path-to-your-mpf>

That is it. Your display, keyboard and mouse belong to you, no annoying pop-up windows to disturb you and Marathon chugs along running the application tests. You can use -text, -html or -xml arguments along with -batch so that the test results are captured in a report.

Note: Running a telnet server is insecure. Modify your windows firewall settings to disable incoming connections. Default is to disable incoming connections from Internet – it should be OK.

Note: It will be possible to automate the log-in part provided we have a telnet client that can take input from a file and send the commands to the server. I don’t know of any. Drop me a line if you know such a client for Windows.

I am writing this post so that it can be a reminder to myself as well as giving information to all of you waiting for a new release of Marathon. Since beginning of August I am full time working for my own company, so I am able to put in lot more effort in the development of Marathon. So what is going to be the release plan? The following are the tentative release dates for new versions of Marathon:

1. Version 0.91 – August 15th
2. Version 0.92 – August 28th
3. Version 0.93 – September 4th
4. Version 1.0beta – September 11th
5. Version 1.0 – September 18th

Version 0.91

The following features are added to Version 0.91:

1. Tabbed editor

   Now you can open multple testscripts/capturescripts and fixtures in the editor multiple tabs. Consider the current functionality as rudimentary. Some more enhancements are planned for 0.92. I faced a funny problem while developing this bit. Somehow, when we put up JEditTextArea components into a tabbed pane, the content of all the tabbed panes with JEditTextArea becomes the same. It turns out that JEditTextArea’s (from jedit-syntax package) uses a static TextAreaDefault object if default constructor is used. And the TextAreaDefaults has a document also!!

2. Docking

   The result window and console are docked to the bottom pane. You can switch between the two. The JUnit panel is docked to the Navigator window. You can open it by clicking on the JUnit button.

3. Capturing output from the Application under test

   The output (stderr/stdout) from the application as well as the test scripts is captured and displayed in the console window. The output from script (using system.err or system.out) is italicised. The stderr output is in red color and stdout is in blue color.

4. Runtime port

   Marathon uses RMI to communicate with the AUT. Setting up of this port was a major headache in the earlier versions. Now this port is dynamically allocated by Marathon, so you do not need to set marathon.runtime.port property.

5. Playing scripts with a delay

   Now you can execute the script with a delay from the toolbar using Slow Play command. The delay can be changed by setting marathon.runtime.default.delay to a different value (default: 1000 – i.e 1second).

6. Edit menu

   Undo/Redo/Cut/Copy/Paste are added to the Edit menu item.

Version 0.92

The concentration for 0.92 is on hardening of Marathon and providing support for Java 5.

1. Run unit tests in a loop.

   I added a junitrunner script and facility so that we can execute the UTs in a loop. Found multiple failures when the UTs are run like this and fixed all of them. At this time all UTs are passing.

2. Support for Java 5

   All the UTs should pass under Java 5. Should be able to run all of them in a loop. Found a problem with Java 5. See Window.dispose() can be a recursive call.

3. ComponentResolver and DefaultNamingConvention need to be updated

   I am planning for simplifying the naming conventions used. If we take into consideration that only components on the toplevel, focused window need to be named at any time – we may not need to consider the container objects within the window. Need to check this out. This will also solve the problem where-in different names are being generated for components in Java 2 and Java 5.

4. Fix some bugs

   Now when a play is stopped by the user by quitting the application, the display window can be restored to normal state by clicking on the ‘stop’ button. Similarly, while recording if the application is exited, the display window comes back to the normal state. (However, the last few clicks might not be recorded).

Version 0.93

1. Floating toolbar and hiding Marathon main window while recording

   I am excited with this feature. I do not know about others, but I am always irritated that the screen real estate is occupied by the Marathon main window while recording test cases. I am planning to hide the main window if you press the record button. A floating (always on top) tool bar will be displayed with stop/insertscript/pause etc. relevant buttons.

   It will also be nice to have a small text area in the toolbar where the current component name hovered on is displayed.

2. Assertions

   I might leave this for the next version (post 1.0), but I will mention the feature here. There are always requests for new assertions to be added to Marathon. If we can provide a mechanism by which the tester can define a new assertion while recording it might be of great value. While recording, we already know the current focused component. When the user opens the context menu, we can provide a option to record a new assertion. That can provide a list of all getters/is<XX> functions from which the user can select an option and can record a new assertion. Giving this a name make this assertion available when recording on similar components.

1.0 Versions

Mostly documentation and more documentation.

So what do you people think? If I missed out something that you feel should be there, please drop a mail to either the list or add a comment here.

Remember that Marathon can be improved only by feedback from its users.