Working With Reverse Engineering Output for Benchmarking and Further Use

Various tools exist to reverse engineer software source code and generate design information, such as UML projections. Each has specific strengths and weaknesses, however no standardised benchmark exists that can be used to evaluate and compare their performance and effectiveness in a systematic manner. To facilitate such comparison we introduce the Reverse Engineering to Design Benchmark (RED-BM), which consists of a comprehensive set of Java-based targets for reverse engineering and a formal set of performance measures with which tools and approaches can be analysed and ranked. When used to evaluate 12 industry standard tools performance figures range from 8.82% to 100% demonstrating the ability of the benchmark to differentiate between tools. Most reverse engineering tools can provide their output in the Extensible Metadata Information (XMI) format. Theoretically this should ensure tool interoperability but in practice the implementation of the XMI standard varies widely to the point where outputs cannot be exchanged between tools. In addition, this severely hinders the systematic usage of reverse engineering tool output, for example in a benchmark or for use in other analysis. To aid the comparison, analysis and further use of reverse engineering XMI output we have developed a parser which can interpret the XMI output format of the most commonly used reverse engineering applications, and is used in a number of tools. These tools offer the facility for standalone examination of one or more XMI files, comparison between outputs for benchmarking or measurement, the use of XMI within Eclipse to generate UML projections in UMLet, and use of reverse engineering output in combination with other sources of relationship information. Given the imperfect performance of the majority of the reverse engineering tools tested by the benchmark a future direction of research is the combination of different sources of information, multiple tool output or other data, to build a more complete and accurate picture of structural relationships within source code.