Recent Changes - Search:

The benchmarking engine: Running all pairwise comparisons

This program automatically runs all the required pairwise comparisons for a given benchmark and a given a set of methods, described into a XML file.

Installation

Requirement

  • You need a Java installation (version 6 and above).
  • You need to have the POSIX program time (that support option -p) installed and accessible (i.e. into your PATH).

Java program

The program is provided as a jar file: braseroComparator_1.1.jar. To use this program, download the jar file onto you computer.

Using the benchmarking engine

Requirement

  • First, each method which is going to be benchmarked has to be properly installed and accessible (i.e. into your PATH).
  • The java adaptor class specific to each method (see section "Adding a new method" below) needs to be accessible by java.
  • The benchmark data set that will be analyzed needs to be properly stored in accessible files (see Storing a benchmark).

Running

The benchmarkign engine can be invoked with the command:

java -jar braseroComparator_1.0.jar [OPTIONS] firstSet [secondSet] tools.xml

where:

  • OPTIONS can be
    • -h: for help
    • -d dir or --directory dir: store the result of the comparison into directory dir. By default dir is equal to the current date.
    • -t n or --thread n: run n comparisons simultaneously
    • -s or --self: secondary structures in the file firstSet are compared against themselves, in which case secondSelf must be omitted.
    • -e a.jar:b.jar:... or --ext a.jar:b.jar:...: add a new method defined into the jar files a, b (see section "Adding a new method" below)...
  • firstSet and secondSet: two files that contain the list of files containing the secondary structures respectively of the sets R and R2 and F2.
  • tools.xml: an xml file describing the different RNA comparison methods that are evaluated.

Note: at any moment, the benchmarking engine can be stopped, and it is possible to re-run it later from where it stopped (with -d pointing to the right directory, and with the same firstSet,secondSet and tools.xml files).

firstSet and secondSet

Theses files must contain a list of paths to valid files, whose format is described in Storing a benchmark. These paths can be absolute or relative in which case you must run the program from the appropriate directory. Here is an example of such a file:

references/d.16.a.P.abyssi.dp
references/d.16.b.E.coli.dp
references/d.16.b.T.maritima.dp
references/d.16.e.A.thaliana.dp
references/d.16.e.H.sapiens.dp
references/d.16.e.S.cerevisiae.dp

Each file is in the dot-parenthesis format and can contain several RNA secondary structures.

tools.xml file format.

The file tools.xml contains a description of each method (i.e. software+options+parameters) in xml format. The dtd for this file (included into the jar file) is:

<?xml version="1.0" encoding="UTF-8"?>

<!ELEMENT brasero (tool+)>
<!ELEMENT tool (options,files?)>
<!ELEMENT options (#PCDATA)>
<!-- files is used to define a list of file to be copied in result directory, usefull to replay the benchmark-->
<!ELEMENT files (file+)>
<!ELEMENT file (#PCDATA)>
<!ATTLIST tool name  CDATA #REQUIRED>
<!ATTLIST tool alias CDATA #IMPLIED>

Here is an example of tools.xml file, for three methods (rnaforester, migal and gardenia):

<?xml version="1.0"?>
<!DOCTYPE brasero PUBLIC "-//BRASERO//DTD Brasero benchmark tools description//EN"  "brasero.dtd">
<brasero>
<!-- RNAforester -->
  <tool name="rnaforester" alias="rnaforester" />

<!-- MiGaL -->
  <tool name="migal" alias="migal">
    <options>
      -M  --hairpin-strict --indel-once
    </options>
  </tool>
<!-- Gardenia -->
  <tool name="gardenia" alias="gardenia">
    <options>
      -S tools_files/gardenia_score.txt
    </options>
    <files>
      <file>tools_files/gardenia_score.txt</file>
    </files>
  </tool>
</brasero>

For each entry tool, the required option name identifies the name of the program for the comparison. The current version of BRASERO includes the following pairwise comparison methods: migal, rnaforester, rnastrat, gardenia, nestedalign, rnadistance, blast, treematching (see below to learn how to extend this list).

The option alias allows to specify a name for each experiment (evaluation of a set of methods on a given benchmark). This is useful to consider the same program with different options or parameters.

The element options records the parameters passed to a specific program.

The element files is a list of file copied into the result directory to keep trace of some parameters that are passed to a program via a file (scoring matrix for example).

Result of a run of the benchmarking engine

The result of a run will be located in the directory specified with the option -d (the date of the run by default). In that directory you will find one directory per method. The name of a directory is specified with the option alias.

In each directory (i.e. for each tool entry of the tools.xml file), there are:

  • a file named min if the best score for the method is the smallest one among all pairwise scores.
  • a file named max if the best score for the method is the largest one among all pairwise scores.
  • a file named options.txt recording the options and parameters used with the software, to define a method.
  • a copy of each file pointed into the entry <files> of the method
  • one directory per reference structure:
    • In each of these directories, there is one file per pairwise comparison.
    • This file is named name1_vs_name2 where name1 is the name of the reference structure (i.e. from the set R described in firstSet) and name2 is the name of the structure of T2 or F2 (described in the secondSet).
    • This file contains the alignment of the two structures in dot-parenthesis format followed by the score of the alignment and the time used by the method to perform the comparison.

Adding a new method

It is possible to extend the comparator with new method by writing two java classes.

Edit - History - Print - Recent Changes - Search
Page last modified on January 12, 2012, at 12:47 PM EST