Recent Changes - Search:

Benchmarking protocol

Introduction

The purpose of a BRASERO benchmark is to evaluate the ability of methods for RNA secondary structure pairwise comparison to properly classify RNA queries with respect to a given RNA family (the benchamrk). It is inspired from the practical problem of identifying into a large databases o RNA secondary structures, candidate RNAs which are close, in terms of secondary structure, to some query RNA secondary structure.

Protocol

In the following, we consider an RNA family named B.

Structure of a benchmark

A benchmark is composed using several sets of RNA sequences and secondary structures:

  • A set R of RNA secondary structures that are known to belong to B.
  • A set T of RNA sequence that are known to belong to F and are different from R.
  • A set T2 of RNA secondary structures obtain by folding the sequences of T (for example using mfold, rnashapes or rnasubopt). For each sequence of T, we include in T2 the optimal predicted secondary structure but also some suboptimal ones.
  • A set F of RNA sequences randomly generated from a noise source, whose lengths have the same distribution than the lengths of the RNAs belonging to R.
  • A set F2 of RNA secondary structures obtained from F in the same way that T2 is obtained from T (same programs/parameters).

A given benchmark consists in the three disjoint sets of RNA secondary structures R, T2 and F2, with the important property that several structures of T2 (resp. R2) can have been obtained from a same sequence from T (resp. R).

Storing a benchmark

In order to be used for evaluating pairwise comparison methods, a benchmark needs to be stored in a set of files as described below.

The structures of the set R are given in the dot-parenthesis format, where each structure is encoded on three lines of text:

  • > name : the name of the RNA
  • ACGU : the sequence of the RNA
  • .() : the secondary structure of the RNA

For the structures of the set T2, the format is almost identical. The only difference is that the name is encoded as follow:

  • > TRUE name % folding info

where

  • name: is the name of the original sequence in T (for example an RFAM access number)
  • folding info: information on how this structure had been generated.

For example, for the genome AF303111 (GenBank access number), positions 1134 to 1201 encode a tRNA, and several folding of these RNA can be encoded in a benchmark file as follows:

> TRUE AF303111: 1134 => 1201 (W) % mfold  P=10 W=3  000
GUUUAUGUAGCUUAAUAGUAAAGCAAGGCACUGAAAAUGCCUAGACGAGUUAUAUAAUUCCAUAAAC
(((((((..((((.......)))).(((((.......)))))..................)))))))
> TRUE AF303111: 1134 => 1201 (W) % mfold  P=10 W=3  001
GUUUAUGUAGCUUAAUAGUAAAGCAAGGCACUGAAAAUGCCUAGACGAGUUAUAUAAUUCCAUAAAC
..(((((((((((....(.....).(((((.......)))))....)))))))))))..........
> TRUE AF303111: 1134 => 1201 (W) % mfold  P=10 W=3  002
GUUUAUGUAGCUUAAUAGUAAAGCAAGGCACUGAAAAUGCCUAGACGAGUUAUAUAAUUCCAUAAAC
(((((((..((((.......)))).(((((.......)))))....(((((....))))))))))))
> TRUE AF303111: 1134 => 1201 (W) % shape  003
gtttatgtagcttaatagtaaagcaaggcactgaaaatgcctagacgagttatataattccataaac
..(((((((((((....((...)).(((((.......)))))....)))))))))))..........
> TRUE AF303111: 1134 => 1201 (W) % shape  004
gtttatgtagcttaatagtaaagcaaggcactgaaaatgcctagacgagttatataattccataaac
(((((((..((((.......)))).(((((.......)))))....(((((....))))))))))))
> TRUE AF303111: 1134 => 1201 (W) % shape  005
gtttatgtagcttaatagtaaagcaaggcactgaaaatgcctagacgagttatataattccataaac
..(((((((((((....((......(((((.......)))))..)))))))))))))..........

The set F2 is encoded in the same way as T2 except that the word TRUE is missing.

Finally, in the benchmark, each set R, T2 and F2 can be stored into a single file or a group of files where each file contains one (or several) secondary structure(s).

Evaluation of pairwise comparison methods

By method or tool we mean a software for the pairwise comparison of RNA secondary structures, together with a set of options and parameters (hence a same software can define several methods, if used with different options or parameters). From now, our goal is to evaluate several methods on a given benchmark. We assume that the result of a comparison of two secondary structures with a method m is a numerical quantity, named the score from now.

First, for each considered method m, we compare each structure of R with each structure of T2 and F2 using m. Then for each sequence s of T and F, we keep the best score obtained over all the comparison of the secondary structures of s included in T2 (or F2) and all the elements of R. We call this score the score of s.

Finally, sequences of T and F are sorted according to the score obtained.

This process is illustrated in the figure below, where A, B, C are the sequences of T, D, E, F, G, H the sequences of F, hte blus RNA secondary structures represent T2 and F2, and the blus secondary structures represent R.

Analysis of the results

A natural way to analyse the results of the process described above is to use them to plot a ROC curve showing the ability to separate the true predictions (sequences of T known to belong to the family B) and the false predictions (sequences of F, that do not belong to B). This curve show the false positive rate vs the true positive rate.

ROC Curve
Edit - History - Print - Recent Changes - Search
Page last modified on December 09, 2010, at 02:49 PM EST