Comparing SBML capable Simulators

Comparing Simulation Results among SBML capable simulators has always been of interest to us. Not knowing the correct result of an SBML simulation, we figured that it would be best to compare what sort of results common simulators would give on a set of curated SBML files.

So we compared the simulators:

• COPASI Version 4.4.26 (development)
• JSim Version 1.6.79
• Jarnac and RoadRunner as available with SBW 2.7.6
• Oscill8 Version 2.0.10 and
• SBML ODE Solver current CVS (as of 10/21/2007)

The models each simulator had to go through were models from the BioModels Database. We used the ninth release (25th September 2007). Each simulator had to simulate the model for 10 seconds, delivering 1000 simulation points. In this comparison we only compared the concentration data returned for floating species.

Here the Result of the Comparison. You will find each model listed, along with concentration plots for each simulator that returned results for the model, and difference plots comparing the simulation results between the simulators, along with a link to the original model at the BioModels Database. If you would like your simulator to appear in the comparison, please send me your simulation results (generated against BioModels Database as described above) in CSV form.

We have also created a New Comparison page, a online application, allowing to specify exactly what to see. The loading times are a bit higher, but the overal experience much better.

As we now have more simulation results coming in, I updated the statistic overview. The results are a bit changed from the ones below:

As no authoritative result set exists, the best we could do for now is to measure how well the simulation results agree on simulators that returned results for a given model.

• This graph shows, that for a large percentage of the model the simulators agree with each other
• An interesting area is the ones with an agreement below 50%. Simulation results in this area are due to some simulators do not support certain features. For example SBML events, are not supported by many simulators, and so simulation results returned for those models fall into certain groups.
• Rather interesting are also the results for biomodel #104 here it would seem that all simulators disagree. Update Thanks to new simulation results submitted by Henning Schmidt this mystery is now solved. Read more
• The models currently listed under 0%, actually cannot be compared right now. In the current version of these tests, we are only considering Floating Species. However these models do not contain any floating species. Until a safe way to determine state-variables in those models is found, they will be left out of the comparison.

We are continuing to evaluate the obtained data to yield more statistics and information.

In order to compare the simulation results, we wrapped the commandline version of COPASI and SBML odeSolver as SBW Simulator. The initial version was based on a C# port of VCells COPASI integration, thanks to Anuradha Lakshminarayana for sending it to me. Please contact me in order to obtain versions of these wrappers.

Oscill8, Jarnac and roadRunner were already SBW enabled and can be obtained from the links above.

For the comparison we wrote commandline tools as well as a viewer application. The procedure works like outlined in the diagram below.

For each model of the BioModels Database

• we either use our wrappers to generate simulation results
• or we receive simulation result from a 3rd party
• these results are then pre-processed by comparing them with all other available results for that model
• a viewer application has been written, that allows to display the results and differences, at any level of detail. The viewer also generates the comparison report.

Jun. 02 2008 Update : This page has been updated by adding a link to the new online viewer application, which provides much better feedback. Currently we are preparing to update the simulation results for the tenth release of the BioModels Database, however we are still waiting for 3rd parties to submit simulation results.

— Frank Bergmann 2008/06/02 18:45

---

Feb. 11 2008 Update : This page has been updated to add the results submitted from the VCell team.

— Frank Bergmann 2008/02/12 10:35

---

Feb. 4 2008 Update : This page has been updated to add the results submitted from BioUML and SBToolBox2. Also the results for roadRunner and COPASI were updated to reflect recent program changes.

— Frank Bergmann 2008/02/04 23:59

---

Dec. 4 2007 Update : This page has been updated to display some results obtained from the comparison.

— Frank Bergmann 2007/12/04 19:47

---

Nov. 15 2007 Update : The generated results have been updated as the results from MathSBML were included. Thanks to Bruce Shapiro for contributing the results.

— Frank Bergmann 2007/11/15 14:01

---

Nov. 11 2007 Update : The generated results have been updated to give access to the CSV files that were used to generate the plots. To access the CSV files select a model and then click on the simulator whose results your interested in under the section: ”Simulators delivering results:”.

— Frank Bergmann 2007/11/11 19:16

---

This work has been inspired on similar work done by the VCell Team. See their comparison page.

Of Interest will also be the Simulator feature comparison.

— Frank Bergmann 2007/10/25 11:28

---

The BioUML group have started a similar comparison effort on their site under Preliminary Results.

— Frank Bergmann 2008/02/05 16:39