Bench2
Version 12 - July 25th, 2009
Made by Laurent Duveau
web Site :
http://www.aldweb.com e-mail :
info@aldweb.comWHAT'S NEW IN THIS VERSION OF BENCH2 ?
Version 12 (July 25th, 2009)
Version 11 (October 17th, 2006)
- Added NaPP ("private preview release")
Version 10 (January 7th, 2006)
- Added PocketC Architect (thanks to Christopher Hill)
- Added Palm Basic
Version 9 (February 23rd, 2005)
- Added Quartus Forth (thanks to Marc Furrer)
- Updated iziBasic results in its release #5 (benchmark runs faster)
- Grouped the tools in 4 categories in the table of results
Version 8 (October 22nd, 2004)
- Added AppForge (thanks to Ingbert Grimpe)
- Changed type (from interpreted to runtime) for HotPaw Basic and SmallBasic as it appears that these 2 tools use an internal bytecode
Version 7 (September 5th, 2004)
- Updated iziBasic results in its release #2 (benchmark runs faster)
Version 6 (July 22nd, 2004)
- Changed the referent device: from Sony Clié N770C to Palm Tungsten C, so I had to run all timings again!
- As a consequence, I started to include ARMlets (Palm OS 5) versions of Bench2
- Updated many developement tools to their latest version: cbasPad5, HotPaw Basic, HSPascal, LyME, OnBoardC, picoBASIC, PLua, PP, Rexx
- Added LaFac HELP, µBAS, µPAS et µC (my own multi-mini-languages developement tool)
- Added iziBasic
Version 5 (February 17th, 2004)
Version 4 (September 18th, 2003)
- Added OrbForms Designer
- Updated PocketC in its latest version
- Updated PP in its latest version
Version 3 (September 1st, 2003)
- Added HB++, brand new Basic compiler and RAD development tool
Version 2 (December 12th, 2002)
- Complete rewrite of the Bench2 routine to better benchmark the development tools.
Thanks to all of the people who provided me with sharp analysis, bright comments and good ideas. - As a consequence, I highly updated this web page!
- Refreshed the results for NSBasic/Palm in its new version 3.0.0d
- Refreshed the results for PP in its new version 1.05a
- Added results for the cbasPad, Jump2, Rexx and Satellite Forms development tools
Version 1 (October 22nd, 2002)
WHAT IS BENCH2 ?
I developed the
Bench2 benchmark routine with three goals in mind that are ordered by level of importance:
- Provide a simple but quite rich example of a routine in different languages, for educational purposes. As a result, the comparizon of the different source codes provided on this page is very interesting.
- Find a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result. Result is more important than means...
- In order to give a playful purpose to the writing of the same routine in different languages, try to evaluate compilers & interpreters efficiency, designed to work on the Palm OS environment where performance and speed are of key importance.
Michael Winikoff already did a very impressive job on this topic on his
PalmPilot Software Development - Alternatives to C page by benchmarking the different development tools with a very simple benchmark routine (that I nickname
Bench1 and this is why my own routine is named
Bench2):
program start
start timer
x = 1000
y = 0
while (x>0) do
x = x-1
y = y+2
end do
stop timer
display execution time
program end
|
This is a very good idea and initiative, but this benchmark is fairly limited as it only tests for single decrementations and incrementations loop on integer numbers. Furthermore, an
intelligent compiler will easily find out that the x and y variables are not used afterwards so it will just skip the execution of the loop (this is the case with the newest version 8 of CodeWarrior... a Delphi compilation would give a warning message and let the programmer decide for the action to take).
So, I felt like refining this benchmark analysis and I tried to extend the benchmark to floating point and integer numbers, tests (if), loops (for), with the following benchmark routine which is a Perfect Numbers finder routine for the first 500 or 1000 integer numbers:
program start
for B=1 to 2 step 1 do
execute PerfectNum routine
end for
program end
PerfectNum routine
routine start
start timer
P = 0
for N=2 to B*500 step 1 do
M = N div 2
S = 1
for D=2 to M step 1 do
if B=1 then if (N/D)=(N div D) then S = S+D
if B=2 then if (N modulo D)=0 then S = S+D
end for
if S=N then P = P+1
end for
stop timer
if P<>3 then execution time = 0
display execution time
routine end
Note : "div" stands for "integer division"
then 3 div 2 = 1 and 3/2 = 1.5
|
The first 4 perfect numbers are: 6, 28, 496 and 8128. Since we look for perfect numbers between 2 and 500 or 1000, P should hold 3 at the end of the benchmark routine. If this were not the case, a time of 0 second will be returned, thus giving the code error signification. Furthermore, this reuse of the P variable makes sure that an
intelligent compiler will not detect that this variable is never used after it has been provided with a value.
Additional ideas were:
- Keep the benchmark program as simple as possible to allow a real comparizon between the different languages.
- Keep it as similar as possible, try to avoid optimizations like inc(P) instead of P:=P+1 in Pascal or S+=D instead of S=S+D in C. I do not know if this has an impact on performance, but I think that it is easier to compare the routine in the different language this way.
- Floating point and integer numbers management in Palm OS is a tricky subject and it brought me a lot of e-mails and comments when I released the initial Bench2 routine that was not doing the modulo test. The DragonBall processor has a native management for 16 bit integers on which some development tools rely when others have developped a 32 bit integers management. So, of course, the second ones can be a little bit slowlier in execution (but not always, for instance PocketStudio runs just as fast using the integer type or the longint type). Most tools rely on the Palm OS APIs for managing float numbers and the same remark as for the integer numbers is valid for simple float numbers or double float numbers. I finally decided to work with the default integer and floating point types provided by the different development tools for the Bench2 test (but, for example, OnBoardC runs faster with the Double type than with the Float type). This is justified by the fact that we are taking into account our second objective (a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result) with particular attention.
- The loop [ if (N modulo D)=0 then S = S+D ] (work on integer numbers - this is the add-on analysis in Bench2 version 2) runs for as much as four times, 249001 times, than the one with [ if (N/D)=(N div D) then S = S+D ] (work on float numbers - this was the only analysis made in Bench2 version 1) which runs 62001 times. This is to take into account comments and feedbacks that came to me after I had posted version 1 of Bench2. Indeed, I was nicely told and I acknoledge that benchmark on floating point number is not always significative. Therefore, Bench2's total benchmark time will be level-headed with this different number of loops execution.
BENCH2 RESULTS
All times given here are those found on my Palm Tungsten C (400 MHz XScale processor) when running
Bench2, either stand-alone PRC program (for compilers), using a runtime (for pseudo-coded languages) or within the interpreter's IDE.
Notes:- the tools running in 68k mode (PACE) are in Blue, those running in native ARM mode are in Red.
- to ease the reading of the results, I have chosen to group the tools into 4 categories:
- less than 10 secondes
- between 10 and 100 secondes
- between 100 and 1000 secondes
- more than 1000 secondes
Dev. Tool Version Language Type Time 1 Time 2 Total Time
--------------------------------------------------------------------------------------------
NaPP preview Pascal compiled 0.11 sec 0.06 sec 0.17 sec
PP 2.08q Pascal compiled 0.14 sec 0.07 sec 0.21 sec
FastBasic 1.2 Basic compiled 0.86 sec 0.09 sec 0.95 sec
CodeWarrior 8.0 C compiled 0.70 sec 0.49 sec 1.19 sec
HSPascal 2.1.0 Pascal compiled 0.82 sec 0.52 sec 1.34 sec
HB++ 1.02 Basic compiled 1.12 sec 0.63 sec 1.75 sec
PP 2.08q Pascal compiled 0.71 sec 1.06 sec 1.77 sec
PocketStudio 1.1 Pascal compiled 1.08 sec 0.89 sec 1.97 sec
OnBoardC 2.4.2 C compiled 1.65 sec 0.58 sec 2.23 sec
Jump2 2.0ß8 Java compiled 1.33 sec 1.26 sec 2.59 sec
FastBasic 1.2 Basic compiled 1.55 sec 1.65 sec 3.21 sec
Quartus Forth 1.2.1 Forth compiled 2.89 sec 1.56 sec 4.45 sec
cbasPad 5.3b0i Basic interpreted 1.07 sec 3.60 sec 4.67 sec
Falch.net / GCC 2.5 C compiled 5.34 sec 0.51 sec 5.85 sec
SuperWaba 4.21b Java runtime 3.06 sec 6.31 sec 9.37 sec
--------------------------------------------------------------------------------------------
AppForge 5.1.1 Basic runtime 12.92 sec 10.01 sec 22.93 sec
CASLpro / GCC 4.1.3 CASL compiled 12.46 sec 18.38 sec 30.84 sec
FastBasic 1.2 Basic interpreted 10.68 sec 37.53 sec 48.21 sec
Sun MIDP (*) 1.0 Java runtime 12.20 sec 42.10 sec 54.30 sec
PLua 1.0 Lua runtime ~12 sec ~50 sec ~62 sec
OrbForms 2.1.1 C runtime 16.14 sec 49.20 sec 65.34 sec
PocketC Architect 4.0.0 C runtime 16.22 sec 49.29 sec 65.51 sec
LaFac - HELP 1.1 HELP runtime 18.73 sec 63.69 sec 82.42 sec
iziBasic 5.0 Basic runtime 21.61 sec 75.80 sec 97.41 sec
--------------------------------------------------------------------------------------------
HotPaw Basic 1.4.1 Basic runtime 29.54 sec 108.50 sec 138.04 sec
SmallBASIC 0.8.2 Basic runtime 45.14 sec 133.27 sec 178.41 sec
LispMe 3.21 Scheme runtime 39.76 sec 140.60 sec 180.36 sec
CASL 4.1.3 CASL runtime 36.16 sec 147.24 sec 184.40 sec
NS Basic/Palm 3.0.0d Basic runtime 44.71 sec 155.11 sec 199.82 sec
LaFac - µBAS 1.1 Basic runtime 47.31 sec 174.29 sec 221.60 sec
LaFac - µC 1.1 C runtime 64.90 sec 245.17 sec 310.07 sec
LaFac - µPAS 1.1 Pascal runtime 64.91 sec 245.20 sec 310.11 sec
PocketC 6.0.0 C runtime 82.51 sec 265.89 sec 348.40 sec
REXX 1.00 Rexx interpreted ~89 sec ~265 sec ~354 sec
LyME 2.8 LME interpreted 124.16 sec 577.78 sec 701.94 sec
Palm Tcl 0.4 Tcl interpreted 203.64 sec 607.21 sec 810.85 sec
--------------------------------------------------------------------------------------------
Palm Basic (**) 1.02 Basic interpreted ~280 sec ~1152 sec ~1432 sec
picoBASIC (*) 1.01 Basic interpreted ~4590 sec ~15931 sec ~20521 sec
--------------------------------------------------------------------------------------------
Satellite Forms 5.0.0 Basic runtime [does not run in ARM devices]
cbasPad 0.97b4 Basic interpreted [does not run in ARM devices]
|
And the same results shown in a graphical way :
Dev. Tool Bar Chart (shorter is better)
--------------------------------------------------------------------------------------------
NaPP 
PP
FastBasic
CodeWarrior 
HSPascal
HB++
PP
PocketStudio
OnBoardC
Jump2
FastBasic
Quartus Forth
cbasPad
Falch.net / GCC
SuperWaba
--------------------------------------------------------------------------------------------
AppForge
CASLpro / GCC
Sun MIDP (*)
PLua
OrbForms
PocketC Archit.
LaFac - HELP
iziBasic
--------------------------------------------------------------------------------------------
HotPaw Basic
SmallBASIC
LispMe
CASL
NS Basic/Palm
LaFac - µBAS
LaFac - µC
LaFac - µPAS
PocketC
REXX
LyME
Palm Tcl
--------------------------------------------------------------------------------------------
Palm Basic (**)
picoBASIC (*)
|
(*) :
picoBASIC and Sun MIDP only work with integer numbers and they do not have a float numbers management. This is why loop [
if B=1 then if (N/D)=(N div D) then S = S+D ]
cannot be run as is. Replacing it with [
if B=1 then if (N-(N/D)*D)=0 then S = S+D ]
gives a a way of not removing these two development tools from the Bench2 comparizon even if they cannot then be compared exactly with the others.(**) :
Palm Basic does not have a modulo operator. This is why loop [
if B=2 then if (N modulo D)=0 then S = S+D ]
cannot be run as is. Replacing it with [
if B=2 then if (N-(N div D)*D)=0 then S = S+D ]
provides with an equivalent formula.CONCLUSIONS
As you could see when reading the previous charts, the results are different depending on what you consider. None of the 3 analysis (overall
Bench2 with 1st and 2nd loops, 1st loop only or 2nd loop only) gives the same podium.
We easily get to see a huge gap between the very first timing (PP in ARM mode) and the very last one (picoBASIC Integer in 68k mode) : 0.21 seconde to be compared to 20521 secondes. This is a gap of more than 5 hours 42 minutes when the first one runs in less that a quarter of a second!!!
If we take into account the second timing (CodeWarrior in 68k mode at 1.19 secondes) and the last "
reasonable" one (Palm Tcl in 68k mode at 810.85 secondes), thus considering that the extremes results are not significative, we still find ourselves with a huge gap of almost 13 minutes and a half to be compared with 1.19 secondes !!!
Without any surprise, we see that compiled languages are the fastest ones, ahead of the runtime ones, themselves performing better than the interpreted ones.
I am personally quite happy to see that my own HELP, µBAS, µPAS and µC languages which are included in
LaFac are in the average (
self-satisfaction !).
I am even very happy to see that
iziBasic is the best among the well-known Basic tools, but with less GUI functionalities.
So, where is the truth? By the way, is there a truth in this exercise?
Let's go back to our initial objectives and remember this one:
find a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result. Result is more important than means....
Taking into account this objective, I let you establish by yourself your very personal podium!
As a conclusion, let's not forget, as George Henne from NSBasic Corporation reminded me in a very right analysis, that
a benchmark like Bench2 needs to be taken in context of a total development project in which the developer, either professional or amateur, should answer to the following questions (that I would gently call the
" 7 HOW questions ") for choosing a development tool:
- How much of the job is CPU bound?
- How long will it take to develop my app?
- How good is the look and feel of the resulting project?
- How compatible will it be with various Palm OS devices?
- How is the support?
- How easy is it to deploy?
- How long has the tools vendor been around? Is he likely to be there for the full life cycle of my product?
If you answer to this questions, you will find your truth. And they are many truths since we all are so different people.
Let's also realize that we are very lucky to have such a choice of developement tools for our Palm devices!
BENCH2 SOURCE CODES AND RESULTS FOR ALL SINGLE DEVELOPMENT TOOLS
Only the final results have been put in this page. For the detail of the source codes, of the links to the development tools and the single results for all these tools, please refer to the
Bench 2 source codes page.
By the way, it is a very instructive exercise for the amateur developer as well as for the professional one, to read the source codes in the different languages. Therefore, I highly invite you to read them.
If you would like to help me extend this list of already benchmarked development tools and are working on one which is not listed here yet or in a newer version, I would be more than happy if you get in touch with me.
I. aldweb
visitors online
VI. Palm Knowledge - 3. Development Tools Benchmark
Print the article
Reactions to this article
![[de]](img/de.gif)
![[es]](img/es.gif)
![[it]](img/it.gif)
![[pt]](img/pt.gif)
Top 
© 2004-2013 
