aldweb

Close I. aldweb

Close II. PC Freeware

Close III. Palm Freeware

Close IV. Palm Shareware

Close V. iziBasic Palm

Close VI. Palm Knowledge

Close VII. Pocket Computer

Close VIII. miniPortail

Search




Newsletter
To receive news about this website, consider subscribing to our Newsletter.
Subscribe
Unsubscribe
298 Subscribers
Webmaster - Infos
Visits

   visitors

   visitors online

article.gifVI. Palm Knowledge - 3. Development Tools Benchmark

Bench2


Version 12 - July 25th, 2009
Made by Laurent Duveau
web Site : http://www.aldweb.com
e-mail : info@aldweb.com


WHAT'S NEW IN THIS VERSION OF BENCH2 ?

Version 12 (July 25th, 2009)
  • Added FastBasic
Version 11 (October 17th, 2006)
  • Added NaPP ("private preview release")
Version 10 (January 7th, 2006)
  • Added PocketC Architect (thanks to Christopher Hill)
  • Added Palm Basic
Version 9 (February 23rd, 2005)
  • Added Quartus Forth (thanks to Marc Furrer)
  • Updated iziBasic results in its release #5 (benchmark runs faster)
  • Grouped the tools in 4 categories in the table of results
Version 8 (October 22nd, 2004)
  • Added AppForge (thanks to Ingbert Grimpe)
  • Changed type (from interpreted to runtime) for HotPaw Basic and SmallBasic as it appears that these 2 tools use an internal bytecode
Version 7 (September 5th, 2004)
  • Updated iziBasic results in its release #2 (benchmark runs faster)
Version 6 (July 22nd, 2004)
  • Changed the referent device: from Sony Clié N770C to Palm Tungsten C, so I had to run all timings again!
  • As a consequence, I started to include ARMlets (Palm OS 5) versions of Bench2
  • Updated many developement tools to their latest version: cbasPad5, HotPaw Basic, HSPascal, LyME, OnBoardC, picoBASIC, PLua, PP, Rexx
  • Added LaFac HELP, µBAS, µPAS et µC (my own multi-mini-languages developement tool)
  • Added iziBasic
Version 5 (February 17th, 2004)
  • Added LispMe
  • Added CASL
Version 4 (September 18th, 2003)
  • Added OrbForms Designer
  • Updated PocketC in its latest version
  • Updated PP in its latest version
Version 3 (September 1st, 2003)
  • Added HB++, brand new Basic compiler and RAD development tool
Version 2 (December 12th, 2002)
  • Complete rewrite of the Bench2 routine to better benchmark the development tools.
    Thanks to all of the people who provided me with sharp analysis, bright comments and good ideas.
  • As a consequence, I highly updated this web page!
  • Refreshed the results for NSBasic/Palm in its new version 3.0.0d
  • Refreshed the results for PP in its new version 1.05a
  • Added results for the cbasPad, Jump2, Rexx and Satellite Forms development tools
Version 1 (October 22nd, 2002)
  • Initial release


WHAT IS BENCH2 ?

I developed the Bench2 benchmark routine with three goals in mind that are ordered by level of importance:
  1. Provide a simple but quite rich example of a routine in different languages, for educational purposes. As a result, the comparizon of the different source codes provided on this page is very interesting.
  2. Find a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result. Result is more important than means...
  3. In order to give a playful purpose to the writing of the same routine in different languages, try to evaluate compilers & interpreters efficiency, designed to work on the Palm OS environment where performance and speed are of key importance.
Michael Winikoff already did a very impressive job on this topic on his PalmPilot Software Development - Alternatives to C page by benchmarking the different development tools with a very simple benchmark routine (that I nickname Bench1 and this is why my own routine is named Bench2):


program start
start timer
x = 1000
y = 0
while (x>0) do
x = x-1
y = y+2
end do
stop timer
display execution time
program end

This is a very good idea and initiative, but this benchmark is fairly limited as it only tests for single decrementations and incrementations loop on integer numbers. Furthermore, an intelligent compiler will easily find out that the x and y variables are not used afterwards so it will just skip the execution of the loop (this is the case with the newest version 8 of CodeWarrior... a Delphi compilation would give a warning message and let the programmer decide for the action to take).

So, I felt like refining this benchmark analysis and I tried to extend the benchmark to floating point and integer numbers, tests (if), loops (for), with the following benchmark routine which is a Perfect Numbers finder routine for the first 500 or 1000 integer numbers:


program start
for B=1 to 2 step 1 do
execute PerfectNum routine
end for
program end

PerfectNum routine
routine start
start timer
P = 0
for N=2 to B*500 step 1 do
M = N div 2
S = 1
for D=2 to M step 1 do
if B=1 then if (N/D)=(N div D) then S = S+D
if B=2 then if (N modulo D)=0 then S = S+D
end for
if S=N then P = P+1
end for
stop timer
if P<>3 then execution time = 0
display execution time
routine end

Note : "div" stands for "integer division"
then 3 div 2 = 1 and 3/2 = 1.5


The first 4 perfect numbers are: 6, 28, 496 and 8128. Since we look for perfect numbers between 2 and 500 or 1000, P should hold 3 at the end of the benchmark routine. If this were not the case, a time of 0 second will be returned, thus giving the code error signification. Furthermore, this reuse of the P variable makes sure that an intelligent compiler will not detect that this variable is never used after it has been provided with a value.

Additional ideas were:
  • Keep the benchmark program as simple as possible to allow a real comparizon between the different languages.
  • Keep it as similar as possible, try to avoid optimizations like inc(P) instead of P:=P+1 in Pascal or S+=D instead of S=S+D in C. I do not know if this has an impact on performance, but I think that it is easier to compare the routine in the different language this way.
  • Floating point and integer numbers management in Palm OS is a tricky subject and it brought me a lot of e-mails and comments when I released the initial Bench2 routine that was not doing the modulo test. The DragonBall processor has a native management for 16 bit integers on which some development tools rely when others have developped a 32 bit integers management. So, of course, the second ones can be a little bit slowlier in execution (but not always, for instance PocketStudio runs just as fast using the integer type or the longint type). Most tools rely on the Palm OS APIs for managing float numbers and the same remark as for the integer numbers is valid for simple float numbers or double float numbers. I finally decided to work with the default integer and floating point types provided by the different development tools for the Bench2 test (but, for example, OnBoardC runs faster with the Double type than with the Float type). This is justified by the fact that we are taking into account our second objective (a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result) with particular attention.
  • The loop [ if (N modulo D)=0 then S = S+D ] (work on integer numbers - this is the add-on analysis in Bench2 version 2) runs for as much as four times, 249001 times, than the one with [ if (N/D)=(N div D) then S = S+D ] (work on float numbers - this was the only analysis made in Bench2 version 1) which runs 62001 times. This is to take into account comments and feedbacks that came to me after I had posted version 1 of Bench2. Indeed, I was nicely told and I acknoledge that benchmark on floating point number is not always significative. Therefore, Bench2's total benchmark time will be level-headed with this different number of loops execution.


BENCH2 RESULTS

All times given here are those found on my Palm Tungsten C (400 MHz XScale processor) when running Bench2, either stand-alone PRC program (for compilers), using a runtime (for pseudo-coded languages) or within the interpreter's IDE.

Notes:
  • the tools running in 68k mode (PACE) are in Blue, those running in native ARM mode are in Red.
  • to ease the reading of the results, I have chosen to group the tools into 4 categories:
    1. less than 10 secondes
    2. between 10 and 100 secondes
    3. between 100 and 1000 secondes
    4. more than 1000 secondes



Dev. Tool Version Language Type Time 1 Time 2 Total Time
--------------------------------------------------------------------------------------------
NaPP preview Pascal compiled 0.11 sec 0.06 sec 0.17 sec
PP 2.08q Pascal compiled 0.14 sec 0.07 sec 0.21 sec
FastBasic 1.2 Basic compiled 0.86 sec 0.09 sec 0.95 sec
CodeWarrior 8.0 C compiled 0.70 sec 0.49 sec 1.19 sec
HSPascal 2.1.0 Pascal compiled 0.82 sec 0.52 sec 1.34 sec
HB++ 1.02 Basic compiled 1.12 sec 0.63 sec 1.75 sec
PP 2.08q Pascal compiled 0.71 sec 1.06 sec 1.77 sec
PocketStudio 1.1 Pascal compiled 1.08 sec 0.89 sec 1.97 sec
OnBoardC 2.4.2 C compiled 1.65 sec 0.58 sec 2.23 sec
Jump2 2.0ß8 Java compiled 1.33 sec 1.26 sec 2.59 sec
FastBasic 1.2 Basic compiled 1.55 sec 1.65 sec 3.21 sec
Quartus Forth 1.2.1 Forth compiled 2.89 sec 1.56 sec 4.45 sec
cbasPad 5.3b0i Basic interpreted 1.07 sec 3.60 sec 4.67 sec
Falch.net / GCC 2.5 C compiled 5.34 sec 0.51 sec 5.85 sec
SuperWaba 4.21b Java runtime 3.06 sec 6.31 sec 9.37 sec
--------------------------------------------------------------------------------------------
AppForge 5.1.1 Basic runtime 12.92 sec 10.01 sec 22.93 sec
CASLpro / GCC 4.1.3 CASL compiled 12.46 sec 18.38 sec 30.84 sec
FastBasic 1.2 Basic interpreted 10.68 sec 37.53 sec 48.21 sec
Sun MIDP (*) 1.0 Java runtime 12.20 sec 42.10 sec 54.30 sec
PLua 1.0 Lua runtime ~12 sec ~50 sec ~62 sec
OrbForms 2.1.1 C runtime 16.14 sec 49.20 sec 65.34 sec
PocketC Architect 4.0.0 C runtime 16.22 sec 49.29 sec 65.51 sec
LaFac - HELP 1.1 HELP runtime 18.73 sec 63.69 sec 82.42 sec
iziBasic 5.0 Basic runtime 21.61 sec 75.80 sec 97.41 sec
--------------------------------------------------------------------------------------------
HotPaw Basic 1.4.1 Basic runtime 29.54 sec 108.50 sec 138.04 sec
SmallBASIC 0.8.2 Basic runtime 45.14 sec 133.27 sec 178.41 sec
LispMe 3.21 Scheme runtime 39.76 sec 140.60 sec 180.36 sec
CASL 4.1.3 CASL runtime 36.16 sec 147.24 sec 184.40 sec
NS Basic/Palm 3.0.0d Basic runtime 44.71 sec 155.11 sec 199.82 sec
LaFac - µBAS 1.1 Basic runtime 47.31 sec 174.29 sec 221.60 sec
LaFac - µC 1.1 C runtime 64.90 sec 245.17 sec 310.07 sec
LaFac - µPAS 1.1 Pascal runtime 64.91 sec 245.20 sec 310.11 sec
PocketC 6.0.0 C runtime 82.51 sec 265.89 sec 348.40 sec
REXX 1.00 Rexx interpreted ~89 sec ~265 sec ~354 sec
LyME 2.8 LME interpreted 124.16 sec 577.78 sec 701.94 sec
Palm Tcl 0.4 Tcl interpreted 203.64 sec 607.21 sec 810.85 sec
--------------------------------------------------------------------------------------------
Palm Basic (**) 1.02 Basic interpreted ~280 sec ~1152 sec ~1432 sec
picoBASIC (*) 1.01 Basic interpreted ~4590 sec ~15931 sec ~20521 sec
--------------------------------------------------------------------------------------------
Satellite Forms 5.0.0 Basic runtime [does not run in ARM devices]
cbasPad 0.97b4 Basic interpreted [does not run in ARM devices]

And the same results shown in a graphical way :


Dev. Tool Bar Chart (shorter is better)
--------------------------------------------------------------------------------------------
NaPP
PP
FastBasic
CodeWarrior
HSPascal
HB++
PP
PocketStudio
OnBoardC
Jump2
FastBasic
Quartus Forth
cbasPad
Falch.net / GCC
SuperWaba
--------------------------------------------------------------------------------------------
AppForge
CASLpro / GCC
Sun MIDP (*)
PLua
OrbForms
PocketC Archit.
LaFac - HELP
iziBasic
--------------------------------------------------------------------------------------------
HotPaw Basic
SmallBASIC
LispMe
CASL
NS Basic/Palm
LaFac - µBAS
LaFac - µC
LaFac - µPAS
PocketC
REXX
LyME
Palm Tcl
--------------------------------------------------------------------------------------------
Palm Basic (**)
picoBASIC (*)

(*) : picoBASIC and Sun MIDP only work with integer numbers and they do not have a float numbers management. This is why loop [ if B=1 then if (N/D)=(N div D) then S = S+D ] cannot be run as is. Replacing it with [ if B=1 then if (N-(N/D)*D)=0 then S = S+D ] gives a a way of not removing these two development tools from the Bench2 comparizon even if they cannot then be compared exactly with the others.
(**) : Palm Basic does not have a modulo operator. This is why loop [ if B=2 then if (N modulo D)=0 then S = S+D ] cannot be run as is. Replacing it with [ if B=2 then if (N-(N div D)*D)=0 then S = S+D ] provides with an equivalent formula.


CONCLUSIONS


As you could see when reading the previous charts, the results are different depending on what you consider. None of the 3 analysis (overall Bench2 with 1st and 2nd loops, 1st loop only or 2nd loop only) gives the same podium.

We easily get to see a huge gap between the very first timing (PP in ARM mode) and the very last one (picoBASIC Integer in 68k mode) : 0.21 seconde to be compared to 20521 secondes. This is a gap of more than 5 hours 42 minutes when the first one runs in less that a quarter of a second!!!
If we take into account the second timing (CodeWarrior in 68k mode at 1.19 secondes) and the last "reasonable" one (Palm Tcl in 68k mode at 810.85 secondes), thus considering that the extremes results are not significative, we still find ourselves with a huge gap of almost 13 minutes and a half to be compared with 1.19 secondes !!!

Without any surprise, we see that compiled languages are the fastest ones, ahead of the runtime ones, themselves performing better than the interpreted ones.

I am personally quite happy to see that my own HELP, µBAS, µPAS and µC languages which are included in LaFac are in the average (self-satisfaction !).
I am even very happy to see that iziBasic is the best among the well-known Basic tools, but with less GUI functionalities.

So, where is the truth? By the way, is there a truth in this exercise?

Let's go back to our initial objectives and remember this one: find a routine that an average amateur programmer could possibly write, not especially optimized but giving the same right result. Result is more important than means....

Taking into account this objective, I let you establish by yourself your very personal podium!

As a conclusion, let's not forget, as George Henne from NSBasic Corporation reminded me in a very right analysis, that a benchmark like Bench2 needs to be taken in context of a total development project in which the developer, either professional or amateur, should answer to the following questions (that I would gently call the " 7 HOW questions ") for choosing a development tool:
  • How much of the job is CPU bound?
  • How long will it take to develop my app?
  • How good is the look and feel of the resulting project?
  • How compatible will it be with various Palm OS devices?
  • How is the support?
  • How easy is it to deploy?
  • How long has the tools vendor been around? Is he likely to be there for the full life cycle of my product?

If you answer to this questions, you will find your truth. And they are many truths since we all are so different people.
Let's also realize that we are very lucky to have such a choice of developement tools for our Palm devices!


BENCH2 SOURCE CODES AND RESULTS FOR ALL SINGLE DEVELOPMENT TOOLS


Only the final results have been put in this page. For the detail of the source codes, of the links to the development tools and the single results for all these tools, please refer to the Bench 2 source codes page.

By the way, it is a very instructive exercise for the amateur developer as well as for the professional one, to read the source codes in the different languages. Therefore, I highly invite you to read them.

If you would like to help me extend this list of already benchmarked development tools and are working on one which is not listed here yet or in a newer version, I would be more than happy if you get in touch with me.

Creation date : 01/01/2003 @ 00:00
Last update : 30/07/2009 @ 21:05
Category : VI. Palm Knowledge


Print the article Print the article

 
react.gifReactions to this article


Reaction #23 

by Emma 06/11/2006 @ 04:30

Coucou !

Pour celles et ceux qui comme moi ce demandais ce que c'était un "benchmark" je vous invite à aller voir la définition sur wikipédia... tout d'un coup tout s'éclair !

C'est par ici : http://fr.wikipedia.org/wiki/Benchmark

Bises ! Emma

Reaction #22 

by Pacman 08/05/2006 @ 18:55

Il serait intéressant d'ajouter Websphere/J9 qui semble être la plateforme poussée par PalmSource en ce moment.

Enfin à titre de comparaison technique surtout, car c'est quand même assez différent du reste du lot...

Reaction #21 

by Miellaby 09/03/2005 @ 10:36

:) oui d'ailleurs je crois que je vais m'arrêter. Merci d'avoir répondu à chaque fois.

Reaction #20 

by aldweb 08/03/2005 @ 20:34

Et bien, cher Miellaby, le moins que l'on puisse dire est que cet article te passionne ou, tout du moins, te fait réagir !

Je suis ennuyé et même vexé que tu ais vu autant de "sous-entendus" dans ces questions qui m'ont été suggérées par un développeur renommé et que j'ai trouvé intéressant de reprendre ici pour bien aider les gens à ne pas faire des conclusions hatives à la vue des seuls résultats exposés dans cette page.

Je n'ai pour ma part jamais cherché quoi que ce soit d'autre dans ce petit exercice de style qu'est Bench2 que ce que je mets dans le préambule.
Ce qu'on pourrait écrire en d'autres termes :
1. surtout découvrir des syntaxes de langages que je connaissais peu
2. et aussi recenser nombre d'outils de développement "exotiques", peu connus et néanmoins fort intéressants

Ma conclusion dit surtout "(...) je vous laisse établir par vous même votre podium personnel !" ou encore "(...) vous trouverez votre vérité. Et il y a bien plusieurs vérités (...)", ce qui laisse bien entendre, et rien sous-entendre dans mon esprit, qu'il y a bien des choix possibles laissés au libre arbitre de chacun.

Ceci étant, je suis tout disposé à collaborer avec toi pour monter un "Bench3" qui propose d'autres vues.

@+
aldweb

Reaction #19 

by Miellaby 08/03/2005 @ 14:16

Pour choisir un langage, je vois les questions suivantes :

"Quelle proportion du travail est liée à la performance du processeur ?"

Sous-entendu : si la vitesse d'exécution est primordial, alors il faut du compilé, et peut-être même de l'assembleur.

Mon opinion : c'est faux pour un programmeur moyen et une application qui manipulent des données complexes (chaîne de caractères, enregistrements structurés) tout en proposant de nombreuses fonctionnalités : classements, recherche, indexation... Dance ce cas : pascal/C/Assembleur = programme lent. En {Langage interprété avancé} = {programme simple et rapide}.

"Combien de temps va me prendre le développement de mon application ?"

Sous-entendu : s'il faut faire vite, alors choisir basic.

Mon opinion : Non, on peut programmer rapidement dans tout langage du moment où on choisit le bon pour le logiciel à concevoir. On peut perdre du temps en Basic à programmer des choses aussi simples q'une pile ou une recherche de données dans des tableaux.

"Quel est le rendu visuel du projet ?
Quelle sera sa compatibilité avec les différentes machines tournant sous Palm OS ? Quelle est la qualité du support de l'éditeur de l'outil de développement ? Quelle est la stabilité et la solidité de l'éditeur ? Quelle est la probabilité qu'il réponde présent pendant toute la durée de vie de mon produit ?"

Ici beaucoup de questions équivalentes, qui sous-entendent que si on veut du pro, il faut un langage cher, pas du freeware. Mais depuis l'émergence d'un troisième acteur: l'open-source, tout cela n'est plus tout à fait vrai.

"Quelle est la facilité de déploiement ?"

Sous-entendu : Source + run-time/interpréteur moins bien que binaire exécutable. Là aussi, ça se discute. Tout dépend de l'installateur.

Mais pour finir, il manque la question que je ne vois nulle part : "Qu'est ce que doit faire votre programme?".

Si c'est un programme peu d'intelligence: BASIC.

S'il s'agit de faire de la "computation" sur des données simples (entier, réel...) : C, Pascal, procédural compilé.

S'il s'agit de faire des traitements avancées sur des données structurées : Lua, Tcl, Python, Lisp, etc...
StartPrevious [ 1 2 3 4 5 6 ] NextEnd
 
Special !



Mobile version of
the aldweb Site

m.aldweb.com


Neither French, nor English?

Try one of these flags then:
[de] [es] [it] [pt]
Weather / Spam / www

Lyon Weather


aldweb against spam


Family's web sites

Friends News
Poll
What do you think of the new design of the aldweb Site?
 
Great !
Beautiful
Average
So ugly...
Results
^ Top ^