Well….several months has passed, but I would like to show you how to do it because I think it may be of help for real systems   So…the idea is the following: whenever there is an error, you can get the context from it, and such context is what is usually written down into a log file (in Pharo this is PharoDebug.log). I will show you two things: 1) how to serialize a debugger in one image and materialize it another one and; 2) how to write down the context into a Fuel file when there is an error so that you can materialize it later in another image.

Installing Fuel

The first step is, of course, install Fuel. The latest stable release is 1.7 but to have better results with this example, I would recommend 1.8. Fuel 1.8 is not released yet it is because we plan to write some stuff in the website. The code is almost finish, so you should load Fuel 1.8 beta1. In my case I am using a normal Pharo 1.3 image:

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel';
load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.8-beta1') load.

Once you have finished loading Fuel, save the image. Let’s call it Fuel.image.

Serializing and materializing a debugger

Now its time to do something hacky in the image so that to open a debugger. Or you can just take a piece of code and debug it. In my example, I opened a workspace and wrote the following:

| a |
a := 'Hello Smalltalk hackers. The universal answer is '.
a := a , '42!'.
Transcript show: a.

Then I select the whole code, right click -> “debug it”. Then I do one “step over” and I stop there before the concatenation with ’42!’.

I am sure there could be better ways, but the simpler way I found to get the debugger instance for this example, is to do a Debugger allInstances first   so… be sure not to have another debugger opened hahaha.  Now…let’s serialize the debugger:

Smalltalk garbageCollect.
FLSerializer
serialize: Debugger allInstances first
toFileNamed: 'debugger.fuel'.

After that, you should have a ‘debugger.fuel’ created in the same directory where the image is. Now close your image (without saving it) and re open it. If everything is fine, we should be able to materialize our debugger and continue debugging. So, let’s try it:

| newDebugger |
newDebugger := FLMaterializer materializeFromFileNamed: 'debugger.fuel'.
newDebugger openFullMorphicLabel: 'Materialized debugger ;)'.

So????  Did it work??  are you as happy as me when I first saw it?   if you check this new opened debugger, you will see its state is correct. For example, the instVar ‘a’ has the correct state. You can now open a Transcript and continue with the debugger as if were the original one.

Of course that even if this simple example works, there are a lot of problems. But I will explain them at the end of the post.

Serializing and materializing errors

In the previous example we have serialized the debugger manually. But imagine the following: you have a production application running. There is an error, and PharoDebug.log is written with all the stack. The user/client send you by email the .log and you open your favorite text editor to try to understand what happened. Now imagine the following: you have a production application running. There is an error, and a PharoDebug.fuel is written with all the stack. The user/client send you by email the file and you open an image, and then materialize and open a debugger. How does it sound? magical?

For this example, we will just change the place where Pharo writes PharoDebug.log when there is an error. That method is #logError:inContext:. What we will do is to add just 2 lines at the beginning to serialize the context:

logError: errMsg inContext: aContext

" we should think about integrating a toothpick here someday"
FLSerializer
serialize: aContext
toFileNamed: 'PharoDebug.fuel'.

self logDuring: [:logger |
logger
nextPutAll: 'THERE_BE_DRAGONS_HERE'; cr;
nextPutAll: errMsg; cr.

aContext errorReportOn: logger.

"wks 9-09 - write some type of separator"
logger nextPutAll: (String new: 60 withAll: $- ); cr; cr.
]

Now yes, let’s execute something that causes an error. What I did is to evaluate 1/0. After that, you should see the file PharoDebug.fuel in the same directory where the image is. You can now close the image and reopen it. And then, let’s reopen de debugger:

| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label:  'This is the new debugger!' contents: nil

Et voilà! Hopefully that worked    Notice that in this example and the previous one, there is nothing in special with the Fuel serialization. You are using the normal API, and all you do is to serialize a debugger or a context as if you were serializing any normal object. Of course, you can also apply this idea to other places. For example, in Seaside you have an error handler. You may want to serialize the error with Fuel there.

Limitation and known problems

• Even if Fuel can fully serialize methods, classes, traits, etc., it is recommended that the image were the contexts/debuggers are serialized and materialized are equal. If you are doing this in a production application, then you can have the same image running locally. The idea is that both images have the same classes and methods installed. This is because, by default, if the object graph to serialize includes compiled methods, classes, class variables, etc., they are all considered as “globals”, which means that we only serialize its global name and then during materialization it is searched in Smalltalk globals. Hence, classes and methods have to be present. Otherwise you have to use Fuel in a way that it serializes classes as well, but that’s more complicated.
• There may be things that affects the debugger which are part of the image and not the serialization, and they may have changed. Imagine for example, a class variable which has changed its value in the image where you serialize. Then it will have a different value in the image where you materialize. Most of these problems also happens if even if you open the debugger later in the same image…some state may have changed…
• The graph reachable from the contexts can be very big. For example, Esteban Lorenzano was doing this for an application and one of the problems is that from the context it was reachable the whole UI…which means lots and lots of objects. In such a case, you can always use Fuel hooks to prune the object graph to serialize.
• Be aware to use exactly the same version of Fuel in both images

Conclusion

All in all, I think that as a very first step, it is very nice that we can serialize this kind of stuff like contexts and debuggers out of the box with Fuel. This could be the infrastructure for a lot of fancy stuff. I don’t think that the debugger materialization can be as reliable as if you were debugging in the original image. I don’t think either that it should replace PharoDebug.log. However, what I do think is that you can add the Fuel serialization just as another way of getting more information about the problem. It’s one more tool you can add to your Smalltalk toolbox

Gofer it
url: 'http://ss3.gemstone.com/ss/ExperimentalBit';
package: 'ConfigurationOfExperimentalBit';
load.

So, please distribute along spanish speaking smaltalkers.

For the translation, there are a lot of people to thank, there were like 15 translators. Special thanks to Gabriala Arevalo who start leading it, and then to Nicolas Paez that continues it.

Besides the translation, Pharo By Example 2 is in the way and there are drafts for already a lot of chapters:

• Metacello
• Settings
• OmniBrowser
• Regular expressions
• Exceptions
• Monticello

That’s all, I hope you enjoy them.

I have worked in industry with Eclipse for 3 years. And I am a geek who stays hours in front of the PC and when I was in Linux/Windows I was using all the time Total Commander or something similar. I used to know all possible shortcuts of all the software I was using in my machine. When I then started to program in Squeak it has like if I come back 20 years in time. I was not able to use the keyboard as I was doing it in the rest of my machine (and Eclipse included). It felt really frustrating. But ok…Smalltalk showed me so many cool stuff that I could live with the lack of proper shortcuts.

Keymapping

Today, I can say Pharo has improved a little in the direction of improving the user experience with the keyboard.  This is thanks to Guillermo Pollito who has been developing Keymapping for a while. And now Débora Fortini has joined. Keymapping is a really small tool that let you define keyboard shortcuts per morph. So you can take any subclass of Morph and define your OWN shortcuts to do what you want. Of course, it may happen that there are conflicts with shortcuts of other morphs or even with global shortcuts of your Operating System. Nevertheless, the framework is there and working.

So….why keymapping rocks?  Well, during Smalltalks I say to Guillermo “If you give me shortcuts to step into, step over, restart, proceed, and go up and down in the Debugger, I pay you several beers”. After 5 minutes, they were already working perfectly. What was needed?  First just create the class:

Object subclass: #KMDebuggerShortcuts
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'Keymapping-Debugger'

And then put a class side method :

buildDebuggerShortcutsOn: aBuilder
<keymap>
(aBuilder shortcut: #stackDown)
category: #DebuggerShortcuts
default: $d command shift, Character arrowDown
do: [ :target | target down ].
(aBuilder shortcut: #stackUp)
category: #DebuggerShortcuts
default: $d command shift, Character arrowUp
do: [ :target | target up ].
(aBuilder shortcut: #proceedDebugger)
category: #DebuggerShortcuts
default: $d command shift, $p
do: [ :target | target  proceed ].
(aBuilder shortcut: #restartDebugger)
category: #DebuggerShortcuts
default: $d command shift, $r
do: [ :target | target  restart ].
(aBuilder shortcut: #stepIntoDebugger)
category: #DebuggerShortcuts
default: $d command shift, $i
do: [ :target | target  send ].
(aBuilder shortcut: #stepOverDebugger)
category: #DebuggerShortcuts
default: $d command shift, $o
do: [ :target | target  doStep ].
(aBuilder shortcut: #stepIntoBlockDebugger)
category: #DebuggerShortcuts
default: $d command shift, $t
do: [ :target | target  stepIntoBlock ].
(aBuilder shortcut: #SeeFullStackDebugger)
category: #DebuggerShortcuts
default: $d command shift, $f
do: [ :target | target  fullStack ].
(aBuilder shortcut: #runHereDebugger)
category: #DebuggerShortcuts
default: $d command shift, $h
do: [ :target | target  runToSelection ].

All shortcuts are defined in the same way: you create a method which receives a builder. The method #shortcut receives a shortcut name and creates an instance of KMKeymapBuilder. Then, sending the message #category:default:do: to the created KMKeymapBuilder, you can define all shortcuts. The first parameter is a category. The second is the combination of keys for the shortcut and the last parameter is the block to evaluate when such shortcut is pressed. Notice that the block receives at least the target object. In this case the target is the debugger.

Now, the last part is to tell the Debugger that it should attach those shortcuts. To do that, we had to overwrite (for example) the method Debugger >> #openFullMorphicLabel:

openFullMorphicLabel: aLabelString
"Open a full morphic debugger with the given label"

| window |
window := UIManager default openDebugger: self fullMorphicLabel: aLabelString.
window attachKeymapCategory: #DebuggerShortcuts targetting: self.
^window

Notice in this method the line number 6. So…after 5 minutes that took Guille to code that, I had the shortcuts I ever wanted.

Available shortcuts

What is important of Keymapping is the infrastructure since it give us the tool to create our own shortcuts. Or even better, integrate in Pharo new shortcuts. Just to give you an idea, KMMonticelloShortcuts has shortcuts for committing, open repository, view changes, etc. KMGlobalShortcuts has shortcuts for saving the image, to open the world menu, to open a workspace, to open a transcript, to open Monticello, to open a class browser, etc… So…come on!!  what are you waiting to give it a try?

How to install it?  As easy as always:

Gofer new
squeaksource: 'ShortWays';
package: 'ConfigurationOfKeymapping';
load.

(ConfigurationOfKeymapping project version: #bleedingEdge) load

Notice that Keymapping has an integration with the Settings framework. So if you open the Setting Browser and go to the category “Keymapping” you will see some of the shortcuts and of course you will be able to modify them. Note that not all shortcuts are yet integrated to the Settings framework since Keymapping is still in development

What I would like in the future

• The VM so far cannot handle some key events of certain keys like functions. Moreover, it seems (but I am not sure) that different VMs (the Windows VM, Linux, Mac OSX) do not handle all the key events the same way. It would be nice if they can be uniform.
• In Pharo, when you do right-click or whatever menu that show you options like “do-it”, “print it”, etc., they all show the shortcut for them. However, if you do change them with KeyMappings, then such menu is not updated. It would be nice if the menu is auto-generated depending on the current configured shortcut.
• It would be nice to have a way of tracking conflicts between shortcuts.

Background and Motivations

To help the Pharo community I find myself using new tools (even if they are not completely finished), using/fighting with latest versions (a.k.a bleeding edge), finding/reporting/fixing/testing bugs and issues. So…. I am downloading hundred of images every day. An image can last me a maximum of a couple of days.

Apart from that, after 3 years with Squeak/Pharo I have my own preferences/settings which are different from the defaults. In addition, I often use tools which are not present in the standard Pharo image. I am also working in several projects at the same time. So….. I spend a lot of time building my own images.

If I am downloading hundred of images every day, an image can last me a maximum of a couple of days, and I spend a lot of time building my own images, then there is something that it is not working.

Moreover, I am lazy, I don’t like loosing time with it, and my memory is bad. Hence, I have come to the solution that I will show you in this post.

Types of Operations and Softwares

To explain you what I did, I will use an example my own projects and tools.

Types of Operations

When I am working with Pharo, there are usually 2 types of operations I perform with a particular project: load and build.

Load means that I will just load a project or package into the image where I am working. No more than that. This is useful when I am working with a project with a particular image, and I just want to load another one to see something.

Build, however, not only loads the project or package but also builds my own image for that project. What does it mean to build my own image? well, I install all the tools I use that are not present by default in Pharo, I set my personal settings, etc.

Types of software

There are also 2 types of software: own software and external tools.

Own software are all those projects I own or that I contribute. For these projects, I develop, hence I always want to load the very last version of all packages. In the example of this post, there is Fuel, DBXTalk, Marea and Cog.

External tools are those development tools, addons, themes, or whatever I install when I build my own images. Since I don’t develop these tools, I don’t want the very last version of every package, but instead the last “stable” version. In the example, there are TilingWindowManager,  ‘SimpleLogger’, ‘CodeStats’, ‘Glamoroust’, ‘Keymapping’, ‘SandcastlesThemes’, etc.

Metacello part of the solution

If you have used Metacello before, you are used to see classes like ConfigurationOfSeaside, ConfigurationOfDBXTalk, ConfigurationOfFuel, ConfigurationOfMoose, etc. What I propose is to that you create your own ConfigurationOf. In my case, I have ConfigurationOfMariano. If you want to follow the post watching the real code in your image, then just evaluate:

Gofer new
url: 'http://ss3.gemstone.com/ss/MarianoBuilder';
package: 'MarianoBuilder';
load.

Such configuration has no versions and only one baseline method. The (summary) of such method is:

baseline10: spec
<version: '1.0-baseline'>

spec for: #pharo do: [

spec blessing: #baseline.

" =========================================
The following the list of projects that I DO DEVELOP. Hence, I always want the bleedingEdge, and most of the times, I want to build an image for them.
========================================="

spec
project: 'Fuel' with: [
spec
className: 'ConfigurationOfFuel';
loads: #('DevelopmentGroup' );
file: 'ConfigurationOfFuel';
versionString: #bleedingEdge;
repository: 'http://ss3.gemstone.com/ss/Fuel' ].

spec
project: 'DBXTalk' with: [
spec
className: 'ConfigurationOfGlorpDBX';
loads: #('default' );
file: 'ConfigurationOfGlorpDBX';
versionString: #bleedingEdge;
repository: 'http://www.squeaksource.com/DBXTalk' ].

"....ACTUALLY, THERE ARE MORE PROJECTS...."

" =========================================
The following the list of TOOLS, ADDONS  that I use for my own images. Since they are not my projects, I usually use the latest available metacello version and not the bleedingEdge.
In addition, not all tools and addons are loaded in all type of images I build. Each image decides what to include.
========================================="

spec
project: 'TilingWindowManager' with: [
spec
className: 'ConfigurationOfTilingWindowManager';
loads: #('default');
file: 'ConfigurationOfTilingWindowManager';
repository: 'http://www.squeaksource.com/TilingWindowManager' ].

spec
project: 'Glamoroust' with: [
spec
className: 'ConfigurationOfGlamoroust';
loads: #('GT-Inspector' 'GT-Playground' );
file: 'ConfigurationOfGlamoroust';
repository: 'http://www.squeaksource.com/glamoroust' ].

spec project: 'Keymapping' with: [
spec
className: 'ConfigurationOfKeymapping';
loads: #('forDeveloper' );
file: 'ConfigurationOfKeymapping';
repository: 'http://www.squeaksource.com/ShortWays' ].

"....ACTUALLY, THERE ARE MORE TOOLS...."

" =========================================
The following is a list of groups for building my own images. Notice that for each image group there should be a class side method #buildXXXImage
========================================="

spec
group: 'FuelImage' with: #( 'BaseImage' 'SandcastlesThemes' 'Fuel');
group: 'MareaImage' with: #( 'BaseImage' 'Marea');
group: 'DBXTalkImage' with: #( 'BaseImage' 'DBXTalk');
group: 'CogVMImage' with: #( 'BaseImage' 'CogVM');

group: 'BaseImage' with: #('TilingWindowManager' 'SimpleLogger' 'CodeStats' 'Glamoroust' 'Keymapping').

].

Forget for a moment the last lines about groups. The basic idea is that in such configuration you group both types of software, your own or the one you contribute, and the external tools. There are a couple of interesting things to notice:

• Whenever I want to install something with Metacello, I never remember whether the ConfigurationOf is in MetacelloRepository or in the same repository of the project. I don’t remember which group/packages I have to load either.  This way such information is stored directly in ConfigurationOfMariano. For example, for Fuel, I have to load ‘DevelopmentGroup’ and the ConfigurationOfFuel is in ‘http://ss3.gemstone.com/ss/Fuel’. For DBXTalk I have to load ‘default’ and the ConfigurationOfGlorpDBX is in ‘http://www.squeaksource.com/DBXTalk’. Conclusion? I don’t have to remember that anymore.
• Notice that for my own projects (Fuel, DBXTalk, Marea, etc) I set #bleedingEdge as the #versionString. This is because of what I have already explained that I always want the last version of each package. In the contrary, for the external tools such as Glamoroust, KeyMapping, etc, I don’t specify anything in #versionString. Hence, the default will be the last stable version, exactly what I want for those cases.

Loading Projects

In the class side of ConfigurationOfMariano there is a category called ‘loading’ and there you can find methods like #loadFuel, #loadDBXTalk, #loadMarea, etc. As an example:

loadFuel
FuelImageBuilder new loadPackages: 'Fuel'

For the moment, forget what FuelImageBuilder does, but imagine it just loads ‘Fuel’ (this is the parameter to #loadPackage:) from ConfigurationOfMariano. Of course, the rest of the #load* methods are the same and they only change the package to load and the builder class.

Conclusion? I can take any Pharo image, Gofer MarianoBuilder and then all I need to do is to send #loadXXX message to ConfigurationOfMariano No need to remember anything nor loosing time. In fact what I do is to have  sticky note in my Mac with this code:

Gofer new
url: 'http://ss3.gemstone.com/ss/MarianoBuilder';
package: 'MarianoBuilder';
load.

(Smalltalk at: #ConfigurationOfMariano) perform: #loadFuel.

I copy paste such code and I change #loadFuel for what I want at that moment.

Building my own images

Now let’s see the interesting part. To build your own images with this approach, there are 2 parts. The first part is done in ConfigurationOfMariano. Take a look to the groups defined in the baseline at the beginning of the post.

Basically, we create a Metacello group for each image we want to build. For example, there is a group ‘FuelImage’, ‘DBXTalkImage’, etc. In such groups we can put all external tools that are exclusively needed when we build images for such project. For example, in Fuel I add ‘SandcastlesThemes’ because for Fuel images I want to use such theme. Besides particular tools that each image needs, there is also a group called ‘BaseImage’, where I put all the external tools I want for all images.  In this example, ‘TilingWindowManager’ ‘SimpleLogger’ ‘CodeStats’ ‘Glamoroust’ and ‘Keymapping’.

So…all we do in the baseline of ConfigurationOfMariano related to building images is just to define which packages/projects to load. Now we have to really build the image. Most of the code for building my own images is quite similar among different projects. For example, the code to build a Fuel image will be quite similar to the one to build a DBXTalk image. The image builder is done in Pharo as well so the easiest solution is to use subclassification to customize the build process for each project. The abstract class is GeneralImageBuilder and there are subclasses such as MareaImageBuilder, FuelImageBuilder, DBXTalkImageBuilder, CogVMImageBuilder, etc.

The last part of ConfigurationOfMariano is the class side category called ‘building’, which contains methods such as #buildFuelImage, #buildDBXTalkImage, etc. You can imagine what they do:

buildFuelImage
FuelImageBuilder new buildImage.

The builder

The main method is #buildImage

buildImage
self loadImagePackages.
self generalImagePreferences.
self customPreferences.
self cleanSystem.
self lastActions.
self saveImage.

The first step is to load all the packages defined as part of the image. Each subclass of GeneralImageBuilder understands #imageGroupNameToLoad  which answers the metacello group defined in ConfigurationOfMariano. For example, FuelImageBuilder >> #imageGroupNameToLoad   just answers ‘FuelImage’.

The second line is to set all preferences/settings which are general to all images. Line 3 is a hook that allow subclasses to set custom settings. For example, Fuel and Marea they both set their own background picture, logo, theme, log file name, etc (check implementors of #customPreferences).

With #cleanSystem we execute some cleanings to let the image in a nice and clean state.  Then #lastActions is another hook where each subclass can execute something at the end. For example, to set my username and password for all SqueakSource repositories, to set a global/shared package-cache for all projects, etc.

Finally, we save the image. In my case, I want that the image is saved with a particular name and in a particular place. For example, I want that all Fuel images are called Fuel.N.image and to be placed in ‘/Users/mariano/PhD/Marea/Fuel/imagenes/’. Implementing #imageDirectory and #baseImageName subclasses can customize that.

Conclusion?   I can take any Pharo image, Gofer MarianoBuilder and just sending #buildFuelImage or #buildXXX I can automatically build my own image with my own tools, settings and customizations. The image will be named and placed exactly where I want.

My final proposal

As said, you can find the code in http://ss3.gemstone.com/ss/MarianoBuilder and the license is MIT. I didn’t call it MarianoBuilder because of ego, but on purpose to make it clear that MarianoBuilder is just for me, it has my own preferences, my own settings, my own projects. What you should copy is the procedure. Copy the idea of using Metacello for something else than managing projects. Create your own ConfigurationOfYou and YourBuilder. Of course, you are very welcome to take a look to MarianoBuilder to get an idea of how to set certain preferences or settings. You are invited to take it as a base and fork it.

There are more things I would like to tell about this topic but the post is already too long so I will continue after…

The talk “Building your own Pharo images with Metacello” was good or at least that’s what I think. Of course, it was a talk that only makes sense for certain people or scenarios. Moreover, it was a “pre-requirement” to know Metacello which n is something not all people know. The talk was in the big auditorium and there were quite a lot of people. The only thing I regret from the presentation is that I forgot to say that the code was MIT and that everybody could take it and adapt it to their own needs. Nevertheless, I added a slide with this information to the final version of the slides (which will be published in the website).

You can find the slides here but notice that most of the presentation was a live demo. You will need to wait for the video if you want to see it   (it was recorded so it should be online soon). I like the topic of this talk so I will try to do a special post about it in the future.

The second talk was the presentation of my paper: “Problems and Challenges when Building an Unused Object Manager for Pharo”. It was in the secondary room so there were a few people (no more than 20). Besides, the talk was in parallel with a nice talk of Hernán Wilkinson. I hope that I least I could explain more or less what I am trying to do for my PhD. The slides of the talk are here.

That’s all folks

CI introduction and Pharo integration

I guess most of you are aware of what Continuous Integration and CI server are. Basically are tools that help us to automatically build systems, run tests, make final binaries, run code critics, etc. One of the most popular ones is Jenkins (previously known as Hudson). We would like to have that for Smalltalk, but the problem is that most of them are targeted for a specific language, for example, for Java.

Fortunately for us, thanks to Yanni Chiu and Lukas Renggli, we have two things: 1) HudsonBuildTools: an integration between Jenkins and Pharo Smalltalk. 2) A builder.

HudsonBuildTools: You can check the package HudsonBuildTools in http://source.lukas-renggli.ch/hudson. Basically, the package is an adaptor between what Java expects and what Smalltalk generates. So from the Smalltalks results of running tests, this package generates the .xml files (probably the JUnit results) that Jenkins expects.  Not only for tests but also for Lint rules, test coverage, etc.

Builder: this is a set of bash files that help us all the process of building Pharo images from command line. Of course, we can then use this builder from Jenkins. The builder takes care of, for example, executing the VM and passing by argument the Smalltalk script to load what you want to load in the image. It also has some cache to improve its performance and some hooks to execute code after or before each build. It makes the integration with Jenkins extremely easy because at the end all you have to do is to create a myLoadScriptFile.st in the folder /scripts and that’s all.

Installing Jenkins

This should be easy. You just go to the jenkins website and you download the latest version for Mac OS. In my case it was here. After executing the .pkg you should have Jenkins installed in /Applications/Jenkins/. If you enter there, you will see only one file: jenkins.war. Where is Jenkins really then?  Well, that’s Jenkins “binary” (search in Google for more details about what a Java .war is). How do you start then Jenkins?  Well, it should be as easy as executing from a command line:

java -jar /Applications/Jenkins/jenkins.war

But of course, in my machine it didn’t start and throw an error saying the port 8009 was already in use…After googling a little bit, I found that I could change the port of the AJP13 protocol. I finally ended up changing both ports AJP13 and HTTP port (because default is 8080 and can be in conflict with some other stuff I have running). For more details about it read this link. So finally I had to start Jenkins like this:

java -jar /Applications/Jenkins/jenkins.war --ajp13Port=8010 --httpPort=7777

With that, I could successfully start Jenkins and I could browse its UI in: http://localhost:7777/.

Installing the builder

Nick Ager did a great post about Jenkins and Smalltalk. And the builder’s README is pretty clear. You should read both. I wont repeat all what they have already explained. I will just mention the problems I had and what I did to solve them.

1) The first problem I found is that build.sh expects that for Mac OSX:  PHARO_VM=”$VM_PATH/MacOS/Squeak VM Opt”. And VM_PATH=”$BASE_PATH/oneclick/Contents” (notice that “uname -s” MAC OSX will answer “Darwin”).  So… I needed to correctly set PHARO_VM to my VM. Hence,  after the “case “$(uname -s)” in” and before “# build configuration” I added the line:

PHARO_VM="/Users/mariano/Pharo/VM/CogMT5.app/Contents/MacOS/Croquet"

So that way I am using that particular VM.

2) The second problem is that from Jenkins we will call the builder, so all those .sh has to be found in $PATH. What I did is to create the following startJenkins.sh:

#!/bin/bash

WARFILE="/Applications/Jenkins/jenkins.war"
LOGFILE=jenkins.log

export PATH="$PATH:/Users/mariano/PhD/Jenkins/builder"

java -jar $WARFILE --ajp13Port=8010 --httpPort=7777 > $LOGFILE 2>&1 &

java -jar /Applications/Jenkins/jenkins.war --ajp13Port=8010 --httpPort=7777

Notice that this is a simple script that I use *locally*. If you are running Jenkins in a server, you may probably want to use nohup or something like that.

Configuring Jenkins and the builder

More about the builder, tests and Jenkins

There were some things which were not easy or were not working out of the box. Hence, I will share them wth you.  Just to clarify if it is not already clear. If you want that Jenkins run tests, then you need not only a .st that loads the test, but also the “buildtools” script, which downloads the HudsonBuildTools. So if you create a job and you want to run tests for it you have to put something like:

build.sh -i Pharo-1.3 -o seaside3 -s seaside3
build.sh -i seaside3 -o seaside3-tests -s buildtools -s seaside3-tests

As you can imagine, for each script you pass with “-s” you need a corresponding .st in /scripts. We are putting two lines (commands) to the “execution shell”. The first takes the Pharo-1.3 image as input, loads seaside with the seaside3.st and then saves the image as seaside3.image. The second line, takes that seaside3.image as input, loads HudsonBuildTools (using buildtools.st), and then loads seaside3 tests and run them. We can see that seaside-tests.st is something like:

HDTestReport runPackages: ((WADevelopment allPackages
    select: [ :each | each name includesSubString: '-Tests-' ])
    collect: [ :each | each name ]).
HDLintReport runPackages: ((WADevelopment allPackages
    reject: [ :each | each name includesSubString: '-Tests-' ])
    collect: [ :each | each name ]).

In this case of Seaside3, the tests are not loaded by seaside-tests.st since they are loaded in seaside3.st. But you can split this in your projects and only load tests in the second image.

Notice that you only need the “buildtools” it if you want to run tests, lint, test coverage, etc. If you just want to build images, then “buildtools” parameter is not needed. You can read the README file for more details.

Non-Interactive UIManager

In my case, I was building an image which opens a Transcript and for some reason it also asks for the author name (it is fixed now). Anyway, one of the problems I found is that since the builder is running the image headless, we should use a non-interactive UIManager. To do that I need to evaluate:  “UIManager default nonInteractiveManager”. What I did is to add such line in /scripts/before.st. That way, all my scripts will include it by default.

Needed plugins

As explained in the README, for both, tests and lint, you need a special Jenkins plugin. To install them you can go from the web interface to “manage Jenkins” -> “manage plugins” or directly go to http://localhost:7777/pluginManager/. Then go to the tab “Available” and install those you need. Then restart Jenkins.

Creating Jenkins users

To create Jenkins users, you have to first enable security. Here you have a nice explanation of all the requires steps to be able to create users for Jenkins.

Test Coverage

If you want to run the test coverage you have to use the latest (VM.r2508 for example) CogVM from Eliot Miranda. TestCoverage uses a hook of the VM which is not working correctly in previous versions of the VM.

Conclusions

All in all between 1 and 2 hours, I was able to have my Jenkins running locally and building my own images. I am not a bash expert and I haven’t installed Jenkins before, so you should be able to do it fast also. If you were thinking that configuring Jenkins to build and test your images was something complicated and only for “gurus” then you should re-think it and give it a try. In my experience it was really straightforward. Of course, to have a real/secure Jenkins running you may need to spend some time giving correct permissions to files, users, etc, etc, etc.

During Smalltalks 2011 I talked to Dale Henrichs who told me that SqueakSource3 was quite stable and has daily backups. SqueakSource3 is a new implementation of SqueakSource but based in Seaside3, Magritte2, etc. Even more (and this is very cool), it is deployed in a GemStone server. Thanks a lot for GemStone for hosting such a service and to all the developers behind SS3.

So, I decided to give it a try. I have started to migrate all my projects or the projects I contribute and I thought I could share that with you.

Migrating projects to SqueakSource3

There are two ways I found in order to migrate packages from one repository to another one. As an example, I will show you how I have migrated Fuel code from http://www.squeaksource.com/Fuel to http://ss3.gemstone.com/ss/Fuel

Pre-optional step

As explained later, you will need to fetch all versions of the packages you want to migrate. Downloading that from squeaksource can be quite slow and you may need a good internet connection. However, if you want to migrate a project you have been development, it is probable that some .mcz are still somewhere in your machine. What you can do is to copy all .mcz files of your machine to the package-cache so that you can get it from there and avoid the network traffic. To do that I often use this bash script:

#!/bin/bash

find /Users/mariano/ -name "*.mcz" -print0 | xargs -0 -I {} cp {} /Users/mariano/Pharo/YOUR-PACKAGE-CACHE/

Using Gofer #fetch and #push

As you know, Gofer is a nice scripting library for Monticello and it provides the methods #fetch (which downloads versions from remote repositories into the local cache) and #push (which uploads local versions from local cache into remote repositories). So the idea is that you can fetch the versions from the original repository and push in the new one. In the example of Fuel, I should do:

Gofer it
squeaksource: 'Fuel';
package: 'Fuel';
package: 'FuelTests';
fetch.

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'Fuel';
package: 'FuelTests';
push.

Notice that such code could take a lot of time since EACH version of EACH package has to be fetched and pushed. So…let your machine, take a coffee and then come back. In this example, I only migrate two packages (‘Fuel’ and ‘FuelTests’) but there are much more packages in the Fuel repository. Having to write 20 lines of #package: can be a lot. So if you are a lazy guy like me, you can do the following hack:

| packagesNames gofer |

packagesNames := (Gofer new
disablePackageCache;
squeaksource: 'Fuel';
allResolved)
collect: [ :each | each packageName ] as: Set.

gofer := Gofer it
squeaksource: 'Fuel'.

packagesNames do: [:each | gofer package: each].
gofer fetch.

gofer := Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel'.

packagesNames do: [:each | gofer package: each].
gofer push.

That way you can fetch and push all packages. Of course, you can always get the list first, remove by hand the packages that you don’t want to migrate, and finally just migrate the ones you want.

Using Gofer and Metacello configurations

Instead of manually choosing which packages to migrate, if you do have Metacello configurations, why not to directly use them? Dale has an excellent post explaining exactly how to do that. Please read it if you are interested in such approach. Basically it consist of migrating only those packages of a Metacello version. Example:

| gofer |
"Identify the source repository"
gofer := Gofer new squeaksource: 'Fuel';
package: 'ConfigurationOfFuel' yourself.
"Traverse all packages defined in baseline '1.7-baseline' of the Fuel configuration"
(ConfigurationOfFuel  project version:  '1.7-baseline') packages do:[:spec |
"Use the #name and #package: messages and Gofer will fetch
all versions of the Monticello package"
gofer package: spec name ].
"Initiate the download with the #fetch command"
gofer fetch.

"Identify the target repository"
gofer := Gofer new url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel' yourself.
"Use the same algorithm for identifying the packages to be pushed"
(ConfigurationOfFuel project version: '1.7-baseline') packages do:[:spec |
gofer package: spec name ].
"Initiate the upload with the #push command"
gofer push.

The only difference with the previous approach is that this way we automatically migrate the packages referenced from the configuration.

Problems I found

Fetching and pushing take a long time, so be patient and try to be with a good internet connection. In my case, I have several errors which I have to deal with:

•  Timeouts: If you get a “ConnectionTimedOut: Data receive timed out.” then it is clear there was a timeout with a certain package. What I did when I had this error is to open the debugger, go down in the stack a little while up to the place where I see it was doing the load of the version, and do a restart. Otherwise, you can always close the debugger and execute the code again, but this will take more time.
• If you get the “Error: file is too short” then it means that there is some package (.mcz file) in your package-cache with zero bytes. Open the debugger, go down in the stack and identify which package/version has the problem. Then just go the package cache and remove such file. Then restart the migration procedure.

So, that’s all. I hope I could make your migration easier

Direct pointers vs object tables

Let’s say we have this code:

| aPoint |
| aPoint := Point x: 10 y: 20.5.

In this case, aPoint has an instance variable that refers to an integer (10) and a float (20.5). How are these references implemented in the VM?

Most virtual machines have an important part whose responsibility is managing the memory, allocating objects, releasing, etc. In Squeak/Pharo VM, such part is called Object Memory. In addition, the Object Memory defines the internal representation of objects, its references, its location, its object header, etc.  Regarding the references implementation, there are two possibilities which are the most common: object tables and direct pointers.

With the first, there is a large table with two entries. When the object aPoint refers to the float 20.5, it means that the instance variable “y” of aPoint has an index in the table where the memory address of the float 20.5 is located. With direct pointers, when aPoint refers to 20.5, it means that the instance variable “y” of aPoint has directly the memory address of 20.5.

There are pros and cons for each strategy but such discussion is out of range for this post. One of the nice things with object tables is that the primitive #become: is really fast since it is just updating one reference. With direct references, the #become: it needs to scan all the memory do detect all the objects that are pointing to a particular one. On the other hand, with object tables, we have to pay the cost of accessing an extra indirection and (I guess) this may impacts on the overall performance of the system. With direct pointers, we do not have that problem. Finally, object table uses more memory since the table itself needs memory. Few months ago there was a nice discussion in the mailing list about the prons and cons.

First Smalltalk VMs used to have an object table, but now most current VMs (included the Squeak/Pharo VM) use direct pointers. The only current VM I am aware of that uses object tables is GemStone. But… they actually have one (virtual) Object Table (OT) per committed transaction!!  How they can do those optimizations and not blowup in terabytes of memory used by OTs? Well, that’s one of GemStone keys   If you are interested in this topic, you can read this thread.

Memory addresses

In the previous paragraphs you learn that each memory address in the Squeak/Pharo VM represents a direct pointer to another object. Well, that’s almost correct. We are missing what it is usually known as “immediate objects”.  Immediate objects are those that are directly encoded in the memory address and do not require an object header nor slots so they consume less memory. In the CogVM there is only one type of immediate object, and it is SmallInteger. What does it mean?

In our example, the instance variable “x” of aPoint does not have a pointer to an instance of SmallInteger with the content 10. Instead, the memory address of “x” has directly encoded the value 10. So there is no instance of SmallInteger. But now, how the VM can known whether an instance variable is a pointer to another object or a SmallInteger? We need to tag a memory address to say “this is a object pointer” or “this is a SmallInteger”. To do that, the VM uses the last bit of the word (32 bits). If such bit is 1, then it is a signed 31-bits SmallInteger. If it is 0, it is a regular object pointer (oop).

Since I told you SmallInteger were encoded in 31 bits and they were signed, it follows that we have 30 bits for the number (one bit is for the sign). Hence, SmallInteger maxVal should be (2 raisedTo: 30) -1, that is, 1073741823. Analogy, SmallInteger minVal answers -1073741824. Number are encoded using the two’s complement. If you want to know more about this, read the excellent chapter that Stéphane Ducasse wrote about it.

Now, regarding object pointers, they always point to the memory address where the object header is. In our example, the instance variable “y” of aPoint, has the memory address of 20.5‘s object header.

As you can imagine, the VM needs to check all the time whether a OOP is really an OOP or an integer:

ObjectMemory >> isIntegerObject: objectPointer

^ (objectPointer bitAnd: 1) > 0

If you have an image with Cog loaded (as I explained in all my posts about building the VM), you can check for its senders…and you will find quite a lot

Previously, I explain you why SmallInteger instances do not have object headers and those instances do not really exist as “objects”. That’s exactly why “SmallInteger instanceCount” answers zero. Each SmallInteger is encoded in different instance variables of different objects.

Another funny fact is why identity is always true with SmallIntegers. Say you have  ’1′ asNumber ==  (4-3), that answers true. Because at the end, the VM calls a regular C’s equality (=), which of course, for 2 equal numbers, it is always true. But of course, if those numbers are actually OOP (a number), if they are equal, then it means they both point to the same object:

StackInterpreter >> bytecodePrimEquivalent

| rcvr arg |
rcvr := self internalStackValue: 1.
arg := self internalStackValue: 0.
self booleanCheat: rcvr = arg.

There are more things where you can notice that SmallInteger is special. In fact, you can browse the class and see some methods it overwrites, like #nextInstance (throwing an error), #shallowCopy, #sizeInMemory, etc. And of course, there are more problems like trying to do a become. For example, (42 become: Date new) throws an error saying it cannot become SmallIntegers.

More immediate objects?

As said, in a word of 32 bits, we only use 1 bit for tagging immediate objects (SmallInteger in the case of the squeak VM). We could use more than 1 bit…but then it means we have fewer bits for the OOP, therefore, the maximum possible memory to address is smaller, because the amount of bits of the OOP limits us in how much memory we can address as maximum.

But….what happens in a 64-bits VM?  I think 63 bits can be more than enough  for memory addresses. So what about using fewer bits for OOP and more for immediate objects?  Say we can use 58 for OOP and 6 for tagging immediate objects. In that example, we have (2 raisedTo: 6) – 1 , that is,  63 different possibilities!!!  So we can not only encode SmallIntegers but also small floats, true, false, nil, characters, etc… Is that all?  No! there are even more ideas. We can not only encode instances of certain class, but also give semantics to the possibility of tagging memory addresses. For example..we could use one of the combinations of tag bits to say that memory address is in fact a proxy. It doesn’t need to be an instance of Proxy, but we just give the semantics that when a memory address finishes with that tag bit, it means that the 58 bits for the OOP is not an OOP but a proxy contents. Such content can be a number representing an offset in a table, an address in secondary memory, etc… The VM could then do something different if the object is a proxy!

Well…all that I mention is not new at all. In fact, Gemstone does something very similar. They use 61 bits for address + 3 for tags. Here is a nice set of videos about Gemstone’s internals.  And in this video you can see what we are speaking here.

Documentation and future posts

I always try to put some links together related to each post I talk about:

• Object Table explanation in the blue book
• Slides and video of “Journey In The VM”
• A Tour of the Squeak Object Engine

In the next post, I will give details about the current Object Header.

Some time ago, they contacted me to do an interview, mainly because Fuel won the ESUG Awards. The interview also included some questions related to my PhD and what I am doing here in France. So… if you are interested in knowing why we have started Fuel, which was the most difficult part, what is a pickle format, etc, I really recommend you to take a look at it.

The interview is here.

See you