Loading projects and building your own images with Metacello

As I have already said, I presented at Smalltalks 2011 a talk called “Building your own images with Metacello”. I thought it could be interesting to write down the details of such presentation for those who couldn’t attend or those who want to follow what I did.

Background and Motivations

To help the Pharo community I find myself using new tools (even if they are not completely finished), using/fighting with latest versions (a.k.a bleeding edge), finding/reporting/fixing/testing bugs and issues. So…. I am downloading hundred of images every day. An image can last me a maximum of a couple of days.

Apart from that, after 3 years with Squeak/Pharo I have my own preferences/settings which are different from the defaults. In addition, I often use tools which are not present in the standard Pharo image. I am also working in several projects at the same time. So….. I spend a lot of time building my own images.

If I am downloading hundred of images every day, an image can last me a maximum of a couple of days, and I spend a lot of time building my own images, then there is something that it is not working.

Moreover, I am lazy, I don’t like loosing time with it, and my memory is bad. Hence, I have come to the solution that I will show you in this post.

Types of Operations and Softwares

To explain you what I did, I will use an example my own projects and tools.

Types of Operations

When I am working with Pharo, there are usually 2 types of operations I perform with a particular project: load and build.

Load means that I will just load a project or package into the image where I am working. No more than that. This is useful when I am working with a project with a particular image, and I just want to load another one to see something.

Build, however, not only loads the project or package but also builds my own image for that project. What does it mean to build my own image? well, I install all the tools I use that are not present by default in Pharo, I set my personal settings, etc.

Types of software

There are also 2 types of software: own software and external tools.

Own software are all those projects I own or that I contribute. For these projects, I develop, hence I always want to load the very last version of all packages. In the example of this post, there is Fuel, DBXTalk, Marea and Cog.

External tools are those development tools, addons, themes, or whatever I install when I build my own images. Since I don’t develop these tools, I don’t want the very last version of every package, but instead the last “stable” version. In the example, there are TilingWindowManager,  ‘SimpleLogger’, ‘CodeStats’, ‘Glamoroust’, ‘Keymapping’, ‘SandcastlesThemes’, etc.

Metacello part of the solution

If you have used Metacello before, you are used to see classes like ConfigurationOfSeaside, ConfigurationOfDBXTalk, ConfigurationOfFuel, ConfigurationOfMoose, etc. What I propose is to that you create your own ConfigurationOf. In my case, I have ConfigurationOfMariano. If you want to follow the post watching the real code in your image, then just evaluate:

Gofer new
url: 'http://ss3.gemstone.com/ss/MarianoBuilder';
package: 'MarianoBuilder';
load.

Such configuration has no versions and only one baseline method. The (summary) of such method is:

baseline10: spec
<version: '1.0-baseline'>

spec for: #pharo do: [

spec blessing: #baseline.

" =========================================
The following the list of projects that I DO DEVELOP. Hence, I always want the bleedingEdge, and most of the times, I want to build an image for them.
========================================="

spec
project: 'Fuel' with: [
spec
className: 'ConfigurationOfFuel';
loads: #('DevelopmentGroup' );
file: 'ConfigurationOfFuel';
versionString: #bleedingEdge;
repository: 'http://ss3.gemstone.com/ss/Fuel' ].

spec
project: 'DBXTalk' with: [
spec
className: 'ConfigurationOfGlorpDBX';
loads: #('default' );
file: 'ConfigurationOfGlorpDBX';
versionString: #bleedingEdge;
repository: 'http://www.squeaksource.com/DBXTalk' ].

"....ACTUALLY, THERE ARE MORE PROJECTS...."

" =========================================
The following the list of TOOLS, ADDONS  that I use for my own images. Since they are not my projects, I usually use the latest available metacello version and not the bleedingEdge.
In addition, not all tools and addons are loaded in all type of images I build. Each image decides what to include.
========================================="

spec
project: 'TilingWindowManager' with: [
spec
className: 'ConfigurationOfTilingWindowManager';
loads: #('default');
file: 'ConfigurationOfTilingWindowManager';
repository: 'http://www.squeaksource.com/TilingWindowManager' ].

spec
project: 'Glamoroust' with: [
spec
className: 'ConfigurationOfGlamoroust';
loads: #('GT-Inspector' 'GT-Playground' );
file: 'ConfigurationOfGlamoroust';
repository: 'http://www.squeaksource.com/glamoroust' ].

spec project: 'Keymapping' with: [
spec
className: 'ConfigurationOfKeymapping';
loads: #('forDeveloper' );
file: 'ConfigurationOfKeymapping';
repository: 'http://www.squeaksource.com/ShortWays' ].

"....ACTUALLY, THERE ARE MORE TOOLS...."

" =========================================
The following is a list of groups for building my own images. Notice that for each image group there should be a class side method #buildXXXImage
========================================="

spec
group: 'FuelImage' with: #( 'BaseImage' 'SandcastlesThemes' 'Fuel');
group: 'MareaImage' with: #( 'BaseImage' 'Marea');
group: 'DBXTalkImage' with: #( 'BaseImage' 'DBXTalk');
group: 'CogVMImage' with: #( 'BaseImage' 'CogVM');

group: 'BaseImage' with: #('TilingWindowManager' 'SimpleLogger' 'CodeStats' 'Glamoroust' 'Keymapping').

].

Forget for a moment the last lines about groups. The basic idea is that in such configuration you group both types of software, your own or the one you contribute, and the external tools. There are a couple of interesting things to notice:

• Whenever I want to install something with Metacello, I never remember whether the ConfigurationOf is in MetacelloRepository or in the same repository of the project. I don’t remember which group/packages I have to load either.  This way such information is stored directly in ConfigurationOfMariano. For example, for Fuel, I have to load ‘DevelopmentGroup’ and the ConfigurationOfFuel is in ‘http://ss3.gemstone.com/ss/Fuel’. For DBXTalk I have to load ‘default’ and the ConfigurationOfGlorpDBX is in ‘http://www.squeaksource.com/DBXTalk’. Conclusion? I don’t have to remember that anymore.
• Notice that for my own projects (Fuel, DBXTalk, Marea, etc) I set #bleedingEdge as the #versionString. This is because of what I have already explained that I always want the last version of each package. In the contrary, for the external tools such as Glamoroust, KeyMapping, etc, I don’t specify anything in #versionString. Hence, the default will be the last stable version, exactly what I want for those cases.

Loading Projects

In the class side of ConfigurationOfMariano there is a category called ‘loading’ and there you can find methods like #loadFuel, #loadDBXTalk, #loadMarea, etc. As an example:

loadFuel
FuelImageBuilder new loadPackages: 'Fuel'

For the moment, forget what FuelImageBuilder does, but imagine it just loads ‘Fuel’ (this is the parameter to #loadPackage:) from ConfigurationOfMariano. Of course, the rest of the #load* methods are the same and they only change the package to load and the builder class.

Conclusion? I can take any Pharo image, Gofer MarianoBuilder and then all I need to do is to send #loadXXX message to ConfigurationOfMariano 🙂 No need to remember anything nor loosing time. In fact what I do is to have  sticky note in my Mac with this code:

Gofer new
url: 'http://ss3.gemstone.com/ss/MarianoBuilder';
package: 'MarianoBuilder';
load.

(Smalltalk at: #ConfigurationOfMariano) perform: #loadFuel.

I copy paste such code and I change #loadFuel for what I want at that moment.

Building my own images

Now let’s see the interesting part. To build your own images with this approach, there are 2 parts. The first part is done in ConfigurationOfMariano. Take a look to the groups defined in the baseline at the beginning of the post.

Basically, we create a Metacello group for each image we want to build. For example, there is a group ‘FuelImage’, ‘DBXTalkImage’, etc. In such groups we can put all external tools that are exclusively needed when we build images for such project. For example, in Fuel I add ‘SandcastlesThemes’ because for Fuel images I want to use such theme. Besides particular tools that each image needs, there is also a group called ‘BaseImage’, where I put all the external tools I want for all images.  In this example, ‘TilingWindowManager’ ‘SimpleLogger’ ‘CodeStats’ ‘Glamoroust’ and ‘Keymapping’.

So…all we do in the baseline of ConfigurationOfMariano related to building images is just to define which packages/projects to load. Now we have to really build the image. Most of the code for building my own images is quite similar among different projects. For example, the code to build a Fuel image will be quite similar to the one to build a DBXTalk image. The image builder is done in Pharo as well so the easiest solution is to use subclassification to customize the build process for each project. The abstract class is GeneralImageBuilder and there are subclasses such as MareaImageBuilder, FuelImageBuilder, DBXTalkImageBuilder, CogVMImageBuilder, etc.

The last part of ConfigurationOfMariano is the class side category called ‘building’, which contains methods such as #buildFuelImage, #buildDBXTalkImage, etc. You can imagine what they do:

buildFuelImage
FuelImageBuilder new buildImage.

The builder

The main method is #buildImage

buildImage
self loadImagePackages.
self generalImagePreferences.
self customPreferences.
self cleanSystem.
self lastActions.
self saveImage.

The first step is to load all the packages defined as part of the image. Each subclass of GeneralImageBuilder understands #imageGroupNameToLoad  which answers the metacello group defined in ConfigurationOfMariano. For example, FuelImageBuilder >> #imageGroupNameToLoad   just answers ‘FuelImage’.

The second line is to set all preferences/settings which are general to all images. Line 3 is a hook that allow subclasses to set custom settings. For example, Fuel and Marea they both set their own background picture, logo, theme, log file name, etc (check implementors of #customPreferences).

With #cleanSystem we execute some cleanings to let the image in a nice and clean state.  Then #lastActions is another hook where each subclass can execute something at the end. For example, to set my username and password for all SqueakSource repositories, to set a global/shared package-cache for all projects, etc.

Finally, we save the image. In my case, I want that the image is saved with a particular name and in a particular place. For example, I want that all Fuel images are called Fuel.N.image and to be placed in ‘/Users/mariano/PhD/Marea/Fuel/imagenes/’. Implementing #imageDirectory and #baseImageName subclasses can customize that.

Conclusion?   I can take any Pharo image, Gofer MarianoBuilder and just sending #buildFuelImage or #buildXXX I can automatically build my own image with my own tools, settings and customizations. The image will be named and placed exactly where I want.

My final proposal

As said, you can find the code in http://ss3.gemstone.com/ss/MarianoBuilder and the license is MIT. I didn’t call it MarianoBuilder because of ego, but on purpose to make it clear that MarianoBuilder is just for me, it has my own preferences, my own settings, my own projects. What you should copy is the procedure. Copy the idea of using Metacello for something else than managing projects. Create your own ConfigurationOfYou and YourBuilder. Of course, you are very welcome to take a look to MarianoBuilder to get an idea of how to set certain preferences or settings. You are invited to take it as a base and fork it.

There are more things I would like to tell about this topic but the post is already too long so I will continue after…

My small talks at Smalltalks 2011

Smalltalks 2011 was, as usual, a very nice conference. I am glad I could attend even when living in France. This time I gave 2 talks as I mentioned in a previous post.

The talk “Building your own Pharo images with Metacello” was good or at least that’s what I think. Of course, it was a talk that only makes sense for certain people or scenarios. Moreover, it was a “pre-requirement” to know Metacello which n is something not all people know. The talk was in the big auditorium and there were quite a lot of people. The only thing I regret from the presentation is that I forgot to say that the code was MIT and that everybody could take it and adapt it to their own needs. Nevertheless, I added a slide with this information to the final version of the slides (which will be published in the website).

You can find the slides here but notice that most of the presentation was a live demo. You will need to wait for the video if you want to see it 😉  (it was recorded so it should be online soon). I like the topic of this talk so I will try to do a special post about it in the future.

The second talk was the presentation of my paper: “Problems and Challenges when Building an Unused Object Manager for Pharo”. It was in the secondary room so there were a few people (no more than 20). Besides, the talk was in parallel with a nice talk of Hernán Wilkinson. I hope that I least I could explain more or less what I am trying to do for my PhD. The slides of the talk are here.

That’s all folks 🙂

smalltalks2011Attendees + 1

Hi. I think I was pretty clear in this post when I described my feelings about ESUG Conference: “What is really great is to meet people. Have you ever sent 1 million emails to someone without even knowing his face?. Is he 70 years old? 20 ? What language does he speak?  Well, ESUG is the perfect conference to meet people by real, face to face. The best part of ESUG happens in the “corridors”, I mean, talking with people between talks, after the conference, in the social event, etc. There will be people who will ask you about your stuff, they will give you feedback and ideas. You will find yourself wanting to give feedback to others. It is a nice circle.”

Even though the previous paragraph describes pretty good what I think about ESUG, it can also be applied to Smalltalks Conference as well. This conference, which takes place in Argentina, has started in 2007 meaning this is the fifth edition. It’s true that the audience is less international than ESUG but that doesn’t mean that the conference is worst or not worth it. The conference is really international, people come from different countries and, in the last editions, all presentations were in English. The conference is free and the attendees are more or less between 150 and 300 people.

The conference is organized by FAST (Smalltalk Argentinian Foundation in Spanish) with several sponsors including ESUG. Since last year, there is not only a “technical track” of presentations, but also a “research track” (a workshop). This means there is a full program committee conformed by several international researchers. Papers are submitted and there is even a journal associated to the workshop. Believe me that for me, a PhD student and Smalltalker at the same time, this is awesome. Java fanatisim does not only exist in industry 😉

I have submitted a paper which was accepted so I am attending the conference. This is very good news for me because it means that I will be going to all existing Smalltalks conferences so far (from 2007 to 2011)! Well, at the same time I will be visiting my country so… 🙂  The paper is about my current development/investigation about managing unused objects. Soon, I will do a post and put the link to the paper.

Apart from the paper, I have also submitted a talk which was accepted as well: “Building your own Pharo images with Metacello”. It doesn’t make sense to said more because it is already explained in the conference website.

I would like to mention the effort that FAST and Smalltalkers are doing this year in order to attract new attendees to the conference. They are doing special “pre-Smalltalks talks” in 2 different universities where they give introduction courses to Smalltalk and the VM. It’s nice to see friends of mine taking care of that 🙂  So… if you want to improve your Smalltalks skills before Smalltalks or you want someone else to take the courses, plase visit this link: http://giti.org.ar/announcements/seminarsofsmalltalk-relatedsubjects. Of course, these courses are free as well.

There is also a Pharo sprint just before Smalltalks. It is on Wednesday 2nd at the Universidad Nacional de Quilmes (the same place where the conference takes place). If you have never attended a Pharo sprint, don’t miss this chance!! It’s the perfect place for everyone who loves Smalltalk from newbies to hackers. You can learn and pair program with really experienced guys of the Pharo board and make a real progress in an open-source project such as Pharo. The link is: http://www.fast.org.ar/smalltalks2011/events

Not enough reasons to come? Well, I may be biased but Argentina is a very nice country with a lot of beautiful places and really friendly people. So, why don’t you come to Smaltalks and take some holidays as well?

See you at ESUG?

You can notice that I’ve added a picture/link of ESUG on the right column of the blog. Even if I am young and I don’t have experience traveling arround the world, I can tell you that ESUG Conference, as well as Smalltalks Conference, are just AWESOME. I have assisted to all Smalltalks from 2007 to 2010 and ESUG 09 and 10. All of them are great and not only because of the talks. The talks are the less important for me 😉

What is really great is to meet people. Have you ever sent 1 million emails to someone without even knowing his face?. Is he 70 years old? 20 ? What language does he speak?  Well, ESUG is the perfect conference to meet people by real, face to face. The best part of ESUG happens in the “corridors”, I mean, talking with people between talks, after the conference, in the social event, etc. There will be people who will ask you about your stuff, they will give you feedback and ideas. You will find yourself wanting to give feedback to others. It is a nice circle.

And the Smalltalk community is unique in the sense that you can assist ESUG and meet the developers of the tools you use to have fun, make a living, study, do research, work, or whatever. You can take a beer with the developers/creators of Pharo, Seaside, AidaWeb, GemStone, Pier, Moose, Squeak/Pharo VM, DBXTalk, Metacello, Glorp, Cincom, VA, etc….. And what is also great about smalltalkers is that a smalltalker is likely to be a very good person. ESUG is friendly and smalltalkers come from all over the world. Come on, join us 🙂

This year ESUG Conference is in Edinburgh, Scotland. All the information you may need, such as list of talks, schedule, maps, venue, etc, is in http://www.esug.org/wiki/pier/Conferences/2011.

Now, regarding the “Journey through the Virtual Machine”, if you like that topic, you have to attend ESUG. There you can meet Igor Stasenko (author of HydraVM, NativeBoost, Hudson VM configurations, CMakeVMMaker, etc),  Esteban Lorenzano (the new maintainer of Mac Squeak VM and Cocoa port), Andres Valloud (VM developer at Cincom), Javier Burroni and Gerardo Richarte (authors of SqueakNOS and the JIT/GC implemented in Smalltalk), etc. Even more, there is a special workshop for you: “Compiling your own VM” by Igor. This is a hand-on tutorial. So, what are you waiting for? In addition, there are even more talks related to VM. Check the schedule for more details.

I will give some talks as well, but I will comment that in another post.

So…see you there?