“New” Tanker – current status

Both ESUG 2012 and GSoC have finished. At ESUG I gave a presentation about Fuel/Tanker (sorry, no slides because it was all demo). I also presented Tanker in the ESUG Awards….we got the 4th place, and only 8 points of difference with the 3er place ūüôā ¬† Anyway…I wanted to make public what exactly we finally managed to do with Tanker, a package exporter and importer that uses Fuel serializer. As I said in a previous post, we have changed a lot how Tanker works internally. In that post also I mentioned that Tanker didn’t support source code export/import, but in this new version, it does support methods source code, class comments, timestamps and everything that is currently being stored in the .changes or .sources file.

Exporting packages

The following figure shows how the export works. The input is a package of code (classes + extension methods), in this example “MyPackage”. The first step is to traverse the classes and methods of the package and do 2 things: a) write the source code (source of methods, class comments, timestamps, etc) to a text-based chunk format sources file (MyPackage.tankst); b) create “definition” objects. These objects represent a kind of object model that store all the necessary information of the code entities to be able to recreate them back ¬†in another image. So we have for example, TAClass, TATrait, TAMethod, TABinding, TAAdditionalMethodState, TATraitAlias, TATraitComposition, etc…

Something key here is that besides all that information, we also store the relative position of the source code of the¬†entity¬†in the “MyPackage.tankst”. So for example, TAMethod instances will have a number representing that relative offset (similar to the pointer that CompiledMethods have in their trailer to .sources/.changes) where its source code is.

Another important detail is that in TAMethod we are storing the bytecodes, because even if we have the source code we want to avoid needing to compile during the import.

Once we have finished writing the sources file and creating the definitions, we just serialize them into a Fuel binary, say “MyPackage.tank”.

Importing packages

The following diagram shows how the import works. The input is the sources file (MyPackage.tankst) and the Fuel binary file (MyPackage.tank). The first step is basically to read the sources file and append all its contents in the .changes file (so that to get the same behavior as if we were installing a package with Monticello). But before doing that, we temporally store/keep the current “end” position of the .changes file ūüėČ ¬†(you already guessed why?).

The second step is to materialize the definitions using Fuel. Something very nice is that the definitions that were serialized, also understand the necessary messages to be installed in the system (#installUsing: aSourcesFile environment: anEnvironment). So we basically tell each class and each extension method to get installed in the system. Those objects will delegate to the rest of the definitions to complete their task (all definitions know what to do when installing them in the system).

Classes and traits will use the class builder to recreate¬†themselves¬†back. The source code was already installed in the .changes, so now we need to fix the “pointers” from our newly created classes/methods to point to the correct place of their source in the .changes file. And this is very easy because their position is just the “end” position of the .changes before installing our package (that value I told you we were temporally storing) + the relative position that was stored in the definition itself ūüôā

So what is the biggest difference with Monticello for example? The key point is that we export bytecodes and we avoid having to compile during import. This make things faster while also being able to install packages without compiler (very useful for bootstrapping for example).

How to install it and use it

Tanker only works in Pharo 2.0 (because we rely on new class builder and layouts) so you first need to grab an image. Then, you can evaluate:

Gofer it
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) perform: #loadDevelopment.
Yes, so far there isn’t a stable version yet, we are waiting for new class builder.¬†Be careful because that will install the new class builder (and change the system to use it) ¬†and it may have some bugs…so take care ūüôā
The export and import look like:
aPackage := TAPackage behaviors: {MyClass. MySecondClass} extensionMethods: {String>>#myExtensionMethod1. Object>>#myExtensionMethod2}.
(TAExport
 package: aPackage
 binariesOn: aBinaryWriteStream
 sourcesOn: aTextSourcesWriteStream)
 run.

(TAImport
 binariesFrom: aBinaryReadStream
 sourcesFrom: aSourcesReadStream)
 run.

However, we do not expect the final user to provide all the list of classes and extension methods. Therefore, we have helper methods to export and import RPackages and PackageInfos. Example:

 TAExport exportRPackageNamed: 'MyProjectCore'.
 TAExport exportRPackageNamed: 'MyProjectTests'.

TAImport importPackageNamed: 'MyProjectCore'.
 TAImport importPackageNamed: 'MyProjectTests'.

Conclusion

So that was all for today. Probably, I will do another blog post where I will show how we can query Metacello to know which packages to export and in which order, some details about the new class builder, some benchmarks while exporting all seaside/pier/magritte (10 seconds to export and 20 to import), and so on. The conclusion we got with this project is that indeed Fuel can be successfully used in yet another completely different domain. If you want to help us, please test it with your own packages. Right now we have only one open issue (need to recompile if superclasses present in the image have reshaped). I guess that soon we will fix this last issue and release the first stable version. In the future we plan to analyze how can we really integrate Tanker with Monticello/Metacello.

Advertisements

ESUG 2012: when the time to pay for the beers has arrived!

Hi guys. In a blog post of last year, I said:

“What is really great is to meet people. Have you ever sent 1 million emails to someone without even knowing his face?. Is he 70 years old? 20 ? What language does he speak?¬† Well, ESUG is the perfect conference to meet people by real, face to face. The best part of ESUG happens in the ‚Äúcorridors‚ÄĚ, I mean, talking with people between talks, after the conference, in the social event, etc. There will be people who will ask you about your stuff, they will give you feedback and ideas. You will find yourself wanting to give feedback to others. It is a nice circle.”

This year, we should have another motivation. How many times have you promised people to buy them a beer if they did XXX? How many times people have promised you a beer for doing YYY? What better place than Belgium to balance the accounts?

I have already done my homework. I own beers to: Alain Plantec and  Henrik Sperre Johansen.

Anyway guys, hope to see you there!


Tanker screencast and image ready for testing

Hi guys. Last days, we submitted Tanker to the ESUG Innovation Technology Awards. As part of that submission, I have created a screencast and an image with the examples.

The screencast gives you an introduction to Tanker and shows you how to export and import packages.¬†I starts with a simple example, then it exports a real library and ends up exporting all Seaside, Magritte and Pier ūüôā ¬† Ahhh yes, sorry for my voice, I know it is ugly hahaha. You can also watch it here:

You can also get the image I used for the screencast, which you can try and experiment.

Have fun,


Tanker: transporting packages with Fuel

Hi all. You may have noticed that Tanker is starting to appear in some mails or in the Pharo issue tracker. Tanker is a project that Martin and I have been developing for a while and we are going to submit it this year to ESUG Innovation Technology Award. Therefore, I thought it would be interesting to explaining what it is, its current status, its goals, etc.

What is “Tanker” and what was “FuelPackageLoader”?

Right now the common way to export and import packages in Pharo is by using Monticello (or doing fileOut, which is almost the same). This ends up exporting the source code and then compiling it during the import.¬†Tanker is a tool to export and import packages of code in a binary way using¬†Fuel serializer. Using Fuel enables us to¬†avoid having to compile from sources during the import. Tanker¬†understands¬†the concept of “packages of code” and the correct integration of them into the system. For example, it initializes classes, sends notifications, etc.

Tanker was first a prototype called “FuelPackageLoader” which was what I used for the¬†example of exporting and importing seaside packages.¬†In the last months, we have renamed the project to “Tanker”. Why? Because we do not want people to think that it is a Fuel project. In fact, Tanker is a simple USER of Fuel. Just as any other code that uses Fuel. This is why we have also moved it to its own repository.

Fuel has a package called “FuelMetalevel”.¬†This package gives Fuel the knowledge of how to correctly serialize and materialize classes, metaclasses, traits, method dictionaries, compiled methods and closures, in other words, all the entities related to code and runtime infrastructure. It only knows how to serialize and materialize correctly. Nothing else. It does not initialize classes, it does not notify the system about the materialized classes, it does not install classes in Smalltalk globals, etc.

Current features, design and missing things

Right now, Tanker provides the following features:

  • It is able to export a package to a .tank file and import it in another image. The input for the export is a TAPackage which basically contains a list of classes and a list of extension methods. We are¬†completely¬†decoupled from the “package representation” (PackageInfo, RPackage, MCPackage, etc). However, we provide an API if you want to directly export from those types of packages.
  • Classes are initialized and installed in Smalltalk globals, events are sent, etc.
  • It has the ability to add additional user-defined objects to the package being exported (this is used, for example, for the Pharo generation from a kernel to store large/heavy class variables, tables and fonts).
  • It supports pre and post load actions represented as closures.

From the design point of view, Tanker:

  • Fully serializes classes and traits (not its “definitions”)
  • Does not use the ClassBuilder during materialization. Tanker itself materializes the “class objects” and sets the data.

So far, we are missing:

  • The possibility to export source code (right now classes and methods do not have source code) and also to install it in the .changes file during import.
  • Some validations during import. For example, the superclass of a class being installed may have changed its shape and, therefore, the classes to install need recompilation (because the bytecodes accessing instVars offset may be shifted or wrong). Or if a class already exists in the image and the shape has changed, we need to update the existing instances.
  • Integration with other tools like Monticello and Metacello.

Results with the “current status”

There are so far 3 real examples of Tanker:

How to install it and use it

Tanker will work only in the bleeding edge of Pharo 2.0. So I first recommend you to get an image from Jenkins. Then, you can install Tanker this way:

Gofer it
 url: 'http://smalltalkhub.com/mc/marianopeck/Tanker/main';
 package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) load.

To export a package and provide yourself the classes and extension methods, you can:

| aPackage aStream |
"Export"
aStream := 'demo.tank' asFileReference writeStream binary.
aPackage := TAPackage behaviors: {TestCase. TestLocalVariable. } extensionMethods: #().
TAPackageStore new storePackage: aPackage on: aStream.

"Import"
aStream := 'demo.tank' asFileReference readStream binary.
TAPackageLoader new loadFrom: aStream.

Then you can also use the API that provides helper methods to RPackage and PackageInfo:

aPackage := TAPackage fromPackageInfoNamed: 'MyPackage'

You also have #fromPackagesInfoNames:, #fromRPackageNamed: and #fromRPackagesNames:. Of course, there are more use-cases, API and scenarios. But, so far, that is the simplest usage. For more examples, browse the class side methods of TankerExamples.

GSoC and “new status”

The results so far are quite promising and not anymore a “proof of concept”. However, we still need to support source code management as well as the already mentioned pending features. Because of this reason, Martin submitted Tanker for the GSoC and¬†fortunately¬†it was accepted. So, right now we are moving to a different design to solve the requirements. ¬†The idea now is NOT to serialize classes and traits, but instead serialize their “definition”. Imagine by “definition” the string used to create them. Then, during import, instead of just materializing class objects, we take the definition and, using the ClassBuilder or similar, we “evaluate” the definitions and we get the new classes.

At the same time, the idea is to export the source code of a package in a file (myPackage.tank.st or something like that) and the binary representation in another file (say myPackage.tank). Then, during import, you should be able to import with or without sources.

Side-effect projects of Tanker

You may be wondering why we didn’t start from the very beginning with the “definitions” way. Well, to be honest, the ClassBuilder is a mess, difficult to understand,¬†maintain¬†and extend. It was really hard trying to use it for our purpose. So the first “side-effect project” of Tanker is to continue pushing the “new ClassBuilder” started by Toon Verwaest¬†based in “slots”. Martin Dias, Guillermo Polito and Camillo Bruni are pushing it and writing tests. I think it could be soon integrated in Pharo and replace the old one. The idea is that Tanker will use this ClassBuilder, for example, to evaluate the definitions.

When we are importing a class, it may happen that the superclass (present in the image where we are importing) has changed its shape (added or removed instVars, change supperclass, etc). If this is true, we have to recompile because the bytecodes accessing intsVars will be a shifted offset. However, recompiling is slow and we don’t want that. Therefore, Tanker will use IR (intermediate representation) which was developed by Marcus Denker and the team working with the new Opal compiler. IR is just a nice model generated from a CompiledMethod. The idea is that we can generate the IR, modify it (bytecodes for instVars accessing, for example) using this nice abstraction and API and then generate back a new CompiledMethod. This is way faster than recompiling. Furthermore, IR is decoupled from Opal so we don’t need whole Opal.

Conclusion

Tanker started as an experiment to see whether Fuel coud be used to export and import packages in a binary way. The proof of¬†concept¬†was quite good so we are now going forward with the source code management and related stuff. It is important to notice that Tanker just “uses” Fuel. Fuel is¬†completely¬†decoupled from Tanker. We think Fuel was well received by the community.¬†We are doing our best so that Tanker gets positive feedback as well.


Reviving CI test failures in local machine

The problem

These days, most serious software developments include a Continuous Integration server which runs tests. A problem appears when tests fail in the server but they do not fail locally. There can be differences in the used operating system, virtual machine, configuration, etc. Let’s take as an example the Jenkins server of Pharo. We use such server to not only build and test the Pharo images but also the VMs. There are 3 slaves (one for each: Windows, Linux and MacOSX) and tests are run in all of them. Still, it is common to have tests that we cannot reproduce locally. Why?

  • Random failures: tests that fail randomly. Of course, we would prefer not having these tests but sometimes we do.
  • Tests that fail because as a side effect of other tests.
  • The OS of the server or even its configuration/infrastructure is different.
  • The used virtual machine can be different (for example, Jenkins uses the VM it builds to test the other jobs).

What do we have now?

So, we have a failure in the server that we cannot reproduce locally. How can we understand what happened? So far, the only thing we have is a piece of a text-based stack trace. For example, let’s take this test failure:

Error Message
Assertion failed
Stacktrace
SocketStreamTest(TestCase)>>signalFailure:
SocketStreamTest(TestCase)>>assert:
SocketStreamTest(TestCase)>>should:raise:
SocketStreamTest>>testUpToAfterCloseSignaling
SocketStreamTest(TestCase)>>performTest

As you can see, this is not that helpful and you may still don’t know what ¬†has happened. Something really useful would be to at least know what where the values of the instance variables involved in that stack… Here is where¬†Camillo Bruni had a nice idea ūüôā

Fuelizing test failures

In Pharo, the stack of the running system is reified also from the language side and we can access them! (we can even modify them). We have instances of MethodContext which hold an instVar ‘sender’ that refers to the next sender in the stack (another MethodContext or nil if it is the last). Apart from ‘sender’, a context also includes the receiver, the method that caused its activation, the arguments and the temporal variables. The¬†Fuel serializer can serialize any type of object including MethodContext. If we can serialize a MethodContext (and closures and methods), we can serialize a stack, right?¬†And what does this mean? Well, it means that we can serialize a debugger with its current state.¬†I have already shown several times (at ESUG Innovation Technology Award and at PharoConf) how we can use Fuel to serialize a debugger (from image X)¬†in the middle of its execution and materialize it in image Y and continue debugging.

Pharo provides ‘exception’ objects and, at the end, test failures are exceptions (TestFailure). We can always ask its “signaler context” to an exception, in other words, the MethodContext that signals it. Once we have¬†that MethodContext, we have all the stack (because that object has a¬†sender and the sender context has a sender and ….). So, how do we serialize that?

context := testFailure signalerContext.
FLSerializer newFull
 serialize: context
 toFileNamed: 'context.fuel'.

So that piece of code will serialize all the stack of contexts including all the transitive closure: receiver, arguments, temporal variables, etc.

Reviving test failures

So we have serialized our test failure on a file. Now we want to revive them in our local machine. The first obvious thing is to materialize the original stack from the file. But then, what do we do with the stack? How can we do something useful with it? Well, Pharo allows us to open a debugger for a particular stack ūüôā . This means we can just open a debugger with the stack of the test failure! To do that:

| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'context.fuel'.
Debugger
 openContext: aContext
 label: 'This is the new debugger!'
 contents: nil

And that opens our nice debugger. Much better than a text-based stack trace, isn’t it?

Caveats when serializing a stack

When you serialize the whole stack, you may find some problems:
  1. The object graph that you serialize and, therefore, the resulting stream size can be really large depending on what the contexts have. Sometimes a context end up in the UI so you end up serializing lots of morphs, colors,¬†forms, etc. If everything is fine, the file should be¬†a couple hundred¬†or thousands KB. If the file ¬†size is in MB…then you may be serializing too much.
  2. Not only the graph is too big, but it also incorporates objects that CHANGE while being serialized (mostly when these are objects from the UI). This will cause Fuel to throw an error saying the graph has changed during serialization.
  3. If 2) happens, then depending where you trigger the fuel serialization, you may end up in a loop. For example, say you want to serialize each error with Fuel. So you change ¬†SmalltalkImage>>logError:inContext: ¬†to write the context with Fuel. Now, if 2) happens and Fuel throws an error, you will try to log that again causing again the serialization… infinitive loop.
  4. Apart from the previous points, there are still more problems. You can read the title “Limitation and known problems” in this post.
So… some workarounds are (still, not sure if they will help in all cases):
  • Deep copy the context before serializing it.
  • If you want to serialize particular contexts (for example, particular domain exceptions), then you may know WHERE to hook to make some instVars transient and, therefore, avoid serializing things you don’t want and that may cause 2).
  • Serialize a PART of the stack.

Jenkins integration

Thanks to Camillo and to Sean P. DeNigris, now Jenkins serializes (for some jobs) each test failure into a file (here¬†you can see how to set up your own Jenkins for Pharo). For example, we have the job “pharo-2.0-tests”. If you select the OS and then a particular build number, you will have an artifact called “Pharo-2.0-AfterRunningTests.zip”. For example, this one:¬†https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip. This zip contains all the .fuel files of all the test failures. Each file is named ClassXXX-testYYY.fuel.

To workaround the problem mentioned in the previous paragraphs (“Caveats when serializing a stack”), we just serialize a part of the stack: from the context that signals the failure up to the test method. Example:

  ...
  performTest
"Start context slice"
> testMyFeatureBla
> ...
> ...
> assert: foo equals: bar
"end context slice"
  assert:
  Exception signal

The idea is to serialize the least number of stack-frames possible while still giving decent debug feedback. To do that, our Jenkins code (HDTestReport>>serializeError: error of: aTestCase) is:

serializeError: error of: aTestCase
 "We got an error from a test, let's serialize it so we can properly debug it later on..."
 | context testCaseMethodContext |

 context := error signalerContext.
 testCaseMethodContext := context findContextSuchThat: [ :ctx|
 ctx receiver == aTestCase and: [ ctx methodSelector == #performTest ]].
 context := context copyTo: testCaseMethodContext.

 [
 FLSerializer newFull
 " use the sender context, generally the current context is not interesting"
 serialize: context sender
 toFileNamed: aTestCase class name asString,'-', aTestCase selector, '.fuel'.
 ] on: Error do: [:err| "simply continue..." ]
During serialization the graph can somehow reach classes of the Jenkins code (like HDTestReport). If you materialize in an image where such class is not present, you will have a Fuel error. For this purpose in the same Pharo-2.0-AfterRunningTests.zip besides having the .fuel files, we also have a Pharo-2.0-AfterRunningTests.image which, as it names says, was saved after having run all tests (meaning it has the Jenkins code installed). This means we can directly use that image to materialize and it will work. The other option is to take another image and install the following before materializing:
Gofer new
 url: 'http://ss3.gemstone.com/ss/CISupport';
 package: 'HudsonBuildTools20';
 load.
This is temporal because soon Jenkins support code will be directly integrated in Pharo.
Anyway, I recommend using the same version of image that was used during serialization. So I think that using directly Pharo-2.0-AfterRunningTests.image is more reliable.

Conclusion

It is clear that there are several caveats. However, I do believe this is yet another step in CI and¬†development. It is just one more tool you have at hand when something is failing in the server and you cannot reproduce it locally. It the worst case, it won’t help but it won’t hurt either. If you have luck, you may find out the cause ūüôā¬†It is incredible all the things you can do when the stack is¬†reified and¬†visible¬†from the language while also being¬†serializable. For me, asking for a text-based stack trace in Smalltalk is like going to a cabaret and ask for a hug. We have so much power that we should take advantage of it.¬†At the end, using a debugger is way better. Anyway, I do not recommend to remove the stack trace information, just adding also the Fuel possibility.

My (past) presentations at PharoConf and (future) talk at ESUG 2012

Hi. As usual, I wanted to share with you the slides of my last talks in case you are interested.

PharoConf

Last month I went to the first PharoConf held in Lille, France, and I gave to talks. One was about using the Fuel serializer for several different hacky things ūüôā¬† You can find the slides here but since most of the presentation was a demo, they are almost useless. The videos of the conference are being processed and will be updated soon. I will update this post once they are finished.

The other talk I gave was about building the Pharo Virtual Machine and you can find the slides here. If you are interesting in the topic, you can see all the blog posts I have written about it.

ESUG 2012

Once again, I will be attending and presenting at ESUG (in Ghent, Belgium). This year I will present something similar to the Fuel talk at PharoConf. As you can see in the ESUG schedule, the abstract of my talk says:

Fuel is an open-source general-purpose object serialization framework developed in Pharo. It is fast, extensible and has an object-oriented design. It can serialize not only plain objects, but also closures, contexts, methods, classes, traits, among others.
This presentation will be mostly a demo with only a few slides. I will show the power of Fuel by using it in several scenarios: rebuilding Pharo from a kernel image, exporting/importing Monticello packages, moving a debugger from one image to another one, persisting (and also import/export) Pier kernels, etc.

So…see you at Ghent?


Smalltalk syntax highlighting for Latex

Hi. For papers I usually write code in black and white. Even when you may want colors in books or thesis, you still may not want to add colors. However, for my thesis (I have to finish it beginning of Augoust!!), I do want to make the code a little bit nicer and so I did some Smalltalk syntax highlighting with the listings package. The listings package provides lots of languages out of the box, but not Smalltalk. Because of that, I searched ways to have it. Not sure if this would be helpful for someone else, but just in case, I wanted to share it with you.
Basically, I can detect (and therefore, choose a color/style): globals/classes (start with uppercase), keywords, comments, strings, symbols and numbers.  At the end of the post, I paste all the latex code so that you can take the code.

Examples:

This is a plain example:

\begin{code}
fuelExample
| sourceArray loadedArray |
sourceArray := Array
with: 'a string'
with: Transcript
with: 42
with: #foo.
"Store to a file"
FLSerializer serialize: sourceArray toFileNamed: 'example.FL'.
"Load from the file"
loadedArray := FLMaterializer materializeFromFileNamed: 'example.FL'.
"Check that the materialized Transcript is the right singleton instance."
loadedArray second show: loadedArray first; flush.
self validateMaterializedGraph: loadedArray.
\end{code}

And the pdf looks like this:

This is how you can define pieces of bold, italics and underline:

\begin{code}
<b>fuelExample</b>
| sourceArray loadedArray |
sourceArray := Array
with: 'a string'
with: Transcript
with: 42
with: #foo.
"Store to a file"
<i>FLSerializer serialize: sourceArray toFileNamed: 'example.FL'.</i>
"Load from the file"
<u>loadedArray := FLMaterializer materializeFromFileNamed: 'example.FL'.</u>
"Check that the materialized Transcript is the right singleton instance."
loadedArray second show: loadedArray first; flush.
self validateMaterializedGraph: loadedArray.
\end{code}

The resulting pdf looks like this:
This is another environment if you want to show number lines:

\begin{codeWithLineNumbers}
<b>fuelExample</b>
| sourceArray loadedArray |
sourceArray := Array
with: 'a string'
with: Transcript
with: 42
with: #foo.
"Store to a file"
<i>FLSerializer serialize: sourceArray toFileNamed: 'example.FL'.</i>
"Load from the file"
<u>loadedArray := FLMaterializer materializeFromFileNamed: 'example.FL'.</u>
"Check that the materialized Transcript is the right singleton instance."
loadedArray second show: loadedArray first; flush.
self validateMaterializedGraph: loadedArray.
\end{codeWithLineNumbers}

And this is the output:

Customization

As you can see at the end, in the latex code, you are able to customize almost everything, the fonts, the fonts sizes, the colors and styles, etc. I won’t go in detail to show you how to customize everything. If you are interested in this post, you probably know latex and therefore you will know what and where to change.

Wanna improve it?

Disclaimer: I am *not* an expert in latex at all, in fact, quite a newbie. To do what I showed in this post, I needed the help of Lukas Renggli and Luc Frabresse, and also surf a lot of forums. If you improve this, please share it with me!! Moreover, I wanted to share this work with that they did in Pharo By Example. But so far I didn’t have time. If you do it, please share it with me!!

For simplicity, I have zipped together the example used in this blog and a .tex with the listings macros.

Code

</pre>
% Source Code
\usepackage{color}
\usepackage{textcomp}
\usepackage{listings}

\definecolor{source}{gray}{0.85}% my comment style
\newcommand{\myCommentStyle}[1]{{\footnotesize\sffamily\color{gray!100!white} #1}}
%\newcommand{\myCommentStyle}[1]{{\footnotesize\sffamily\color{black!100!white} #1}}

% my string style
\newcommand{\myStringStyle}[1]{{\footnotesize\sffamily\color{violet!100!black} #1}}
%\newcommand{\myStringStyle}[1]{{\footnotesize\sffamily\color{black!100!black} #1}}

% my symbol style
\newcommand{\mySymbolStyle}[1]{{\footnotesize\sffamily\color{violet!100!black} #1}}
%\newcommand{\mySymbolStyle}[1]{{\footnotesize\sffamily\color{black!100!black} #1}}

% my keyword style
\newcommand{\myKeywordStyle}[1]{{\footnotesize\sffamily\color{green!70!black} #1}}
%\newcommand{\myKeywordStyle}[1]{{\footnotesize\sffamily\color{black!70!black} #1}}

% my global style
\newcommand{\myGlobalStyle}[1]{{\footnotesize\sffamily\color{blue!100!black} #1}}
%\newcommand{\myGlobalStyle}[1]{{\footnotesize\sffamily\color{black!100!black} #1}}

% my number style
\newcommand{\myNumberStyle}[1]{{\footnotesize\sffamily\color{brown!100!black} #1}}
%\newcommand{\myNumberStyle}[1]{{\footnotesize\sffamily\color{black!100!black} #1}}

\lstset{
language={},
% characters
tabsize=3,
escapechar={!},
keepspaces=true,
breaklines=true,
alsoletter={\#},
literate={\$}{{{\$}}}1,
breakautoindent=true,
columns=fullflexible,
showstringspaces=false,
% background
frame=single,
aboveskip=1em, % automatic space before
framerule=0pt,
basicstyle=\footnotesize\sffamily\color{black},
keywordstyle=\myKeywordStyle,% keyword style
commentstyle=\myCommentStyle,% comment style
frame=single,%
backgroundcolor=\color{source},
% numbering
stepnumber=1,
numbersep=10pt,
numberstyle=\tiny,
numberfirstline=true,
% caption
captionpos=b,
% formatting (html)
moredelim=[is][\bfseries]{<b>}{</b>},
moredelim=[is][\textit]{<i>}{</i>},
moredelim=[is][\underbar]{<u>}{</u>},
moredelim=[is][\color{red}\uwave]{<wave>}{</wave>},
moredelim=[is][\color{red}\sout]{<del>}{</del>},
moredelim=[is][\color{blue}\underbar]{<ins>}{</ins>},
% smalltalk stuff
morecomment=[s][\myCommentStyle]{"}{"},
% morecomment=[s][\myvs]{|}{|},
morestring=[b][\myStringStyle]',
moredelim=[is][]{<sel>}{</sel>},
moredelim=[is][]{<rcv>}{</rcv>},
moredelim=[is][\itshape]{<symb>}{</symb>},
moredelim=[is][\scshape]{<class>}{</class>},
morekeywords={true,false,nil,self,super,thisContext},
identifierstyle=\idstyle,
}

\makeatletter
\newcommand*\idstyle[1]{%
\expandafter\id@style\the\lst@token{#1}\relax%
}
\def\id@style#1#2\relax{%
\ifnum\pdfstrcmp{#1}{\#}=0%
% this is a symbol
\mySymbolStyle{\the\lst@token}%
\else%
\edef\tempa{\uccode`#1}%
\edef\tempb{`#1}%
\ifnum\tempa=\tempb%
% this is a global
\myGlobalStyle{\the\lst@token}%
\else%
\the\lst@token%
\fi%
\fi%
}
\makeatother
%\newcommand{\ct}{\lstinline[backgroundcolor=\color{white}]}
%\newcommand{\needlines}[1]{\Needspace{#1\baselineskip}}
\newcommand{\lct}{\texttt}

\lstnewenvironment{code}{%
 \lstset{%
 % frame=lines,
 frame=single,
 framerule=0pt,
 mathescape=false
 }%
 \noindent%
 \minipage{\linewidth}%
}{%
 \endminipage%
}%
\lstnewenvironment{codeWithLineNumbers}{%
 \lstset{%
 % frame=lines,
 frame=single,
 framerule=0pt,
 mathescape=false,
 numbers=left
 }%
 \noindent%
 \minipage{\linewidth}%
}{%
 \endminipage%
}%

&nbsp;

\newenvironment{codeNonSmalltalk}
{\begin{alltt}\sffamily}
{\end{alltt}\normalsize}
<pre>

Pharo: Tips and Tricks

Hi guys. It is going to be 4 years since I have started to develop with Squeak/Pharo. In the last months, talking with people or by reading the mailing list, I noticed that I may know some tricks and tips of Pharo that not everybody know. If you are an advanced Pharaoer, you probably won’t learn anything new but, if you are a beginner or so, this post may help you.

I have structured this post so it can be easy to follow. There are different categories and, for each of them, there are several tricks. Each trick has a detailed name so you can quickly scan them and only read those you are interested in.

UPDATE: James Robertson has started creating videos for some of the tips I mention here. So for those that there are already videos, I will paste the link. Thanks James!

Searching

“Method source with it”: Have you ever needed to search a string in EVERY place of an image? I mean… in methods’ source code, class comments, etc. I need this all the time. To do this in Pharo, type what you want to search, select that text -> right-click -> extended search -> method source with it.¬† Notice that you can type whatever you want. For example, you can search ‘text’ (with the quotes included) or #whatever. Whatever you type will be literally searched. You can also see the other options under “Extended search” because they are also useful. Video here.

Spotlight: This is a new tool in Pharo 2.0 similar to Mac’s spotlight. For those of you who miss Algernon (like me), this is what you are looking for. Stop writing classes/method in code panes, workspaces or whatever just to browse them. Typing in code panes sucks because then you have to remove it or discard it. Opening a workspace takes too much time and then you end up full of windows. So… to try it, just SHIFT+ENTER and the input will appear in the right top corner of the image. The text inputs work exactly as the code completion (so it works for classes and methods). It has the same behavior as if you were typing in a workspace. Once you have selected something from the list, just enter to open a browser with it. Video here.

Finder examples: Sometimes I have some receiver object and I know what answer I want to get but I don’t remember which is the message. For example, last week I wanted to know which was the method to remove the prefix from a filename. So, open the Finder and, in the list of options, select “Examples”. Then, in the input pane, write ‘whatever.fuel’.’whatever’ and hit search. Wow! Pharo told me the method I was looking for is #withoutPeriodSuffix. For more details about this tool, print ‘FinderUI methodFinderExplanation’. AWESOME, isn’t it? Now, being serious, the idea is nice but the implementation…mmmm. Just check method MethodFinder class >> #initialize2. Of course you cannot try all selectors of the image because there can be lots of bad problems. However, having all the selectors hardcoded in a method is not nice either. Video here.

Settings

How to script them? It usually happens that you need to set your own settings from code (and not from the Settings Browser). For example, when setting them with StartupLoader. But how do you know for each setting how to do it by code? Easy. Open Setting Browser -> go to the setting you want -> right-click -> Browse. Or, you can just use the shortcut of browsing (CMD+B in Mac). This will open the method that declares the setting and has the pragma. In such method, you can search the setting you were looking for and you will find the selector and class that set/get the value of the setting. Video here.

StartupLoader: Wanna run scripts from startup with Pharo? See my previous post!

Changing logo, desktop, theme and friends: Did you know you can change all of them? You can build your own images that look completely different from the default one. Example:

PolymorphSystemSettings setDesktopLogoWith:  (ImageMorph withForm: (ImageReadWriter formFromFileNamed: '/Users/mariano/PhD/Marea/Fuel/fuel kit logos/logo-fuel-header-title.png')).
PolymorphSystemSettings desktopImageFileName: '/Users/mariano/PhD/Marea/Fuel/fotos/1997_Need_For_Speed_2_Special_Edition.jpg'.
UIThemeW2K beCurrent.

Gives this ugly image:

System fonts: Sometimes you don’t like the fonts that come out of the box with Pharo. In other cases, if you have non UTF characters, then you may be forced to use a different font. To do that, you have to use “Free Type”. Go to the Settings Browser -> type “font” in the search -> check the item “Use Free type…”. Now, you can click on each of the standard fonts and a pop-up will appear and show you all the fonts available from your OS. Notice that if you make the code font too big, bringing up tools will slow to a crawl. Anyway, here is a crazy example:

Facing problems

Copy the stacktrace from a debugger to clipboard: There are lots of time when I just want to paste the stacktrace of a debugger to clipboard. One way is to go and see the PharoDebug.log. Problem is that since I use new images all the time, I always need to print “Smalltalk imagePath”, then go there and open the file. So… if you are in a debugger, you can do right-click -> Copy to clipboard and that’s all ūüôā¬† Video here.

Interrupt key: Sometimes your image goes into an infinitive loop or something similar and you want to interrupt what it is doing. To do that, you can press the interrupt shortcut and it will try to interrupt the processes and open a debugger. For more details, read the Pharo book.

Send kill signal: If your image “hangs” and you are lucky, the interruption key will help you. If you are not, then you have a different way to know what was happening. You can KILL the VM by sending a special signal and the VM will print the Smalltalk stacktrace of the current processes in the OS console (stdout) or in a .dump file (located in the image path). This is a good last resort when nothing else is working. For this trick, you have to be using latest Cog VM since it is quite new I think. Example:

kill -s SIGUSR1 4943

Where 4943 is the PID of the VM you want to kill.

Recovering lost code: Probably, one of the things Smalltalkers are always proud of saying is “In Smalltalk you never lose code” hehhehe. So… your vm crashed and you didn’t save your code? Well, there are different ways of recovering the code.

1) Tools -> Recover lost changes. There you can select the date from where to start watching the changes. Once you select a point in time, you can go item by item seeing what you want to recover. What I usually do is multi-select all the ones I want to recover and then I click on “file in selections”. That will install back those lost changes. Notice that this is thanks to the .changes file so you need it in order to recover changes. Also notice that even Do-Its are saved.

2) If your image does not even start, are you screwed up? No! You can even browse the changes and recover your lost data in ANOTHER image. Just open another image and then drag and drop the .changes of the image you lost the data. Then repeat the option 1).

3) You can always take your prefer text editor, open the .changes file and copy paste ūüėČ

Video here.

System reporter: This is a small tool that gives you information about the image, VM and OS you are running. This is also very useful when you want to report an issue. To access it, System -> System Reporter. Video here.

Monticello

History to view changes: I don’t feel proud of discovering this functionality just quite recently hahahah. In several opportunities, I load version XXX (or nothing) of a package and I want to see the differences between YYY and ZZZ. I knew the “Changes” option from Monticello but that compares a version of the repository to your installed version. So, I always needed to load the oldest version of the one I wanted to compare (YYY in this case) and then select ZZZ and do the “Changes”. That sucks because of the time it takes but also because I need to load a different version (maybe I have changes in local version). So…. to do this, open a repository, select the version you want (ZZZ in this case), press the button “History” and, in the pop-up, choose which version you wanna compare to. Then, right-click -> view changes -> MyPackage.mariano.42. Video here.

Global package-cache: The package-cache is created by default as a subdirectory where the image is. For me, this is totally useless since the package-cache should be global, that is, shared between all your images. Therefore, I recommend you to change the directory of the package-cache. You can even do it in your startup preferences as I do:

| sharedPackageCacheDirectory |
sharedPackageCacheDirectory := (FileDirectory on: '/Users/mariano/Pharo/localRepo/')
assureExistence;
yourself.
MCCacheRepository default directory: sharedPackageCacheDirectory.

Notice that depending on the amunt of mcz files, you may notice a slowdown when using Monticello Browser. A couple of things have been done in Pharo 2 regarding this problem but it is not completly solved yet.

But you don’t want to start with an empty package-cache when you already have lots of .mcz around your machine, do you? So, you can run this small script:

#!/bin/bash
find /Users/mariano/ -name "*.mcz" -print0 | xargs -0 -I {} cp {} /Users/mariano/Pharo/localRepo/

Video here.

Setting your username and password: If you have a project which does not allow global commit access and you use Metacello, then you will understand my problem. You downloaded your project, you change something and, when you are going to commit, you get a nice error saying that you don’t have access grrr. Then you open the Monticello repo description and you notice that this is obvious because you didn’t put your username and password. Metacello allows us to put that in the ConfigurationOfYourProject but you may not want to put it there if this is public. Therefore, I did a script (also attached to my startup preferences) that fixes this for me. It is a hack but here it is:

(MCRepositoryGroup default  repositories
select: [:each | (each isKindOf: MCHttpRepository)
and: [((each locationWithTrailingSlash includesSubString: 'www.squeaksource.com')
or: [each locationWithTrailingSlash includesSubString: 'http://ss3.gemstone.com/ss/'])]
]) do: [:each |
each
user: 'MMP';
password: ((FileDirectory default oldFileNamed: '/Users/mariano/Pharo/repositoriesPassword.txt') contents).
].

I guess the code is self-explanatory.

Random useful options: There are a couple of useful operations with Monticello. To revert your local version to another one, right button on a package -> revert package -> choose version. To copy a version of a package from one repo to another one, select the version -> button copy -> select target repo. Do you already have a repository and you want to add it to another different package? Adding again the repo is boring… right-click the repo -> add to package -> select package.

Miscellaneous

There is a top-right arrow!: Maybe you never saw this arrow before but open, for example, a workspace and you will see that, in the top-right of the window, there is an arrow. You can click on it and a menu with options will appear. This menu depends on the window/tool so you should go window by window and discover new stuff ūüôā Video here.

Recover text from closed workspaces: If you closed a workspace and you had something useful there, you can still recover it. Click on the top-right arrow (as described in the previous item) -> previous contents -> select the piece of text you want to recover ūüôā ¬†Video here.

Halt once: Sometimes you want to debug a code and you want to halt in a method. However, the method can be complicated and can be sent lots of time. You can even hang your image. This is typical when debugging with Morphic. To achieve this, we have a #haltOnce. You use it in the same was as #halt with the only difference that you have to tell the system to start taking into account such halt. To activate that, go to System -> Enable halt/inspect once.  Then, it will just halt only once. Of course you can do it by code (Halt enableHaltOnce and Halt disableHaltOnce). Another useful message is #halfIf: [xxx] which is easier than xxx ifTrue: [self halt]. Video here.

Writing to stout and reading form stdin:  Writing to stdout is really easy:

FileStream stdout lf; nextPutAll: 'zaraza'; lf.

Reading from stdin is also easy but I recommend you to read this thread. Video here.

Shortcut explanation for beginners: Do you know that when you see in a menu a shortcut between parenthesis, like (o) (which is lowercase), it means cmd+o. But, if it is (O), it is cmd+shift+o hahahah. Quite mysterious the first time ūüėČ

Creating your own shortcuts: Did you know you can create your own shortcuts? And, what’s more, that there are already lots of shortcuts for global actions like opening a transcript, a workspace, Monticello, etc, etc, etc. For more details, check the Keymapping project.

Why GC is not collecting a particular object?: Sometimes you want to understand why the GC is not removing certain object. Pharo let us explore the pointers to a particular object meaning we can trace the references and therefore understand who is referring to the object. To do this, explore your object ->right-click -> explore pointers. Or, by code, “yourObject pointersTo”. Video here.

Debugging with closures and loops: When you are debugging, sometimes you need to step into a closure. this happens, for example, when you are iterating/filtering/whatever a collection. In those cases, you have to use the “Through” button of the debugger.

Cleaning the system: “ScriptLoader new cleanUpForRelease” generates a really clean image. You can see that method to understand what it does. The resulting image is much smaller than the original one. You also have “ScriptLoader new cleanUpForProduction” which not only cleans, but also removes some stuff (like tests). That image is very small and useful for deploying applications. Video here.

How to turn on Halo’s?¬†¬†Contrary to what people think, Halos are present in Pharo but only when you really want them ūüėČ In Mac, by pressing Apple-Shift-Click and, in Linux/Windows, by pressing Shift-Alt-Click. In Pharo 2.0 the two-way cycling (i.e. the way it was in Squeak) can be restored via “Morph cycleHalosBothDirections: true.”. ¬†Video here.

Nautilus

Enable and disable plugins: Nautilus provides several plugins and, of course, you can build your own. To see the available list of plugins and add or remove them,  click in the top right arrow of a Nautilus browser -> Nautilus Plugin Manager. There, you can add and remove plugins with absolute freedom.

Get list of available shortcuts: Nautilus can show you the list of available shortcuts. To do that, click in the top right arrow of a Nautilus browser -> Shortcuts descriptions. You can then select from the list which group of shortcuts you want to see. Video here.

Conclusion

So I hope that you could learn at least ONE trick! If you didn’t, I am sorry. You should tell me new tricks ūüôā¬† If you did learn something, then excellent. Do you have more tricks to share with me?¬† There are some tricks in the Pharo collaborative book.

See you soon!


StartupLoader: running startUp scripts in Pharo

Hi. Some time ago, I wrote a post about how I build images for my projects. I am downloading new images all the time and because of that I used to have 2 problems: 1) I needed to load several external packages to the standard Pharo image; 2) I needed to set my own settings and preferences. Today, Pharo 2.0 (which is in development and unstable state) includes most of the packages I always needed to install: shout, code completion, nice browser (Nautilus or OB), refactoring integration, spotlight, etc.  So nowadays I only had problem 1).

In this post, I will show you how I solve that problem using StartupLoader, a nice utility present in Pharo (since Pharo 1.4). IMPORTANT: Everything I mention in this blog is taking into account the “new version” I did of StartupLoader. Before writing this post I have improved it and therefore you need at least a Pharo 2.0 version 20071.

UPDATE: Stephane Ducasse read,  converted and improved this blog post into a new chapter for a future Pharo book. You can find a draft here StartupPreferences.pier.pdf. 

Why we need StartupLoader?

How can we execute something at startup with Pharo? There have traditionally been two known ways (and now in Pharo there is a new third option and it is the reason of this post):

1) Send a .st file as parameter to the VM, that is, you execute the VM like this:

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/Pharo/images/Pharo-1.4.image startup.st

You execute the VM, pass the Pharo image as parameter and then the file. This will be executed during the startup. What’s the problem with this? If I want to always execute the startup in different images I always need to open the image this way, from command line. I just want to open an image with a double-click. Moreover, this file is hand-coded and not versioned in Monticello (what I would like). Besides, there are more limitations as I mention later in this blog.

2) Register your own class in Smalltalk startup list and implement #startUp message to do whatever you want to do. The problem is that my class is not present in the distributed images of Pharo. Therefore I need to manually first load my own code. Same problem: too much work.

New new tool

Remember my problem: I am downloading new images all the time. Having to manually set up my preferences is boring and time-consuming.

The new StartupLoader class searches for and executes .st files from certain locations.¬† To find these it searches for a ‚Äė.config‚Äô folder in the folder where the image file sits.¬† Then it looks in the next folder up, then up again and so on until reaching the root folder. ¬†When a ‘.config’ folder is found, StartupLoader looks within this for a ‘pharo’ folder. This contains the startup scripts common to all versions of Pharo, and also optionally a folder per Pharo version holding startup scripts suitable for that version only. ¬†So a typical directory layout might be…

.../some/folders/pharo/Content/Resources/pharo.image.
.../some/folders/pharo/Content/Resources/startup.st
.../some/folders/.config/pharo/author.st
.../some/folders/.config/pharo/useSharedCache.st
.../some/folders/.config/pharo/1.4/mystartupFor14only.st
.../some/folders/.config/pharo/2.0/mystartupFor20only.st
(**Note however that ‘.config’ is an invalid filename on Windows, so ‘..config’ is used instead)

IMPORTANT: I said that StartupLoader will search for a folder ‘.config’ starting from the image directory until the root of the filesystem. What happens if no folder is found? It creates ‘.config’ in the image folder. However, I recommend you create the ‘.config’ following the standard, that is, in the $HOME.

To know the real values for you…
Print the result of ‚ÄúFileDirectory preferencesGeneralFolder‚ÄĚ which holds the startup scripts common to all versions of Pharo.
Print the result of “FileDirectory preferencesVersionFolder” which holds the startup scripts specific to the version of the current image.

The order of the search is from the most general to the most specific:

  1. General preferences folder: This is general for all Pharo versions This folder is shared for all the images you open. In my case (in MaxOSX) and Pharo 2.0, it is¬†‘/Users/mariano/.config/pharo/’. In this place, StartupLoader will load ALL existing .st files. This type of startup is useful when we have something to execute for all images of all Pharo versions.
  2. Preference version folder: This is a specific folder for a specific Pharo version. In my case it is ¬†‘/Users/mariano/.config/pharo/2.0/’. This type of startup is useful when we have something to execute for all images of a specific Pharo version.
  3. The image folder: The startup only searches for a file called ‘startup.st’. So if you have such a file in the same directory where the image is, then such script will be executed automatically. This type of startup is usually used to do something that is application-specific or something that only makes sense for the specific image you are using. Now you might ask why we don’t search the image folder for multiple .st files. ¬†This is because it is normal for the image folder to contain .st files not related to started – such as from any file out. ¬†Using one specific file ‘startup.st‘ avoids this while still allowing an image delivered to a client to run a script upon execution on a new system. Be careful if you already were sending your own ‘startup.st’ as parameter to the VM because it will be executed twice ūüėČ

As you can see the order is from the most general to the most specific. Moreover, it does not stop when it finds files in any of them. So all are searched and executed. More specific scripts can even override stuff set in more general ones. It works more or less the same way as variables in UNIX with .bashrc /etc/envirorment, etc…

So you know where the system will search startup files. Now you can directly put your code there and it will be automatically executed during startup. Great!!! So that’s all?¬† we just write scripts there?¬† Of course not! ūüėČ

StartupActions

Directly putting code in the files is easy, however, it is not the best choice. For example, what happens if there are certain scripts you want to execute only once on a certain image but some other code that you want to execute each time the image starts? To solve that, among other things, we have the reification of StartupAction. Let’s see how we use them:

| items |
items := OrderedCollection new.
items add: (StartupAction
name: 'Basic settings'
code: [
Author fullName: 'MarianoMartinezPeck'.
Debugger alwaysOpenFullDebugger: true.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'basicSettings.st'.

What we do first is to create an instance of StartupAction using the message #name:code:. We pass as argument a name and the Smalltalk code we want to run inside a block closure. In this example, I just set my username and I put a setting to always open the debugger (no pre-debugger window). So far nothing weird.

The magic comes with the last line, the message #addAtStartupInPreferenceVersionFolder: items named: aFileName¬† receives a list of startup actions and a filename and stores the actions in a file with the passes argument. So in this case we have only one action called ‘Basic Settings’ and it will be placed in a file called ‘basicSettings.st’. But where? in which of the 3 folders described previously is it placed? well…it depends on the message. In this case, we used the #addAtStartupInPreferenceVersionFolder:named:¬† (notice the “InPreferenceVersionFolder”). So it put the files in 2). In addition, you can use the messages #addAtStartupInGeneralPreferenceFolder:named: which stores files in 1) and #addAtStartupInImageDirectory: which stores in 3). Notice that with the first two messages we can specify the file name but with the last one we can’t. Remember the last one is always called ‘startup.st’. If you are lazy and don’t want to think the name yourself, you can just use #addAtStartupInPreferenceVersionFolder: which creates a file called ‘startupPharoNN.st’ or #addAtStartupInGeneralPreferenceFolder: that creates a file named ‘startupPharo.st’.

StartupLoader

We saw that when executing the message #addAtStartupInPreferenceVersionFolder: items named:aFilename or any of its variant, a file is created with the code we want to evaluate. Then, when the system starts it will find our file and execute our code. But, how is the resulting file? Exactly as the code we provided? No! Look how our example file ‘/Users/mariano/.config/pharo/2.0/basicSettings.st’ is generated:

StartupLoader default executeAtomicItems: {
StartupAction name: 'Basic settings' code: [Author fullName: 'MarianoMartinezPeck'.
Debugger alwaysOpenFullDebugger: true].
}.

So as you can see, the file is generated by sending a collection of actions to “StartupLoader default executeAtomicItems:”. In this example the collection has only one action, but it would have more if our example has more. So now the StartupLoader will execute all the actions found in the file. Do we execute all actions? No! Actions can be built with a property of “runOnce”. So if an action has already been executed in the current image before the last save, it is not executed again. Executed actions are stored in the singleton instance #default of StartupLoader. Therefore, you have to save the image. If an action generates an error the action is NOT registered as executed. In addition, errors are also stored in the singleton of StartupLoader so you can query them after starting the image by inspecting the result of “StartupLoader default errors”.

Advanced example

As an advanced example, I want to show you the script I am using for my images. For that, I have this Smalltalk code:

setPersonalStartUpPrefernces
"self setPersonalStartUpPrefernces"
| items |
items := OrderedCollection new.

items add: (StartupAction name: 'General Preferences for all Pharo versions' code: [
FileStream stdout lf; nextPutAll: 'Setting general preferences for all Pharo versions'; lf.
Author fullName: 'MarianoMartinezPeck'.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInGeneralPreferenceFolder: items named: 'generalSettings.st'.

items add: (StartupAction name: 'Settings' code: [
FileStream stdout lf; nextPutAll: 'Setting general preferences'; lf.
UITheme currentSettings fastDragging: true.
CodeHolder showAnnotationPane: true.
MCCodeTool showAnnotationPane: true.
Deprecation raiseWarning: true.
Debugger alwaysOpenFullDebugger: true.
Parser warningAllowed: false.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'settings.st'.

items removeAll.
items add: (StartupAction name: 'Nautilus' code: [
FileStream stdout lf; nextPutAll: 'Executing Nautilus related stuff'; lf.
Nautilus pluginClasses add: { NautilusBreadcrumbsPlugin. #top }.
Nautilus pluginClasses add: { AnnotationPanePlugin. #middle }.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
] runOnce: true).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'nautilus.st'.

items removeAll.
items add: (StartupAction name: 'Monticello related stuff' code: [
| sharedPackageCacheDirectory |
FileStream stdout lf; nextPutAll: 'Executing Monticello related stuff'; lf.
sharedPackageCacheDirectory := (FileDirectory on: '/Users/mariano/Pharo/localRepo/')
assureExistence;
yourself.
MCCacheRepository default directory: sharedPackageCacheDirectory.
MCDirectoryRepository defaultDirectoryName: '/Users/mariano/Pharo/localRepo/'.
(MCRepositoryGroup default  repositories
select: [:each | (each isKindOf: MCHttpRepository)
and: [((each locationWithTrailingSlash includesSubString: 'www.squeaksource.com')
or: [each locationWithTrailingSlash includesSubString: 'http://ss3.gemstone.com/ss/'])]
]) do: [:each |
each
user: 'MMP';
password: ((FileDirectory default oldFileNamed: '/Users/mariano/Pharo/repositoriesPassword.txt') contents).
].
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'monticello.st'.

Basically, I have 4 files to customize stuff: 1) general settings;¬† 2) settings for Pharo 2.0; 3) nautilus and 4) monticello related stuff. 1) is for all Pharo versions. So far I am just setting my username. 2) 3) and 4) are for Pharo 2.0 (just because I know they work in Pharo 2.0 and I am not sure if they work in other versions). For nautilus, I don’t want to add the plugins each time (because it would add the plugin several times) so I create a StartupAction using the message #name: nameOfItem code: code runOnce: aBoolean¬† passing a true to aBoolean.

Using the tool

How to split your stuff in files and actions?

So you may have noticed that: a) #addAtStartupInPreferenceVersionFolder: and friends expect a list of actions; b) you can have multiples files. So, how do you split your code? From what I can see in the framework, there is no restriction. You can have as many actions per files and as many files as you wish. An action has a block of closure that can contain as much code as you want.

I found that one way of splitting your code is when some actions need to be executed only once and some other each time. Another reason may be some code which may be expected to fail for some reason. If it fails, the code after the line that generated the error won’t be executed. Hence, you may want to split that code to a separate action.

How to version and work with this tool?

The way I found to work with this stuff is to have my own class GeneralImageBuilder (put whatever name you want). In such class I have the mentioned method #setPersonalStartUpPrefernces (from the advanced example). So I use Monticello to save and load that project. Then, whenever I want to create the script files and add them to their correct directory, I just evaluate that method.

Be careful with the cache!

In order to support the “runOnce:”, actions are stored in the singleton instance of StartupLoader. After an action is executed (if executed correctly), the action is stored and marked as “executed”. It may happen that later on you modify the scripts by hand, or you change the rules and re-store them or some kind of black magic. So…if you change some of these, I recommend to do 2 things:

  • Remove all existing files from the preference directories (no script is removed automatically). Check methods #remove* in StartupLoader.
  • Remove the existing stored actions in StartupLoader. Check method #cleanSavedActionsAndErrors and #removeAllScriptsAndCleanSavedActions.

Conclusion

I think that the tool is very nice. It is just 3 classes and a very few methods. I have been improving it recently but still, there could be more improvements. Wanna help?¬† I wanted to summarize this post and write a better class comment, but I am running out of free time. In addition, it would be nice to have some tests ūüėČ

I want to thank to Benjamin Van Ryseghem for doing the first version of the tool and to Ben Coman for fixing the preference folder in Windows and for discussing with me about the performed improvements.

Hope you like it!


Building Pharo from PharoKernel in 10 seconds

Hi. Last week I was playing with Pharo kernel images and I thought it could be interesting to document here what we (I am not alone!) were doing. First, the context:

Context of Pharo, PharoCore and PharoKernel

In older versions of Pharo, there were always 2 distributions: Pharo (a.k.a PharoDev) and PharoCore. The latter was a small core with just the basic stuff and very few development tools. It was perfect for deployment purposes for example. Then, on top of such core, we build PharoDev which added several packages for development: OmniBrowser, code completion, Shout, refactorings, etc, etc, etc. Since Pharo 1.4 (and now in Pharo 2.0) we now have only one image which is more similar to what we know as PharoDev. Of course the image can also be shrinked and get the core.

As said, the PharoCore was great for deployment because it was small. However, Pavel Krivanek was not satisfied, and hence started to work on “KernelImage”. The image is now known as PharoKernel. Such image contains a real small core and it is only about 2 MB. The image has to run headless. Can you imagine Seaside running in 4.6 MB? Just try Seaside on PharoKernel.

The first challenge was to be able to really shrink a Pharo image to a Kernel image. This does not happen in one afternoon. Pavel has done *a lot* of work improving the modularization of the system to reduce the dependencies between packages (to be able to correctly unload packages). So PharoKernel is the result of a HUGE work.

The second step was how to take a PharoKernel and to reload and *re initialize* everything so that it works. You know…unloading Morphic (remember PharoKernel is headless) and correctly load back and initialize it is not easy. In summary, it is a really complex process.

So far, PharoKernel was working. But there was something that we didn’t like: we still needed the Compiler in PharoKernel since to load code back we needed the source and therefore the compiler.

FuelPackageLoader arrived

So…Pavel didn’t give up and continue his efforts. Fuel is a general-purpose binary serializer but in addition, we have packages like FuelPackageLoader that let us export classes and packages in a binary way. This is what I used in the post I show you how to export and import Seaside. For more details of Fuel, FuelMetalevel and FuelPackageLoader read this link. The idea is that know we can export in a binary way. So when we are serializing a class, we serialize the class itself with the method dictionary, compiled methods, subclasses, instance variables, etc.¬† And to import we do not need the Compiler! we just materialize ūüėČ ¬† In addition, the export and import is much faster.

Wanna try it yourself?

The following are the needed steps if you want to try yourself:

1) Create a working directory where you will place all your stuff.

2) Download a Pharo 1.4 image and a PharoKernel 1.4 image and move them to your working directory.

3) Download the scripts to do this experiment. The scripts are in GIT together with all the scripts used by the Pharo Jenkins. The easiest way is to clone the repository:

git clone https://git.gitorious.org/pharo-build/pharo-build.git

4) Move the files (export.st, load.st, initCore.st and load.sh) from /XXX/pharo-build/scripts/pharo/Kernel-2.0/FuelPackageLoader to your working directory.

If you follow the steps correctly you should have something like this:

ūüėČ ls -la
total 93112
drwxr-xr-x  13 mariano  staff       442 Apr 27 23:07 .
drwxr-xr-x@ 17 mariano  staff       578 Apr 27 23:06 ..
-rw-r--r--@  1 mariano  staff  10772559 Apr 27 20:35 Pharo-1.4.changes
-rw-r--r--   1 mariano  staff  15863104 Apr 26 12:05 Pharo-1.4.image
-rw-r--r--@  1 mariano  staff   2488141 Apr 26 12:17 PharoKernel-1.4.changes
-rw-r--r--@  1 mariano  staff   2272532 Apr 26 12:17 PharoKernel-1.4.image
-rw-r--r--@  1 mariano  staff  16235372 Oct 19  2009 PharoV10.sources
-rw-r--r--   1 mariano  staff      4023 Apr 27 20:34 export.st
-rw-r--r--   1 mariano  staff      9215 Apr 27 20:29 initCore.st
-rwxr-xr-x   1 mariano  staff       151 Apr 27 20:29 load.sh
-rw-r--r--   1 mariano  staff      3548 Apr 27 20:29 load.st
drwxr-xr-x  14 mariano  staff       476 Apr 26 19:27 pharo-build

5) Now we take the Pharo image and we export almost all core (except what is already present in PharoKernel) using Fuel. The script first downloads Fuel. Then we use Fuel for both things: export some class variables and fonts, but also the code (packages). If you are lazy you can see the files online. So for example, these lines export fonts and a class variable:

FLSerializer serialize: (TextStyle named: 'Bitmap DejaVu Sans') toFileNamed: 'dejavu.fuel'.
FLSerializer serialize: (UCSTable classPool at: #JISX0208Table) toFileNamed: '#jisX0208Table.fuel'.

And this is how we export packages:

packageNames := 'Ring-Core-Containers
Ring-Core-Kernel' lines.

FileStream forceNewFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageStore new storeOn: aStream packages: packageNames.].

In this example we are exporting two packages (Ring-Core-Containers and Ring-Core-Kernel) with the same stream (‘ring.fuel’ file). If you want to only export one package you can use instead the message #storeOn:packageNamed:.

So its time to take the Pharo image and export everything. To do that we need to run the image from command line and send the export.st file as argument.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/Pharo-1.4.image export.st

6) Once we have exported, we should have several .fuel files in our working directory. The biggest file is pharo-core.fuel and it is about 5 MB. The export should have taken approx. 10 seconds (considering also the time to download Fuel).

7) As you can see in load.st, the way to import a package with Fuel is:

FileStream readOnlyFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageLoader new loadFrom: aStream contents readStream].

Now we take a PharoKernel image and we load the files.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image load.st

This step should also take less than 10 seconds. You can notice that now PharoKernel is not 2MB anymore but more about 13MB ūüėČ

8) Once packages has been loaded, we have to correctly initialize the system. FuelPackageLoader has a setting to send class side #initialize or not. For most cases it works. But in other cases (like the case of PharoKernel or any Smalltalk boostrap), the initialize has to be done in a careful order. Therefore, the initialize is done manually in a script called initCore.st. That scripts not only sends the #initialize to classes but also performs all the necessary actions to get back a headfull (not headless) working Pharo image. So the step now is to run:

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image initCore.st

9) Finally!!!! If everything was fine, we should now have a working Pharo image built from a PharoKernel. You can just open¬†PharoKernel-1.4.image and give it a try ūüôā

Conclusions

  • It is possible to have a minimal image without compiler and boostrap from there a bigger image.
  • The performance seems quite good so far (of course we still need to add lots of things)
  • FuelMetalevel (the package to serialize and materialize classes) is working really well since it could serialize and materialize almost all classes and traits from Pharo.

Known limitations

For this experiment of exporting and importing packages we are using FuePackageLoader. This is a prototype and we still have lots of missing features. In fact, that’s why Martin is now as a student in the GSoC project ūüėȬ† The current limitations are:

  • We are not exporting source code, timestamp, class comments, etc.
  • We are not doing all needed validations nor recompiling in those cases that may be necessary.
  • We are not updating instances if the classes already existed in the image.
  • and more…. (read here for more details).

So, that’s all.¬† I hope you had fun. See you