Monthly Archives: April 2012

Building Pharo from PharoKernel in 10 seconds

Hi. Last week I was playing with Pharo kernel images and I thought it could be interesting to document here what we (I am not alone!) were doing. First, the context:

Context of Pharo, PharoCore and PharoKernel

In older versions of Pharo, there were always 2 distributions: Pharo (a.k.a PharoDev) and PharoCore. The latter was a small core with just the basic stuff and very few development tools. It was perfect for deployment purposes for example. Then, on top of such core, we build PharoDev which added several packages for development: OmniBrowser, code completion, Shout, refactorings, etc, etc, etc. Since Pharo 1.4 (and now in Pharo 2.0) we now have only one image which is more similar to what we know as PharoDev. Of course the image can also be shrinked and get the core.

As said, the PharoCore was great for deployment because it was small. However, Pavel Krivanek was not satisfied, and hence started to work on “KernelImage”. The image is now known as PharoKernel. Such image contains a real small core and it is only about 2 MB. The image has to run headless. Can you imagine Seaside running in 4.6 MB? Just try Seaside on PharoKernel.

The first challenge was to be able to really shrink a Pharo image to a Kernel image. This does not happen in one afternoon. Pavel has done *a lot* of work improving the modularization of the system to reduce the dependencies between packages (to be able to correctly unload packages). So PharoKernel is the result of a HUGE work.

The second step was how to take a PharoKernel and to reload and *re initialize* everything so that it works. You know…unloading Morphic (remember PharoKernel is headless) and correctly load back and initialize it is not easy. In summary, it is a really complex process.

So far, PharoKernel was working. But there was something that we didn’t like: we still needed the Compiler in PharoKernel since to load code back we needed the source and therefore the compiler.

FuelPackageLoader arrived

So…Pavel didn’t give up and continue his efforts. Fuel is a general-purpose binary serializer but in addition, we have packages like FuelPackageLoader that let us export classes and packages in a binary way. This is what I used in the post I show you how to export and import Seaside. For more details of Fuel, FuelMetalevel and FuelPackageLoader read this link. The idea is that know we can export in a binary way. So when we are serializing a class, we serialize the class itself with the method dictionary, compiled methods, subclasses, instance variables, etc.  And to import we do not need the Compiler! we just materialize 😉   In addition, the export and import is much faster.

Wanna try it yourself?

The following are the needed steps if you want to try yourself:

1) Create a working directory where you will place all your stuff.

2) Download a Pharo 1.4 image and a PharoKernel 1.4 image and move them to your working directory.

3) Download the scripts to do this experiment. The scripts are in GIT together with all the scripts used by the Pharo Jenkins. The easiest way is to clone the repository:

git clone https://git.gitorious.org/pharo-build/pharo-build.git

4) Move the files (export.st, load.st, initCore.st and load.sh) from /XXX/pharo-build/scripts/pharo/Kernel-2.0/FuelPackageLoader to your working directory.

If you follow the steps correctly you should have something like this:

😉 ls -la
total 93112
drwxr-xr-x  13 mariano  staff       442 Apr 27 23:07 .
drwxr-xr-x@ 17 mariano  staff       578 Apr 27 23:06 ..
-rw-r--r--@  1 mariano  staff  10772559 Apr 27 20:35 Pharo-1.4.changes
-rw-r--r--   1 mariano  staff  15863104 Apr 26 12:05 Pharo-1.4.image
-rw-r--r--@  1 mariano  staff   2488141 Apr 26 12:17 PharoKernel-1.4.changes
-rw-r--r--@  1 mariano  staff   2272532 Apr 26 12:17 PharoKernel-1.4.image
-rw-r--r--@  1 mariano  staff  16235372 Oct 19  2009 PharoV10.sources
-rw-r--r--   1 mariano  staff      4023 Apr 27 20:34 export.st
-rw-r--r--   1 mariano  staff      9215 Apr 27 20:29 initCore.st
-rwxr-xr-x   1 mariano  staff       151 Apr 27 20:29 load.sh
-rw-r--r--   1 mariano  staff      3548 Apr 27 20:29 load.st
drwxr-xr-x  14 mariano  staff       476 Apr 26 19:27 pharo-build

5) Now we take the Pharo image and we export almost all core (except what is already present in PharoKernel) using Fuel. The script first downloads Fuel. Then we use Fuel for both things: export some class variables and fonts, but also the code (packages). If you are lazy you can see the files online. So for example, these lines export fonts and a class variable:

FLSerializer serialize: (TextStyle named: 'Bitmap DejaVu Sans') toFileNamed: 'dejavu.fuel'.
FLSerializer serialize: (UCSTable classPool at: #JISX0208Table) toFileNamed: '#jisX0208Table.fuel'.

And this is how we export packages:

packageNames := 'Ring-Core-Containers
Ring-Core-Kernel' lines.

FileStream forceNewFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageStore new storeOn: aStream packages: packageNames.].

In this example we are exporting two packages (Ring-Core-Containers and Ring-Core-Kernel) with the same stream (‘ring.fuel’ file). If you want to only export one package you can use instead the message #storeOn:packageNamed:.

So its time to take the Pharo image and export everything. To do that we need to run the image from command line and send the export.st file as argument.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/Pharo-1.4.image export.st

6) Once we have exported, we should have several .fuel files in our working directory. The biggest file is pharo-core.fuel and it is about 5 MB. The export should have taken approx. 10 seconds (considering also the time to download Fuel).

7) As you can see in load.st, the way to import a package with Fuel is:

FileStream readOnlyFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageLoader new loadFrom: aStream contents readStream].

Now we take a PharoKernel image and we load the files.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image load.st

This step should also take less than 10 seconds. You can notice that now PharoKernel is not 2MB anymore but more about 13MB 😉

8) Once packages has been loaded, we have to correctly initialize the system. FuelPackageLoader has a setting to send class side #initialize or not. For most cases it works. But in other cases (like the case of PharoKernel or any Smalltalk boostrap), the initialize has to be done in a careful order. Therefore, the initialize is done manually in a script called initCore.st. That scripts not only sends the #initialize to classes but also performs all the necessary actions to get back a headfull (not headless) working Pharo image. So the step now is to run:

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image initCore.st

9) Finally!!!! If everything was fine, we should now have a working Pharo image built from a PharoKernel. You can just open PharoKernel-1.4.image and give it a try 🙂

Conclusions

  • It is possible to have a minimal image without compiler and boostrap from there a bigger image.
  • The performance seems quite good so far (of course we still need to add lots of things)
  • FuelMetalevel (the package to serialize and materialize classes) is working really well since it could serialize and materialize almost all classes and traits from Pharo.

Known limitations

For this experiment of exporting and importing packages we are using FuePackageLoader. This is a prototype and we still have lots of missing features. In fact, that’s why Martin is now as a student in the GSoC project 😉  The current limitations are:

  • We are not exporting source code, timestamp, class comments, etc.
  • We are not doing all needed validations nor recompiling in those cases that may be necessary.
  • We are not updating instances if the classes already existed in the image.
  • and more…. (read here for more details).

So, that’s all.  I hope you had fun. See you

Advertisements

Random news

Hi. Sorry for being away from the blog for so long but I was on holidays and, when I came back, I dedicate all my time to submitting a paper. This post is just a summary of some recent news. In case you are already aware, then just discard this post 😉

Pharo 1.4 was released

Finally. After a lot of work, Pharo 1.4 was released. You can read the list of actions done here. Pharo is not perfect and, of course, there is still a lot of room for improvement but, from my point of view, the system is getting better with each release. Some of the improvements are in the UI or in direct tools so users notice them. However, most of the changes are in the infrastructure, that is, under the hood so they are not always perceived by the user. One way to notice this is the fact that we are able to build a lot of great stuff using this new infrastructure. As an example, I refer to projects such as Ring, RPackage, Nautilus, Zinc, Fuel, Opal, NativeBoostAthens, Moose, Spec, DrGeo, etc.

Pharo Conf 2012

For the first time ever, we  will have a Pharo Conference. It will be held in Lille, France on the 24th and 25 th of May 2012. Pharo Conf 2012 is organized by the Pharo core team of INRIA and the IA team of Ecole des Mines de Douai, both of which I am proud to be part of. For more information you can read this blog post. The main goal of the conference is to bring companies, developers and users together. Yo can register for the social event here.

International Workshop – IWST 2012 program committee

This is kind of personal but, since I am really happy, I wanted to share the news with you. I will be part of the program committee of IWST 2012 collocated in ESUG. It is the first time for me so I hope I will enjoy it.

Young Engineer Position on Smalltalk VM and Related Technology

This looks to me like the best job offer ever. It is an engineer position in RMoD team (in Lille, France) which is focused on virtual machine related work. As researchers, we need a system that allows us to explore new ideas. The job of the engineer is to help us by improving the infrastructure we use for our daily work. You can read more about it here. Wanna apply?

13 GSoC accepted projects (Fuel included!)

ESUG has received 13 slots for the GSoC 2012. This means that there were 13 accepted Smalltalk projects. I think these kind of projects are always useful. I was part of the ESUG SummerTalk (in two opportunities as a student and once as a mentor) and I think it is really worthy. As a  student, you can learn a lot and hopefully also give something useful to the community. Happily for me, one of the accepted projects is FuelPackageLoader. Even if I am not as official “mentor” in the GSoC, I will be there 🙂  What we are going do is described here. Basically, we will continue the prototype we have started. I talked about that when I showed how to import and export packages with Fuel (meaning to load Seaside in 10 seconds).

Ok, so that’s all for now. Talk to you soon!