Tag Archives: Fuel

LZ4 binding for Pharo

Hi guys. In the last days I wrote a Pharo binding for the LZ4 compressor (thanks to Camillo Bruni for pointing out), and so I wanted to share it. The main goal of LZ4 is to be really fast in compressing and uncompressing but not to obtain the biggest compression ratio possible.

The main reason why I wrote this binding is for Fuel serializer, with the idea of compressing/uncompressing the serialization (ByteArray) of a graph. Hopefully, with a little bit of overhead (for compressing and uncompressing), we gain a lot in writing to the stream (mostly with files and network). However, the binding is not coupled with Fuel at all.

I have documented all the steps to install and run LZ4 in Pharo here.  Please, if you give it a try, let me know if it worked or if you had problems.

I would also like to do some more benchmarks with it, because so far I only did a few. So if you have benchmarks to share with me, please do it.

So far LZ4 does not provide a streaming like API. We tried with Camillo to build a streaming API in Pharo (like ZLibWriteStream, GZipWriteStream, etc) but the results were not good enough. So we are still analyzing this.

Ahhh yes, for the binding I use Native Boost FFI, so I guess I will wrote a post soon to explain how to wrap a very simple library with NB.

See you,

Advertisements

“New” Tanker – current status

Both ESUG 2012 and GSoC have finished. At ESUG I gave a presentation about Fuel/Tanker (sorry, no slides because it was all demo). I also presented Tanker in the ESUG Awards….we got the 4th place, and only 8 points of difference with the 3er place 🙂   Anyway…I wanted to make public what exactly we finally managed to do with Tanker, a package exporter and importer that uses Fuel serializer. As I said in a previous post, we have changed a lot how Tanker works internally. In that post also I mentioned that Tanker didn’t support source code export/import, but in this new version, it does support methods source code, class comments, timestamps and everything that is currently being stored in the .changes or .sources file.

Exporting packages

The following figure shows how the export works. The input is a package of code (classes + extension methods), in this example “MyPackage”. The first step is to traverse the classes and methods of the package and do 2 things: a) write the source code (source of methods, class comments, timestamps, etc) to a text-based chunk format sources file (MyPackage.tankst); b) create “definition” objects. These objects represent a kind of object model that store all the necessary information of the code entities to be able to recreate them back  in another image. So we have for example, TAClass, TATrait, TAMethod, TABinding, TAAdditionalMethodState, TATraitAlias, TATraitComposition, etc…

Something key here is that besides all that information, we also store the relative position of the source code of the entity in the “MyPackage.tankst”. So for example, TAMethod instances will have a number representing that relative offset (similar to the pointer that CompiledMethods have in their trailer to .sources/.changes) where its source code is.

Another important detail is that in TAMethod we are storing the bytecodes, because even if we have the source code we want to avoid needing to compile during the import.

Once we have finished writing the sources file and creating the definitions, we just serialize them into a Fuel binary, say “MyPackage.tank”.

Importing packages

The following diagram shows how the import works. The input is the sources file (MyPackage.tankst) and the Fuel binary file (MyPackage.tank). The first step is basically to read the sources file and append all its contents in the .changes file (so that to get the same behavior as if we were installing a package with Monticello). But before doing that, we temporally store/keep the current “end” position of the .changes file 😉  (you already guessed why?).

The second step is to materialize the definitions using Fuel. Something very nice is that the definitions that were serialized, also understand the necessary messages to be installed in the system (#installUsing: aSourcesFile environment: anEnvironment). So we basically tell each class and each extension method to get installed in the system. Those objects will delegate to the rest of the definitions to complete their task (all definitions know what to do when installing them in the system).

Classes and traits will use the class builder to recreate themselves back. The source code was already installed in the .changes, so now we need to fix the “pointers” from our newly created classes/methods to point to the correct place of their source in the .changes file. And this is very easy because their position is just the “end” position of the .changes before installing our package (that value I told you we were temporally storing) + the relative position that was stored in the definition itself 🙂

So what is the biggest difference with Monticello for example? The key point is that we export bytecodes and we avoid having to compile during import. This make things faster while also being able to install packages without compiler (very useful for bootstrapping for example).

How to install it and use it

Tanker only works in Pharo 2.0 (because we rely on new class builder and layouts) so you first need to grab an image. Then, you can evaluate:

Gofer it
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) perform: #loadDevelopment.
Yes, so far there isn’t a stable version yet, we are waiting for new class builder. Be careful because that will install the new class builder (and change the system to use it)  and it may have some bugs…so take care 🙂
The export and import look like:
aPackage := TAPackage behaviors: {MyClass. MySecondClass} extensionMethods: {String>>#myExtensionMethod1. Object>>#myExtensionMethod2}.
(TAExport
 package: aPackage
 binariesOn: aBinaryWriteStream
 sourcesOn: aTextSourcesWriteStream)
 run.

(TAImport
 binariesFrom: aBinaryReadStream
 sourcesFrom: aSourcesReadStream)
 run.

However, we do not expect the final user to provide all the list of classes and extension methods. Therefore, we have helper methods to export and import RPackages and PackageInfos. Example:

 TAExport exportRPackageNamed: 'MyProjectCore'.
 TAExport exportRPackageNamed: 'MyProjectTests'.

TAImport importPackageNamed: 'MyProjectCore'.
 TAImport importPackageNamed: 'MyProjectTests'.

Conclusion

So that was all for today. Probably, I will do another blog post where I will show how we can query Metacello to know which packages to export and in which order, some details about the new class builder, some benchmarks while exporting all seaside/pier/magritte (10 seconds to export and 20 to import), and so on. The conclusion we got with this project is that indeed Fuel can be successfully used in yet another completely different domain. If you want to help us, please test it with your own packages. Right now we have only one open issue (need to recompile if superclasses present in the image have reshaped). I guess that soon we will fix this last issue and release the first stable version. In the future we plan to analyze how can we really integrate Tanker with Monticello/Metacello.


Tanker screencast and image ready for testing

Hi guys. Last days, we submitted Tanker to the ESUG Innovation Technology Awards. As part of that submission, I have created a screencast and an image with the examples.

The screencast gives you an introduction to Tanker and shows you how to export and import packages. I starts with a simple example, then it exports a real library and ends up exporting all Seaside, Magritte and Pier 🙂   Ahhh yes, sorry for my voice, I know it is ugly hahaha. You can also watch it here:

You can also get the image I used for the screencast, which you can try and experiment.

Have fun,


Tanker: transporting packages with Fuel

Hi all. You may have noticed that Tanker is starting to appear in some mails or in the Pharo issue tracker. Tanker is a project that Martin and I have been developing for a while and we are going to submit it this year to ESUG Innovation Technology Award. Therefore, I thought it would be interesting to explaining what it is, its current status, its goals, etc.

What is “Tanker” and what was “FuelPackageLoader”?

Right now the common way to export and import packages in Pharo is by using Monticello (or doing fileOut, which is almost the same). This ends up exporting the source code and then compiling it during the import. Tanker is a tool to export and import packages of code in a binary way using Fuel serializer. Using Fuel enables us to avoid having to compile from sources during the import. Tanker understands the concept of “packages of code” and the correct integration of them into the system. For example, it initializes classes, sends notifications, etc.

Tanker was first a prototype called “FuelPackageLoader” which was what I used for the example of exporting and importing seaside packages. In the last months, we have renamed the project to “Tanker”. Why? Because we do not want people to think that it is a Fuel project. In fact, Tanker is a simple USER of Fuel. Just as any other code that uses Fuel. This is why we have also moved it to its own repository.

Fuel has a package called “FuelMetalevel”. This package gives Fuel the knowledge of how to correctly serialize and materialize classes, metaclasses, traits, method dictionaries, compiled methods and closures, in other words, all the entities related to code and runtime infrastructure. It only knows how to serialize and materialize correctly. Nothing else. It does not initialize classes, it does not notify the system about the materialized classes, it does not install classes in Smalltalk globals, etc.

Current features, design and missing things

Right now, Tanker provides the following features:

  • It is able to export a package to a .tank file and import it in another image. The input for the export is a TAPackage which basically contains a list of classes and a list of extension methods. We are completely decoupled from the “package representation” (PackageInfo, RPackage, MCPackage, etc). However, we provide an API if you want to directly export from those types of packages.
  • Classes are initialized and installed in Smalltalk globals, events are sent, etc.
  • It has the ability to add additional user-defined objects to the package being exported (this is used, for example, for the Pharo generation from a kernel to store large/heavy class variables, tables and fonts).
  • It supports pre and post load actions represented as closures.

From the design point of view, Tanker:

  • Fully serializes classes and traits (not its “definitions”)
  • Does not use the ClassBuilder during materialization. Tanker itself materializes the “class objects” and sets the data.

So far, we are missing:

  • The possibility to export source code (right now classes and methods do not have source code) and also to install it in the .changes file during import.
  • Some validations during import. For example, the superclass of a class being installed may have changed its shape and, therefore, the classes to install need recompilation (because the bytecodes accessing instVars offset may be shifted or wrong). Or if a class already exists in the image and the shape has changed, we need to update the existing instances.
  • Integration with other tools like Monticello and Metacello.

Results with the “current status”

There are so far 3 real examples of Tanker:

How to install it and use it

Tanker will work only in the bleeding edge of Pharo 2.0. So I first recommend you to get an image from Jenkins. Then, you can install Tanker this way:

Gofer it
 url: 'http://smalltalkhub.com/mc/marianopeck/Tanker/main';
 package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) load.

To export a package and provide yourself the classes and extension methods, you can:

| aPackage aStream |
"Export"
aStream := 'demo.tank' asFileReference writeStream binary.
aPackage := TAPackage behaviors: {TestCase. TestLocalVariable. } extensionMethods: #().
TAPackageStore new storePackage: aPackage on: aStream.

"Import"
aStream := 'demo.tank' asFileReference readStream binary.
TAPackageLoader new loadFrom: aStream.

Then you can also use the API that provides helper methods to RPackage and PackageInfo:

aPackage := TAPackage fromPackageInfoNamed: 'MyPackage'

You also have #fromPackagesInfoNames:, #fromRPackageNamed: and #fromRPackagesNames:. Of course, there are more use-cases, API and scenarios. But, so far, that is the simplest usage. For more examples, browse the class side methods of TankerExamples.

GSoC and “new status”

The results so far are quite promising and not anymore a “proof of concept”. However, we still need to support source code management as well as the already mentioned pending features. Because of this reason, Martin submitted Tanker for the GSoC and fortunately it was accepted. So, right now we are moving to a different design to solve the requirements.  The idea now is NOT to serialize classes and traits, but instead serialize their “definition”. Imagine by “definition” the string used to create them. Then, during import, instead of just materializing class objects, we take the definition and, using the ClassBuilder or similar, we “evaluate” the definitions and we get the new classes.

At the same time, the idea is to export the source code of a package in a file (myPackage.tank.st or something like that) and the binary representation in another file (say myPackage.tank). Then, during import, you should be able to import with or without sources.

Side-effect projects of Tanker

You may be wondering why we didn’t start from the very beginning with the “definitions” way. Well, to be honest, the ClassBuilder is a mess, difficult to understand, maintain and extend. It was really hard trying to use it for our purpose. So the first “side-effect project” of Tanker is to continue pushing the “new ClassBuilder” started by Toon Verwaest based in “slots”. Martin Dias, Guillermo Polito and Camillo Bruni are pushing it and writing tests. I think it could be soon integrated in Pharo and replace the old one. The idea is that Tanker will use this ClassBuilder, for example, to evaluate the definitions.

When we are importing a class, it may happen that the superclass (present in the image where we are importing) has changed its shape (added or removed instVars, change supperclass, etc). If this is true, we have to recompile because the bytecodes accessing intsVars will be a shifted offset. However, recompiling is slow and we don’t want that. Therefore, Tanker will use IR (intermediate representation) which was developed by Marcus Denker and the team working with the new Opal compiler. IR is just a nice model generated from a CompiledMethod. The idea is that we can generate the IR, modify it (bytecodes for instVars accessing, for example) using this nice abstraction and API and then generate back a new CompiledMethod. This is way faster than recompiling. Furthermore, IR is decoupled from Opal so we don’t need whole Opal.

Conclusion

Tanker started as an experiment to see whether Fuel coud be used to export and import packages in a binary way. The proof of concept was quite good so we are now going forward with the source code management and related stuff. It is important to notice that Tanker just “uses” Fuel. Fuel is completely decoupled from Tanker. We think Fuel was well received by the community. We are doing our best so that Tanker gets positive feedback as well.


Reviving CI test failures in local machine

The problem

These days, most serious software developments include a Continuous Integration server which runs tests. A problem appears when tests fail in the server but they do not fail locally. There can be differences in the used operating system, virtual machine, configuration, etc. Let’s take as an example the Jenkins server of Pharo. We use such server to not only build and test the Pharo images but also the VMs. There are 3 slaves (one for each: Windows, Linux and MacOSX) and tests are run in all of them. Still, it is common to have tests that we cannot reproduce locally. Why?

  • Random failures: tests that fail randomly. Of course, we would prefer not having these tests but sometimes we do.
  • Tests that fail because as a side effect of other tests.
  • The OS of the server or even its configuration/infrastructure is different.
  • The used virtual machine can be different (for example, Jenkins uses the VM it builds to test the other jobs).

What do we have now?

So, we have a failure in the server that we cannot reproduce locally. How can we understand what happened? So far, the only thing we have is a piece of a text-based stack trace. For example, let’s take this test failure:

Error Message
Assertion failed
Stacktrace
SocketStreamTest(TestCase)>>signalFailure:
SocketStreamTest(TestCase)>>assert:
SocketStreamTest(TestCase)>>should:raise:
SocketStreamTest>>testUpToAfterCloseSignaling
SocketStreamTest(TestCase)>>performTest

As you can see, this is not that helpful and you may still don’t know what  has happened. Something really useful would be to at least know what where the values of the instance variables involved in that stack… Here is where Camillo Bruni had a nice idea 🙂

Fuelizing test failures

In Pharo, the stack of the running system is reified also from the language side and we can access them! (we can even modify them). We have instances of MethodContext which hold an instVar ‘sender’ that refers to the next sender in the stack (another MethodContext or nil if it is the last). Apart from ‘sender’, a context also includes the receiver, the method that caused its activation, the arguments and the temporal variables. The Fuel serializer can serialize any type of object including MethodContext. If we can serialize a MethodContext (and closures and methods), we can serialize a stack, right? And what does this mean? Well, it means that we can serialize a debugger with its current state. I have already shown several times (at ESUG Innovation Technology Award and at PharoConf) how we can use Fuel to serialize a debugger (from image X) in the middle of its execution and materialize it in image Y and continue debugging.

Pharo provides ‘exception’ objects and, at the end, test failures are exceptions (TestFailure). We can always ask its “signaler context” to an exception, in other words, the MethodContext that signals it. Once we have that MethodContext, we have all the stack (because that object has a sender and the sender context has a sender and ….). So, how do we serialize that?

context := testFailure signalerContext.
FLSerializer newFull
 serialize: context
 toFileNamed: 'context.fuel'.

So that piece of code will serialize all the stack of contexts including all the transitive closure: receiver, arguments, temporal variables, etc.

Reviving test failures

So we have serialized our test failure on a file. Now we want to revive them in our local machine. The first obvious thing is to materialize the original stack from the file. But then, what do we do with the stack? How can we do something useful with it? Well, Pharo allows us to open a debugger for a particular stack 🙂 . This means we can just open a debugger with the stack of the test failure! To do that:

| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'context.fuel'.
Debugger
 openContext: aContext
 label: 'This is the new debugger!'
 contents: nil

And that opens our nice debugger. Much better than a text-based stack trace, isn’t it?

Caveats when serializing a stack

When you serialize the whole stack, you may find some problems:
  1. The object graph that you serialize and, therefore, the resulting stream size can be really large depending on what the contexts have. Sometimes a context end up in the UI so you end up serializing lots of morphs, colors, forms, etc. If everything is fine, the file should be a couple hundred or thousands KB. If the file  size is in MB…then you may be serializing too much.
  2. Not only the graph is too big, but it also incorporates objects that CHANGE while being serialized (mostly when these are objects from the UI). This will cause Fuel to throw an error saying the graph has changed during serialization.
  3. If 2) happens, then depending where you trigger the fuel serialization, you may end up in a loop. For example, say you want to serialize each error with Fuel. So you change  SmalltalkImage>>logError:inContext:  to write the context with Fuel. Now, if 2) happens and Fuel throws an error, you will try to log that again causing again the serialization… infinitive loop.
  4. Apart from the previous points, there are still more problems. You can read the title “Limitation and known problems” in this post.
So… some workarounds are (still, not sure if they will help in all cases):
  • Deep copy the context before serializing it.
  • If you want to serialize particular contexts (for example, particular domain exceptions), then you may know WHERE to hook to make some instVars transient and, therefore, avoid serializing things you don’t want and that may cause 2).
  • Serialize a PART of the stack.

Jenkins integration

Thanks to Camillo and to Sean P. DeNigris, now Jenkins serializes (for some jobs) each test failure into a file (here you can see how to set up your own Jenkins for Pharo). For example, we have the job “pharo-2.0-tests”. If you select the OS and then a particular build number, you will have an artifact called “Pharo-2.0-AfterRunningTests.zip”. For example, this one: https://ci.lille.inria.fr/pharo/view/Pharo%202.0/job/pharo-2.0-tests/Architecture=32,OS=mac/lastSuccessfulBuild/artifact/Pharo-2.0-AfterRunningTests.zip. This zip contains all the .fuel files of all the test failures. Each file is named ClassXXX-testYYY.fuel.

To workaround the problem mentioned in the previous paragraphs (“Caveats when serializing a stack”), we just serialize a part of the stack: from the context that signals the failure up to the test method. Example:

  ...
  performTest
"Start context slice"
> testMyFeatureBla
> ...
> ...
> assert: foo equals: bar
"end context slice"
  assert:
  Exception signal

The idea is to serialize the least number of stack-frames possible while still giving decent debug feedback. To do that, our Jenkins code (HDTestReport>>serializeError: error of: aTestCase) is:

serializeError: error of: aTestCase
 "We got an error from a test, let's serialize it so we can properly debug it later on..."
 | context testCaseMethodContext |

 context := error signalerContext.
 testCaseMethodContext := context findContextSuchThat: [ :ctx|
 ctx receiver == aTestCase and: [ ctx methodSelector == #performTest ]].
 context := context copyTo: testCaseMethodContext.

 [
 FLSerializer newFull
 " use the sender context, generally the current context is not interesting"
 serialize: context sender
 toFileNamed: aTestCase class name asString,'-', aTestCase selector, '.fuel'.
 ] on: Error do: [:err| "simply continue..." ]
During serialization the graph can somehow reach classes of the Jenkins code (like HDTestReport). If you materialize in an image where such class is not present, you will have a Fuel error. For this purpose in the same Pharo-2.0-AfterRunningTests.zip besides having the .fuel files, we also have a Pharo-2.0-AfterRunningTests.image which, as it names says, was saved after having run all tests (meaning it has the Jenkins code installed). This means we can directly use that image to materialize and it will work. The other option is to take another image and install the following before materializing:
Gofer new
 url: 'http://ss3.gemstone.com/ss/CISupport';
 package: 'HudsonBuildTools20';
 load.
This is temporal because soon Jenkins support code will be directly integrated in Pharo.
Anyway, I recommend using the same version of image that was used during serialization. So I think that using directly Pharo-2.0-AfterRunningTests.image is more reliable.

Conclusion

It is clear that there are several caveats. However, I do believe this is yet another step in CI and development. It is just one more tool you have at hand when something is failing in the server and you cannot reproduce it locally. It the worst case, it won’t help but it won’t hurt either. If you have luck, you may find out the cause 🙂 It is incredible all the things you can do when the stack is reified and visible from the language while also being serializable. For me, asking for a text-based stack trace in Smalltalk is like going to a cabaret and ask for a hug. We have so much power that we should take advantage of it. At the end, using a debugger is way better. Anyway, I do not recommend to remove the stack trace information, just adding also the Fuel possibility.

My (past) presentations at PharoConf and (future) talk at ESUG 2012

Hi. As usual, I wanted to share with you the slides of my last talks in case you are interested.

PharoConf

Last month I went to the first PharoConf held in Lille, France, and I gave to talks. One was about using the Fuel serializer for several different hacky things 🙂  You can find the slides here but since most of the presentation was a demo, they are almost useless. The videos of the conference are being processed and will be updated soon. I will update this post once they are finished.

The other talk I gave was about building the Pharo Virtual Machine and you can find the slides here. If you are interesting in the topic, you can see all the blog posts I have written about it.

ESUG 2012

Once again, I will be attending and presenting at ESUG (in Ghent, Belgium). This year I will present something similar to the Fuel talk at PharoConf. As you can see in the ESUG schedule, the abstract of my talk says:

Fuel is an open-source general-purpose object serialization framework developed in Pharo. It is fast, extensible and has an object-oriented design. It can serialize not only plain objects, but also closures, contexts, methods, classes, traits, among others.
This presentation will be mostly a demo with only a few slides. I will show the power of Fuel by using it in several scenarios: rebuilding Pharo from a kernel image, exporting/importing Monticello packages, moving a debugger from one image to another one, persisting (and also import/export) Pier kernels, etc.

So…see you at Ghent?


Building Pharo from PharoKernel in 10 seconds

Hi. Last week I was playing with Pharo kernel images and I thought it could be interesting to document here what we (I am not alone!) were doing. First, the context:

Context of Pharo, PharoCore and PharoKernel

In older versions of Pharo, there were always 2 distributions: Pharo (a.k.a PharoDev) and PharoCore. The latter was a small core with just the basic stuff and very few development tools. It was perfect for deployment purposes for example. Then, on top of such core, we build PharoDev which added several packages for development: OmniBrowser, code completion, Shout, refactorings, etc, etc, etc. Since Pharo 1.4 (and now in Pharo 2.0) we now have only one image which is more similar to what we know as PharoDev. Of course the image can also be shrinked and get the core.

As said, the PharoCore was great for deployment because it was small. However, Pavel Krivanek was not satisfied, and hence started to work on “KernelImage”. The image is now known as PharoKernel. Such image contains a real small core and it is only about 2 MB. The image has to run headless. Can you imagine Seaside running in 4.6 MB? Just try Seaside on PharoKernel.

The first challenge was to be able to really shrink a Pharo image to a Kernel image. This does not happen in one afternoon. Pavel has done *a lot* of work improving the modularization of the system to reduce the dependencies between packages (to be able to correctly unload packages). So PharoKernel is the result of a HUGE work.

The second step was how to take a PharoKernel and to reload and *re initialize* everything so that it works. You know…unloading Morphic (remember PharoKernel is headless) and correctly load back and initialize it is not easy. In summary, it is a really complex process.

So far, PharoKernel was working. But there was something that we didn’t like: we still needed the Compiler in PharoKernel since to load code back we needed the source and therefore the compiler.

FuelPackageLoader arrived

So…Pavel didn’t give up and continue his efforts. Fuel is a general-purpose binary serializer but in addition, we have packages like FuelPackageLoader that let us export classes and packages in a binary way. This is what I used in the post I show you how to export and import Seaside. For more details of Fuel, FuelMetalevel and FuelPackageLoader read this link. The idea is that know we can export in a binary way. So when we are serializing a class, we serialize the class itself with the method dictionary, compiled methods, subclasses, instance variables, etc.  And to import we do not need the Compiler! we just materialize 😉   In addition, the export and import is much faster.

Wanna try it yourself?

The following are the needed steps if you want to try yourself:

1) Create a working directory where you will place all your stuff.

2) Download a Pharo 1.4 image and a PharoKernel 1.4 image and move them to your working directory.

3) Download the scripts to do this experiment. The scripts are in GIT together with all the scripts used by the Pharo Jenkins. The easiest way is to clone the repository:

git clone https://git.gitorious.org/pharo-build/pharo-build.git

4) Move the files (export.st, load.st, initCore.st and load.sh) from /XXX/pharo-build/scripts/pharo/Kernel-2.0/FuelPackageLoader to your working directory.

If you follow the steps correctly you should have something like this:

😉 ls -la
total 93112
drwxr-xr-x  13 mariano  staff       442 Apr 27 23:07 .
drwxr-xr-x@ 17 mariano  staff       578 Apr 27 23:06 ..
-rw-r--r--@  1 mariano  staff  10772559 Apr 27 20:35 Pharo-1.4.changes
-rw-r--r--   1 mariano  staff  15863104 Apr 26 12:05 Pharo-1.4.image
-rw-r--r--@  1 mariano  staff   2488141 Apr 26 12:17 PharoKernel-1.4.changes
-rw-r--r--@  1 mariano  staff   2272532 Apr 26 12:17 PharoKernel-1.4.image
-rw-r--r--@  1 mariano  staff  16235372 Oct 19  2009 PharoV10.sources
-rw-r--r--   1 mariano  staff      4023 Apr 27 20:34 export.st
-rw-r--r--   1 mariano  staff      9215 Apr 27 20:29 initCore.st
-rwxr-xr-x   1 mariano  staff       151 Apr 27 20:29 load.sh
-rw-r--r--   1 mariano  staff      3548 Apr 27 20:29 load.st
drwxr-xr-x  14 mariano  staff       476 Apr 26 19:27 pharo-build

5) Now we take the Pharo image and we export almost all core (except what is already present in PharoKernel) using Fuel. The script first downloads Fuel. Then we use Fuel for both things: export some class variables and fonts, but also the code (packages). If you are lazy you can see the files online. So for example, these lines export fonts and a class variable:

FLSerializer serialize: (TextStyle named: 'Bitmap DejaVu Sans') toFileNamed: 'dejavu.fuel'.
FLSerializer serialize: (UCSTable classPool at: #JISX0208Table) toFileNamed: '#jisX0208Table.fuel'.

And this is how we export packages:

packageNames := 'Ring-Core-Containers
Ring-Core-Kernel' lines.

FileStream forceNewFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageStore new storeOn: aStream packages: packageNames.].

In this example we are exporting two packages (Ring-Core-Containers and Ring-Core-Kernel) with the same stream (‘ring.fuel’ file). If you want to only export one package you can use instead the message #storeOn:packageNamed:.

So its time to take the Pharo image and export everything. To do that we need to run the image from command line and send the export.st file as argument.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/Pharo-1.4.image export.st

6) Once we have exported, we should have several .fuel files in our working directory. The biggest file is pharo-core.fuel and it is about 5 MB. The export should have taken approx. 10 seconds (considering also the time to download Fuel).

7) As you can see in load.st, the way to import a package with Fuel is:

FileStream readOnlyFileNamed: 'ring.fuel' do: [:aStream |
aStream binary.
FLPackageLoader new loadFrom: aStream contents readStream].

Now we take a PharoKernel image and we load the files.

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image load.st

This step should also take less than 10 seconds. You can notice that now PharoKernel is not 2MB anymore but more about 13MB 😉

8) Once packages has been loaded, we have to correctly initialize the system. FuelPackageLoader has a setting to send class side #initialize or not. For most cases it works. But in other cases (like the case of PharoKernel or any Smalltalk boostrap), the initialize has to be done in a careful order. Therefore, the initialize is done manually in a script called initCore.st. That scripts not only sends the #initialize to classes but also performs all the necessary actions to get back a headfull (not headless) working Pharo image. So the step now is to run:

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/PhD/Marea/Fuel/PharoKernelExperiments/blog/PharoKernel-1.4.image initCore.st

9) Finally!!!! If everything was fine, we should now have a working Pharo image built from a PharoKernel. You can just open PharoKernel-1.4.image and give it a try 🙂

Conclusions

  • It is possible to have a minimal image without compiler and boostrap from there a bigger image.
  • The performance seems quite good so far (of course we still need to add lots of things)
  • FuelMetalevel (the package to serialize and materialize classes) is working really well since it could serialize and materialize almost all classes and traits from Pharo.

Known limitations

For this experiment of exporting and importing packages we are using FuePackageLoader. This is a prototype and we still have lots of missing features. In fact, that’s why Martin is now as a student in the GSoC project 😉  The current limitations are:

  • We are not exporting source code, timestamp, class comments, etc.
  • We are not doing all needed validations nor recompiling in those cases that may be necessary.
  • We are not updating instances if the classes already existed in the image.
  • and more…. (read here for more details).

So, that’s all.  I hope you had fun. See you


Random news

Hi. Sorry for being away from the blog for so long but I was on holidays and, when I came back, I dedicate all my time to submitting a paper. This post is just a summary of some recent news. In case you are already aware, then just discard this post 😉

Pharo 1.4 was released

Finally. After a lot of work, Pharo 1.4 was released. You can read the list of actions done here. Pharo is not perfect and, of course, there is still a lot of room for improvement but, from my point of view, the system is getting better with each release. Some of the improvements are in the UI or in direct tools so users notice them. However, most of the changes are in the infrastructure, that is, under the hood so they are not always perceived by the user. One way to notice this is the fact that we are able to build a lot of great stuff using this new infrastructure. As an example, I refer to projects such as Ring, RPackage, Nautilus, Zinc, Fuel, Opal, NativeBoostAthens, Moose, Spec, DrGeo, etc.

Pharo Conf 2012

For the first time ever, we  will have a Pharo Conference. It will be held in Lille, France on the 24th and 25 th of May 2012. Pharo Conf 2012 is organized by the Pharo core team of INRIA and the IA team of Ecole des Mines de Douai, both of which I am proud to be part of. For more information you can read this blog post. The main goal of the conference is to bring companies, developers and users together. Yo can register for the social event here.

International Workshop – IWST 2012 program committee

This is kind of personal but, since I am really happy, I wanted to share the news with you. I will be part of the program committee of IWST 2012 collocated in ESUG. It is the first time for me so I hope I will enjoy it.

Young Engineer Position on Smalltalk VM and Related Technology

This looks to me like the best job offer ever. It is an engineer position in RMoD team (in Lille, France) which is focused on virtual machine related work. As researchers, we need a system that allows us to explore new ideas. The job of the engineer is to help us by improving the infrastructure we use for our daily work. You can read more about it here. Wanna apply?

13 GSoC accepted projects (Fuel included!)

ESUG has received 13 slots for the GSoC 2012. This means that there were 13 accepted Smalltalk projects. I think these kind of projects are always useful. I was part of the ESUG SummerTalk (in two opportunities as a student and once as a mentor) and I think it is really worthy. As a  student, you can learn a lot and hopefully also give something useful to the community. Happily for me, one of the accepted projects is FuelPackageLoader. Even if I am not as official “mentor” in the GSoC, I will be there 🙂  What we are going do is described here. Basically, we will continue the prototype we have started. I talked about that when I showed how to import and export packages with Fuel (meaning to load Seaside in 10 seconds).

Ok, so that’s all for now. Talk to you soon!


Moving contexts and debuggers between images with Fuel

Hi guys. During ESUG 2011, at the Awards, I was showing Fuel. The week before such event I was thinking what I could show to the people. This was a challenge because showing a serializer can be plain boring. I was working at home that afternoon, and suddenly I thought: “What happens if I try to serialize a living debugger and materialize it in another image?” After 5 minutes, really, you will see it takes only 5 minutes, I notice that such crazy idea was working OUT OF THE BOX. Even if I knew Fuel supported serialization of methods, contexts, closures, classes, etc…I was surprised that it worked from the first try. I was so happy that I tried to explain to my poor wife what I had just done hahahah. Unfortunately, she told me it was too abstract and that understanding the garbage collector was easier (I promise she really understands what the garbage collector does hahhahaha).

Well….several months has passed, but I would like to show you how to do it because I think it may be of help for real systems 😉  So…the idea is the following: whenever there is an error, you can get the context from it, and such context is what is usually written down into a log file (in Pharo this is PharoDebug.log). I will show you two things: 1) how to serialize a debugger in one image and materialize it another one and; 2) how to write down the context into a Fuel file when there is an error so that you can materialize it later in another image.

Installing Fuel

The first step is, of course, install Fuel. The latest stable release is 1.7 but to have better results with this example, I would recommend 1.8. Fuel 1.8 is not released yet it is because we plan to write some stuff in the website. The code is almost finish, so you should load Fuel 1.8 beta1. In my case I am using a normal Pharo 1.3 image:

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel';
load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.8-beta1') load.

Once you have finished loading Fuel, save the image. Let’s call it Fuel.image.

Serializing and materializing a debugger

Now its time to do something hacky in the image so that to open a debugger. Or you can just take a piece of code and debug it. In my example, I opened a workspace and wrote the following:

| a |
a := 'Hello Smalltalk hackers. The universal answer is '.
a := a , '42!'.
Transcript show: a.

Then I select the whole code, right click -> “debug it”. Then I do one “step over” and I stop there before the concatenation with ’42!’.

I am sure there could be better ways, but the simpler way I found to get the debugger instance for this example, is to do a Debugger allInstances first 😉  so… be sure not to have another debugger opened hahaha.  Now…let’s serialize the debugger:

Smalltalk garbageCollect.
FLSerializer
serialize: Debugger allInstances first
toFileNamed: 'debugger.fuel'.

After that, you should have a ‘debugger.fuel’ created in the same directory where the image is. Now close your image (without saving it) and re open it. If everything is fine, we should be able to materialize our debugger and continue debugging. So, let’s try it:

| newDebugger |
newDebugger := FLMaterializer materializeFromFileNamed: 'debugger.fuel'.
newDebugger openFullMorphicLabel: 'Materialized debugger ;)'.

So????  Did it work??  are you as happy as me when I first saw it? 🙂  if you check this new opened debugger, you will see its state is correct. For example, the instVar ‘a’ has the correct state. You can now open a Transcript and continue with the debugger as if were the original one.

Of course that even if this simple example works, there are a lot of problems. But I will explain them at the end of the post.

Serializing and materializing errors

In the previous example we have serialized the debugger manually. But imagine the following: you have a production application running. There is an error, and PharoDebug.log is written with all the stack. The user/client send you by email the .log and you open your favorite text editor to try to understand what happened. Now imagine the following: you have a production application running. There is an error, and a PharoDebug.fuel is written with all the stack. The user/client send you by email the file and you open an image, and then materialize and open a debugger. How does it sound? 🙂 magical?

For this example, we will just change the place where Pharo writes PharoDebug.log when there is an error. That method is #logError:inContext:. What we will do is to add just 2 lines at the beginning to serialize the context:

logError: errMsg inContext: aContext

" we should think about integrating a toothpick here someday"
FLSerializer
serialize: aContext
toFileNamed: 'PharoDebug.fuel'.

self logDuring: [:logger |
logger
nextPutAll: 'THERE_BE_DRAGONS_HERE'; cr;
nextPutAll: errMsg; cr.

aContext errorReportOn: logger.

"wks 9-09 - write some type of separator"
logger nextPutAll: (String new: 60 withAll: $- ); cr; cr.
]

Now yes, let’s execute something that causes an error. What I did is to evaluate 1/0. After that, you should see the file PharoDebug.fuel in the same directory where the image is. You can now close the image and reopen it. And then, let’s reopen de debugger:

| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label:  'This is the new debugger!' contents: nil

Et voilà! Hopefully that worked 🙂   Notice that in this example and the previous one, there is nothing in special with the Fuel serialization. You are using the normal API, and all you do is to serialize a debugger or a context as if you were serializing any normal object. Of course, you can also apply this idea to other places. For example, in Seaside you have an error handler. You may want to serialize the error with Fuel there.

Limitation and known problems

  • Even if Fuel can fully serialize methods, classes, traits, etc., it is recommended that the image were the contexts/debuggers are serialized and materialized are equal. If you are doing this in a production application, then you can have the same image running locally. The idea is that both images have the same classes and methods installed. This is because, by default, if the object graph to serialize includes compiled methods, classes, class variables, etc., they are all considered as “globals”, which means that we only serialize its global name and then during materialization it is searched in Smalltalk globals. Hence, classes and methods have to be present. Otherwise you have to use Fuel in a way that it serializes classes as well, but that’s more complicated.
  • There may be things that affects the debugger which are part of the image and not the serialization, and they may have changed. Imagine for example, a class variable which has changed its value in the image where you serialize. Then it will have a different value in the image where you materialize. Most of these problems also happens if even if you open the debugger later in the same image…some state may have changed…
  • The graph reachable from the contexts can be very big. For example, Esteban Lorenzano was doing this for an application and one of the problems is that from the context it was reachable the whole UI…which means lots and lots of objects. In such a case, you can always use Fuel hooks to prune the object graph to serialize.
  • Be aware to use exactly the same version of Fuel in both images 😉

Conclusion

All in all, I think that as a very first step, it is very nice that we can serialize this kind of stuff like contexts and debuggers out of the box with Fuel. This could be the infrastructure for a lot of fancy stuff. I don’t think that the debugger materialization can be as reliable as if you were debugging in the original image. I don’t think either that it should replace PharoDebug.log. However, what I do think is that you can add the Fuel serialization just as another way of getting more information about the problem. It’s one more tool you can add to your Smalltalk toolbox 🙂


Interview about Fuel for ClubSmalltalk

Hi guys. ClubSmalltalk is a very nice website which has a lot of information regarding Smalltalk. You can see interviews, posts, jobs offers, etc. There is also a mailing list in Spanish which has been the most active Smalltalk/Spanish mailing list in the last years. Anyway….you can see it all by yourself in the website.

Some time ago, they contacted me to do an interview, mainly because Fuel won the ESUG Awards. The interview also included some questions related to my PhD and what I am doing here in France. So… if you are interested in knowing why we have started Fuel, which was the most difficult part, what is a pickle format, etc, I really recommend you to take a look at it.

The interview is here.

See you