Monthly Archives: September 2011

Importing and exporting packages with Fuel

Background

Hi guys. One month ago, I was running like crazy preparing my presentations for ESUG 2011. I had 3 talks and also the Fuel presentation in the awards. So…I was thinking about examples and demos to show with Fuel. I finally showed how to serialize a debugger and continue debugging it in another image, how to serialize a graph and store it in a Riak NOSQL database, how to use SandtoneDB with Fuel, etc. But I was not the only one running. Martin Dias, main author of Fuel, who was in the other part of the world, was also coding. He was trying to finish what I will show today: to be able to export and import packages using Fuel. Reality is different from movies. Hence, he could finally make it work, one day after the awards 😦

Since that day, we only focused in Fuel itself and we left the tool there. This week I went to work to Lille and Igor asked me to show him such tool. The tool at that moment was outdated (using an old version of Fuel) and quite buggy. Today, we have something better 🙂

UPDATE: As part of his excellent blog and screencasts, James Robertson has recorded a screencast where he follows the steps of this blog post. I recommend to take a look to it as well. Thanks a lot James!

What is the tool about?

As you may know, Fuel is a plain object graph serializer. No more than that. However, Fuel supports correct serialization of Class, Trait, MethodDictionary, CompiledMethod, BlockClosure, MethodContext, etc. Hence, creating a tool on top of that which allows one to export and import packages, is quite easy. In fact the tools is right now about 220 lines of code.

The tool is called Fuel Package Loader, but its name may change in the future. Basically what it does is to take a PackageInfo, extract the classes and extension methods, create a FLPackage instance and serialize that. During materialization (the import), we need to do extra tasks like registering classes in Smalltalk globals, in categories/organizations, notify about class creation, send class side #initialize, etc. In other words, this tool adds package semantics to a plain serialization.

Installing Fuel and Package Loader

As always, you have two options, pick up the latest bleeding edge build from our Jenkis job, or install with Metacello. For this post, use the Metacello way. The last stable version of Fuel is 1.6. But we needed to fix a couple of things on it in other to make FuelPackageLoader to work. However, Fuel right now is not “stable enough” to release 1.7 (we need to fix few things). Hence, I have created a temporal version 1.6.1 tagged as #development specially for this post. All this is because I want the post to be reproducible. So, pick up a Pharo 1.3 image (sorry, for older Pharo images we need to fix something very stupid but I don’t have time, but for 1.7 it will be fixed), and execute:

Gofer new
squeaksource: 'Fuel';
package: 'ConfigurationOfFuel';
load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.6.1') load: #('Core' 'PackageLoaderWithTests').

That will load Fuel, FuelPackageLoader and all their tests. Now, one of the best tests we can do for this project is to import and export Seaside web framework. Why? Because it is really big, it has a lot of packages, dependencies between them, a lot of important class side #initialize, etc. So…if we are able to import/export seaside, we should be able to import/export most of the existing projects.

Save this image as ‘Fuel.image’.

Exporting packages

The first basic and naive idea we had when dealing with packages was to map one to one between Monticello packages and Fuel packages. Knowing which packages of Seaside are needed, in which other, and all the dependencies it has, is quite complicated. So…we can use Metacello for that. In the following example, we will install Seaside using Metacello and the regular packages of Monticello. But, we will at the same time, creat an associated Fuel package for each Monticello package. So, execute the following and wait between 10 and 30 minutes (if you are lazy, go directly to the import step where I provide the binaries):

FLMetacelloStore default
capturePackagesDuring: [
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfSeaside30';
load.
((Smalltalk at: #ConfigurationOfSeaside30) project version: '3.0.6')
load: #('Core' 'Tests' 'Zinc-Seaside' )]
prefixed: 'seaside'.

In this case, we have loaded the group ‘Core’ of Seaside, together with its tests and the Zinc seaside adaptor (just because I like it). Once that finishes, you can execute:

(ZnZincServerAdaptor port: 4242)
codec: GRPharoUtf8Codec new;
start.

And now if you open the internet browser and go to http://localhost:4242/  you should see the nice Seaside welcome page.

Now, pay attention that a directory called ‘FuelMetacello’ was created in the same directory where the .image is. Inside that directory you can see all the generated fuel packages. Each of them are named with the pattern ‘prefix-packageName-fuel’. For example, ‘seaside-Scriptaculous-Core.fuel’, ‘seaside-Seaside-Flow.fuel’, etc. Those would be the equivalents to the Monticello .mcz and we will import them after.

Finally, save this image as ‘FuelSeasideExporter.image’

Importing packages

To import packages, all we need is to read the file and materialize them. But the problem is, in which order? mmmm Metacello used to know that, but not us. Well, this is why during export, we have also written a file that contains the script to import back. The file is called ‘prefix-load-script.txt’ (seaside-load-script.txt in this case) and it is also in the ‘FuelMetacello’ directory. If you open it, you can see its contents:

#('seaside-ConfigurationOfGrease.fuel' 'seaside-ConfigurationOfKomHttpServer.fuel' 'seaside-ConfigurationOfSPort2.fuel' 'seaside-ConfigurationOfSwazoo2.fuel' 'seaside-ConfigurationOfZincHTTPComponents.fuel' 'seaside-ConfigurationOfRefactoringBrowser.fuel' 'seaside-Grease-Core.fuel' 'seaside-Grease-Pharo-Core.fuel' 'seaside-Grease-Tests-Core.fuel' 'seaside-Grease-Tests-Pharo-Core.fuel' 'seaside-Sport.fuel' 'seaside-Swazoo.fuel' 'seaside-Zinc-HTTP.fuel' 'seaside-Zinc-Tests.fuel' 'seaside-Seaside-Core.fuel' 'seaside-Seaside-Pharo-Core.fuel' 'seaside-Seaside-Component.fuel' 'seaside-Seaside-Canvas.fuel' 'seaside-Seaside-Pharo-Canvas.fuel' 'seaside-RSS-Core.fuel' 'seaside-Javascript-Core.fuel' 'seaside-Javascript-Pharo-Core.fuel' 'seaside-Comet-Core.fuel' 'seaside-Prototype-Core.fuel' 'seaside-Scriptaculous-Core.fuel' 'seaside-JQuery-Core.fuel' 'seaside-JQuery-UI.fuel' 'seaside-Seaside-Email.fuel' 'seaside-Seaside-Pharo-Email.fuel' 'seaside-Seaside-HTML5.fuel' 'seaside-Seaside-InternetExplorer.fuel' 'seaside-Seaside-Session.fuel' 'seaside-Seaside-RenderLoop.fuel' 'seaside-Seaside-Tools-Core.fuel' 'seaside-Seaside-Flow.fuel' 'seaside-Seaside-Examples.fuel' 'seaside-RSS-Examples.fuel' 'seaside-Seaside-Widgets.fuel' 'seaside-Seaside-Tools-Web.fuel' 'seaside-Seaside-Pharo-Tools-Web.fuel' 'seaside-Seaside-Environment.fuel' 'seaside-Seaside-Pharo-Environment.fuel' 'seaside-Seaside-Development.fuel' 'seaside-Scriptaculous-Components.fuel' 'seaside-Seaside-Swazoo.fuel' 'seaside-Zinc-Seaside.fuel' 'seaside-Seaside-Adaptors-Swazoo.fuel' 'seaside-Seaside-Tests-Core.fuel' 'seaside-Seaside-Tests-Pharo-Core.fuel' 'seaside-Seaside-Tests-Session.fuel' 'seaside-Seaside-Tests-RenderLoop.fuel' 'seaside-Seaside-Tests-Component.fuel' 'seaside-Seaside-Tests-Canvas.fuel' 'seaside-Seaside-Tests-Environment.fuel' 'seaside-RSS-Tests-Core.fuel' 'seaside-Javascript-Tests-Core.fuel' 'seaside-Javascript-Tests-Pharo-Core.fuel' 'seaside-Seaside-Tests-Email.fuel' 'seaside-Seaside-Tests-Functional.fuel' 'seaside-Seaside-Tests-Pharo-Functional.fuel' 'seaside-Seaside-Tests-Flow.fuel' 'seaside-Seaside-Welcome.fuel' 'seaside-Seaside-Pharo-Welcome.fuel' 'seaside-Prototype-Tests-Core.fuel' 'seaside-Scriptaculous-Tests-Core.fuel' 'seaside-Scriptaculous-Tests-Components.fuel' 'seaside-JQuery-Tests-Core.fuel' 'seaside-JQuery-Tests-UI.fuel' 'seaside-Seaside-Tests-HTML5.fuel' 'seaside-Seaside-Tests-InternetExplorer.fuel' 'seaside-Seaside-Tests-Examples.fuel' 'seaside-Seaside-Tests-Tools-Web.fuel' 'seaside-Seaside-Tests-Development.fuel' 'seaside-Seaside-Tests-UTF8.fuel' 'seaside-Seaside-Tests-Welcome.fuel' 'seaside-DynamicBindings.fuel' 'seaside-KomServices.fuel' 'seaside-KomHttpServer.fuel' 'seaside-Seaside-Tools-OmniBrowser.fuel' 'seaside-Seaside-Pharo-Tools-OmniBrowser.fuel' 'seaside-Seaside-FileSystem.fuel' 'seaside-Seaside-Tests-FileSystem.fuel' 'seaside-Seaside-Pharo-Continuation.fuel' 'seaside-Seaside-Tests-Pharo-Continuation.fuel' 'seaside-Seaside-Pharo-Flow.fuel' 'seaside-Seaside-Pharo-Development.fuel' 'seaside-Seaside-Tests-Pharo-Development.fuel' 'seaside-Seaside-Adaptors-Comanche.fuel' 'seaside-Comet-Pharo-Core.fuel' 'seaside-Comet-Examples.fuel' 'seaside-Comet-Tests-Core.fuel' 'seaside-Seaside-Tests-Adaptors-Comanche.fuel' 'seaside-Grease-Slime.fuel' 'seaside-Grease-Tests-Slime.fuel' 'seaside-Seaside-Slime.fuel' 'seaside-Seaside-Tests-Slime.fuel')
do: [ :packageFileName |
(FileDirectory default directoryNamed: 'FuelMetacello')
oldFileNamed: packageFileName
do: [ :aStream | FLPackageLoader new loadFrom: aStream binary ]]
displayingProgress: [ :packageFileName | 'Fuel importing ', packageFileName ]

So…. if you have followed the previous steps, then just open ‘Fuel.image’ (a new Pharo 1.3 image where Fuel is loaded but seaside is not), paste the previous code into a workspace and execute it. Of course that it depends on the machine, but here (Mac Book Pro i5), it only takes 10 seconds.

If you were a lazy gut who didn’t export the package, ok, here you have a zip of FuelMetacello. Just extract it and let the directory next to the .image.

It seems it was too fast to be working. So, if you don’t believe me, execute again the Zinc code to start a server (if you let the FuelSeasideExporter.image open, take another port) and then go to the browser. You should see, again the welcome page:

Ok, you still don’t believe it? Open the TestRunner, select all the seaside related tests and run it. They will be all green, or at least it should be exactly the same results as when running the tests from the FuelSeasideExporter image (where seaside was loaded with Metacello).

Why we put each package in a separate file? First because it was easier, and second because we wanted to be as much similar as possible with Monticello packages. In addition, this way, you can choose which packages to load. However, that doesn’t mean we are not able to serialize all packages in the same file, say seaside.fuel hahah. That’s possible, and it is easy since you can just download one file and you are done.

What the tool does and does not

As you may imagine the idea is that maybe in the future we can replace Monticello’ mcz with Fuel packages. Of course, we really far from there, but this was the first step. We tool right now:

  • Sends class side #initialize to classes.
  • Add classes to superclass instVar ‘subclasses’.
  • Send notifications that a class was created

What it does not do so far (but we will try to do it in the future) is :

  • Validations: we do not do validations at all. For example, we do not check whether a class we are materializing is already present in the image or if the global name was already taken. So we do not do class migration and things like that. In addition, we don’t validate class pools uniqueness and so on. So…right now the tool only works if you load in a clean image where you know there cannot be problems.
  • Source code is not serialized. That means, when you import and browse the classes, you will see the decompiled source.
  • .changes is not modified at all when loading a package.
  • Traits are partially supported and not really tested (soon will be fixed).

Conclusions

This tool is similar to VisualWorks Parcels, but there is a big difference. Parcels was designed with the idea in mind of managing code. Hence, it is a little bit “coupled” with that. It is difficult to use Parcels as a plain serializer. Fuel is a general-purpose serializer and coupled with nothing. It is an infrastructure where we can then build tools on top of that. FuelPackageLoader is just an example of that. In 220 lines of code we where able to have something more or less working for importing and exporting packages. We can export seaside (96 packages!!!) in 12 seconds and import them in 10. Ok, we still need to add a lot of missing features that will probably have overhead. But so far, the results are promising.

Do not use this tool for production. It is just an experiment and it is really really in development. Just try it for fun. If you want to help us, try importing/exporting your own packages instead of seaside and let us know if you have problems!

Finally, a big clap clap to Martin who has been working a lot in Fuel and the package loader.

Advertisements

Community driven VM

Hi. This is a short post to tell you that I am very happy with the work it is being done related to the Squeak virtual machine.

A couple of years ago, John Macintosh ported the Mac VM to the iPhone. Now with all the iOS it should be easy to make it work not only in the iPhone but also in iPad, etc… Then after Esteban Lorenzano continue with the effort, and even worked in projects like Mars and Deimos.

Some time ago, Andreas Raab started a port of the SqueakVM to the Android.  Dimitry Golubovsky took that work and continue with it and improve it a lot. It can now work even with Android tablets. Few weeks, he announced a first beta release that you can find here. The home webpage of such project is here and there are also some nice screenshots.

Yoshiki Ohshima started to port the SqueakVM to the Google Native Client (NaCl platform), that is, you can run Pharo/Squeak directly in the Chrome browser! isn’t that cool?  Now, Javier Pimas and Guido Chari, who are doing an internship with INRIA – RMOD, are following his work and now they have made it work!  you can run Pharo on Chrome!  of course, this is just the start and there are still a lot of problems and limitations.

And I can imagine the SqueakVM is already working in a lot of different devices I am not even aware of. So why I am writing this? Because I am really surprised about this fact. Most of those ports/forks were done by one or a few persons and sometimes even in free time. And they succeeded. How common is this in other programming languages?  I don’t know, but I am impressed.

In addition, I am happy with the idea that more and more people can jump into the VM world and compile the VM because that helps us as a community. Just as an example, Alain Plantec could build the VM and finally fix a problem with the FT2Plugin. Friedrich Dominicus could build the VM and fix a problem with SerialPort.  Laurent Laffont is the mentor of the project SmallHarbour and they needed some specific changes in the SecutiryPlugin which were done by SeasideHosting. I helped him to merge those changes to the latest code of Cog, and it worked! Now they have their own clone in gitorious and they can easily build its own SmallHarbour VM.

Sean P. DeNigris accidentally put a halt in a method which was called from the startup of the system, which generated a system crash. He was not able to run again the image, but he has code there he needed to rescue. Guess what? he build the VM in debug mode, modify the VM in the method execution and put an if saying “if the selector is #halt, then fetch next bytecode”!! And it worked. That is amazing.

So….to sum up, what I want to say is that from my point of view it is extremely important what is happening arround Cog VM and the community. Having an easy and uniform way to compile the VM, a Jenkis server that automatically builds the VM, a open-source repository like gitorious where everybody can work, etc. are really important. But the most important one, is to DOCUMENT. No matter if you use git, CMakeVMMaker, Jenkis or whatever. The key point is to document the process and that everybody can do it. That’s all I have tried with my first posts of this blog. My post about how to build the VM from scratch has already like 800 visists….So…hopefully more people can build Cog VM now.

But notice that people are aware of that. It is not by chance that Stéphane Ducasse has organized a “Deep into Smalltalk” school with INRIA, or that we have a long “Compiling your own VM” tutorial at ESUG 2011.

Happy VM hacking 🙂


ESUG 2011 presentations’ slides

Thanks to Marcus Denker, all the slides of the presentations of ESUG 2011 are available in two places: to directly download from: http://esug.org/data/ESUG2011/ and to view or download from slideshare:   http://www.slideshare.net/esug/

As every year, ESUG has not only the presentations of the conference itself but also from the International Workshop in Smalltalk Technologies (IWST). So, the slides of the presentations of the conference are in http://esug.org/data/ESUG2011/Slides/ and http://www.slideshare.net/esug/presentations. The ones of IWST are in http://esug.org/data/ESUG2011/IWST/PRESENTATIONS/ and http://www.slideshare.net/esug/presentations. In addition, the proceedings of IWST are in: http://esug.org/data/ESUG2011/IWST/Proceedings.pdf

As I have explained in a previous post, I gave 3 presentations: one (DBXTalk) for the conference and two (Fuel and Ghost proxies) for the IWST.

The first one was the one of Fuel. It was a pity because the printed schedule of the conference was not in sync with the real time of the talk. Hence, some people missed it. Anyway, the slides can be downloaded here and in slideshare.  You can also download the paper from here. If you are interested in Fuel, I really recommend to see the slides and to read the paper.

The second talk was also as part of the IWST and was about Ghost, a proxy implementation I did on top of Pharo. The slides can be downloaded here and in slideshare.  Again, you can also download the paper from here.

Finally, on Friday we did (together with Santiago Bragañolo and Guillermo Polito) a presentation of DBXTalk. The slides can be downloaded here and in slideshare. The live demo….well maybe we will have the video in a near future.

See you


Fuel: First place at ESUG Innovation Technology Awards

ESUG Innovation Technology Awards presentations

Hi guys. As I have already told you before, I finally presented Fuel at ESUG Innovation Technology Awards. The awards ceremony was quite good. Some people came to the table where I was and asked things about Fuel. Some didn’t know at all what Fuel was about (neither the internals of a general serializer while others knew a lot about serializers. I got interesting talks and some good ideas. Now there are even more items in my infinitive to-do list.

BTW, thanks Adriaan van Os for his wonderful pictures.

Now….showing a serializer is quite boring. What would you show?  Serializing and materializing an object graph is plain boring. So I though I could show something more interesting such as:

  1. Type some code on a workspace, debug it and then, in the middle of the debugging, serialize the debugger and its content into a file. Then I opened another image, I materialized the file and the debugger just opened. I could continue the debugging in this new image, exactly in the same place where it was serialized and with all the temp variables with its correct state. People were amazed about this. It is fun it took me literally 2″  to test it and see it works. There is NOTHING special needed for Fuel to test that.
  2. Following the previous item, Esteban A. Maringolo recommend me in the Pharo mailing list to do more or less the same but with Pharo errors. Pharo writes down an exception into PharoDebug.log (this is done in #logError:inContext:). So what I did is to modify such method so that it also serializes the context into a PharoDebug.fuel. Then, for the demo, I raised an exception so that the PharoDebug.fuel was written. Finally, I could take another image, materialize such file and debug the exception 🙂   That was kind of fun. Of course there are a lot of limitations in this but it was fun anyway.
  3. I showed how to use Fuel with a Riak NoSQL database. This is basically what I explained in a previous post.
  4. I showed how to speed up SandstoneDB by using Fuel instead of SmartRefStream. Again, this was the same as my previous post.

So…that was pretty much what I showed from Fuel. There were much more things I wanted to show but didn’t have time. For example, Martin Dias (main author of Fuel) could export seaside packages, load them in another image and start Seaside!!!

So far so good. Which was the problem? While we (submitters of ESUG Awards) were presenting our projects, Instantiations offered a reception with drinks (wine and scotch) and food. Grrrr I couldn’t take anything. Well…. almost anything. Henrik Johansen took my place a couple of minutes while I took a glass of wine. I may be wrong but someone told me that someone put a nice text in a workspace saying something like “Will be back in a couple of minutes” hahaha. Awesome.

ESUG Innovation Technology Awards Ceremony

The ceremony of the awards was on Wednesday (2 days after the presentation). This happened in the “City Chambers” in Edinburgh. There was a very nice reception (food and drinks again) in a really nice place. Before announcing the winners, “The Lord Provost and Council of the City of Edinburgh” give a nice speech. He (George Grubb) explained Edinburgh was proud of receiving ESUG and the talk was quite fun. Finally, he gave the diploma to each of the winners.

So….as you may guess from the title, Fuel got the first place. First of all, I want to really thanks to all the people who vote for us. There were a lot of interesting projects out there. Second, I would like to thanks Martin Dias. HE is the real Fuel developer. He has been doing a great job with Fuel and I just help him as much as I can. He deserves the honor as much as I do (and even more!). Finally, thanks to ESUG for sponsoring Martin with SummerTalk and to Stéphane Ducasse for bringing Fuel to life.

Paper accepted and slides of the presentation

Apart from presenting Fuel at the ESUG Innovation Technology Awards, we have also submitted a paper to the International Workshop on Smalltalk Technologies. You can get the paper here. The paper was accepted, invited to do an extended journal version of it and then I gave a presentation in the ESUG conference. The talk was more or less good but as always I have something to learn. This time, the problem was the I had too many slides. I was on time (ok, maybe 5 minutes more hahahhah) but I have to be too fast. Next time: less slides.

Anyway, the slides are available here and also in slideshare.