Tag Archives: Monticello

Pharo By Example in Spanish and PBE2 new chapters!

Hi. This is a short post in case someone has missed the info. The Pharo By Example’s spanish translation was finally done. Each of the 14 chapters were translated. Notice that even that, the translation is not 100% ready. It is a draft version and it still needs some passes along the whole book.  Anyway, the link is this: https://gforge.inria.fr/frs/download.php/28829/PBE1-Spanish-FullDraft-2011-08-01.zip

So, please distribute along spanish speaking smaltalkers.

For the translation, there are a lot of people to thank, there were like 15 translators. Special thanks to Gabriala Arevalo who start leading it, and then to Nicolas Paez that continues it.

Besides the translation, Pharo By Example 2 is in the way and there are drafts for already a lot of chapters:

That’s all, I hope you enjoy them.


Migrating projects to SqueakSource3

Hi guys. I am one of the guys who is always criticizing www.squeaksource.com because it is down all the time. And when it is not down it is loosing versions. So it is not fair that I continue doing so if there is a new already working solution out there: SqueakSource3 (http://ss3.gemstone.com/ss).

During Smalltalks 2011 I talked to Dale Henrichs who told me that SqueakSource3 was quite stable and has daily backups. SqueakSource3 is a new implementation of SqueakSource but based in Seaside3, Magritte2, etc. Even more (and this is very cool), it is deployed in a GemStone server. Thanks a lot for GemStone for hosting such a service and to all the developers behind SS3.

So, I decided to give it a try. I have started to migrate all my projects or the projects I contribute and I thought I could share that with you.

Migrating projects to SqueakSource3

There are two ways I found in order to migrate packages from one repository to another one. As an example, I will show you how I have migrated Fuel code from http://www.squeaksource.com/Fuel to http://ss3.gemstone.com/ss/Fuel

Pre-optional step

As explained later, you will need to fetch all versions of the packages you want to migrate. Downloading that from squeaksource can be quite slow and you may need a good internet connection. However, if you want to migrate a project you have been development, it is probable that some .mcz are still somewhere in your machine. What you can do is to copy all .mcz files of your machine to the package-cache so that you can get it from there and avoid the network traffic. To do that I often use this bash script:

#!/bin/bash

find /Users/mariano/ -name "*.mcz" -print0 | xargs -0 -I {} cp {} /Users/mariano/Pharo/YOUR-PACKAGE-CACHE/

Using Gofer #fetch and #push

As you know, Gofer is a nice scripting library for Monticello and it provides the methods #fetch (which downloads versions from remote repositories into the local cache) and #push (which uploads local versions from local cache into remote repositories). So the idea is that you can fetch the versions from the original repository and push in the new one. In the example of Fuel, I should do:

Gofer it
squeaksource: 'Fuel';
package: 'Fuel';
package: 'FuelTests';
fetch.

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'Fuel';
package: 'FuelTests';
push.

Notice that such code could take a lot of time since EACH version of EACH package has to be fetched and pushed. So…let your machine, take a coffee and then come back. In this example, I only migrate two packages (‘Fuel’ and ‘FuelTests’) but there are much more packages in the Fuel repository. Having to write 20 lines of #package: can be a lot. So if you are a lazy guy like me, you can do the following hack:

| packagesNames gofer |

packagesNames := (Gofer new
disablePackageCache;
squeaksource: 'Fuel';
allResolved)
collect: [ :each | each packageName ] as: Set.

gofer := Gofer it
squeaksource: 'Fuel'.

packagesNames do: [:each | gofer package: each].
gofer fetch.

gofer := Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel'.

packagesNames do: [:each | gofer package: each].
gofer push.

That way you can fetch and push all packages. Of course, you can always get the list first, remove by hand the packages that you don’t want to migrate, and finally just migrate the ones you want.

Using Gofer and Metacello configurations

Instead of manually choosing which packages to migrate, if you do have Metacello configurations, why not to directly use them? Dale has an excellent post explaining exactly how to do that. Please read it if you are interested in such approach. Basically it consist of migrating only those packages of a Metacello version. Example:

| gofer |
"Identify the source repository"
gofer := Gofer new squeaksource: 'Fuel';
package: 'ConfigurationOfFuel' yourself.
"Traverse all packages defined in baseline '1.7-baseline' of the Fuel configuration"
(ConfigurationOfFuel  project version:  '1.7-baseline') packages do:[:spec |
"Use the #name and #package: messages and Gofer will fetch
all versions of the Monticello package"
gofer package: spec name ].
"Initiate the download with the #fetch command"
gofer fetch.

"Identify the target repository"
gofer := Gofer new url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel' yourself.
"Use the same algorithm for identifying the packages to be pushed"
(ConfigurationOfFuel project version: '1.7-baseline') packages do:[:spec |
gofer package: spec name ].
"Initiate the upload with the #push command"
gofer push.

The only difference with the previous approach is that this way we automatically migrate the packages referenced from the configuration.

Problems I found

Fetching and pushing take a long time, so be patient and try to be with a good internet connection. In my case, I have several errors which I have to deal with:

  •  Timeouts: If you get a “ConnectionTimedOut: Data receive timed out.” then it is clear there was a timeout with a certain package. What I did when I had this error is to open the debugger, go down in the stack a little while up to the place where I see it was doing the load of the version, and do a restart. Otherwise, you can always close the debugger and execute the code again, but this will take more time.
  • If you get the “Error: file is too short” then it means that there is some package (.mcz file) in your package-cache with zero bytes. Open the debugger, go down in the stack and identify which package/version has the problem. Then just go the package cache and remove such file. Then restart the migration procedure.

So, that’s all. I hope I could make your migration easier 🙂


Importing and exporting packages with Fuel

Background

Hi guys. One month ago, I was running like crazy preparing my presentations for ESUG 2011. I had 3 talks and also the Fuel presentation in the awards. So…I was thinking about examples and demos to show with Fuel. I finally showed how to serialize a debugger and continue debugging it in another image, how to serialize a graph and store it in a Riak NOSQL database, how to use SandtoneDB with Fuel, etc. But I was not the only one running. Martin Dias, main author of Fuel, who was in the other part of the world, was also coding. He was trying to finish what I will show today: to be able to export and import packages using Fuel. Reality is different from movies. Hence, he could finally make it work, one day after the awards 😦

Since that day, we only focused in Fuel itself and we left the tool there. This week I went to work to Lille and Igor asked me to show him such tool. The tool at that moment was outdated (using an old version of Fuel) and quite buggy. Today, we have something better 🙂

UPDATE: As part of his excellent blog and screencasts, James Robertson has recorded a screencast where he follows the steps of this blog post. I recommend to take a look to it as well. Thanks a lot James!

What is the tool about?

As you may know, Fuel is a plain object graph serializer. No more than that. However, Fuel supports correct serialization of Class, Trait, MethodDictionary, CompiledMethod, BlockClosure, MethodContext, etc. Hence, creating a tool on top of that which allows one to export and import packages, is quite easy. In fact the tools is right now about 220 lines of code.

The tool is called Fuel Package Loader, but its name may change in the future. Basically what it does is to take a PackageInfo, extract the classes and extension methods, create a FLPackage instance and serialize that. During materialization (the import), we need to do extra tasks like registering classes in Smalltalk globals, in categories/organizations, notify about class creation, send class side #initialize, etc. In other words, this tool adds package semantics to a plain serialization.

Installing Fuel and Package Loader

As always, you have two options, pick up the latest bleeding edge build from our Jenkis job, or install with Metacello. For this post, use the Metacello way. The last stable version of Fuel is 1.6. But we needed to fix a couple of things on it in other to make FuelPackageLoader to work. However, Fuel right now is not “stable enough” to release 1.7 (we need to fix few things). Hence, I have created a temporal version 1.6.1 tagged as #development specially for this post. All this is because I want the post to be reproducible. So, pick up a Pharo 1.3 image (sorry, for older Pharo images we need to fix something very stupid but I don’t have time, but for 1.7 it will be fixed), and execute:

Gofer new
squeaksource: 'Fuel';
package: 'ConfigurationOfFuel';
load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.6.1') load: #('Core' 'PackageLoaderWithTests').

That will load Fuel, FuelPackageLoader and all their tests. Now, one of the best tests we can do for this project is to import and export Seaside web framework. Why? Because it is really big, it has a lot of packages, dependencies between them, a lot of important class side #initialize, etc. So…if we are able to import/export seaside, we should be able to import/export most of the existing projects.

Save this image as ‘Fuel.image’.

Exporting packages

The first basic and naive idea we had when dealing with packages was to map one to one between Monticello packages and Fuel packages. Knowing which packages of Seaside are needed, in which other, and all the dependencies it has, is quite complicated. So…we can use Metacello for that. In the following example, we will install Seaside using Metacello and the regular packages of Monticello. But, we will at the same time, creat an associated Fuel package for each Monticello package. So, execute the following and wait between 10 and 30 minutes (if you are lazy, go directly to the import step where I provide the binaries):

FLMetacelloStore default
capturePackagesDuring: [
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfSeaside30';
load.
((Smalltalk at: #ConfigurationOfSeaside30) project version: '3.0.6')
load: #('Core' 'Tests' 'Zinc-Seaside' )]
prefixed: 'seaside'.

In this case, we have loaded the group ‘Core’ of Seaside, together with its tests and the Zinc seaside adaptor (just because I like it). Once that finishes, you can execute:

(ZnZincServerAdaptor port: 4242)
codec: GRPharoUtf8Codec new;
start.

And now if you open the internet browser and go to http://localhost:4242/  you should see the nice Seaside welcome page.

Now, pay attention that a directory called ‘FuelMetacello’ was created in the same directory where the .image is. Inside that directory you can see all the generated fuel packages. Each of them are named with the pattern ‘prefix-packageName-fuel’. For example, ‘seaside-Scriptaculous-Core.fuel’, ‘seaside-Seaside-Flow.fuel’, etc. Those would be the equivalents to the Monticello .mcz and we will import them after.

Finally, save this image as ‘FuelSeasideExporter.image’

Importing packages

To import packages, all we need is to read the file and materialize them. But the problem is, in which order? mmmm Metacello used to know that, but not us. Well, this is why during export, we have also written a file that contains the script to import back. The file is called ‘prefix-load-script.txt’ (seaside-load-script.txt in this case) and it is also in the ‘FuelMetacello’ directory. If you open it, you can see its contents:

#('seaside-ConfigurationOfGrease.fuel' 'seaside-ConfigurationOfKomHttpServer.fuel' 'seaside-ConfigurationOfSPort2.fuel' 'seaside-ConfigurationOfSwazoo2.fuel' 'seaside-ConfigurationOfZincHTTPComponents.fuel' 'seaside-ConfigurationOfRefactoringBrowser.fuel' 'seaside-Grease-Core.fuel' 'seaside-Grease-Pharo-Core.fuel' 'seaside-Grease-Tests-Core.fuel' 'seaside-Grease-Tests-Pharo-Core.fuel' 'seaside-Sport.fuel' 'seaside-Swazoo.fuel' 'seaside-Zinc-HTTP.fuel' 'seaside-Zinc-Tests.fuel' 'seaside-Seaside-Core.fuel' 'seaside-Seaside-Pharo-Core.fuel' 'seaside-Seaside-Component.fuel' 'seaside-Seaside-Canvas.fuel' 'seaside-Seaside-Pharo-Canvas.fuel' 'seaside-RSS-Core.fuel' 'seaside-Javascript-Core.fuel' 'seaside-Javascript-Pharo-Core.fuel' 'seaside-Comet-Core.fuel' 'seaside-Prototype-Core.fuel' 'seaside-Scriptaculous-Core.fuel' 'seaside-JQuery-Core.fuel' 'seaside-JQuery-UI.fuel' 'seaside-Seaside-Email.fuel' 'seaside-Seaside-Pharo-Email.fuel' 'seaside-Seaside-HTML5.fuel' 'seaside-Seaside-InternetExplorer.fuel' 'seaside-Seaside-Session.fuel' 'seaside-Seaside-RenderLoop.fuel' 'seaside-Seaside-Tools-Core.fuel' 'seaside-Seaside-Flow.fuel' 'seaside-Seaside-Examples.fuel' 'seaside-RSS-Examples.fuel' 'seaside-Seaside-Widgets.fuel' 'seaside-Seaside-Tools-Web.fuel' 'seaside-Seaside-Pharo-Tools-Web.fuel' 'seaside-Seaside-Environment.fuel' 'seaside-Seaside-Pharo-Environment.fuel' 'seaside-Seaside-Development.fuel' 'seaside-Scriptaculous-Components.fuel' 'seaside-Seaside-Swazoo.fuel' 'seaside-Zinc-Seaside.fuel' 'seaside-Seaside-Adaptors-Swazoo.fuel' 'seaside-Seaside-Tests-Core.fuel' 'seaside-Seaside-Tests-Pharo-Core.fuel' 'seaside-Seaside-Tests-Session.fuel' 'seaside-Seaside-Tests-RenderLoop.fuel' 'seaside-Seaside-Tests-Component.fuel' 'seaside-Seaside-Tests-Canvas.fuel' 'seaside-Seaside-Tests-Environment.fuel' 'seaside-RSS-Tests-Core.fuel' 'seaside-Javascript-Tests-Core.fuel' 'seaside-Javascript-Tests-Pharo-Core.fuel' 'seaside-Seaside-Tests-Email.fuel' 'seaside-Seaside-Tests-Functional.fuel' 'seaside-Seaside-Tests-Pharo-Functional.fuel' 'seaside-Seaside-Tests-Flow.fuel' 'seaside-Seaside-Welcome.fuel' 'seaside-Seaside-Pharo-Welcome.fuel' 'seaside-Prototype-Tests-Core.fuel' 'seaside-Scriptaculous-Tests-Core.fuel' 'seaside-Scriptaculous-Tests-Components.fuel' 'seaside-JQuery-Tests-Core.fuel' 'seaside-JQuery-Tests-UI.fuel' 'seaside-Seaside-Tests-HTML5.fuel' 'seaside-Seaside-Tests-InternetExplorer.fuel' 'seaside-Seaside-Tests-Examples.fuel' 'seaside-Seaside-Tests-Tools-Web.fuel' 'seaside-Seaside-Tests-Development.fuel' 'seaside-Seaside-Tests-UTF8.fuel' 'seaside-Seaside-Tests-Welcome.fuel' 'seaside-DynamicBindings.fuel' 'seaside-KomServices.fuel' 'seaside-KomHttpServer.fuel' 'seaside-Seaside-Tools-OmniBrowser.fuel' 'seaside-Seaside-Pharo-Tools-OmniBrowser.fuel' 'seaside-Seaside-FileSystem.fuel' 'seaside-Seaside-Tests-FileSystem.fuel' 'seaside-Seaside-Pharo-Continuation.fuel' 'seaside-Seaside-Tests-Pharo-Continuation.fuel' 'seaside-Seaside-Pharo-Flow.fuel' 'seaside-Seaside-Pharo-Development.fuel' 'seaside-Seaside-Tests-Pharo-Development.fuel' 'seaside-Seaside-Adaptors-Comanche.fuel' 'seaside-Comet-Pharo-Core.fuel' 'seaside-Comet-Examples.fuel' 'seaside-Comet-Tests-Core.fuel' 'seaside-Seaside-Tests-Adaptors-Comanche.fuel' 'seaside-Grease-Slime.fuel' 'seaside-Grease-Tests-Slime.fuel' 'seaside-Seaside-Slime.fuel' 'seaside-Seaside-Tests-Slime.fuel')
do: [ :packageFileName |
(FileDirectory default directoryNamed: 'FuelMetacello')
oldFileNamed: packageFileName
do: [ :aStream | FLPackageLoader new loadFrom: aStream binary ]]
displayingProgress: [ :packageFileName | 'Fuel importing ', packageFileName ]

So…. if you have followed the previous steps, then just open ‘Fuel.image’ (a new Pharo 1.3 image where Fuel is loaded but seaside is not), paste the previous code into a workspace and execute it. Of course that it depends on the machine, but here (Mac Book Pro i5), it only takes 10 seconds.

If you were a lazy gut who didn’t export the package, ok, here you have a zip of FuelMetacello. Just extract it and let the directory next to the .image.

It seems it was too fast to be working. So, if you don’t believe me, execute again the Zinc code to start a server (if you let the FuelSeasideExporter.image open, take another port) and then go to the browser. You should see, again the welcome page:

Ok, you still don’t believe it? Open the TestRunner, select all the seaside related tests and run it. They will be all green, or at least it should be exactly the same results as when running the tests from the FuelSeasideExporter image (where seaside was loaded with Metacello).

Why we put each package in a separate file? First because it was easier, and second because we wanted to be as much similar as possible with Monticello packages. In addition, this way, you can choose which packages to load. However, that doesn’t mean we are not able to serialize all packages in the same file, say seaside.fuel hahah. That’s possible, and it is easy since you can just download one file and you are done.

What the tool does and does not

As you may imagine the idea is that maybe in the future we can replace Monticello’ mcz with Fuel packages. Of course, we really far from there, but this was the first step. We tool right now:

  • Sends class side #initialize to classes.
  • Add classes to superclass instVar ‘subclasses’.
  • Send notifications that a class was created

What it does not do so far (but we will try to do it in the future) is :

  • Validations: we do not do validations at all. For example, we do not check whether a class we are materializing is already present in the image or if the global name was already taken. So we do not do class migration and things like that. In addition, we don’t validate class pools uniqueness and so on. So…right now the tool only works if you load in a clean image where you know there cannot be problems.
  • Source code is not serialized. That means, when you import and browse the classes, you will see the decompiled source.
  • .changes is not modified at all when loading a package.
  • Traits are partially supported and not really tested (soon will be fixed).

Conclusions

This tool is similar to VisualWorks Parcels, but there is a big difference. Parcels was designed with the idea in mind of managing code. Hence, it is a little bit “coupled” with that. It is difficult to use Parcels as a plain serializer. Fuel is a general-purpose serializer and coupled with nothing. It is an infrastructure where we can then build tools on top of that. FuelPackageLoader is just an example of that. In 220 lines of code we where able to have something more or less working for importing and exporting packages. We can export seaside (96 packages!!!) in 12 seconds and import them in 10. Ok, we still need to add a lot of missing features that will probably have overhead. But so far, the results are promising.

Do not use this tool for production. It is just an experiment and it is really really in development. Just try it for fun. If you want to help us, try importing/exporting your own packages instead of seaside and let us know if you have problems!

Finally, a big clap clap to Martin who has been working a lot in Fuel and the package loader.