Monthly Archives: September 2012

“New” Tanker – current status

Both ESUG 2012 and GSoC have finished. At ESUG I gave a presentation about Fuel/Tanker (sorry, no slides because it was all demo). I also presented Tanker in the ESUG Awards….we got the 4th place, and only 8 points of difference with the 3er place šŸ™‚ Ā  Anyway…I wanted to make public what exactly we finally managed to do with Tanker, a package exporter and importer that uses Fuel serializer. As I said in a previous post, we have changed a lot how Tanker works internally. In that post also I mentioned that Tanker didn’t support source code export/import, but in this new version, it does support methods source code, class comments, timestamps and everything that is currently being stored in the .changes or .sources file.

Exporting packages

The following figure shows how the export works. The input is a package of code (classes + extension methods), in this example “MyPackage”. The first step is to traverse the classes and methods of the package and do 2 things: a) write the source code (source of methods, class comments, timestamps, etc) to a text-based chunk format sources file (MyPackage.tankst); b) create “definition” objects. These objects represent a kind of object model that store all the necessary information of the code entities to be able to recreate them back Ā in another image. So we have for example, TAClass, TATrait, TAMethod, TABinding, TAAdditionalMethodState, TATraitAlias, TATraitComposition, etc…

Something key here is that besides all that information, we also store the relative position of the source code of theĀ entityĀ in the “MyPackage.tankst”. So for example, TAMethod instances will have a number representing that relative offset (similar to the pointer that CompiledMethods have in their trailer to .sources/.changes) where its source code is.

Another importantĀ detailĀ is that in TAMethod we are storing the bytecodes,Ā becauseĀ even if we have the source code we want to avoid needing to compile during the import.

Once we have finished writing the sources file and creating the definitions, we just serialize them into a Fuel binary, say “MyPackage.tank”.

Importing packages

The following diagram shows how the import works. The input is the sources file (MyPackage.tankst) and the Fuel binary file (MyPackage.tank). The first step is basically to read the sources file and append all its contents in the .changes file (so that to get the same behavior as if we were installing a package with Monticello). But before doing that, we temporally store/keep the current “end” position of the .changes file šŸ˜‰ Ā (you already guessed why?).

The second step is to materialize the definitions using Fuel. Something very nice is that the definitions that were serialized, also understand the necessary messages to be installed in the system (#installUsing: aSourcesFile environment: anEnvironment). So we basically tell each class and each extension method to get installed in the system. Those objects will delegate to the rest of theĀ definitionsĀ to complete their task (all definitions know what to do when installing them in the system).

Classes and traits will use the class builder to recreateĀ themselvesĀ back. The source code was already installed in the .changes, so now we need to fix the “pointers” from our newly created classes/methods to point to the correct place of their source in the .changes file. And this is very easy because their position is just the “end” position of the .changes before installing our package (that value I told you we were temporally storing) + the relative position that was stored in the definition itself šŸ™‚

So what is the biggest difference with Monticello for example? The key point is that we export bytecodes and we avoid having to compile during import. This make things faster while also being able to install packages without compiler (very useful for bootstrapping for example).

How to install it and use it

Tanker only works in Pharo 2.0 (because we rely on new class builder and layouts) so you first need to grab an image. Then, you can evaluate:

Gofer it
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
(Smalltalk at: #ConfigurationOfTanker) perform: #loadDevelopment.
Yes, so far there isn’t a stable version yet, we are waiting for new class builder.Ā Be careful because that will install the new class builder (and change the system to use it) Ā and it may have some bugs…so take care šŸ™‚
The export and import look like:
aPackage := TAPackage behaviors: {MyClass. MySecondClass} extensionMethods: {String>>#myExtensionMethod1. Object>>#myExtensionMethod2}.
 package: aPackage
 binariesOn: aBinaryWriteStream
 sourcesOn: aTextSourcesWriteStream)

 binariesFrom: aBinaryReadStream
 sourcesFrom: aSourcesReadStream)

However, we do not expect the final user to provide all the list of classes and extension methods. Therefore, we have helper methods to export and import RPackages and PackageInfos. Example:

 TAExport exportRPackageNamed: 'MyProjectCore'.
 TAExport exportRPackageNamed: 'MyProjectTests'.

TAImport importPackageNamed: 'MyProjectCore'.
 TAImport importPackageNamed: 'MyProjectTests'.


So that was all for today. Probably, I will do another blog post where I will show how we can query Metacello to know which packages to export and in which order, some details about the new class builder, some benchmarks while exporting all seaside/pier/magritte (10 seconds to export and 20 to import), and so on. The conclusion we got with this project is that indeed Fuel can be successfully used in yet another completely different domain. If you want to help us, please test it with your own packages. Right now we have only one open issue (need to recompile if superclasses present in the image have reshaped). I guess that soon we will fix this last issue and release the first stable version. In the future we plan to analyze how can we really integrate Tanker with Monticello/Metacello.