“New” Tanker – current status

Both ESUG 2012 and GSoC have finished. At ESUG I gave a presentation about Fuel/Tanker (sorry, no slides because it was all demo). I also presented Tanker in the ESUG Awards….we got the 4th place, and only 8 points of difference with the 3er place :)   Anyway…I wanted to make public what exactly we finally managed to do with Tanker, a package exporter and importer that uses Fuel serializer. As I said in a previous post, we have changed a lot how Tanker works internally. In that post also I mentioned that Tanker didn’t support source code export/import, but in this new version, it does support methods source code, class comments, timestamps and everything that is currently being stored in the .changes or .sources file.

Exporting packages

The following figure shows how the export works. The input is a package of code (classes + extension methods), in this example “MyPackage”. The first step is to traverse the classes and methods of the package and do 2 things: a) write the source code (source of methods, class comments, timestamps, etc) to a text-based chunk format sources file (MyPackage.tankst); b) create “definition” objects. These objects represent a kind of object model that store all the necessary information of the code entities to be able to recreate them back  in another image. So we have for example, TAClass, TATrait, TAMethod, TABinding, TAAdditionalMethodState, TATraitAlias, TATraitComposition, etc…

Something key here is that besides all that information, we also store the relative position of the source code of the entity in the “MyPackage.tankst”. So for example, TAMethod instances will have a number representing that relative offset (similar to the pointer that CompiledMethods have in their trailer to .sources/.changes) where its source code is.

Another important detail is that in TAMethod we are storing the bytecodes, because even if we have the source code we want to avoid needing to compile during the import.

Once we have finished writing the sources file and creating the definitions, we just serialize them into a Fuel binary, say “MyPackage.tank”.

Importing packages

The following diagram shows how the import works. The input is the sources file (MyPackage.tankst) and the Fuel binary file (MyPackage.tank). The first step is basically to read the sources file and append all its contents in the .changes file (so that to get the same behavior as if we were installing a package with Monticello). But before doing that, we temporally store/keep the current “end” position of the .changes file ;)  (you already guessed why?).

The second step is to materialize the definitions using Fuel. Something very nice is that the definitions that were serialized, also understand the necessary messages to be installed in the system (#installUsing: aSourcesFile environment: anEnvironment). So we basically tell each class and each extension method to get installed in the system. Those objects will delegate to the rest of the definitions to complete their task (all definitions know what to do when installing them in the system).

Classes and traits will use the class builder to recreate themselves back. The source code was already installed in the .changes, so now we need to fix the “pointers” from our newly created classes/methods to point to the correct place of their source in the .changes file. And this is very easy because their position is just the “end” position of the .changes before installing our package (that value I told you we were temporally storing) + the relative position that was stored in the definition itself :)

So what is the biggest difference with Monticello for example? The key point is that we export bytecodes and we avoid having to compile during import. This make things faster while also being able to install packages without compiler (very useful for bootstrapping for example).

How to install it and use it

Tanker only works in Pharo 2.0 (because we rely on new class builder and layouts) so you first need to grab an image. Then, you can evaluate:

Gofer it
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) perform: #loadDevelopment.
Yes, so far there isn’t a stable version yet, we are waiting for new class builder. Be careful because that will install the new class builder (and change the system to use it)  and it may have some bugs…so take care :)
The export and import look like:
aPackage := TAPackage behaviors: {MyClass. MySecondClass} extensionMethods: {String>>#myExtensionMethod1. Object>>#myExtensionMethod2}.
(TAExport
 package: aPackage
 binariesOn: aBinaryWriteStream
 sourcesOn: aTextSourcesWriteStream)
 run.

(TAImport
 binariesFrom: aBinaryReadStream
 sourcesFrom: aSourcesReadStream)
 run.

However, we do not expect the final user to provide all the list of classes and extension methods. Therefore, we have helper methods to export and import RPackages and PackageInfos. Example:

 TAExport exportRPackageNamed: 'MyProjectCore'.
 TAExport exportRPackageNamed: 'MyProjectTests'.

TAImport importPackageNamed: 'MyProjectCore'.
 TAImport importPackageNamed: 'MyProjectTests'.

Conclusion

So that was all for today. Probably, I will do another blog post where I will show how we can query Metacello to know which packages to export and in which order, some details about the new class builder, some benchmarks while exporting all seaside/pier/magritte (10 seconds to export and 20 to import), and so on. The conclusion we got with this project is that indeed Fuel can be successfully used in yet another completely different domain. If you want to help us, please test it with your own packages. Right now we have only one open issue (need to recompile if superclasses present in the image have reshaped). I guess that soon we will fix this last issue and release the first stable version. In the future we plan to analyze how can we really integrate Tanker with Monticello/Metacello.

About these ads

3 responses to ““New” Tanker – current status

  • Frank Shearar

    Does that mean one could use Tanker in, say, Squeak, with the understanding that one would have to rewrite the part that loads stuff into the image (because Squeak doesn’t use Pharo’s ClassBuilder)?

    • marianopeck

      Hi Frank. It seems I was not clear. Tanker does not extend or needs anything in particular from the new class builder. We just believe it is way better, modular, and OO than the classical one. Therefore, we thought using it was a push on the new class builder, fix it (because we needed to fix some stuff), write lots of tests (it used to have no test), and we plan to integrate it in Pharo (maybe 2.1 or whatever). But in Squeak (or even older versions of Pharo 2.0) you can still use the old class builder. In fact, it is used in only ONE method of all Tanker: TAClass>>installUsing:environment: and it should be very easy to change it to use the old ClassBuilder.

  • Frank Shearar

    Hi Mariano, thanks for clearing up my question. I’m glad to hear that Tanker doesn’t care about how classes get built, and thanks for caring about having Fuel/Tanker in older/other images!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 26 other followers

%d bloggers like this: