Tanker: transporting packages with Fuel

Hi all. You may have noticed that Tanker is starting to appear in some mails or in the Pharo issue tracker. Tanker is a project that Martin and I have been developing for a while and we are going to submit it this year to ESUG Innovation Technology Award. Therefore, I thought it would be interesting to explaining what it is, its current status, its goals, etc.

What is “Tanker” and what was “FuelPackageLoader”?

Right now the common way to export and import packages in Pharo is by using Monticello (or doing fileOut, which is almost the same). This ends up exporting the source code and then compiling it during the import. Tanker is a tool to export and import packages of code in a binary way using Fuel serializer. Using Fuel enables us to avoid having to compile from sources during the import. Tanker understands the concept of “packages of code” and the correct integration of them into the system. For example, it initializes classes, sends notifications, etc.

Tanker was first a prototype called “FuelPackageLoader” which was what I used for the example of exporting and importing seaside packages. In the last months, we have renamed the project to “Tanker”. Why? Because we do not want people to think that it is a Fuel project. In fact, Tanker is a simple USER of Fuel. Just as any other code that uses Fuel. This is why we have also moved it to its own repository.

Fuel has a package called “FuelMetalevel”. This package gives Fuel the knowledge of how to correctly serialize and materialize classes, metaclasses, traits, method dictionaries, compiled methods and closures, in other words, all the entities related to code and runtime infrastructure. It only knows how to serialize and materialize correctly. Nothing else. It does not initialize classes, it does not notify the system about the materialized classes, it does not install classes in Smalltalk globals, etc.

Current features, design and missing things

Right now, Tanker provides the following features:

  • It is able to export a package to a .tank file and import it in another image. The input for the export is a TAPackage which basically contains a list of classes and a list of extension methods. We are completely decoupled from the “package representation” (PackageInfo, RPackage, MCPackage, etc). However, we provide an API if you want to directly export from those types of packages.
  • Classes are initialized and installed in Smalltalk globals, events are sent, etc.
  • It has the ability to add additional user-defined objects to the package being exported (this is used, for example, for the Pharo generation from a kernel to store large/heavy class variables, tables and fonts).
  • It supports pre and post load actions represented as closures.

From the design point of view, Tanker:

  • Fully serializes classes and traits (not its “definitions”)
  • Does not use the ClassBuilder during materialization. Tanker itself materializes the “class objects” and sets the data.

So far, we are missing:

  • The possibility to export source code (right now classes and methods do not have source code) and also to install it in the .changes file during import.
  • Some validations during import. For example, the superclass of a class being installed may have changed its shape and, therefore, the classes to install need recompilation (because the bytecodes accessing instVars offset may be shifted or wrong). Or if a class already exists in the image and the shape has changed, we need to update the existing instances.
  • Integration with other tools like Monticello and Metacello.

Results with the “current status”

There are so far 3 real examples of Tanker:

How to install it and use it

Tanker will work only in the bleeding edge of Pharo 2.0. So I first recommend you to get an image from Jenkins. Then, you can install Tanker this way:

Gofer it
 url: 'http://smalltalkhub.com/mc/marianopeck/Tanker/main';
 package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) load.

To export a package and provide yourself the classes and extension methods, you can:

| aPackage aStream |
"Export"
aStream := 'demo.tank' asFileReference writeStream binary.
aPackage := TAPackage behaviors: {TestCase. TestLocalVariable. } extensionMethods: #().
TAPackageStore new storePackage: aPackage on: aStream.

"Import"
aStream := 'demo.tank' asFileReference readStream binary.
TAPackageLoader new loadFrom: aStream.

Then you can also use the API that provides helper methods to RPackage and PackageInfo:

aPackage := TAPackage fromPackageInfoNamed: 'MyPackage'

You also have #fromPackagesInfoNames:, #fromRPackageNamed: and #fromRPackagesNames:. Of course, there are more use-cases, API and scenarios. But, so far, that is the simplest usage. For more examples, browse the class side methods of TankerExamples.

GSoC and “new status”

The results so far are quite promising and not anymore a “proof of concept”. However, we still need to support source code management as well as the already mentioned pending features. Because of this reason, Martin submitted Tanker for the GSoC and fortunately it was accepted. So, right now we are moving to a different design to solve the requirements.  The idea now is NOT to serialize classes and traits, but instead serialize their “definition”. Imagine by “definition” the string used to create them. Then, during import, instead of just materializing class objects, we take the definition and, using the ClassBuilder or similar, we “evaluate” the definitions and we get the new classes.

At the same time, the idea is to export the source code of a package in a file (myPackage.tank.st or something like that) and the binary representation in another file (say myPackage.tank). Then, during import, you should be able to import with or without sources.

Side-effect projects of Tanker

You may be wondering why we didn’t start from the very beginning with the “definitions” way. Well, to be honest, the ClassBuilder is a mess, difficult to understand, maintain and extend. It was really hard trying to use it for our purpose. So the first “side-effect project” of Tanker is to continue pushing the “new ClassBuilder” started by Toon Verwaest based in “slots”. Martin Dias, Guillermo Polito and Camillo Bruni are pushing it and writing tests. I think it could be soon integrated in Pharo and replace the old one. The idea is that Tanker will use this ClassBuilder, for example, to evaluate the definitions.

When we are importing a class, it may happen that the superclass (present in the image where we are importing) has changed its shape (added or removed instVars, change supperclass, etc). If this is true, we have to recompile because the bytecodes accessing intsVars will be a shifted offset. However, recompiling is slow and we don’t want that. Therefore, Tanker will use IR (intermediate representation) which was developed by Marcus Denker and the team working with the new Opal compiler. IR is just a nice model generated from a CompiledMethod. The idea is that we can generate the IR, modify it (bytecodes for instVars accessing, for example) using this nice abstraction and API and then generate back a new CompiledMethod. This is way faster than recompiling. Furthermore, IR is decoupled from Opal so we don’t need whole Opal.

Conclusion

Tanker started as an experiment to see whether Fuel coud be used to export and import packages in a binary way. The proof of concept was quite good so we are now going forward with the source code management and related stuff. It is important to notice that Tanker just “uses” Fuel. Fuel is completely decoupled from Tanker. We think Fuel was well received by the community. We are doing our best so that Tanker gets positive feedback as well.

About these ads

7 responses to “Tanker: transporting packages with Fuel

  • Stéphane Ducasse

    This looks really amazing!
    Crazy guys I love you

  • Sebastian

    Really cool stuff indeed!

    I am looking forward to using it as a way to “share” (distribute) Pharo based projects.

    Could it be used that way?
    (export the packages, and re-import in a clean image)

    • marianopeck

      Hi Sebastian. Yes, definitely, it can be used to share Pharo based projects. You can share packages of code, but also you can use Fuel to share “plain objects”. That’s the exact goal: export packages and import them in another image. Right now we have completely refactored our work and now we do support sources. We still need to merge and refactor a couple of things and one small fix. In a couple of weeks, it will be ready and we will release the first version. I would wait for that and then give it a try.
      Cheers,

      • Sebastian

        Will wait then.

        I am very excited about many of the new Pharo projects that were started in the last few months (cairo graphics, NativeBoost, Fuel/Tank, AndroidVM, pluggable DSLs, continuous integration, GIT interface, and many many more).

        When they all mature, Smalltalk development will no doubt see a new era…

      • marianopeck

        Indeed, there a lot of really cool projects around. And yes, I am also waiting for that new era :)

  • “New” Tanker – current status « Mariano Martinez Peck

    [...] to do with Tanker, a package exporter and importer that uses Fuel serializer. As I said in a previous post, we have changed a lot how Tanker works internally. In that post also I mentioned that Tanker [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 26 other followers

%d bloggers like this: