Moving contexts and debuggers between images with Fuel

Hi guys. During ESUG 2011, at the Awards, I was showing Fuel. The week before such event I was thinking what I could show to the people. This was a challenge because showing a serializer can be plain boring. I was working at home that afternoon, and suddenly I thought: “What happens if I try to serialize a living debugger and materialize it in another image?” After 5 minutes, really, you will see it takes only 5 minutes, I notice that such crazy idea was working OUT OF THE BOX. Even if I knew Fuel supported serialization of methods, contexts, closures, classes, etc…I was surprised that it worked from the first try. I was so happy that I tried to explain to my poor wife what I had just done hahahah. Unfortunately, she told me it was too abstract and that understanding the garbage collector was easier (I promise she really understands what the garbage collector does hahhahaha).

Well….several months has passed, but I would like to show you how to do it because I think it may be of help for real systems ;)  So…the idea is the following: whenever there is an error, you can get the context from it, and such context is what is usually written down into a log file (in Pharo this is PharoDebug.log). I will show you two things: 1) how to serialize a debugger in one image and materialize it another one and; 2) how to write down the context into a Fuel file when there is an error so that you can materialize it later in another image.

Installing Fuel

The first step is, of course, install Fuel. The latest stable release is 1.7 but to have better results with this example, I would recommend 1.8. Fuel 1.8 is not released yet it is because we plan to write some stuff in the website. The code is almost finish, so you should load Fuel 1.8 beta1. In my case I am using a normal Pharo 1.3 image:

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel';
load.
((Smalltalk at: #ConfigurationOfFuel) project version: '1.8-beta1') load.

Once you have finished loading Fuel, save the image. Let’s call it Fuel.image.

Serializing and materializing a debugger

Now its time to do something hacky in the image so that to open a debugger. Or you can just take a piece of code and debug it. In my example, I opened a workspace and wrote the following:

| a |
a := 'Hello Smalltalk hackers. The universal answer is '.
a := a , '42!'.
Transcript show: a.

Then I select the whole code, right click -> “debug it”. Then I do one “step over” and I stop there before the concatenation with ’42!’.

I am sure there could be better ways, but the simpler way I found to get the debugger instance for this example, is to do a Debugger allInstances first ;)  so… be sure not to have another debugger opened hahaha.  Now…let’s serialize the debugger:

Smalltalk garbageCollect.
FLSerializer
serialize: Debugger allInstances first
toFileNamed: 'debugger.fuel'.

After that, you should have a ‘debugger.fuel’ created in the same directory where the image is. Now close your image (without saving it) and re open it. If everything is fine, we should be able to materialize our debugger and continue debugging. So, let’s try it:

| newDebugger |
newDebugger := FLMaterializer materializeFromFileNamed: 'debugger.fuel'.
newDebugger openFullMorphicLabel: 'Materialized debugger ;)'.

So????  Did it work??  are you as happy as me when I first saw it? :)  if you check this new opened debugger, you will see its state is correct. For example, the instVar ‘a’ has the correct state. You can now open a Transcript and continue with the debugger as if were the original one.

Of course that even if this simple example works, there are a lot of problems. But I will explain them at the end of the post.

Serializing and materializing errors

In the previous example we have serialized the debugger manually. But imagine the following: you have a production application running. There is an error, and PharoDebug.log is written with all the stack. The user/client send you by email the .log and you open your favorite text editor to try to understand what happened. Now imagine the following: you have a production application running. There is an error, and a PharoDebug.fuel is written with all the stack. The user/client send you by email the file and you open an image, and then materialize and open a debugger. How does it sound? :) magical?

For this example, we will just change the place where Pharo writes PharoDebug.log when there is an error. That method is #logError:inContext:. What we will do is to add just 2 lines at the beginning to serialize the context:

logError: errMsg inContext: aContext

" we should think about integrating a toothpick here someday"
FLSerializer
serialize: aContext
toFileNamed: 'PharoDebug.fuel'.

self logDuring: [:logger |
logger
nextPutAll: 'THERE_BE_DRAGONS_HERE'; cr;
nextPutAll: errMsg; cr.

aContext errorReportOn: logger.

"wks 9-09 - write some type of separator"
logger nextPutAll: (String new: 60 withAll: $- ); cr; cr.
]

Now yes, let’s execute something that causes an error. What I did is to evaluate 1/0. After that, you should see the file PharoDebug.fuel in the same directory where the image is. You can now close the image and reopen it. And then, let’s reopen de debugger:

| aContext |
aContext := FLMaterializer materializeFromFileNamed: 'PharoDebug.fuel'.
Debugger openContext: aContext label:  'This is the new debugger!' contents: nil

Et voilà! Hopefully that worked :)   Notice that in this example and the previous one, there is nothing in special with the Fuel serialization. You are using the normal API, and all you do is to serialize a debugger or a context as if you were serializing any normal object. Of course, you can also apply this idea to other places. For example, in Seaside you have an error handler. You may want to serialize the error with Fuel there.

Limitation and known problems

  • Even if Fuel can fully serialize methods, classes, traits, etc., it is recommended that the image were the contexts/debuggers are serialized and materialized are equal. If you are doing this in a production application, then you can have the same image running locally. The idea is that both images have the same classes and methods installed. This is because, by default, if the object graph to serialize includes compiled methods, classes, class variables, etc., they are all considered as “globals”, which means that we only serialize its global name and then during materialization it is searched in Smalltalk globals. Hence, classes and methods have to be present. Otherwise you have to use Fuel in a way that it serializes classes as well, but that’s more complicated.
  • There may be things that affects the debugger which are part of the image and not the serialization, and they may have changed. Imagine for example, a class variable which has changed its value in the image where you serialize. Then it will have a different value in the image where you materialize. Most of these problems also happens if even if you open the debugger later in the same image…some state may have changed…
  • The graph reachable from the contexts can be very big. For example, Esteban Lorenzano was doing this for an application and one of the problems is that from the context it was reachable the whole UI…which means lots and lots of objects. In such a case, you can always use Fuel hooks to prune the object graph to serialize.
  • Be aware to use exactly the same version of Fuel in both images ;)

Conclusion

All in all, I think that as a very first step, it is very nice that we can serialize this kind of stuff like contexts and debuggers out of the box with Fuel. This could be the infrastructure for a lot of fancy stuff. I don’t think that the debugger materialization can be as reliable as if you were debugging in the original image. I don’t think either that it should replace PharoDebug.log. However, what I do think is that you can add the Fuel serialization just as another way of getting more information about the problem. It’s one more tool you can add to your Smalltalk toolbox :)

About these ads

16 responses to “Moving contexts and debuggers between images with Fuel

  • Udo Schneider

    That’s insane! :-)

    I already had a hard time explaining live debugging to the curly braces guys. Can you guess how hard will be for them to swallow the fact that you can move a living debugger incl. state from one image to the other? Have some mercy :-)

    • Sebastian Nozzi

      Smalltalk magic at work indeed!

      Very promising, and well explained post. Thanks!

      • marianopeck

        Thanks Sebastian. Indeed, Smalltalk magic. Imagine that you can not only serialize contexts and closures, but also processes. I haven’t time to go further in the details, but imagine fancy stuff about distributed work by passing arround processes or closures to execute, or remote debugging, or Seaside sessions replications (no need anymore of server afinity), or ….

      • Sebastian Nozzi

        No matter how “imperfect”, this is still infinitely better than being confronted with a “dead” (and often obscure) stack-trace in a log file.

        More than once on a Java project we wish we could have had access to the “live” stack-trace, examine any stack-frame at will, see the values of parameters, fields, local variables, etc.

        In projects that deploy stable “staging” releases the stack-trace COULD be opened in a copy of the production image. This would likely save countless of man-hours lost to trying to diagnose the problem in an environment radically different than your developer desk (production vs. development).

        If Smalltalk was more popular, it would be really interesting to make a study as to where most man-hours get lost: automatic seamless safe refactoring (on which Eclipse would seem to have some kind of edge) or problem diagnosing (where Smalltalk definitely has the edge).

        I like to think about Smalltalk-based diagnosing vs. Java-based (or any other environment not supporting live exploration) with this analogy: whereas Java-based diagnosing would be already getting the dead body on which you are NOT EVEN ALLOWED to practice an autopsy, Smalltalk-based is more similar to going to the doctor to get an examination (the doctor can examine anything to any depth he likes, to the molecular level if he so wishes, interact with your “illness”, and treat it on the fly ;-)

      • marianopeck

        Amen. Nice analysis. Indeed, it is an interesting topic. In my case I will contribute by continue building the base infrastructure (Fuel for example). Hopefully could be people in the future that build such funcy stuff.

    • marianopeck

      hahahahaha nice comment :) Sorry, I have no mercy for them ;) Now, seriously. I am sure that what I showed does not work 100% of the cases nor that it is really reliable. However, the fact of at least being able to do it for certain scenarios is amazing. In these kind of cases is when I realize the magic and the advantages of having reification of contexts, methods and classes in the image… Maybe the curly braces guys do not get the advantages of having first class objects for these kind of stuff. Until you show them something cool :)

  • Géal

    That’s amazing! Reproducing a bug takes a lot of time, so jumping right in the debugger is a big productivity gain :)

    Will you use it on http://code.google.com/p/pharo/ ?

  • François Stephany

    Mariano, do you know the size of Esteban Lorenzano’s serialization you were talking about?

    That’s pure curiosity. Have you ever imagined to have something like http://airbrakeapp.com? That would be a great business idea (not sure about the volume of potential Smalltalkers client though…)

    • marianopeck

      Hi Francois. Esteban has 2 interesting problems which were somehow related:
      1) He was doing a DESKTOP application with Glamour. From the context of the error, he could reach his UI/Windows/Browsers objects. For this app, he had A LOT of data inside the image (kind of a small database in memory). And such lot of data was reachable from the UI. Hence…everything was serialized. In this case the file was approx. 7MB. The main problem was not that one but the following.
      2) Since the graph to serialize was big (approx. 7MB) it took between 1 and 2 seconds. Problem was that the UI could change during that time. And if the graph changes while Fuel is serializing it…then there will be an error.

      The solution was pretty easy since we used a hook of Fuel to make transient the root of his “in-memory database” in the UI. The file becomed 500kb and the stack was written instantaniously.
      I wasn’t aware of airbrakeapp. It looks interesting. The only problem is that it doesn’t look open source, right? I would love something like that for Pharo/Seaside.

  • renemendoza

    Mariano: There is an opensource airbrake compliant server called errbit: https://github.com/errbit/errbit

  • Reviving CI test failures in local machine « Mariano Martinez Peck

    [...] In Pharo, the stack of the running system is reified also from the language side and we can access them! (we can even modify them). We have instances of MethodContext which hold an instVar ‘sender’ that refers to the next sender in the stack (another MethodContext or nil if it is the last). Apart from ‘sender’, a context also includes the receiver, the method that caused its activation, the arguments and the temporal variables. The Fuel serializer can serialize any type of object including MethodContext. If we can serialize a MethodContext (and closures and methods), we can serialize a stack, right? And what does this mean? Well, it means that we can serialize a debugger with its current state. I have already shown several times (at ESUG Innovation Technology Award and at PharoConf) how we can use Fuel to serialize a debugger (from image X) in the middle of its execution and materialize it i…. [...]

  • Smalltalk: Links, News And Resources (4) « Angel ”Java” Lopez on Blog

    [...] contexts and debuggers between images with Fuel « Mariano Martinez Peck http://marianopeck.wordpress.com/2012/01/19/moving-contexts-and-debuggers-between-images-with-fuel/ Hi guys. During ESUG 2011, at the Awards, I was showing Fuel. The week before such event I was [...]

  • Pharo 2.0 Released | Playing with Objects

    [...] Pharo wants to be fast. And that’s something NativeBoost and Fuel achieve. That’s why you can find them included by default in the system. NativeBoost (by Igor Stasenko) gives us the ability to execute machine code from the language side, and a new generation FFI with callbacks. Use it with caution :). Fuel, written by Mariano Martinez Peck and Martin Dias, is a cool object serializer focusing on fast deserialization (materialization), and the ability to serialize any kind of objects: Block closures? yes. Contexts? yes. Complete debuggers so we can restore them and debug failures in other environments? YES. [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 26 other followers

%d bloggers like this: