Tag Archives: CompiledMethod

Named Primitives

In the previous post we saw different things: what is a primitive and some examples, their impact on CompiledMethod instances, pragmas, etc. Continuing with this “Journey through the Virtual Machine”, today I will talk about Named Primitives.

Important data from previous post

What is important for this post is a summary of what a primitive is. As we saw, there are methods that can be implemented in such a way that they call a Virtual Machine primitive. To declare the information related to which primitive to use, we use Pragmas. Example of the method #class:

Object >> class
"Primitive. Answer the object which is the receiver's class. Essential. See
Object documentation whatIsAPrimitive."

<primitive: 111>
self primitiveFailed

In this case, the primitive is the number 111. The primitive is implemented in the CORE of the Virtual Machine. This core is written in Slang, a subset of Smalltalk.  To see how to map primitive numbers to their implementation we can see the method StackInterpreter >> #initializePrimitiveTable. In this example, for example, we can see it is mapped to the method #primitiveClass. But don’t confuse, this is NOT a regular method. This is part of the VM (the package VMMaker) and that method is automatically translated to C while building the VM.

For more details, please read the previous posts of this blog.

Named Primitives vs. Numbered Primitives

Again, in the previous post, we saw a “weird” method like:

FileDirectory >> primDeleteFileNamed: aFileName
"Delete the file of the given name. Return self if the primitive succeeds, nil otherwise."

    <primitive: 'primitiveFileDelete' module: 'FilePlugin'>
    ^ nil

Which are the differences between this primitive and the previous one (#class)? Well…let’s see:

With “numbered primitives” like #class, those primitives are implemented in the VM core, that is, the code of the primitives is inside Interpreter classes. There is a table kept in the VM that maps numbers to methods which are then translated to C functions. The only thing is needed to know from image side to call a primitive is the primitive number. In addition, these primitives cannot be loaded dynamically and hence, it is not easy to extend the VM with new primitives. If that is desired one need to build a new VM wich such primitive and distribute that VM.

Named primitives are different. They can be written with Slang as well, but they are not part of what I call the “VM core”. The methods that implement those primitives are not part of the Interpreter classes. Instead, they are written in different classes: plugins. What is needed to know from image side to call a named primitive is the name and its module. What is a module? Let’s say that it is the plugin name. Contrary to numbered primitives, named ones can be loaded dynamically and hence, it is easy to extend the VM with new primitives. One can generate the binaries of the plugin and distribute it with the regular VM. Named primitives can reside in an external library (.so on Unix, DLL on Windows, etc).

Named Primitives / Plugins / Pluggable Primitives

So…do they all mean the same?  Yes, at least for me, they all represent something similar. For me, named and pluggable primitives are the same concept. And I see a plugin like a set of named/pluggable primitives.

When someone says “this is done with a plugin” or “did you load the plugin”, they refer to that. Even if in a future post we will see how to implement our custom plugin, I will give a small introduction.

Plugins are translated to a different C file, not to the same C file of the VM (result of Interpreter classes translation). In fact, plugins are translated and placed in the directory /src/plugin. Each plugin is implemented in the VM as a subclass of InterpreterPlugin. Just for fun, inspect “InterpreterPlugin allSubclasses”. Usually, a plugin needs functionality provided by the VM core. For this purpose, the class InterpreterPlugin has an instance variable InterpreterProxy, which acts as its name says, as a proxy to the Interpreter (the vm). InterpreterProxy provides only the methods that the VM wants to provide to primitives. Some examples are #fetchInteger:ofObject:, #pop:thenPush:, #superclassOf:, etc….So, plugins can only use those provided methods of the interpreter.

We saw that from the image side, named primitives are implemented using the following pragma: “<primitive: ‘primitiveXXX’ module: ‘YYYPlugin’>”. For example, “<primitive: ‘primitiveFileDelete’ module: ‘FilePlugin’>”. The first parameter is the primitive name, which has to map to the method that implementes such primitive (notice the difference with the table for numbered primitives). So in this case, there must be a method (implemented in Slang) called #primitiveFileDelete. The second parameter is the plugin name. A plugin is rified as a subclass of InterpreterPlugin and the plugin name can be defined by implementing the method #moduleName. If a plugin does not do that then the class name is used by default, as it happens with FilePlugin. So….FilePlugin is a subclass of InterpreterPlugin and implements the method #primitiveFileDelete, which looks like:

primitiveFileDelete

| namePointer nameIndex nameSize  okToDelete |

<export: true>

namePointer := interpreterProxy stackValue: 0.
(interpreterProxy isBytes: namePointer)
ifFalse: [^ interpreterProxy primitiveFail].
nameIndex := interpreterProxy firstIndexableField: namePointer.
nameSize := interpreterProxy byteSizeOf: namePointer.
"If the security plugin can be loaded, use it to check for permission.
If not, assume it's ok"
sCDFfn ~= 0
ifTrue: [okToDelete := self cCode: ' ((sqInt (*)(char *, sqInt))sCDFfn)(nameIndex, nameSize)'.
okToDelete
ifFalse: [^ interpreterProxy primitiveFail]].
self
sqFileDeleteName: nameIndex
Size: nameSize.
interpreterProxy failed
ifFalse: [interpreterProxy pop: 1]

How plugins are compiled with the VM, as well as telling the VM which plugins to compile, is explained in a previous posts such as this one and this one.

Plugins: internal or external?

Plugins can be compiled in two ways: internal or external. Notice that it is just the way they are compiled, but the way they are written is the same: using SLANG. Each plugin is a class subclass of InterpreterPlugin or SmartSyntaxInterpreterPlugin. A plugin can then be compiled in the mentioned ways.

Internal plugins are linked together with the core of the classical VM, that is, the binaries of the plugins are put together with the binary of the VM. So for the final user, there is just one binary representing the VM. External plugins are distributed as separate shared library (a .dll in windows, a .so in Unix, etc). The functions (remember that slang is then translated to C so what we coded as methods will become C functions hahaha) of the shared libraries representing the plugins are accessed using system calls.

Which one to use?  Well, that depends on what the developer of the plugin wants. In my case I usually try to build them externally since you don’t need to do anything at all to the VM. It is easier to distribute: just compile the plugin and use it with a regular VM. And from security point of view they are even simpler to eliminate or disable, just removing the binary file.

But not everything is pink in this world. Unfortunately, there are some plugins that cannot be compiled in both ways, but with one in particular. Most existing plugins are optional. Nevertheless, there are some plugins that are mandatory for the core of the VM, that is, the VM cannot run without those plugins. There are lots of available plugins. Which ones are needed? Some plugins only work in certain Operating System. Some only work not even in certain OS but also in a particular version. Plugins may need different compiler flags in different OS. Etc…

To solve the problem of knowing all that, CMakeVMMaker provides an easy way to compile the plugins of a VM. I assume you have been following this “journey” so you read how to compile the VM from scratch in https://marianopeck.wordpress.com/2011/04/10/building-the-vm-from-scratch-using-git-and-cmakevmmaker/ and https://marianopeck.wordpress.com/2011/04/16/building-the-vm-second-part/. So if you installed ConfigurationOfCog, you installed CMakeVMMaker as well. Check for the methods #defaultInternalPlugins and #defaultExternalPlugins. Each CMakeVMMaker configuration class implements those methods correctly. Each of them knows which plugins should be compiled and whether internally or externally. So, the user, someone who wants to build the VM, doesn’t need to worry about that. In addition, CMakeVMMaker let us customize which plugins to use with the method #internalPlugins: and #externalPlugins.

I know, I know.  You want to write and compile your own plugin? Ok, there will be a future post about that. But if you want to try it, check subclasses of InterpreterPlugin or SmartSyntaxInterpreterPlugin  (I recommend the last one since makes a lot of stuff simpler) and then build the VM with something like:

| config |
config := CogUnixConfig new.
config externalPlugins: (config externalPlugins copyWith: #MyHackyFirstPlugin).
config generateWithSources.

Named Primitives and their relation to CompiledMethod

In the previous post we saw that methods that contained a numbered primitive have something special in the CompiledMethod instance: the penultimate literal does not have the Symbol containing the selector but instead an instance of AdditionalMethodState which has a pragma with the primitive information. In the case of numbered primitives we have that, but in addition, there is one more special object in the first literal of the CompiledMethod. That object is an Array that with 4 elements. The first is the plugin name, which is answered by #moduleName (what you put in the module:). The second one is the selector. The third is the session ID which is obsolete, not used anymore, and hence it is usually zero. The last one, is the function index (Integer) in a table that resides in the VM: externalPrimitiveTable. As far as I understood, such table and this index is used as a cache. What is funny is that the VM writes that index in the CompiledMethod instance. For more details, read the method #primitiveExternalCall.

Links

As always, if there are more links or documentation about them please let me know and I will add it.


Primitives, Pragmas, Literals and their relation to CompiledMethods

What is a primitive?

Do you want to know the answer?  Just do what we always do in Smalltalk: browse code 🙂   Open your image and browse the method #whatIsAPrimitive. You can read the following information there:

“Some messages in the system are responded to primitively. A primitive response is performed directly by the interpreter rather than by evaluating expressions in a method. The methods for these messages indicate the presence of a primitive response by including <primitive: xx> before the first expression in the method.  

Primitives exist for several reasons. Certain basic or ‘primitive’ operations cannot be performed in any other way. Smalltalk without primitives can move values from one variable to another, but cannot add two SmallIntegers together. Many methods for arithmetic and comparison between numbers are primitives. Some primitives allow Smalltalk to communicate with I/O devices such as the disk, the display, and the keyboard.  Some primitives exist only to make the system run faster; each does the same thing as a certain Smalltalk method, and its implementation as a primitive is optional.  

When the Smalltalk interpreter begins to execute a method which specifies a primitive response, it tries to perform the primitive action and to return a result. If the routine in the interpreter for this primitive is successful, it will return a value and the expressions in the method will not be evaluated. If the primitive routine is not successful, the primitive ‘fails’, and the Smalltalk expressions in the method are executed instead. These expressions are evaluated as though the primitive routine had not been called. 

The Smalltalk code that is evaluated when a primitive fails usually anticipates why that primitive might fail. If the primitive is optional, the expressions in the method do exactly what the primitive would have done (See Number @). If the primitive only works on certain classes of arguments, the Smalltalk code tries to coerce the argument or appeals to a superclass to find a more general way of doing the operation (see SmallInteger +). If the primitive is never supposed to fail, the expressions signal an error (see SmallInteger asFloat). 

Each method that specifies a primitive has a comment in it. If the primitive is optional, the comment will say ‘Optional’. An optional primitive that is not implemented always fails, and the Smalltalk expressions do the work instead.  If a primitive is not optional, the comment will say, ‘Essential’. Some methods will have the comment, ‘No Lookup’. See Object >> #howToModifyPrimitives for an explanation of special selectors which are not looked up. 

For the primitives for +, -, *, and bitShift: in SmallInteger, and truncated in Float, the primitive constructs and returns a 16-bit LargePositiveInteger when the result warrants it. Returning 16-bit LargePositiveIntegers from these primitives instead of failing is optional in the same sense that the LargePositiveInteger arithmetic primitives are optional. The comments in the SmallInteger primitives say, ‘Fails if result is not a SmallInteger’, even though the implementor has the option to construct a LargePositiveInteger. For further information on primitives, see the ‘Primitive Methods’ part of the chapter on the formal specification of the interpreter in the Smalltalk book.”

Primitives examples

As we will see later in another post, in the object header of every object (except compact classes) there is a pointer to its class (another object). Hence, accessing to that pointer of the object header has to be done by a primitive:

Object >> class
"Primitive. Answer the object which is the receiver's class. Essential. See
Object documentation whatIsAPrimitive."

<primitive: 111>
self primitiveFailed

We read in the comment of the method #whatIsAPrimitive that what it is after <primitive: XXX> is ONLY called when the primitive fails. In this case, when that happens, the code “self primitiveFailed” will be executed: there is nothing we can do from image side if this primitive fails. Notice that the declaration of <primitive: XXX> has to be first in the method. The only possible thing before that is comments and declare temp variables: it is not possible to write code before that. So, this is not possible:

Object >> class
"Primitive. Answer the object which is the receiver's class. Essential. See
Object documentation whatIsAPrimitive."
Transcript show: '#class was called!!'.
<primitive: 111>
self primitiveFailed

Another example of a primitive:

SmallInteger >> bitOr: arg
"Primitive. Answer an Integer whose bits are the logical OR of the
receiver's bits and those of the argument, arg.
Numbers are interpreted as having 2's-complement representation.
Essential.  See Object documentation whatIsAPrimitive."

<primitive: 15>
self >= 0 ifTrue: [^ arg bitOr: self].
^ arg < 0
ifTrue: [(self bitInvert bitAnd: arg bitInvert) bitInvert]
ifFalse: [(self bitInvert bitClear: arg) bitInvert]

In this case, if the primitive fails, this method tries to resolve its task in Smalltalk code. Sometimes this works and it means that this primitive is for improving performance, but not mandatory (as it is the case with #class).  In other cases, the code after the primitive (written in Smalltalk) will fail for sure if the primitive has already failed. However, such code is put in Smalltalk with documentation purposes. You can imagine what such primitive does (and why it could fail) in the VM side (Slang/C) by looking its possible code in Smalltalk.

Two important literals

When we talked about CompiledMethod and literals I forgot to mention that there are 2 literals in every CompiledMethod that are really important. CompiledMethod can answer to the messages #methodClass (which answers the class where such CompiledMethod is installed) and #selector (which answers the method’s selector). How can both methods be implemented in CompiledMethod if they don’t hold such information?  Ok, they do hold such information as literals. The LAST literal of every CompiledMethod is an Association where the key is the class name and the value the class object. The penultimate literal stores the selector. So if we explore “Date >> #month”:

Literal 3 is the last one and points to the Association and literal 2 is the penultimate and points to the selector.

So you can now understand the methods:

CompiledMethod >> methodClass
"answer the class that I am installed in"
^self numLiterals > 0
ifTrue: [ (self literalAt: self numLiterals) value ]
ifFalse: [ nil ]

 

CompiledMethod >> selector
"Answer a method's selector.  This is either the penultimate literal,
or, if the method has any properties or pragmas, the selector of
the MethodProperties stored in the penultimate literal."
| penultimateLiteral |
^(penultimateLiteral := self penultimateLiteral) isMethodProperties
ifTrue: [penultimateLiteral selector]
ifFalse: [penultimateLiteral]

Forget for the moment the #isMethodProperties.

Pragmas and CompiledMethods

Now…when we talk about the <primitive: XXX>, what’s that??  it is not a regular message send. How can that be compiled by the Compiler? Ok, these are called “Method tags” and their goal is to store metadata of the method. If you are a java developer, method tags can be “similar” to Java annotations. In Pharo Smalltalk, one implementation of method tags is called “Pragmas”. I won’t discuss the advantages or disadvantages of Pragmas against other method tag implementations, or whether to use pragmas o regular subclassification, etc.

For more information about Pragmas, check the class comment of Pragma class and the tests like PragmaTest, MethodPragmaTest, etc. Nowadays, Pragmas are used in Pharo in several places like the new settings framework, the world menu, Metacello, HelpSystem, etc.

Ok…nice. But how are they really stored in a CompiledMethod? Let’s explore “SmallInteger >> #bitOr:”.

So….as we can see in the explorer, at compiling time the Compiler creates an instance of AdditionalMethodState and such object is placed in the penultimate literal. The class comment of AdditionalMethodState says: “I am class holding state for compiled methods. All my instance variables should be actually part of the CompiledMethod itself, but the current implementation of the VM doesn’t allow this.  Currently I hold the selector and any pragmas or properties the compiled method has.  Pragmas and properties are stored in indexable fields; pragmas as instances of Pragma, properties as instances of Association.”

AdditionalMethodState has two named instance variables: ‘method’ and ‘selector’. No explanation needed here. But since the class format is variable (do you remember them from my old post?) it can also store indexable fields. In this case, pragmas are stored that way. Hence, an instance of Pragma is stored in AdditionalMethodState and that’s what we can see in the explorer. A Pragma instance has 3 instance variables:  ‘method keyword arguments’.

But the AdditionalMethodState instance is put in the penultimate literal and that’s where the “selector” should be found. How can “CompiledMethod >> #selector”  work with them? If we now take again a look to such method (look above), you will see there is a “isMethodProperties ifTrue: [penultimateLiteral selector]”. Of course, AdditionalMethodState answers true to isMethodProperties and hence the selector is asked to itself (which in fact is an instance variable of it).

Primitives and their impact in CompiledMethod

Since primitives uses Pragma, the first effect is to have an AdditionalMethodState in the penultimate literal instead of a selector. The second effect, is that the primitive number is stored in the CompiledMethod header. You can send the message #primitive and get the value. For example, “(SmallInteger >> #bitOr:) primitive” -> 15. If the method has no primitive then zero is answered. Example, (TestCase >> #assert:) primitive -> 0.

When the VM executes a CompiledMethod it checks whether it is a primitive method or not (checking whether the value in the object header is zero or bigger). If it is, the VM searches in a table and dispatches the  primitive associated to the number.

How primitives are map to the VM side?

Continuing with SmallInteger >> #bitOr:, the primitive number is 15. How can we know the code of such primitive in the VM side? Time to open an image with VMMaker (if you don’t know how to do it read the title “Prepared image for you” in this post). The VM keeps a table that maps primitive numbers with selectors implemented in the interpreter class. The most useful advice here is to check the method that initialices such table: #initializePrimitiveTable. So we can take a look:

For our example of #class the primitive number was 111. In such table 111 maps to #primitiveClass. So we can browse its code. Remember that this code is written in SLANG and it is part of the VMMaker package (check my previous posts for details).

primitiveClass
| instance |
instance := self stackTop.
self pop: argumentCount+1 thenPush: (objectMemory fetchClassOf: instance)

(SmallInteger >> #bitOr:)  has primitive number 15, which maps to #primitiveBitOr, which code is:

primitiveBitOr
| integerReceiver integerArgument |
integerArgument := self popPos32BitInteger.
integerReceiver := self popPos32BitInteger.
self successful
ifTrue: [self push: (self positive32BitIntegerFor:
(integerReceiver bitOr: integerArgument))]
ifFalse: [self unPop: 2]

So..you have learnt how to map primitive numbers with methods in VM side 🙂   You already know how to do that for primitives and bytecodes now. Congrats!!!

Future explanations

Browse de method #primDeleteFileNamed: and you will see something like:

primDeleteFileNamed: aFileName
"Delete the file of the given name. Return self if the primitive succeeds, nil otherwise."

^ nil

what’s that primitive? where is the number?  Can I create my own primitive? Sure! We will see how to do that in a future post 🙂


Introduction to Smalltalk bytecodes

Hi all. It this post I will give you a quick overview and introduction to bytecodes. I won’t talk that much because this topic is well explained in the Blue book, in the code, etc. In the previous posts we saw that a CompiledMethod is all about bytecodes and literals (ok, and a header and a trailer). To really follow the post, I recommend you to have an image with VMMaker. If you don’t know how to do it, please see the title “Prepare the image” of a previous post.

Bytecodes introduction

Let’s start from the beginning: What is a bytecode?  A bytecode is a compact and platform neutral representation of machine code instructions, interpreted by VM. When you code and then save a method in Smalltalk, the Compiler generates an instance of CompiledMethod. Your method’s code is decomposed by the Compiler into a set of basic instructions so that the VM can interpret them.

We have also seen that a CompiledMethod is just an array of bytes. And “BYTEcode” has the prefix “byte”. So, as you can imagine, every bytecode is represented by a byte. One bytecode, one byte (sure?? mmmm). One byte, 8 bits, 2^8 -1 = 255 possible different bytecodes.

Imagine that we code the following basic method:

MyClass >> foo
self name.

To see the bytecodes there are two possibilities: print the answer of  the message #symbolic to the CompiledMethod instance. For example, (MyClass >> #foo) symbolic or use the SystemBrowser, button “View” -> “byte codes”. The bytecodes from the previous method are:

17 <70> self
18 <D0> send: name
19 <87> pop
20 <78> returnSelf

So…how do we interpret such symbolic representation?

Understanding bytecodes printing

Let’s start from left to right. First “column” is a number. In this case, from 17 to 20. What do those number mean?  Explore or inspect the CompiledMethod:

So, those number represent just the position in the whole array. We said a CM (CompiledMethod) was an array of bytes. So, those number represent the position. The CM has two regions: the literal frame (the first bytes of the CM where the literals are stored) and the bytecodes. These numbers are called “Program Counter” (PC) when they are in the bytecode part. For example, if we send the message #endPC to this CM instance, we will get 20 which is the last byte of the CM that represents bytecodes. The next one, 21, is already representing the trailer. The same way, #initialPC answers 17. And how those two methods are implemented?  the #initialPC uses the header’s encoded information such as the number of literals. And #endPC delegates to its trailer since he knows the size of the trailer.

The second column is an hexadecimal surrounded by <> which represents the bytecode number. For example, <70>,, etc. This hexadecimal represent the unique number of bytecode. ’70’ is the bytecode push receiver, ‘D0’ is a send bytecode, 85 pop stack, and push receiver. Since these numbers are encoded in 1 byte it follows that there are 255 possible differnt type of bytecodes.

The third column is just a text describing the type of bytecode.

If we analyze now the bytecodes generated for our simple method that does a “self name” we have that:  the first bytecode (number 17) it just pushes the receiver (self) into the stack. We need that in the stack because in the next bytecode we send a message to it. The second bytecode, 18, sends the message #name to what it is in the stack. When this bytecode finishes, it pushes in the stack the result of the send. But our method doesn’t do anything with it and instead it just answers self (because there is not explicit return). So, we need to do a pop first, bytecode number 19, to remove the result from stack and let the receiver in the top of the stack. And now, we can finally do the return with bytecode number 20.

Mapping bytecodes from image side to VM side

So far we saw how the bytecodes in the CM look like, but we didn’t see how they are map to the VM. So, take your image with VMMaker loaded, and inspect the method #initializeBytecodeTable. You will see that such method is something like this:

The table follows much more but I cut it for the post. So as you see it is just a table that maps numbers to methods 😉  We saw that the symbolic representation has 3 columns, and the second one which was surrounded by <> and an hexadecimal value represents the number of the bytecode. That number is exactly this one used in this table, with the difference that it is in decimal. So, for example the bytecode “<78> returnSelf”, if we translate 78 from hexa to decimal (just print 16r78)  we get 120, which maps to the method #returnReceiver. So, you can now just browse the method and look what it does 🙂  Remember that this is part of VMMaker and this code is written is SLANG. For more details read my old posts.

StackInterpreter >> returnReceiver
localReturnValue := self receiver.
self commonReturn

You have now learned how to see the bytecodes of a method and how to see its implementation in the VM. Cool!!  You deserve a break (or a beer) 🙂

Did you notice that some bytecodes are mapped directly to one only number (like #returnReceiver) but some other like #pushLiteralVariableBytecode are mapped to a range of numbers?  We will see later why.

More complicated bytecodes

Now, let’s see a more advance method, for example, this one:

fooComplicated: aBool and: aNumber
| something aName |
aName := self name.
Transcript show: aName.
aBool
ifTrue: [ ^ aName ].
^ nil

Which generates the following bytecodes:

25 <70> self
26  send: name
27 <6B> popIntoTemp: 3
28 <42> pushLit: Transcript
29 <13> pushTemp: 3
30  send: show:
31 <87> pop
32 <10> pushTemp: 0
33 <99> jumpFalse: 36
34 <13> pushTemp: 3
35 <7C> returnTop
36 <7B> return: nil

There are a couple of new bytecodes in this method. Bytecode 27, does a pop of the return of “self name” and push it in the temp number 3. Notice that temps are both, parameters and temp variables. In this case, temp 0 is ‘aBool’, temp 1 ‘aNumber’, temp 2 ‘something’ and temp 3 is ‘aName’. Bytecode 28 needs to push the literal “Transcript” into the stack since in the next bytecode a message is sent to it. Bytecode 29 pushes ‘aName’ to the stack since it will be the parameter for the send.  Bytecode 30 does the send and 31 does a pop because we don’t do anything with the return of the message.

With bytecode 32 we put ‘aBool’ into the stack and then….then…shouldn’t we have something like 33 send: ifTrue:ifFalse:  ???  Yes, we should. But the compiler does an optimization and replaces the message send by a jump bytecode. In this case, a jump bytecode saying that when it is false jump to bytecode number 36 which does the return nil. Otherwise (if true), continue with bytecode 34 which pushes ‘aName’ into the stack and finally bytecode 35 does the return of the top of the stack (where we can ‘aName’).

How do we represent parameters in bytecodes?

We shouldn’t forget that btyecodes are just a number between 0 and 255. The bytecode <78> returnSelf  is number 120, which was we can see in #initializeBytecodeTable it is mapped by  (120 returnReceiver). Does this method requires any kind of parameter? No. It just returns the receiver. Now,  let’s analyze the bytecode <6B> popIntoTemp: 3 from the previous example. 16r6B -> 107.  Ok, cool. So, number 107 does a pop and puts that into the temp number 3. But….all what we have in the CompiledMethod is the bytecode, the byte that contains the number 107. Nothing more. Imagine the method in the VM that is mapped to this bytecode….how can it knows the temp number?   The same with bytecode “<42> pushLit: Transcript”. It is just a number,  66. Where is the “Transcript” stored?

So…the generic question is, if we only have a bytecode number, how do we solve the bytecodes that require some parameters ? Ok, this will sound a little weird. Or smart?  The truth is that this missing information is sometimes (we will see later why sometimes) encoded using offsets in the range of bytecodes. Let’s see the example of the  <6B> popIntoTemp: 3, which is bytecode number 107. In #initializeBytecodeTable, we can see: “(104 111 storeAndPopTemporaryVariableBytecode)“. So we have a range of bytecode between 104 and 111. In this case we want to do a pop and put the result in the temp number 3. If we can assume that 104 is for temp 0, 105, temp1, 106 temp2 and 107 temp3 🙂 we now understand why out bytecode number is 107. That bytecode number encodes that the number of the temp is the 3. The method storeAndPopTemporaryVariableBytecode  will be able to get a diff between the current number (107) and the start of this range (104) and finally know that the temp number is the 3th.

The same happens with the other example <42> pushLit: Transcript, number 66. In #initializeBytecodeTable we can see “( 64  95 pushLiteralVariableBytecode)“. 64 is for literal at 1, 65 at 2, and 66 at 3.   Now, do (MyClass >> #fooComplicated:and:) literalAt: 3 -> #Transcript->Transcript  🙂

Now, if we analyze “(104 111 storeAndPopTemporaryVariableBytecode)” we can understand that the maximum amount of temporal variable is 7 (111-104). However, in this post, we saw the class comment of CompiledMethod says “(index 18)    6 bits:    number of temporary variables (#numTemps)”. That means that maximum number of temps is 2^6-1=63 . So…..something is wrong. Let’s find out what.

Extended bytecodes

(104 111 storeAndPopTemporaryVariableBytecode)”  supports 7 temps, but CompiledMethod class comment says 63 are supported. So, let’s create a method with more than 7 temps and let’s see its bytecodes. If we continue with our example we now modify the method to this:

fooComplicated: aBool and: aNumber
| something aName a b c d e f g h i j k |
d := self name.
Transcript show: aName.
aBool
ifTrue: [ ^ aName ].
^ nil

In this case, “d := self name” we are assigning ‘name’ to ‘d’ which is the temp number 7 (remember they start in 0 and the parameters are count together with the temp variables). Hence, the bytecodes are:

 25 <70> self
 26  send: name
 27 <6F> popIntoTemp: 7
 28 <42> pushLit: Transcript
 29 <13> pushTemp: 3
 30  send: show:
 31 <87> pop
 32 <10> pushTemp: 0
 33 <99> jumpFalse: 36
 34 <13> pushTemp: 3
 35 <7C> returnTop
 36 <7B> return: nil

Now, if we just change “d := self name.” to “e := self name.” we would be using the parameter number 8. What would happen? Ok, if you change it you will see that the bytecode changes from “27 <6F> popIntoTemp: 7” to “27 <82 48> popIntoTemp: 8”. Chan! Chan! Chan! What is that???? It seems the bytecode is using in fact 2 bytecodes? (82 and 48).

These kind of bytecodes are called “extended bytecodes”. If we check in #initializeBytecodeTable bytecode 16r82 = 130 is #extendedStoreAndPopBytecode. So at least we know which method is it. Now, what does the 2 bytecode mean (48 in our example) ?  Somehow such second byte should tell us the number of the temp (8 in our example). If we do 16r48 = 72 and check the bytecode 72 we get it is #pushLiteralVariableBytecode, which doesn’t seem to be correct. So, this second byte does not represent a bytecode. Instead, it represents just a byte that encodes information. That information is usually a type and an index, both encoded in one single byte.

In this particular example of #extendedStoreAndPopBytecode uses 2 bits for a type and 6 bits for an index:

extendedStoreBytecode
| descriptor variableType variableIndex association |
<inline: true>
descriptor := self fetchByte.
self fetchNextBytecode.
variableType := descriptor >> 6 bitAnd: 3.
variableIndex := descriptor bitAnd: 63.
variableType = 0 ifTrue:
[^objectMemory storePointer: variableIndex ofObject: self receiver withValue: self internalStackTop].
variableType = 1 ifTrue:
[^self temporary: variableIndex in: localFP put: self internalStackTop].
variableType = 3 ifTrue:
[association := self literal: variableIndex.
^objectMemory storePointer: ValueIndex ofObject: association withValue: self internalStackTop].
self error: 'illegal store'.
^nil

We can see that in our example, 16r48 = 72.  (72 >> 6) bitAnd: 3  -> 1. So, type is 1. The 3 is because it uses 2 bits for the type (2^2-1=3). And 72 bitAnd: 63 -> 8  (which correctly is the number of temp we need). 63 is because 2^6-1=63.  As you can notice, each bytecode is responsible of decoding the information of the second byte. The compiler of course, needs to correctly generate the bytecodes. #extendedStoreAndPopBytecode was an example so that you can understand and learn, but there are much more extended bytecodes. There are even “single extended bytecodes” and “double extended bytecodes”.

Why do we need extended bytecodes?

Well, I am not an expert at all in this subject but I can guess it is because of the size of CompiledMethod. In the previous example of the extended bytecode, it uses two bytes instead of one, as we can see in the explorer:

Notice that bytecode number 27 occupies two (28 is not shown). At the beginning we saw that when we have a “range” in the bytecodes it means that the difference encodes a number, usually an index. But if we need a range of bytecodes for the maximum supported, we would need much more than 255. Hence, more bytes per bytecode.  Since most methods in Smalltalk are short and encode few number of instance variables, parameters, temporal variables, etc, it was decided to use 255 and just use more bytes per bytecode for those cases that was needed. For example:

(CompiledMethod allInstances select: [:each | each numTemps > 7]) size  -> 1337
CompiledMethod instanceCount -> 75786
((1337 * 100) / 75786) asFloat -> 1.7641780803842397

So…only a 1.76% of the CompiledMethod of my image have more than 7 temporal variables. And remember that this was just an example, but there are extended bytecodes for more things. Maybe (I have no idea) with today computers this is not worth it and maybe having 3 or 4 bytes for every bytecode is enough. But since it is like this and working correctly, why to change it?

Groups of bytecodes

Since the specification of the blue book of Smalltalk-80, bytecodes are known to be grouped in different groups. And since the core of Squeak/Pharo VM is implemented in a subset of Smalltalk called SLANG and since we have classes with methods that represents Interpreters….how would these groups be represented? Of course, as method categories!!! So, you can map each of the following group, with a method category of an Interpreter class:

  • Stack manipulation bytecodes: all things related to push and pop.
  • Message sending bytecodes: bytecodes that are used when sending messages.
  • Return bytecodes: are used for different kinds of return.
  • Jump bytecodes are related to conditionals

Look the attached screenshot:

Books and links

In this post it is easy: just read the blue book 🙂   As always, you can download it in pdf from http://stephane.ducasse.free.fr/FreeBooks.html or directly browse the web version provided by Eliot Miranda. For a bytecode introduction read the end of chapter 26  and for more details the whole chapter 28. Notice that the specification has changed a bit from the 80’s and there are now there are more bytecodes but the general idea is still valid.


Playing with CompiledMethod

The today’s stop of this Journey through the VM is about CompiledMethods. In the previous post I explained the different class formats and specially, the unique format of CompiledMethod. Today we are going deeper with them and we will see why they are even more special 😉

Summary of the previous post: CompiledMethod instances are internally represented in the VM as bytes. However, CompiledMethod is the only class in the system that mixes pointers (for the literals) with bytes (for the bytecodes). So those bytes encodes both things.

Inspecting a CompiledMethod

What is the normal way to learn something in Smalltalk? Open your image and check senders, references, or someone who does more or less what you need and try to understand it. In the previous post, I showed you how inspecting or exploring a CompiledMethod give us a lot useful information like the header, the literals, the bytecodes and the trailer. Example:

So this means that at least the Inspector and the Explorer can have access to the CompiledMethod and understand its internal. Let’s take the Inspector (we could have taken also the explorer in which case take a look to CompiledMethod >> #explorerContents). When we inspect a CompiledMethod, the inspector class that is used is CompiledMethodInspector. So, first point, there is a special inspector class for CompiledMethod. Otherwise, if we inspect it with a normal inspector, for example if we do “BasicInspector openOn: (MyClass >> #testSomething)” we have something like this:

CompiledMethodInspector has two important methods:

CompiledMethodInspector >> fieldList

| keys |
keys := OrderedCollection new.
keys add: 'self'.
keys add: 'all bytecodes'.
keys add: 'header'.
1 to: object numLiterals do: [ :i |
keys add: 'literal', i printString ].
object initialPC to: object size do: [ :i |
keys add: i printString ].
^ keys asArray
CompiledMethodInspector  >> selection

| bytecodeIndex |
selectionIndex = 0 ifTrue: [^ ''].
selectionIndex = 1 ifTrue: [^ object ].
selectionIndex = 2 ifTrue: [^ object symbolic].
selectionIndex = 3 ifTrue: [^ object headerDescription].
selectionIndex <= (object numLiterals + 3)
ifTrue: [ ^ object objectAt: selectionIndex - 2 ].
bytecodeIndex := selectionIndex - object numLiterals - 3.
^ object at: object initialPC + bytecodeIndex - 1

So…as you can see in the code, “keys add: ‘all bytecodes’.”  maps to “selectionIndex = 2 ifTrue: [^ object symbolic].“, and “keys add: ‘header’.” to “selectionIndex = 3 ifTrue: [^ object headerDescription].“.  What we should learn from this, is that CompiledMethod >> #symbolic answers a string which nicely shows the bytecodes. So for example, if we have the method:

MyClass >> testSomething
TestCase new.
self name.
Transcript show: 'The answer is:', 42.

Then, “(MyClass >> #testSomething) symbolic” answers the following:

41 <40> pushLit: TestCase
42  send: new
43 <87> pop
44 <70> self
45  send: name
46 <87> pop
47 <43> pushLit: Transcript
48 <25> pushConstant: ''The answer is:''
49 <26> pushConstant: 42
50  send: ,
51  send: show:
52 <87> pop
53 <78> returnSelf

Don’t worry for the moment about the first number in each column (for the interested guys it is the PC -> program counter) and the hexadecimal between <>  (it is the bytecode number in hexa). I will explain that in a future post.

This method #symbolic could be the same used by the SystemBrowser when you select “View” -> “Bytecodes”.  From the previous example, we can also learn that CompiledMethod implements methods like #numLiterals, #objectAt:, #initialPC, etc.  Imagine the CompiledMethod as an array of bytes…how can you determinate which part is literals and which one is bytecodes?  How the #numLiterals can be implemented in CompiledMethod if it is just an array of bytes?

CompiledMethod header

It may be already obvious that CompiledMethods have a header. But be careful, CompiledMethod have both, the normal object header every object has, and then a special header which is just the first word (32 bits -> 4 bytes) of the byte array. So this header is just before the literals and the bytecodes. As we can read in the class comment of CompiledMethod:

“The header is a 30-bit integer with the following format:

(index 0)    9 bits:    main part of primitive number   (#primitive)
(index 9)    8 bits:    number of literals (#numLiterals)
(index 17)    1 bit:    whether a large frame size is needed (#frameSize)
(index 18)    6 bits:    number of temporary variables (#numTemps)
(index 24)    4 bits:    number of arguments to the method (#numArgs)
(index 28)    1 bit:    high-bit of primitive number (#primitive)
(index 29)    1 bit:    flag bit, ignored by the VM  (#flag)”

Ok, with this comment you may notice the limits imposed in methods. For example, 9 bits for a primitive it means (2^9) -1=511. BTW, I think this class comment is outdated and now there are 11 bits for primitive index, so it is (2^11) – 1 = 2 047. But you get the idea…. anyway, it is not likely that you have ever reached any of these limits.

Who is responsable of generating such header in the CompiledMethod?  In this post, I told you that usually the input for the Compiler was a string representing the source code and the result was a CompiledMethod instance. Hence, the Compiler takes care about creating such CompiledMethod header. Notice that this header is not only used from the image side but also from the VM. Check (in the VMMaker) implementors and senders of #argumentCountOf:, #literal:ofMethod:, #primitiveIndexOf:, #tempCountOf:, etc.

CompiledMethod trailer

Something you should be asking yourself is where the source code is stored?  I mean, when you open a browser and see the source code of a method, where does it come from?  because in the CompiledMethod we saw that only literals and bytecodes are stored, not source code. So???  Ok… the source code is stored in two files: .sources and .changes. The “old” methods’ source code is in the .sources file and the “new” method’s source code in the .changes. You can browse #condenseChanges and #condenseSources for details. So far so good. But…. how a CompiledMethod instance is map to its source code in the file?  Excellent question Mariano 🙂

The same way there is a special header for CompiledMethod, there is a trailer. So far the trailer has been used only for getting the source code of the method. Some time ago, this trailer was one word size (4 bytes) and it encoded a number which was the offset in the .sources/.changes file. That number represent both things: the offset in the file, and a flag to say from which file (if .changes or .sources). Check for example the method #filePositionFromSourcePointer:. In addition, the logic of encoding and decoding the trailer was implemented in the CompiledMethod class.

In today’s Pharo images (and Squeak), this is not true anymore. There are two big differences with the “old” approach:

  1. The trailer was reified with the class CompiledMethodTrailer.
  2. There are different kind of trailers implemented and up to 255 possibilities. The implemented kinds are: normal source pointer, temp names (the decompiler can use such temp names when getting the source so that to generate a source code more similar to the original one), variable length (for example when .changes is bigger than 32MB), etc. For more details, check #trailerKinds. Of course, the most common type is “SourcePointer”.

CompiledMethod is a chunk of bytes (this is why it is a subclass from ByteArray), and its format is “bytes”, so it means it cannot define normal instance variables.  So how can it have a CompiledMethodTrailer?  Ok, it works this way: when a CompiledMethod is being created (usually by the Compiler), a specific CompiledMethodTrailer instance is also created. That instance of CompiledMethodTrailer has to be created with a specific type (source pointer, temp names, etc). Once the CompiledMethod is almost ready the trailer instance is encoded as bytes in the CompiledMethod instance, and then it is garbage collected. Later on, when someone ask to the CompiledMethod for its source code (using the method #getSource), it delegates to a trailer instance. But there is not trailer instance as this moment. So….every time the source code is needed, the CompiledMethod creates an instance of a trailer. But notice that it is up to the CompiledMethodTrailer to know how many bytes are the trailer, how to decode it and what do the bytes represent (if a source pointer, an array of temp names, etc). Finally, the trailer answers the source code of the method. So, the CompiledMethod just has:

CompiledMethod >> trailer
"Answer the receiver's trailer"
^ CompiledMethodTrailer new method: self

The CompiledMethodTrailer just read the last byte, it checks in an internal table to see which kind of trailer is it, and then perform the correct method to decode the information. The amount of bytes used by the trailer and what they represent, depends on the kind of trailer.

method: aMethod

| flagByte |

data := size := nil.
method := aMethod.
flagByte := method at: (method size).

"trailer kind encoded in 6 high bits of last byte"
kind := self class trailerKinds at: 1+(flagByte>>2).

"decode the trailer bytes"
self perform: ('decode' , kind) asSymbol.

"after decoding the trailer, size must be set"
[size notNil] assert.

Depending on the type of trailer,  CompiledMethodTrailer will finally execute one of the methods encode* when the CompiledMethod is being created, and decode* when asking its source code.

A question to all of you….wouldn’t it make sense to rename CompiledMethodTrailer to MethodSource ? because trailers has been always use only for that….

Decompiling CompiledMethods

Why source code is not stored in CompiledMethod? From my point of view, there are 2 main reasons:

  1. Because as its class name suggests, they reify COMPILED methods, not source methods or whatever name you want to use.
  2. Memory and security reasons. When you deploy an application written in C, do you include source code? no. And in Java? no. So why we would do it in Smalltalk? Remember that the way to “deploy” a Smalltalk application is providing an .image.

The ideal approach would be to have the sources in development and to be able to remove them when deploying. Smalltalk allows us that. Just remove the .sources file and that’s all 🙂 Your image continues to work as if nothing has happened. But sometimes we have a bug in our application and we want to be able to browse the code. Guess what? Smalltalk provides that also 😉  Let’s try it (don’t try in Pharo1.3 because there is a bug. Use anyone before 1.3). Create a method anywhere, for example:

testSomething: aaa with: bbb and: ccc
| name |
TestCase new.
(4 = 3)
ifTrue: ["I am a nice comment, don't remove meeee pleaeee!"]
ifFalse: ["I like this way of formatting my code"].

name := self name.
Transcript show: 'The answer is:', 42.

Now, close your image. Rename the .changes file (creating a new method will ensure that the source pointer points to the .changes and not to .sources) so that it is not found. Open your image again, and you may have the popup saying that the .changes couldn’t be find. No problem. Accept it.

Now, if you open the system browser, you can browse any method of the image. And only whose source were in the .changes will look similar to this:

testSomething: t1 with: t2 and: t3
| t4 |
TestCase new.
4 = 3.
t4 := self name.
Transcript show: 'The answer is:' , 42

What you are seeing is not the source code of the method but instead the decompiled one. The compiler is able to decompile a CompiledMethod (using the bytecodes and literals) and get the possible “source code”. However, the decompiler source is not exactly the same as the original source. Note that the decompiler the only thing it has for a method is the bytecodes and the literals. Hence, the decompiled code does not have:

  • Temporal variable and parameters  names. Since they are not stored in the CompiledMethod they are lost. Both temps and parameters are replaced in the decompiled code with “t1”, “t2”, etc.
  • Comments are lost (they are not stored in the CompiledMethod).
  • Code formatting (tabs and spaces) is lost.

The cool thing is that even with the decompiled code we can get an idea of the code, debug it, and probably find the bug we were looking for.

Notice that the source code of “old” methods are stored not in the .changes but in the .sources. So, even removing .changes there are methods which get the source from the .sources file. Therefore, we can also remove the .sources and that way all methods in the image will be decompiled if you try to browse them.

Depending of what and where you are deploying, getting rid of .sources and .changes could be worth it.

CompiledMethod equality

What do you expect the following expression to answer:

(Boolean>>#&) = (Boolean>>#|)

Ok…you need to see the source code?

& aBoolean
"Evaluating conjunction. Evaluate the argument. Then answer true if
both the receiver and the argument are true."

self subclassResponsibility
| aBoolean
"Evaluating disjunction (OR). Evaluate the argument. Then answer true
if either the receiver or the argument is true."

self subclassResponsibility

So? true or false? TRUEEEE!!! that is true. And why? if they have different comments, they have different selectors! So? who cares about that? we are talking about COMPILED methods. Are the bytecodes the same? yes. Are the literals the same? yes.  So they are the same compiled method. Point. So….you would really be careful when putting CompiledMethods in Sets, Dictionaries or things like that. Example:

InstructionClient methods size -> 27
InstructionClient methods asSet size -> 21

Conclusion: use an IdentitySet or a IdentiyDictionary if you want to avoid problems.

Sorry for the long post, but there is too much to talk about CompiledMethods. In the next post we will talk a little more about bytecodes.


Class formats and CompiledMethod uniqueness

Before going deeper with CompiledMethods I would like to talk a little bit about class formats. Unfortunately, I didn’t find class formats documented more than in code and method comments. If you know a source of documentation of this topic, please let me know.

Class format

From my point of view, the class format is a really internal and implementative detail of the VM. The class format defines the structure (layout) of the instances of a class, in the VM. In the previous post, I said: “In the internal representation of the Virtual Machine, objects are a chunck of memory. They have an object header which (there will be a whole post about it) can be between one and three words, and following the object header, there are slots (normally of 32 or 64 bytes) that are memory addresses which usually (we will see why I didn’t say always) represent the instance variables.”

So, usually, that is, “normal” structure, an object has a fixed amount of instance variables which are just pointers to other objects. In this case, those “slots” (which are one word size, that is 32 or 64 bits) contain the memory address (pointer) of the header of the object they point to. But that’s not the only possibility, another object (like a Collection instance), do not have a fixed number of instance variables, but instead it is variable. And the representation is not always pointers (a word), but it can also be bytes. In summary, what changes is how it is represented the chunk of memory of an object.

Different class formats

  • Normal: there is a fixed amount of instance variables and each of them is just a pointer to another object. Notice that not only the amount of pointers is fixed by the amount of instance variables, but also, the pointer is always the same, one world (32 or 64 bits). Examples are any normal class like TestCase, Browser, True, Integer, etc.
  • Bytes: it means that the chunk of memory of an object is represented in a variable sequence of individual bytes.  Examples: ByteArray, ByteString, ByteSymbol, LargePositiveInteger, LargeNegativeInteger, etc.
  • Words: it is similar to “Bytes”, in the way that it is variable, but it is represented by a sequence of words instead. Notice that “Normal” also encodes the pointers in words, but in that case, the amount of those words is fixed and second they represent pointers. In this case, the amount of words is variable and they do not represent pointers to objects. Examples are Bitmap, WideString, WideSymbol, WordArray, FloatArray, etc.
  • Weak: when an object has weak references it means that its pointers to other objects don’t count for the Garbage Collector. So the GC removes and object when nobody else non-weak point to it. Weak format can be applied to both, variable and fixed formats. For example, WeakFinalizerItem has a normal format, but weak. On the contrary, WeakArray has a variable format and weak.
  • Variable: this is like “Normal” but where the pointers are not fixed, but instead variable. It can also be seen as “Words” but there each word does represent a pointer. Examples: BlockClosure, MethodDictionary, etc.
  • CompiledMethod: Chan! Chan! Chan! Yes, CompiledMethod class has its own format. Do you understand already why I wanted to talk about this before CompiledMethods?  But we will let the explanation to the end of the post…

Now…if you want to check by yourself, check the method Behavior >> #typeOfClass, it answers a symbol uniquely describing the format of the receiver class:

Behavior >> typeOfClass
"Answer a symbol uniquely describing the type of the receiver"
self instSpec = CompiledMethod instSpec ifTrue:[^#compiledMethod]. "Very special!"
self isBytes ifTrue:[^#bytes].
(self isWords and:[self isPointers not]) ifTrue:[^#words].
self isWeak ifTrue:[^#weak].
self isVariable ifTrue:[^#variable].
^#normal.

So you can do for example:

TestCase typeOfClass -> #normal
ByteArray typeOfClass -> #bytes
Bitmap typeOfClass -> #words
WeakArray typeOfClass -> #weak
BlockClosure typeOfClass -> #variable

Or you can inspect all classes of a certain type:

(Smalltalk allClasses select: [:each | each typeOfClass = #weak ]) inspect

Now, if you take a look to the method #typeOfClass we can see that it ask to itself whether it is bytes, or bits, or pointers, etc…In addition, notice the word “uniquely” in the comment of the method #typeOfClass. This means that the same class can be several “things” at the same time. For example:

Bitmap isVariable -> true
Bitmap isWords -> true
Bitmap isPointers -> false

BlockClosure isVariable -> true
BlockClosure isWords -> true
BlockClosure isPointers -> true

That example shows that all those classes that are “Words” or “Bytes” are also “variable”. Ahhh and btw…those variable classes supports Behavior >> #new: sizeOfVariables. Most classes in the Collection‘s hierarchy  are variable.

Prepare the image

In my post about compiling the VM I told you to use a PharoCore image since it was the “recommended” way. However, in the second post about building the VM, I provided you with a PharoDev 1.2.1 image ready to load Cog and its VMMaker branch.  So, even if you are not going to compile the VM, I recommend you to load Cog and VMMaker so that you could follow some of my comments. In addition, since we are not going to build the VM for a couple of posts, but instead understanding it, you can save this image and you will be able to use it in the next posts. Just thake the image and evaluate:

Deprecation raiseWarning: false.
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfCog';
load.
((Smalltalk at: #ConfigurationOfCog) project version: '2.0') load.

Class format encoding in classes and instances

If you see all those methods like #isBytes, #isVariable, #isPointers, etc (all those methods in the category ‘testing’ in Behavior class) you will notice that they all send #instSpec (instance specification I guess) at the end. And this method looks like this:

Behavior >> instSpec
^ (format bitShift: -7) bitAnd: 16rF

And a couple of examples:

TestCase instSpec -> 1
ByteArray instSpec -> 8
CompiledMethod instSpec -> 12

“format” is the instVar of Behavior, and as it says its getter method “Answer an Integer that encodes the kinds and numbers of variables of  instances of the receiver.”. So the number just alone is not really useful, but taking some bits from it yes, like #instSpec
, #instSize, #indexIfCompact, etc. So…the class encodes this information in an integer which is the “format” instVar.

But what happens to their instances?  Imagine that the VM for different tasks needs to how the format of a particular object. Fetching its class every time may be expensive. So where is such information stored? To answer, we will take our image and browse the “core” of the VM. Let’s see the method ObjectMemory >> formatOf:

formatOf: oop
"       0      no fields
1      fixed fields only (all containing pointers)
2      indexable fields only (all containing pointers)
3      both fixed and indexable fields (all containing pointers)
4      both fixed and indexable weak fields (all containing pointers).
5      unused
6      indexable word fields only (no pointers)
7      indexable long (64-bit) fields (only in 64-bit images)
8-11      indexable byte fields only (no pointers) (low 2 bits are low 2 bits of size)
12-15     compiled methods:
# of literal oops specified in method header,
followed by indexable bytes (same interpretation of low 2 bits as above)
"
<inline: true>
^((self baseHeader: oop) >> 8 ) bitAnd: 16rF

As you can see, there are 16 possible formats, encoded from 0 to 15 in 4 bits of the Object Header. The line “^((self baseHeader: oop) >> 8 ) bitAnd: 16rF” is the one that takes those 4 bits from the Object Header of the OOP (object pointer) received by parameter.

If you now browse the class comment of ObjectMemory, you will read it says that there are 4 bits for the object format. As you can guess, that number that represents the format is what we get in the image side with the method #instSpec. Notice that at the beginning of the post described all the different types of format and they were 6, but here we have 16 possibilities.  Ok, some are for optimizations (for example the number zero means that the object has no instVar, hence the GC can stop there while doing the mark and trace instead of trying to follow non-existent pointers), some are not used (like the number 5), some are only for 64 bits (number 7), the format for “bytes” uses 4 numbers, and CompiledMethod also uses 4 numbers.

Don’t get confused:  In the image side, we have an instVar which is called “format” in Behavior that keeps an integer with both, what WE call format plus the amount of variables. What we call format, is the method #instSpec in the image (which in fact gets the format from the “format” instVar). Finally, the VM agree with us, the method is #formatOf:  and it refers to what we call format. All in all, the instVar “format” of Behavior is misleading. Don’t get confused.

Finally, if you are curious you can check senders of #formatOf: and you will see all the places where the VM needs to know the format of an object.

Creating classes with a special format

We saw all the details of the class formats but we didn’t see how to create a class with a special one. In the previous post, I told you the way to create a subclass in Smalltalk was, of course, by sending a message. In this case, a message to the desired superclass. The method was Class >> #subclass:instanceVariableNames:classVariableNames:poolDictionaries:category: .  Now, if you check in the category of that method, that is, ‘subclass creation’ you will see much more methods like:

  • #variableSubclass: t instanceVariableNames: f classVariableNames: d poolDictionaries: s category: cat
  • #variableByteSubclass: t instanceVariableNames: f classVariableNames: d poolDictionaries: s category: cat
  • #variableWordSubclass: t instanceVariableNames: f  classVariableNames: d poolDictionaries: s category: cat
  • #weakSubclass: t instanceVariableNames: f  classVariableNames: d poolDictionaries: s category: cat

So…you image what each of those methods do, don’t you? If we want to confirm our suspicion, take a look to the definition of the classes. For example, we saw that Bitmap was “words” and ByteArray was “bytes”, hence:

ArrayedCollection variableWordSubclass: #Bitmap
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'Graphics-Primitives'

And:

ArrayedCollection variableByteSubclass: #ByteArray
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'Collections-Arrayed'

Do you notice the difference?  🙂    It is important to note also that there must be some validation. For example, if I define a class as variable with bytes, I shouldn’t be able to declare instance variables to that class, because I cannot mix both (only CompiledMethod do that!!!). So for example, if you try to do:

TestCase variableByteSubclass: #MarianoArray
instanceVariableNames: ' size '
classVariableNames: ''
poolDictionaries: ''
category: 'Collections-Arrayed'

You will get an error that says ‘cannot make a byte subclass of a class with named fields’. These validation are done by ClassBuilder.

CompiledMethod format

All this post was just to explain you the following 😉  As I said, CompiledMethod has a very special class format, and we can read it in his own class comment “My instances are methods suitable for interpretation by the virtual machine.  This is the only class in the system whose instances intermix both indexable pointer fields and indexable integer fields.”  This means that CompiledMethod was created with the message #variableByteSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:    and in addition:

CompiledMethod isBytes -> true
CompiledMethod isWords -> false  "lying!! he also includes words for pointers "
CompiledMethod isPointers -> false  "lying!! he also includes words for pointers"

So…the system thinks CompiledMethod is just a “Bytes” but it is not, it is a mix between bytes and pointers (words). The pointers are used to point to the literals and this part of the CompiledMethod is known as the “Literal Frame”. In fact, you will notice that the literals usually include a few type of objects: Symbols (for selectors), Association (for classes and globals), SmallInteger, ByteString for string constants, etc. The “bytes” part is the part used to encode the bytecodes (so it means we have only 255 possible bytecodes???  stay tuned…) . Example:

MyClass >> testSomething
TestCase new.
self name.
Transcript show: 'The answer is:', 42.

If you now inspect the literals, you can see something like this:

(MyClass >>#testSomething) literals ---->>>{(#TestCase->TestCase). #name. #show:. (#Transcript->Transcript). #,. 'The answer is:'. 42. #testSomething. (#MyClass->MyClass)}

So…those are regular objects: (#TestCase->TestCase)  is an Association, #name a Symbol, ‘I am hungry’ a Bytestring, 42 a SmallInteger, etc. Think this:  if you explore any of those objects and check for the pointers to them, will you see the CompiledMethod of #testSomething as one of the pointers to them??? we will see the answer next post, but basically it depends whether the tool takes into consideration or not this special magic of CompiledMethod.

Mmmm now I wonder which are the possible classes for literals? …if my Smalltalk doesn’t fail me:

(CompiledMethod allInstances
inject: OrderedCollection new
into: [:allTypesOfLiterals :aCompiledMethod | allTypesOfLiterals addAll: ((aCompiledMethod literals collect: [:aLiteral | aLiteral class]) asSet ); yourself  ]) asSet.

Prints: ” a Set(Float Association ByteArray WideString LargeNegativeInteger AdditionalMethodState Character ByteSymbol Fraction ByteString SmallInteger Array ScaledDecimal LargePositiveInteger)”

If the format is “Bytes” and there is supposed to be no pointers, how it is possible that we can ask for an object (a literal for example)?  Ok…if you see CompiledMethod >> objectAt:   it delegates to a primitive. But since now you know how to download VMMaker and go to StackInterpreter class >> initializePrimitiveTable  and see that the primitive method is in fact called #primitiveObjectAt and you can see the code of what it does (hint: CompiledMethod has a header which contains the amount of literals among other stuff).

To conclude, let’s say that CompiledMethod format is “Bytes” but in fact it is the only class in the system that mixes pointers (for the literals) with bytes (for the bytecodes). Because of this, and another couple of reasons, CompiledMethod is aunique  quite special class.

Finally, I let you homework 😉 If we inspect/explore a ByteArray, we get something like this:

However, if we explore a CompiledMethod we get an explorer that show us the literals and the bytecodes in a nice way. Like this one:

How do you think the Explorer can do such thing?  and the Inspector ?

See you


Smalltalk reflective model

Hi. I am sure the title of this post is horrible, but I didn’t find anything better. The idea is simple: in this part of the journey, we will talk about bytecodes, primitives, CompiledMethods, FFI, plugins, etc… But before going there, I would like to write some bits about what happens first in the image side. These may be topics everybody know, so in that case, just skip the post and wait for the next one 😉  My intention is that anyway can follow my posts.

A really quick intro to Smalltalk reflective model

The reflective model of Smalltalk is easy and elegant. As we can read in Pharo by Example, there are two important rules:  1) Everything is an object; 2) Every object is instance of a class. Since classes are objects and every object is an instance of a class, it follows that classes must also be instances of classes. A class whose instances are classes is called a metaclass. Whenever you create a class, the system automatically creates a metaclass. The metaclass defines the structure and behavior of the class that is its instance. The following picture shows a minimized reflective model of Smalltalk. Notice that for clarification purposes this diagram shows only a part of it.

A class contains a name, a format, a method dictionary, its superclass, a list of instance variables, etc. The method dictionary is a map where keys are the methods names (called selectors in Smalltalk) and the values are the compiled methods which are instances of CompiledMethod.

When an object receives a message, the Virtual Machine has to do first what it is commonly called as the Method Lookup. This consist of searching the message through the hierarchy chain of the receiver’s class. For each class in the chain, it checks whether the selector is included or not in the MethodDictionary.  If it is not, it continues searching forward in the chain until it finds a method or sends the #doesNotUnderstand: message in case it was not found in the whole hierarchy. When a method is found, it is directly executed.

To understand these topics, I really recommend the two wonderful chapters in Pharo By Example book: Chapter 13 “Classes and Metaclasses” and Chapter 14 “Reflection”. They are both a “must read” if you are more or less new with these topics.

In the internal representation of the Virtual Machine, objects are a chuck of memory. They have an object header which (there will be a whole post about it) can be between one and three words, and following the object header, there are slots (normally of 32 or 64 bytes) that are memory addresses which usually (we will see why I didn’t say always) represent the instance variables. The object header contains bits for the Garbage Collector usage, the hash, the format, a pointer to its class, etc.

Classes and Metaclasses

How do you create a class in Smalltalk? In other languages, you normally create a new text file that after you compile. But in Smalltalk, as we are used to, everything happens by a message send. So, to create a new class you tell to the superclass, “Can you create this subclass with this name, these instance variables and this category please?”. So, when you take a browser and you do a “Ctrl + s” of this code:

Object subclass: #MyClass
instanceVariableNames: ''
classVariableNames: ''
poolDictionaries: ''
category: 'MyCategory'

The only thing you do, is to send the message #subclass:instanceVariableNames:classVariableNames:poolDictionaries:category: to Object. If fact, you can take that piece of code, evaluate it in a Workspace, and you will get the same results 🙂

You can see implementors and you will logically find one in Class. Which should be the result of such message sent?  two things: a new class and a new metaclass. Do the following test:

Metaclass instanceCount ->  3710
Class allSubclasses size -> 3710

Now, create a new class, and inspect again:

Metaclass instanceCount ->  3711
Class allSubclasses size -> 3711

The problem with Metaclasses is that they are implicit, so they are very difficult to understand. Imagine that you create a class User, then its class is “User class”. The unique instance of “User class” is “User”. And at the same time, “User class” is an instance of Metaclass. So….complicated, but if you want to understand them, take a look to the chapters I told you.  How it is done?  it is not really important for the purpose of this post, but it uses the ClassBuilder and also the Compiler (check senders of #compilerClass).

Creating a method

We saw what happens when we create a class. And when you save a method from the browser? what happens ? In a nutshell what happens is that the Smalltalk Compiler does its magic, that is, it receives as an input a string that represents the source code, and as a result you get a CompiledMethod instance. A CompiledMethod contains all the instructions (bytecodes) and information (literals) that the VM needs to interpret and execute such method.

Let’s see it by ourself. Take your image, create a dumy class and then put a breakpoint at the beginning of Behavior >> #compile:classified:notifying:trailer:ifFail:. Now, type the following method and accept it:

testCompiler
Transcript show: 'all this code will be compiled'.

Once you accept such code, the debugger should appear. You can analyze the stacktrace if you want. Notice the arguments that the methods has: compile: code classified: category notifying: requestor trailer: bytes ifFail: failBlock.  So, I told you that the basic idea was to send a piece of code as text and get the CompiledMethod instance. The parameter “code” should be the code of the method we type, and yes, it is a ByteString. If you go step by step with the debugger, and inspect the result, that is, “CompiledMethodWithNode generateMethodFromNode: methodNode trailer: bytes.”  you will see it answers the CompiledMethodWithNode instance to which you can ask “method” and it is the CompiledMethod instnace.  Of course, that method should be the same you get after when doing “MyClass methodDict at: #testCompiler”.

The rest of the parameters are the category in which the method should be, the requestor (someone to notify about this event), the trailer bytes (we will see this later on), and a block to execute if there is an error.

CompiledMethods

In Smalltalk compiled methods are first-class objects (classes too!), in this case instances of CompiledMethod class. However, the class CompiledMethod is quite special and a little differet from the rest. But we will see this later on….What it is important for the moment, is to know that a CompiledMethod contains a list of bytecodes and a list of  literals. Bytecodes are instructions. A method is decomposed in a set of bytecodes, which are grouped in five categories: pushes, stores, sends, returns, and jumps. Literals are all those objects and selectors that are needed by the bytecodes but they are not instance variables of the receiver, hence they need to be stored somewhere.

For example, with our previous example of #testCompiler, we will have a bytecode (among others) for sending the message #show:  and we will have the ‘Transcript’ and the selector name ‘show:’ in the literals. As an exersise, inspect the CompiledMethod instance. You can just evaluate: “(MyClass >> #testCompiler) inspect”. But….”exploring” is usually better that “inspect” for compiled methods…I let you see the differences 🙂  Anyway, you will see something like this:

In the next post we will play a bit deeper with CompiledMethods so don’t worry.

The Compiler

My knowledge of the Compiler is quite limited, but is is important to notice that the Compiler does much more things than the one I have said. In a compiler, there are usually several steps like parse the code, validate it, get an intermediate representation, and finally create the CompiledMethod instance. The compiler needs to know how to translate our Smalltalk code to bytecodes understood by the VM.

In Squeak and Pharo, the compiler is mostly implemented in the class Compiler. It seems it is quite difficult to understand and it has some limitations and difficulties to get intermediate representation of the code. Because of that and probably much other reasons, the community started to work in a new compiler called Opal (which at the beginning was called NewCompiler).

Links

  • Blue book: When talking about the VM and Smalltalk in general, the bible is the book “Smalltalk-80: The Language and Its Implementation”. You can find it in pdf in Stéphane Ducasse free books page, and directly in html (actually, only the chapters 26-30) in Eliot Miranda webpage. Those chapters are the part of the “Implementation” so everything that is related to the VM is there.
  • Regarding Compiler, CompiledMethod, etc, you can read in the blue book this and this. About bytecodes, an intro here and in details here.
  • Pharo By Example book: Chapter 13 “Classes and Metaclasses” and Chapter 14 “Reflection”.
  • A Tour of the Squeak Object Engine” gives an excellent overview of the VM, including a description about CompiledMethod, bytecodes and friends.
  • Opal Compiler.