Tag Archives: XCode

How to debug the VM?

Hi. Whether you are experimenting and hacking in the VM (where it is likely that some things will go wrong) or you are running an application in production, it is always useful to know how to debug the VM.  In this post, we will see how to compile the VM with all the debug information, how to run the VM from GDB (the GNU Project Debugger) and how to debug it using an IDE’s debugger.

Before going further, I would like to tell you something I did last weekend (apart from sleeping quite a lot).  Laurent Lauffont, creator of the PharoCasts, started a new sequence of screencasts called ” PharoCasts with Experts“. The idea is to do a more or less 1 hr “interview” about certain topic. Laurent calls by Skype to the “expert” and they can talk, ask questions, etc. Using TeamViewer, the “expert” shares his desktop to Laurent, who is in addition recording the screencast. The screencast is finally edited and uploaded to http://www.pharocasts.com . Ok, I am not a VM expert at all, but last week we did a 1:30 screencast about compiling and debugging the VM 🙂   This screencast contemplates all what we have learned in the previous posts and what you will learn today.

So…let’s start the show.

Prepared image for you

In previous posts, I told you that Pharo guys recommend us to use the latest PharoCore for building the CogVMs. There are several reasons for this but the basic idea is that they want the VM can be build in latest versions, and if cannot, then fix it as soon as possible. But we have a much better testers than you, and it is called Hudson. So…I don’t want to force you to use the latest unstable PharoCore (which in fact, it is unstable sometimes). Even more, if you are starting in the VM world, I don’t want to add even more problems. So…from now onward, I have prepared and image so that you can use. At the time of this writing, there is no PharoDev 1.3 yet, but all the VM tools doesn’t load directly in a Pharo 1.2. There are 3 little changes that are needed. So…you can grab a PharoDev 1.2.1 and file in these changes, or directly load this image that I have prepared for you. This image has only those 3 changes. It does not contain any Cog or CMakeVMMaker package, that’s your homework 😉   The good news is that since we use a Dev image now we have all the development tools like refactorings, code completion, code highlighting, etc.

When do we need to debug the VM?

I think it is a good idea to start thinking when we need to debug the VM. In my little experience, I have found the following reasons:

  1. When there is a crash and you cannot find why it was.
  2. When you are developing something on the VM.
  3. For learning purposes.
  4. When you are just a hacky geek 🙂

I think most people will need the first one.

Debugging and optimizing a C program

If you are reading this post, you are probably a Smalltalker. And if you are a Smalltalker, I know how much you love the debugger. We all do. But there are guys that are not as lucky as we are 😉 Now, being serious, how do we debug a program written in C ? I mean, the VM, at the end, is a program compiled in C and executed like any other program. I am not a C expert so I will do a quick introduction.

When you normally compile a C program the binary you get does not contain the C source code used to generate such binaries. In addition, the names of the variables or the functions may be lost. Hence, it is really hard to be able to debug a program. This is why C compilers (like GCC) usually provide a way to compile a C program with “Debugging Symbols“, which add the necessary information for debugging to the binaries. Normally, as we will see today, these symbolic symbols are included directly in the binary. However, they can be sometimes separated in a different file. The GCC flag for the debugging symbols is -g  where -g3 is the maximum and -g0 the minimal (none). There are much more flags related to debugging but that is not the main topic of this post.

In addition to the debugging, a C compiler usually provides what they call “Compiler Optimizations”. This means that the compiler try to optimize the resulted binary in different ways: speed, memory, size, etc. When compiling for debugging it is common to disable optimizations, why? because, as its documentation says some variables you declared may not exist at all; flow of control may briefly move where you did not expect it; some statements may not be executed because they compute constant results or their values were already at hand; some statements may execute in different places because they were moved out of loops. The GCC flag for the optimization is -O  where -O3 is the maximum and -O0 no optimization (default).

But we are Smalltalkers, so much better if someone can take care about all these low-level details. In this case, CMakeVMMaker is that guy.

Building a Debug VM

As I told you in the previous post, there are several CMakeVMMAker configurations classes, and some of them are for compiling a debug VM. Examples: MTCocoaIOSCogJitDebugConfig, StackInterpreterDebugUnixConfig, CogDebugUnixConfig, etc. What is the difference between those configurations and the ones we have used so far ? The only difference is the compiler flags. These “Debug” VMs use special flags for debugging. Normally, these flags include things like -g3 (maximum debugging information), -O0 (no optimization), etc. For more details, check implementors of #compilerFlagsDebug. But…since there were a couple of fixes in the latest commit, I recommend you to load the version “CMakeVMMaker-MarianoMartinezPeck.94” of the CMakeVMMaker package.

So…how do we build the debug VM?  Exactly the same way as the regular (“deploy”)  VMs (I have explained that here and here), with only one difference. Guess which one? Yes, of course, use the debug configuration instead of the normal one. That means that in Mac for example, instead of doing “MTCocoaIOSCogJitConfig  generateWithSources”  we do “MTCocoaIOSCogJitDebugConfig  generateWithSources”. And that’s all. Once you finish all the build steps, you have a debug VM.

What are you waiting? Fire up your image and create a debug VM 😉  Now…how do you know if you really succeeded or not to create a debug VM?  Easy: it will be much slower and the VM binary will be bigger (because of the symbolic symbols addition). In a regular CogMTVM, if you evaluate “1 tinyBenchmarks” you may get something arround  ‘713091922 bytecodes/sec; 103597027 sends/sec’ whereas with a debug VM something arround ‘223580786 bytecodes/sec; 104885548 sends/sec’. So..as you can notice, a debug VM is at least 3 times slower.

Debugging the VM with GDB

Disclaimer: I am not a GDB expert, since most of the times I debug from the IDE (XCode). I will give you a quick introduction. So…we have compiled our with all the debug flags which means that the VM binaries now have extra information so that we can debug. GDB is a command line debugger from GNU Project and it is the “standard” when debugging C programs. In addition, it works in most OS. Notice that for Windows there are not VMMakeVMMaker debug configurations. I think this is just because nobody tried it. As far as I know gdb works with MSYS, so it may be a matter of just implementing #compilerFlagsDebug for Windows confs and try to make it work. If you give it a try and it works, please let me know! So far, I did a little test with some compiler flags like -g3 and -O0 and it compiles and runs. But (as we will see later) when I do CTRL+C instead of get the gdb prompt, I kill GDB heheheh. Googling, seems to be a known problem.

So….open a console/terminal. The following are the minimal steps to run the VM under GDB in Mac OS:

cd blessed/results/CogMTVM.app/Contents/MacOS/
#If you compiled a StackVM or CogVM instead of a CogMTVM, then the .app and the executable will have another name
gdb CogMTVM

In Linux/Windows since you don’t have the .app directory of the Mac OS, it should be something like this;

cd blessed/results
#If you compiled a StackVM or CogVM instead of a CogMTVM, then the executable will those names instead
gdb CogMTVM

With the previous step, you should have arrived to a gdb console that looks like this:


So now we need to start the VM. In Mac, the VM automatically raises the File Selection popup that lets you choose which image to run. So the following line is enough:

(gdb) run

Choose the image and continue. In the Linux  you have to specify to the VM the .image to run by parameter. So, you have to do something like this:

(gdb) run  /home/mariano/Pharo/images/Pharo1.3.image

The idea is that you send by parameter the .image you want to run. So…at this point you have already launched your image, its time to play a bit. Open a Transcript. Open a workspace and evaluate:

9999 timesRepeat: [Transcript show: 'The universal answer is 42.']

Now…come back to the console, and do a CTRL+C. This will get a SIGINT Interruption and in the GDB console you should get something like:

Program received signal SIGINT, Interrupt.
0x9964a046 in __semwait_signal ()

What happened?  I am not an expert, but with such interruption we can “pause” the VM execution and the gdb takes the control. If you see your image at this moment, you will se that you cannot do anything in it. It is like “frozen”. If you now do:

(gdb) continue

You can also type “c” instead of “continue”. The image should continue running, and you Transcript should continue printing the universal answer 😉   So…what is cool about being able to interrupt the VM?  We can, for example:

  • Inspect the value of some variables and stack. Type “help data” and “help stack” in the gdb prompt.
  • Put breakpoints in the code and then after go step by step.
  • Invoke functions from our VM (this is really cool!!)

Let’s see a couple of examples….hit CTRL+C again to get the control of the gdb prompt and do:

(gdb) bt

Which should look similar this (why similar? because it depends where you stop it):

(gdb) bt
#0  0x0002dbd0 in longAtPointerput (ptr=0x22efa6e4 "\017??\024?\001", val=349999375) at sqMemoryAccess.h:82
#1  0x0002daf2 in longAtput (oop=586131172, val=349999375) at sqMemoryAccess.h:99
#2  0x0006a2ae in sweepPhase () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44356
#3  0x0003d8b4 in incrementalGC () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17846
#4  0x00069f61 in sufficientSpaceAfterGC (minFree=0) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44214
#5  0x00032e56 in checkForEventsMayContextSwitch (mayContextSwitch=1) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:11398
#6  0x0003ccad in handleStackOverflowOrEventAllowContextSwitch (mayContextSwitch=1) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17346
#7  0x000326c4 in ceStackOverflow (contextSwitchIfNotNil=525336580) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:11136
#8  0x1f40032e in ?? ()
#9  0x0006ab4c in threadSchedulingLoop (vmThread=0x828a00) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44714
#10 0x0003dd96 in initialEnterSmalltalkExecutive () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17970
#11 0x0003ebdc in initStackPagesAndInterpret () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:18435
#12 0x00022211 in interpret () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:2037
#13 0x00072142 in -[sqSqueakMainApplication runSqueak] (self=0x211090, _cmd=0x121c80) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms/iOS/vm/Common/Classes/sqSqueakMainApplication.m:172
#14 0x96339cbc in __NSFirePerformWithOrder ()
#15 0x91b9ee02 in __CFRunLoopDoObservers ()
#16 0x91b5ad8d in __CFRunLoopRun ()
#17 0x91b5a464 in CFRunLoopRunSpecific ()
#18 0x91b5a291 in CFRunLoopRunInMode ()
#19 0x920bde04 in RunCurrentEventLoopInMode ()
#20 0x920bdaf5 in ReceiveNextEventCommon ()
#21 0x920bda3e in BlockUntilNextEventMatchingListInMode ()
#22 0x9307378d in _DPSNextEvent ()
#23 0x93072fce in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#24 0x93035247 in -[NSApplication run] ()
#25 0x9302d2d9 in NSApplicationMain ()
#26 0x0006fca5 in main (argc=2, argv=0xbffff864, envp=0xbffff870) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms/iOS/vm/Common/main.m:52

Guess what it was doing ? Sure…an incremental GC 😉

(gdb) call printAllStacks()

Which should look similar to the following (notice that it is not the same place as the previous example and this is because the stacktrace in the middle of the incremental Garbage Collector was too long to put in the post, so I did a “bt” in another moment):

(gdb) call printAllStacks()
Process 0x2197b5f8 priority 10
0xbff650b0 I ProcessorScheduler class>idleProcess 526237088: a(n) ProcessorScheduler class
0xbff650d0 I [] in ProcessorScheduler class>startUp 526237088: a(n) ProcessorScheduler class
0xbff650f0 I [] in BlockClosure>newProcess 563590428: a(n) BlockClosure

Process 0x219edf1c priority 50
0xbff5f0b0 I WeakArray class>finalizationProcess 526237628: a(n) WeakArray class
0xbff5f0d0 I [] in WeakArray class>restartFinalizationProcess 526237628: a(n) WeakArray class
0xbff5f0f0 I [] in BlockClosure>newProcess 564059712: a(n) BlockClosure

Process 0x2067e180 priority 80
0xbff600d0 M Delay class>handleTimerEvent 526243268: a(n) Delay class
0xbff600f0 I Delay class>runTimerEventLoop 526243268: a(n) Delay class
543678476 s [] in Delay class>startTimerEventLoop
543678752 s [] in BlockClosure>newProcess

Process 0x2197b430 priority 60
0xbff610b0 I SmalltalkImage>lowSpaceWatcher 527603592: a(n) SmalltalkImage
0xbff610d0 I [] in SmalltalkImage>installLowSpaceWatcher 527603592: a(n) SmalltalkImage
0xbff610f0 I [] in BlockClosure>newProcess 563589972: a(n) BlockClosure

Process 0x219eba78 priority 60
0xbff62030 M [] in Delay>wait 564050624: a(n) Delay
0xbff62050 M BlockClosure>ifCurtailed: 565737144: a(n) BlockClosure
0xbff6206c M Delay>wait 564050624: a(n) Delay
0xbff62084 M InputEventPollingFetcher>waitForInput 526769836: a(n) InputEventPollingFetcher
0xbff620b0 I InputEventPollingFetcher(InputEventFetcher)>eventLoop 526769836: a(n) InputEventPollingFetcher
0xbff620d0 I [] in InputEventPollingFetcher(InputEventFetcher)>installEventLoop 526769836: a(n) InputEventPollingFetcher
0xbff620f0 I [] in BlockClosure>newProcess 564050332: a(n) BlockClosure

Process 0x212b6170 priority 40
0xbff5e03c M [] in Delay>wait 565740688: a(n) Delay
0xbff5e05c M BlockClosure>ifCurtailed: 565741260: a(n) BlockClosure
0xbff5e078 M Delay>wait 565740688: a(n) Delay
0xbff5e098 M WorldState>interCyclePause: 529148776: a(n) WorldState
0xbff5e0b4 M WorldState>doOneCycleFor: 529148776: a(n) WorldState
0xbff5e0d0 M PasteUpMorph>doOneCycle 527791904: a(n) PasteUpMorph
0xbff5e0f0 I [] in Project class>spawnNewProcess 528638248: a(n) Project class
556491024 s [] in BlockClosure>newProcess

Notice that in this case we are calling from GDB a exported function of the VM. Where does it come from?  For the moment, take a look to #printAllStacks. Now, it is time to kill the executable and finally to quit gdb:

(gdb) kill
(gdb) Quit

Notice that after doing the “kill” you can call “run” again an start everything again.  Hopefully, you got an idea of how to debug the VM. However, using gdb from command line is not the only option.

Debugging the VM with XCode

In the previous post we saw that CMake provides “generators”, i.e, a way to generate specific makefiles for different IDEs. Hence, we can generate makefiles for our IDE and debug the VM from there. Remember that executing “cmake help” in a console shows the help and at the end, there is a list of the available generators. In this particular example, I will use XCode in Mac OS. To do so, we need to follow the same steps than compiling a debug VM (that is, the same as building a release VM but using a debug CMakeVMMaker conf) but using the XCode generator in particular. That means that instead of doing “cmake .” we do “cmake -GXcode”. Notice that if we previously build another VM you will need to remove the CMake cache: /blessed/build/CMakeCache.txt. So..

cd blessed/build
cmake -G Xcode

Notice that it is not needed to do a “make” since we will do that from inside XCode. “cmake -G Xcode” generates a /blessed/build/CogMTVM.xcodeproj (the name will change if you generate a CogVM or a StackVM) that we directly open with XCode. Now…normally we should be able to select the target Debug (on the top left) and then from the menu “Build -> Build and Debug – Breakpoints On”. That should compile the VM and automatically run it with GDB. However, I found two problems:

  • External plugins are not correctly placed. I tried to fix it but I couldn’t. They are placed in blessed/results/CogMTVM.app/Contents/Resources/Debug  but they should be together in blessed/results/Debug/CogMTVM.app/Contents/Resources. The problem is that when XCode compiles, it adds a “Debug” in the directory when compiling with the “debug target” and the same for “release”. I have spent two days completely trying to solve it from the CMakeVMMaker and I couldn’t. If you know how to do it, please let me know. Anyway, the workaround is to copy all those .dylib from the first directory to the second one.
  • No matter that you change the settings in the project (like compiler flags, gcc version, etc) it will always the values from the CMake generated makefiles. This is a pity because I would like to change settings from XCode…

Now you are able to “Build -> Build and Debug – Breakpoints On”. If everything goes well, you should be able to open an image and the VM will be running with gdb. You will notice there is a little gdb button that opens a gdb console where you can do exactly the same as  if it were by command line. In addition, there is another button for the debugger. I attach a screenshot.

So…there you are, you are running the VM in debug mode from gdb and inside XCode. Interesting things from the gdb console is that you have buttons for “Pause” (what we did with gdb but from command line with the CTRL+C) and for “Continue” (what we type from gdb command line). And you have even a “Restart”.

Now….the nice thing about being able to debug the VM with XCode is the ability to easily put breakpoints in the code. What where? in which file? Without giving much details (because that’s the topic of another post), we will say that the “VM Core” (basically Interpreter classes + ObjectMemory classes) is translated into a big .c file called gcc3x-cointerpmt.c (in case of cogMT). A regular CogVM will be called “gcc3x-cointerp.c” the same as for StackVM. You can check this files by yourself. You should know where they are. Remember?  yes, they are (if you did the default configuration) in /blessed/src/vm  (for Cocoa configurations it may be in /blessed/stacksrc/vm). In such folder you will see that there is another file cointerpmt.c (or cointerp.c). Which is the difference between the one that has the “gcc3x” and the one that has not? for me moment let’s just say the one with the “gcc3x” has some optimizations (Gnuifier) that were automatically done in the C code and with the intention of compiling such sources with a gcc3x compiler.

We already know which file to debug “most of the times”. Maybe you need to debug another C file like plugins or the machine code generator, but most of the times, you will debug the “VM core” which is the file gcc3x-cointerpmt.c or its variant. Now what you have to do is to open it and put a breakppoint. On the top part of XCode you have a list of the files included in the project. Search for that one and select it. Be careful that XCode is really slow with this big file. Let’s put a breakpoint in the method lookup when the #doesNotUnderstand is thrown. This is done in the function “static sqInt lookupMethodInClass(sqInt class)” and here is a screenshot:

Once you set breakpoints you can “Build and Debug”. Now, instead of testing by executing something in a workspace, we will create a simple method anywhere (why not to use the workspace is not at my knowledge right now, sorry). So…I create the method foo in TestCase doing like this:

TestCase >> foo
self methodNonExistent.

And then I executed (this can be done in a workspace):

TestCase new foo

Once this is done, the VM should have been paused and you should have available the gdb prompt. You can now open the debugger or GDB console and do whatever you want. With the Debugger you can do “Step Over”, “Step Into”, “Step Out”, etc…And with the GDB console you can do exactly the same like if you were executing gdb from a terminal. Here is the screenshot:

There are much things I would like to talk about regarding debugging a VM, but the post is already too long and I think this is enough to start. I recommend you to watch the screencast.  Further in this blog sequence, we will do a second part. Now, it is time to understand some internal parts of the Squeak/Pharo VM.


Building the VM – Second Part

Hi folks. I guess that some readers do not like all the building part and want to go directly to see the VM internals. But it is really important that you understand how to change the VM, compile it or even debug it. Otherwise, you’ll be very limited.

This post is mostly about a couple of things that I wanted to mention in the previous post, but I couldn’t because it was already too long. If you read such post, you may think that compiling the VM from scratch is a lot of work and steps. But the post was long because of my explanations and because of my efforts in making it reproducible. This is why I would like to do a summary of how to compile the VM.

Summary of VM build

Assuming that you have already installed Git + CMake + GCC, then the following are the needed steps to compile the Cog VM:

mkdir newCog
cd newCog
git clone --depth 1 git://gitorious.org/cogvm/blessed.git
cd blessed/image
wget --no-check-certificate http://www.pharo-project.org/pharo-download/unstable-core
"Or we manually download with an Internet Browser the latest PharoCore
image from that URL and we put it in blessed/image

Then we open the image with a Cog VM (which we can get from here or here) and we evaluate:

Deprecation raiseWarning: false.
 Gofer new
 squeaksource: 'MetacelloRepository';
 package: 'ConfigurationOfCog';
(Smalltalk at: #ConfigurationOfCog) project latestVersion load.
"Notice that even loading CMakeVMaker is not necessary anymore
since it is included just as another dependency in ConfigurationOfCog"
MTCocoaIOSCogJitConfig generateWithSources.
"Replace this CMMakeVMMaker configuration class for the one that suites your OS
like CogUnixConfig and CogMsWindowsConfig"

Now, come back to the terminal and do:

cd newCog/blessed/build
cmake .
# Or  cmake . -G"MSYS Makefiles"  if you are in Windows

And that’s all, in “blessed/results” (in Windows it should be under “blessed/build/results”) you should have the CogVM binary. I know that you probably are a lazy guy, but if you really want to take advantage and learn in my posts, I strongly recommend you to follow those steps. All along this sequence of posts, we will debug and modify the VM (change GC, method lookup, create our own primitives and plugins, etc). Once you have Git and CMake, I promise the process takes less than 5 minutes.

Available CogVMs

Remember that all these posts is what I called “Journey through the VM”, so we will probably go and come back between different posts 🙂  In the first post,under the title “CogVM and current status” I explained the different flavors of CogVMs and the main features of them:

  1. Real and optimized block closure implementation. This is why from the image side blocks are now instances of BlockClosure instead of BlockContext.
  2. Context-to-stack mapping.
  3. JIT (just in time compiler) that translates Smalltalk compiled methods to machine code.
  4. PIC (polymorphic inline caching).
  5. Multi-threading.

What is the big difference between StackVM and CogVM? Well, Stack VM implements 1) and 2). And Cog VM is on top of the Stack VM and adds 3) and 4). Finally, there is CogMTVM which is on top of Cog VM and adds multi-threading support for external calls (like FFI for example).

In addition, Cog brings also some refactors. For example, in Interpreter VM, the Interpreter was a subclass of ObjectMemory. That was necessary in order to easily translate to C. In Cog, there are new classes like CoInterpreter and NewObjectMemory. But the good news is that we can have composition!! The CoInterpreter (which is a new class from Cog) has an instance variable that is the object memory (in this case an instance of NewObjectMemory). This was awesome and required changes in the SLANG to C translator.

As said, in the VMMaker part of the VM, what we called the “core”, there are mainly two important classes: Interpreter and ObjectMemory. Read the first post for details of their responsibilities. In Cog, there are a couple of differences:

  1. As said, the Cog Interpreter class do not subclass from ObjectMemory, but instead it is an instance variable.
  2. In Cog there isn’t only one Interpreter class like in the old VM. In fact each Cog VMs I told you (StackVM, CogVM, CogVMMT) has its own Interpreter class (StackInterpreter, CoInterpreter and CoInterpreterMT). Come on!! Don’t be lazy, take you image and browse them 🙂
  3. In Cog, there are not only those Interpreter classes that I have already told you, but also several more that are just for a design point of view, i.e, they are not Interpreter classes that should be used for compiling the VM. They are for example, to reuse code or to better simulate them. Examples, CoInterpreterPrimitives, StackInterpreterPrimitives, InterpreterPrimitives, etc. And then, of course, we have the Interpreter simulators, but that’s another story for another post.

So…if you are paying attention to this blog you may be asking yourself which Interpreter class you should use? My advice, and this is only my advice, is that you should normally use the “last one”. In this case, the CogVMMT. The few reasons I find not to use the last one are:

  1. If you are running on a hardware where Cog JIT is not supported. For example, for the iPhone the StackVM is usually used.
  2. When you are doing hacky things with the VM and you want to be sure there is no problems with JIT, PIC, etc. This is my case…
  3. Maybe for learning purposes the CogVM or CogVMMT is far much complicated than the StackVM or InterprertVM.
  4. The “last one” may not be the most stable one. So if you are in a production application you may want to deploy with a CogVM rather than a CogVM that has been released just now.

But apart from that, you will probably use the “last one” available. Just to finish with this little section, I let you a screenshot of a part of the Cog VMs hierarchy.

CMakeVMaker available configurations

In the previous post we saw what CMMakeVMMaker configurations do: 1) generate VM sources from VMMaker and 2) generate CMake files. 1) depends on which Cog (StackVM, CogVM and CogVM) we want to build, which plugins, etc. And 2) depends not only in which CogVM but also in the OS (the CMake files are not the same for each Operating System) and other things, like whether we are compiling for debug mode or not, whether we are using Carbon on Cococa library in Mac, etc. So…imagine the combination of: which CogVM, which OS, and whether debug mode or not. It gives us a lot of possibilities 🙂

The design decision to solve this in the CMakeVMake project was to create specific “configuration” classes. To summarize, there are at least one class for VM/OS. So you have, for example, CogUnixConfig (which is a CogVM, for Unix and “release”), CogDebugUnixConfig, MTCogUnixConfig, StackInterpreterUnixConfig, StackInterpreterDebugUnixConfig. And then for the rest of the OS is the same: CogMsWindowsConfig, StackInterpreterMsWindowsConfig, MTCogMsWindowsConfig, etc….So, your homework: browse the categories ‘CMakeVMMaker-Windows’, ‘CMakeVMMaker-Unix’ and  ‘CMakeVMMaker-IOS’. Look at the available classes. To learn, check implementors of #compilerFllags, #defaultInternalPlugins, #interpreterClass, etc…To test, take the debug variant, follow the same procedure as always, and you compile a debug VM with all the debugging symbols and no optimization 🙂

Which one you should use? I have already answered, but imagine you want the “last one”, then they are MTCocoaIOSCogJitConfig, MTCogUnixConfig and MTCogMsWindowsConfig.It doesn’t matter which configuration you choose, all you need to normally do is send the #generateWithSoources.

This design decision has a couple of advantages from my point of view:

  1. It is extremelly easy to customize. And in fact, there are already examples: CogUnixNoGLConfig (which doesn’t links against OpenGL so it works perfect unless you use Balloon3D or Croquet plugins), CogFreeBSDConfig (specially for BSD since it has a couple of differences in the compiler flags), etc.
  2. YOU can subclass and change what you want: default internal or external plugins, compiler flags, etc.
  3. It is easy for a continuous integration server like Hudson to build different targets.

Customizing CMakeVMMaker configurations

I told you that you can subclass from a specific class and overwrite the compiler flags, the default plugins and if they should be internal or external, etc. However, CMMakeVMaker can be parametrized in several ways while using them. In the building instructions at the beginning of this blog, I told you to move your Pharo image to blessed/image. And as I explained in the previous post that was in order to let CMakeVMaker take the defaults directories and make it work out of the box. But in fact, it is not necessary at all to move your image. You can download the “platforms code” in some place and the image elsewhere. Notice that these changes (the ability to customize each direcotry) has been commited in new versions of the CMakeVMMaker package. So, if you want to really try the followin code, make sure to have CMakeVMMaker-MarianoMartinezPeck.94. You can get it using Monticello Browser or Gofer.

So, you can do something like this:

"The image where this code is being run can be in ANY place"
MTCocoaIOSCogJitConfig new
srcDir: '/Users/mariano/Pharo/generateCode/src';
platformsDir: '/Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms';
buildDir: '/Users/mariano/Pharo/vms/build';
"The resources directory is only needed for Mac"
resourcesDir: '/Users/mariano/Pharo/vm/git/cogVM2/blessed/macbuild/resources';
outputDir: '/Users/mariano/binaries/results';

The “platformsDir” must  map with “platforms” directory that we downloaded with Git, it cannot be choosed randomly. The same with the “resourcesDir” (which in fact is only for Mac). The rest of the directories (src, build and output) are not created by VMMaker nor Git. They are just directories that I have created by my own and I want to use them instead of the default.

And I’ve created this shortcut also:

"The image where this code is being run can be in ANY place"
MTCocoaIOSCogJitDebugConfig new
defaultDirectoriesFromGitDir: '/Users/mariano/Pharo/vm/git/cogVM1/blessed';

That way, I don’t need to move my image to blessed/image. BTW, don’t try this with Windows confs because there still a problem. Anyway, despite from that we can also customize things using #internalPlugins:,  #externalPlugins, etc.

Synchronization between platform code (Git) and VMMaker

In this post, I told you the problems I have seen so far with “process” of the Interpreter VM + SVN for platform code. And I also told you how this new process (CMake + Git ) helps a bit in some of those problems. From my point of view there are a couple of things that have improved the process:

  1. Platform code and VMMaker are be in sync: when people (Igor, Esteban, etc) commit a new version to Git, they make sure that the VMMaker part is working.
  2. Documentation of that synchronization: in the previous post, I told you to load version ‘1.5’ of ConfigurationOfCog. Suppose I didn’t tell you that, how do you know for a certain Git version, which version of ConfigurationOfCog you should use?  Check in blessed/codegen-scripts/LoadVMMaker.st  and you have exactly the piece of code you should execute to get the working VMMaker with that specific version of GIT. So…this means that when someone commits the Git repository and such changes require a new VMMaker version, then such developer needs to create a new version of ConfoigurationOfCog, and modify LoadVMMaker.st.  Now that you know this, the steps I told you at the beginning of this posts can be automatic, can’t they?  someone say uncle Hudson?  Yes, of course!!
  3. Git is easier in the fact that people can fork, hack, experiment, tests, and then push changes into the blessed.

Hudson for building VMs

Pharo has a continuous integration server with Hudson: http://ci.pharo-project.org/. And as you can see here, there are a lot of targets for CogVMs. Basically, for every single commit in Git, Hudson builds all those images. How? Following nearly the same steps I told you at the beginning of this post. It creates StackVMs, CogVMs and CogVMs for every OS. In fact, there are no Windows builds yet because this week they are getting the Windows slave. But the confs and the procure is working…So it is just a matter of getting the Windows box.

Conclusion: you don’t need to wait one year an a half to get a VM with a bug fix, nor you don’t need to compile it by yourself. With Hudson, they are built for every commit.

Hudson traceability

We saw how we can trace from platform code to VMMaker. Now, how to know how was every Hudson VM build ? Easy:

  1. Go to http://ci.pharo-project.org
  2. Choose a target in the “Cog” tab. For example, I choose “Mac Cog Cocoa”
  3. Follow the link, for example Cog Unix,  and there you can see two artifacts:
  • a built VM
  • a source code tarball, which is used to build that VM (in this example, CocoaIOSCogJitConfig-sources.tar.gz)

If you Download the source code archive and unpack it into your local directory what would you expect?? Of course, a copy of the git directory plus the Pharo image generated to build such VM. Such image is in build/ subdirectory and it is called generator.image and was the used to generate source code (located in src/ subdirectory) and CMake configuration files (located in build/ subdirectory). Isn’t this cool ?

CMake generators

Did I already tell you that I am also a CMake newbie? Ok…just in case 😉  Anyway, imagine CMake like a tool where we can set things, parameters, variables, directories, etc, in some files (which in our case they are auto-generated by CMakeVMMaker) and then from those “general” files we can generate specific and different makefiles. So, from the same CMake files we can generate different kind of makefiles,  i.e, we can generate makefiles the way some IDE except them to be. CMake call this ability “generators”. And the way to create makefiles with a specific generator is like this:

cmake -G "Generator Name"

Does that sound familiar?? Of course! We have already used them for MSYS in Windows. The cool thing is that there are generators for several IDEs. And this is just GREAT. For example, I can create makefiles and a project for XCode (the C IDE for MacOS). Just doing:

cmake -G Xcode

creates a XCode project for CogVM which is  in /blessed/build/CogMTVM.xcodeproj. You don’t have an idea how cool is this. This mean you can open XCode and everything is set and working out of the box for the CogVM. You can put breakpoints, inspect C code, compile, debug, everything….Before, this was much more complicated because the .xcodeproj file was versioned in the SVN and this file usually keeps some file locations or things like that and in my experience, it was always a pain to make it work.

When you use a particular generator for an IDE (like Xcode, Eclipse, KDevelop, Vsual Studio, etc, you usually don’t do the “make” by hand. So, after invoking cmake, you won’t need to do a make. Instead, you have to compile from the IDE itself (which should have the correct makefiles).

How do you know which are the available generators?  just type:

cmake --help

and at the end you’ll find a section that says “The following generators are available on this platform:”  and each of them has a name and a description. What you need to pass to the -G parameter is the name. Notice that as the help says, it automatically shows the generators available in YOUR platform (OS).  Some examples:

cmake -G KDevelop3
cmake -G "Eclipse CDT4 - Unix Makefiles"
cmake -G "Visual Studio 10"
cmake -G "Borland Makefiles"

When the name includes more than one word you must use the double quotes.

So…the 2 main advantages I see from CMake to our process is: cross compiling, and be able to automatically create makefiles for IDEs. Sorry I couldn’t try with other IDE than Xcode. If you try it and it works, let me know 🙂

In the next post we will so how to debug the VM and some related tricks. After that post, we will probably start to see the VM internals since you will have already all the needed tools.