Tag Archives: build

StartupLoader: running startUp scripts in Pharo

Hi. Some time ago, I wrote a post about how I build images for my projects. I am downloading new images all the time and because of that I used to have 2 problems: 1) I needed to load several external packages to the standard Pharo image; 2) I needed to set my own settings and preferences. Today, Pharo 2.0 (which is in development and unstable state) includes most of the packages I always needed to install: shout, code completion, nice browser (Nautilus or OB), refactoring integration, spotlight, etc.  So nowadays I only had problem 1).

In this post, I will show you how I solve that problem using StartupLoader, a nice utility present in Pharo (since Pharo 1.4). IMPORTANT: Everything I mention in this blog is taking into account the “new version” I did of StartupLoader. Before writing this post I have improved it and therefore you need at least a Pharo 2.0 version 20071.

UPDATE: Stephane Ducasse read,  converted and improved this blog post into a new chapter for a future Pharo book. You can find a draft here StartupPreferences.pier.pdf

Why we need StartupLoader?

How can we execute something at startup with Pharo? There have traditionally been two known ways (and now in Pharo there is a new third option and it is the reason of this post):

1) Send a .st file as parameter to the VM, that is, you execute the VM like this:

/Users/mariano/Pharo/VM/Pharo.app/Contents/MacOS/Pharo /Users/mariano/Pharo/images/Pharo-1.4.image startup.st

You execute the VM, pass the Pharo image as parameter and then the file. This will be executed during the startup. What’s the problem with this? If I want to always execute the startup in different images I always need to open the image this way, from command line. I just want to open an image with a double-click. Moreover, this file is hand-coded and not versioned in Monticello (what I would like). Besides, there are more limitations as I mention later in this blog.

2) Register your own class in Smalltalk startup list and implement #startUp message to do whatever you want to do. The problem is that my class is not present in the distributed images of Pharo. Therefore I need to manually first load my own code. Same problem: too much work.

New new tool

Remember my problem: I am downloading new images all the time. Having to manually set up my preferences is boring and time-consuming.

The new StartupLoader class searches for and executes .st files from certain locations.  To find these it searches for a ‘.config’ folder in the folder where the image file sits.  Then it looks in the next folder up, then up again and so on until reaching the root folder.  When a ‘.config’ folder is found, StartupLoader looks within this for a ‘pharo’ folder. This contains the startup scripts common to all versions of Pharo, and also optionally a folder per Pharo version holding startup scripts suitable for that version only.  So a typical directory layout might be…

.../some/folders/pharo/Content/Resources/pharo.image.
.../some/folders/pharo/Content/Resources/startup.st
.../some/folders/.config/pharo/author.st
.../some/folders/.config/pharo/useSharedCache.st
.../some/folders/.config/pharo/1.4/mystartupFor14only.st
.../some/folders/.config/pharo/2.0/mystartupFor20only.st
(**Note however that ‘.config’ is an invalid filename on Windows, so ‘..config’ is used instead)

IMPORTANT: I said that StartupLoader will search for a folder ‘.config’ starting from the image directory until the root of the filesystem. What happens if no folder is found? It creates ‘.config’ in the image folder. However, I recommend you create the ‘.config’ following the standard, that is, in the $HOME.

To know the real values for you…
Print the result of “FileDirectory preferencesGeneralFolder” which holds the startup scripts common to all versions of Pharo.
Print the result of “FileDirectory preferencesVersionFolder” which holds the startup scripts specific to the version of the current image.

The order of the search is from the most general to the most specific:

  1. General preferences folder: This is general for all Pharo versions This folder is shared for all the images you open. In my case (in MaxOSX) and Pharo 2.0, it is ‘/Users/mariano/.config/pharo/’. In this place, StartupLoader will load ALL existing .st files. This type of startup is useful when we have something to execute for all images of all Pharo versions.
  2. Preference version folder: This is a specific folder for a specific Pharo version. In my case it is  ‘/Users/mariano/.config/pharo/2.0/’. This type of startup is useful when we have something to execute for all images of a specific Pharo version.
  3. The image folder: The startup only searches for a file called ‘startup.st’. So if you have such a file in the same directory where the image is, then such script will be executed automatically. This type of startup is usually used to do something that is application-specific or something that only makes sense for the specific image you are using. Now you might ask why we don’t search the image folder for multiple .st files.  This is because it is normal for the image folder to contain .st files not related to started – such as from any file out.  Using one specific file ‘startup.st‘ avoids this while still allowing an image delivered to a client to run a script upon execution on a new system. Be careful if you already were sending your own ‘startup.st’ as parameter to the VM because it will be executed twice 😉

As you can see the order is from the most general to the most specific. Moreover, it does not stop when it finds files in any of them. So all are searched and executed. More specific scripts can even override stuff set in more general ones. It works more or less the same way as variables in UNIX with .bashrc /etc/envirorment, etc…

So you know where the system will search startup files. Now you can directly put your code there and it will be automatically executed during startup. Great!!! So that’s all?  we just write scripts there?  Of course not! 😉

StartupActions

Directly putting code in the files is easy, however, it is not the best choice. For example, what happens if there are certain scripts you want to execute only once on a certain image but some other code that you want to execute each time the image starts? To solve that, among other things, we have the reification of StartupAction. Let’s see how we use them:

| items |
items := OrderedCollection new.
items add: (StartupAction
name: 'Basic settings'
code: [
Author fullName: 'MarianoMartinezPeck'.
Debugger alwaysOpenFullDebugger: true.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'basicSettings.st'.

What we do first is to create an instance of StartupAction using the message #name:code:. We pass as argument a name and the Smalltalk code we want to run inside a block closure. In this example, I just set my username and I put a setting to always open the debugger (no pre-debugger window). So far nothing weird.

The magic comes with the last line, the message #addAtStartupInPreferenceVersionFolder: items named: aFileName  receives a list of startup actions and a filename and stores the actions in a file with the passes argument. So in this case we have only one action called ‘Basic Settings’ and it will be placed in a file called ‘basicSettings.st’. But where? in which of the 3 folders described previously is it placed? well…it depends on the message. In this case, we used the #addAtStartupInPreferenceVersionFolder:named:  (notice the “InPreferenceVersionFolder”). So it put the files in 2). In addition, you can use the messages #addAtStartupInGeneralPreferenceFolder:named: which stores files in 1) and #addAtStartupInImageDirectory: which stores in 3). Notice that with the first two messages we can specify the file name but with the last one we can’t. Remember the last one is always called ‘startup.st’. If you are lazy and don’t want to think the name yourself, you can just use #addAtStartupInPreferenceVersionFolder: which creates a file called ‘startupPharoNN.st’ or #addAtStartupInGeneralPreferenceFolder: that creates a file named ‘startupPharo.st’.

StartupLoader

We saw that when executing the message #addAtStartupInPreferenceVersionFolder: items named:aFilename or any of its variant, a file is created with the code we want to evaluate. Then, when the system starts it will find our file and execute our code. But, how is the resulting file? Exactly as the code we provided? No! Look how our example file ‘/Users/mariano/.config/pharo/2.0/basicSettings.st’ is generated:

StartupLoader default executeAtomicItems: {
StartupAction name: 'Basic settings' code: [Author fullName: 'MarianoMartinezPeck'.
Debugger alwaysOpenFullDebugger: true].
}.

So as you can see, the file is generated by sending a collection of actions to “StartupLoader default executeAtomicItems:”. In this example the collection has only one action, but it would have more if our example has more. So now the StartupLoader will execute all the actions found in the file. Do we execute all actions? No! Actions can be built with a property of “runOnce”. So if an action has already been executed in the current image before the last save, it is not executed again. Executed actions are stored in the singleton instance #default of StartupLoader. Therefore, you have to save the image. If an action generates an error the action is NOT registered as executed. In addition, errors are also stored in the singleton of StartupLoader so you can query them after starting the image by inspecting the result of “StartupLoader default errors”.

Advanced example

As an advanced example, I want to show you the script I am using for my images. For that, I have this Smalltalk code:

setPersonalStartUpPrefernces
"self setPersonalStartUpPrefernces"
| items |
items := OrderedCollection new.

items add: (StartupAction name: 'General Preferences for all Pharo versions' code: [
FileStream stdout lf; nextPutAll: 'Setting general preferences for all Pharo versions'; lf.
Author fullName: 'MarianoMartinezPeck'.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInGeneralPreferenceFolder: items named: 'generalSettings.st'.

items add: (StartupAction name: 'Settings' code: [
FileStream stdout lf; nextPutAll: 'Setting general preferences'; lf.
UITheme currentSettings fastDragging: true.
CodeHolder showAnnotationPane: true.
MCCodeTool showAnnotationPane: true.
Deprecation raiseWarning: true.
Debugger alwaysOpenFullDebugger: true.
Parser warningAllowed: false.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'settings.st'.

items removeAll.
items add: (StartupAction name: 'Nautilus' code: [
FileStream stdout lf; nextPutAll: 'Executing Nautilus related stuff'; lf.
Nautilus pluginClasses add: { NautilusBreadcrumbsPlugin. #top }.
Nautilus pluginClasses add: { AnnotationPanePlugin. #middle }.
FileStream stdout lf; nextPutAll: 'Finished'; lf.
] runOnce: true).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'nautilus.st'.

items removeAll.
items add: (StartupAction name: 'Monticello related stuff' code: [
| sharedPackageCacheDirectory |
FileStream stdout lf; nextPutAll: 'Executing Monticello related stuff'; lf.
sharedPackageCacheDirectory := (FileDirectory on: '/Users/mariano/Pharo/localRepo/')
assureExistence;
yourself.
MCCacheRepository default directory: sharedPackageCacheDirectory.
MCDirectoryRepository defaultDirectoryName: '/Users/mariano/Pharo/localRepo/'.
(MCRepositoryGroup default  repositories
select: [:each | (each isKindOf: MCHttpRepository)
and: [((each locationWithTrailingSlash includesSubString: 'www.squeaksource.com')
or: [each locationWithTrailingSlash includesSubString: 'http://ss3.gemstone.com/ss/'])]
]) do: [:each |
each
user: 'MMP';
password: ((FileDirectory default oldFileNamed: '/Users/mariano/Pharo/repositoriesPassword.txt') contents).
].
FileStream stdout lf; nextPutAll: 'Finished'; lf.
]).
StartupLoader default addAtStartupInPreferenceVersionFolder: items named: 'monticello.st'.

Basically, I have 4 files to customize stuff: 1) general settings;  2) settings for Pharo 2.0; 3) nautilus and 4) monticello related stuff. 1) is for all Pharo versions. So far I am just setting my username. 2) 3) and 4) are for Pharo 2.0 (just because I know they work in Pharo 2.0 and I am not sure if they work in other versions). For nautilus, I don’t want to add the plugins each time (because it would add the plugin several times) so I create a StartupAction using the message #name: nameOfItem code: code runOnce: aBoolean  passing a true to aBoolean.

Using the tool

How to split your stuff in files and actions?

So you may have noticed that: a) #addAtStartupInPreferenceVersionFolder: and friends expect a list of actions; b) you can have multiples files. So, how do you split your code? From what I can see in the framework, there is no restriction. You can have as many actions per files and as many files as you wish. An action has a block of closure that can contain as much code as you want.

I found that one way of splitting your code is when some actions need to be executed only once and some other each time. Another reason may be some code which may be expected to fail for some reason. If it fails, the code after the line that generated the error won’t be executed. Hence, you may want to split that code to a separate action.

How to version and work with this tool?

The way I found to work with this stuff is to have my own class GeneralImageBuilder (put whatever name you want). In such class I have the mentioned method #setPersonalStartUpPrefernces (from the advanced example). So I use Monticello to save and load that project. Then, whenever I want to create the script files and add them to their correct directory, I just evaluate that method.

Be careful with the cache!

In order to support the “runOnce:”, actions are stored in the singleton instance of StartupLoader. After an action is executed (if executed correctly), the action is stored and marked as “executed”. It may happen that later on you modify the scripts by hand, or you change the rules and re-store them or some kind of black magic. So…if you change some of these, I recommend to do 2 things:

  • Remove all existing files from the preference directories (no script is removed automatically). Check methods #remove* in StartupLoader.
  • Remove the existing stored actions in StartupLoader. Check method #cleanSavedActionsAndErrors and #removeAllScriptsAndCleanSavedActions.

Conclusion

I think that the tool is very nice. It is just 3 classes and a very few methods. I have been improving it recently but still, there could be more improvements. Wanna help?  I wanted to summarize this post and write a better class comment, but I am running out of free time. In addition, it would be nice to have some tests 😉

I want to thank to Benjamin Van Ryseghem for doing the first version of the tool and to Ben Coman for fixing the preference folder in Windows and for discussing with me about the performed improvements.

Hope you like it!

Advertisements

How to debug the VM?

Hi. Whether you are experimenting and hacking in the VM (where it is likely that some things will go wrong) or you are running an application in production, it is always useful to know how to debug the VM.  In this post, we will see how to compile the VM with all the debug information, how to run the VM from GDB (the GNU Project Debugger) and how to debug it using an IDE’s debugger.

Before going further, I would like to tell you something I did last weekend (apart from sleeping quite a lot).  Laurent Lauffont, creator of the PharoCasts, started a new sequence of screencasts called ” PharoCasts with Experts“. The idea is to do a more or less 1 hr “interview” about certain topic. Laurent calls by Skype to the “expert” and they can talk, ask questions, etc. Using TeamViewer, the “expert” shares his desktop to Laurent, who is in addition recording the screencast. The screencast is finally edited and uploaded to http://www.pharocasts.com . Ok, I am not a VM expert at all, but last week we did a 1:30 screencast about compiling and debugging the VM 🙂   This screencast contemplates all what we have learned in the previous posts and what you will learn today.

So…let’s start the show.

Prepared image for you

In previous posts, I told you that Pharo guys recommend us to use the latest PharoCore for building the CogVMs. There are several reasons for this but the basic idea is that they want the VM can be build in latest versions, and if cannot, then fix it as soon as possible. But we have a much better testers than you, and it is called Hudson. So…I don’t want to force you to use the latest unstable PharoCore (which in fact, it is unstable sometimes). Even more, if you are starting in the VM world, I don’t want to add even more problems. So…from now onward, I have prepared and image so that you can use. At the time of this writing, there is no PharoDev 1.3 yet, but all the VM tools doesn’t load directly in a Pharo 1.2. There are 3 little changes that are needed. So…you can grab a PharoDev 1.2.1 and file in these changes, or directly load this image that I have prepared for you. This image has only those 3 changes. It does not contain any Cog or CMakeVMMaker package, that’s your homework 😉   The good news is that since we use a Dev image now we have all the development tools like refactorings, code completion, code highlighting, etc.

When do we need to debug the VM?

I think it is a good idea to start thinking when we need to debug the VM. In my little experience, I have found the following reasons:

  1. When there is a crash and you cannot find why it was.
  2. When you are developing something on the VM.
  3. For learning purposes.
  4. When you are just a hacky geek 🙂

I think most people will need the first one.

Debugging and optimizing a C program

If you are reading this post, you are probably a Smalltalker. And if you are a Smalltalker, I know how much you love the debugger. We all do. But there are guys that are not as lucky as we are 😉 Now, being serious, how do we debug a program written in C ? I mean, the VM, at the end, is a program compiled in C and executed like any other program. I am not a C expert so I will do a quick introduction.

When you normally compile a C program the binary you get does not contain the C source code used to generate such binaries. In addition, the names of the variables or the functions may be lost. Hence, it is really hard to be able to debug a program. This is why C compilers (like GCC) usually provide a way to compile a C program with “Debugging Symbols“, which add the necessary information for debugging to the binaries. Normally, as we will see today, these symbolic symbols are included directly in the binary. However, they can be sometimes separated in a different file. The GCC flag for the debugging symbols is -g  where -g3 is the maximum and -g0 the minimal (none). There are much more flags related to debugging but that is not the main topic of this post.

In addition to the debugging, a C compiler usually provides what they call “Compiler Optimizations”. This means that the compiler try to optimize the resulted binary in different ways: speed, memory, size, etc. When compiling for debugging it is common to disable optimizations, why? because, as its documentation says some variables you declared may not exist at all; flow of control may briefly move where you did not expect it; some statements may not be executed because they compute constant results or their values were already at hand; some statements may execute in different places because they were moved out of loops. The GCC flag for the optimization is -O  where -O3 is the maximum and -O0 no optimization (default).

But we are Smalltalkers, so much better if someone can take care about all these low-level details. In this case, CMakeVMMaker is that guy.

Building a Debug VM

As I told you in the previous post, there are several CMakeVMMAker configurations classes, and some of them are for compiling a debug VM. Examples: MTCocoaIOSCogJitDebugConfig, StackInterpreterDebugUnixConfig, CogDebugUnixConfig, etc. What is the difference between those configurations and the ones we have used so far ? The only difference is the compiler flags. These “Debug” VMs use special flags for debugging. Normally, these flags include things like -g3 (maximum debugging information), -O0 (no optimization), etc. For more details, check implementors of #compilerFlagsDebug. But…since there were a couple of fixes in the latest commit, I recommend you to load the version “CMakeVMMaker-MarianoMartinezPeck.94” of the CMakeVMMaker package.

So…how do we build the debug VM?  Exactly the same way as the regular (“deploy”)  VMs (I have explained that here and here), with only one difference. Guess which one? Yes, of course, use the debug configuration instead of the normal one. That means that in Mac for example, instead of doing “MTCocoaIOSCogJitConfig  generateWithSources”  we do “MTCocoaIOSCogJitDebugConfig  generateWithSources”. And that’s all. Once you finish all the build steps, you have a debug VM.

What are you waiting? Fire up your image and create a debug VM 😉  Now…how do you know if you really succeeded or not to create a debug VM?  Easy: it will be much slower and the VM binary will be bigger (because of the symbolic symbols addition). In a regular CogMTVM, if you evaluate “1 tinyBenchmarks” you may get something arround  ‘713091922 bytecodes/sec; 103597027 sends/sec’ whereas with a debug VM something arround ‘223580786 bytecodes/sec; 104885548 sends/sec’. So..as you can notice, a debug VM is at least 3 times slower.

Debugging the VM with GDB

Disclaimer: I am not a GDB expert, since most of the times I debug from the IDE (XCode). I will give you a quick introduction. So…we have compiled our with all the debug flags which means that the VM binaries now have extra information so that we can debug. GDB is a command line debugger from GNU Project and it is the “standard” when debugging C programs. In addition, it works in most OS. Notice that for Windows there are not VMMakeVMMaker debug configurations. I think this is just because nobody tried it. As far as I know gdb works with MSYS, so it may be a matter of just implementing #compilerFlagsDebug for Windows confs and try to make it work. If you give it a try and it works, please let me know! So far, I did a little test with some compiler flags like -g3 and -O0 and it compiles and runs. But (as we will see later) when I do CTRL+C instead of get the gdb prompt, I kill GDB heheheh. Googling, seems to be a known problem.

So….open a console/terminal. The following are the minimal steps to run the VM under GDB in Mac OS:

cd blessed/results/CogMTVM.app/Contents/MacOS/
#If you compiled a StackVM or CogVM instead of a CogMTVM, then the .app and the executable will have another name
gdb CogMTVM

In Linux/Windows since you don’t have the .app directory of the Mac OS, it should be something like this;

cd blessed/results
#If you compiled a StackVM or CogVM instead of a CogMTVM, then the executable will those names instead
gdb CogMTVM

With the previous step, you should have arrived to a gdb console that looks like this:

(gdb)

So now we need to start the VM. In Mac, the VM automatically raises the File Selection popup that lets you choose which image to run. So the following line is enough:

(gdb) run

Choose the image and continue. In the Linux  you have to specify to the VM the .image to run by parameter. So, you have to do something like this:

(gdb) run  /home/mariano/Pharo/images/Pharo1.3.image

The idea is that you send by parameter the .image you want to run. So…at this point you have already launched your image, its time to play a bit. Open a Transcript. Open a workspace and evaluate:

9999 timesRepeat: [Transcript show: 'The universal answer is 42.']

Now…come back to the console, and do a CTRL+C. This will get a SIGINT Interruption and in the GDB console you should get something like:

Program received signal SIGINT, Interrupt.
0x9964a046 in __semwait_signal ()
(gdb)

What happened?  I am not an expert, but with such interruption we can “pause” the VM execution and the gdb takes the control. If you see your image at this moment, you will se that you cannot do anything in it. It is like “frozen”. If you now do:

(gdb) continue

You can also type “c” instead of “continue”. The image should continue running, and you Transcript should continue printing the universal answer 😉   So…what is cool about being able to interrupt the VM?  We can, for example:

  • Inspect the value of some variables and stack. Type “help data” and “help stack” in the gdb prompt.
  • Put breakpoints in the code and then after go step by step.
  • Invoke functions from our VM (this is really cool!!)

Let’s see a couple of examples….hit CTRL+C again to get the control of the gdb prompt and do:

(gdb) bt

Which should look similar this (why similar? because it depends where you stop it):

(gdb) bt
#0  0x0002dbd0 in longAtPointerput (ptr=0x22efa6e4 "\017??\024?\001", val=349999375) at sqMemoryAccess.h:82
#1  0x0002daf2 in longAtput (oop=586131172, val=349999375) at sqMemoryAccess.h:99
#2  0x0006a2ae in sweepPhase () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44356
#3  0x0003d8b4 in incrementalGC () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17846
#4  0x00069f61 in sufficientSpaceAfterGC (minFree=0) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44214
#5  0x00032e56 in checkForEventsMayContextSwitch (mayContextSwitch=1) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:11398
#6  0x0003ccad in handleStackOverflowOrEventAllowContextSwitch (mayContextSwitch=1) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17346
#7  0x000326c4 in ceStackOverflow (contextSwitchIfNotNil=525336580) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:11136
#8  0x1f40032e in ?? ()
#9  0x0006ab4c in threadSchedulingLoop (vmThread=0x828a00) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:44714
#10 0x0003dd96 in initialEnterSmalltalkExecutive () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:17970
#11 0x0003ebdc in initStackPagesAndInterpret () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:18435
#12 0x00022211 in interpret () at /Users/mariano/Pharo/vm/git/cogVM2/blessed/src/vm/gcc3x-cointerpmt.c:2037
#13 0x00072142 in -[sqSqueakMainApplication runSqueak] (self=0x211090, _cmd=0x121c80) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms/iOS/vm/Common/Classes/sqSqueakMainApplication.m:172
#14 0x96339cbc in __NSFirePerformWithOrder ()
#15 0x91b9ee02 in __CFRunLoopDoObservers ()
#16 0x91b5ad8d in __CFRunLoopRun ()
#17 0x91b5a464 in CFRunLoopRunSpecific ()
#18 0x91b5a291 in CFRunLoopRunInMode ()
#19 0x920bde04 in RunCurrentEventLoopInMode ()
#20 0x920bdaf5 in ReceiveNextEventCommon ()
#21 0x920bda3e in BlockUntilNextEventMatchingListInMode ()
#22 0x9307378d in _DPSNextEvent ()
#23 0x93072fce in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#24 0x93035247 in -[NSApplication run] ()
#25 0x9302d2d9 in NSApplicationMain ()
#26 0x0006fca5 in main (argc=2, argv=0xbffff864, envp=0xbffff870) at /Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms/iOS/vm/Common/main.m:52
(gdb)

Guess what it was doing ? Sure…an incremental GC 😉

(gdb) call printAllStacks()

Which should look similar to the following (notice that it is not the same place as the previous example and this is because the stacktrace in the middle of the incremental Garbage Collector was too long to put in the post, so I did a “bt” in another moment):

(gdb) call printAllStacks()
Process 0x2197b5f8 priority 10
0xbff650b0 I ProcessorScheduler class>idleProcess 526237088: a(n) ProcessorScheduler class
0xbff650d0 I [] in ProcessorScheduler class>startUp 526237088: a(n) ProcessorScheduler class
0xbff650f0 I [] in BlockClosure>newProcess 563590428: a(n) BlockClosure

Process 0x219edf1c priority 50
0xbff5f0b0 I WeakArray class>finalizationProcess 526237628: a(n) WeakArray class
0xbff5f0d0 I [] in WeakArray class>restartFinalizationProcess 526237628: a(n) WeakArray class
0xbff5f0f0 I [] in BlockClosure>newProcess 564059712: a(n) BlockClosure

Process 0x2067e180 priority 80
0xbff600d0 M Delay class>handleTimerEvent 526243268: a(n) Delay class
0xbff600f0 I Delay class>runTimerEventLoop 526243268: a(n) Delay class
543678476 s [] in Delay class>startTimerEventLoop
543678752 s [] in BlockClosure>newProcess

Process 0x2197b430 priority 60
0xbff610b0 I SmalltalkImage>lowSpaceWatcher 527603592: a(n) SmalltalkImage
0xbff610d0 I [] in SmalltalkImage>installLowSpaceWatcher 527603592: a(n) SmalltalkImage
0xbff610f0 I [] in BlockClosure>newProcess 563589972: a(n) BlockClosure

Process 0x219eba78 priority 60
0xbff62030 M [] in Delay>wait 564050624: a(n) Delay
0xbff62050 M BlockClosure>ifCurtailed: 565737144: a(n) BlockClosure
0xbff6206c M Delay>wait 564050624: a(n) Delay
0xbff62084 M InputEventPollingFetcher>waitForInput 526769836: a(n) InputEventPollingFetcher
0xbff620b0 I InputEventPollingFetcher(InputEventFetcher)>eventLoop 526769836: a(n) InputEventPollingFetcher
0xbff620d0 I [] in InputEventPollingFetcher(InputEventFetcher)>installEventLoop 526769836: a(n) InputEventPollingFetcher
0xbff620f0 I [] in BlockClosure>newProcess 564050332: a(n) BlockClosure

Process 0x212b6170 priority 40
0xbff5e03c M [] in Delay>wait 565740688: a(n) Delay
0xbff5e05c M BlockClosure>ifCurtailed: 565741260: a(n) BlockClosure
0xbff5e078 M Delay>wait 565740688: a(n) Delay
0xbff5e098 M WorldState>interCyclePause: 529148776: a(n) WorldState
0xbff5e0b4 M WorldState>doOneCycleFor: 529148776: a(n) WorldState
0xbff5e0d0 M PasteUpMorph>doOneCycle 527791904: a(n) PasteUpMorph
0xbff5e0f0 I [] in Project class>spawnNewProcess 528638248: a(n) Project class
556491024 s [] in BlockClosure>newProcess
(gdb)

Notice that in this case we are calling from GDB a exported function of the VM. Where does it come from?  For the moment, take a look to #printAllStacks. Now, it is time to kill the executable and finally to quit gdb:

(gdb) kill
(gdb) Quit

Notice that after doing the “kill” you can call “run” again an start everything again.  Hopefully, you got an idea of how to debug the VM. However, using gdb from command line is not the only option.

Debugging the VM with XCode

In the previous post we saw that CMake provides “generators”, i.e, a way to generate specific makefiles for different IDEs. Hence, we can generate makefiles for our IDE and debug the VM from there. Remember that executing “cmake help” in a console shows the help and at the end, there is a list of the available generators. In this particular example, I will use XCode in Mac OS. To do so, we need to follow the same steps than compiling a debug VM (that is, the same as building a release VM but using a debug CMakeVMMaker conf) but using the XCode generator in particular. That means that instead of doing “cmake .” we do “cmake -GXcode”. Notice that if we previously build another VM you will need to remove the CMake cache: /blessed/build/CMakeCache.txt. So..

cd blessed/build
cmake -G Xcode

Notice that it is not needed to do a “make” since we will do that from inside XCode. “cmake -G Xcode” generates a /blessed/build/CogMTVM.xcodeproj (the name will change if you generate a CogVM or a StackVM) that we directly open with XCode. Now…normally we should be able to select the target Debug (on the top left) and then from the menu “Build -> Build and Debug – Breakpoints On”. That should compile the VM and automatically run it with GDB. However, I found two problems:

  • External plugins are not correctly placed. I tried to fix it but I couldn’t. They are placed in blessed/results/CogMTVM.app/Contents/Resources/Debug  but they should be together in blessed/results/Debug/CogMTVM.app/Contents/Resources. The problem is that when XCode compiles, it adds a “Debug” in the directory when compiling with the “debug target” and the same for “release”. I have spent two days completely trying to solve it from the CMakeVMMaker and I couldn’t. If you know how to do it, please let me know. Anyway, the workaround is to copy all those .dylib from the first directory to the second one.
  • No matter that you change the settings in the project (like compiler flags, gcc version, etc) it will always the values from the CMake generated makefiles. This is a pity because I would like to change settings from XCode…

Now you are able to “Build -> Build and Debug – Breakpoints On”. If everything goes well, you should be able to open an image and the VM will be running with gdb. You will notice there is a little gdb button that opens a gdb console where you can do exactly the same as  if it were by command line. In addition, there is another button for the debugger. I attach a screenshot.

So…there you are, you are running the VM in debug mode from gdb and inside XCode. Interesting things from the gdb console is that you have buttons for “Pause” (what we did with gdb but from command line with the CTRL+C) and for “Continue” (what we type from gdb command line). And you have even a “Restart”.

Now….the nice thing about being able to debug the VM with XCode is the ability to easily put breakpoints in the code. What where? in which file? Without giving much details (because that’s the topic of another post), we will say that the “VM Core” (basically Interpreter classes + ObjectMemory classes) is translated into a big .c file called gcc3x-cointerpmt.c (in case of cogMT). A regular CogVM will be called “gcc3x-cointerp.c” the same as for StackVM. You can check this files by yourself. You should know where they are. Remember?  yes, they are (if you did the default configuration) in /blessed/src/vm  (for Cocoa configurations it may be in /blessed/stacksrc/vm). In such folder you will see that there is another file cointerpmt.c (or cointerp.c). Which is the difference between the one that has the “gcc3x” and the one that has not? for me moment let’s just say the one with the “gcc3x” has some optimizations (Gnuifier) that were automatically done in the C code and with the intention of compiling such sources with a gcc3x compiler.

We already know which file to debug “most of the times”. Maybe you need to debug another C file like plugins or the machine code generator, but most of the times, you will debug the “VM core” which is the file gcc3x-cointerpmt.c or its variant. Now what you have to do is to open it and put a breakppoint. On the top part of XCode you have a list of the files included in the project. Search for that one and select it. Be careful that XCode is really slow with this big file. Let’s put a breakpoint in the method lookup when the #doesNotUnderstand is thrown. This is done in the function “static sqInt lookupMethodInClass(sqInt class)” and here is a screenshot:

Once you set breakpoints you can “Build and Debug”. Now, instead of testing by executing something in a workspace, we will create a simple method anywhere (why not to use the workspace is not at my knowledge right now, sorry). So…I create the method foo in TestCase doing like this:

TestCase >> foo
self methodNonExistent.

And then I executed (this can be done in a workspace):

TestCase new foo

Once this is done, the VM should have been paused and you should have available the gdb prompt. You can now open the debugger or GDB console and do whatever you want. With the Debugger you can do “Step Over”, “Step Into”, “Step Out”, etc…And with the GDB console you can do exactly the same like if you were executing gdb from a terminal. Here is the screenshot:

There are much things I would like to talk about regarding debugging a VM, but the post is already too long and I think this is enough to start. I recommend you to watch the screencast.  Further in this blog sequence, we will do a second part. Now, it is time to understand some internal parts of the Squeak/Pharo VM.


Building the VM – Second Part

Hi folks. I guess that some readers do not like all the building part and want to go directly to see the VM internals. But it is really important that you understand how to change the VM, compile it or even debug it. Otherwise, you’ll be very limited.

This post is mostly about a couple of things that I wanted to mention in the previous post, but I couldn’t because it was already too long. If you read such post, you may think that compiling the VM from scratch is a lot of work and steps. But the post was long because of my explanations and because of my efforts in making it reproducible. This is why I would like to do a summary of how to compile the VM.

Summary of VM build

Assuming that you have already installed Git + CMake + GCC, then the following are the needed steps to compile the Cog VM:

mkdir newCog
cd newCog
git clone --depth 1 git://gitorious.org/cogvm/blessed.git
cd blessed/image
wget --no-check-certificate http://www.pharo-project.org/pharo-download/unstable-core
"Or we manually download with an Internet Browser the latest PharoCore
image from that URL and we put it in blessed/image

Then we open the image with a Cog VM (which we can get from here or here) and we evaluate:

Deprecation raiseWarning: false.
 Gofer new
 squeaksource: 'MetacelloRepository';
 package: 'ConfigurationOfCog';
 load.
(Smalltalk at: #ConfigurationOfCog) project latestVersion load.
"Notice that even loading CMakeVMaker is not necessary anymore
since it is included just as another dependency in ConfigurationOfCog"
MTCocoaIOSCogJitConfig generateWithSources.
"Replace this CMMakeVMMaker configuration class for the one that suites your OS
like CogUnixConfig and CogMsWindowsConfig"

Now, come back to the terminal and do:

cd newCog/blessed/build
cmake .
# Or  cmake . -G"MSYS Makefiles"  if you are in Windows
make

And that’s all, in “blessed/results” (in Windows it should be under “blessed/build/results”) you should have the CogVM binary. I know that you probably are a lazy guy, but if you really want to take advantage and learn in my posts, I strongly recommend you to follow those steps. All along this sequence of posts, we will debug and modify the VM (change GC, method lookup, create our own primitives and plugins, etc). Once you have Git and CMake, I promise the process takes less than 5 minutes.

Available CogVMs

Remember that all these posts is what I called “Journey through the VM”, so we will probably go and come back between different posts 🙂  In the first post,under the title “CogVM and current status” I explained the different flavors of CogVMs and the main features of them:

  1. Real and optimized block closure implementation. This is why from the image side blocks are now instances of BlockClosure instead of BlockContext.
  2. Context-to-stack mapping.
  3. JIT (just in time compiler) that translates Smalltalk compiled methods to machine code.
  4. PIC (polymorphic inline caching).
  5. Multi-threading.

What is the big difference between StackVM and CogVM? Well, Stack VM implements 1) and 2). And Cog VM is on top of the Stack VM and adds 3) and 4). Finally, there is CogMTVM which is on top of Cog VM and adds multi-threading support for external calls (like FFI for example).

In addition, Cog brings also some refactors. For example, in Interpreter VM, the Interpreter was a subclass of ObjectMemory. That was necessary in order to easily translate to C. In Cog, there are new classes like CoInterpreter and NewObjectMemory. But the good news is that we can have composition!! The CoInterpreter (which is a new class from Cog) has an instance variable that is the object memory (in this case an instance of NewObjectMemory). This was awesome and required changes in the SLANG to C translator.

As said, in the VMMaker part of the VM, what we called the “core”, there are mainly two important classes: Interpreter and ObjectMemory. Read the first post for details of their responsibilities. In Cog, there are a couple of differences:

  1. As said, the Cog Interpreter class do not subclass from ObjectMemory, but instead it is an instance variable.
  2. In Cog there isn’t only one Interpreter class like in the old VM. In fact each Cog VMs I told you (StackVM, CogVM, CogVMMT) has its own Interpreter class (StackInterpreter, CoInterpreter and CoInterpreterMT). Come on!! Don’t be lazy, take you image and browse them 🙂
  3. In Cog, there are not only those Interpreter classes that I have already told you, but also several more that are just for a design point of view, i.e, they are not Interpreter classes that should be used for compiling the VM. They are for example, to reuse code or to better simulate them. Examples, CoInterpreterPrimitives, StackInterpreterPrimitives, InterpreterPrimitives, etc. And then, of course, we have the Interpreter simulators, but that’s another story for another post.

So…if you are paying attention to this blog you may be asking yourself which Interpreter class you should use? My advice, and this is only my advice, is that you should normally use the “last one”. In this case, the CogVMMT. The few reasons I find not to use the last one are:

  1. If you are running on a hardware where Cog JIT is not supported. For example, for the iPhone the StackVM is usually used.
  2. When you are doing hacky things with the VM and you want to be sure there is no problems with JIT, PIC, etc. This is my case…
  3. Maybe for learning purposes the CogVM or CogVMMT is far much complicated than the StackVM or InterprertVM.
  4. The “last one” may not be the most stable one. So if you are in a production application you may want to deploy with a CogVM rather than a CogVM that has been released just now.

But apart from that, you will probably use the “last one” available. Just to finish with this little section, I let you a screenshot of a part of the Cog VMs hierarchy.

CMakeVMaker available configurations

In the previous post we saw what CMMakeVMMaker configurations do: 1) generate VM sources from VMMaker and 2) generate CMake files. 1) depends on which Cog (StackVM, CogVM and CogVM) we want to build, which plugins, etc. And 2) depends not only in which CogVM but also in the OS (the CMake files are not the same for each Operating System) and other things, like whether we are compiling for debug mode or not, whether we are using Carbon on Cococa library in Mac, etc. So…imagine the combination of: which CogVM, which OS, and whether debug mode or not. It gives us a lot of possibilities 🙂

The design decision to solve this in the CMakeVMake project was to create specific “configuration” classes. To summarize, there are at least one class for VM/OS. So you have, for example, CogUnixConfig (which is a CogVM, for Unix and “release”), CogDebugUnixConfig, MTCogUnixConfig, StackInterpreterUnixConfig, StackInterpreterDebugUnixConfig. And then for the rest of the OS is the same: CogMsWindowsConfig, StackInterpreterMsWindowsConfig, MTCogMsWindowsConfig, etc….So, your homework: browse the categories ‘CMakeVMMaker-Windows’, ‘CMakeVMMaker-Unix’ and  ‘CMakeVMMaker-IOS’. Look at the available classes. To learn, check implementors of #compilerFllags, #defaultInternalPlugins, #interpreterClass, etc…To test, take the debug variant, follow the same procedure as always, and you compile a debug VM with all the debugging symbols and no optimization 🙂

Which one you should use? I have already answered, but imagine you want the “last one”, then they are MTCocoaIOSCogJitConfig, MTCogUnixConfig and MTCogMsWindowsConfig.It doesn’t matter which configuration you choose, all you need to normally do is send the #generateWithSoources.

This design decision has a couple of advantages from my point of view:

  1. It is extremelly easy to customize. And in fact, there are already examples: CogUnixNoGLConfig (which doesn’t links against OpenGL so it works perfect unless you use Balloon3D or Croquet plugins), CogFreeBSDConfig (specially for BSD since it has a couple of differences in the compiler flags), etc.
  2. YOU can subclass and change what you want: default internal or external plugins, compiler flags, etc.
  3. It is easy for a continuous integration server like Hudson to build different targets.

Customizing CMakeVMMaker configurations

I told you that you can subclass from a specific class and overwrite the compiler flags, the default plugins and if they should be internal or external, etc. However, CMMakeVMaker can be parametrized in several ways while using them. In the building instructions at the beginning of this blog, I told you to move your Pharo image to blessed/image. And as I explained in the previous post that was in order to let CMakeVMaker take the defaults directories and make it work out of the box. But in fact, it is not necessary at all to move your image. You can download the “platforms code” in some place and the image elsewhere. Notice that these changes (the ability to customize each direcotry) has been commited in new versions of the CMakeVMMaker package. So, if you want to really try the followin code, make sure to have CMakeVMMaker-MarianoMartinezPeck.94. You can get it using Monticello Browser or Gofer.

So, you can do something like this:

"The image where this code is being run can be in ANY place"
MTCocoaIOSCogJitConfig new
srcDir: '/Users/mariano/Pharo/generateCode/src';
platformsDir: '/Users/mariano/Pharo/vm/git/cogVM2/blessed/platforms';
buildDir: '/Users/mariano/Pharo/vms/build';
"The resources directory is only needed for Mac"
resourcesDir: '/Users/mariano/Pharo/vm/git/cogVM2/blessed/macbuild/resources';
outputDir: '/Users/mariano/binaries/results';
generateSources;
generate.

The “platformsDir” must  map with “platforms” directory that we downloaded with Git, it cannot be choosed randomly. The same with the “resourcesDir” (which in fact is only for Mac). The rest of the directories (src, build and output) are not created by VMMaker nor Git. They are just directories that I have created by my own and I want to use them instead of the default.

And I’ve created this shortcut also:

"The image where this code is being run can be in ANY place"
MTCocoaIOSCogJitDebugConfig new
defaultDirectoriesFromGitDir: '/Users/mariano/Pharo/vm/git/cogVM1/blessed';
generateSources;
generate.

That way, I don’t need to move my image to blessed/image. BTW, don’t try this with Windows confs because there still a problem. Anyway, despite from that we can also customize things using #internalPlugins:,  #externalPlugins, etc.

Synchronization between platform code (Git) and VMMaker

In this post, I told you the problems I have seen so far with “process” of the Interpreter VM + SVN for platform code. And I also told you how this new process (CMake + Git ) helps a bit in some of those problems. From my point of view there are a couple of things that have improved the process:

  1. Platform code and VMMaker are be in sync: when people (Igor, Esteban, etc) commit a new version to Git, they make sure that the VMMaker part is working.
  2. Documentation of that synchronization: in the previous post, I told you to load version ‘1.5’ of ConfigurationOfCog. Suppose I didn’t tell you that, how do you know for a certain Git version, which version of ConfigurationOfCog you should use?  Check in blessed/codegen-scripts/LoadVMMaker.st  and you have exactly the piece of code you should execute to get the working VMMaker with that specific version of GIT. So…this means that when someone commits the Git repository and such changes require a new VMMaker version, then such developer needs to create a new version of ConfoigurationOfCog, and modify LoadVMMaker.st.  Now that you know this, the steps I told you at the beginning of this posts can be automatic, can’t they?  someone say uncle Hudson?  Yes, of course!!
  3. Git is easier in the fact that people can fork, hack, experiment, tests, and then push changes into the blessed.

Hudson for building VMs

Pharo has a continuous integration server with Hudson: http://ci.pharo-project.org/. And as you can see here, there are a lot of targets for CogVMs. Basically, for every single commit in Git, Hudson builds all those images. How? Following nearly the same steps I told you at the beginning of this post. It creates StackVMs, CogVMs and CogVMs for every OS. In fact, there are no Windows builds yet because this week they are getting the Windows slave. But the confs and the procure is working…So it is just a matter of getting the Windows box.

Conclusion: you don’t need to wait one year an a half to get a VM with a bug fix, nor you don’t need to compile it by yourself. With Hudson, they are built for every commit.

Hudson traceability

We saw how we can trace from platform code to VMMaker. Now, how to know how was every Hudson VM build ? Easy:

  1. Go to http://ci.pharo-project.org
  2. Choose a target in the “Cog” tab. For example, I choose “Mac Cog Cocoa”
  3. Follow the link, for example Cog Unix,  and there you can see two artifacts:
  • a built VM
  • a source code tarball, which is used to build that VM (in this example, CocoaIOSCogJitConfig-sources.tar.gz)

If you Download the source code archive and unpack it into your local directory what would you expect?? Of course, a copy of the git directory plus the Pharo image generated to build such VM. Such image is in build/ subdirectory and it is called generator.image and was the used to generate source code (located in src/ subdirectory) and CMake configuration files (located in build/ subdirectory). Isn’t this cool ?

CMake generators

Did I already tell you that I am also a CMake newbie? Ok…just in case 😉  Anyway, imagine CMake like a tool where we can set things, parameters, variables, directories, etc, in some files (which in our case they are auto-generated by CMakeVMMaker) and then from those “general” files we can generate specific and different makefiles. So, from the same CMake files we can generate different kind of makefiles,  i.e, we can generate makefiles the way some IDE except them to be. CMake call this ability “generators”. And the way to create makefiles with a specific generator is like this:

cmake -G "Generator Name"

Does that sound familiar?? Of course! We have already used them for MSYS in Windows. The cool thing is that there are generators for several IDEs. And this is just GREAT. For example, I can create makefiles and a project for XCode (the C IDE for MacOS). Just doing:


cmake -G Xcode

creates a XCode project for CogVM which is  in /blessed/build/CogMTVM.xcodeproj. You don’t have an idea how cool is this. This mean you can open XCode and everything is set and working out of the box for the CogVM. You can put breakpoints, inspect C code, compile, debug, everything….Before, this was much more complicated because the .xcodeproj file was versioned in the SVN and this file usually keeps some file locations or things like that and in my experience, it was always a pain to make it work.

When you use a particular generator for an IDE (like Xcode, Eclipse, KDevelop, Vsual Studio, etc, you usually don’t do the “make” by hand. So, after invoking cmake, you won’t need to do a make. Instead, you have to compile from the IDE itself (which should have the correct makefiles).

How do you know which are the available generators?  just type:

cmake --help

and at the end you’ll find a section that says “The following generators are available on this platform:”  and each of them has a name and a description. What you need to pass to the -G parameter is the name. Notice that as the help says, it automatically shows the generators available in YOUR platform (OS).  Some examples:

cmake -G KDevelop3
cmake -G "Eclipse CDT4 - Unix Makefiles"
cmake -G "Visual Studio 10"
cmake -G "Borland Makefiles"

When the name includes more than one word you must use the double quotes.

So…the 2 main advantages I see from CMake to our process is: cross compiling, and be able to automatically create makefiles for IDEs. Sorry I couldn’t try with other IDE than Xcode. If you try it and it works, let me know 🙂

In the next post we will so how to debug the VM and some related tricks. After that post, we will probably start to see the VM internals since you will have already all the needed tools.


Building the VM from scratch using Git and CMakeVMMaker

So…this is the post all hackers were waiting for! Take a beer, open your Pharo image and prepare your terminal 🙂   In this post, we will see how to build a VM from scratch. I will use Mac OS along this post, but I will also explain how to build in Linux and Windows (yes, I had to do that for this post!). I will follow the instructions together with you so that they are correct and working heehhehe. In addition, I am new with both Git and CMake…so if I say something wrong, please correct me.

The VM is a huge beast and no matter all the effort people do to ease its build, it can fail. But don’t get disappointed, if there are problems, please ask!

Remember (I told you this in the previous post) that there are different ways of building the from scratch. In this case we will use GIT (instead of the SVN repository) and CMakeVMMaker (instead of VMMakerTool) and we will build a Cog VM. These tools are part of what I called “new infrastructure” in the previous post.

Installing necessary tools on each OS

For all the OS we need a couple of things: Git client, CMake, and gcc/make. I will tell you what I think is needed in each OS. I know it is painful, but at least, you have to do it only once…Don’t be lazy. If this get’s longer, you have the excuse for the second beer 😉

Mac OS

  1. “Just” (4gb) installing the whole XCode is enough for compiling. XCode package comes with everything: gcc, make, cmake, etc. You can download it from Apple’s website. For Lion or XCode 4.2 users, read the last section of the blog (“Problems I have found so far”) .
  2. The Git client for Mac OS. Try to put Git in PATH.
  3. To check if everything is installed, open a terminal and try to execute the different commands: git, make and cmake.

Linux

  1. You need to install the development tools such as gcc, make, etc. In Ubuntu, you need to install the “buildessential” package (sudo apt-get install build-essential).
  2. Install cmake -> if it is not already included something like “sudo apt-get install cmake” should work in all Ubuntu and forks.
  3. Install Git client (in my Ubuntu this was with “sudo apt-get install git-core”). You may want to put Git in PATH. Usually, when installing with a package system it automatically adds binaries to the PATH.
  4. To check if everything is installed, open a terminal and try to execute the different commands: git, make and cmake.

Windows

  1. Download and install MinGW and MSYS, with C/C++ compiler support:  http://www.mingw.org/wiki/msys. To install mingw+msys using single, painless step, one should download latest installer from here: http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/mingw-get-inst/
    Today, the mingw-get-inst-20110530 is latest one.
  2. Download and install Git: http://code.google.com/p/msysgit/ During the installation, I choose the option “run git from from the windows command prompt”. Optional: add git to the PATH variable so that you can see git from msys. To do this, add path to git for msys:
    Control panel -> System -> System Properties / Advanced  [ Environment Variables ].   There should be already: ‘C:\Program Files\Git\cmd’. Add ‘C:\Program Files\Git\bin’  Notice that the path may not be exactly ‘C:\Program Files\Git\’ but similar…
  3. Install CMake: http://www.cmake.org/cmake/resources/software.html (during installation, in install options , make sure that you choose to add CMake to PATH).
  4. To check if everything is installed, open MSYS program (which should look like a UNIX terminal) and try to execute the different commands: git, make and cmake.

From now, when we refer to “terminal” we refer to whatever it is: iTerm and friends in Mac OS, Unix/Linux terminals, and MSYS in Windows.

Downloading platform code from GIT

If and only if you plan to modify and publish your cahnges, you will need to create an account in https://gitorious.org/ in case you don’t have one already. If you just want to download a project, then you don’t need an account. Gitorious is not more than a nice, free and public GIT hosting. Creating an account there is a little tricky because you need to SSH key. But try it and if you have problems check in the web because there is a lot of documentation and blog posts about that. Once you have your account working, continue with this…

The goal of this post is not to talk about GIT so we will do it fast 🙂 If you are reading this post, you may probably be a Smalltalk hacker. A real Smalltalk hacker doesn’t use a UI program, but instead command line jeheheh. Seriously, for this post it is easier to just tell you the git commands instead of using a Git front end. So..all git commands should be run from a command line. What are you waiting for? Open a terminal go to your prefer directory and create your workspace for today:

mkdir cogVM
cd cogVM

The Cog VM repository is https://gitorious.org/cogvm. We need to download the “platform code” from there. To do this, we have two options. The first one is to just clone the CogVM project to your local directory. This option is the most common and it is what you will do if you just want to load CogVM (and not modify it or commit changes). It is like doing a “svn co” for the SVN guys.

git clone git://gitorious.org/cogvm/blessed.git

Normally, you can also pass the argument “–depth 1” and do “git clone –depth 1 git://gitorious.org/cogvm/blessed.git”. This is just for avoiding to download all the history and it just downloads the HEAD (at least that’s what I think it does). In this post we are not going to use “–depth 1” and I will explain later why not. The second option is to clone Cog VM to your own Git repository. This has to be done from the Git website (maybe it can be done from command line but I don’t know it): login in Gitorius, search the project CogVM in the search input, select the CogVM project and you will find a “clone repository” button. Click on it and wait. Once it finishes, you will have your own fork of CogVM, and the Git repository is something like this: git://gitorious.org/~marianopeck/cogvm/marianopecks-blessed.git. Now, the last step is to clone from your own fork to your local directory (as we did in the first option). This should be something like:

git clone git://gitorious.org/~marianopeck/cogvm/marianopecks-blessed.git

For this post, I recommend to take the first option if you may be a beginner. I told you that I wanted my posts to be reproducible. With the previous commands, you will clone the latest version in the repository. Since I don’t know when you are going to do it (if there is someone), I would like that you load the specific version I know it works. What I am suggesting is doing a kind of “svn co http://xxxx -r 2202”. I checked how to do this in git, and it seems not to provide a clone of a specific version. Instead, you just clone (from the latest one) and then you checkout or revert to a previous one. Execute:

cd blessed
git checkout f3fe94c828f66cd0e7c37cfa3434e384ff65915e

Notice that you can do this because we have downloaded the full history of the repository. If we would have added the “–depth 1” parameter during the first clone, then we should be having an error like “fatal: reference is not a tree: f3fe94c828f66cd0e7c37cfa3434e384ff65915e”.

f3fe94c828f66cd0e7c37cfa3434e384ff65915e is the commit hash of the version I want. You can do “git log” to see the latest commits or “git rev-parse HEAD” to see the last one.

Ok, if you could successfully load everything from Gitorious you should have something like this:

mariano @ Aragorn : ~/Pharo/vm/git/cogVM/blessed
$ls -la
total 72
drwxr-xr-x  16 mariano  staff    544 Apr  7 00:14 .
drwxr-xr-x   3 mariano  staff    102 Apr  7 00:07 ..
drwxr-xr-x  13 mariano  staff    442 Apr  7 00:14 .git
-rw-r--r--   1 mariano  staff   6651 Apr  7 00:13 .gitignore
-rw-r--r--   1 mariano  staff     41 Apr  7 00:13 CHANGES
-rw-r--r--   1 mariano  staff   1112 Apr  7 00:13 LICENSE
-rw-r--r--   1 mariano  staff  13597 Apr  7 00:13 README
-rw-r--r--   1 mariano  staff     17 Apr  7 00:13 VERSION
drwxr-xr-x   8 mariano  staff    272 Apr  7 00:13 codegen-scripts
drwxr-xr-x  13 mariano  staff    442 Apr  7 00:13 cygwinbuild
drwxr-xr-x   3 mariano  staff    102 Apr  7 00:13 image
drwxr-xr-x  24 mariano  staff    816 Apr  7 00:13 macbuild
drwxr-xr-x   7 mariano  staff    238 Apr  7 00:14 platforms
drwxr-xr-x   4 mariano  staff    136 Apr  7 00:14 processors
drwxr-xr-x   8 mariano  staff    272 Apr  7 00:14 scripts
drwxr-xr-x   6 mariano  staff    204 Apr  7 00:14 unixbuild

I have highlighted two lines that represent two important directories: “/platforms” and “/image”. For the moment, lets explain what “/platforms” is…come on, you should guess it! Yes, there, in that folder, it is the famous “platform code”. You can enter to such directory and see the C code by yourself.

Downloading Cog and dependencies

So far we have loaded from Git the “platform code”. We are missing the other part, VMMaker. In the previous post I told you it may not be necessary to always download VMMAker and generate sources, because such sources may also be commited in the repository. Having the auto-generated source code in the repository is a trade-off, it has advantages and disadvantages. In this “new infrastrucutre” under Git, it was decided to remove it. So if you want to compile the VM you have to load VMMaker and translate it to C. You can read the explanations here and here.

So…we need to download VMMaker (Cog branch) and translate it to C. But of course, we first need a Smalltalk image. I told you in the previous post that I want all my posts to be reproducible. So, take this PharoCore 1.3 image. Notice that zip comes not only with the .image and .sources files, but also the .sources. If you don’t know what .sources file is, you should read Pharo By Example chapter 1, section 1.1 “Getting Started” 🙂  You need the .sources because it is necessary in order to generate the sources of the VMMaker. The image of such zip can be opened with both, Interpreter VM and Cog VM. However, if you open it with Cog, and you save it, then you won’t be able to run it with the Interpreter VM (this was fixed in the latest code of Interpreter VM but there isn’t yet an official VM release for all OS that contains this fix). Thus, I recommend you to run the image with a CogVM. If you don’t have one already, you can download this one.

Now, let’s load VMMaker branch for Cog and all its dependencies. You have already learned that Cog has dependencies in other packages and to solve that, among other problems, we use a Metacello configuration for it. The following code may take time because since we are evaluating this in a PharoCore image where Metacello is not present, Metacello needs to be installed first. In addition, VMMaker is a big package…Fortunatly we are running with a CogVM 🙂    So, take the image and evaluate:

Deprecation raiseWarning: false.
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfCog';
load.
((Smalltalk at: #ConfigurationOfCog) project version: '1.5') load.

Gofer new
squeaksource: 'VMMaker';
package: 'CMakeVMMaker';
version: 'CMakeVMMaker-MarianoMartinezPeck.83';
load.

IMPORTANT: How do you know you have to load version ‘1.5’ ? Yes, because I am telling you hahha. But how do I know that 1.5 is the one that works with the current version of Git?  This is answered in the next post under the title “Synchronization between platform code (Git) and VMMaker”.

One of the cool things I like from Metacello configurations is the possibility to query them. For example, do you want to all the packages that are installed while doing the previous code? Just inspect or print:


(ConfigurationOfCog project version: '1.5') packages.
ConfigurationOfCog project versions.

Now, if you are curious about how defining versions and baselines is achieved in Metacello, take a look to the methods ConfigurationOfCog >>#version15:  and ConfigurationOfCog >>#baseline13:.

Generating VM sources and CMake files

What have we done so far? We have just download all the VMMaker branch for Cog with all its required dependencies. Now, its turn to translate from SLANG to C and to generate the CMake outputs so that we can compile after. This C code generated from VMMaker is known in the Squeak/Pharo VM world as “the sources”. But don’t confuse, the platform code is also source code…but anyway, when usually someone says the “sources” it means the autogenerated C code from VMMaker. This is why this C code is placed (by default) in /src.

To do both things (generate VM sources and CMake outputs), we use one of the available CMakeVMMaker configurations. Metacello configurations are represented by classes (instead of XML like maven or similar package management systems). How do you think CMakeVMMaker configurations are represented?? Of course!! with classes too. So, we need to find the accurate class for us. I won’t go further now because that’s topic of another post, but for the moment, lets use these classes CogUnixConfig, CogMsWindowsConfig and CocoaIOSCogJitConfig depending in which OS you are.

A little remark for Mac users: you can notice that there are two categories with CMake Mac configurations, ‘CMakeVMMaker-MacOS’ and ‘CMakeVMMaker-IOS’. The first one, is for Mac OS versions that use Carbon library. ‘CMakeVMMaker-IOS’ contains CMake configurations that use Cocoa instead, which is the new library for Mac OS. Carbon is legazy, and it may be removed soon in next MacOS versions. So, the new ones, and the ones you should use, are those configurations under ‘CMakeVMMaker-IOS’.

These configurations are flexible enough to set specific directories for sources, platforms, results, etc. In addition, if you follow certain conventions (defaults), the build is more automatic. For the purpose of this post, we will follow the conventions and use the expected default directories. The only real convention we should follow is that the .image should be in a subdirectory of the directory where you downloaded the GIT code. In my case (see the bash example at the beginning of the post), it is ~/Pharo/vm/git/cogVM/blessed.  So, I moved the .image to ~/Pharo/vm/git/cogVM/blessed/image. You can create your own directory  ~/Pharo/vm/git/cogVM/generator  and place it there. The only requirement is that the ‘platforms’ directory is found in ‘../platforms’.  So…did you move your image?   Perfect, let’s continue.

No…wait. Why you need ‘platforms’ directory if we are not really compiling right now?  Ask yourself…do you think VMMaker translation to C needs the platform code?  Nooo!  So…we only need the platform directory for the second part, for  CMake. Now yes, we continue…take the Pharo image (which should have been moved) and evaluate:

"CocoaIOSCogJitConfig is an example. If you are not in Mac, replace it with CogMsWindowsConfig or CogUnixConfig"
CocoaIOSCogJitConfig  new
"Using VMMaker we translate Cog to C"
generateSources;
"We generate all the CMake necessary directories and files"
generate.

Ok…As my comments say, #generateSources uses VMMaker class to translate from SLANG to C. Instead of using a UI (VMMakerTool) we directly translate by code…but..do you remember that for compiling the VM we needed to say which plugins to compile and whether to compile them like internal or external? Ok…At that moment I told you that most of the developers shouldn’t be aware of that. In this case, CMakeVMMaker does the job for us. We will come later to this topic, but if you want to know which plugins are compiled, check implementors of #internalPlugins and #externalPlugins. Once again, CMakeVMMaker has defaults for these things, but you can customize and change them.

Where is the C generated code ? By default (yes, it can be changed) is placed in ‘../src’. In my example, it should be in ~/Pharo/vm/git/cogVM/blessed/src and should look like this:

mariano @ Aragorn : ~/Pharo/vm/git/cogVM/blessed/src
ls -la
total 16
drwxr-xr-x   6 mariano  staff   204 Apr  9 14:55 .
drwxr-xr-x  18 mariano  staff   612 Apr  9 14:55 ..
-rw-r--r--@  1 mariano  staff   776 Apr  9 14:55 examplePlugins.ext
-rw-r--r--@  1 mariano  staff    83 Apr  9 14:55 examplePlugins.int
drwxr-xr-x  43 mariano  staff  1462 Apr  9 14:55 plugins
drwxr-xr-x  11 mariano  staff   374 Apr  9 14:55 vm

In a future post, we will go deeper in how is the C translated cog…but if you want to take a look, go ahead!! Inspect the file /src/vm/cointerp.c  for example 🙂   So…do you already love SLANG? hehehe

With the method #generate we create all the directories and files needed by CMake so that we can after use CMake to generate different makefiles. You will notice that this method creates a directory /build. In my case, it is ~/Pharo/vm/git/cogVM/blessed/build. If you check inside that directory, there are a couple of important files generated for CMake (so that we can use it after), such as CMakeLists.txt, directories.cmake, etc. If you are curious, take a look to them.

If you are interested, I strongly recommend you to take a look to both methods: #generateSources and #generate. Now that I have explained the two big steps, I can tell you that there is a shortcut:

CocoaIOSCogJitConfig generateWithSources

Using CMake and compiling the VM

We are almost done…we already have all the necessary C code, and all the CMake files and directories. The next step is to just use CMake. I am newbie in both Git and CMake. But as far as I could see, CMake is a wonderful tool for being able to generate different makefiles from the same “definition”. So, we have already tell CMake which was our code, the directories, compiler flags, ec, etc. Then CMake can take such information and generate different makefiles: UNIX makefiles, MSYS (for Windows), XCode, Visual Studio, etc…In this post we will see just how to use regular Unix makefiles.

Now…come back to your terminal. We need to first go the /build directory and then execute CMake. In MacOS and Linux, evaluate:

cd build
cmake .

Now, in Windows we are compiling in MSYS, so we need to create special makefiles for it. The way to do this with CMake is using the parameter -G”Generator Name” where “Generator Name” is “MSYS Makefiles” in this case. So, in MSYS (in Windows) we evaluate:

cd build
 cmake . -G"MSYS Makefiles"

Once that is done, we have created all the necessary makefiles. Now, the last pending thing is just to “make”. It is the same whether you’re on windows or not. The makefiles have been generated, son only a make is needed:

make

Hopefully, you didn’t receive any compilation error and you can find your VM binary in /results (in Windows it is under /build/results) . Again, in my case it is ~/Pharo/vm/git/cogVM/blessed/results.

Problems I have found so far

Linux

It seems that the default CogUnixConfig needs OpenGL dev files (headers) and libs.This is because some plugins like Croquet or Balloon3D require such lib. And those plugins are being included by default in CogUnixConfig. So in my case I’ve got the error “The file was not found sqUnixOpenGL.h” which I fixed by installing the dev package:

sudo apt-get install mesa-common-dev

Then, I have a problem at linking time, “/usr/bin/ld: cannot find -lGL”, which I solved by doing:

delete /usr/lib/libGL.so
cd /usr/lib/
sudo ln -s libGL.so.1.2 libGL.so

Notice that other people have experienced the same problem but with the libSM.so (lSM) and libICE.so (lICE). Such problem could be resolved the same way. You may  want to use the command “locate” first to see where the library is and then do the “ln”.

Another solution (but I couldn’t test it by myself) to avoid using “ln”, could be to simply install the package libgl1-mesa-dev (sudo apt-get install libgl1-mesa-dev).

After having done all that, I then realised there is a special  CogUnixNoGLConfig  which seems to avoid linking to OpenGL. The VM will work until you use something like Croquet or the Ballon3D. For a more detailed explanation, read this thread.

If you have the error “alsa/asoundlib.h: No such file or directory” then you should install libasound2-dev, for example, by doing “sudo apt-get install libasound2-dev”. Be aware that depending on your Linux distro and what packages you have installed on in, you may require to install a couple of packages or non. If you have a problem with a C header file not found (a .h file) you will probably need to install the “dev” package of the project. And if what it is not found is a library (a .so for example), then it is likely you will need to install the package that contains such libs. How do you know which package contains a specific header file or lib ? I have no idea. I always go to Google and I find the answer.

Windows

As you can read in this thread I could generate the VM in Windows by myself but I had a problem at compile time: “cc1.exe: error: invalid option `no-fused-madd'”. To solve that problem I edited the method CPlatformConfig >> configureFloatMathPlugin:.  I’ve changed the line “maker addDefinitions: ‘-O0 -mno-fused-madd’.”  to “maker addDefinitions: ‘-O0’.”. The effects of this change, may be that Croquet doesn’t work properly. For further details, read another thread.

Ok, this is the end of the post. I still remember (ok, only two years ago) the first day I could compile the VM. I felt so hacky and happy at the same time. I remember I chatted with my SqueakDBX companions about that ehehhehe. So…if I could help you to successfully build your own first VM, you own me a beer (yes, I will be at ESUG hahaha). If you spent a couple of hours and you couldn’t…..mmmm… ask the mailing list 🙂    Seriously, as I told you, compiling the VM from scratch is complicated, even with all the effort made to ease that. If you had problems, please ask in the mailing lists. They will probably be able to help you, and your questions will make this process easier in a near future.

In the next post we will see some advanced topics of compiling the VM, and after that we will start to take a look to the VM inside. Once again, thanks Esteban, Igor, and everbody who answered my questions in the mailing list.

Mac OSX

As far as I understand, if you update to the latest XCode (4.2) from an older version (in Snow Leopard) it still  includes the GCC but it sets LLVM as the default compiler. In a fresh installation of Lion/XCode, not only LLVM is the default compiler but also GCC is not installed. To check which compiler area you using, execute in a console “gcc –version”. If it says something like “i686-apple-darwin11-gcc-4.2.1” then it is correct, you are using GCC. If it says something like “686-apple-darwin11-llvm-gcc-4.2” then it means you are using LLVM by default.

If GCC is not installed, you can install it via Mac Ports by doing “sudo port install apple-gcc42”.  You can follow http://stackoverflow.com/a/8593831/424245 to get it to appear in  Xcode, the last two steps will probably look like:
a) sudo ln -s /opt/local/bin/gcc-apple-4.2 /Developer/usr/bin/gcc-4.2
b) sudo ln -s /opt/local/bin/g++-apple-4.2 /Developer/usr/bin/g++-4.2

The CogVM needs GCC and cannot be compiled right now with LLVM so you have to use GCC. There are a couple of possibilities:
1) Change the default compiler for the whole system. To do this you have to edit the symbolic link of gcc to point to the real GCC and not LLVM. I do not recommend that much this option since you may be affecting the whole system.
2) When you do the cmake, reacher than simply do “cmake .” do: “cmake -D CMAKE_C_COMPILER=gcc-4.2 -D CMAKE_CXX_COMPILER=g++-4.2 . ” If that doesn’t work, try “cmake -D CMAKE_C_COMPILER=gcc-apple-4.2 -D CMAKE_CXX_COMPILER=g++-apple-4.2 .”. I would recommend this option for those who are building for the first time.
3) Instead of doing 2) by hand, you can use a patch in CMakeVMMaker that does that for you. In such a case you should use the config CogMTCocoaIOSGCC42Config. Notice, however, that such patch was added in the latests versions of CMakeVMMaker. In this post, I put specific versions of each part of the VM building system. Therefore, if you want to use the latest version of CMakeVMMaker you should also use the latest code from git and from ConfigurationOfCog.

For more details see this thread and this one in the VM mailing list: