Building the VM from scratch using Git and CMakeVMMaker

So…this is the post all hackers were waiting for! Take a beer, open your Pharo image and prepare your terminal 🙂   In this post, we will see how to build a VM from scratch. I will use Mac OS along this post, but I will also explain how to build in Linux and Windows (yes, I had to do that for this post!). I will follow the instructions together with you so that they are correct and working heehhehe. In addition, I am new with both Git and CMake…so if I say something wrong, please correct me.

The VM is a huge beast and no matter all the effort people do to ease its build, it can fail. But don’t get disappointed, if there are problems, please ask!

Remember (I told you this in the previous post) that there are different ways of building the from scratch. In this case we will use GIT (instead of the SVN repository) and CMakeVMMaker (instead of VMMakerTool) and we will build a Cog VM. These tools are part of what I called “new infrastructure” in the previous post.

Installing necessary tools on each OS

For all the OS we need a couple of things: Git client, CMake, and gcc/make. I will tell you what I think is needed in each OS. I know it is painful, but at least, you have to do it only once…Don’t be lazy. If this get’s longer, you have the excuse for the second beer 😉

Mac OS

1. “Just” (4gb) installing the whole XCode is enough for compiling. XCode package comes with everything: gcc, make, cmake, etc. You can download it from Apple’s website. For Lion or XCode 4.2 users, read the last section of the blog (“Problems I have found so far”) .
2. The Git client for Mac OS. Try to put Git in PATH.
3. To check if everything is installed, open a terminal and try to execute the different commands: git, make and cmake.

Linux

1. You need to install the development tools such as gcc, make, etc. In Ubuntu, you need to install the “build–essential” package (sudo apt-get install build-essential).
2. Install cmake -> if it is not already included something like “sudo apt-get install cmake” should work in all Ubuntu and forks.
3. Install Git client (in my Ubuntu this was with “sudo apt-get install git-core”). You may want to put Git in PATH. Usually, when installing with a package system it automatically adds binaries to the PATH.
4. To check if everything is installed, open a terminal and try to execute the different commands: git, make and cmake.

Windows

1. Download and install MinGW and MSYS, with C/C++ compiler support:  http://www.mingw.org/wiki/msys. To install mingw+msys using single, painless step, one should download latest installer from here: http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/mingw-get-inst/
   Today, the mingw-get-inst-20110530 is latest one.
2. Download and install Git: http://code.google.com/p/msysgit/ During the installation, I choose the option “run git from from the windows command prompt”. Optional: add git to the PATH variable so that you can see git from msys. To do this, add path to git for msys:
   Control panel -> System -> System Properties / Advanced  [ Environment Variables ].   There should be already: ‘C:\Program Files\Git\cmd’. Add ‘C:\Program Files\Git\bin’  Notice that the path may not be exactly ‘C:\Program Files\Git\’ but similar…
3. Install CMake: http://www.cmake.org/cmake/resources/software.html (during installation, in install options , make sure that you choose to add CMake to PATH).
4. To check if everything is installed, open MSYS program (which should look like a UNIX terminal) and try to execute the different commands: git, make and cmake.

From now, when we refer to “terminal” we refer to whatever it is: iTerm and friends in Mac OS, Unix/Linux terminals, and MSYS in Windows.

Downloading platform code from GIT

If and only if you plan to modify and publish your cahnges, you will need to create an account in https://gitorious.org/ in case you don’t have one already. If you just want to download a project, then you don’t need an account. Gitorious is not more than a nice, free and public GIT hosting. Creating an account there is a little tricky because you need to SSH key. But try it and if you have problems check in the web because there is a lot of documentation and blog posts about that. Once you have your account working, continue with this…

The goal of this post is not to talk about GIT so we will do it fast 🙂 If you are reading this post, you may probably be a Smalltalk hacker. A real Smalltalk hacker doesn’t use a UI program, but instead command line jeheheh. Seriously, for this post it is easier to just tell you the git commands instead of using a Git front end. So..all git commands should be run from a command line. What are you waiting for? Open a terminal go to your prefer directory and create your workspace for today:

mkdir cogVM
cd cogVM

The Cog VM repository is https://gitorious.org/cogvm. We need to download the “platform code” from there. To do this, we have two options. The first one is to just clone the CogVM project to your local directory. This option is the most common and it is what you will do if you just want to load CogVM (and not modify it or commit changes). It is like doing a “svn co” for the SVN guys.

git clone git://gitorious.org/cogvm/blessed.git

Normally, you can also pass the argument “–depth 1” and do “git clone –depth 1 git://gitorious.org/cogvm/blessed.git”. This is just for avoiding to download all the history and it just downloads the HEAD (at least that’s what I think it does). In this post we are not going to use “–depth 1” and I will explain later why not. The second option is to clone Cog VM to your own Git repository. This has to be done from the Git website (maybe it can be done from command line but I don’t know it): login in Gitorius, search the project CogVM in the search input, select the CogVM project and you will find a “clone repository” button. Click on it and wait. Once it finishes, you will have your own fork of CogVM, and the Git repository is something like this: git://gitorious.org/~marianopeck/cogvm/marianopecks-blessed.git. Now, the last step is to clone from your own fork to your local directory (as we did in the first option). This should be something like:

git clone git://gitorious.org/~marianopeck/cogvm/marianopecks-blessed.git

For this post, I recommend to take the first option if you may be a beginner. I told you that I wanted my posts to be reproducible. With the previous commands, you will clone the latest version in the repository. Since I don’t know when you are going to do it (if there is someone), I would like that you load the specific version I know it works. What I am suggesting is doing a kind of “svn co http://xxxx -r 2202”. I checked how to do this in git, and it seems not to provide a clone of a specific version. Instead, you just clone (from the latest one) and then you checkout or revert to a previous one. Execute:

cd blessed
git checkout f3fe94c828f66cd0e7c37cfa3434e384ff65915e

Notice that you can do this because we have downloaded the full history of the repository. If we would have added the “–depth 1” parameter during the first clone, then we should be having an error like “fatal: reference is not a tree: f3fe94c828f66cd0e7c37cfa3434e384ff65915e”.

f3fe94c828f66cd0e7c37cfa3434e384ff65915e is the commit hash of the version I want. You can do “git log” to see the latest commits or “git rev-parse HEAD” to see the last one.

Ok, if you could successfully load everything from Gitorious you should have something like this:

mariano @ Aragorn : ~/Pharo/vm/git/cogVM/blessed
$ls -la
total 72
drwxr-xr-x  16 mariano  staff    544 Apr  7 00:14 .
drwxr-xr-x   3 mariano  staff    102 Apr  7 00:07 ..
drwxr-xr-x  13 mariano  staff    442 Apr  7 00:14 .git
-rw-r--r--   1 mariano  staff   6651 Apr  7 00:13 .gitignore
-rw-r--r--   1 mariano  staff     41 Apr  7 00:13 CHANGES
-rw-r--r--   1 mariano  staff   1112 Apr  7 00:13 LICENSE
-rw-r--r--   1 mariano  staff  13597 Apr  7 00:13 README
-rw-r--r--   1 mariano  staff     17 Apr  7 00:13 VERSION
drwxr-xr-x   8 mariano  staff    272 Apr  7 00:13 codegen-scripts
drwxr-xr-x  13 mariano  staff    442 Apr  7 00:13 cygwinbuild
drwxr-xr-x   3 mariano  staff    102 Apr  7 00:13 image
drwxr-xr-x  24 mariano  staff    816 Apr  7 00:13 macbuild
drwxr-xr-x   7 mariano  staff    238 Apr  7 00:14 platforms
drwxr-xr-x   4 mariano  staff    136 Apr  7 00:14 processors
drwxr-xr-x   8 mariano  staff    272 Apr  7 00:14 scripts
drwxr-xr-x   6 mariano  staff    204 Apr  7 00:14 unixbuild

I have highlighted two lines that represent two important directories: “/platforms” and “/image”. For the moment, lets explain what “/platforms” is…come on, you should guess it! Yes, there, in that folder, it is the famous “platform code”. You can enter to such directory and see the C code by yourself.

Downloading Cog and dependencies

So far we have loaded from Git the “platform code”. We are missing the other part, VMMaker. In the previous post I told you it may not be necessary to always download VMMAker and generate sources, because such sources may also be commited in the repository. Having the auto-generated source code in the repository is a trade-off, it has advantages and disadvantages. In this “new infrastrucutre” under Git, it was decided to remove it. So if you want to compile the VM you have to load VMMaker and translate it to C. You can read the explanations here and here.

So…we need to download VMMaker (Cog branch) and translate it to C. But of course, we first need a Smalltalk image. I told you in the previous post that I want all my posts to be reproducible. So, take this PharoCore 1.3 image. Notice that zip comes not only with the .image and .sources files, but also the .sources. If you don’t know what .sources file is, you should read Pharo By Example chapter 1, section 1.1 “Getting Started” 🙂  You need the .sources because it is necessary in order to generate the sources of the VMMaker. The image of such zip can be opened with both, Interpreter VM and Cog VM. However, if you open it with Cog, and you save it, then you won’t be able to run it with the Interpreter VM (this was fixed in the latest code of Interpreter VM but there isn’t yet an official VM release for all OS that contains this fix). Thus, I recommend you to run the image with a CogVM. If you don’t have one already, you can download this one.

Now, let’s load VMMaker branch for Cog and all its dependencies. You have already learned that Cog has dependencies in other packages and to solve that, among other problems, we use a Metacello configuration for it. The following code may take time because since we are evaluating this in a PharoCore image where Metacello is not present, Metacello needs to be installed first. In addition, VMMaker is a big package…Fortunatly we are running with a CogVM 🙂    So, take the image and evaluate:

Deprecation raiseWarning: false.
Gofer new
squeaksource: 'MetacelloRepository';
package: 'ConfigurationOfCog';
load.
((Smalltalk at: #ConfigurationOfCog) project version: '1.5') load.

Gofer new
squeaksource: 'VMMaker';
package: 'CMakeVMMaker';
version: 'CMakeVMMaker-MarianoMartinezPeck.83';
load.

IMPORTANT: How do you know you have to load version ‘1.5’ ? Yes, because I am telling you hahha. But how do I know that 1.5 is the one that works with the current version of Git?  This is answered in the next post under the title “Synchronization between platform code (Git) and VMMaker”.

One of the cool things I like from Metacello configurations is the possibility to query them. For example, do you want to all the packages that are installed while doing the previous code? Just inspect or print:

(ConfigurationOfCog project version: '1.5') packages.
ConfigurationOfCog project versions.

Now, if you are curious about how defining versions and baselines is achieved in Metacello, take a look to the methods ConfigurationOfCog >>#version15:  and ConfigurationOfCog >>#baseline13:.

Generating VM sources and CMake files

What have we done so far? We have just download all the VMMaker branch for Cog with all its required dependencies. Now, its turn to translate from SLANG to C and to generate the CMake outputs so that we can compile after. This C code generated from VMMaker is known in the Squeak/Pharo VM world as “the sources”. But don’t confuse, the platform code is also source code…but anyway, when usually someone says the “sources” it means the autogenerated C code from VMMaker. This is why this C code is placed (by default) in /src.

To do both things (generate VM sources and CMake outputs), we use one of the available CMakeVMMaker configurations. Metacello configurations are represented by classes (instead of XML like maven or similar package management systems). How do you think CMakeVMMaker configurations are represented?? Of course!! with classes too. So, we need to find the accurate class for us. I won’t go further now because that’s topic of another post, but for the moment, lets use these classes CogUnixConfig, CogMsWindowsConfig and CocoaIOSCogJitConfig depending in which OS you are.

A little remark for Mac users: you can notice that there are two categories with CMake Mac configurations, ‘CMakeVMMaker-MacOS’ and ‘CMakeVMMaker-IOS’. The first one, is for Mac OS versions that use Carbon library. ‘CMakeVMMaker-IOS’ contains CMake configurations that use Cocoa instead, which is the new library for Mac OS. Carbon is legazy, and it may be removed soon in next MacOS versions. So, the new ones, and the ones you should use, are those configurations under ‘CMakeVMMaker-IOS’.

These configurations are flexible enough to set specific directories for sources, platforms, results, etc. In addition, if you follow certain conventions (defaults), the build is more automatic. For the purpose of this post, we will follow the conventions and use the expected default directories. The only real convention we should follow is that the .image should be in a subdirectory of the directory where you downloaded the GIT code. In my case (see the bash example at the beginning of the post), it is ~/Pharo/vm/git/cogVM/blessed.  So, I moved the .image to ~/Pharo/vm/git/cogVM/blessed/image. You can create your own directory  ~/Pharo/vm/git/cogVM/generator  and place it there. The only requirement is that the ‘platforms’ directory is found in ‘../platforms’.  So…did you move your image?   Perfect, let’s continue.

No…wait. Why you need ‘platforms’ directory if we are not really compiling right now?  Ask yourself…do you think VMMaker translation to C needs the platform code?  Nooo!  So…we only need the platform directory for the second part, for  CMake. Now yes, we continue…take the Pharo image (which should have been moved) and evaluate:

"CocoaIOSCogJitConfig is an example. If you are not in Mac, replace it with CogMsWindowsConfig or CogUnixConfig"
CocoaIOSCogJitConfig  new
"Using VMMaker we translate Cog to C"
generateSources;
"We generate all the CMake necessary directories and files"
generate.

Ok…As my comments say, #generateSources uses VMMaker class to translate from SLANG to C. Instead of using a UI (VMMakerTool) we directly translate by code…but..do you remember that for compiling the VM we needed to say which plugins to compile and whether to compile them like internal or external? Ok…At that moment I told you that most of the developers shouldn’t be aware of that. In this case, CMakeVMMaker does the job for us. We will come later to this topic, but if you want to know which plugins are compiled, check implementors of #internalPlugins and #externalPlugins. Once again, CMakeVMMaker has defaults for these things, but you can customize and change them.

Where is the C generated code ? By default (yes, it can be changed) is placed in ‘../src’. In my example, it should be in ~/Pharo/vm/git/cogVM/blessed/src and should look like this:

mariano @ Aragorn : ~/Pharo/vm/git/cogVM/blessed/src
ls -la
total 16
drwxr-xr-x   6 mariano  staff   204 Apr  9 14:55 .
drwxr-xr-x  18 mariano  staff   612 Apr  9 14:55 ..
-rw-r--r--@  1 mariano  staff   776 Apr  9 14:55 examplePlugins.ext
-rw-r--r--@  1 mariano  staff    83 Apr  9 14:55 examplePlugins.int
drwxr-xr-x  43 mariano  staff  1462 Apr  9 14:55 plugins
drwxr-xr-x  11 mariano  staff   374 Apr  9 14:55 vm

In a future post, we will go deeper in how is the C translated cog…but if you want to take a look, go ahead!! Inspect the file /src/vm/cointerp.c  for example 🙂   So…do you already love SLANG? hehehe

With the method #generate we create all the directories and files needed by CMake so that we can after use CMake to generate different makefiles. You will notice that this method creates a directory /build. In my case, it is ~/Pharo/vm/git/cogVM/blessed/build. If you check inside that directory, there are a couple of important files generated for CMake (so that we can use it after), such as CMakeLists.txt, directories.cmake, etc. If you are curious, take a look to them.

If you are interested, I strongly recommend you to take a look to both methods: #generateSources and #generate. Now that I have explained the two big steps, I can tell you that there is a shortcut:

CocoaIOSCogJitConfig generateWithSources

Using CMake and compiling the VM

We are almost done…we already have all the necessary C code, and all the CMake files and directories. The next step is to just use CMake. I am newbie in both Git and CMake. But as far as I could see, CMake is a wonderful tool for being able to generate different makefiles from the same “definition”. So, we have already tell CMake which was our code, the directories, compiler flags, ec, etc. Then CMake can take such information and generate different makefiles: UNIX makefiles, MSYS (for Windows), XCode, Visual Studio, etc…In this post we will see just how to use regular Unix makefiles.

Now…come back to your terminal. We need to first go the /build directory and then execute CMake. In MacOS and Linux, evaluate:

cd build
cmake .

Now, in Windows we are compiling in MSYS, so we need to create special makefiles for it. The way to do this with CMake is using the parameter -G”Generator Name” where “Generator Name” is “MSYS Makefiles” in this case. So, in MSYS (in Windows) we evaluate:

cd build
 cmake . -G"MSYS Makefiles"

Once that is done, we have created all the necessary makefiles. Now, the last pending thing is just to “make”. It is the same whether you’re on windows or not. The makefiles have been generated, son only a make is needed:

make

Hopefully, you didn’t receive any compilation error and you can find your VM binary in /results (in Windows it is under /build/results) . Again, in my case it is ~/Pharo/vm/git/cogVM/blessed/results.

Problems I have found so far

Linux

It seems that the default CogUnixConfig needs OpenGL dev files (headers) and libs.This is because some plugins like Croquet or Balloon3D require such lib. And those plugins are being included by default in CogUnixConfig. So in my case I’ve got the error “The file was not found sqUnixOpenGL.h” which I fixed by installing the dev package:

sudo apt-get install mesa-common-dev

Then, I have a problem at linking time, “/usr/bin/ld: cannot find -lGL”, which I solved by doing:

delete /usr/lib/libGL.so
cd /usr/lib/
sudo ln -s libGL.so.1.2 libGL.so

Notice that other people have experienced the same problem but with the libSM.so (lSM) and libICE.so (lICE). Such problem could be resolved the same way. You may  want to use the command “locate” first to see where the library is and then do the “ln”.

Another solution (but I couldn’t test it by myself) to avoid using “ln”, could be to simply install the package libgl1-mesa-dev (sudo apt-get install libgl1-mesa-dev).

After having done all that, I then realised there is a special  CogUnixNoGLConfig  which seems to avoid linking to OpenGL. The VM will work until you use something like Croquet or the Ballon3D. For a more detailed explanation, read this thread.

If you have the error “alsa/asoundlib.h: No such file or directory” then you should install libasound2-dev, for example, by doing “sudo apt-get install libasound2-dev”. Be aware that depending on your Linux distro and what packages you have installed on in, you may require to install a couple of packages or non. If you have a problem with a C header file not found (a .h file) you will probably need to install the “dev” package of the project. And if what it is not found is a library (a .so for example), then it is likely you will need to install the package that contains such libs. How do you know which package contains a specific header file or lib ? I have no idea. I always go to Google and I find the answer.

Windows

As you can read in this thread I could generate the VM in Windows by myself but I had a problem at compile time: “cc1.exe: error: invalid option `no-fused-madd'”. To solve that problem I edited the method CPlatformConfig >> configureFloatMathPlugin:.  I’ve changed the line “maker addDefinitions: ‘-O0 -mno-fused-madd’.”  to “maker addDefinitions: ‘-O0’.”. The effects of this change, may be that Croquet doesn’t work properly. For further details, read another thread.

Ok, this is the end of the post. I still remember (ok, only two years ago) the first day I could compile the VM. I felt so hacky and happy at the same time. I remember I chatted with my SqueakDBX companions about that ehehhehe. So…if I could help you to successfully build your own first VM, you own me a beer (yes, I will be at ESUG hahaha). If you spent a couple of hours and you couldn’t…..mmmm… ask the mailing list 🙂    Seriously, as I told you, compiling the VM from scratch is complicated, even with all the effort made to ease that. If you had problems, please ask in the mailing lists. They will probably be able to help you, and your questions will make this process easier in a near future.

In the next post we will see some advanced topics of compiling the VM, and after that we will start to take a look to the VM inside. Once again, thanks Esteban, Igor, and everbody who answered my questions in the mailing list.

Mac OSX

As far as I understand, if you update to the latest XCode (4.2) from an older version (in Snow Leopard) it still  includes the GCC but it sets LLVM as the default compiler. In a fresh installation of Lion/XCode, not only LLVM is the default compiler but also GCC is not installed. To check which compiler area you using, execute in a console “gcc –version”. If it says something like “i686-apple-darwin11-gcc-4.2.1” then it is correct, you are using GCC. If it says something like “686-apple-darwin11-llvm-gcc-4.2” then it means you are using LLVM by default.

If GCC is not installed, you can install it via Mac Ports by doing “sudo port install apple-gcc42”.  You can follow http://stackoverflow.com/a/8593831/424245 to get it to appear in  Xcode, the last two steps will probably look like:
a) sudo ln -s /opt/local/bin/gcc-apple-4.2 /Developer/usr/bin/gcc-4.2
b) sudo ln -s /opt/local/bin/g++-apple-4.2 /Developer/usr/bin/g++-4.2

The CogVM needs GCC and cannot be compiled right now with LLVM so you have to use GCC. There are a couple of possibilities:
1) Change the default compiler for the whole system. To do this you have to edit the symbolic link of gcc to point to the real GCC and not LLVM. I do not recommend that much this option since you may be affecting the whole system.
2) When you do the cmake, reacher than simply do “cmake .” do: “cmake -D CMAKE_C_COMPILER=gcc-4.2 -D CMAKE_CXX_COMPILER=g++-4.2 . ” If that doesn’t work, try “cmake -D CMAKE_C_COMPILER=gcc-apple-4.2 -D CMAKE_CXX_COMPILER=g++-apple-4.2 .”. I would recommend this option for those who are building for the first time.
3) Instead of doing 2) by hand, you can use a patch in CMakeVMMaker that does that for you. In such a case you should use the config CogMTCocoaIOSGCC42Config. Notice, however, that such patch was added in the latests versions of CMakeVMMaker. In this post, I put specific versions of each part of the VM building system. Therefore, if you want to use the latest version of CMakeVMMaker you should also use the latest code from git and from ConfigurationOfCog.

For more details see this thread and this one in the VM mailing list: