Migrating projects to SqueakSource3

Hi guys. I am one of the guys who is always criticizing www.squeaksource.com because it is down all the time. And when it is not down it is loosing versions. So it is not fair that I continue doing so if there is a new already working solution out there: SqueakSource3 (http://ss3.gemstone.com/ss).

During Smalltalks 2011 I talked to Dale Henrichs who told me that SqueakSource3 was quite stable and has daily backups. SqueakSource3 is a new implementation of SqueakSource but based in Seaside3, Magritte2, etc. Even more (and this is very cool), it is deployed in a GemStone server. Thanks a lot for GemStone for hosting such a service and to all the developers behind SS3.

So, I decided to give it a try. I have started to migrate all my projects or the projects I contribute and I thought I could share that with you.

Migrating projects to SqueakSource3

There are two ways I found in order to migrate packages from one repository to another one. As an example, I will show you how I have migrated Fuel code from http://www.squeaksource.com/Fuel to http://ss3.gemstone.com/ss/Fuel

Pre-optional step

As explained later, you will need to fetch all versions of the packages you want to migrate. Downloading that from squeaksource can be quite slow and you may need a good internet connection. However, if you want to migrate a project you have been development, it is probable that some .mcz are still somewhere in your machine. What you can do is to copy all .mcz files of your machine to the package-cache so that you can get it from there and avoid the network traffic. To do that I often use this bash script:


find /Users/mariano/ -name "*.mcz" -print0 | xargs -0 -I {} cp {} /Users/mariano/Pharo/YOUR-PACKAGE-CACHE/

Using Gofer #fetch and #push

As you know, Gofer is a nice scripting library for Monticello and it provides the methods #fetch (which downloads versions from remote repositories into the local cache) and #push (which uploads local versions from local cache into remote repositories). So the idea is that you can fetch the versions from the original repository and push in the new one. In the example of Fuel, I should do:

Gofer it
squeaksource: 'Fuel';
package: 'Fuel';
package: 'FuelTests';

Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'Fuel';
package: 'FuelTests';

Notice that such code could take a lot of time since EACH version of EACH package has to be fetched and pushed. So…let your machine, take a coffee and then come back. In this example, I only migrate two packages (‘Fuel’ and ‘FuelTests’) but there are much more packages in the Fuel repository. Having to write 20 lines of #package: can be a lot. So if you are a lazy guy like me, you can do the following hack:

| packagesNames gofer |

packagesNames := (Gofer new
squeaksource: 'Fuel';
collect: [ :each | each packageName ] as: Set.

gofer := Gofer it
squeaksource: 'Fuel'.

packagesNames do: [:each | gofer package: each].
gofer fetch.

gofer := Gofer it
url: 'http://ss3.gemstone.com/ss/Fuel'.

packagesNames do: [:each | gofer package: each].
gofer push.

That way you can fetch and push all packages. Of course, you can always get the list first, remove by hand the packages that you don’t want to migrate, and finally just migrate the ones you want.

Using Gofer and Metacello configurations

Instead of manually choosing which packages to migrate, if you do have Metacello configurations, why not to directly use them? Dale has an excellent post explaining exactly how to do that. Please read it if you are interested in such approach. Basically it consist of migrating only those packages of a Metacello version. Example:

| gofer |
"Identify the source repository"
gofer := Gofer new squeaksource: 'Fuel';
package: 'ConfigurationOfFuel' yourself.
"Traverse all packages defined in baseline '1.7-baseline' of the Fuel configuration"
(ConfigurationOfFuel  project version:  '1.7-baseline') packages do:[:spec |
"Use the #name and #package: messages and Gofer will fetch
all versions of the Monticello package"
gofer package: spec name ].
"Initiate the download with the #fetch command"
gofer fetch.

"Identify the target repository"
gofer := Gofer new url: 'http://ss3.gemstone.com/ss/Fuel';
package: 'ConfigurationOfFuel' yourself.
"Use the same algorithm for identifying the packages to be pushed"
(ConfigurationOfFuel project version: '1.7-baseline') packages do:[:spec |
gofer package: spec name ].
"Initiate the upload with the #push command"
gofer push.

The only difference with the previous approach is that this way we automatically migrate the packages referenced from the configuration.

Problems I found

Fetching and pushing take a long time, so be patient and try to be with a good internet connection. In my case, I have several errors which I have to deal with:

  •  Timeouts: If you get a “ConnectionTimedOut: Data receive timed out.” then it is clear there was a timeout with a certain package. What I did when I had this error is to open the debugger, go down in the stack a little while up to the place where I see it was doing the load of the version, and do a restart. Otherwise, you can always close the debugger and execute the code again, but this will take more time.
  • If you get the “Error: file is too short” then it means that there is some package (.mcz file) in your package-cache with zero bytes. Open the debugger, go down in the stack and identify which package/version has the problem. Then just go the package cache and remove such file. Then restart the migration procedure.

So, that’s all. I hope I could make your migration easier 🙂

3 thoughts on “Migrating projects to SqueakSource3

  1. Just to point out an alternative to get all your packages if you want to migrate: the latin american squeaksource mirror has all packages of public projects available for download as well. You can easily get them via http, e.g. by using the wget command-line tool for unix/macos.

    The URL for the mirror is http://dsal.cl/squeaksource Instructions on how it can be used for ‘normal’ situations are on that page as well.


    1. Hi Johan. Thanks for the pointer. Indeed, I may have used such mirror instead of squeaksource to do the migration. Now, regarding WHY I didn’t choose to migrate to this squeaksource mirror instead of SS3? I think this mirror is a very nice idea and everything that helps to decrease the overhead in squeaksource is good. So, thanks for such a mirror. That being said, I choose SS3 because I want to make progress in the technology used. Seaside 2.8 is almost dead for example. So if there is a new implementation that uses new frameworks (seaside 3.0, magritte2, etc), then I will go for it. And in this case, there is even more more thing and it is the fact of being hosted in a GemStone server. I think such approach can scale much more than a plain pharo/squeak image/vm.


      1. Of course, the mirror was never intended as a replacement of Squeaksource, nor it it useful for everything, eg. it is read-only. As you say, it’s just a way to decrease the load on squeaksource, plus to have faster load times here down under.

        I am all for better/newer/faster squeaksource replacements. I would be even happy to mirror them at some point if it helps …


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s