Years ago I saw a really good presentation by Rob Pike about a little known language called Newsqueak. I can’t remember if I had anything more than a mild interest in concurrent programming, but that talk got my attention. I read all I could find about concurrency in Newsqueak. Fun ideas are fun to play with.

I wrote a short article about channels, which are Newsqueak’s construct for synchronization between threads. I tried to emulate behavior of channels in C# via locking and signaling. I was aware that there probably is no 1:1 mapping between what Newsqueak calls processes and managed threads in .NET (though I’m not sure to this day how many processes could Newsqueak spawn before bringing machine to a halt) but the exercise was fun enough to do it anyway.

Now, this was long enough ago, that C# didn’t have syntax for lambdas and was missing all the concurrency goodness that .NET 4 brings. I decided to update the code, expecting it to be a 5 minute operation. It took a bit longer.

Code to be upgraded consisted of 2 projects: a Channel<t> class and a nice example adapted from the presentation mentioned above. Implementation of channel didn’t need any changes, but the example (an interesting take on Eratosthenes prime sieve) was spawning System.Threading.Threads. Replacing occurrences of

new Thread(foo).Start();

with

Task.Factory.StartNew(foo);

resulted in this:

The Y axis on the graph is the number of contentions. In other words, the program spend 94% of its time on synchronization. A disaster.

Fixing this took me a couple of minutes but it is not obvious. The prime sieve works by starting new Task for filtering out multiplies of a certain number. The filters are chained together and every integer which comes out at the end, has to go through all of them. Newly discovered primes cause a new filter to be added; making sure multiplies of this new prime get rejected from now on.

New filters are chugging along, each in a new Task created via a factory as shown above. So why all the contention?

Tasks are scheduled on the thread pool and apparently the default hints passed to the scheduler are not enough. We can fix that by being more explicit about the nature of our task:

  Task.Factory.StartNew(Filter, arg, TaskCreationOptions.LongRunning);

our filters are kept alive as long as the program is running, therefore the TaskCreationOptions.LongRunning.

While entertaining, the prime sieve is obviously abusing Tasks and managed threads; cost of synchronization is greater than the cost of the computation performed in each thread. But finding out the differences between System.Threading.Thread and System.Threading.Tasks.Task, the new recommended method of spawning new threads from .NET 4.0, may come in handy. Especially if one decides to make some changes in older code.

I pushed the revised version on github: prime-sieve-cs.

Side note: I’ve been meaning to have a look at Go language, which I know expands on themes present in Newsqueak. I finally found some time and want to write down some thoughts on Go, Clojure and other things.

Mike Acton from Insomniac Games, creators of Ratchet & Clank, gave a talk at this years Game Developer’s Conference. This blog post bares the same title as the presentation.

Though I wish I could hear the talk, the slides (pdf) from the talk available from Insomniac’s blog seem to capture the main points well.

Instead of designing your code around the model of the world (presumably with OO), Acton suggests to come up with way to transform the data given the constraints of the platform (hardware). They also equate World Modeling to “self helf books for programming.”

Thinking about your program as a series of transforms is the standard mind-set employed people doing functional style programming. Sometimes the language doesn’t leave you much choice, because everything is immutable. In game programming you have as much mutable state as you can. But according to Mike Acton, thinking in terms of explicit data transforms can bring us a lot of value: in performance and ease of maintenance.

That’s just a quick summary, see the slides for a more complete debunking of the Three Big lies.

In case of Clojure, that means using arrays or importing mutable classes from an existing Java library - assuming the library already has classes with properties/fields that fit our needs. Otherwise, we have to write some Java code.

Dropping to another language for performance is pretty disappointing. But sometimes it’s the only sane choice.

I’ve created gen-struct library to avoid some of those situations. This simple macro, which in reality is a modified version of the gen-class macro created by Rich Hickey, allows you to create simple, mutable and immutable structures. You can customize only one aspect of those classes: their fields. Here’s an example:

(gen-struct
 :name my.test.struct
 :mutable-fields [[float x]
                  [float y]]
 :final-fields [[int timeout]
                [static int MAX_TIMEOUT :is 1000]
                [java.io.File input]])

gen-struct automatically generates a constructor which lets you initialize all the final, non-static fields. In the above example, single constructor accepting an int value and a File object would be generated (you pass the arguments in order in which they were declared in the macro).

gen-struct classes need to be compiled, just like with gen-class.

This is an ugly duckling, since it doesn’t allow for inheritance, probably not everyone will be happy with no possibility of overriding equals, and static final fields can only be of primitive type. At least it doesn’t let you mix state and behavior in one entity.

My main motivation for creating it was that some of my experiments in clj-processing needed a lot of mutation.

I’m doing mostly .NET related stuff right now, I only have time for Clojure on weekends. I’m trying to get a feel for getting better performance out of Clojure code; yesterday I’ve dug up an old code sample, but now I see I was doing a bunch of subtle things wrong. Rewrite ahead!

Back in December I’ve begun to prototype a Swing app in Java (I needed auto-completion to get used to the giantic API). Right now I’m in the process of building it from scratch in Clojure. I want to see how far I can go without dropping down to Java, and how to combine Java and Clojure in one project effectively (mostly from the design perspective).

Processing is a java library for doing visualizations and various media related stuff. A while ago I did a Clojure wrapper for it. You can get it from github: clj-processing.

Cloak

A couple of months ago I saw examples of config files of ruby’s rake. Rake is a build tool, like ant or msbuild, but unlike those, it doesn’t involve editing XML files. And thanks to ruby’s quite flexible syntax, the resulting configuration files don’t hurt your eyes.

I don’t know ruby and didn’t really feel like learning it just for one tool. Googling for something similar to rake in the python universe didn’t yield any results.

Since I’m investing much time in Clojure lately, I decided that I might as well write something useful in it. Hence Cloak, a simplistic automation tool written in Clojure. It’s heavily inspired by rake; I had rake’s docs opened most of the time when writing the app.

Here’s how to use it. You put a file named CLOAK in your project’s directory, and in it you define tasks, like this:

(task :task-name [:other-task :another]
    (when (exists? "some.file")
        (rm  "some.file"))
    (sh "command arg1 arg2 arg3" :dofail))

Task name should be a keyword. To run it from the command line, type cloak task-name, without the semicolon (assuming cloak is the name of the shell script that launches cloak). Second argument to task is an optional list of dependencies, which can be other tasks or files. For example [:compile "classes/my.lib"]. To refer to a file, use it’s file name. Apart from regular tasks, you can specify file tasks:

 (file "index.html" ["template.xml" "content/main.txt"]
       (sh "genhtml template.xml content/main.txt index.html"))

file tasks differ from regular tasks in one detail: they compare modification time of the target and all its dependencies. Task is executed only if mtime of target < mtime of source.

Cloak is not a replacement for maven or ant. It would take me ages to reach the level of functionality of those tools. I just wanted something simple, to automate the generation of my website (among other things). So far, it’s still a work in progress but it’s usable enough for me to use it with my own projects.

Utility Libs

I’ve also put my utility libs on github. Cloak requires them (but has them in in a jar, so no need to download them separately). There’s some IO and math related stuff in there.

On a personal note, that was a good year (I finished school, among other things). On the professional level, well, I didn’t work anywhere, was too busy with my thesis. But in September I started sending out my CV and found something really nice. In Denmark.

I’m starting in January, that’s why I won’t have much time at the beginning of next year for fancy retrospectives. I’ll spend 1+ year in Denmark. I always wanted to live there, at least for a while. I mean, they invented Lego, best toy ever.

But till then, I have lots of free time and so far I’m spending it mostly on reading and programming. I’m gradually replacing Python with Clojure as my weapon of choice and I’ll post something on that topic shortly.

MathTalker has pretty steep requirements for a web app: Firefox browser and special math fonts (download links on MathTalker page). Displaying math formulas is possible thanks to ASCIIMathML library, which converts ASCII to MathML. I know about the plug-in that lets you render MathML in IE, but I decided to drop support for that browser after spending a couple of hours trying to make the UI work.

App Engine

App Engine SDK comes with a little web framework and that’s what I used. You can also use Django, but I wanted to keep things simple. The built-in framework is nothing special, but it doesn’t get in your way. What requires more attention is Google’s Datastore - their distributed database. Honestly, I used only relational databases so far (oh, and flat files…), so I had to watch one or two talks from Google IO conference to get the idea.

What pissed me off the most was lack of support for nested transactions and some bugs in existing transaction code. To run some piece of code, you have to pass a function to db.run_in_transaction(), but whenever I tried to make the code more generic (as in: not to write two almost identical functions) stuff exploded. It’s pretty bad, because creating an entity and updating a counter should be a single atomic action. But they aren’t, so you end up hoping for the best.

JavaScript

Little bit of history: At first I hated it, then I really liked it. Now I’m just using it and ignoring its deficiencies. And regarding the period that I liked JS: it was the time that JS libraries were becoming really, really good, and I was just impressed (shocked, even) that it’s possible to build anything with it without cursing like mad.

But even JS 1.7, which has generators and list comprehensions, is pretty verbose, and you can only take it so far as jQuery - the best attempt for creating domain specific language in JavaScript. Great for traversing/manipulating DOM.

Another thing: I used Emacs with js2-mode. The style of formating code encouraged (somewhat) by jQuery drives Emacs insane and beaks the indentation. At the beginning it bothered me, but then I stopped nesting function definitions and js2-mode let me do my work, and my code looked more readable. I don’t know if that was how js2-mode supposed to work or if it was just a bug, but it’s done more good than harm.

Blueprint

I’ve also tried out Blueprint CSS framework. I’ve used Yahoo’s CSS reset library before (and I don’t want to work without CSS reset ever again), but never Blueprint. It’s pretty good, but somehow feels not clean enough. I guess it provides to much defaults for my taste. But I’ll probably use it again someday.