Bring It On

It’s been more than a year since Powershell 1.0 is out. The way Powershell is marketed you’d guess that the only people using it are hard-core administrators who need to automate a lot of work across many machines.

Far from it, Powershell is useful to any serious developer and maybe even enthusiast user. Yes, it does administrative tasks well, but it’s also useful for automation of many other tasks too.

I’ve seen two scenarios lately where Powershell shines: builds (as in software builds) and throw-away helper projects.

I will discuss the current state of a couple of build systems in more detail in a another post, but I will say here that both NAnt and MSBuild are not the greatest possible tools for the problem. You need to face a bit more than a simple console or Windows Forms project to really see why complex build rules end up unreadable or quirky in either of just mentioned systems. I attribute this to the basic design flaw of both: NAnt and MSBuild are highly declarative XML files that are “interpreted” by an engine. In comparison, both rake (Ruby based) and Powershell are full fledged languages.

Thus expressing intent, especially complex rules, is a lot easier in Powershell than in MSBuild for example. You don’t have to fit into MSBuild limited set of tasks nor to build an extension task (this is a primary way to extend MSBuild) in C#, compile, deploy and reference the assembly with the extension in the MSBuild build file.

Example: what follows is a Powershell script that transforms XML using XSLT.

param ([string]$inputXmlFilePath = $(read-host "Please specify the input XML file"),
       [string]$xslFilePath = $(read-host "Please specify the XSLT file"),
       [string]$outputXmlFilePath = $(read-host "Please specify the output XML file"),
       [string]$xparam
)

$xslFilePath = resolve-path $xslFilePath 
$inputXmlFilePath = resolve-path $inputXmlFilePath 
$outputXmlFilePath = join-path (split-path $inputXmlFilePath) $outputXmlFilePath 

$xslt = New-Object System.Xml.Xsl.XslCompiledTransform
$xslt.Load($xslFilePath, [System.Xml.Xsl.XsltSettings]::TrustedXslt, (New-Object System.Xml.XmlUrlResolver))

if($xparam) {
  $argList = New-Object System.Xml.Xsl.XsltArgumentList
  $argList.AddParam("xparam", "", $xparam)
  $outStream = New-Object System.IO.FileStream $outputXmlFilePath, Create, Write
  $xslt.Transform($inputXmlFilePath, $argList, $outStream)
  $outStream.Close()
}
else {
  $xslt.Transform($inputXmlFilePath, $outputXmlFilePath)
}

It's a bit limited in the sense that it doesn't pass more than one extra parameter to the XSLT, but this can easily be remedied. Since Powershell can use any .NET class, it was trivial to build this.

Another scenario where I found Powershell useful is for small throw-away helper projects. Instead of cluttering your solution with these projects, you need one file in one of the projects and you’re done. The concrete example: I built a simple TCP/IP server that would listen on a port and when a client connects would serve a content of a text file “replaying” one of the real server/client sessions from before. Simple, effective, works.

Finally, if you haven’t, do yourself a favor and download PowerGUI. Personally, I don’t use the main GUI at all, just the script editor. But the editor works really well, is simple and has a killer feature – Powershell debugging. Happy scripting!

The headline of this post is a quote. I last read it in a most-excellent book that I wrote about before called “Founders at work”. In short, the book is about success of the many startups that did something great (some subsequently crumbled) in the last fifteen years or so.

For some reason, this quote really resonated with me. Despite all the information I’ve gathered before I even thought about going at it on my own, I’ve still made many small mistakes that could have been easily avoided. More often than not, I see this pattern repeat itself with other entrepreneurs too. We either get too enchanted by the technology, we misjudge the size of the market or we continue “brushing the diamond” (trying to produce the perfect piece of technology instead of selling something that works).

It’s easy to blame the circumstances. If only one of the partners did or didn’t do something, if only market wasn’t in a slump, if only the exchange rate of your primary currency wasn’t so bad at the moment etc. But more often then not, we are the enemy, we’re the ones who shot ourselves.

 

I ran into a great question on the Business of Software forum the other day. I give it to you in its entirety:

I want to make a rich web application. Ajax is out of question (too many horror stories). I want to choose 1 platform and stick with it. Requirements are:

1. Widespread install base (I guess Silverlight's already out!)
2. RAD. Need to write an app from inception fast.
3. Has a lot of third-party/free code available on the net.

BTW I'm a C++ Win32 developer, and I would definitely prefer Silverlight *if* it had a higher install base.

I'M ESPECIALLY LOOKING FOR INSIGHTS ON PEOPLE WHO HAVE FIRST HAND EXPERIENCE IMPLEMENTING ONE OR MORE OF THE AFOREMENTIONED TECHNOLOGIES. Yea, I just yelled.

This is a great question for several reasons: the poster tells you why one of the technologies (AJAX) is not on the list (even if you disagree with the reason). He tells you about the criteria for choosing the platform. He gives you his background. Finally, he preempts any “evangelist” comments – do give advice if you’ve used one or more of the technologies above.

Unfortunately, the poster is in a “cheap, fast, good: pick two” situation. None of the technologies listed will satisfy all or even most of his requirements.

I was a C++ Win32 developer, just like him. I have used Java (for applets) and WPF (a bit of Silverlight). I only know what I’ve read about Flash. Here are my 2 euro cents…

Silverlight in its deployed version 1.0 (no CLR) is, for his needs, useless, regardless of the adoption. What he would most likely want is Silverlight 2.0 (CLR included), but that’s months away and then there’s the question of adoption and tools. Personally, I don’t think adoption is what he should be worried about, but that’s another story. Tools (RAD) should be fine for him as most will be integrated in Visual Studio where he should feel right at home. Third-party code will be scarce.

Flash has a great install base (anecdotally, around 90%). Tools appear to be very RAD. I don’t think there’s much third-party/free code available on the net as Flash is not so much about code. Mainly, as far as I can tell, the problem with Flash is the host language. It’s some kind of JavaScript mutant called ActionScript. It appears to be faster than JavaScript in the latest incarnation of Flash, but it’s still a script language. Him being an ex-C++ developer, I don’t see him using a script language and being productive with it. But that really is the only problem with the Flash.

Java has a widespread install base, but you never know which version users have installed. I’ve seen point releases of Java breaking applications and that is to me one of the biggest problems with Java. RAD is questionable (depends on your definition of RAD), but on the other hand if the applet is graphics intensive, he’ll be able to trivially transfer all the GDI/GDI+ knowledge accumulated over the years. There is a lot of third-party and even more free code on the net, so much that it might be a problem to pick which library to use, not to find if there’s one to suit his needs.

All in all, it’s tough for Win32 desktop developers trying to build RIA. Java feels great as a language (if you have C++ experience), but there are deployment issues and RAD aspect. Silverlight looks even better, but it’s not ready. Flash uses awkward script language. Let’s not forget that unlike Java which still uses rather traditional GUI concepts, both Flash and Silverlight are fully vector-enabled and use substantially different GUI concepts and this takes time getting used to.

My recommendation? If possible, wait for Silverlight. If not, go with Java.

My trusty Dell laptop is now 3.5 years old and the batteries have basically died at this point. The extra battery for the drive bay (slide DVD out, slide extra battery in, enjoy several more hours of unplugged goodness) can’t be charged while the primary battery only lasts for an hour or so.

A quick search shows that the batteries are insanely expensive. Roughly, they cost 120€ each, 240€ total, which is around $350! (yeah, this is mostly due to the weak dollar, but still). For comparison, the laptop cost me around 2500€ at the time (it was top of the line and that’s why I can still comfortably use it today) so the new batteries would be 10% of that price.

I wasn’t ready to pay that much so I turned to eBay. Guess what – I found the same batteries, brand new, for a quarter of the price. Both were about 64€ with shipping from Hong Kong.  As expected, the batteries do not have Dell official sticker, but are for all intents and purposes identical to the ones I already have. You can see the ID and serial number of the primary battery on the screenshot.

The new primary battery lasted almost 4 hours yesterday, exactly what it should for light Internet browsing, news reading etc. I had LCD brightness to the max, Wi-Fi on and an external hard drive attached so I wasn’t really saving power.

I guess that the batteries sold by other “official” sources still come from the same group of vendors as this one, except that they have “official” sticker (Dell’s or somebody else’s). I don’t like paying for branding and I’m extremely happy with my purchase.

A few sites on the Internet show you how to replace the actual batteries inside the laptop packaging. This is a slightly tricky process that requires soldering and careful manipulation of the battery package (so that it can fit back into the laptop), not to mention the possibility to fry the battery and/or laptop if mishandled. With batteries as cheap as this, there’s no need to opt for do-it-yourself whatsoever.

The headline is actually a partial quote, here’s the full quote:

Duplicating software is about as close to legally printing money as a company can get; profit margins regularly exceed 80 percent.

It comes from Jeff Atwood and is (in Internet years) quite old, from April 2007.

The blog post this appears in unfortunately talks about a different topic than I want to touch here, but here’s link for completeness.

This quote demonstrates a common misconception about software development that I will demonstrate using a well-known* Slashdot presentation style:

1. Build the application
2. Duplicate it (no cost to do so)
3. Sell it
4. Distribute it (very low cost if done over Internet)
5. ?
6. Profit!

The problem with this reasoning is that software is not a vacuum cleaner – it is never “done” and off to the duplication facility we go. There are bugs to fix, weird UI interactions to solve, new functionality to add (that seems obvious once a customer points to it), new marketing venues to try, new markets to win (often by adding features, if nothing else than language support) etc.

Therefore, the investment (in money and time) never ends, especially if you are a µISV or a small company. Yes, unlike traditional manufacturing, you don’t have to incur the costs of raw materials, warehouses and traditional transport (by truck or a train). But you still have to pay for computers (hosting), bandwidth, support… We just replaced one set of problems with another.

This does not mean that building software is harder than growing tomato. At least we don’t have to look into the sky and pray that there is (or there isn’t) rain. But unlike tomato production where clearly there is a cycle of seasons, our job never stops.

*That is, well-known if you’re a geek and read Slashdot. Despite all the mindless Microsoft bashing, I find Slashdot amusing and occasionally (quite rare these days) thought provoking.

So you have a Hotmail account (who doesn’t) and you want to use it just like you use most if not all of the other accounts – through your desktop email client. It is understood that if you are one of the proponents of the Web based email clients that this post does not apply to you.

What you’d expect is that you’ll just create a new account in your email client (most clients support many accounts), set POP3/IMAP parameters, set aside a separate folder in the email Inbox and that’s it, yes?

Wrong. Hotmail is not accessible via simple POP3 supposedly due to the large number of spammers that would then more easily abuse the service. I wonder how come Google offered POP3 access to everyone for the GMail service if this was such a huge problem.

Microsoft decided to create a project called Outlook Connector. It’s an add-in for Outlook 2003/2007 (only these two clients are supported) and to justify the creation of the whole thing they threw in a couple of extra features like synchronization of contacts, tasks and notes.

While the additional features sound nice, I wonder why this functionality isn’t integrated in the Outlook in the first place (even if it's a paid service). Most people just want to access their email and nothing more and for this task, the Connector is a horrible tool.

Connector’s UI is ugly, inconsistent with the rest of the Outlook and looks like something welded on the regular Outlook’s UI. It’s a 4+MB download that replicates basic account management UI already present in the client. Finally, it only works with Outlook (duh!) as the name implies.

A simple POP3 access for any email client would be just fine (if Google can do it, Microsoft can do it too). Why this product exists is beyond me. I think the programming effort should have been spent on better spam fighting techniques on the server and not burdening end-users will pointless add-ons.

Lambda expressions available in the new C# 3.0 compiler are mostly used in the context of LINQ queries, like this:

internal class Product {
  internal int ID { get; set; }
}

var products = new List<Product>();
products.Add(new Product() { ID = 1 });
products.Add(new Product() { ID = 5 });
// continue to fill in products
var one1 = from p in products
           where p.ID == 1
           select p;

It’s not obvious, but the above query is equivalent to (also note that one1’s type is IEnumerable<Product>, not Product):

var one2 = products.Where(p => p.ID == 1);

The part p => p.ID == 1 is the lambda expression and in this case (lambdas are more general than a simple delegate substitute) can be substituted by a C# 2.0 feature called anonymous delegate:

var one3 = products.Where(delegate(Product p) { return p.ID == 1; });

One advantage of lambdas is already obvious: we didn't have to specify the type of the parameter of the delegate, plus there was no need to type return. To a certain extent, this is actually a feature of the new compiler (better type inference), but it does not work in all cases (it can’t), so sometimes (not here, but works nevertheless) you must qualify the parameter with a type, like this:

var one4 = products.Where((Product p) => p.ID == 1);

It is relatively easy to conclude from this that the syntax for lambdas with no parameters is likely:

() => /* some criteria, note that parenthesis are required */

This is probably obvious to many developers (especially those using the help files and not deducing things like this), but the reason I mention it here is I've seen many LINQ articles that use anonymous delegates as if the author did not know that lambdas can have no parameters too. For completion, here's how a syntax of an anonymous delegate with no parameters looks like:

delegate { /* statements */ }

The age of multi-core processors is here. It’s not because we’ve just started getting multi-core processors but because most consumers today will get one without even knowing it. Just like most users today will possess 64–bit processors but because their OS is 32–bit they won’t know it, nor care.

We developers have to care if the multi-core is becoming mainstream because that is the only cheap way to scale. If your code is CPU bound you should try your best to distribute the work over as many cores as the machine has. This is not easy to solve right and a lot of work will be in the libraries anyway so why not create something all of us can use?

Microsoft is working on exactly this. The December CTP of Parallel Extensions is here and it’s looking really good. It’s just a library that you add to your project and all of the sudden there are several ways in which you can trivially parallelize your code.

The emphasis is on trivially – most experienced developers could do the same (or close) kind of work if given enough time but why invent hot water? Parallelism is here to stay, having a library in the .NET Framework is the way to go.

But all this theorizing will not convince you so I’ll give you a concrete example. Let’s say you have a server with lots of data (doesn’t matter which). This data is served to clients over a slow network, thus you’d like to transmit as little as possible. You decide that you should compute a hash of data so that when clients connect to you and send you the hash, if it matches, you won’t send them anything because the data is the same.

Computing a hash is not cheap though. In fact, let’s have a look at how fast it is on a modern 64–bit Core2 Duo running at 2.8GHz with plenty of DDR2 RAM running at 800MHz. The benchmark (whose source will follow shortly) hashes the data in the memory:

Generating 512 MB of random data...
....done.
Calculating hashes sequentually
MD5 hashes 371,82 MB/sec
MD5 hashes 379,82 MB/sec
SHA1 hashes 341,56 MB/sec
SHA256 hashes 68,31 MB/sec
SHA384 hashes 104,79 MB/sec
SHA512 hashes 105,11 MB/sec

The fastest one (MD5) crunches about 380MB/s. Since the memory throughput (theoretical) is about 6400MB/s, this code is CPU bound; if we had a faster processor we’d be able to hash more data. Using a simpler hash algorithm would also help, but let’s assume you only want to use what’s already available in the .NET BCL and you don’t care if the hash is cryptographically secure.

Let’s first take a look at the code:

using System;
using System.Diagnostics;
using System.Security.Cryptography;
using System.Threading;

namespace HashSpeedTest {
 class Program {
  private const int MB = 1024 * 1024;
  private static int _sampleSize;
  private static bool _parallelExecution;
  private static readonly Stopwatch _stopWatch = new Stopwatch();
  static void Main(string[] args) {
   if (args.Length < 2 || !int.TryParse(args[0], out _sampleSize)
    || (args[1] != "par" && args[1] != "seq")) {
    Console.WriteLine("Usage: HashSpeedTest <data_size> <style>");
    Console.WriteLine("<data_size>: size of test data in megabytes (32, 64 or more)");
    Console.WriteLine("<style>: 'par' for parallel and 'seq' for sequential execution (no quotes)");
    return;
   }
   _sampleSize *= MB;
   _parallelExecution = args[1] == "par";
   byte[] data = new byte[_sampleSize];
   Console.WriteLine("Generating {0} MB of random data...", _sampleSize / MB);
   new RNGCryptoServiceProvider().GetBytes(data);
   Console.WriteLine("...done.");
   Console.WriteLine("Calculating hashes {0}", _parallelExecution ? "in parallel" : "sequentually");
   string[] algorithms = new string[] { "MD5", "MD5", "SHA1", "SHA256", "SHA384", "SHA512" };
   foreach (var algo in algorithms) {
    Measure(data, algo);
   }
  }
  private static void Measure(byte[] data, string algo) {
   _stopWatch.Reset();
   int iterations = _sampleSize / MB;
   _stopWatch.Start();
   if (_parallelExecution) {
    Parallel.For(0, iterations, i => {
     var hash = HashAlgorithm.Create(algo);
     byte[] result = hash.ComputeHash(data, i * MB, MB);
    });
   }
   else {
    var hash = HashAlgorithm.Create(algo);
    for (int i = 0; i < iterations; ++i) {
     byte[] result = hash.ComputeHash(data, i * MB, MB);
    }
   }
   _stopWatch.Stop();
   double speed = 1000.0 / (_stopWatch.ElapsedMilliseconds / (double)iterations);
   Console.WriteLine("{0} hashes {1:f2} MB/sec", algo, speed);
  }
 }
}

The main work is done in the Measure method. We use StopWatch class to measure passed time, then hash 1MB of data several times and then calculate the speed of hashing in MB/s. The Main method just prepares the list of names of hash routines, parses parameters, displays usage etc.

Now look at the if(_parallelExecution) branch – it’s almost identical to the non-parallel version. The only difference is usage of Parallel.For instead of keyword for and the fact that we create hasher object for each hash computation – since some of the calculations will be done in parallel, we don’t want to worry about the shared state of the hash algorithm.

What do we gain? Let’s see the results of the parallel run:

Generating 512 MB of random data...
....done.
Calculating hashes in parallel
MD5 hashes 660,65 MB/sec
MD5 hashes 719,10 MB/sec
SHA1 hashes 650,57 MB/sec
SHA256 hashes 132,47 MB/sec
SHA384 hashes 203,26 MB/sec
SHA512 hashes 203,42 MB/sec

Just as expected, we get about 190% of single core performance when we use both cores (excellent result)! Even better is that we almost didn’t have to do anything. But the real gain is this: put this code on a 4–core machine and it will be almost twice as fast – we get scalability for free. As you add cores, they get used. Note: you might have noticed that MD5 is calculated twice in the example: that’s because there seems to be a small performance penalty as the Parallel Extensions library is initializing, just discard the first result and look at the second.

This isn’t the silver bullet of the multi-threaded programming though as it mostly solves CPU bound problems. If you are I/O bound you’ll need a different set of techniques. But this is a great step in the right direction and it’s still just a library. Go get it and try it out yourself.

This would not have been news if it wasn’t for the fact that my wife never went to computer school nor has worked in a programming related field, ever.

You see, my wife is a girly girl. The one that thinks first about what to wear and which makeup to apply when she goes somewhere. The one that does her nails 7 times a week or more. The one that likes pink things. The one that makes the best damn cake in the area. You get my point.

Yet, she is now a certified developer. It wasn’t easy – we started with Charles Petzold’s most excellent book Code, then moved on to object-oriented design in general, then finally to the concrete programming language (VB.NET). We developed a couple of small projects and then bought her the material for self-study preparations for the exam. For one of the tests she attended the 4 week course, for the other she just read the books. I helped as much as I could, but she did all the hard work.

I’m so proud. One more developer in the family, yay! On the other hand, she’s a VB person and I am a C# person, so some rivalry is to be expected

Quite often I want to quickly try something out – make a class or two, just a couple of simple methods that exercise certain kind of functionality I am trying to produce.

Running this code in the context of a real solution which consists of dozens of projects only muds the waters. It gets hard to separate and execute just the portions of the code you’re writing and may even throw away.

For cases like this I used to use Snippet Compiler – a simple yet powerful editor and mini-IDE for C#, VB.NET and even JScript.NET. It is convenient because it does not generate project files and other residual files, starts and runs quickly and even reroutes console output to its own output panel. Just perfect for small try-out tests.

Snippet Compiler today is quite dated though and has not been actively developed. But yesterday my good friend and développeur extraordinaire Dejan “sold” me on another excellent alternative – Visual C# Express. Version 2008 just came out and this time you can download a single ISO image of a DVD with all of the Express versions in one convenient package.

It’s great. Works almost exactly the same as its more feature rich cousin editions (Standard and Professional) but starts quickly and just like Snippet Compiler does not leave trail on a disk (see screenshot to the right). You get intellisense, all of the designers, build system… everything is there, except for one thing: multi-targeting (you can only use .NET 3.5). Still, for small test projects this is not an issue at all.

Visual C# 2008 Express, just like all the other Express editions is completely free, no registration is required and it doesn’t expire.

Thanks, Microsoft.