Just A Programmer We’re Just Programmers…

30May/111

Remembering those IT practioners that have served

Posted by Justin

Today we observe Memorial Day in the United States. I happened to have spent part of my Memorial Day returning from a *minication* in Dallas, Texas. On my flight home, it was announced that a female WWII veteran who served in the WASP program was flying with us. Since flight time is often blogging time for me, I decided to write this article to express my thanks to all of those who have served the United States in uniform. Thank you to all of you!  I also wanted to use this space to thank those who at one point served in the US military who are now civilian programmers, sysadmins or DBAs.

Naturally the list must start off with Rear Admiral Grace Hopper, responsible amongst other things for COBOL and the term debugging. Honorable mentions must follow for the sci-fi writer Robert A Heinlein and his wife Virginia. While not programmers by profession, they were both veterans and inspirations to many in our field. Although the three of them have sailed beyond the sunset, I am thankful for their service.

In addition, one cannot forget those that have served and are still living. Microsoftie and University of Washington Professor Buck Woody served in the Air Force, as did Audrey Hammonds, half of datachix. Thank you both for your service!

Finally, a special thanks to those who serve in a civilian IT role while still in the active reserves like Jonathan Kehayias of SQLSkills, better know as SQLPoolboy to some. Thank you Jonathan, and those like you for your continuing service.

This list is by no means exhaustive, just the people I could think of. I invite everyone to write in the comments below a personal message of thanks to fellow programmers, sysadmins and dbas that have served in the US military.

14May/114

Using MongoDB in Powershell

Posted by Justin

Readers of this blog know that I’ve been using MongoDB for a while, and I’ve recently become very excited about Powershell. Well recently I’ve been able to combine the two together for pure dynamically typed, schema-less, non-relational awesomeness. Such awesomeness is begging to be shared.

To get started you will need version 1.1 of the official 10gen CSharp driver for Mongodb. At the time of this writing 1.1 has not been released, so your going to have to pull the latest version from github and compile it yourself. The minimal revision of the trunk you will need is this one. First download and install the MSI (or download the source, build the MSI and install it). Then open up your favorite text editor or Powershell IDE. Personally I am a fan of Powershell ISE or plain old vim, with some configuration modifications. Now we will load the bson and driver DLLs via the Add-Type cmdlet.

# Check to see if we are running the 64 bit version of Powershell.
# See http://stackoverflow.com/questions/2897569/visual-studio-deployment-project-error-when-writing-to-registry
if ([intptr]::size -eq 8) {
    $mongoDriverPath = (Get-ItemProperty "HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v3.5\AssemblyFoldersEx\MongoDB CSharpDriver 1.0").'(default)';
}
else {
    $mongoDriverPath = (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\.NETFramework\v3.5\AssemblyFoldersEx\MongoDB CSharpDriver 1.0").'(default)';
}
Add-Type -Path "$($mongoDriverPath)\MongoDB.Bson.dll";
Add-Type -Path "$($mongoDriverPath)\MongoDB.Driver.dll";

Since the Csharp Driver MSI is 32 bits, it creates the registry entries in the Wow6432Node. Therefore, we have to check to see if we are running in the 32 or 64 bit version of Powershell . Credit to an anonymous commenter on the msgoodies blog for providing this size of a pointer trick to determine if you are running a 32 or 64 bit system.

The next thing we want to do is to create a BSON document. This is surprisingly easy.

[MongoDB.Bson.BsonDocument] $doc = @{
    "_id"= [MongoDB.Bson.ObjectId]::GenerateNewId();
    "FirstName"= "Justin";
    "LastName"= "Dearing";
    "PhoneNumbers"= [MongoDB.Bson.BsonDocument] @{
        'Home'= '718-555-1212';
        'Mobile'= '646-555-1212';
    };
};

As you can see Powershell can convert a HashTable to a BsonDocument. This is because of the public constructor BsonDocument(IDictionary hashTable). Powershell can use these one parameter constructors to cast an object. You can use the same Hashtable trick for the QueryDocument and UpdateDocument classes.

Now that we have our BsonDocument, its time to perform basic crud operations.

$db = [MongoDB.Driver.MongoDatabase]::Create('mongodb://localhost/powershell');
$collection = $db['example1'];
Write-Host "Insert";
$collection.Insert($doc);
$collection.FindOneById($doc['_id']);

$updates = @{'$set' = @{'email'= 'justin@mongodb.org'}};
$query = @{"_id"= $doc['_id']}

Write-Host "Update";
$collection.Update([MongoDB.Driver.QueryDocument]$query, [MongoDb.Driver.UpdateDocument]$updates);
$collection.FindOneById($doc['_id']);

Write-Host "Delete";
$collection.Remove([MongoDB.Driver.QueryDocument]$query);
$collection.FindOneById($doc['_id']);

This gives us the output below.

_id                                                         4dc5e5a43b42bd41d0154679
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
FirstName                                                   Justin
Delete

PS E:\src\gist\gist-854911> powershell.exe -file .\MongoTest.ps1

Insert

Name                                                        Value
----                                                        -----
_id                                                         4dc636c33b42bd3d7c8dc3a4
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
FirstName                                                   Justin
Update
FirstName                                                   Justin
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
_id                                                         4dc636c33b42bd3d7c8dc3a4
email                                                       justin@mongodb.org
Delete

As you can see, its not very hard to use the 10Gen MongoDB Csharp driver from within Powershell. Using Powershell with the MongoDb C-Sharp driver has many possibilities. First of all, adhoc mongodb queries from inside of powershell. Secondly,  The code for this example is available in its entirety here.

8May/111

Setx where have you been all my life?

Posted by Justin

For me, the reference implementation of mixed emotions is the combination of anger, relief, joy, and frustration when “Why isn’t there a way to do X?!” becomes “How come no one ever told me about Y?!” This past Friday, I got to experience that entire bag of emotion thanks to setx.exe.

Setx (technetss64) is a command line utility that sets environmental variables permanently on windows. This behavior is distinct from the set command (technetss64) which only affects the current cmd.exe session. To clarify, there are three levels of environment variables:

  • Machine level All users on a given machine see these
  • User Level Each individual user on a system has a set of these variables
  • Session Level When you actually spawn a cmd.exe process, it allows you to have a set of transient variables for the session

Now, until I knew about setx, I had two ways of setting environment variables permanently. The first was to go through several layers of the windows GUI. The second, preferable to me, method was to use PowerShell as I illustrate elsewhere on this blog. However, that method requires a lot of keystrokes or some aliasing. Setx however simplifies the syntax quite nicely.

One thing to note about setx, as per ss64, it is available on windows 7 and through the resource kits. If your windows installation does not have setx.exe, try installing a resource kit.

 

6Apr/110

Scaling MongoDB: The review

Posted by Justin

Back in September, I posted the definitive review of MongoDB: The Definitive Guide by10gen employees Kristina Chodorow (blog | twitter) and Mike Dirolf (blog | twitter). Since then the book has continued to serve me well, and I even bought the electronic version. Therefore I was excited to learn that Kristina had solo authored a second O’Reilly volume on MongoDB, Scaling MongoDB.

The Short of It

The ebook weighs in at 64 pages. You can get  a dead-tree copy via print on demand, but I recommend against it. Sharding in mongodb is a moving target. For example, right around publication time, the default chunk size of shards was changed from 200MB to 64MB. Therefore, if you rushed out and bought a print on demand copy, you would end up being stuck with a paper with that wrong piece of information. Ebook owners however can download an updated copy.

The writing style in the book is very matter of fact, and quite readable. It takes a certain talent to be able to use terms like cardinality and illustrate sharding with interval notation without sounding unnecessarily academic. Kristina demonstrates said talent here.

My one complaint about the book is that it does not discuss the problems with running the shard server on windows. However, as Kristina pointed out to me on twitter, the book also does not deal with writing init.d or upstart scripts for mongos on unix, so windows is not being singled out here.

Conclusion

Despite my one complaint, the book is quite comprehensive in its 64 pages. If I ever do need to use sharding in production, I will be reading the book very closely, and I’d recommend anyone else thinking about sharding mongodb do the same.

30Mar/110

An Array of one item in Powershell

Posted by Justin

Powershell is a great scripting language. One of its greatest features is that you have the full power of the .NET framework at your disposal when you use it. However, Powershell is a dynamically language, and the .NET CLR was written for static languages. Because of this, sometimes when you call CLR assemblies in Powershell, the types do not get resolved correctly.

I ran into this recently while calling New-Object to invoke the MemoryStream constructor that takes an array of bytes as a parameter. Powershell did not automatically turn byte[] into object[] {byte[]}. Instead I got an error.

Windows PowerShell New-Object MemoryStream error

I initially solved my problem the hard way. I created a List<byte>, add the the array to the list and called List.ToArray(). Eventually, I figured out how to do it correctly thanks to this MSDN Powershell Tip of the Week. To make an array of one item you precede it with a comma (I tried placing the comma afterwards during my initial experiments).  Since the argument list is not the first parameter of New-Object, I needed to wrap it in parenthesis. So I ended up with New-Object System.IO.MemoryStream (,$bytes) which worked as illustrated below.

Windows PowerShell New-Object MemoryStream success

Tagged as: No Comments
27Mar/110

My ideas for The Atlantis Interactive Schema Engine

Posted by Justin

I first became aware of the open source Atlantis Interactive Schema Engine on Febuary 24th 2011 thanks to a retweet by Paul Randal. Its a .NET assembly which contains most of the “heavy lifting” code related to the schema operations in their various products. Matt Whitfield, the sole proprietor of Atlantis Interactive, is going to setup an account on codeplex or one of the other forges so people can post bugs, feature requests, and of course patches. Personally, I have several ideas for feature requests. However, I have no channel at the moment for these requests besides emailing Matt. While Matt has been very responsive, I wanted the conversation to be more public, I talked to Matt and he agreed it was a good idea to create put them on this blog post. Please use the comments to add your own bugs, feature requests, or patches, or comment on my feature requests.

Now, here are the feature requests.

Integration with ILSpy

ILSpy is a free and open source .NET decompiler. Development was accelerated by the fact that Redgate Reflector ceased to be free. It would be great if the library could decompile CLR assemblies to C# with this tool. It would be great to produce a diff of the decompiled source of two CLR assemblies.

Powershell related improvements

Powershell Cmdlets

I’ve written about using the schema engine with  powershell previously. Its a relatively pleasant experience, but it could be improved with powershell cmdlets. I’m pretty new to powershell, so I don’t have a clear direction on where I’d like these to go. However, I might consider taking the initiative on this one.

Shortened namespaces for the Enums

Basically, in powershell you have no using statements, and the Enums that are used as parameters are several namespaces deep. It might not make sense to change them. However, I am just throwing this out there.

More flexibility for creating SchemaReaders

I’m looking to be able to do a few things:

  • Pass an existing connection object
  • Create a pooled connection string. While that doesn’t make sense for desktop apps, it might make sense if I was using the library from a web app. Being able to sync databases from a web app would be really cool.

Get the existing connection string from  SchemaCollection

Its self explanatory.

Be able to script a schema to a Microsoft Sql Server Project

I’d like to be able to point the library at a database and have a function in SchemaCollection that will script everything to a visual studio database project. Initially it would be easiest to hard code this functionality. However, it would probably be best if it was template (t4?) based so that you could easily support other database versions.

Naturally a long term possibility would be to be able to do a schema sync between databases and sql server projects.

Support for Mono

I actually started a port of this. Its very easy to get it to compile (comment out like 3 lines), but you do lose Geospatial support. For this reason, it might make sense to have separate mono and .NET builds. In my limited time testing it I could not get my linux vm to connect to SQL server, so I don’t know if it would actually work. I want to pursue this path a bit myself, even if it just leads to me filing upstream bugs in mono.

Support for other databases

It seems like the object model is pretty well organized. That being said I realize this exercise would be far from mechanical. I don’t think cross database compares will ever be practical on a whole database level.  Below are the specific databases I’d like to see supported.

Microsoft Access

I don’t actually see much of a case for this other than “it can be done” and all windows machines have support for the JetSQL engine.

SQLite

Support for SQLite in .net is very good via System.Data.SQLite.

Postgres

Its been a while since I used NPgsql, but its pretty stable and robust. That being said, there will always be edge cases here since postgres is so customizable and extensible.

Conclusion

So thats a tall list of requests. When we have an issue tracker I will be transferring them all there. Until then, use the comments.

26Mar/110

A BSON ObjectId implementation in Javascript

Posted by Justin

Recently I spent a few days implementing the BSON ObjectId data type used by MongoDB in javascript so I could generate ObjectIds in a a web browser. I originally had a specific problem to solve. However, I ended reworking my approach in that instance so I did not have to generate ObjectIds in the browser. Despite this, the code is still a valid approach to solving my original problem. The code is available on the justaprogrammer github org, The project is called ObjectId.js. Although the git repository contains a sample html file to illustrate its usage, all you need is the javascript file, and if you want to support IE6, json2.js.

How It Works

Originally, I wanted an implementation of ObjectId that could interact with the format that the WCF DataContractJsonSerializer serialized objects of the type MongoDB.Bson.ObjectId in the official 10gen MongoDB C# driver. This format looked like this:

{
    "timestamp":1298346771,
    "machine":8775338,
    "pid":20092,
    "increment":0
}

Now, it was trivial a JSON object that looked like this. The real issue was filling those 4 values in a sensible manner. I ended up settling on the following:

  • timestamp: This is supposed to be seconds since the unix epoch. Javascript represents time as milliseconds since the UNIX epoch. Therefore all I had to do was set this to Math.floor(new Date().valueOf() / 1000)
  • machine: This is normally the first three bytes of the md5 hash of the hostname. Since I can’t access that from the browser I store a random number in html5 local storage and a cookie for fallback. This makes the machine id probably consistent for a given combination of machine, logon and browser. Naturally, I could use something like evercookie to make the machine part of the object id more sticky. However, I felt this was “good enough”
  • pid: Pid is generated every time ObjectId.js is executed, so usually once per page. This means that pid changes each page load, but remains consistent across Ajax calls on a single page reload.
  • increment: This was simple. pid++ every time I generate a new ObjectId.

So if you need to generate ObjectId’s in javascript, check out this class. If you find a bug or have an improvement, fork and send a pull request. Happy coding!.

26Mar/110

How many ways can we ‘script’ a stored proc or UDF from the command line?

Posted by Justin

Update: Added link to the scripts on github.

If you’ve worked with SQL Server for any length of time, you’ve probably written some stored procedures and user defined functions. If you’re like me, you’ve written many. Usually, when you edit a stored proc or UDF. you do it from Visual Studio or Sql Server Management Studio (SSMS). However, what if you want to script the sproc or udf from the command line? Well, there are quite a few ways to skin this cat, and I’ll go through a few.

Using sqlcmd

The simplest way to do this is the command line client that ships with SQL server 2005 and up, sqlcmd, and a system stored proc called sp_helptext. For our example lets have sp_helptext script itself.

sqlcmd -S .\sqlexpress2k8r2  -d master -h-1 -s"" -W -w 1024 -Q "EXEC sp_helptext 'sp_help'" -o sp_helptext.sql

Lets breakdown the arguments:

  • -S .\sqlexpress2k8 this is the named instance of sql server I am connecting to. If you want to connect to the local default instance, you can omit this
  • -d master This is the database we want to connect to. This can be omitted if you want to script an object from the users default catalog.
  • -h-1 -s”" -W -w 1024 These format the output of sqlcmd in a way that makes sense for this task. If you want to understand these options, look at the sqlcmd msdn page, or this stackoverflow question.
  • -Q “EXEC sp_helptext ‘sp_helptext’” -Q means “execute this T-SQL and exit.
  • -o sp_helptext.sql This writes the command output to the text file sp_helptext.sql

Using a batch file

Now we could easily wrap this in a batch file as I demonstrate here:

C:\Users\zippy\Documents\deleteme>type scriptProcOrUdf.cmd
@echo off
sqlcmd -S %1  -d %2 -h-1 -s"" -W -w 1024 -Q "EXEC sp_helptext '%3'" -o "%3.sql"
So we saved ourselves a few keystrokes with a 2 line batch file.

Using Powershell

Batch files work, but they’re so 2002. Powershell is the CLI of choice these days on windows. So lets see how we would do this in powershell:

So apparently it took me 26 lines of Powershell (if you remove the BSD License) to accomplish what I could have done in a 2 line batch file. Whats the point you might be asking? Well, honestly if you just want the quickest way to script stored procedures to a file from the command line, I’d use the batch file. However, if you wanted to make changes to this script, for example, make it script all stored procedures and udfs in the database. You’d have an easier job with Powershell. You could turn the body of the script into a Poweshell function that takes a SqlConnection object and object name as a parameter. You could then get a list of all the stored procedures and functions with the T-SQL statement SELECT [ROUTINE_NAME], [ROUTINE_TYPE], [ROUTINE_DEFINITION] FROM [INFORMATION_SCHEMA].[ROUTINES]. I’ll leave writing that script as an exercise to the reader.

Using Powershell and “Original Thinking” to do it in less than 26 LOC

In the fictional Star Trek universe, Denny Crane, James T. Kirk cheated on the Kobayashi Maru by reprogramming the no-win scenario to allow him to win. He was awarded for his original thinking. Our scenario is less dire. However, like Kirk, we can change the rules. I chose to use a the open source .NET assemble Atlantis.SchemaEngine.dll. This dll is used for synchronizing schema between two SQL server databases. We don’t want to synchronize or compare anything. However, the library synchronizes database schema’s by generating SQL scripts. Therefore, it can generate DDL for any script-able object in a database. So lets take a look at our leaner, meaner script.

So we went from 26 to 15 lines. That’s a savings of 42%. Some of those are pretty long lines since we have to fully qualify our namespaces in Powershell. However, its a net simplification compared to making the ADO.NET calls directly.

Conclusion

I showed you 4 ways to script a stored proc. I also left plenty of room for you the reader to explore on your own.  The code for ScriptProc.ps1 and Script-Object.ps1 lives in the justaprogrammer powershell github repo. If you have any improvements, feel free to leave a comment or pingback below. Also, pull requests are always welcome. Happy Scripting!

2Feb/112

RedGate Reflector 7 news roundup

Posted by Justin

Below is a series of links related to todays news that RedGate will cease to offer a free version of its .NET Reflector product.

Stackoverflow Links

There are some stackoverflow questions that might be in need of updating as a result of the fallout of this.

In addition, hopefully there will be some activity on the Reflector section of alternativeto.net.

Tagged as: , 2 Comments
2Feb/112

RedGate is no longer free? I say open source it!

Posted by Justin

As of today, RedGate’s .NET Reflector is no longer free. It is now $35. In an interview their CEO claimed they can not justify dedicating any developer resources to the product under the current free model.

I am a big fan of Redgate’s SQL Compare tool, it makes DB deploys as easy as code deploys. I’ve used .NET Reflector, although naturally its not the sort of thing I use on a regular basis. I had mixed reactions at first. I do think that $35 is a fair price for the tool. I am glad that they will continue to develop it, and I have no doubt that will happen if a revenue stream is realized from the product. That being said, perhaps it would be better to open source it.