Just A Programmer We're just programmers

27Oct/119

Continuous Integration with TFS2010, MSDeploy & VSDBCMD

After I created an account at Cytanium, I quickly discovered the glory of One-Click Publish. After setting up my source control in TFS. All I could think of, was getting TFS to do the same thing for me, but even I knew that wouldn’t be enough. I wanted TFS to deploy my database changes as well. Well, there is that Schema Compare utility in Visual Studio, I know it can figure out and deploy changes to a database. I figured if Visual Studio can do all of these things, I should be able to as well… shouldn’t I?

Custom Build Process Templates

Very soon after you start to ask these kinds of questions, you find yourself looking at TFS Build Process Templates. The templates are designed in Windows Workflow and thanks to a few good blogs out there this is a rather approachable topic. Most of my starting information came from this blog sequence: Ewald Hofman’s – Customize Team Build 2010.

Before you start, save yourself the pain, take Ewald’s advice and add the Team Foundation Build Activities to your Toolbox. This will shave 15 seconds off the time to open every template.

Exposing Arguments and Providing Metadata

More lessons from Ewald on adding arguments and adding more complex arguments. We will use them and see how the values are supplied a bit later on.

  • Continuous Integration Database Deployment
    • ciVSDBCMDConnectionString: “Database Connection String”
    • ciVSDBCMDManifestFile: “Database Manifest File”
    • ciVSDBCMDTargetDatabase: “Target Database”
  • Continuous Integration Website Deployment
    • ciMSDeployWebProject: “Web Project”
    • ciMSDeployUsername: “MSDeploy Credentials: Username”
    • ciMSDeployPassword: “MSDeploy Credentials: Password”
    • ciMSDeployWebService: “MSDeploy Web Service Url”
    • ciMSDeployWebApplication: “Web Application Name”

Customizing the Template

Under your Team Project, go to the folder BuildProcessTemplates, and branch DefaultTemplate.xaml to ContinuousIntegrationTemplate.xaml.

Open it up and and take a look around. It will help if you take a top down approach, Collapse All, using the controls in the top right. Locate the AgentScope Activity named “Run On Agent” and expand it. Now look for the open space after the “Try Compile, Test, and Associate Work Item” Sequence Activity; this is where we will start adding our custom functionality.

In this space add a If Activity and name it: “If Build and Test Succeeded”. In its condition we are going to add the following VB statement:

BuildDetail.CompilationStatus = BuildPhaseStatus.Succeeded And
    (BuildDetail.TestStatus = BuildPhaseStatus.Succeeded Or
        BuildDetail.TestStatus = BuildPhaseStatus.Unknown)

In the Then clause we drop a sequence named: “Deploy Database and Website”. Inside that sequence, go ahead and create a “Deploy Database” Sequence and a “Deploy Website” Sequence. In the else clause we can create a WriteBuildWarning with the message: “Deployment Skipped”. It should all look like this in the end…

Deploy Database

A large part of my introduction to VSDBCMD (not to mention a large part of this idea) came from this Visual Studio Walkthrough on the topic. If you are willing to work through the various issues with using Database Deployment, it is worth it.

I happen to have Visual Studio installed on my TFS Server, if you dont, copy this application from your Visual Studio install to all your TFS Build Agents.

C:Program Files (x86)Microsoft Visual Studio 10.0VSTSDBDeploy

We are going to use an InvokeProcess Activity to call VSDMCMD.exe with the following code for arguments property. The manifest file will contain the Database name and Connection String that was configured in the project, but chances are you are going to want to override this per environment.

String.Format("/a:Deploy /dd+ /dsp:Sql" _
     & " /manifest:""{0}{1}""" _
     & " /p:TargetDatabase=""{2}"" /cs:""{3}""", _
     BuildDetail.DropLocation, ciVSDBCMDManifestFile, _
     ciVSDBCMDTargetDatabase, ciVSDBCMDConnectionString)

Be sure to properly name the InvokeProcess Activity, as well as provide WriteBuildMessage and WriteBuildError Actvities for the standard and error output.

Deploy Website

Deploying databases was actually a lot easier than I expected, which is fortunate, because deploying websites was a lot more challenging.

Lets add a few variables to the Deploy Website Sequence:

  • Name: websitePublishDirectoryPath
  • Variable type: String
  • Default:
    String.Format("{0}Website Deployment", BinariesDirectory)
  • Name: msDeployManifestFilePath
  • Variable type: String
  • Default:
    String.Format("{0}{1}.manifest", websitePublishDirectoryPath, ciMSDeployWebProject)

As we will continue by defining a Sequence named “Prepare Deployment”. This sequence is responsible for preparing a folder to publish. So we have to create the directory copy the output to it and perform any Web.config transformations. First lets declare a variable in this scope:

  • Name: platformConfiguration
  • Variable type: Microsoft.TeamFoundation.Build.Workflow.Activities.PlatformConfiguration
  • Default:
    BuildSettings.PlatformConfigurations.First
  • Name: websiteSourceDirectory
  • Variable type: String
  • Default:
    String.Format("{0}_PublishedWebsites{1}", BinariesDirectory, ciMSDeployWebProject)
  • Name: websiteDestinationDirectory
  • Variable type: Microsoft.TeamFoundation.Build.Workflow.Activities.PlatformConfiguration
  • Default:
    String.Format("{0}{1}", websitePublishDirectoryPath, ciMSDeployWebProject)

One CreateDirectory Activity to create the folder named in variable `websitePublishDirectory`, one CopyDirectory Activity to copy `websiteSourceDirectory` to `websiteDestinationDirectory` and an additonal Sequence named “Find and Transform Web.config Files”

Click for the full workflow

Yet Another Web.config Transform

There seems to be a lot of places to find good information on the Web.config
transformation topic. So I’m not reinventing anything here. Generally I looked for web.config files and any matching transformation files. An MSBuild Activity is used to perform the transformation. Finally delete all transform files, and continue. It’s important, but rather boring… and this blog post is long enough already.

Web Deploy 2.0

Hopefully you can have someone else configure Web Deploy 2.0 for you, but since you are reading this, I doubt that is the case. On learn.iis.net, I found the best set of tutorials to install, configure, (and most importantly) debug. If you are having problems with the setup, don’t hesitate to enable tracing of failed requests. Once you are able to deploy with Visual Studio you are ready to continue… right? I mean, the MSDeploy command doesn’t seem too intimidating. After a day and change, I was ready to scream uncle. Until I found this StackOverflow Question, I was unable to figure out the arguments of the command that Visual Studio used to initiate the deployment.

First we have to generate a msdeploy manifest file. An InvokeMethod Activity to call `System.IO.File.WriteAllText` with the following expression

String.Format("<?xml version=""1.0"" encoding=""utf-8""?>" _
    & "<sitemanifest>" _
        & "<contentPath path=""{0}{1}"" />" _
        & "<setAcl path=""{0}{1}"" setAclResourceType=""Directory"" />" _
        & "<setAcl path=""{0}{1}"" setAclUser=""anonymousAuthenticationUser"" " _
            & "setAclResourceType=""Directory"" />" _
    &"</sitemanifest>", _
    websitePublishDirectoryPath, ciMSDeployWebProject)

Finally we can execute MSDeploy with an InvokeProcess Activity and the following arguments.

String.Format("-source:manifest='{0}'" _
              & " -dest:auto,ComputerName='{1}?site={2}',UserName='{3}',Password='{4}',IncludeAcls='False',AuthType='Basic'" _
              & " -verb:sync -disableLink:AppPoolExtension -disableLink:ContentExtension -disableLink:CertificateExtension" _
              & " -setParam:kind='ProviderPath',scope='contentPath',match='^{5}\{6}$',value='{2}'" _
              & " -setParam:kind='ProviderPath',scope='setAcl',match='^{5}\{6}$',value='{2}' -allowUntrusted", _
              msDeployManifestFilePath, ciMSDeployWebService, ciMSDeployWebApplicationName, _
              ciMSDeployUsername, ciMSDeployPassword, _
              websitePublishDirectoryPath.Replace("", "\").Replace(" ", " ").Replace(".", "."), _
              ciMSDeployWebProject.Replace(" ", " ").Replace(".", "."))
Configure and Queue the Build

We can finally see the light at the end of the tunnel here. Since we provided the metadata for our arguments, we get a rather impressive display of customization.

Queue up your build and hopefully you wont have to do too much debugging to get this to work for yourself.

Conclusion

After seeing this work, it was all worth it. Of course, this is all very custom, I have my xaml attached here as an example, but you will have to be prepared to work with it to make it work for you. Drop me a line, I’ll help you out if I can.

ContinuousIntegrationTemplate.zip

15Oct/110

Synchronously Await an Async Workflow Using Observable.guard

The other day, I couldn’t figure out how to Synchronously wait for an Async workflow to complete. I was pretty sure I needed to use Observable.guard, I just couldn’t wrap my head around how to use it. This morning I was on a mission to figure it out.

open System
open FSharp.Control

let event = new Event<_>()

let createObservableByExecutingAsync asyncToExecute =
    event.Publish
    |> Observable.guard(fun _ -> 

        let asyncOperation = async {
            asyncToExecute
            |> Async.RunSynchronously

            event.Trigger(())
        }
        
        printfn "Start Executing Async Operation"

        Async.Start asyncOperation

        printfn "End Executing Async Operation"
    )

let someAsyncOperation = async {
        printfn "Async Operation"
    }

let asyncWorker = createObservableByExecutingAsync someAsyncOperation

printfn "Start Waiting for Async Result"

Async.AwaitObservable asyncWorker
|> Async.RunSynchronously

printfn "End Waiting for Async Result"

Console.ReadLine()
|> ignore

This bit of code produces the most beautiful output of…

Start Waiting for Async Result 
Start Executing Async Operation
End Executing Async Operation
Async Operation
End Waiting for Async Result

Figuring that out makes me feel better, but the bigger realization was that I didn’t need Observable.guard at all. I needed Async.AwaitObservable.

open System
open FSharp.Control
open RateLimiting
 
type Messages =
    | Message of string
 
type Responses =
    | Response of string
 
let operationFunction (message: Messages) =
    match message with
    | Message(input) -> Response(sprintf "Response: %s" input)
 
let agent =
    RateLimitedAgent(operationFunction, 4, 1000)
 
let event = new Event<_>()

let someAsyncOperation (response:Responses) = async {
        printfn "Async Response"
        event.Trigger(())
    }

agent.QueueItem({payload = Message("Awaited Payload") ; response = Some(someAsyncOperation)})
|> Async.Start

printfn "Start Waiting for Async Result"

Async.AwaitObservable event.Publish
|> Async.RunSynchronously

printfn "End Waiting for Async Result"

Console.ReadLine()
|> ignore
Start Waiting for Async Result
Async Response
End Waiting for Async Result

I love watching this stuff work.

Async Wait Operation

14Oct/112

Rate Limiting a Function with Async Workflows

So I am trying to deal with an API that says I cannot make more than 4 requests per second. This is a really good use for F# since it has such great support for Asynchronous Workflows. I took a little inspiration from this post by Luca Bolognese.

module RateLimiting

open System
open FSharp.Control

type RateLimitedMessage<'a,'b> = {payload: 'a ; response: ('b -> Async<unit>) option}

type RateLimitedAgent<'a,'b>(
                                operation: 'a -> 'b,
                                workerCount, 
                                ?minExecutionTime: int,
                                ?errorHandler: Exception -> 'a -> Async<unit>
                            ) = class
    let blockingQueue = new BlockingQueueAgent<RateLimitedMessage<'a, 'b>>(workerCount)
    let semaphore = new System.Threading.Semaphore(workerCount, workerCount)

    let errorHandler = defaultArg errorHandler (fun _ _ -> async{()})
    let minExecutionTime = defaultArg minExecutionTime -1

    let createWorker () =
        async {
            while true do
                semaphore.WaitOne()
                |> ignore

                let! message = blockingQueue.AsyncGet()

                let messageProcessor =
                    async {
                            try
                                let response = operation message.payload
                                if message.response.IsSome then
                                    do! message.response.Value response
                            with 
                            | ex ->
                                errorHandler ex message.payload
                                |> Async.Start
                        }

                   
                seq {
                        yield messageProcessor
                        yield async {
                                do! Async.Sleep(minExecutionTime)
                                semaphore.Release()
                                    |> ignore
                            }
                    }
                |> Async.Parallel
                |> Async.Ignore
                |> Async.Start
            }

    do
        (fun _ -> createWorker ())
        |> Seq.init workerCount
        |> Async.Parallel
        |> Async.Ignore
        |> Async.Start

    member x.QueueItem item =
        blockingQueue.AsyncAdd item
end

RateLimitedAgent

Messages are passed in through a BlockingQueueAgent from fsharpx’s Agent Library. This allows users to choose whether or not they wait until messages are picked up for processing.

Here is an example of where we don’t wait at all. Although, we throw in some random exceptions for some fun.

open System
open System.Diagnostics
open System.Threading
open FSharp.Control
open RateLimiting

let counterCreationData name ctype = new CounterCreationData(name, "", ctype)

let counterCategoryName = "Test Rate Limiter"
let requestsProcessedPerSecondCounterItem = "Requests Processed per second"
let requestsQueuedPerSecondCounterItem = "Requests Queued per second"
let requestsQueuedCounterItem = "Number of Items Queued"

let performanceCounters = [|
        counterCreationData requestsProcessedPerSecondCounterItem PerformanceCounterType.RateOfCountsPerSecond32 
        counterCreationData requestsQueuedPerSecondCounterItem PerformanceCounterType.RateOfCountsPerSecond32 ;
        counterCreationData requestsQueuedCounterItem PerformanceCounterType.NumberOfItems32
    |]

let removeCounters () = PerformanceCounterCategory.Delete(counterCategoryName)

let installCounters () =
    if PerformanceCounterCategory.Exists(counterCategoryName)
    then
        removeCounters ()

    PerformanceCounterCategory.Create(counterCategoryName, "", PerformanceCounterCategoryType.SingleInstance, new CounterCreationDataCollection(performanceCounters)) 
    |> ignore 

installCounters()
    
let createCounter name =
    let curProc = Process.GetCurrentProcess()
    new PerformanceCounter(counterCategoryName, name, false)

let requestProcessedSpeedCounter = createCounter requestsProcessedPerSecondCounterItem
let requestQueuedSpeedCounter = createCounter requestsQueuedPerSecondCounterItem

let queueCounter = createCounter requestsQueuedCounterItem
queueCounter.RawValue <- 0L

let rndSeed = new Random()

type Messages =
    | Message of string

type Responses =
    | Response of string

let operationFunction (message: Messages) =
    match rndSeed.Next(10) with
    | x when x = 1 -> failwith "Error"
    | _ ->
        match message with
        | Message(input) -> Response(sprintf "Response: %s" input)

let errorHandler (ex:Exception) message = async {
                                    requestProcessedSpeedCounter.Increment()
                                    |> ignore

                                    queueCounter.Decrement()
                                    |> ignore

                                    printfn "Error"
                                }

let agent =
    RateLimitedAgent(operationFunction, 4, 1000, errorHandler)

let responseFunction (someArg:Responses) = async{
    match someArg with
    | Response(response) ->
        requestProcessedSpeedCounter.Increment()
        |> ignore

        queueCounter.Decrement()
        |> ignore

        printfn "Responded: %s" response
    }

let createDisposable name =
    let x = ref 1
    let rnd = new Random(rndSeed.Next())

    async {
        while true do               
            agent.QueueItem({payload = Message(sprintf "%s Request %i" name (!x)) ; response = Some(responseFunction)})
            |> Async.Start
            
            requestQueuedSpeedCounter.Increment()
            |> ignore

            queueCounter.Increment()
            |> ignore

            x := !x + 1

            do! Async.Sleep(550)
        }
    |> Async.StartDisposable

let disposable1 = createDisposable "Automatic 1"
let disposable2 = createDisposable "Automatic 2"
let disposable3 = createDisposable "Automatic 3"
let disposable4 = createDisposable "Automatic 4"

Console.ReadLine()
|> ignore

disposable1.Dispose()
disposable2.Dispose()
disposable3.Dispose()
disposable4.Dispose()

Console.ReadLine()
|> ignore

So this is just about what I was looking for. I say just about because apparently I can get spikes where I go over limit. I have a strong belief this is a calculation error within Performance Monitor, but if we were really concerned about it, we could increase the worker cooldown time to 1.1s.

Because it’s fun, I’ve toyed some with the operation function to include a random delay, just to see the results.

let operationFunction (message: Messages) =
    Thread.Sleep(rndSeed.Next(50,2500))
    match rndSeed.Next(10) with
    | x when x = 1 -> failwith "Error"
    | _ ->
        match message with
        | Message(input) -> Response(sprintf "Response: %s" input)

Not too shabby..

Messages are bundled with an optional Async response. This allows the caller to provide an Async function to handle the response. If a user were interested, I’m pretty sure this is a more likely and useful target to block your Async Workflow for.

I’m sure I have to use Observable.guard like Thomas Petricek reffered to here and here.

So, create an Async Workflow, make it throw an event, add the message to the agent, then wait until the event is observed. Sounds simple enough, but I just can’t wrap my head around the mechanics of it at this point. I think I’m going to ask Thomas to help me out.

Edited 10/17/2011
As per comments below

9Oct/111

HashTables and New-Object -Property in Powershell

There are many uses for the New-Object cmdlet. One of the more obvious uses is to transform a HashTable to a PSObject, perhaps to be pipelined into Format-Table.

New-Object PSObject -Property @{
    Name = "Justin Dearing";
    Occupation = ".NET Developer"
}

That is a convenient output formatting trick because you can then pass it to an Out-* or Format-* cmdlet. However, this trick has uses beyond formatting. For example, lets say you encapsulated web service calls into a PowerShell module via the New-WebServiceProxy cmdlet. If the parameters of the service methods of the object in question aren’t .NET primitives like strings and ints, you will need to figure out how your wrapper functions will take those parameters. Once again New-Object -Property and a HashTable. Consider this contrived example:

$service = New-WebServiceProxy http://somedomain/webservices/something.svc?wsdl  -Namespace SomeService

function Get-Something ([HashTable]$SomethingRequest) {
    $request = New-Object SomeService.GetSomethingRequest -Property $SomethingRequest;
    return $service.GetSomethingRequest($SomethingRequest);
}

Note that you might want to use Parameter Sets to allow the user of your module to pass either a HashTable or the native request object in production.

One final note, you need to be aware that New-Object’s -Property parameter doesn’t handle nested HashTable’s. To Illustrate, consider the following PowerShell snippet:

Add-Type -TypeDefinition @"

public class Child {
	string name;
	public string Name {
		get { return name; }
		set { name = value; }
	}

	string email;
	public string Email {
		get { return email; }
		set { email = value; }
	}
}

public class Container {
	string name;
	public string Name {
		get { return name; }
		set { name = value; }
	}

	private Child myChild;
	public Child MyChild {
	 	get { return myChild; }
		set { myChild = value; }
	}
}
"@

$container = New-Object Container -Property @{
	Name = 'My Container'
	MyChild = @{
		Name = 'Justin Dearing'
		Email = 'justin@dearing.com'
	}
}

This will return the following error:

New-Object : The value supplied is not valid, or the property is read-only. Change the value, and then try again.
At C:UsersjdearingAppDataLocalTempe072a665-0543-4b85-9489-e843b9b932e5.ps1:32 char:24
+ $container = New-Object <<<< Container -Property @{
 + CategoryInfo : InvalidData: (:) [New-Object], Exception
 + FullyQualifiedErrorId : InvalidValue,Microsoft.PowerShell.Commands.NewObjectCommand

The workaround for this is you have to call New-Object on $container.MyChild first as follows:

Add-Type -TypeDefinition @"

public class Child {
	string name;
	public string Name {
		get { return name; }
		set { name = value; }
	}

	string email;
	public string Email {
		get { return email; }
		set { email = value; }
	}
}

public class Container {
	string name;
	public string Name {
		get { return name; }
		set { name = value; }
	}

	private Child myChild;
	public Child MyChild {
	 	get { return myChild; }
		set { myChild = value; }
	}
}
"@

$container = @{
	Name = 'My Container'
	MyChild = @{
		Name = 'Justin Dearing'
		Email = 'justin@dearing.com'
	}
}

$container.MyChild = New-Object Child -Property $container.MyChild;
$container = New-Object Container -Property $container;

Naturally, your code can get very verbose if your class is very complicated. You could probably write a script that reflects on the object type and handles nested HashTables. I’ll leave that as an exercise to the reader.

These three examples should help you gain a better understanding of the New-Object cmdlet and HashTable’s in PowerShell.