A PowerShell based FTP runner for TeamCity builds

This blog post can be seen in continuation of my last post proposing a deployment strategy for Drupal installations, or an independent post in itself. The objective is to create a Build Runner for TeamCity builds that can upload/push (all or selectively) files/folders affected in each build to one or more remote servers over FTP for deployment.

The intent initially was indeed to automate deployment for Drupal installations but the (Powershell) scripts are written to provide a completely autonomous mechanism to trigger FTP uploads to remote servers for files/folders affected in each TeamCity build. The end-result is a set of PowerShell scripts that can be configured as a build step to achieve the stated objective.

To provide a bit of a background on why PowerShell was chosen, I originally intended to write a native TeamCity build runner taking cues from the code of existing runners open-sourced by JetBrains. However, I did not had a TeamCity dev enviornment available at hand for native Java-based TeamCity development. The fact that .Net provides a very friendly FtpWebRequest class that makes creating a custom FTP client a breeze, turned the tide in favour of a .Net based solution. And finally our TeamCity servers being all Windows based, sealed the decision in favor of PowerShell (further aided by the availability of a very powerful PwerShell build runner in TeamCity out-of-the-box).

The title of the post can technically be considered a bit inaccurate as its now amply clear we are not creating a new build runner in its purest sense but a set of scripts riding on the PowerShell build runner. But I would still like to perceive it as a FTP runner for TeamCity riding on the PowerShell build runner 🙂

Okay so now, these were the design objectives that I thought the FTP runner should satisfy:

1. Deep integration with TeamCity utilizing all customizability TeamCity has to offer (including elaborate build FTP messages for each build).
2. Build-oriented: a TeamCity build is usually triggered by commits to a version control system. The TeamCity admin can decide if each commit should trigger a new build or specify a set of conditions to combine multiple commits into a single build. The runner should integrate with this TeamCity’s behavior and should be able to cope with single, multiple or even no commits in a build or when builds are triggered manually.
3. The runner should only push the files/folders affected in a build to the remote server (and not the entire content under source control).
4. If multiple commits are combined together in a single build and a file changed multiple times in those commits, only a single upload for that file should be trigerred (for the last version of that file included in the build).
5. Some source control systems do not track directories explicitly (e.g. Git) in the metadata. They only track files and directories are assumed to be present implicitly for the paths where files exist (and the SCM client is expected to automatically create the parent directory for each path where a file exists).
   This means no directory information would be recorded in TeamCity or SCM logs when files are added/removed. The runner should be able to atleast automatically create directories for paths where files are added/modified on the remote server if they do not already exist.
6. The FTP parameters (remote FTP url, ftp credentials etc) should not be hard-coded in the scripts and configurable via TeamCity’s interface.
7. Ability to selectively push sub-folders from the content under version control to the remote server (and not necessarily track the entire source for pushing out remotely).
8. Ability to define mappings for source control folders and remote FTP folders. So for examples you should be able to send content from /folder1 in the content under source-control to folder2/sub-folder on the remote server.
9. For specific builds, ability to temporarily suspend FTP upload and not perform any action for those builds at all.

I would like to mention here that I found a third-party FTP runner available for TeamCity, and I played with it too. However it did not satisfy many of the listed objectives and the single most important reason for rejecting it out-right was it was trying to re-upload the entire content under source control to the remote server for each TeamCity build. It might be okay doing this for specific use-cases but isn’t acceptable generally I would say; and wasn’t at all for us considering our use-case of automating Drupal deployments.

Now some insights into TeamCity’s behaviour and internals much of which I discovered while writing these scripts for FTP deployment (and each successive discovery increased the wow factor for TeamCity, considering how carefully its thought out for extensibility. DISCLAIMER: this is not sponsored content , but my own assessment after what I worked with and found).

1. TeamCity provides three types of parameters for each build, Configuration parameters, System properties and Environment variables. There are subtle but important aspects of the scope and existence perimeter of each type of variables which usually would affect which type to opt when defining a custom parameter. More details on these parameter types here.
2. TeamCity maintains a list of all files/directories affected by each build in a text file whose path is accessible via system property teamcity.build.changedFiles.file.
3. This file only exists during the life-time of a build and is removed once a build completes.
4. The file contains a single line for each file/folder affected in any of the commits in the build.
   1. A file/folder affected by multiple commits in a single build is still listed only once.
   2. Whether folders are listed depends upon whether your SCM tracks folders explicitly.
   3. Each line contains 3 parts separated by colon (:) in this order:
      1. File/folder path relative to repo root.
      2. Type of change for a file/folder(more details here).
      3. Last Revision identifier for the file/folder in the build (revision identifier for example would be Revision number in SVN and commit hash in Git).
   4. The file would still exist if a build contains no SCM revisions, but would be empty in that case.

   I have attached a sample such file with this blog post (changedFiles7913905745611206951.txt).

5. TeamCity provides all system properties (and some other build parameters) available as a .ini formatted file whose path is provided as a system property, teamcity.build.properties.file as well as via environment variable, TEAMCITY_BUILD_PROPERTIES_FILE. Although undocumented, the same set of properties is also made available as a xml file whose path can be calculated by suffixing “.xml” extension to the value returned by teamcity.build.properties.file system property or TEAMCITY_BUILD_PROPERTIES_FILE environment variable. Again both these files only exist for the duration of a build.

   Any custom system properties defined are included in both these files and thus can be accessed in build scripts. I have attached both sample ini and xml build properties files below. These also contain custom parameters that are required by the powershell FTP upload scripts.

6. TeamCity provides a very exhaustive support for build scripts to write to TeamCity’s build log, more details available here.

This was enough enlightment about TeamCity support needed to write our FTP upload PS scripts that would achieve the stated objectives. However I would like to describe the parameters these scripts need before discussing the scripts themselves. All in all, the scripts utilize 3 required and 2 optional parameters:

1. ftpBaseUrl (required): The base url relative to which affected files/folders are pushed. This could be a server root (e.g. ftp://example.com) or a sub-directory (e.g. ftp://example.com/somefolder/).
2. ftpUsername (required): The username for the remote FTP server.
3. ftpPassword (required): The password for the remote FTP server.
4. ftpPathMappings (optional): This parameter allows you to re-map files/folders from your repo source to different files/folders on the remote server; or selectively push specific files/folders only to the remote server. You can enter multiple re-mapping rules semi-colon separated.

   Please note all paths should be relative to your repo source on one side and ftpBaseUrl parameter value on the other side. I would take a few examples to demonstrate the flexibility provided by this configuration parameter.

   1. In its simplest form, you can omit specifying any value for this parameter. In this case, all files/folders affected in a build from source repo would be pushed to remote server in the same hierarchy. The scripts would automatically take care of creating the folders as needed if they don’t exist.
   2. /folder1=>/folder2
      The above rule would cause mapping of folder1 in source repo to folder2 on remote ftp server, thus meaning any content from under folder1 in source repo goes to folder2 on ftp server.

      Please also note if this parameter value is non-empty, only those paths in the source repo affected by a build are pushed to the ftp server that satisfy one of the re-mapping rules specified by this property value. All other paths affected by the build are ignored. Therefore in this case, if /web/readme.txt file is affected by a build, it won’t be pushed to the remote server.

   3. /folder1=>/folder2;folder3=>folder4/sub-folder
      Content affected by the build under folder1 in source repo goes to folder2 on remote server and from folder3 in source repo goes to folder4/sub-folder on remote server. All other affected paths not satisfying any of these re-mapping rules are ignored. 
   4. /folder1=>/folder2;folder3=>folder4/sub-folder;/=>/
      The first two re-mapping rules work exactly the same is described in previous example. The interesting thing is all other affected paths not satisfying the first 2 re-mapping rules are pushed in their source hierarchy as is to the remote server because of the third re-mapping rule /=>/, which affectively says upload all other affected paths not matching any of the previous re-mapping rules as is to the remote server.

      Please note if you are selectively re-mapping content and pushing the rest of the content as is, then the “match-all” re-mapping rule (i.e. /=>/) should be the last rule in the semi-colon separated list. This also means more specific rules (e.g. folder1/sub-folder=>folder2) should occur before less specific rules (e.g. folder1=>folder3) if they have a common source path prefix (folder1 in the last example). 

5. ftpSuspendUploads (optional): This is a boolean (checkbox type) parameter which when set causes the scripts to not process FTP uploads for affected paths for builds where this parameter is set to true. Please note this parameter’s type should be set to checkbox with true as its Checked value and anything for Unchecked value.

Because of the way TeamCity parameters work, you should specify all these 5 parameters as System properties on your build. Starting with TeamCity 8.1, you are able to edit Spec for a parameter and select Password for a parameter’s type which masks the parameter’s value all over the TeamCity’s interface but making it available to build scripts in the properties files. You would want to specify type Password for both ftpUsername and ftpPassword parameters and possibly for ftpBaseUrl too. The screenshot to the right demonstrates how you can edit Spec for a parameter and choose type as password (please click on the screenshot to enlarge).

 

I would now jump to the scripts and give a brief overview of each of the files that comprise the set of scripts performing the actual FTP upload functionality (attached with this blog post as TeamCity.FtpRunner.zip). Please note only TeamCity.FtpRunner.zip contains the actual scripts, all other attachments are sample TeamCity files to help understand the working of the scripts and need not be deployed.

1. The first file you should take a look at from the .zip package is FtpHelper.cs. It is simply a wrapper for .Net’s FtpWebRequest class making it easier to invoke all FTP functionality from PowerShell in a single line taking care of plumbing for creating FTP requests and parsing FTP reponses.
   Despite all the flexibility of PS, I felt is was still easier to write a non-trivial class in C#. PowerShell’s Add-Type support made consuming the wrapper class in PS a breeze.
2. TeamCity.FtpHelper.ps1: This is a helper script making it easier to intergrate with TeamCity. Currently it provides utility functions making it easier to write to TeamCity’s build log.
3. TeamCity.FtpRunner.ps1: This is the main script that does all the heavy-lifting. This is what you would point to when creating the PowerShell build step in TeamCity. The script implements all the logic for TeamCity integration and triggering FTP functionality, the background of which has been provided above.
   You don’t really need to study the scripts if all your are interested in is just configuring a build for FTP uploads. For the nerds, feel free to open and dive into the actual logic.
4. License.txt: The scripts’ use is governed by Microsoft’s Public License and the file contains the text of the license.

The below screenshot demonstrates sample configuration for a PowerShell build step to utilize these scripts (please click to enlarge the screenshot):

Now comes the question, how would you get the scripts on your build server. There are a couple of ways of doing that:

1. You can either manually copy them to the build server at a known location and then configure the build step pointing the script path to that location.
2. Or you can put the scripts into your repo itself in some location that is not pushed to the remote ftp server. This usually means you will have to specify ftpPathMappings to exclude the scripts themselves from being uploaded to the remote server (unless you want these scripts also to be uploaded). If you do not want to use ftpPathMappings, then the more appropriate way is to copy them to the build server manually at a known location.

I have provided detailed steps to create a Configuration Template for these FTP runner scripts and subsequently creating a Build Configuration based on the template in my other blog post that I do not feel like repeating all over here (so please follow the links for getting assistance with creating the configuration template and build configuration to actually utilize these scripts).

The scripts are also available as a GitHub repo at the following url in case you are interested in checking-out the commit log or contributing to the scripts:
https://github.com/imbibe/TeamCityPsFtpRunner

As always, please feel free to use the comment section to let me know what you think about the approach and if you need help with anything related to this blog post.