The client project I'm currently working on is quite large. There are over 90 different screens required in the application, and regression testing all of these, with the different data variants and scenarios is not something I'd like to attempt by hand, and is not something I would expect my client to pay for me (or anyone for that matter) to do. Both unit and application tests can be automated, and I figured that this was something that I should do for this project to ensure changes I introduce don't break existing functionality, and any bugs found won't be introduced.
To solve this challenge I have introduced a Continuous Integration (CI) server to my workflow. My goals were to have unit and application tests automatically executed when an commit is made to a specific branch in a (local) git repository, and if both of these phases are successful, have the application packaged up, ready for distribution. I also wanted this successful build to be deployed on a daily basis to TestFlight so my client (and any other human testers in the future) could pick it up on their devices. I also wanted code coverage reporting, for both the unit and application tests, so I can see what parts of the application logic and screen flows are actually being tested to give me a level of confidence that the right stuff is getting the attention. The application tests also needed to be executed in a headless manner - I don't have a physically separate machine (or VM) to run the CI server in, so I don't want the simulator popping up and distracting me.
Warning - this is long and involved. I've also written this after the fact. I hope I've captured all the steps but if something doesn't work for you please let me know and I'll try to help and update this guide.
I used a set of existing tools to help with this
And this is how I did it. I cobbled together all the tools I needed with help from various blogs, stack overflow answers, and vendor documentation. I've referenced the sources where I can. Hat tips to all.
You'll need ruby for compiling Frank, and I recommend you use rvm for this. Paste the following into a shell and follow the instructions.
bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
Extra install instructions (head builds, multi-user etc) are available at http://beginrescueend.com/rvm/install/
You'll need to close and open shell after following the instructions (including modification of your .bash_profile). You'll also need to install your favourite ruby. I recommend installing 1.9.2 with rvm install 1.9.2
You'll also need rake (for managing Frank's build process), so you can get that with gem install rake
I used homebrew to install Jenkins. It's a good package manager for OSX, and well maintained. Instructions are at https://github.com/mxcl/homebrew/wiki/installation but you can just paste the following into a shell and it'll install.
/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"
gcovr (and python) are needed to generate the coverage files from the unit and application test output. If you already have a modern python installation (2.7) then skip the python installation step and just install gcovr with your own copy of easy_install
You can use homebrew to install python. Do so with:
brew install python
Then you will need easy_install (or the distribute tools)
curl -O http://peak.telecommunity.com/dist/ez_setup.py
/usr/local/bin/python ez_setup.py
You will then need to get gcovr.
/usr/local/share/python/easy_install gcovr
Thanks to homebrew, the installation step is very easy. Just paste the following into the shell and it'll install.
brew install jenkins
There are a couple of configuration steps that are required as well. Thanks to http://mattonrails.wordpress.com/2011/06/08/jenkins-homebrew-mac-daemo/ for the configuration required.
First of all find an ID that is free on your system. dscl . -search /Users uid 600 searches for users with ID 600, and dscl . -search /Groups gid 600 looks for groups with ID 600. Change the number until you find an empty ID. Then (with appropriate ID changes)...
sudo mkdir /var/jenkins
sudo /usr/sbin/dseditgroup -o create -r 'Jenkins CI Group' -i 600 _jenkins
sudo dscl . -append /Groups/_jenkins passwd "*"
sudo dscl . -create /Users/_jenkins
sudo dscl . -append /Users/_jenkins RecordName jenkins
sudo dscl . -append /Users/_jenkins RealName "Jenkins CI Server"
sudo dscl . -append /Users/_jenkins uid 600
sudo dscl . -append /Users/_jenkins gid 600
sudo dscl . -append /Users/_jenkins shell /usr/bin/false
sudo dscl . -append /Users/_jenkins home /var/jenkins
sudo dscl . -append /Users/_jenkins passwd "*"
sudo dscl . -append /Groups/_jenkins GroupMembership _jenkins
sudo chown -R jenkins /var/jenkins
Create a file at /Library/LaunchDaemons/org.jenkins-ci.plist with the following contents (check the version number in directory for Jenkins!). This will ensure Jenkins starts when the system boots.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>Jenkins</string>
<key>ProgramArguments</key>
<array>
<string>/usr/bin/java</string>
<string>-jar</string>
<string>/usr/local/Cellar/jenkins/1.428/lib/jenkins.war</string>
</array>
<key>OnDemand</key>
<false/>
<key>RunAtLoad</key>
<true/>
<key>UserName</key>
<string>jenkins</string>
</dict>
</plist>
and load it with sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
At this point, Jenkins should be installed and running. Head to http://localhost:8080 and it should be running.
In the Jenkins menu, click on Manage Jenkins, Manage Plugins, and then Available. Install the following plugins and make sure the "Restart Jenkins when installation is complete and no jobs are running" checkbox (on the screen after the "Install" button) is checked.
Install the github plugins if your project is on github. I won't be covering configuration of this here, but it should be relatively simple.
Once these are installed and Jenkins has restarted, you'll need to configure the plugins. In the Jenkins menu, click on Manage Jenkins, Configure System and fill in the parameters for your git installation(s) and Xcode Builder paths. Click Save and we're ready to go.
We're going to take a little departure from Jenkins for the moment to prep Xcode for integration. We are going to setup our project for unit tests and Frank.
If you don't have one already, you'll need to set up a new target for Unit Tests in your Xcode project. Select your top level project in the in Project Navigator, and then "Add Target". Under iOS -> Other choose the "Cocoa Touch Unit Testing Bundle". Give it a name and then write some tests.
The other thing we'll need to do here is set up Code Coverage. In the Build Phases area of your new Target, under Link Binary With Libraries, hit the + and select Add Other. Then navigate to /Developer/usr/lib and select libprofile_rt.dylib. This is the library that enables the profiling goodness. After this, select the Build Settings area, and set "Generate Test Coverage Files" and "Instrument Program Flow" both to Yes in the column for your Unit Test target. Ensure that "Library Search Paths" includes $(DEVELOPER_DIR)/usr/lib, but this should be there already.
You should be set up for code coverage now. If you want to check this is working, ensure your build target (up the top on the right on the Run & Stop buttons) is set to your Unit Test target and iPhone Simulator, then build and test your unit test target. Then, open Organizer, choose the Projects item from the toolbar, select your project, and then click the little arrow next to the Derived Data directory. This will open the build location in Finder. In here, open the selected directory, then navigate to Build/Intermediates/<Your Project>.build/Debug-iphonesimulator/<Your Unit Test Target>.build/Objects-normal/i386/ and in there there should be a set of .gcno (generated at build) and .gcda (generated when your test target executed and finished) files. These are the code coverage files. If you'd like to have a look at them before we integrate back into Jenkins, get CoverStory and open them up.
Now we'll set up the application test target with Frank. First of all, we need to install Frank and cucumber.
We will need to use a customised version of Frank to enable the iOS Simulator to exit after execution of the tests. The standard build does not include a method to terminate an application so we'll need to build this in. Thanks to Martin Hauner at http://softnoise.wordpress.com/2010/11/14/ios-running-cucumberfrank-with-code-coverage-in-hudson/ for the tip on this. I've forked frank and included this exit method, and you can get it from https://github.com/rickerbh/Frank/tree/exitCommand with the command git clone git@github.com:rickerbh/Frank.git and then switch to the exitCommand branch. You'll also need to git submodule init and git submodule update, and then check the submodules that are pulled in as I recall the submodules have submodules :-/
Once all that is done, you'll need to be in the root directory of Frank that you cloned, and compile my branch of Frank with the following command.
rake build_lib
After this is finished, there should be a file at dist/libFrank.a. This is the customised library that we'll need to use with the exit command built in.
(Full Frank installation instructions available at http://www.testingwithfrank.com/installing.html - I'll paraphrase here with a couple of sightly different steps for the custom library and code coverage inclusion) For convenience of installation, I actually installed the proper Frank gem rather than my customised build. You can do this with gem install frank-cucumber. Then, cd to your project directory, and run frank-skeleton. This installs Frank in your project directory. It also copies a version of the official libFrank.a file into Frank/. You'll need to replace that with the version that we built.
To add Frank to your Xcode project, you'll need a new target (you don't want the Frank server installed in the Release version of your application). Duplicate your main application target by right clicking on it and selecting Duplicate. Rename the new target "<Your app name> Frankified". Then, add the Frank directory (that was created when you ran frank-skeleton) to your Xcode project. Ensure that it's only added to your frankified target, not your main application target. Add CFNetwork.framework to the Frankified "Link Binary With Libraries" section of the Build Phases. Then, add -all_load and -ObjC to the "Other Linker Flags" Build Setting.
To enable code coverage for Frank, add --coverage to the "Other Linker Flags" Build Setting, and set "Generate Test Coverage Files" and "Instrument Program Flow" both to Yes in the column for your Frankified target.
If you build and run the Frankified target for the iPhone simulator of your application, it should build OK and start the simulator. Head to http://localhost:37265 and you should see the Symbiote browser of your iPhone application.
If you want to see if the coverage is working, head to Build/Intermediates/<Your Project>.build/Debug-iphonesimulator/<Your Frankified Target>.build/Objects-normal/i386/ just like with the Unit Tests area above.
You should then write some tests in cucumber and make sure they work.
I had a lot of trial and error (mostly error actually) getting coverage working with Frank. I had to play a bit with the "Library Search Paths" Build Setting so it could find the correct coverage library to include. I have $(Developer)/Platforms/iPhoneOS.platform/Developer/usr/lib added in the setting, but can't recall if this is required or not - apologies for this.
If you are getting the .gcno created but not the .gcda files, it could be an issue of your application not exiting correctly when the simulator terminates (as the .gcda files are only written when the application terminates). If this is the case, head into the Info area for your Frankified build target. Add/Set the "Application does not run in background" property to the application, and set it to YES. This will ensure the app terminates when the home button is pressed, and the .gcda files are created.
I also had an issue when the application was attempting to write the .gcda files. It was complaining that fopen$UNIX2003 called from function llvm_gcda_start_file. I found this entry in stackoverflow and created the c methods in my main.m file and it enabled the application to write the files correctly.
You may also need to enable the Accessibility inspector in the iPhone Simulators Settings app under General > Accessibility - instructions from Apple.
If you are getting a message that says Couldn't register XXXX with the bootstrap server. Error: unknown error code. This generally means that another instance of this process was already running or is hung in the debugger. this means that either the Simulator is already running (it must be quit for the headless build to run) or that there is a previous version of the headless test exection that has hung. You can find these with ps -ef | grep Fran (Fran for frankified). Kill them with the kill command.
There will be a file in your Xcode project under the Frank directory at support/env.rb - this will contain some environmental settings that are used for executing the cucumber tests. Replace the content with the below with the appropriate text replacements. You will need different settings for manual command line based test execution, and the Jenkins based tests, so there is an environment based conditional in the file. Default is for command line based testing.
For the replacement text for
require 'frank-cucumber'
ENV['TESTING_ENV'] ||= 'command_line'
environment = ENV['TESTING_ENV']
if environment == 'command_line'
BASE_DIR = "<Your home directory>/Library/Developer/Xcode/DerivedData/<Your long and random derived data dir from Xcodes project organizer>/"
APP_BUNDLE_PATH = "#{BASE_DIR}Build/Products/Debug-iphonesimulator/<Your frankified target>.app"
APP_DIR = "#{BASE_DIR}Build/Intermediates/<Your project name>.build/Debug-iphonesimulator/<Your frankified target>.build"
elsif environment == 'jenkins'
BASE_DIR = "<Your Jenkins install location>/jobs/<Your UI Testing Jenkins Job Name>/workspace/"
APP_BUNDLE_PATH = "#{BASE_DIR}build/Debug-iphonesimulator/<Your frankified target>.app"
APP_DIR = "#{BASE_DIR}build/<Your project name>.build/Debug-iphonesimulator/<Your frankified target>.build"
end
#### Common ####
SDK_DIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk"
APP_BINARY = "#{APP_BUNDLE_PATH}/<Your frankified target>"
USER_DIR = "iPhone Simulator/User"
PREF_DIR = "#{USER_DIR}/Library/Preferences"
In your Xcode project under the Frank directory, there will be another directory named step_definitions. This should contain a .rb file that has some ruby and cucumber/frank definitions it it. We need to add a couple of things to that file. Again thanks to Martin Hauner here.
Add the following lines to the top of the ruby file in the step_definitions directory. I'll explain what these are...
require 'fileutils'
ACCESIBILITY_PLIST = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>ApplicationAccessibilityEnabled</key>
<true/>
</dict>
</plist>
PLIST
Before do
# check that pwd contains the "build" dir as we are creating
# items relative to it.
#Dir["build"].length.should == 1
# make sure we do start with a clean environment
FileUtils.remove_dir("#{USER_DIR}",true)
pwd = "#{Dir.pwd}"
prefdir = "#{PREF_DIR}"
FileUtils.mkdir_p prefdir
File.open("#{PREF_DIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
f <<ACCESIBILITY_CONTENT
end
ENV['SDKROOT'] = "#{SDK_DIR}"
ENV['DYLD_ROOT_PATH'] = "#{SDK_DIR}"
ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDK_DIR}"
ENV['TEMP_FILES_DIR'] = "#{APP_DIR}"
ENV['CFFIXED_USER_HOME'] = "#{pwd}/#{USER_DIR}"
end
After do
frankly_exit
end
def launch_app_headless
@apppid = fork do
exec(APP_BINARY, "-RegisterForSystemEvents")
end
wait_for_frank_to_come_up
end
def frankly_exit
get_to_uispec_server('exit')
# calling exit in the app will not return any response
# so we simply catch the error caused by exiting.
rescue EOFError
end
Given /^I launch the headless app$/ do
launch_app_headless
end
The Before block gets executed before any tests are run. The accessibility plist/content section sets up a file that enables the accessibility settings for the simulator. This is created every time the cucumber tests are run to ensure accessibility is in the correct state. The SDKROOT, DYLD_ROOT_PATH, IPHONE_SIMULATOR_ROOT, TEMP_FILES_DIR, and CFFIXED_USER_HOME are all directories for the simulator to function correctly. Thanks to Matt Gallagher's blog entry for aiding my understanding of these.
The launch_app_headless method adds a flag when launching the application so it launches headlessly. If you wanted, you could actually add a conditional so that the headless method is only used when launching under jenkins.
The frankly_exit method will call the new exit method that we built into libFrank.a. This is called from the After block, which gets called after the tests are executed.
That should be all the configuration you need. If you alter your frank/cucumber tests to use the "Given I launch the headless app" call to start the application, you should now be able to run it in a headless manner. Execute cucumber in your Frank directory inside your Xcode project dir to test it out.
The way I have structured my tasks in jenkins is that I have 4 different jobs. I have a unit test execution job that triggers off a git push. If this is successful, I have the UI test job that executes. If this is successful, I then package and archive the binary that was generated from that push. The last job that is configured is a daily distribution of the last successfully tested application to a group on TestFlight.
Navigate to Jenkins and click on "New Job". Give it a name (I called mine "Project Unit Tests"), and select the "Build a free-style software project" radio button, then click OK.
You should now be in a screen to configure the build settings for your unit test target. In the source code management area, choose git. Enter your repo name (something like git@localhost:my-project.git). If you have a specific branch you want to build from put it in the "Branches to build" box (mine is */develop). In the "Build Triggers" area below, select "Poll SCM". I've set my schedule to * * * * * meaning it'll look every minute for a new push.
Navigate down the screen until you find the "Add build step" button. Click it and select Xcode. Then, fill in the following boxes.
/Developer/Platform/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk/DEBUG${HOME}/Library/Keychains/login.keychainThat's all the Xcode build information set up. Now, to generate the coverage test files. Add an "Execute Shell" build step with the "Add build step" button. In the "Command" box, call gcovr (which gcovr to find your own install location) with /usr/local/share/python/gcovr -r "<Your Jenkins install location>/jobs/Project Unit Tests/workspace" --exclude '.*UnitTests.*' --xml > "<Your Jenkins install location>/jobs/Project Unit Tests/workspace/coverage.xml"
In the Post-build Actions section, check the "Archive the artifacts" box and set the "Files to archive" field to build/Debug-iphoneos/*.ipa
Check the "Publish Cobertura Coverage Report" box and set the "Cobertura xml report pattern" to **/coverage.xml.
Also check the "Publish JUnit test result report" box and set the "Test report XMLs" to test-reports/*.xml.
Click the Save button down the bottom, and then attempt to run your unit tests manually. The code should be checked out from your git repo, build the unit test target, run the tests and produce the unit test reports and coverage results.
Navigate to Jenkins and click on "New Job". Give it a name (I called mine "Project UI Tests"), and select the radio button to copy the unit test job that was previously set up. In the Configuration screen for the new job, alter the following fields.
Change the existing "Execute shell" step to use your UI Test build job name rather than the unit test job name. /usr/local/share/python/gcovr -r "<Your Jenkins install location>/jobs/Project UI Tests/workspace" --exclude '.*UnitTests.*' --xml > "<Your Jenkins install location>/jobs/Project UI Tests/workspace/coverage.xml"
Create a new "Execute shell build step", and click and drag it (with the little 4x4 set of boxes on the right of the "Execute shell" label on screen) to move it inbetween the Xcode step and the gcovr step. Insert the following commands in the shell box.
source <Your home dir>/.rvm/environments/<your ruby version>
export TESTING_ENV="jenkins"
cd "<Your Jenkins install location>/jobs/Project UI Tests/workspace/Frank"
cucumber -f junit --out ../test-reports
The source line sets up the environment for rvm so the frank gem is included correctly - set it to the appropriate ruby version in the ~/.rvm/environments directory. Mine is ~/.rvm/environments/ruby-1.9.2-p290
Uncheck the "Archive the artifacts" option, and click save. You can now attempt to run your application tests via Frank. This will check out the code, build the Frankified target, execute the tests and then process the unit test and coverage reports.
FYI - my unit test reports for this step are empty. If someone figures out how to get cucumber to put something useful in them please let me know!
Similar to the application test job, create a new job that copies the Unit Test job. I called mine Project Developer Build. In the Configuration screen for the new job, alter the following fields.
Remove the execute shell build step with the Delete button.
Uncheck the Cobertura and Unit test report generation Post-build actions. Change the Archive files location to build/Release-iphoneos/*.ipa
Click save, and now whenever both your unit and application test steps pass, you'll have a version of the application built and archived ready to go to TestFlight.
Thanks to the Shine Technologies team for their blog entry that helped here.
Create another free-style software project job (last one, I promise!) in Jenkins - I've called mine Project TestFlight Deployment. Set the build triggers to "Build periodically" and set the schedule to whatever time you want to upload the application to TestFlight. Mine is set to 10pm (0 22 * * *).
Add an Execute shell build step. This will call curl to upload the application to TestFlight. You will need your API token and Team token from Testflight. Set the Command to the following
cd ../..
curl http://testflightapp.com/api/builds.json -F file=@Project\ TestFlight/lastSuccessful/archive/build/Release-iphoneos/Project\ TestFlight-Release.ipa -F api_token=’<api token>’ -F team_token=’<team token>’ -F notes=’This is an auto deploy build of the develop branch with Release configuration’ -F notify=True -F distribution_lists=’<name of test distribution list>’
The ipa that's being uploaded there is the version that was saved in the last build job. For the name, just look inside the Project TestFlight job and it'll have the name of the "Last Successful Artifact" - this is what you need to upload.
That should be it.
If you experience issues with this, have corrections or useful troubleshooting steps then please let me know. This process was a bit of a pain to get working correctly so I hope this guide is useful for someone. You should also follow me on twitter @rickerbh.