The client project I’m currently working on is quite large. There are over 90 different screens required in the application, and regression testing all of these, with the different data variants and scenarios is not something I’d like to attempt by hand, and is not something I would expect my client to pay for me (or anyone for that matter) to do. Both unit and application tests can be automated, and I figured that this was something that I should do for this project to ensure changes I introduce don’t break existing functionality, and any bugs found won’t be introduced.
To solve this challenge I have introduced a Continuous Integration (CI) server to my workflow. My goals were to have unit and application tests automatically executed when an commit is made to a specific branch in a (local) git repository, and if both of these phases are successful, have the application packaged up, ready for distribution. I also wanted this successful build to be deployed on a daily basis to TestFlight so my client (and any other human testers in the future) could pick it up on their devices. I also wanted code coverage reporting, for both the unit and application tests, so I can see what parts of the application logic and screen flows are actually being tested to give me a level of confidence that the right stuff is getting the attention. The application tests also needed to be executed in a headless manner - I don’t have a physically separate machine (or VM) to run the CI server in, so I don’t want the simulator popping up and distracting me.
Warning - this is long and involved. I’ve also written this after the fact. I hope I’ve captured all the steps but if something doesn’t work for you please let me know and I’ll try to help and update this guide.
I used a set of existing tools to help with this
- CI Server - Jenkins
- Unit Tests - SenTest/OCUnit - it’s baked right into Xcode and meets my needs. It’s hard for me to justify using something else like GHUnit because of the pre-integration.
- Application (UI) Tests - Frank - I wanted to use a behaviour driven approach to UI testing, as well as something my client could actually specify tests in. Frank is cucumber based, so uses an english language syntax, making it easy for non-developers to specify and understand tests.
- Deployment - Curl - low tech, but the TestFlight API is simple to use, so nothing more complex is really required here. It could be wrapped in a Jenkins plugin, but I’m not the guy to create that…
And this is how I did it. I cobbled together all the tools I needed with help from various blogs, stack overflow answers, and vendor documentation. I’ve referenced the sources where I can. Hat tips to all.
Installation of Prerequisites
You’ll need ruby for compiling Frank, and I recommend you use rvm for this. Paste the following into a shell and follow the instructions.
bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
Extra install instructions (head builds, multi-user etc) are available at http://beginrescueend.com/rvm/install/
You’ll need to close and open shell after following the instructions (including modification of your
.bash_profile). You’ll also need to install your favourite ruby. I recommend installing 1.9.2 with
rvm install 1.9.2
You’ll also need rake (for managing Frank’s build process), so you can get that with
gem install rake
I used homebrew to install Jenkins. It’s a good package manager for OSX, and well maintained. Instructions are at https://github.com/mxcl/homebrew/wiki/installation but you can just paste the following into a shell and it’ll install.
/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"
Install Python and gcovr
gcovr (and python) are needed to generate the coverage files from the unit and application test output. If you already have a modern python installation (2.7) then skip the python installation step and just install gcovr with your own copy of easy_install
You can use homebrew to install python. Do so with:
brew install python
Then you will need easy_install (or the distribute tools)
You will then need to get gcovr.
Install and configure Jenkins
Thanks to homebrew, the installation step is very easy. Just paste the following into the shell and it’ll install.
brew install jenkins
There are a couple of configuration steps that are required as well. Thanks to http://mattonrails.wordpress.com/2011/06/08/jenkins-homebrew-mac-daemo/ for the configuration required.
Create a service account
First of all find an ID that is free on your system.
dscl . -search /Users uid 600 searches for users with ID 600, and
dscl . -search /Groups gid 600 looks for groups with ID 600. Change the number until you find an empty ID. Then (with appropriate ID changes)…
1 2 3 4 5 6 7 8 9 10 11 12 13
Starting up Jenkins
Create a file at
/Library/LaunchDaemons/org.jenkins-ci.plist with the following contents (check the version number in directory for Jenkins!). This will ensure Jenkins starts when the system boots.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
and load it with
sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
At this point, Jenkins should be installed and running. Head to http://localhost:8080 and it should be running.
In the Jenkins menu, click on Manage Jenkins, Manage Plugins, and then Available. Install the following plugins and make sure the “Restart Jenkins when installation is complete and no jobs are running” checkbox (on the screen after the “Install” button) is checked.
- Jenkins Cobertura Plugin - for code coverage report processing
- Jenkins GIT plugin - for integration with git
- Xcode Integration - for execution and understanding of Xcode files
Install the github plugins if your project is on github. I won’t be covering configuration of this here, but it should be relatively simple.
Once these are installed and Jenkins has restarted, you’ll need to configure the plugins. In the Jenkins menu, click on Manage Jenkins, Configure System and fill in the parameters for your git installation(s) and Xcode Builder paths. Click Save and we’re ready to go.
Configure your Xcode project
We’re going to take a little departure from Jenkins for the moment to prep Xcode for integration. We are going to setup our project for unit tests and Frank.
If you don’t have one already, you’ll need to set up a new target for Unit Tests in your Xcode project. Select your top level project in the in Project Navigator, and then “Add Target”. Under iOS -> Other choose the “Cocoa Touch Unit Testing Bundle”. Give it a name and then write some tests.
The other thing we’ll need to do here is set up Code Coverage. In the Build Phases area of your new Target, under Link Binary With Libraries, hit the + and select Add Other. Then navigate to
/Developer/usr/lib and select
libprofile_rt.dylib. This is the library that enables the profiling goodness. After this, select the Build Settings area, and set “Generate Test Coverage Files” and “Instrument Program Flow” both to Yes in the column for your Unit Test target. Ensure that “Library Search Paths” includes
$(DEVELOPER_DIR)/usr/lib, but this should be there already.
You should be set up for code coverage now. If you want to check this is working, ensure your build target (up the top on the right on the Run & Stop buttons) is set to your Unit Test target and iPhone Simulator, then build and test your unit test target. Then, open Organizer, choose the Projects item from the toolbar, select your project, and then click the little arrow next to the Derived Data directory. This will open the build location in Finder. In here, open the selected directory, then navigate to
Build/Intermediates/<Your Project>.build/Debug-iphonesimulator/<Your Unit Test Target>.build/Objects-normal/i386/ and in there there should be a set of
.gcno (generated at build) and
.gcda (generated when your test target executed and finished) files. These are the code coverage files. If you’d like to have a look at them before we integrate back into Jenkins, get CoverStory and open them up.
Application (UI) Tests with Frank
Now we’ll set up the application test target with Frank. First of all, we need to install Frank and cucumber.
We will need to use a customised version of Frank to enable the iOS Simulator to exit after execution of the tests. The standard build does not include a method to terminate an application so we’ll need to build this in. Thanks to Martin Hauner at http://softnoise.wordpress.com/2010/11/14/ios-running-cucumberfrank-with-code-coverage-in-hudson/ for the tip on this. I’ve forked frank and included this exit method, and you can get it from https://github.com/rickerbh/Frank/tree/exitCommand with the command
git clone [email protected]:rickerbh/Frank.git and then switch to the exitCommand branch. You’ll also need to
git submodule init and
git submodule update, and then check the submodules that are pulled in as I recall the submodules have submodules :-/
Once all that is done, you’ll need to be in the root directory of Frank that you cloned, and compile my branch of Frank with the following command.
After this is finished, there should be a file at
dist/libFrank.a. This is the customised library that we’ll need to use with the exit command built in.
(Full Frank installation instructions available at http://www.testingwithfrank.com/installing.html - I’ll paraphrase here with a couple of sightly different steps for the custom library and code coverage inclusion) For convenience of installation, I actually installed the proper Frank gem rather than my customised build. You can do this with
gem install frank-cucumber. Then,
cd to your project directory, and run
frank-skeleton. This installs Frank in your project directory. It also copies a version of the official
libFrank.a file into
Frank/. You’ll need to replace that with the version that we built.
To add Frank to your Xcode project, you’ll need a new target (you don’t want the Frank server installed in the Release version of your application). Duplicate your main application target by right clicking on it and selecting Duplicate. Rename the new target “\
frank-skeleton) to your Xcode project. Ensure that it’s only added to your frankified target, not your main application target. Add
CFNetwork.framework to the Frankified “Link Binary With Libraries” section of the Build Phases. Then, add
-ObjC to the “Other Linker Flags” Build Setting.
To enable code coverage for Frank, add
--coverage to the “Other Linker Flags” Build Setting, and set “Generate Test Coverage Files” and “Instrument Program Flow” both to Yes in the column for your Frankified target.
If you build and run the Frankified target for the iPhone simulator of your application, it should build OK and start the simulator. Head to http://localhost:37265 and you should see the Symbiote browser of your iPhone application.
If you want to see if the coverage is working, head to
Build/Intermediates/<Your Project>.build/Debug-iphonesimulator/<Your Frankified Target>.build/Objects-normal/i386/ just like with the Unit Tests area above.
You should then write some tests in cucumber and make sure they work.
Troubleshooting Frank and Code Coverage
I had a lot of trial and error (mostly error actually) getting coverage working with Frank. I had to play a bit with the “Library Search Paths” Build Setting so it could find the correct coverage library to include. I have
$(Developer)/Platforms/iPhoneOS.platform/Developer/usr/lib added in the setting, but can’t recall if this is required or not - apologies for this.
If you are getting the
.gcno created but not the
.gcda files, it could be an issue of your application not exiting correctly when the simulator terminates (as the
.gcda files are only written when the application terminates). If this is the case, head into the Info area for your Frankified build target. Add/Set the “Application does not run in background” property to the application, and set it to YES. This will ensure the app terminates when the home button is pressed, and the
.gcda files are created.
I also had an issue when the application was attempting to write the
.gcda files. It was complaining that
fopen$UNIX2003 called from function llvm_gcda_start_file. I found this entry in stackoverflow and created the c methods in my main.m file and it enabled the application to write the files correctly.
You may also need to enable the Accessibility inspector in the iPhone Simulators Settings app under General > Accessibility - instructions from Apple.
If you are getting a message that says
Couldn't register XXXX with the bootstrap server. Error: unknown error code. This generally means that another instance of this process was already running or is hung in the debugger. this means that either the Simulator is already running (it must be quit for the headless build to run) or that there is a previous version of the headless test exection that has hung. You can find these with
ps -ef | grep Fran (Fran for frankified). Kill them with the
Configuring Frank for Headless Tests under Jenkins
There will be a file in your Xcode project under the Frank directory at
support/env.rb - this will contain some environmental settings that are used for executing the cucumber tests. Replace the content with the below with the appropriate text replacements. You will need different settings for manual command line based test execution, and the Jenkins based tests, so there is an environment based conditional in the file. Default is for command line based testing.
For the replacement text for
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
In your Xcode project under the Frank directory, there will be another directory named
step_definitions. This should contain a .rb file that has some ruby and cucumber/frank definitions it it. We need to add a couple of things to that file. Again thanks to Martin Hauner here.
Add the following lines to the top of the ruby file in the step_definitions directory. I’ll explain what these are…
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
Before block gets executed before any tests are run. The accessibility plist/content section sets up a file that enables the accessibility settings for the simulator. This is created every time the cucumber tests are run to ensure accessibility is in the correct state. The
CFFIXED_USER_HOME are all directories for the simulator to function correctly. Thanks to Matt Gallagher’s blog entry for aiding my understanding of these.
launch_app_headless method adds a flag when launching the application so it launches headlessly. If you wanted, you could actually add a conditional so that the headless method is only used when launching under jenkins.
frankly_exit method will call the new exit method that we built into
libFrank.a. This is called from the
After block, which gets called after the tests are executed.
That should be all the configuration you need. If you alter your frank/cucumber tests to use the “Given I launch the headless app” call to start the application, you should now be able to run it in a headless manner. Execute
cucumber in your Frank directory inside your Xcode project dir to test it out.
Setting up the jobs in Jenkins
The way I have structured my tasks in jenkins is that I have 4 different jobs. I have a unit test execution job that triggers off a git push. If this is successful, I have the UI test job that executes. If this is successful, I then package and archive the binary that was generated from that push. The last job that is configured is a daily distribution of the last successfully tested application to a group on TestFlight.
Setting up the unit test job in Jenkins
Navigate to Jenkins and click on “New Job”. Give it a name (I called mine “Project Unit Tests”), and select the “Build a free-style software project” radio button, then click OK.
You should now be in a screen to configure the build settings for your unit test target. In the source code management area, choose git. Enter your repo name (something like
[email protected]:my-project.git). If you have a specific branch you want to build from put it in the “Branches to build” box (mine is
*/develop). In the “Build Triggers” area below, select “Poll SCM”. I’ve set my schedule to
* * * * * meaning it’ll look every minute for a new push.
Navigate down the screen until you find the “Add build step” button. Click it and select Xcode. Then, fill in the following boxes.
- Clean before build - check this box. I like clean builds before testing.
- Target - set this to the unit test target name from Xcode (no escaping of spaces required)
- SDK - set this to the SDK you want to build for. Mine is
- Configuration - set this to
- Keychain path - this should already be set to
That’s all the Xcode build information set up. Now, to generate the coverage test files. Add an “Execute Shell” build step with the “Add build step” button. In the “Command” box, call gcovr (
which gcovr to find your own install location) with
/usr/local/share/python/gcovr -r "<Your Jenkins install location>/jobs/Project Unit Tests/workspace" --exclude '.*UnitTests.*' --xml > "<Your Jenkins install location>/jobs/Project Unit Tests/workspace/coverage.xml"
In the Post-build Actions section, check the “Archive the artifacts” box and set the “Files to archive” field to
Check the “Publish Cobertura Coverage Report” box and set the “Cobertura xml report pattern” to
Also check the “Publish JUnit test result report” box and set the “Test report XMLs” to
Click the Save button down the bottom, and then attempt to run your unit tests manually. The code should be checked out from your git repo, build the unit test target, run the tests and produce the unit test reports and coverage results.
Setting up the application test job in Jenkins
Navigate to Jenkins and click on “New Job”. Give it a name (I called mine “Project UI Tests”), and select the radio button to copy the unit test job that was previously set up. In the Configuration screen for the new job, alter the following fields.
- Build Triggers - set this to be “Build after other projects are built” and type in the name of your unit test job.
- Build Triggers - uncheck poll scm
- Target - Set this to your Frankified target name from Xcode
Change the existing “Execute shell” step to use your UI Test build job name rather than the unit test job name.
/usr/local/share/python/gcovr -r "<Your Jenkins install location>/jobs/Project UI Tests/workspace" --exclude '.*UnitTests.*' --xml > "<Your Jenkins install location>/jobs/Project UI Tests/workspace/coverage.xml"
Create a new “Execute shell build step”, and click and drag it (with the little 4x4 set of boxes on the right of the “Execute shell” label on screen) to move it inbetween the Xcode step and the gcovr step. Insert the following commands in the shell box.
1 2 3 4
The source line sets up the environment for rvm so the frank gem is included correctly - set it to the appropriate ruby version in the ~/.rvm/environments directory. Mine is ~/.rvm/environments/ruby-1.9.2-p290
Uncheck the “Archive the artifacts” option, and click save. You can now attempt to run your application tests via Frank. This will check out the code, build the Frankified target, execute the tests and then process the unit test and coverage reports.
FYI - my unit test reports for this step are empty. If someone figures out how to get cucumber to put something useful in them please let me know!
Setting up the archive job
Similar to the application test job, create a new job that copies the Unit Test job. I called mine Project Developer Build. In the Configuration screen for the new job, alter the following fields.
- Build Triggers - set this to be “Build after other projects are built” and type in the name of your application/UI test job.
- Build Triggers - uncheck poll scm
- Target - Set this to your application target name from Xcode (mine is Project TestFlight - I have a specific target that includes the TestFlight SDK)
- Configuration - I have this set to release. Be sure to setup and test the signing identities correctly in Xcode for this. You must use the adhoc profile that should be used in TestFlight.
- Build IPA - check this.
Remove the execute shell build step with the Delete button.
Uncheck the Cobertura and Unit test report generation Post-build actions. Change the Archive files location to
Click save, and now whenever both your unit and application test steps pass, you’ll have a version of the application built and archived ready to go to TestFlight.
Distributing to TestFlight
Thanks to the Shine Technologies team for their blog entry that helped here.
Create another free-style software project job (last one, I promise!) in Jenkins - I’ve called mine Project TestFlight Deployment. Set the build triggers to “Build periodically” and set the schedule to whatever time you want to upload the application to TestFlight. Mine is set to 10pm (
0 22 * * *).
Add an Execute shell build step. This will call curl to upload the application to TestFlight. You will need your API token and Team token from Testflight. Set the Command to the following
The ipa that’s being uploaded there is the version that was saved in the last build job. For the name, just look inside the Project TestFlight job and it’ll have the name of the “Last Successful Artifact” - this is what you need to upload.
That should be it.
End - finally
If you experience issues with this, have corrections or useful troubleshooting steps then please let me know. This process was a bit of a pain to get working correctly so I hope this guide is useful for someone. You should also follow me on twitter @rickerbh.