Jenkins was installed using Brew (a 3rd party package manager that makes OSX package management feel more like Debian and includes a lot of 3rd party Linux packages) rather than use the native Jenkins OSX installer because it's easier to use as a headless installation as a build server should be. They both install projects and application code in different places, so don't install both ways or you'll have conflicts. Create a separate Jenkins user when installing it and give it a separate home directory.
Other software installed included:
- XCode (for building iOS apps...yes, this means the build server has to be a Mac OSX Machine...Apple made sure of that)-:
- Android SDK (for building Android apps)
- Genymotion (for running Android unit tests because the Android emulator is too slow)
- Crashlytics SDK (we use this on both places to track crash logs...how we use it deserves a separate blog post)
- Vagrant (for our server-side unit/build testing)
- Chef (how we automate server builds...another topic worth a separate blog post) pinned to match our AWS version (our backend runs on Amazon) via "sudo true && curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -v VERSION"
For Jenkins Plugins, we used the following:
- Android Emulator (for running Android unit tests on release builds)
- Android Lint (for graphs of lint results on Android code)
- Gradle (for Android builds after we switched to Android Studio)
- XCode (for iOS builds)
- Jenkins GIT Client (for pulling our source code off Bitbucket for each build)
- Jenkins Email Extension (for sending out build failure notifications)
- HipChat (for sending out build success/failure notifications)
- HockeyApp (for uploading to HockeyApp which we use for installing our apps on test devices)
- Workspace Cleanup (for cleaning up the workspace before each build starts)
For both Android/iOS, we have a "Dev" build job that watches for any code updates in Git and immediately triggers a build which runs lint and unit tests. This provides early warning that the code base might be in a non-buildable state or might have build issues. It's also a given that sometimes, an engineer might check in code without running unit tests or lint locally first, even though they're supposed to. :-)
The next group of jobs is for testing which is built daily at 7:30am before everyone gets in so we can include any late night changes (some engineers are night owls) that have been made; the jobs can also be kicked off by QA if they need to verify a bug fix immediately. We have different versions of our backend named, appropriately, Development, Staging, and Production. We need builds of the client apps that tie into these different back ends depending on the target audience. Developers get a special "Toggle" version that lets them select whichever back end they need to test with, including what can be an unstable/raw Development backend w/ the latest changes. QA (and 3rd party testing services like UTest) can use the "Staging" backend for their testing without clogging up the Production servers with their test activity. And alpha/beta test users can use the latest version of the app against their Production accounts.
During the build process, the different servers are specified using GCC_PREPROCESSOR_DEFINITIONS on iOS, and by using Gradle Flavors for the Android build. Note that we elected to use build flavors on Android instead of the preprocessor equivalent of changing a constant at compile time by modifying a file; this was simpler for our Jenkins builds, but doesn't exclude any of the debug code from being included in the binary file.
These jobs are done for different branches that may be in progress (usually next major feature release and current minor fix release). When a branch is considered ready for release, it's merged into the Release branch.
Finally, Release versions are built off the Release branch when code is merged into it. These jobs provide the binaries that are submitted to the various app stores. We don't put the official signing keys on the Jenkins machine, so these jobs just prepare a binary (an XArchive for iOS and an obfuscated APK for Android) that can be signed by the official "keymaster". This extra precaution is needed because if a signing key is ever compromised, an entirely new app has to be released to the app store and previous users can't be migrated over. For this part of the build process, the binaries and debug symbols and obfuscation maps are also copied over to an archive directory so we can get a useful stack trace via Crashlytics for crashes in the field.
The Jenkins server was set up to have multiple ways to notify people that builds failed. If your machine is able to send email directly, you could send a msg via a shell command. However, Jenkins also has a plugin to send mail via mandrillapp.com because it's low enough volume to use their free account; this allows notifications to be sent w/o worrying about whether you have a local mail server configured, etc. When a job fails, the build.log is also sent out with the email so we can see what the problem is without having to log into VPN and look at the Jenkins web console.
In addition, we use HipChat for our company instant messaging. The Jenkins HipChat plugin allows Jenkins to put messages directly into your instant messaging client and is more immediate than email and provides a backup channel in case the email server is down.
HockeyApp for App Installation
iOS apps require a "store" as a source to install from unlike Android which can be installed by sending users an APK binary file. Because of this, there are various 3rd party purveyors of this service. We formerly used Testflight but Apple bought them and killed off Android support; Testflight's job versions were tied to the package name so they weren't as flexible as HockeyApp's App ID's anyways.
The one kink we've had is that HockeyApp depends on version numbering to show you the latest version (they only let you download one per Jenkins job) that can be downloaded from their installer app and this version is based on the version number of the app which should increment with each build. On iOS, if you want to put extra information in (e.g., the git hash so you know what the code was built from), you have to put that information into the CFBundleShortVersionName instead of the CFBundleVersionName string via Plistbuddy.
Initially, we used the Android Emulator to run our unit/integration tests, but each build cycle took 25 minutes. This was not tolerable because you're supposed to find out whether someone's checkin broke the build ASAP. By switching to Genymotion, this was brought down to a much better 4 minutes, much of which is taken up by Gradle and Proguard.
Gradle is used for invoking unit tests since our switch to Android Studio. Unfortunately, none of the code coverage tools work with it yet (they only work w/ Maven), but we'll integrate them as soon as they are compatible.
This was an interesting exercise unto itself because it was my first experience with Apple's app signing system. Unlike with Android where you have a signing key inside a standalone Java keystore file and validation on Google's App store, Apple's signing system is heavily tied into the OS as well as the iOS device. Apple's keystore is built into the OS and unless you sign it w/ a release key or enterprise key, you have to include/identify all the devices that can install the app. While it's more flexible than Google's version which is essentially just a release key, it's totally confusing to figure out why a test device can't install an app even though it can download it. This only applies Adhoc keys and is useful for preventing apps from being leaked onto other devices you don't have control of (the Enterprise key essentially behaves like the Android keys in that any device can install an app signed with it).
iOS Adhoc Key Signing
Set up for this is worth documenting since it's so confusing to get right. The AdHoc profile has to be created on the Apple Developers Portal and downloaded as <something>.mobileprovision. Note that you can name this anything you want, but you’ll have to look inside for the UUID value. The UUID value is then added as a parameter to the xcodebuild command line as “PROVISION_PROFILE=<UUID>”. The provisioning profile files are stored in “~/Library/MobileDevice/Provisioning\ Profiles/”.
You also have to add the distribution key to your keychain. This is one of the most painful things to do because it depends on whether Jenkins runs as a real user or a headless service; it’s simplest to set it up as a regular user because you can then check everything by hand. The keychains are in ~/Library/Keychains of this user. To build the special keychain in that directory, do this to create one named ios.keychain (we’re creating a new keychain because we have to unlock it in Jenkins using a plaintext password) but before you do this, download the AppleWWDRCA root certificate from their developer site:
security create-keychain ios.keychain
security default-keychain -s ios.keychain
security import AppleWWDRCA.cer -k ios.keychain -A
security import <distribkey>.p12 -k ios.keychain -P ThePasswordYouSetWhenExporting -A
The Jenkins workflow should mirror what your testing/QA/build workflow is, even as it evolves. This setup has changed from the initial setup to include a Staging version and also making the Dev version select the server instead of locking it to Dev servers because our workflow and needs changed and I'm sure it will change more to match any future changes. As such, you do need a Jenkins caretaker who will help evolve your Jenkins setup...