Built to Scale: The Mozilla Release Engineering toolbox
Mozilla builds and tests software at tremendous scale, for projects such as Firefox for Desktop and Android, and Firefox OS. A single code "check in" consumes over 200 compute hours to complete builds as well as correctness and performance tests on multiple platforms. We recently ran a record 73,000 test jobs in a single day. This talk will discuss the open source tools we use to optimize and automate the management of the hardware and software required to handle this scale of release engineering activity on our hybrid in-house/AWS infrastructure. I'll describe how to mitigate the inherent instability of running continuous tests on hundreds of mobile devices and producing reliable results. I'll also discuss how we recover when a change breaks everything.