by Leland Takamine and Gergely Orosz

We’ve worked for years at the cutting edge of the mobile industry at mobile-first startups and industry-leading organizations like Uber and Google. We also collaborated with mobile engineering experts from Twitter through Flipkart to many others when writing the book Building Mobile Apps at Scale. We’ve explored the best practices — and constraints — of the mobile industry. Time and again, we’ve come back again to the main constraint that gets in the way of most mobile teams. It is this:

Mobile engineering has a teething problem with shipping with confidence. Most native mobile teams struggle to move as confidently as backend and web teams do. They struggle to move fast while also keeping a high quality bar.

We started mobile.dev to build a platform that solves these problems. But what is the biggest issue that holds back teams from shipping confidently?

Building software for mobile is fundamentally different from web or backend development. While the programming paradigm, the tools, and the platform all vary slightly, the main distinction is the constraint on the release cycle.

Shipping to production on mobile is slower than on most other platforms. Additionally, on mobile, more regressions tend to be discovered after shipping to production. These regressions are more expensive because of the slower release cycle compared to web or backend teams. The slower release cycle also results in longer delays rolling out fixes to end users.

It often takes 2–3 weeks to ship a code change into the hands of the first few customers with native mobile apps. Why so long? Mobile apps are distributed as binaries: once they go out, mistakes are hard to correct. Native apps also have to go through an app store review process. All these constraints have forced mobile teams to do more manual testing before release, and manage a more gradual rollout in the app stores.

Detecting and resolving issues with mobile apps take months at most companies. Following a weeks-long staged rollout, analytics are collected from customers running the app. Regressions like crashes, performance issues, memory leaks, network usage, and battery drain are often detected months later — if at all — and take further months to fix and validate.

What if we challenged the long iteration cycle of engineers detecting and resolving mobile quality issues? Before we do so, let’s understand the traditional approach most software teams take to keep the quality bar high.

The Traditional Approach to Shipping With Confidence

Having worked with dozens of engineers on various teams across the industry on mission-critical projects, we noticed common patterns on teams which ship with confidence. These observations hold not just for mobile, but across backend, mobile, and other platforms.

1. Detect issues close to real-time, through monitoring. The quicker and faster a team is able to detect issues, the more confidence the same team has in making code changes. Teams ensure they can detect issues in a variety of ways. These range from automated checks running on every build, all the way to real-time monitoring of systems.

Mobile teams that ship with confidence tend to invest heavily in monitoring mobile flows; tracking events like crashes and latencies and keeping track of custom business flows and business events.

2. Identify regressions with confidence, through alerting. When a regression is detected in real-time, most teams have an alerting mechanism in place. This could be a PagerDuty alert, an automated chat message, or a ticket created automatically.

Getting alerting right is a difficult but crucial step, and is always done through a series of iterations. Teams tend to start with alerting that is either not granular enough, or is too noisy. Through lots of investment of time and effort, teams that ship with confidence end up with systems that almost always alert when an outage occurs, and rarely fire false positive alerts.

3. Resolve issues swiftly with feature flags. Once a regression is identified, how easy is it to fix quickly? In most cases, this maps to how easy is it to revert the offending change?

For backend and web teams, reverting changes in the codebase can fix an issue in production almost immediately. For mobile, this is only possible if most code changes are rolled out behind feature flags. More mature mobile teams do exactly this; it was also how we operated at Uber. This meant that we could resolve nearly all new regressions in almost real-time, by simply rolling back a feature flag controlling the offending code.

4. Limit the blast radius through dogfooding and staged rollouts. When the damage is done, does it affect all customers, or only a few? If code changes are shipped to all customers right away, this puts additional pressure on the engineering team and they’ll push their changes with less confidence. What if they impact the whole business?

Most software teams use structured rollouts, canaries, and experimentation to ensure that changes are initially rolled out to a limited group of users. When things look good, the rollout is increased. More mature teams automate this process.

Teams that ship with confidence on mobile, typically don’t start the staged rollout with customers. They test out the app with recent changes through dogfooding. Some teams recruit beta users as the next target group for rollout. If there are no regressions in this group, they roll out changes to the customer base.

Shifting Left on Mobile Quality

Let’s take another look at the four traditional strategies to ship confidently on all platforms, including on mobile:

1. Detect issues close to real-time

2. Identify regressions with confidence

3. Resolve issues swiftly

4. Limit the blast radius

Close to real-time detection of all issues that could impact customers has traditionally been rare on mobile. Most teams use static analysis and write automated tests that detect functional issues, validating that the app works correctly. But what about detecting non-functional issues, such as ensuring the app works in an optimal way? Non-functional issues include:

• Performance problems with app startup time, responsiveness, or battery drain
• Overly large binary size, or increased disk usage
• Networking problems
• Memory leaks
• Security issues

Detecting non-functional issues is difficult to do for most mobile teams. Teams who monitor these non-functional app issues tend to do so only after the app is in production. For example, companies like Uber sample the performance of the application running in production and then act on these performance metrics.

Detecting quality issues after shipping to customers will inherently result in a larger blast radius for any regression. The later quality issues are detected, the more customers they impact. Also, because mobile releases are inherently slow, resolving these issues will likely take weeks, at best.

In order to decrease the time to detect non-functional quality issues, a shift left on mobile quality is needed. These non-functional quality issues need to be detected before the app is released to production.

Today, most teams do non-functional testing after the development phase, often when changes are already in production. Non-functional testing is often ad-hoc, like one-off security audits or occasional reviews of the app for memory leaks.

Shifting left on mobile quality means bringing forward quality tests to be as close to the development phase as possible. This might mean bringing some steps to be part of the rollout stage or the dogfooding/beta testing stage. We will have shifted left on mobile quality when these steps are part of the development workflow:

So how can we shift left on mobile quality? How can we flag quality issues to be visible during the development phase?

Shifting left on mobile quality needs a new set of tools. The specialized tools largely do not exist today that integrate into the development workflow and flag non-functional regressions such as performance, networking, memory leaks, security or app size regressions.

mobile.dev is the Shift-Left in Mobile

We started mobile.dev because working at large companies, we have observed the beginnings of the shift left on mobile quality. Mobile-first companies on the cutting edge of the industry are already building their own tools for this. These allow them to bring quality checks closer to when the code is being written. They are forced to invest heavily internally because there are no vendor solutions with these capabilities.

At mobile.dev, we are building the platform that will allow mobile teams to shift left on mobile quality. Our first product helps teams to shift left on mobile performance and is already used by teams at Reddit and other mobile-first companies.

Our mobile performance platform does not just shift left on performance, it introduces a new level of measurement accuracy and determinism. The performance monitoring solutions some companies use in production rely on sampling of production application metrics. Issues surfaced by these tools are often hard to reproduce, and the measurements can be inaccurate. We built our performance regression detection solution for accuracy and debuggability.

Mobile performance is just the start. We’re kicking off with performance because it’s one of the most challenging problems to shift left on, and one that has an immediate impact for companies.

We will continue building platforms and tools that enable engineering teams to detect and resolve quality regressions as part of their development process. Our mission at mobile.dev is to set a new standard for mobile development, which empowers all companies to deliver high-quality mobile applications to their users.

We won’t be done until we enable mobile teams to iterate quickly and confidently on mobile quality, allowing teams to move to this stage:

Shift-Left is the Strategy to Keep Winning in Mobile

The mobile industry is at an inflection point. More and more companies are going all-in on mobile. What was good enough in mobile quality just a few years ago, is already subpar for users today.

Shipping just as fast — or faster — than the competition is becoming table stakes. To ship frequently, without introducing quality regressions, mobile quality needs to shift left, in order to be as close to the engineering phase as possible.

We have first-hand experience of the massive impact an organization can make by shifting left in their quality process. At Reddit, we’ve introduced performance regression detection after every build — instead of the traditional way of measuring performance ad-hoc, when the app has shipped to production. These changes are resulting in better user experiences and translate to better business results.

Shifting left is one of the few strategies that any established mobile team can use to improve their ability to iterate faster and outpace the competition.

Today, some companies are still investing in in-house engineering teams to build custom tooling for shifting left on mobile quality. And while those who invest in building these tools typically are ahead of their competition, building the right tooling is a technically difficult thing. .

We understand how challenging it is to build this tooling because we’ve built similar in-house tools for years. Many companies which start to build their own tools, later backtrack. Either they realize the challenges run deeper than they expected, or they simply cannot justify the cost of building and maintaining these tools in-house.

While building tools to shift left on quality is hard to justify within a single company because of the cost and expertise needed, it makes perfect sense when you serve a whole industry. At mobile.dev we are doing exactly this; building a platform to enable mobile organizations across the industry to shift left.

So what does the future look like for companies that shift left on mobile quality? It’s a world where mobile engineers can ship code quickly and with full confidence. They no longer worry about functional or non-functional regressions. The mobile.dev toolset warns them of unintended regressions and educates them on how and why to avoid or resolve these issues, before their code ships.

World-class tooling is available to any organization that wants it. You no longer have to invest millions in a team of world-class mobile platform engineers. Even with a small team, you can compete on mobile quality with giants like Facebook or Uber.

In this future, amazing and seamless native mobile applications are a concrete reality, not the aspirations they often are today.Gone are apps that crash midway through flows, get stuck on an unresponsive screen, or inexplicably drain the battery.

We’re hiring to build the future: Come build it with us. If you’re an engineer passionate about building the next generation of tooling that world-class teams depend on, get in touch with us. Check out our openings and connect with the authors Leland (LinkedIn, Twitter) and Gergely (LinkedIn, Twitter).