New features including the first AI-powered commands, improved sharding, landscape iOS support, and many bugfixes and stability improvements.

It’s been more than three months since the last Maestro v1.37.0 release. Today we’re happy to present you with v1.39.0.

New AI-powered commands

This release brings the first commands: assertWithAI and assertNoDefectsWithAI.

“What? AI-powered commands? When should I even use them?”. Well, we’re glad you ask!

Maestro’s commands are great: reliable and deterministic. But the real world is not like that, and sometimes it’s really hard, or even impossible, to express some things as conditions.

Take a look at this simple example below. To us, humans, it’s clear that this screen has the problem of overlapping buttons (apart from being real ugly, but let’s not focus on that):

Both buttons are visible and can be tapped on, so a normal command like assertVisible would pass and tapOn would also work. How to catch it? Well, you can just use assertNoDefectsWithAI:

- assertNoDefectsWithAI

It really couldn’t get be any simpler. Now, you can actually be notified that something is clearly wrong.

If you want to perform a specific assertion, then there’s also assertWithAI. Thanks to it, you can really go wild with assertions!

- assertWithAI:
    assertion: A picture of a cute bunny is visible
    optional: true

And guess what — it just works! Take a look:

Check out our docs for configuring AI in Maestro. TL;DR is that you have to bring your own OpenAI (or Anthropic) API key and export it as environment variable.

How we got this idea?

We looked at the issue tracker and currently, the issue with the largest number of 👍 is #1222, whose author asks for a new command to be added to compare screenshots across runs to ensure UI doesn’t break.

Screenshot testing has various names: the native Android/iOS community refers to them as snapshot tests, the Flutter community uses the term golden tests. There’s also baseline tests floating around. We consider them all to be the same concept.

The issue makes sense, and we even started implementing it, but then we asked ourselves: can we try to do something different here?

The main problem with screenshot testing is having to maintain the baseline — the “correct” screenshots that have to be checked into the git repo and maintained. We’ve experienced this pain ourselves, and thought about how we could get rid of the baseline. It wasn’t long until we settled on using LLMs.

What’s next for AI-powered commands?

It’s still early days for Maestro’s integration with AI, but feedback from our early adopters convinced us we’re going in the right direction. very promising. We’re seeing immense potential in this feature and are excited for improving it.

Of course, rest assured — we still want to add support for plain, old, screenshot-based testing. We recognize it’s useful in many scenarios that require predictability — so stay tuned!

Parallel execution improvements

PR #1955 by Tarek Belkahia

In this release, there are 2 new, mutually-exclusive options you can pass to the maestro test command: --shard-all <N> and --shard-split <N>, where N is the number of devices you want to run your tests on.

— shard-split <N> is the same as the now-deprecated --shard <N> option (which will be removed in a future release) — it splits your test suite into chunks and runs each chunk on a different device, speeding up your testing.

The new --shard-all is a bit different. Let’s say you want to make sure that all your flows pass on a small phone, a big phone (does anyone still remember the word “phablet”?), and a tablet. --shard-all lets you do just that — run all the flows on each connected device.

The CLI output for running sharded tests was also improved:

iOS landscape mode

This was a long-standing issue, but no more!

Notable bugfixes

• Allow env vars in setLocation and travel commands (#1988 by Prasanta Biswas)
• Fix using integers from JavaScript outputs causing a deserialization error (#1788 by Muhammed Furkan Boran)
• Fix no tests being run when flowsOrder specified all tests in the workspace (#2003)
• Fix confusing error message “BlockingCoroutine is cancelling” (#2036)

Check out the changelog for a full list.

Upcoming deprecation of Rhino JavaScript engine

The default JavaScript engine in Maestro right now is RhinoJS. It works for simple scripts, but doesn’t implement the modern ECMAScript standard — in fact, it stopped at ES5, which greatly reduces its usefulness.

Fortunately, Maestro also supports GraalJS — a much more capable JavaScript engine that supports ECMAScript 2022. GraalJS has been available in Maestro since version 1.30.0, released in June 2023. Going forward, we will make GraalJS the default JavaScript engine, and then remove RhinoJS.

Check out this issue to learn how we will move ahead with this change. We advise switching to GraalJS as soon as possible — and if you find any regressions in the process, please create an issue and let us know!

Wrapping up

And that’s a wrap for this release!

We’d like to thank all members of our community who tested the new changes, and created issues with feature requests and bug reports.

Special thanks goes to Tarek Belkahia— a long-time member of Maestro community and an avid open-source contributor. Most of bugfixes and new features in this release are the result of his job. Thanks a ton, Tarek!