1.1 How to write a test script for your game

In this tutorial, we will use a game based on the Unity engine, The big fish is coming as an example to show you how to use our AirtestIDE to quickly write your first game automation test script. It is not as hard as it might seem.

Before we start, we strongly recommend that you read the 5 minutes to get started automation test tutorial provided in the official website, which roughly shows the connection from the mobile phone to writing scripts, running scripts, and generating reports, as well as a detailed introduction to game scripting.

This article may take more than 10 minutes to read carefully (but it will save you a lot of detours!)

Start!

Suppose now that we have AirtestIDE installed on the computer, and connect an Android phone to the computer via USB cable (please refer to the connection method tutorial), and also open the game you want to test: The big fish is coming.

When everything is ready, we will see the following on AirtestIDE:

image

Image recognition based script

First, we will show you how to use image recognition to write scripts, we provide an open source framework Airtest, which is very intuitive, to find out our target elements by image recognition on the current game screen and operate on it.

Airtest touch statement

For example, our first test case is: Click the “go through” button on the screen to enter the level selection screen, then select “First pass”.

This use case can be done using the touch statement. In AirtestIDE you can generate a statement by clicking the touch button in the Airtest helper window and then selecting the button:

image

For this use case, we wrote two lines of code:

touch(image)

touch(image)

Run the code, the effect is that the “pass through” button and the “first pass” button in the game are pressed successively. The principle of the code operation is shown in the figure below:

image

Advanced reading: touch statement in the API documentation

The code of the touch statement is as simple and intuitive as it seems. It looks for the coordinates that match the content of the picture in the current game screen. If found, it will click on the coordinates. If no matching picture is found, an exception will be thrown.

If you want to do more complicated operations, you can get more details in the API documentation of Airtest. For example, to learn more parameters and usage. Since Airtest is cross-platform, all the interfaces are defined in airtest.core.api. You could start in the section airtest.core.api to understand the touch interface

In the interface documentation, the description of the touch statement is: clicks on the screen of the device. It is cross-platform: can be used for Android, Windows and iOS. The common parameters are: the coordinates of the click position: v (or a picture, as airtest will find the position coordinates of the picture on the screen) and the number of clicks: times. There are some other platform-specific parameters.

Since we are using an Android phone to record the script in this example, we will continue to consult the platform-related chapters in the documentation. Find the touch interface in the airtest.core.android.android module. You can see the touch on the android platform interface accepts 2 parameters, one is pos, which is the coordinates of the actual click position, and the other is duration, which can control the length of time we click on the screen. The default is 0.01 second, which is a light touch, but with longer duration it can become a long press.

This is another example of the touch statement. We replaced the image with coordinates and added a parameter duration, so this line of code long presses a coordinate position for one second:

touch((500, 600), duration=1)
If you feel that it is a bit troublesome to consult the API documentation, you can directly hover the mouse over each button in the Airtest Assist Window of AirtestIDE, and you will directly see the parameter information corresponding to this statement. Of course, we also highly recommend reading Airtest source code.

More action statements

In addition to touch, we also provide several other common operation statements, that you can use to achieve more complex operations:

  • swipe, to slide from one location to another
  • wait, to wait for an image to appear in the screen
  • exists,to determine whether there is an image in the screen
  • text,to call the input method and enter a paragraph of text

In addition to the operation statement, we also need to verify that the running result is correct during the process of writing the test script. Airtest provides a dedicated assertion statement for us to verify:

  • assert_exists,to assert that the image exists on the current screen
  • assert_not_exists, to assert that the image does not exist

A complete test case

Now that you know the common statements, you can implement a use case like this with a small piece of code:

  • Open our game to be tested
  • Select game mode
  • Swipe the screen to the far right and select the level you want to play “20th”
  • Verify that you have successfully entered the level 20 in the preparation screen

Sample code:

image

In this code, we show the vast majority of the Airtest statements, which make it easy to write logically complex automated test scripts.

Further reading: improving the success rate of running code

Airtest is not complicated to get started. It can be said that it is easy to get started with a little Python script base. However, as more and more script code is produced, we will find a very serious problem: our script run results sometimes seem to be out of our control. For example, we want to first determine whether there is an icon on the current screen. Only when it exists, the next step should be performed, otherwise, if there is no clear image on the screen, the step should not be performed. However, Airtest still thinks that there is an existing icon. When you open the report, you find that the content of another area on the screen is judged to be the icon we are looking for.

This problem is caused by the operation principle of Airtest, since it uses image recognition technology to find the corresponding image in the current game screen. However, the image recognition system can still not achieve the accuracy of human eye. It can only find, as far as possible, an image that best meets the expected one. This often leads to images that we do not think exist, but Airtest thinks they do, or some icon we think we can find on the screen, Airtest thinks it does not exist.

Therefore, after writing the script, we can make it run a few times and then improve the part with low success rate. This has a few notable improvements:

  • When taking the screenshoots try to ensure that the captured images are highly recognizable and independent. For example, when capturing a button image, try not to bring too much noise into the background in order to avoid problems of recognition when the background changes.
  • In AirtestIDE, you can change the threshold of image recognition by double-clicking the image and modifying the value of threshold. The higher the threshold, the higher requirement of accuracy for image matching. You can refer to [document address](http:/ /airtest.netease.com/docs/docs_AirtestIDE-en_US/2_airtest_script/3_image_editor.html#id3)
  • When recognizing an image, Airtest will first convert the image to grayscale and then recognize it. The result may not be as accurate as expected when judging certain gray icons that become colored when hovering over them. However, we can force the use of color images by double-clicking on the image and checking the RGB option in the settings.
  • Airtest will try to adapt to different resolutions of the phone as much as possible, but in some games there may be custom resolution adaptation rules, which can be customized according to the resolution of the game, the strategy to follow can be found here
  • If there is a large number of repetitive, very similar icons stacked together, it is possible that the recognition is not good. In our eyes, perhaps the text on each icon is different, but in the ‘eyes’ of Airtest they are too similar. We can try to modify the screenshot and modify it to a more recognizable image with some other background styles.
  • We provide a convenient automatic recording function, which can directly convert all current operations into code step by step. Nevertheless, in some cases, the automatically intercepted images are not ideal, and the screenshots need to be manually adjusted.

New use cases and challenges

Airtest satisfies our need to write game automation test scripts quickly and easily. However, its core technology is based on image recognition, and in addition to the lower success rate proposed in the previous chapter, we find it somewhat inadequate to test some of the more complex requirements.

For example, for this use case:

  • Open the mall
  • Upgrade “invincible bubble” item, which will deduct a certain amount of gold coins
  • Verify that the gold coins in the backpack were successfully deducted

image

As shown in the figure, although we can easily identify the gold coin control by taking a screenshot, the values inside are difficult to obtain. Similarly, we can write the touch statement to click the upgrade button (although some tricks are needed to distinguish which one is the “upgrade” button for the “invincible bubble”), it is difficult to verify whether the backpack gold coin is correctly deducted by the “400” gold coin marked in the image.

The good news is that we have another solution to better solve this kind of problem, which is a framework based on control search Poco

Poco based on control recognition

Let’s put aside the use cases mentioned above and take a brief look at the difference between Poco and Airtest.

image

This is a screenshot of the Poco plugin in AirtestIDE. We use Poco to accurately locate the position of the elements on the current game screen in the actual UI tree. For example we can get the name of the “Go” button, coordinates, etc. We can also get this button by writing a certain filter statement, and then click and perform other operations on it.

Suppose we now want to click on the “Go” button in the screen, using Airtest, we will write a statement like this:

touch(image)

In Poco, the statements written are quite different, but simple and elegant:

poco("Go").click()

This is a simplified diagram that compares the operating principles of Airtest and Poco: image

As you can see, the biggest difference between Poco is that there is a Poco-SDK module. We need to embed Poco-SDK into the game under test to get the UI tree smoothly and for the next parsing and processing operations.

How to access Poco

In this chapter, we will show you how to get your game connected to Poco and enjoy its power.

Poco The following modes are currently supported (not just the game engine)

In short, for game testing, if you want to use Poco, you need to connect Poco-SDK to the game you want to test according to the access document. If you have questions about the document, you can invite your project team to help with reading and accessing.

Start the handwriting Poco script

Ok, let’s assume that with the help of the project team, we have already connected the The big fish is coming game to Poco’s Unity3D version of the SDK. The next scripting is very simple:

image

The basic usage process should be:

  • In the Poco Assist window select the corresponding mode, for example here we select Unity
  • The script editing window will prompt if to insert the corresponding initialization statement, click yes
  • The contents of the initialization statement of poco (two lines) will be inserted in the blank script.
  • Wait a few seconds, the UI tree structure of the current game screen will be displayed in the Poco Assist window
  • Click the node on the UI tree to display the node content in the lower log window; double-click the node on the UI tree to automatically insert the code corresponding to the node into the edit box; click the pause button to freeze the current UI tree and view more detailed information easily.
  • After getting the node we need, we can manipulate it, with instructions like .click() and .swipe()

Next, going back to the use case we mentioned before that was difficult to do with Airtest:

  • Open the mall
  • Upgrade “invincible bubble” item, which will deduct a certain amount of gold coins
  • Verify that the gold coins in the backpack were successfully deducted

We can write a sample script in poco like this:

# Please be sure to start the app under test before initializing poco
start_app("fish.package")

# Initialize Poco, very important!
from poco.drivers.unity3d import UnityPoco
poco = UnityPoco()

money = int(poco("Money").child('Text').get_text())

# .get_text() Can help us get its value
cost = int(poco("2024").child("Upgrade").child("NonFullPanel").child("Cost").child("Number").get_text())

# Click the upgrade button for the item under test, please note the end ".click()"
poco("2024").child("Upgrade").child("NonFullPanel").child("UpgradeBtn").click()

current_money = int(poco("Money").child('Text').get_text())

assert_equal(money-cost, current_money, "The gold coin has to be deducted successfully")

For beginners, there are a few things that can go wrong:

  • Poco initialization should be done after the game is started, because poco needs to communicate with the poco-sdk in the game, you need to wait for the game to initialize poco-sdk before you can initialize poco
  • Please be sure to choose the correct Poco mode, first figure out what engine your game is made on, and then perform the corresponding SDK access. After the access is successful, you can start using it.
  • The UI object in poco is just a agent, which needs to be further manipulated (eg .click(), or .exists()) to achieve the desired effect, for example:
# For example, this is a bad way to determine that the if statement with the Go button in the screen is spelled incorrectly
# because Poco("Go") is a UIObjectProxy object
if Poco("Go"):
    Poco("Go").click()
# The correct way to write it is
if Poco("Go").exists():
    Poco("Go").click()

Advanced reading: how to improve Poco scripts

The code before uses the Poco plugin of AirtestIDE to automatically generate the corresponding node selection of the code. This is done by simply double-clicking on the node in the UI tree. Additionally, we provide an automatic recording option. This function can directly record the current mouse operation as a statement. This is very easy for beginners, but, it should be noted that sometimes the automatically generated code may not meet our needs.

For example, in the item selection screen of The big fish is coming, there are multiple nodes with price tags, as shown in the figure:

image

If you use AirtestIDE to select a node directly, the node selection code generated after double-clicking may be like this:

# Gold coin tags for magnetic bubble items
poco(text="600")

At first glance, this line of automatically generated code is concise and clear, and can also fulfill our needs for “get magnetic bubble item corresponding to the value of gold the gold”, but this code directly uses text="600" as the selection criteria. If the value of this node changes, or if there are other items that cost 600 gold coins, the result of this line of code may not meet our expectations, because it can not accurately locate the specific node we want.

We need to have some understanding of the Poco selector, to modify it by writing your own certain conditions.

We suggest:

  • To read our Introduction to Use Cases, that provides demo games and applications and access to Poco - SDK, in order to experience the fastest way to get started with Poco
  • Read the Poco documentation carefully and learn how to use Poco to select UI objects, for mastering the selector which allow us to write a UI selection code that better suits our needs.
  • The documentation of Operation UI Objects describes some of the most common operations, which can be good to get started
  • After getting started, you can read Poco instance API and in more depth Poco proxy object API, as well as other sections of the API documentation

Summary

This article introduces some basic guidelines for writing game test scripts using Airtest and Poco, and answers to some of the most frequently asked questions. For more advanced content, please refer to the official documentation manual.

It’s worth noting that Poco and Airtest are both Python libraries. In essence, the scripts we write are all ordinary Python scripts, so they can be combined to other python libraries to achieve our test needs more accurately and to write more complex and powerful scripts.