Writing a Swarmbot script with AI: should it find text or match an image?
Tell Swarmbot AI which clue on the phone screen stays reliable: a label, an icon, or both. Check the match before asking the script to tap.
A Swarmbot script can look for text or match a small image on the phone's current screen. If you are asking Swarmbot AI to write that script, you do not have to start with a function name. Start by telling it which clue on the page stays reliable. Open the project and script assistant in Swarmbot's script development view, select one authorized test device, and describe the page and target. Ask for a finding plan before any tap is added.
Choose the clue that identifies the right target
Suppose a card is labelled “Pending,” and that label stays the same on the phones you use. Ask AI to identify the card by text first. If the target has only a stable icon and no useful text, a small crop from the target phone may be more helpful. If both the text and icon change, describe a more reliable page state or leave the step for human confirmation. A guessed screen coordinate is a poor substitute for a clear target.
You could tell the Swarmbot script assistant:
The phone is on the order list. I need the view icon on the “Pending” card. First confirm the card by its text, then look for the icon crop from this script project near that card. Return the text and icon locations for me to check. If the text is absent, there is more than one matching icon, or the two are on different cards, stop and explain why. Do not tap yet. Use only functions in the current Swarmbot script documentation.
Swarmbot's script runtime provides text finding and image finding. Image matching compares a crop with parts of the current screen; OpenCV's template matching guide explains that basic idea. A visually similar icon does not tell the script which business item you meant. When both clues matter, establish the page and label first, then narrow the image search to the relevant area.
Use a crop when the shape is the dependable part
If a control is an icon without a label, or a label changes with the app language, ask AI to use an image resource in the project. Capture a distinctive small area from the target device's current screen. Check the resource name, screen orientation, and display scale. Swarmbot's current image finding does not automatically resize a crop for a different resolution, so check it again after switching phones or changing the interface. The image crop guide shows how to prepare one.
Several identical icons on one page call for more context, not more copies of the same crop. Tell AI which card the icon belongs to, what text is nearby, and where to search. The similar button guide walks through that case.
Ask for a match before an action
Have the assistant write the finding and result checks first, without a tap. Ask what happens when nothing is found, when several results appear, and when a result is outside the area you described. Compare the reported result with the test phone's screen. Only then ask AI to add the tap and a check of the next page.
This separates three jobs: text identifies the section, an image identifies the control, and the current screen confirms the result. An AI message saying “found it” is not, by itself, evidence that the script reached the right place.