Skip to content

scripting.md

Latest commit

 

History

History
577 lines (424 loc) · 51.8 KB

scripting.md

File metadata and controls

577 lines (424 loc) · 51.8 KB
Raw

Writing SandFly scripts

A script is a Lua file that changes how SandFly works in one app or in every app, or adds commands to SandFly. This document describes the whole script API, version 4.

Scripts run on phones and tablets. Watches and TVs can't run them.

What a script is

A script is one of these:

  • A .lua file, up to 1 MB.
  • A .zip file with a main.lua in it, plus anything main.lua uses: other Lua files, sounds, and translations. The zip can be up to 5 MB, up to 5 MB unpacked, and hold up to 200 files.

To install a script, go to Apps > Scripts in SandFly settings and choose Import script, or open or share a script file or an https link to one with SandFly. SandFly asks before downloading from a link another app sends. SandFly lists what the script asks to do before it installs it. Importing a file with the id of a script that's already installed updates that script and keeps its data and settings; SandFly always asks first, and says so if the author changed.

The install dialog starts with where the script came from (a file, a link, or a script source) and whether SandFly signed it. SandFly signs its own scripts, those in aaron-gh/sandfly-scripts, with a key only SandFly has; the signature is the last line of main.lua and covers every file of the script. A signed script shows Signed by SandFly. wherever it came from. A script changed after signing shows that its signature doesn't match. Other scripts show Not signed by SandFly. The author in a manifest is whatever its writer put there, so only the signature shows that a script is SandFly's own.

To find scripts, choose Script sources on the Scripts screen and Add source. A source is the address of a repository, or of a branch or folder in one, on GitHub, GitLab, Codeberg, Gitea (gitea.com) or Bitbucket:

  • github.com/owner/repository or github.com/owner/repository/tree/main/scripts
  • gitlab.com/group/repository or gitlab.com/group/subgroup/repository/-/tree/main/scripts
  • codeberg.org/owner/repository or codeberg.org/owner/repository/src/branch/main/scripts
  • gitea.com/owner/repository, written as for Codeberg
  • bitbucket.org/workspace/repository or bitbucket.org/workspace/repository/src/main/scripts

Only public repositories on these services work. The source's .lua and .zip files are listed. Choosing one downloads it and shows the same install dialog as Import script. Remove source on a source's screen removes it from the list; scripts already installed from it stay installed.

SandFly checks installed scripts for a newer version in the source they came from once a day, and Check for updates on the Scripts screen checks now. With Update scripts automatically on, a newer version is installed without asking when it is by the same author, asks for no new permissions, is made for this SandFly, and comes from the script's own source; scripts installed before SandFly kept their source are updated this way only from SandFly's own scripts. A newer version from SandFly's own scripts installs without asking only if SandFly signed it. Any other newer version is listed under Updates on the Scripts screen, and installs after the usual dialog. A script is found in a source by the file it came from, or, if SandFly doesn't know it, by a file named after its id.

Scripts run in Lua 5.4 through LuaJava. A script has the string, table, math, utf8 and coroutine libraries, os.time, os.clock, os.date and os.difftime, json for JSON text, and the sf table described below. It has no io, debug or package, no access to files, and loads only Lua text, not compiled Lua. Unless it has the java permission, it can't reach Android or Java; everything goes through sf.

A first script

local manifest = {
  id = 'hello',
  name = 'Hello',
  version = '1.0',
  author = 'You',
  minApiVersion = 1,
  testedApiVersion = 2,
  apps = { 'com.android.settings' },
  permissions = { 'speech' },
}

sf.onAppEnter(function()
  sf.speak('Hello from a script')
end)

examples/example-plugin.lua is a longer one. It also comes with SandFly: choose Add the example script on the Scripts screen.

The manifest

main.lua must contain local manifest = { ... } before its code. SandFly reads it without running the script, so it can only hold plain values: text, numbers, true, false and tables. No variables, function calls or ...

A table with only listed values, such as { 'speech', 'screen' }, is a list. A table with only named fields, such as { id = 'x' }, is a record. A table can't have both.

Field Meaning
id Required. Lowercase letters, digits, dots, dashes and underscores, starting with a letter or digit, up to 64 characters. A script with the same id replaces the installed one.
name Required. Shown in the Scripts list, and at the start of the title of every dialog the script shows.
author Required.
minApiVersion Required. The lowest script API version the script works with. This document describes version 4. Use the highest version marked on anything the script uses, such as 3 for a script that uses anything marked Version 3.
testedApiVersion Required. The highest script API version you tested with. SandFly warns before installing a script tested only with a version older than the last one that changed existing calls. Versions 2, 3 and 4 only add to version 1, so scripts tested with 1 get no warning.
version The script's own version, as text, such as '1.2'. The default is '1'. SandFly updates installed scripts from their source when this is higher, compared number by number, so raise it with each change.
description Shown on the script's page and before it's installed.
homepage An http or https address. The script's page has a button that opens it.
onDemand Version 4. true loads the script only to run one of its commands. See Scripts loaded on demand.
apps Where the script runs. Leave it out for a script that runs in every app. See Where a script runs.
permissions A list of the permissions the script needs.
settings Settings the user can change on the script's page.
rules Rules that change items without running any code.
commands Commands the user can run with a gesture, keys, the SandFly menu or reading controls.
navigation Reading controls that move between items of a kind.

Where a script runs

apps is a package name, or a list. Each entry is a package name or a record:

apps = {
  'com.example.mail',
  { package = 'com.example.chat', activity = '.ConversationActivity' },
  { package = 'com.example.chat', window = 'Settings' },
},
  • activity limits the script to one screen. It's the activity's full class name, or the end of it. SandFly learns the activity from the app's window changes. Until it has seen one, such as just after SandFly is updated or the phone restarts with the app open, the script runs in all of the app's screens.
  • window limits it to windows whose title contains the text.

A script with apps is loaded when one of its apps comes to the front and unloaded when it leaves, so anything it keeps in variables is lost then. Use sf.storage for what should last.

A script without apps runs in every app and is loaded as long as it's turned on, unless it's loaded on demand.

Scripts never change SandFly's own screens. Rules, speech changes, gestures, keys, menu items and reading controls from scripts don't apply there, and a script can't name SandFly in apps. While a SandFly screen is in front, calls that need the screen, actions, system or dialogs permission raise an error, sf.speak doesn't interrupt, and scripts get no events from SandFly itself. This is so that Apps > Scripts always works, whatever a script does. A script with the java permission can get past all of this.

Scripts are also unloaded while the screen is off, while SandFly is paused, and while Turn off all scripts is on. Timers and requests end when a script is unloaded.

Permissions

A script can only use what its manifest asks for. The user sees the list before installing, and can turn each permission off on the script's page. A call that needs a permission the script doesn't have raises an error; a hook or rule that needs one is skipped, with a warning in the script's log.

Permission What it allows
screen Reading the screen: sf.screen, the items given to hooks, commands and actions, sf.speech.last(), recognizing text, and the onFocus, onAnnouncement, onTextChange, onContentChange, onScroll, onSelectionChange, onClick, onLongClick and onInputFocus hooks. A script for all apps also needs it to learn which app is in front: sf.app(), onAppEnter, onAppLeave and onWindowChange.
actions Clicking, scrolling, typing and moving focus: item:click(), item:perform(), item:focus(), item:setText() and the other item actions, sf.touch, sf.keys.type, sf.keys.press, and rule actions other than speak.
speech Speaking with sf.speak and sf.message, sf.speech.stop(), sf.braille, rules that change an item, sf.rules, changing what a hook was given to say, and the onSpeech hook.
passwords The text of password fields. Without it, that text is left out.
input Commands, sf.bind, navigation, sf.commands.run, sf.keys.passAll, the onActions hook, and actions in rules.
network sf.fetch and sf.translate.
clipboard sf.clipboard.
system sf.system, sf.resume, sf.device.setVolume and sf.device.setRingerMode.
notifications The onNotification hook.
events The onEvent hook, which sees every accessibility event in every app.
dialogs sf.ui.
java LuaJava's java module. See Java.

SandFly marks passwords, network, clipboard, system, notifications, events, dialogs and java as ones to allow only for trusted scripts.

Java

With the java permission, a script gets LuaJava's java module: java.import, java.new, java.proxy and the rest. It can call any Java class in SandFly, with all of SandFly's access, in every app, and no other permission limits it. Install such a script only from someone you trust.

Anything the sf table doesn't offer needs this permission, such as saving a screenshot to a file or sharing a file with another app.

local Build = java.import('android.os.Build')
sf.speak('This is a ' .. Build.MODEL)

Calls into the script hold the script's Lua state, so a Java object made with java.proxy and called from another thread waits until the script's current call ends. Don't wait in the script for something that needs SandFly's main thread. After the script unloads, calling a method of an object made with java.proxy does nothing and gives nil, 0 or false, so a script should remove anything it registered with Android before it unloads, in onAppLeave for example.

In a script loaded on demand, each object made with java.proxy keeps the script loaded until the first call to one of its methods has finished, or for 30 seconds, for up to 100 objects at once. A listener that stays registered needs a script without onDemand. A table or function passed straight to a Java method doesn't count, so use java.proxy for those.

Rules

A rule changes an item without running code. Rules are the cheapest way to fix an app, and they keep working while the script is busy. Rules that change an item need the speech permission.

rules = {
  { id = 'send_button', label = 'Send' },
  { match = { className = 'ImageView', parentId = 'toolbar' }, hide = true },
  { id = 'row', group = true, hint = 'Double-tap to open' },
},

A rule names its item with id, with match, or with both:

  • id is the item's view id, either the short form (send_button) or the full one (com.example:id/send_button).
  • match is a query.

window and activity limit a rule to windows whose title contains the text, or to an activity whose class name ends with it. A rule with activity applies while the activity isn't known yet, as described for apps.

A rule can set any of these:

Field Effect
label The item's name, used where the app gave none or a poor one.
speak Everything SandFly says for the item, in place of its name, role and state.
hide true hides the item: swiping skips it. 'all' hides what's inside it too. A hidden item is still there, so a script can click it.
silence true stops SandFly speaking the item's changes, selections and announcements, such as a progress bar's updates. 'all' silences what's inside it too, such as the seek bar in a grouped row. Focusing the item still speaks it.
role The item's role, such as 'button'. An unknown role is an error that lists the roles.
state The item's state, such as 'Selected'.
hint The usage hint.
heading true or false.
group true reads the item and everything in it as one item.
readBefore, readAfter Moves the item in the reading order to just before or after another item, named by id or query. Use one, not both.
actions Items to add to the item's Actions menu. See below.
activate A query for the item inside it that a double-tap presses, such as the part of a grouped post that opens it.
liftToActivate true activates the item when the finger lifts from it after touching it, as Lift to activate does, whatever that setting is.
longPressAfter For an item with liftToActivate, a number of milliseconds from 100 to 10,000. Holding the finger on the item for that time long-presses it, and lifting the finger then does nothing more. Items that can't be long-pressed are left alone.
touchToActivate true activates the item by tapping it on the screen instead of sending it an accessibility click, for controls that ignore that click. Needs Android 13 or later and Handle gestures in SandFly; elsewhere the item is clicked as usual.

When several rules match an item, the first one that sets a field wins for that field. Rules from scripts for the app come before rules from scripts for all apps, and sf.rules.add comes before the manifest.

To find an item's id, class and role, assign Inspect focused item to a gesture, or add it to the SandFly menu in SandFly menu, under Controls. Copy screen tree copies every item on screen as JSON.

Actions in rules

{
  id = 'message_row',
  actions = {
    { title = 'Archive', click = 'archive_button' },
    { title = 'Read time', speak = { textOf = 'timestamp' } },
  },
}

Each action has a title and one of click, longClick, focus, scrollForward, scrollBackward or speak. All but speak take the item to act on, as an id or a query, and need the actions permission. speak takes text, or { textOf = item }, which needs screen. Actions in rules need the input permission.

SandFly looks for the item inside the focused item first, and then on the whole screen. In a list where every row has the same button, the second search finds the first row's button, so use the onActions hook there and search from the row yourself.

Queries

A query finds items. It's an id as text, or a record with any of these; an item must match them all.

Field Matches
id The view id, short or full.
text The item's text, exactly.
textContains Part of the item's text or content description, ignoring case.
contentDescription The content description, exactly.
className The end of the class name, so 'Button' matches android.widget.Button.
role The role the app gave the item.
clickable true or false.
parentId The view id of the item's parent.
inside A query that one of the items around it must match: its parent, the parent's parent, and so on. { className = 'Button', inside = { textContains = 'play video' } } matches every button in a video's row, and not the row itself. inside can't hold another inside. Version 2: a list of queries, each of which one of the items around it must match. { clickable = true, inside = { { clickable = true }, { id = 'results' } } } matches every item that can be clicked inside another one in the list results, and not the list's rows themselves.
has A query that one of the items inside it must match, up to 6 levels down and 60 items. { parentId = 'results', has = { textContains = 'Subscribe to' } } matches a row in a list that holds a Subscribe button. has can't hold has or inside.
checkable, checked, longClickable, focusable, focused, selected, enabled, editable, scrollable, visible, heading, password Version 2. true or false. { className = 'CheckBox', checked = false } matches unchecked check boxes.

Items

Hooks, commands and sf.screen give you items. An item is a snapshot: its fields don't change when the screen does. Call item:refresh() for a new one.

Fields: text, contentDescription, hint, stateDescription, id, className, role, packageName, windowId, bounds (left, top, right, bottom), childCount, selectionStart, selectionEnd, checkable, checked, clickable, longClickable, focusable, focused, accessibilityFocused, selected, enabled, editable, password, scrollable, visible, heading, and actions, a list of { id, label }. A field the item doesn't have is nil.

Version 2 adds:

Field Value
roleDescription The role the app names itself, such as a web page's "tab".
error The error text of a field.
tooltip, paneTitle, containerTitle Text.
uniqueId An id that stays the same while the item exists, on Android 13 and later.
multiLine, dismissable, contentInvalid, textSelectable, importantForAccessibility, screenReaderFocusable true or false.
maxTextLength, inputType, drawingOrder Numbers.
liveRegion 'polite', 'assertive' or nil.
range For sliders and progress bars: { min, max, current, type }, where type is 'integer', 'float', 'percent' or 'indeterminate'.
collection For lists, grids and tables: { rows, columns, hierarchical }.
collectionItem For their rows and cells: { row, column, rowSpan, columnSpan, heading, rowTitle, columnTitle }. row and column count from 1.
Method Does
item:parent(), item:children(), item:child(index) The items around it. child(1) is the first.
item:find(query), item:findAll(query, limit) Items inside it. find gives the first, or nil. limit is 50 if left out.
item:refresh() A new snapshot, or nil if the item is gone.
item:click(), item:longClick(), item:scrollForward(), item:scrollBackward() Performs the action, and gives whether it worked.
item:perform(action, args) Performs click, longClick, scrollForward, scrollBackward, expand, collapse, dismiss, copy, cut, paste, select, a custom action by its label, or any action by its number. Version 2 adds clearSelection, focusInput, clearFocus, showOnScreen, scrollUp, scrollDown, scrollLeft, scrollRight, pageUp, pageDown, pageLeft, pageRight, contextClick, imeEnter, pressAndHold, showTooltip, hideTooltip and showTextSuggestions, and these, which take args: setText (text), setSelection (start, end), setProgress (value), scrollToPosition (row, column), nextAtGranularity and previousAtGranularity (unit, extend), and nextHtmlElement and previousHtmlElement (element).
item:setText(text) Replaces the text of an edit field.
item:focus() Moves SandFly's focus to the item.
item:showOnScreen(), item:expand(), item:collapse(), item:dismiss() Version 2. Performs the action.
item:setSelection(start, end) Version 2. Selects text in an edit field, or moves its cursor when end is left out. Positions count from 0.
item:moveCursor(unit, direction, extend) Version 2. Moves the cursor by 'character', 'word', 'line', 'paragraph' or 'page'. direction is 'next' or 'previous'; extend = true selects the text passed over.
item:setProgress(value) Version 2. Sets a slider to a value between its range.min and range.max.
item:scrollTo(row, column) Version 2. Scrolls a list or grid to a row and column, counted from 1.
item:nextHtmlElement(element, direction) Version 2. Moves in web content to the next or previous element of a kind, such as 'HEADING', 'LINK', 'BUTTON', 'LANDMARK' or 'TABLE'.
item:tap(), item:doubleTap(), item:longPress(ms) Version 2. Touches the middle of the item, as a finger would, for items with no click action. Needs actions.
item:labeledBy(), item:labelFor() Version 2. The item that labels this one, or that this one labels, or nil.
item:recognizeText() Version 2. Recognizes text in the item's part of the screen.

a == b is true when two snapshots are the same item, and tostring(item) gives its text, description, id or class.

The actions and focus() wait for the result. Where a call can't wait, such as in a speech hook, they still act, and give nil.

SandFly keeps the last 500 items a script was given. Using an older one raises "That item is no longer available".

Waiting

Some calls wait for an answer: dialogs, sf.fetch, sf.sleep, an item's actions, touches and keys, sf.commands.run, sf.screen.waitFor, recognizing text, sf.translate, sf.speech.last and sf.speak with wait. A script waits only where SandFly runs it as a task: the main file, hooks other than the four speech hooks and onActions, commands, actions, timers, dialog callbacks, and functions given to sf.run. While one task waits, others run. The script's own coroutines can wait too, inside a task.

sf.onCommand('delete', function()
  if sf.ui.confirm('Delete the message?') then
    sf.screen.find('delete_button'):click()
  end
end)

A call that waits raises an error where it can't, such as in onFocus, because those hooks must answer at once. Start a task with sf.run(function() ... end) to do it later.

The sf table

Hooks

Each of these takes a function and returns a function that removes it again.

Hook Called Arguments
sf.onFocus Before SandFly speaks a focused item. The item, and the text SandFly will say.
sf.onSpeech Before SandFly says anything. The text.
sf.onNotification Before SandFly speaks a notification or toast. { package, toast, title, text, category }, and the text.
sf.onAnnouncement Before SandFly speaks an app's announcement. { package, text }, and the text.
sf.onAppEnter, sf.onAppLeave When the app in front changes, or the script loads or unloads. { package, activity, window }.
sf.onWindowChange When a window opens or changes. { package, activity, window }.
sf.onTextChange When text in an edit field changes. { package, node, text, before, from, added, removed, password }.
sf.onContentChange Shortly after part of the screen changes. { package, node, changeTypes }.
sf.onEvent For every accessibility event. { type, package, className, text, contentDescription, beforeText, password, windowId, time }.
sf.onSettingChange When the user changes one of the script's settings. The key and the new value.
sf.onButton When the user presses a button setting. The key.
sf.onActions When SandFly builds the Actions menu for an item. The item. Return a list of { title, run }; run is called with the item, and can wait.
sf.onScroll Version 2. Shortly after a list or page scrolls. { package, node, from, to, count, scrollX, scrollY, deltaX, deltaY }. from and to are the first and last items shown, counted from 1, and count the number of items, where the app gives them.
sf.onSelectionChange Version 2. When the cursor moves or the selection changes in a text. { package, node, text, start, end, length, password }.
sf.onClick, sf.onLongClick Version 2. When an item is clicked or long-clicked, by anyone. { package, node }.
sf.onInputFocus Version 2. When an item gets input focus, such as an edit field the keyboard types into. { package, node }.

The first four can change what's said. Return:

  • Text, or { speak = text }, to say that instead.
  • { before = text, after = text } to add to what's said.
  • false or { silent = true } to say nothing.
  • Nothing, to leave it alone.

Changing speech needs the speech permission. SandFly waits 30 milliseconds for these hooks, then speaks without them, so keep them fast. They can't wait. Speech hooks take priority over queued background work, and all listeners share that deadline, including queue time. A hook cannot interrupt Lua already running. If the script is in the middle of something that takes longer, such as a command, SandFly doesn't wait at all.

Commands

A command is something the user runs. Declare it in the manifest:

commands = {
  { id = 'unread', title = 'Next unread', gesture = 'swipeUpThenRight', keys = 'sf+u' },
  { id = 'summary', title = 'Read summary', menu = true },
  { id = 'messages', title = 'Messages', control = true },
},
  • gesture is a gesture name or a list of them. The names are in Gestures.
  • keys is keys or a list. Keys start with sf, SandFly's modifier key, then any of shift, ctrl, alt and meta, then a key: 'sf+shift+n'. A second key can follow a comma: 'sf+t, s'. Version 2. Keys can leave out sf: 'k', 'shift+n', 'space', 't, s'. They don't apply while an edit field has input focus.
  • menu = true adds the command to the SandFly menu.
  • control = true adds it to the reading controls. Swiping up or down with the control selected runs the command with direction set to 'previous' or 'next'.

Commands need the input permission. The user can turn each gesture and key off on the script's page, or assign others. While a script runs, its gestures and keys come before SandFly's own, except for the gesture assigned to Turn all scripts on or off.

sf.onCommand('unread', function(event)
  -- event.id, event.via ('gesture', 'keys', 'menu', 'control' or 'command'), event.direction, event.node
end)

event.node is the focused item. Return false, before waiting for anything, to have SandFly do what the gesture or keys normally do.

Version 2. For keys without sf, SandFly waits up to 30 milliseconds for the command. If it returns false in that time, before waiting for anything, the keys go to the app. Otherwise they don't, and returning false later does nothing. Holding the keys down runs the command again for each key repeat. Arrow key navigation, while on, takes the arrow keys before scripts.

sf.bind(command, { gesture = name }) or sf.bind(command, { keys = keys }) adds a gesture or keys to a declared command, and sf.unbind removes one.

Version 4. Every command of a script that is turned on is listed in SandFly settings under Commands > Scripts, as the script's name and the command's title. From there the user assigns it to a gesture, a keyboard shortcut, the SandFly menu, a custom menu, a custom reading control or an app's gestures, as with SandFly's own commands, and event.via is 'command'. These assignments are SandFly's and aren't shown on the script's page. A command needs only id and title, so a script can leave every assignment to the user:

commands = {
  { id = 'search', title = 'Search Wikipedia' },
},

If a command runs while its script is off, in an app the script doesn't list, or on a SandFly screen, SandFly speaks "Script not running on this screen".

Scripts loaded on demand

Version 4. With onDemand = true, a script uses no memory until one of its commands runs. SandFly loads it, runs the command, and unloads it about 2 seconds after it has no dialog or window open, no timer set, nothing it is waiting for and no java.proxy object still to be called. examples/wikipedia-search.lua is one.

Variables are lost when the script unloads, so use sf.storage for what should last. Hooks run only while the script is loaded. The manifest can't have apps, rules, navigation or button settings, each command has only id and title, sf.bind and sf.unbind raise an error, and the script's page doesn't list its commands.

Navigation

navigation = { { title = 'Unread messages', match = { className = 'UnreadRow' } } },

Each entry becomes a reading control that moves to the next or previous item matching id or match. It needs the input permission.

Speech, sound and vibration

  • sf.speak(text, { interrupt, rate, pitch, language, wait }). rate and pitch are from 0.25 to 4, where 1 is the user's own setting. language is a tag such as 'de'. With wait = true, it waits until the text is said, and gives true, or false if it was interrupted or not said within 2 minutes. The text is said while media plays, during calls, while another app records from the microphone and during dictation, but not while speech is muted.
  • sf.message(text, { interrupt }) Version 2. Says the text, interrupting by default, and shows it on a connected braille display.
  • sf.speech.stop() Version 2. Stops what SandFly is saying.
  • sf.speech.isSpeaking() Version 2. Whether SandFly is speaking.
  • sf.speech.last() Version 2. Waits, and gives the text of the last thing SandFly said. Needs screen.
  • sf.braille.show(text) Version 2. Shows text on a connected braille display for a while, as SandFly shows an app's announcements. Needs speech.
  • sf.playSound(name) plays one of SandFly's sounds (focus, actionable, click, longClick, scroll, listEnter, listExit, end, enter, typo, beep) or a sound file in the script's zip, such as 'sounds/chime.wav'.
  • sf.playTone(frequency, ms, { volume }) plays a tone. ms is 100 if left out.
  • sf.playAudio(samples, { sampleRate, volume }) plays a list of samples from -1 to 1, up to 10 seconds. sampleRate is from 8,000 to 48,000, and 22,050 if left out.
  • sf.vibrate(pattern) vibrates for a number of milliseconds, or a list of up to 20 that alternates vibration and pause, up to 5 seconds.

Sounds and vibration follow the user's Sound feedback and Vibration feedback settings. A script can play 4 sounds at once; more are skipped.

Screen

  • sf.screen.focused() is the item with SandFly's focus.
  • sf.screen.root() is the top item of the active window.
  • sf.screen.find(query) and sf.screen.findAll(query, limit) search the active window. A search looks at up to 3,000 items, returns up to 200, and stops when the script runs out of time.
  • sf.screen.inputFocused() Version 2 is the item with input focus, such as the edit field the keyboard types into, or nil.
  • sf.screen.at(x, y) Version 2 is the innermost visible item at a point on the screen, in pixels, in the top window there.
  • sf.screen.windows() Version 2 lists the windows on screen, top first: { id, type, title, layer, active, focused, bounds, root }. type is 'application', 'inputMethod', 'system', 'accessibilityOverlay', 'splitScreenDivider' or 'magnificationOverlay', and root is the window's top item. window.root:find(query) searches a window that isn't active, such as the keyboard. SandFly's own windows, such as its menu, are not listed.
  • sf.screen.waitFor(query, timeout, interval) Version 2 waits until an item matching the query is on screen and gives it, or gives nil after timeout milliseconds, 5,000 if left out and at most 60,000. It looks every interval milliseconds, at least 100, and 250 if left out.
  • sf.screen.click(query) Version 2 finds an item and clicks it, or taps it if it has no click action. It gives whether that worked, and needs actions too.
  • sf.screen.recognizeText() Version 2. Recognizes text on the whole screen.
  • sf.app() is { package, label, activity, window, meta } for the app in front. label is the app's name. activity is nil until SandFly has seen the app change window. meta holds the meta-data in the app's manifest that has a true or false, number or text value, such as sf.app().meta['com.example.flag'].

Recognizing text

Version 2. sf.screen.recognizeText() and item:recognizeText() take a screenshot and recognize the Latin-script text in it, on the device. They wait, need screen and Android 11 or later, and give { text, blocks }. Each block is { text, bounds, lines }, and each line { text, bounds }, in screen pixels, so a script can tap a line with sf.touch.tap. A script can recognize text twice at once. Android allows only a few screenshots a second, so SandFly tries again twice when it refuses one, and raises an error after that.

Touch and keys

Version 2. These need the actions permission. They wait for the result where they can, and give whether it worked. Touching the screen needs Android 11 or later. While a script's touch is under way, touches on the screen pass through to apps, so SandFly doesn't take the script's touches as its own gestures.

  • sf.touch.tap(x, y), sf.touch.doubleTap(x, y) and sf.touch.longPress(x, y, ms) touch a point on the screen, in pixels. ms is 1,000 if left out.
  • sf.touch.swipe(x1, y1, x2, y2, ms) moves a finger from one point to the other in ms milliseconds, 300 if left out.
  • sf.touch.gesture(strokes) performs up to 10 strokes at once, for gestures with more fingers. Each stroke is { points, start, duration }: a list of up to 100 { x, y } points, followed in duration milliseconds, starting start milliseconds after the gesture begins. A gesture lasts up to 10 seconds.
  • sf.keys.type(text) types text where the cursor is in the edit field with input focus, replacing any selection. Into a password field, or with no edit field, it types through Shizuku, which takes only text without line breaks, up to 1,000 characters.
  • sf.keys.press(keys) presses keys through Shizuku, one after another. keys is a list, or names separated by spaces: Android's key names with or without KEYCODE_, such as 'enter', 'tab', 'KEYCODE_MEDIA_PLAY_PAUSE', and up, down, left, right, backspace, delete, esc, home, end, pageup, pagedown, playpause, next, previous, volumeup and volumedown.

Keys and typing through Shizuku need Shizuku running, with SandFly allowed in it. Otherwise they raise an error.

Version 3. sf.keys.passAll(on) sends every key to Android and the app in front, as Turn SandFly keyboard input on or off does: the SandFly key, its shortcuts, browse mode keys, arrow key navigation, the review cursor and the keys of scripts, this one's too. Only the keys of Turn SandFly keyboard input on or off still work. sf.keys.passAll() or sf.keys.passAll(true) starts it, and sf.keys.passAll(false) stops it. It needs the input permission.

It lasts while the script is loaded: a script with apps stops passing keys when its app leaves the front, and a script for all apps passes them in every app until it stops. Keys are never passed on SandFly's own screens. The keys of Turn SandFly keyboard input on or off stop it until the script calls sf.keys.passAll again. SandFly says nothing when a script starts or stops passing keys, so a script that does it should tell the user.

-- A game that uses every key.
local manifest = {
  id = 'example.game-keys', name = 'Game keys', author = 'Example',
  minApiVersion = 3, testedApiVersion = 3,
  apps = 'com.example.game',
  permissions = { 'input', 'speech' },
}

sf.keys.passAll()
sf.message('Keys go to the game')

SandFly's commands

Version 2. A script can run any command that a gesture or keys can be assigned to in SandFly, such as moving to the next heading, reading from the top or changing the reading control.

  • sf.commands.list() lists them: { id, title, category }, in the order of Commands in SandFly settings.
  • sf.commands.run(id) runs one, as its gesture would, and gives whether it ran. It needs the input permission, and doesn't work on SandFly's own screens. Commands that pause or silence SandFly, act on the system (Back, Home, recent apps, screenshots, notifications, volume, lock screen, power menu), use the microphone or SandFly Pro also need system; commands that copy, cut, paste or open the clipboard history also need clipboard.
sf.commands.run('NEXT')          -- Next item
sf.commands.run('READ_FROM_TOP') -- Read from top

Turn all scripts on or off, Turn SandFly keyboard input on or off, Open app and app shortcut commands, and the commands of scripts are not in the list.

Rules from code

sf.rules.add(rule) adds a rule and returns a number for sf.rules.remove(number). sf.rules.clear() removes all added rules. A script can add up to 200.

Storage

sf.storage keeps data between loads: get(key, fallback), set(key, value), remove(key), keys() and clear(). Values are text, numbers, true, false and tables of them. A script can store about 512,000 characters. The user can clear it with Clear stored data.

Settings

settings = {
  { key = 'sayApp', type = 'switch', title = 'Say the app', default = true },
  { key = 'voice', type = 'list', title = 'Voice', options = { 'low', { value = 'hi', label = 'High' } } },
  { key = 'name', type = 'text', title = 'Your name', summary = 'Used in greetings' },
  { key = 'count', type = 'number', title = 'How many' },
  { key = 'reset', type = 'button', title = 'Reset counts' },
},

Settings appear on their own screen on the script's page. sf.settings.get(key) reads one, sf.settings.all() reads them all, and sf.settings.set(key, value) changes one. sf.settings.setOptions(key, options) replaces the options of a list. A setting's value can be up to 16,384 characters.

Version 3. sf.settings.setTitle(key, text) and sf.settings.setSummary(key, text) change the title or summary that a setting shows, such as "Slot 1: Rewind, YouTube" on a button, up to 500 characters. nil goes back to the manifest's. A list shows its chosen option instead of a summary.

Dialogs and windows

All of these need the dialogs permission, and all but dialog and window wait. Every dialog's and window's title starts with the script's name, so that it can't be taken for the app's own or the system's.

  • sf.ui.alert(message, { title })
  • sf.ui.confirm(message, { title, ok, cancel }) gives true or false.
  • sf.ui.prompt(title, { text, hint }) gives the text, or nil if cancelled.
  • sf.ui.choose(title, options, { selected }) gives the chosen value, or nil. A list has 1 to 500 options.
  • sf.ui.dialog({ title, items, onClose }) shows a form with up to 50 items, and gives the form at once.
  • sf.ui.window({ title, items, onClose }) Version 2 does the same in a window that fills the screen, for a script that is an app of its own, such as a notepad. Back closes it.
  • sf.ui.text(title, text) Version 2 shows text that can be read by line, word or character and selected, and waits until it's closed. Use it for results too long to say.

Version 2: dialogs and windows open over the ones already open, so a script can ask sf.ui.confirm from a window and the window stays open under it. Up to 8 are open at once; opening another closes the oldest. They close when the script is unloaded, such as when the app the script runs in leaves the screen, so keep what the user typed in sf.storage.

Each form item has a type, an id, and usually a label:

Type Other fields
label Version 2: selectable = true makes the text readable by character, word and line, and selectable.
heading Version 2. A label that is a heading, for moving between parts of a window.
edit value, hint, number, password, multiline, onChange. Version 2: readOnly = true holds text to read and select, not to change; onSubmit is called with the text when the keyboard's Done or Enter key is pressed in a field of one line.
switch value, onChange
list items, value, onChange. One of the items can be chosen.
items Version 2. Rows to open, such as the notes of a notepad: items, a list of values or { value, label }, up to 500; empty, the text shown when there are none; and actions, names of actions on each row, which are in SandFly's Actions menu. onSelect is called with the row's value when it's double-tapped, onLongPress when it's long-pressed, and onAction with the action's name and the row's value.
slider Version 2. min, max, step and value, 0, 100 and 1 if left out; onChange.
button close = true closes the form; onClick

onChange is called with the new value and the form, and onClick with the form. onSelect, onLongPress and onSubmit get the form after the value, and onAction after the action and value. All of them can wait.

The form has form:get(id), form:values(), form:set(id, { value, label, items, enabled, visible }), form:close(), and form:wait(), which waits until the form closes and gives the final values. Version 2 adds:

  • form:add(item, { after }) adds an item at the end, or after the item with the id after, and gives its id.
  • form:remove(id) removes an item.
  • form:focus(id) moves SandFly's focus to an item, or to the first row of an items list.
  • form:setTitle(title) changes the title.

set with items changes the rows of an items list too, and with actions its actions.

examples/notepad.lua is a notepad: a window lists the notes, double-tapping one opens it in a window of its own, and the Actions menu copies or deletes one.

Clipboard, system and network

  • sf.clipboard.get() and sf.clipboard.set(text).
  • sf.system.back(), home(), recents(), notifications(), quickSettings(), powerDialog(), lockScreen(), takeScreenshot() and allApps(). takeScreenshot() takes Android's own screenshot, and the image doesn't reach the script.
  • sf.system.openApp(packageName).
  • sf.system.apps() Version 2 lists the apps with a launcher icon: { package, label }, by name.
  • sf.system.openUrl(url) Version 2 opens an http, https, mailto, tel, sms, smsto, geo or market address in the app for it.
  • sf.system.share(text, { title, subject }) Version 2 opens Android's share sheet with the text.
  • sf.system.openSettings(page, packageName) Version 2 opens a page of Android's settings: settings, wifi, bluetooth, mobileData, airplaneMode, nfc, location, sound, display, battery, batterySaver, accessibility, apps, defaultApps, notifications, doNotDisturb, dateTime, language, keyboard, storage, security, privacy, developer, about, castScreen or hotspot, or, with a package name, app or appNotifications for one app.
  • sf.system.startActivity(intent) and sf.system.sendBroadcast(intent) Version 2 send an intent: { action, data, type, package, className, categories, extras }. extras holds text, numbers, true and false. A broadcast needs package. Neither can reach SandFly itself. startActivity gives whether an app opened it.
  • sf.resume() resumes a paused SandFly. It needs the system permission.
  • sf.fetch(url, { method, headers, body, timeout }) waits for an https address and gives the response. method is GET, POST, PUT, DELETE, HEAD or OPTIONS. A body that isn't text is sent as JSON. The response has status, statusText, ok, url, headers (with lowercase names), body, and response:json(). A request body can be 1 MB and a response 5 MB, the timeout is at most 60 seconds, and a script can have 8 requests going at once.

Device

Version 2. Reading these needs no permission.

  • sf.device.info() is { manufacturer, model, androidVersion, sdk, sandflyVersion, screen }, where screen is { width, height, density, orientation } and orientation is 'portrait' or 'landscape'.
  • sf.device.battery() is { level, charging, full, plugged }. level is a percentage, and plugged is 'ac', 'usb', 'wireless', 'dock' or nil.
  • sf.device.network() is { connected, wifi, cellular, ethernet, vpn, metered }.
  • sf.device.volume(stream) is { level, min, max, muted } for 'media', 'ring', 'notification', 'alarm', 'call', 'system' or 'accessibility'. The stream is 'media' if left out. sf.device.setVolume(stream, level) changes it, and needs system.
  • sf.device.ringerMode() is 'normal', 'vibrate' or 'silent'. sf.device.setRingerMode(mode) changes it, and needs system. Android refuses some changes while Do Not Disturb is on, and the call raises an error then.

Translating text

Version 2. sf.translate(text, { from, to }) waits, and gives { text, from, to }, with the service chosen in SandFly's Translation settings. from and to are language tags such as 'fr'. Without from, the service finds the language; without to, the text is translated into the language chosen there. It needs the network permission, and raises an error if the service isn't set up or fails. With on-device translation, it waits for the languages to download.

Translations

Put a file for each language in the zip's locales folder, named by language tag: locales/de.json, locales/pt-BR.json. Each maps the English text to the translation:

{
  "Next unread": "Nächste ungelesene",
  "{count} message": { "one": "{count} Nachricht", "other": "{count} Nachrichten" }
}

SandFly translates the manifest's names, titles, labels and hints with it. In code:

  • sf._(text, params) translates text and fills in {name} placeholders.
  • sf.ngettext(singular, plural, count, params) picks the plural form for the language.
  • sf.locale() is the device's language tag, and sf.language() the one in use.

Everything else

  • sf.apiVersion is the script API version of this SandFly.
  • sf.manifest() is the script's manifest.
  • print(...) and sf.log(...) write to the script's Log, as do sf.log.debug, sf.log.info, sf.log.warn and sf.log.error. They reach Android's log only at the level set in SandFly's developer settings.
  • sf.setTimeout(fn, ms, ...) and sf.setInterval(fn, ms, ...) run a function later, or again and again, and give a number for sf.clearTimer. sf.sleep(ms) waits. A script can have 100 timers at once.
  • sf.run(fn, ...) runs a function as a task, so that it can wait.
  • sf.clock() Version 2 is a count of milliseconds that only goes up, for timing. os.time() gives the date and time.
  • sf.readFile(path) Version 2 gives the text of a file in the script's zip, such as 'data/words.txt', up to 1 MB, or nil if there is none.
  • json.encode(value) and json.decode(text), also as sf.json. An empty table is encoded as [], and json.null stands for null.
  • require(name) loads another Lua file in the script's zip, once: require('lib.words') loads lib/words.lua.

Gestures

swipeUp, swipeDown, swipeLeft, swipeRight, swipeUpThenDown, swipeDownThenUp, swipeLeftThenRight, swipeRightThenLeft, swipeUpThenLeft, swipeUpThenRight, swipeDownThenLeft, swipeDownThenRight, swipeLeftThenUp, swipeLeftThenDown, swipeRightThenUp, swipeRightThenDown, doubleTap, doubleTapAndHold, twoFingerTap, twoFingerDoubleTap, twoFingerTripleTap, twoFingerTapAndHold, twoFingerDoubleTapAndHold, twoFingerTripleTapAndHold, twoFingerSwipeUp, twoFingerSwipeDown, twoFingerSwipeLeft, twoFingerSwipeRight, twoFingerRotateClockwise, twoFingerRotateCounterclockwise, threeFingerTap, threeFingerDoubleTap, threeFingerTripleTap, threeFingerQuadrupleTap, threeFingerTapAndHold, threeFingerDoubleTapAndHold, threeFingerTripleTapAndHold, threeFingerSwipeUp, threeFingerSwipeDown, threeFingerSwipeLeft, threeFingerSwipeRight, fourFingerTap, fourFingerDoubleTap, fourFingerTripleTap, fourFingerDoubleTapAndHold, fourFingerSwipeUp, fourFingerSwipeDown, fourFingerSwipeLeft, fourFingerSwipeRight, fingerprintSwipeUp, fingerprintSwipeDown, fingerprintSwipeLeft and fingerprintSwipeRight.

SandFly's own gestures, which it recognizes only while a command or a script uses them: edgeSwipeUpFromBottom, edgeLongSwipeUpFromBottom, edgeSwipeDownFromTopLeft, edgeLongSwipeDownFromTopLeft, edgeSwipeDownFromTopRight, edgeLongSwipeDownFromTopRight, edgeSwipeRightFromLeftTop, edgeLongSwipeRightFromLeftTop, edgeSwipeRightFromLeftBottom, edgeLongSwipeRightFromLeftBottom, edgeSwipeLeftFromRightTop, edgeLongSwipeLeftFromRightTop, edgeSwipeLeftFromRightBottom, edgeLongSwipeLeftFromRightBottom, edgeSwipeUpAlongLeft, edgeSwipeDownAlongLeft, edgeSwipeUpAlongRight, edgeSwipeDownAlongRight, edgeSwipeLeftAlongTop, edgeSwipeRightAlongTop, edgeSwipeLeftAlongBottom, edgeSwipeRightAlongBottom, twoFingerPinch, twoFingerSpread, threeFingerPinch, threeFingerSpread, fourFingerPinch, fourFingerSpread, twoFingerScrub, and the swipe-and-hold gestures swipeUpAndHold, swipeDownAndHold, swipeLeftAndHold, swipeRightAndHold, swipeUpThenDownAndHold, swipeDownThenUpAndHold, swipeLeftThenRightAndHold, swipeRightThenLeftAndHold, swipeUpThenLeftAndHold, swipeUpThenRightAndHold, swipeDownThenLeftAndHold, swipeDownThenRightAndHold, swipeLeftThenUpAndHold, swipeLeftThenDownAndHold, swipeRightThenUpAndHold and swipeRightThenDownAndHold. Slides along an edge that is an edge menu go to the menu instead.

Limits

  • A script has 2 seconds to load, and 2 seconds for each call into it. Hooks that change speech share 30 milliseconds. onActions has 250 milliseconds, and its items reach the menu only if it answers within 50. Searching the screen counts toward that time.
  • A script can use 32 MB of memory. string.rep can make text up to 8 MB.
  • SandFly checks the time and memory every 10,000 Lua instructions, and stops the call with an error that pcall can't catch. A single library call, such as a pattern match on long text with several .- or .*, runs to its end, so keep patterns simple on long text.
  • SandFly turns a script off after 5 errors in a minute, or after it runs out of time 3 times in 5 minutes, and says so. SandFly also turns a script off when one call holds up scripts for more than 20 seconds, and starts the other scripts again.
  • A manifest can have up to 500 rules, 100 of them with match, 50 commands and 20 navigation entries.
  • Text for speech and dialogs is cut at 20,000 characters, and braille text at 1,000. Keys in sf.storage and settings are 1 to 200 characters.

If a script goes wrong

  • The script's page has its Log, with what it wrote and every error.
  • Turn off all scripts on the Scripts screen stops every script without changing which ones are on.
  • Turn all scripts on or off does the same from a gesture. Assign it in gesture settings before trying a script you don't trust yet. Scripts can't take over the gesture it's assigned to.
  • Scripts change nothing in SandFly's own settings, so you can always get to the Scripts screen. A script with the java permission can.