Android app testing list rules

Last update : 19/12/2022 (hiatus)

Introduction

Previously, my testings are a bit too inconsistent IMO, with the old rules sprawled below :

• Latest Android branch (A11 at the moment) will be used for testing.
• microG & FakeStore used will be the latest variant possible, with all required (and "required" ones like sms, phone, & contacts) permissions granted.
• By default, the latest version of the app will be tested.
• Due to the random nature of my testings, some entries may be obsolete.
• Unless noted otherwise, Google device registration (GDR) is disabled.
• Any apps with a Git repo and/or F-Droid release (for example, AFWall+ & AdAway) will generally work well regardless of microG / GMS availability.
• For location modules, I will use Mozilla, DejaVu, LocalGSM, & Nominatim.
• Unless noted otherwise, MagiskHide / DenyList is either disabled for most apps, or enabled for some soydev PlayStore apps with internet dependencies like online games & transaction apps. No mention of SafetyNet since there's no way to know if it's the cause of an app not running.

That, combined with the extremely abstract nature of my testings (and its results), results in an app testing list that just sucks from all angles. Not detailed enough as there's no knowing when the results are last tested (meaning obsolete results aren't discovered without me giving the effort), and not enough standards to allow the testing to be reproduced. Well, let's hope this rules fixes it.

Basic Rules

These rules are applied universally.

• Testing will always be done on the latest official version of the app whenever possible. However, as I test things at my own pace, some entries can be obsolete.
• Root access is enabled (using latest official Magisk at the time) by default, as I don't believe any app should give a damn about root access without extremely good reason.
However, should an app be a snowflake regarding root access, Zygisk & DenyList will be enabled for them.


• Aside from Zygisk & DenyList (which are inbuilt starting with Magisk 24.0), no other Magisk modules are allowed for the app testing.
• My hosts file is used for this testing, applied via AdAway. Magisk's systemless hosts aren't enabled.
• Aside from the hosts, captive portal provider changes, & root access; no other changes are allowed for the ROM used for testing. Any debloating is tolerated as long as the bloat is uninstalled / disabled through the GUI, without ADB shell and/or app managers such as SD Maid, Warden, & MyAndroidTools.
• ROMs used for testing are userdebug builds with release keys.
LineageOS & Lineage-microG are my general go-to for these testings; with Lineage-microG receiving an exception for its dev-keys.


• ROMs used for testing must be clean flashed. No dirty flashes allowed.

microG Rules

These rules are added in whenever testing in the microG environment.

• By default (and whenever possible), all testing will be done using the latest microG & FakeStore releases. All required permissions will be granted as well.
• Google device registration (as well as notifications) is disabled
• No usage of Play Store and/or any Google accounts on the device being tested.
• If the app being tested requires geolocation API, I will use DejaVu, Nominatim, & LocalGSM/MergedWiFiNLP.

Template

Here's the template for my testings (and their explanations) :

• App / package name : As it appears in the app information.
• Price / source : Where the app can be obtained, and whether it is paid or not.
• Testing date : The day the app gets tested.
• Tested version : The version of the app being tested.
• Vanilla tests : Tests run in the Vanilla environment, applying my rules above
• microG tests : Tests run in the microG environment, applying my rules above
• Issues / additional notes : Lists issues causing the app to not run, my own personal issues with the app, and/or any noteworthy stuff that don't belong in the results

An example of a (not-so-troublesome) paid app being tested :

• App / package name : BH:Legacy / com.MikaMobile.BattleheartLegacy
• Price / source : Paid (Play Store)
• Testing date : 24/12/2021
• Tested version : 1.5.3
• Vanilla tests : Runs perfectly, no license checking issues
• microG tests : Runs perfectly, no license checking issues
• Issues / additional notes :
• None at the moment

An example of a troublesome paid app being tested :

• App / package name : GRID Autosport / com.feralinteractive.gridas
• Price / source : Paid (Play Store)
• Testing date : 24/12/2021
• Tested version : 1.9.1RC3
• Vanilla tests : Immediately chokes due to license checking
• microG tests : Immediately chokes due to license checking
• Issues / additional notes :
• Enforces a license check which depends on internet & Goolag ID

An example of a free app being tested :

• App / package name : mpv / is.xyz.mpv
• Price / source : Free (GitHub releases / F-Droid)
• Testing date : 24/12/2021 (2021-10-28 build)
• Tested version : 2021-10-28 build (GitHub releases)
• Vanilla tests : Runs perfectly
• microG tests : Runs perfectly
• Issues / additional notes :
• While there are no issues running the app as a video viewer, its file browser leaves much to be desired.

An example of a troublesome free app being tested :

• App / package name : Fate/Grand Order (English) / com.aniplex.fategrandorder.en
• Price / source : Free (Play Store (APKPure))
• Testing date : 24/12/2021
• Tested version : 2.24.0
• Vanilla tests : Does not run (root / root manager / USB debugging detected) (A12-Graphene) / App refuses to download resources (root / root manager / USB debugging not detected)
• microG tests : Does not run (root / root manager / USB debugging detected) / App refuses to download resources (root / root manager / USB debugging not detected)
• Issues / additional notes :
• Refuses to run if root, root manager, and/or USB debugging is detected
• Uses Play Asset Delivery, essentially making it unable to run on anything without GApps & Goolag ID.
• Online-only game

Further notes

• Currently, the devices for testing aren't finalized yet. I am planning to have a similar setup for both Vanilla & microG where both are equally rooted & has my hosts file applied.
• To-do list :
• Find ROM with release keys, userdebug build, & prebuilt microG (failed candidates currently are /e/, Lineage-microG, CalyxOS)

Android app testing list

Main Page