Methodology
We test iOS and Android apps in two ways. In static analysis, we look at an app's code without running it. In dynamic analysis, we run the app on a test phone and record what it sends over the internet. Static analysis shows what an app contains. Dynamic analysis shows what it sent. On this page we keep the two apart.
We use four methods:
| Method | What it shows | What it leaves open |
|---|---|---|
| Rule scan | A known piece of code, setting or text is present in the app's files | Whether that code runs, and what it sends |
| Deep scan or question | An AI model's reading of the code, with each finding linked to a file | Whether anything was sent. The model can also be wrong |
| Network capture | The requests the app sent while it ran on a real Android or iOS test phone, and the content of each request | Code that stayed idle during the test |
| Teardown | A study of one app, based on its decompiled code (code turned back into a form people can read) and its recorded traffic | Other apps, and other versions of the same app |
A finding is a technical observation, not legal advice. The limits of each method are set out under Limits.
Code in the app versus data sent
Rule scans and Deep scans examine what is in the code. For example, a scan can find a tracking library inside an app. A library, also called an SDK (software development kit), is a package of ready-made code from another company. A match is evidence that the app includes the library. Whether the library sent anything is shown by recorded traffic.
A network capture is a recording of what the app sent. In a capture we can see, for example, that the test phone's advertising ID (a code that ad companies use to recognize a phone) was sent to an ad company's server. We can also see when it was sent, and whether that was before or after the app's consent screen was accepted.
Our scanner rejects any rule whose title or statement uses the words "sent", "transmits" or "shares with". The one exception is a rule marked as an inference: a conclusion drawn from the code, rather than a direct match. We keep those words for what recorded traffic shows.
Rule scans
In a rule scan, we check an app's files against a fixed set of rules. A rule is a short file that describes one thing to look for: a library, a permission (access the app asks for, such as the camera), a setting, a secret key written into the code, or a line of text in the store listing.
- Android. The scanner reads the app's package file (APK) directly. It reads the compiled code (class names, method calls and text strings) and the manifest (the file that lists the app's permissions and parts). It also reads the resources, the other files and the native libraries (code built for the phone's processor). The store listing is read too.
- iOS. The scanner reads the app's program files (text strings, class names, symbols and linked libraries) and its JavaScript bundles. It also reads three settings files: Info.plist (the app's main settings file), the entitlements (the system features the app is allowed to use) and the privacy manifest (the file Apple asks apps to include to declare data use).
The rules are matched against the app's files without running the app. With the same app file and the same rules, the findings are always the same. Each rule file is fingerprinted, and a scan runs only when every rule file matches its recorded fingerprint.
With each finding we save the rule, the version of that rule, the file in the app where the match was found, and the matching text. We also save the line number when it is known. When the same thing is found in several places, we report one finding and keep every location.
Each rule has a severity level, set by hand when the rule is written: info, low, medium, high or critical. Info is used mostly for rules that show something is present, such as a library. Higher levels are used when the thing found is itself a problem. For example, a private key in PEM format (a common text format for keys) in the app's files is high. An Amazon Web Services key pair found in the same app as the web address of an Amazon S3 storage bucket is critical.
Deep scans and questions
In a Deep scan, we decompile the app: we turn its compiled code back into code a person can read. Then we give the code, the manifest and the resources to an AI model. The model writes each finding with the name of the file it came from. In the scan results, these are labeled "Model findings" and kept apart from rule findings.
Before a Deep scan's score is set, a finding that matches a known false positive (a pattern of finding known to be wrong) can be set aside or given a lower severity. Findings of a real secret (such as a key in the code) or a real way around the app's security are exempt from this step. Some kinds of finding have a maximum severity. A second AI check of the cited code can set aside a finding judged, with high confidence, to be a false positive.
A question is a plain-language question about one app, such as "Does this app read the clipboard?". An AI model searches the decompiled code to answer it.
Before anyone sees an answer to a question, it is checked:
- Every finding must cite a file that exists in the app's code. The quoted text must be in that file, and the cited line must exist.
- An answer that says code "sent", "shared", "transmitted" or "uploaded" something is rejected.
- If the answer fails, the run is marked as failed and the answer stays hidden. The reasons are saved.
In a second AI review pass, a model marks each finding as confirmed, a false positive, inconclusive (the evidence is unclear), or "needs runtime" (it needs a test on a running phone). The reason is saved with the mark. The same code checks apply to this pass. A finding marked as a false positive stays visible with that mark.
An AI finding is a lead until someone checks it against the file it cites.
Ultra-deep is a longer Deep scan with more steps. First, it lists what is in the app. Next, it follows the code from the app's starting points to find which parts can run. Then each finding is investigated, and a separate step tries to show it is wrong before it is scored. When too little of the app could be read, the result is NOT RATED, with no score.
Network captures
For a network capture, we run the app on one of our real Android or iOS test phones and record its traffic (the data it sends and receives over the internet). A session lasts from 60 seconds to 7 days and is recorded in segments of 5 minutes to 6 hours; one session can cover a group of up to 20 apps.
- The test phones have full system access, so our own security certificate can be installed on the phones. With that certificate in place, we can decrypt (unscramble) the app's encrypted HTTPS traffic and read it.
- The app is operated automatically by a navigator, an AI model that reads the screen and taps. Each tap, swipe and typed value is recorded as an event with its source. Nothing is bought.
- When the navigator is stuck, it asks for help, and a researcher can take over the phone live from the browser. Each input from a researcher is recorded with the account that made it.
Each session runs under conditions the researcher sets:
- the exit: the country or US state the phone's traffic is sent from, or the researcher's own proxy (HTTP or SOCKS5). The exit's location is checked against ipinfo.io and the regional internet registry's record of the address.
- the persona: an age from 5 to 99, the device language and a GPS location
- tracking allowed or refused at the system prompt
- on some Android phones, a simulated Wi-Fi network and mobile carrier
- for sign-up flows, a lab mailbox (an email inbox we run for test accounts)
Before each session, we read values that identify the phone: its advertising ID, its device IDs, its location, and the internet (IP) address its traffic is sent from. We search the recorded traffic for those values. When one is found, we can see which server it was sent to. That is how we can state that a specific value, such as the phone's location, was sent to a specific server.
For each session we keep a video of the screen, each request and response, and a traffic file in the standard HAR format (HTTP Archive, a file type that browsers and testing tools use to save web traffic). The video, the taps and the requests are on one clock, so each request can be placed against the screen shown when it was sent. Requests are split into those sent before the app's consent screen was accepted and those sent after. Ad responses are matched to the ad shown on the screen. Request bodies of some SDKs are encoded in a format private to the SDK; bodies from SDKs such as Unity Ads, Vungle, Firebase, AppsFlyer and Google Mobile Ads are decoded so the fields can be read.
The sessions of a project can be searched together by host, path, header or body value, with filters by app and session, and the results exported as CSV.
Captures are counted in device hours. A device hour is one hour of one phone recording a session, counted by the minute from the moment the session is ready. Prices per device hour are on the Pricing page.
Captures have limits:
- A connection that cannot be decrypted is recorded as failed, with its server address and without its content. Certificate pinning is one possible cause: the app accepts only its own certificate, so ours is refused. A failed connection in the recording has no cause attached.
- Traffic to some phone system services and some sign-in pages is passed through without decryption, so the phone and the sign-in keep working. That traffic is outside the recording.
- An ad can be tapped by mistake during a session. Such a tap is logged as an accidental click.
- A session records the screens the app showed during that run. Data from parts of the app that stayed idle is outside the recording.
- A value absent from a capture was absent from that session's traffic.
Teardowns
A teardown is an in-depth study of one app, published under Filip Luchianenco's name. It combines the decompiled code and the recorded traffic. For some teardowns, the traffic was recorded by hand, apart from the automated captures; for the Temu teardown, it was recorded on an iPhone. The method of each teardown is described near its top. All teardowns are listed at /teardowns/, each with its publication date. The publishing policy sets out how each piece is checked.
Commissioned reports
Commissioned work (a question sweep or a wide-net investigation, see Work with us) is delivered as reports per app, per company or per theme. Each finding in these reports is marked with one of three status words:
- Established. The finding is in the recorded request or in the cited code.
- Probable. The evidence is consistent with the finding, and one link is missing; the report names that link.
- Open. The finding is undecided on the evidence so far.
Each finding is linked to its evidence: the code file and line, or the recorded request, with screenshots from the session video.
iOS and Android
| Android | iOS | |
|---|---|---|
| Rule scan | Yes | Yes |
| Deep scan and questions | Yes | Yes |
| Automated network capture | Yes | Yes |
Apps from Apple's App Store are encrypted. To read one, we decrypt it on a test iPhone with full system access. We scan free apps only: we buy no apps, on either platform.
Our rule pack (the full set of rules) has 761 Android rules and 72 iOS rules. An Android scan checks for more things than an iOS scan.
Trust checks and red flags
After a Deep scan, an AI model reviews the app against six trust checks. For each check, the model uses the scan's findings and the decompiled app. That includes the manifest, the privacy manifest and any privacy text inside the app. Each check ends as met, not met, or open. A check is met when the evidence contains no concrete sign of a problem, so "met" describes the evidence we had. A check is left open when the evidence is mixed.
- Secure by design. Not met when there is any critical or high security finding. Examples: a secret key or token in the code, an encrypted connection that is set up wrong, weak encryption, SQL injection or path traversal (flaws that let an attacker reach data or files the app should keep apart), or a web page in the app that can call the app's own functions.
- Data minimization. Covers when tracking starts: tracking libraries started before the user agrees, the presence of a way to opt out, and how the advertising ID is handled.
- Manifest mismatch. Covers whether the app's own disclosures match what the code does. The disclosures are its privacy text, its privacy manifest and the data types it declares.
- User control. Covers whether the user can download a copy of the account's data and delete the account. An app that has no accounts and keeps its data on the phone meets this check.
- Truly local. Applies only to apps that say they work offline or keep data on the phone. Not met when sensitive data is sent from the phone to any server outside the user's control, including the app maker's own. Sensitive data here means health, money, biometrics (such as fingerprints or face data), precise location, contacts, messages or photos. Standard analytics IDs and the advertising ID are allowed.
- Child-safe. Applies only to apps presented as made for children. Covers ad code, analytics and any collection of personal data from a young user.
The AI review applies one standard: a check fails, or a red flag is raised, only when the app's own code sets up or uses the behavior. A library or a setting on its own is too little. The review is AI output and can be wrong.
Red flags are seven specific problems. Any one of them makes an app unTRUSTED:
- Bandwidth resold. The app has code that can route other people's internet traffic through the phone's connection.
- Location for sale. The app has code written to send precise location to servers linked to companies that buy and resell location data.
- Clipboard read. The app reads the clipboard when the user has not chosen to paste.
- Always listening. The app can use the microphone while it is in the background (open but off screen).
- False manifest. The app's privacy declarations list less than is found in its code.
- Persistent ID. The app stores an ID that stays on the phone even after the app is deleted and installed again.
- Kids-safety concern. An app marketed to children has tracking, ads or personal data collection aimed at them.
The first six red flags are set to open in two cases: when no finding of medium or higher severity is cited for the flag, and when the app's own main code could not be decompiled. A flag can also be set to open when the same pattern was reviewed before and judged to be no real problem. The kids-safety flag stays as the AI review set it.
Label and score
A completed Deep scan gets a label and a score, shown in the signed-in app (the part of the site behind a login). Every scan an account starts is private, on every plan. A scan can be opened by the account that started it, by the other accounts on its team or project, and by CanITrustThat staff. A free rule scan can be opened only by the account that started it and by CanITrustThat staff.
The label comes from the trust checks and red flags:
| Label | When |
|---|---|
| unTRUSTED | Any red flag, any critical issue, or two or more checks not met |
| NOT ASSESSED | No checks were run, a check is still open, or the result is held for review, and the app is not unTRUSTED |
| TRUSTish | Exactly one check not met, and no red flag |
| TRUSTED | Every check that applied was met, and no red flag |
A critical issue is any finding of critical severity, except findings marked unverified (guesses the model could not back up).
The score is a number from 0 to 100. It is set in three steps.
1. Starting score. The starting score comes from the number of findings at each severity. Findings marked unverified are left out here too. Each severity has a weight: critical 15, high 7, medium 3, low 1, info 0.2. The more findings of one severity, the less each extra one counts. For each severity, the weight is multiplied by ln(1 + count), the natural log of one plus the number of findings. The five results are added up and taken away from 100. The result is rounded, with 0 as the lowest value.
We work out a separate score for two areas. Security covers data security, network security and code security. Privacy covers privacy, permission use and third-party libraries.
Example: privacy has 1 high and 3 medium findings. Its starting score is 100 − 7 × ln 2 − 3 × ln 4 = 91. Security has 1 critical and 2 high findings. Its starting score is 100 − 15 × ln 2 − 7 × ln 3 = 82.
2. AI adjustment. An AI model reviews the findings and moves each area's score up or down. The target range is −20 to +15 points for most apps, and −35 to +15 for medical, financial and identity apps. An adjustment can fall outside that range. If this step fails, the starting scores become the final scores, and step 3 is skipped.
3. Caps for privacy apps. After the AI adjustment, some scores of privacy apps, such as VPNs, proxies and ad blockers, have a maximum. An app counts as a privacy app when its name, package name, description or category contains words such as "VPN" or "privacy". For example, its privacy score is capped at 30 when a privacy finding of medium or higher severity mentions location, and at 35 when a finding mentions advertising or ad networks.
In the overall score, the weaker area counts more: 70% of the lower area score plus 30% of the higher one. With the starting scores above, that is 0.7 × 82 + 0.3 × 91 = 85.
The label and the score are worked out separately, so they can disagree. A single red flag makes an app unTRUSTED even when its score is high.
Rule pack
The rule pack contains 833 rules in 39 families (groups by topic), such as ad libraries, analytics, permissions, encryption, data brokers and children's privacy. 478 rules are tagged with a law or an app store policy. Another 291 are helper rules; the result of each is evidence for a law-tagged rule.
The rules are also grouped into 52 rule sets. Examples are COPPA (the US law on children's privacy online) and Google Play's policy for family apps. In a project (a private group of apps that a customer scans together), each app gets one of five results per rule set: likely non-compliant, possibly non-compliant, likely compliant, not applicable, or not assessed. An app is possibly non-compliant when its build contains a trace, such as an ad library started at launch, that makes it likely non-compliant if one more condition outside the build is true: the user's region, the recorded traffic or the privacy policy text. The result names the trace and the evidence still needed. These results are shown only to the accounts on that project and to CanITrustThat staff. "Likely" and "possibly" mark our reading of the evidence. Whether a law was broken is for a court or a regulator to decide.
Some rules detect the app's framework (the toolkit it was built with), such as React Native or Flutter. They report which parts of the app the scanner could not read.
Limits
Every method here can be wrong in two ways.
False positives (a finding that is wrong). A rule can match text that looks like a tracking library but is unused code. An AI model can misread code. To reduce this, each rule finding includes the matched text and the file, so it can be checked by hand. Answers to questions must quote a real file. In the AI review pass, false positives are marked, with a reason.
Missed findings. A rule scan finds what its rules describe. Code that is hidden, encrypted, or downloaded after the app is installed is outside what a static scan reads. A network capture contains traffic from the code that ran during the session, and some connections cannot be decrypted. To reduce this, we combine code scans with recorded traffic, and each scan lists the parts of the app that could not be read.
Questions these methods leave to others:
- why a company built its app this way
- what a company does with data after it receives it
- whether a law was broken; that is for a court or a regulator
When a scan fails or is partial
A scan can fail for these reasons:
- the store has no app with that ID
- the store lists the app but does not serve the download
- the app is paid
- the app fails to download or decompile
- the scan needs an iOS phone and none is online
- our analysis service is unavailable, or the scan stops between two stages
- the scan takes too long
When a scan fails, we show the reason and whether trying again may help.
When only part of the app could be read, the scan is marked as partial, with this note: "Part of this build was not read. Findings cover only the parts that were."
Reporting an error
Anyone can report an error by email to [email protected]. For a wrong finding, please include the app, its version, the finding, the file it cites, and why it is wrong.
When we correct a teardown or a blog post, the corrected piece shows the date of the correction and what changed. A company can send a reply to a teardown or post; we add the reply to the piece, with its date. The publishing policy sets out how corrections and replies are handled.
Each new scan uses the version of the app that is in the store at that time. Two scans of the same app can be compared to see what changed between versions.
Independence
Findings come from an app's code, store listing and privacy policy, and from its network traffic when we record it; we earn money from paid plans and commissioned work, we accept donations, and no payment changes a finding, a score, a label or a published page. Our rules on money, AI use and corrections are in the publishing policy.