Threat Modeling and Security Governance for On-Device AI in Regulated Enterprise Mobile Apps (2026)

Securing AI on mobile devices in regulated enterprises requires a governance framework built specifically for on-device inference. Cloud-AI security guidance covers API authentication, prompt injection at the network layer, and data-in-transit controls. None of those controls apply when a CoreML .mlmodel, TensorFlow Lite flatbuffer, or GGUF-format small language model runs entirely on the device. The attack surface shifts from the network to the binary itself, and most enterprise security teams are still using cloud-AI governance templates that miss this entirely.

Why Does On-Device AI Create a Distinct Threat Surface That Cloud-AI Governance Misses?

On-device deployment breaks every assumption that cloud-AI security guidance is built on. The model binary is physically present on attacker-controlled hardware, there is no server-side rate limiting on inference calls, and the model itself may be a regulated data artifact subject to HIPAA and FINRA rules. Cloud governance templates address none of these three conditions.

Three differences define the gap. First, a user who jailbreaks an iPhone or roots an Android device has direct filesystem access to the app bundle. Second, an attacker can run millions of queries against a local model with no throttling, no logging, and no detection. Third, a model trained on regulated data carries PHI and PII risk as a file, not just as a runtime process.

Why Is the Model a Data Artifact, Not Just Code?

A trained ML model is a compressed representation of its training data. Fredrikson et al. demonstrated in their 2015 CCS paper "Model Inversion Attacks that Exploit Confidence Information" that model inversion attacks can reconstruct recognizable training samples from model weights alone. Subsequent work by Carlini et al. (2021, "Extracting Training Data from Large Language Models") confirmed the same risk applies to large language models. If a fraud-detection model was trained on real transaction records, or a clinical NLP model was trained on patient notes, the model file carries PHI and PII risk even if no raw data is stored on the device.

This reframes the compliance question. The model is not just an application asset. It is a regulated data artifact that requires the same classification, encryption, access control, and disposal treatment as the underlying records.

How Does Standard Tooling Expose Model Files?

CoreML stores models as .mlmodel or .mlpackage bundles inside the app's Frameworks or Resources directory. TensorFlow Lite uses a flatbuffer-serialized .tflite file, typically in assets/. Small language models in GGUF or GGML format are single binary files, often several gigabytes, stored in the app's document directory or a custom path.

All three formats are readable by open-source tools without any special capability. netron visualizes the full model architecture and layer weights from any of these formats. frida can hook CoreML's MLModel initializer at runtime to dump the decrypted model from memory on a jailbroken device. llm-tools and llama.cpp can load and query GGUF models directly once extracted. An attacker with physical device access and a jailbreak needs no proprietary tooling.

How Does the Four-Category Threat Model Cover On-Device AI Risk?

Every CISO approving an on-device AI feature needs a threat model that goes beyond STRIDE's original scope. The four categories below map to the specific attack patterns that on-device deployment introduces.

Threat Category	Likelihood	Regulated Impact	Mitigation Priority
Model Extraction (static + query-based)	High	IP loss, potential PHI exposure via inversion	Critical
Adversarial Input Manipulation	Medium	Misclassification in clinical/fraud contexts	High
Membership Inference	Medium	Direct HIPAA/FINRA exposure	High
Supply Chain: OTA Model Substitution	Low-Medium	Malicious inference, data exfiltration	Critical

Category 1: Model Extraction. Static extraction means copying the model file from the app bundle or device storage. On an unencrypted build, this requires only ipainstaller or adb pull. Query-based model stealing works even on encrypted models: an attacker runs thousands of crafted inference calls and uses the outputs to reconstruct approximate weights. On-device deployment makes static extraction trivially easy without proper encryption, and query-based attacks are harder to detect because there is no server log.

Category 2: Adversarial Input Manipulation. Adversarial inputs cause misclassification in contexts where the cost of a wrong answer is high. A clinical image classifier that misses a malignancy because of a pixel-level perturbation, or a fraud-detection model that approves a transaction because of a crafted input feature vector, are both Category 2 failures. On-device small language models face an additional risk: prompt injection. Without a server-side content filter, a malicious input string can override system prompt instructions directly in the model's context window.

Category 3: Membership Inference. An attacker who obtains the model file can probe it to determine whether a specific patient record or financial transaction was in the training set. The attack works by comparing the model's confidence on a candidate record against its confidence on synthetic records with similar features. High confidence on the candidate is statistical evidence of training set membership. This is a direct HIPAA and FINRA exposure, and it requires no special hardware.

Category 4: OTA Model Update Supply Chain. Most on-device AI deployments fetch updated model weights from a CDN or object storage bucket. If the download pipeline does not verify a cryptographic signature on the model package before loading it, an adversary who compromises the CDN or performs a DNS hijack can substitute malicious weights. The app loads the substituted model silently. No code change is required. This is the highest-severity low-probability risk in the threat model.

How Should You Implement Encryption and Secure Enclave Protection for Model Files?

Encryption is necessary but not sufficient. The implementation details determine whether the protection holds against a motivated attacker with device access. The priority order is: Secure Enclave key storage first, runtime-only decryption second, and integrity verification on every model load third.

iOS: Secure Enclave Key Storage and Runtime Decryption

Store the AES-256 decryption key in the Secure Enclave using SecKeyCreateRandomKey with kSecAttrTokenIDSecureEnclave. The key never leaves the Secure Enclave in plaintext. At runtime, decrypt the .mlmodel into a temporary in-memory buffer and pass it to MLModel(contentsOf:configuration:) using the in-memory URL. Never write the decrypted model to disk, not even to the app's temporary directory.

CoreML's native model encryption, introduced in iOS 14, protects the model from being read outside the app sandbox. It does not prevent a jailbroken device from dumping the decrypted model from memory using Frida's Memory.readByteArray() after the model is loaded. This is the gap most teams miss. The decrypted model exists in process memory for the duration of the app session, and a Frida script targeting the MLModel class can extract it in seconds on a jailbroken device.

Mitigations for the memory dump gap include: jailbreak detection at model load time (using the iOS App Security Compliance Enterprise 2026 controls as a baseline), encrypting individual layer weights so that a raw memory dump produces an unusable artifact, and splitting the model across multiple files that are assembled in memory only at inference time.

Android: Keystore-Backed Encryption and TFLite Loading

Use EncryptedFile from Jetpack Security with an Android Keystore-backed AES256_GCM key. TensorFlow Lite models must be loaded as MappedByteBuffer. Decrypt the model into a ByteBuffer in memory and pass it directly to Interpreter(byteBuffer, options). Do not write the decrypted bytes to a file in getCacheDir() or getFilesDir().

Android's fragmentation creates an additional complication. Keystore-backed encryption is reliable on devices running Android 9 and above with a hardware-backed keystore. On older devices or devices without a Trusted Execution Environment, the key may be stored in software, which reduces the protection level. The Android App Security Compliance Enterprise 2026 framework covers device attestation checks that confirm hardware-backed key storage before the model is decrypted.

Model Update Integrity: Ed25519 Signature Verification

Every downloaded model update package must include a detached Ed25519 signature. Verify the signature against a pinned public key embedded in the app binary before the model is loaded. If verification fails, reject the update, log the failure with the download URL and a timestamp, and continue running the previously verified model version.

The pinned public key should be rotated on a defined schedule, with the new key distributed through a signed app update rather than a CDN fetch. This prevents an attacker who compromises the CDN from also substituting the verification key. These controls map directly to SOC 2 CC6.1 (logical access controls over assets) and HIPAA §164.312(a)(2)(iv) (encryption and decryption of ePHI). Document the mapping explicitly in your control narrative.

What Audit Trails and Model Versioning Evidence Will SOC 2 and FINRA Examiners Request?

SOC 2 and FINRA examiners have begun requesting four specific categories of evidence for on-device AI deployments. Cloud-AI articles skip this section because cloud providers handle logging infrastructure. On-device AI means the enterprise builds its own audit trail from scratch, typically at a cost of 15 to 25 engineering days for initial implementation and 3 to 5 days per quarter for ongoing maintenance.

1. A complete version history of every model deployed to production devices, including the deployment date, the deploying engineer's identity, validation test results, and a named CISO or designated approver sign-off.
2. Evidence that model updates were tested against adversarial input benchmarks before deployment, with documented pass/fail results and an acceptable error rate defined for the specific use case.
3. Inference event logs that prove the model running at the time of a specific business decision was the approved version, not a substituted or corrupted one.
4. Training data provenance records: what datasets, what date ranges, what de-identification method was applied, and a SHA-256 hash of the validation dataset.

What Is the Privacy Paradox in On-Device Audit Logging?

You must log enough to satisfy auditors but cannot log raw inference inputs if those inputs contain PHI or PII. The resolution is a privacy-preserving audit pattern: log the model version hash (SHA-256 of the model binary), the inference timestamp, the device attestation token from Android SafetyNet or Apple DeviceCheck, and the output confidence bucket (high/medium/low, not the raw score). Never log the raw input.

This pattern gives auditors proof that the approved model version was running at a specific time on an attested device, without creating a secondary PHI store in your logging infrastructure.

WORM-Compliant Log Storage

Logs generated on-device must sync to a WORM-compliant backend at next connection. AWS S3 Object Lock in Compliance mode and Azure Immutable Blob Storage both satisfy this requirement. The append-only constraint prevents retroactive log deletion, which is the specific control SOC 2 CC7.2 and FINRA Rule 4370 are looking for when they ask about audit trail integrity.

Model lineage metadata, including the training data provenance record and the formal sign-off linking to your change management system, should be stored in the same WORM backend with a retention period that matches your longest applicable regulatory requirement. For FINRA-regulated firms, that is typically six years.

What Does the 14-Gate CISO Approval Checklist Cover?

This checklist covers five areas: pre-deployment security, compliance verification, operational readiness, ongoing governance, and vendor risk. It attaches directly to a change management ticket. Each item is a binary pass/fail gate, and no item is optional for regulated deployments. Enterprises that engage a specialist firm to implement this framework typically spend $40,000 to $90,000 for the initial build-out, depending on the number of target platforms and regulatory regimes in scope.

Pre-Deployment Security Gates (Items 1-5)

1. Formal threat model completed using the four-category framework above, signed by the security lead and stored in the change management record.
2. Model file encryption verified in both iOS and Android builds via an automated test that attempts to read the model file from outside the app sandbox and confirms the attempt fails.
3. Adversarial robustness test suite executed with documented pass/fail results. The acceptable error rate must be defined for the specific use case before testing begins, not after.
4. Runtime memory dump test performed on a jailbroken iOS device and a rooted Android device to confirm the decrypted model cannot be extracted in a usable form.
5. Model update pipeline integrity verified: Ed25519 signature check confirmed in code review, with a test case that loads a model with an invalid signature and confirms rejection.

Compliance Gates (Items 6-9)

6. Data residency confirmation: a legal memo confirming that on-device inference satisfies data localization requirements for every deployment geography, including EU (GDPR Article 44), California (CCPA), and any applicable sector-specific rules.
7. Training data lineage review: the security and privacy team have reviewed dataset provenance for PHI/PII contamination risk and documented the review outcome.
8. Regulatory counsel has reviewed model outputs for advice-giving or diagnostic language that triggers additional licensing obligations. This applies specifically to FINRA (investment advice) and FDA Software as a Medical Device rules.
9. Model artifact classified in the data inventory under an appropriate retention and disposal policy, with a defined procedure for remote model invalidation if a device is lost or compromised.

Operational Gates (Items 10-12)

10. Incident response runbook written for a model compromise scenario, including a remote model disable capability that can push a signed "null model" update to all affected devices within a defined RTO.
11. Model rollback procedure tested end-to-end with a documented RTO. The rollback must be testable without requiring a full app store release cycle.
12. Monitoring cadence defined for model drift and anomalous inference pattern detection, with an owner assigned and an escalation path documented.

Ongoing Governance (Items 13-14)

13. Quarterly model re-validation scheduled with a named owner. The schedule must be in the team's sprint planning system, not just a document.
14. Third-party model vendor, if applicable, assessed under the vendor risk management program with contractual security requirements covering model provenance, update signing, and breach notification.

Items 6, 8, and 14 are the ones cloud-AI checklists omit. Cloud providers absorb those responsibilities contractually. On-device AI puts them back on the enterprise. The CISO Mobile App Approval Checklist Enterprise 2026 covers the broader mobile app approval process; these 14 items are the AI-specific overlay that sits on top of that baseline.

The teams that ship on-device AI into regulated environments without a compliance incident are the ones who treat the model file as a regulated data artifact from day one, not as a software dependency. The governance overhead is real and finite. A team that completes this checklist once builds the repeatable process that makes every subsequent model deployment faster and lower-risk than the first.