Qa Testing Mobile

QA Mobile Testing

Design and execute reliable, cost-aware mobile testing across iOS and Android (native + cross-platform).

Quick Start

• Fill assets/mobile-test-plan.md to define risk, layers, and gates.
• Fill assets/device-matrix.md from analytics to pick Tier 1/2/3 coverage.
• Use references/framework-comparison.md to choose automation frameworks.
• Use references/flake-management.md to set a flake budget, reruns, and quarantine rules.

Scope

• Define mobile test strategy across iOS and Android.
• Plan device matrix, OS coverage, and risk tiers.
• Choose automation frameworks and CI + device lab setup.
• Address performance, network/offline, backgrounding, and permissions.
• Define pre-release gates, staged rollout, and store readiness checks.

When NOT to Use

• Platform-specific iOS test command details -> qa-testing-ios
• Platform-specific Android test command details -> qa-testing-android

Inputs

• Platforms, supported OS versions, and device targets.
• App type (native, cross-platform, hybrid/webview).
• Critical user flows and risk areas.
• Distribution channels and release cadence.
• Existing test tooling, CI, and device lab access (Firebase Test Lab, BrowserStack, AWS Device Farm).
• Observability and rollout controls (Crashlytics/Sentry, performance/RUM, feature flags, staged rollout).
• Test data strategy (seed/reset, test accounts, environment parity).

Workflow

1. Define quality risks and SLIs (crash-free, ANR, startup time, key flow success). 2. Build a device matrix from analytics; keep PR gates emulator/simulator-first. 3. Choose frameworks (default: XCUITest + Espresso/Compose; add cross-platform only when it reduces total cost). 4. Build test layers: unit, integration/contract, UI smoke, targeted E2E on real devices. 5. Add mobile-specific coverage: permissions, background/foreground, deep links, offline/poor network. 6. Add performance checks (startup, scrolling/jank, memory) and accessibility checks. 7. Set flake budget, rerun limits, quarantine policy, and failure triage (artifacts + reproducibility). 8. Define release gates + store readiness; ship via staged rollout with monitoring + rollback.

Outputs

• Mobile test strategy and device matrix.
• Automation plan and framework selection.
• Test case inventory with priorities.
• Release readiness checklist.
• CI pipeline and reporting plan.

Quality Checks

• Keep UI tests focused on critical flows; keep suites small and fast.
• Separate device specific bugs from logic regressions.
• Track flake rate per test/device; quarantine and fix top offenders.
• Verify permissions, notifications, and background behavior.
• Prefer stable selectors (accessibility IDs/test tags), not localized text.

Templates

• assets/device-matrix.md for OS and device coverage.
• assets/mobile-test-plan.md for test scope and automation.
• assets/release-readiness-checklist.md for release gates.

Resources

• references/framework-comparison.md for choosing between XCUITest, Espresso/Compose, Appium, Detox, Maestro, and Flutter testing.
• references/flake-management.md for flake control guidance.
• references/device-farm-strategies.md for cloud device farm selection and cost optimization.
• references/mobile-performance-testing.md for startup, jank, memory, and battery testing.
• references/cross-platform-test-patterns.md for React Native, Flutter, and KMP testing patterns.
• data/sources.json for curated documentation and device lab links.

Related Skills

• Use qa-testing-ios for iOS specific depth (XCTest, Swift Testing, simctl).
• Use qa-testing-android for Android specific depth (Espresso, Compose Testing, UIAutomator).
• Use qa-testing-playwright for web and webview testing.
• Use software-mobile for mobile architecture guidance.

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Device Matrix

Tier Definitions

• Tier 1: Top 70-80% of user traffic. Must pass before release. Run in CI on every PR.
• Tier 2: Next 15-20% of traffic. Run before major releases. Manual or nightly CI.
• Tier 3: Long tail devices. Exploratory testing only. Monitor crash reports.

Selection Criteria

Use analytics to identify top devices by:

• Active users per device model
• OS version distribution
• Screen size / resolution clusters
• Market-specific devices (regional variants)

Example Matrix

Replace examples with your actual analytics (device model, OS distribution, and regional mix).

OS Version Coverage

Update Cadence

• Review matrix quarterly or after major OS releases.
• Update Tier 1 when analytics show traffic shifts.
• Archive devices dropping below 1% traffic.

Mobile Test Plan

App Overview

• App Name:
• App Type: Native / Cross-platform (React Native, Flutter) / Hybrid WebView
• Platforms: iOS, Android, or both
• Min OS Policy: iOS (typically N-2) / Android (typically N-4), or your product policy

Critical User Flows

Prioritize flows by business impact:

Test Layers

Unit Tests

• Business logic, data transformations, utilities.
• Target: stable coverage on critical logic (coverage % is secondary to risk reduction).
• Run on every commit.

Integration Tests

• API contracts, local database, caching.
• Run on PR merge.

UI / E2E Tests

• Critical flows on Tier 1 devices.
• Smoke suite: 10-15 tests, < 10 min.
• Full regression: 50-100 tests, < 30 min.

Mobile-Specific Scenarios

Automation Stack

CI Integration

# Example: GitHub Actions mobile CI
mobile-tests:
  runs-on: macos-latest
  steps:
    - uses: actions/checkout@v4
    - name: Run iOS tests
      run: xcodebuild test -scheme App -destination 'platform=iOS Simulator,name=iPhone 16' -resultBundlePath TestResults.xcresult
    - name: Upload results
      uses: actions/upload-artifact@v4
      with:
        name: test-results
        path: TestResults.xcresult

# Example: Android instrumentation on emulator
- name: Run Android tests
  run: ./gradlew connectedDebugAndroidTest

Notes:

• Pick a simulator/emulator name that exists in your CI image; treat device names in this template as examples.
• Always upload artifacts needed for triage (iOS .xcresult, Android logcat, screenshots/video when available).

Release Gates

Reporting

• Test results in CI artifacts.
• Crash monitoring: Firebase Crashlytics / Sentry.
• Performance: Firebase Performance, Android Macrobenchmark, iOS XCTest metrics.
• Weekly QA summary with flake rate, runtime, and coverage trends.

Mobile Release Readiness Checklist

Pre-release validation checklist for iOS and Android apps.

Build and Signing

• [ ] Build signed with production certificate
• [ ] App version and build number incremented
• [ ] Bundle ID / package name correct for production
• [ ] Entitlements and capabilities configured correctly
• [ ] ProGuard/R8 obfuscation enabled (Android)
• [ ] Third-party SDK versions reviewed (crash/analytics/ads)

Functional Testing

• [ ] All critical user journeys pass on release build
• [ ] Smoke test suite passes (login, core features, checkout)
• [ ] Deep links and universal links work
• [ ] Push notifications deliver and open correctly
• [ ] In-app purchases complete successfully
• [ ] Analytics events fire correctly

Stability

• [ ] Crash-free rate meets threshold (target: 99.5%+)
• [ ] No new crashes in beta/TestFlight testing
• [ ] Memory usage within acceptable limits
• [ ] No memory leaks detected in critical flows
• [ ] App launch time within target (<2s cold start)
• [ ] Android ANR rate within SLO (if tracked)

Device Coverage

• [ ] Tested on minimum supported OS version
• [ ] Tested on smallest supported screen size
• [ ] Tested on largest supported screen size
• [ ] Tested on tablet if supported
• [ ] Tested on low-end device representative

Network and Offline

• [ ] Offline mode works correctly
• [ ] Poor network (2G/3G) handling verified
• [ ] Network transitions (WiFi to cellular) handled
• [ ] API timeout handling tested
• [ ] Retry logic works for transient failures

Permissions and Privacy

• [ ] All permission prompts display correctly
• [ ] Permission denial handled gracefully
• [ ] Privacy manifest updated (iOS)
• [ ] Required-reason APIs (iOS) documented and justified
• [ ] Data collection disclosure accurate
• [ ] GDPR/CCPA consent flows work

Platform-Specific (iOS)

• [ ] App Transport Security exceptions documented
• [ ] Background modes configured correctly
• [ ] App Tracking Transparency prompt if needed
• [ ] Sign in with Apple compliance (if any third-party login)
• [ ] Widget extensions work if applicable
• [ ] Watch app syncs if applicable

Platform-Specific (Android)

• [ ] Target SDK meets Play Store requirements
• [ ] Adaptive icons display correctly
• [ ] Split APKs / App Bundle configured
• [ ] Battery optimization handling tested
• [ ] Accessibility services compatibility checked
• [ ] Play Console Data safety form reviewed (if applicable)
• [ ] App links verified (Android App Links / Digital Asset Links)

Store Metadata

• [ ] Screenshots updated for new features
• [ ] App description updated
• [ ] What's New / Release Notes drafted
• [ ] Keywords and categories reviewed
• [ ] Privacy policy URL valid
• [ ] Support URL valid
• [ ] App Review / Play review notes include required test credentials (if needed)

Rollout Plan

• [ ] Staged rollout percentage defined (5% -> 20% -> 50% -> 100%)
• [ ] Rollback plan documented
• [ ] Monitoring dashboards ready
• [ ] On-call schedule confirmed
• [ ] Hotfix process documented

Sign-Off

Cross-Platform Mobile Test Patterns

Test patterns and architecture for cross-platform mobile apps: React Native (Detox), Flutter (integration_test), Kotlin Multiplatform, and Appium as a universal layer.

Contents

• Shared Test Layer Architecture
• Platform-Specific Test Isolation
• Detox for React Native
• Flutter Integration Testing
• Kotlin Multiplatform Test Sharing
• Appium Cross-Platform Patterns
• Test Data Sharing Strategies
• Abstraction Layers for Platform Differences
• CI Matrix Configuration
• When to Write Shared vs Platform-Specific Tests
• Migration Guide
• Related Resources

---

Shared Test Layer Architecture

The Cross-Platform Test Pyramid

                    ┌──────────────┐
                    │   E2E/UI     │  Platform-specific (Detox, XCUITest, Espresso)
                    │  (per-plat)  │  or cross-platform (Appium)
                   ─┼──────────────┼─
                  │  Integration   │  Shared logic tests + platform API mocks
                  │   (shared)     │
                ──┼────────────────┼──
              │      Unit Tests      │  Shared business logic (Jest, Dart test, KMP)
              │       (shared)       │
            ──┴──────────────────────┴──

Layer Ownership

---

Platform-Specific Test Isolation

When Platform Tests Diverge

Not all behavior is identical across iOS and Android. Isolate tests for:

ALWAYS PLATFORM-SPECIFIC:
  ✓ Push notification handling (APNs vs FCM)
  ✓ Deep link / Universal Link / App Link resolution
  ✓ Permissions dialogs (different UI per OS)
  ✓ Background task scheduling (BGTaskScheduler vs WorkManager)
  ✓ Biometric authentication (Face ID vs fingerprint)
  ✓ File system paths and storage APIs
  ✓ Platform-specific UI components (UIKit / Jetpack Compose wrappers)

SAFE TO SHARE:
  ✓ Business logic and validation rules
  ✓ API integration (same endpoints)
  ✓ Navigation flows (screen sequence)
  ✓ Form input and data entry
  ✓ Authentication flows (login, logout, token refresh)
  ✓ Search, filter, sort behavior

Test File Organization

tests/
├── shared/                    # Cross-platform tests
│   ├── auth.test.ts           # Login/logout flows
│   ├── search.test.ts         # Search behavior
│   └── checkout.test.ts       # Purchase flow
├── ios/                       # iOS-only tests
│   ├── push-notifications.test.ts
│   ├── face-id.test.ts
│   └── universal-links.test.ts
├── android/                   # Android-only tests
│   ├── push-notifications.test.ts
│   ├── fingerprint.test.ts
│   └── app-links.test.ts
└── helpers/
    ├── platform.ts            # Platform detection + abstraction
    └── fixtures.ts            # Shared test data

---

Detox for React Native

Setup

# Install Detox CLI and dependencies
npm install -D detox @types/detox jest-circus
npx detox init

# Build for testing
npx detox build --configuration ios.sim.debug
npx detox build --configuration android.emu.debug

Configuration

// .detoxrc.js
module.exports = {
  testRunner: {
    args: { $0: 'jest', config: 'e2e/jest.config.js' },
    jest: { setupTimeout: 120000 },
  },
  apps: {
    'ios.debug': {
      type: 'ios.app',
      binaryPath: 'ios/build/Build/Products/Debug-iphonesimulator/MyApp.app',
      build: 'xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build',
    },
    'android.debug': {
      type: 'android.apk',
      binaryPath: 'android/app/build/outputs/apk/debug/app-debug.apk',
      testBinaryPath: 'android/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk',
      build: 'cd android && ./gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug',
    },
  },
  devices: {
    simulator: { type: 'ios.simulator', device: { type: 'iPhone 15' } },
    emulator: { type: 'android.emulator', device: { avdName: 'Pixel_8_API_34' } },
  },
  configurations: {
    'ios.sim.debug': { device: 'simulator', app: 'ios.debug' },
    'android.emu.debug': { device: 'emulator', app: 'android.debug' },
  },
};

Detox Test Patterns

// e2e/auth.test.ts - cross-platform Detox test
import { device, element, by, expect } from 'detox';

describe('Authentication', () => {
  beforeAll(async () => {
    await device.launchApp({ newInstance: true });
  });

  beforeEach(async () => {
    await device.reloadReactNative();
  });

  it('should login with valid credentials', async () => {
    await element(by.id('email-input')).typeText('user@example.com');
    await element(by.id('password-input')).typeText('password123');
    await element(by.id('login-button')).tap();

    await waitFor(element(by.id('dashboard-screen')))
      .toBeVisible()
      .withTimeout(5000);
  });

  it('should show error for invalid credentials', async () => {
    await element(by.id('email-input')).typeText('bad@example.com');
    await element(by.id('password-input')).typeText('wrong');
    await element(by.id('login-button')).tap();

    await expect(element(by.text('Invalid credentials'))).toBeVisible();
  });
});

Detox Flake Reduction

---

Flutter Integration Testing

Setup

# pubspec.yaml
dev_dependencies:
  integration_test:
    sdk: flutter
  flutter_test:
    sdk: flutter

Test Structure

// integration_test/app_test.dart
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:my_app/main.dart' as app;

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('Authentication', () {
    testWidgets('login with valid credentials', (tester) async {
      app.main();
      await tester.pumpAndSettle();

      // Enter credentials
      await tester.enterText(find.byKey(const Key('email-input')), 'user@example.com');
      await tester.enterText(find.byKey(const Key('password-input')), 'password123');
      await tester.tap(find.byKey(const Key('login-button')));
      await tester.pumpAndSettle(const Duration(seconds: 3));

      // Verify dashboard
      expect(find.byKey(const Key('dashboard-screen')), findsOneWidget);
    });

    testWidgets('show error for invalid credentials', (tester) async {
      app.main();
      await tester.pumpAndSettle();

      await tester.enterText(find.byKey(const Key('email-input')), 'bad@example.com');
      await tester.enterText(find.byKey(const Key('password-input')), 'wrong');
      await tester.tap(find.byKey(const Key('login-button')));
      await tester.pumpAndSettle(const Duration(seconds: 3));

      expect(find.text('Invalid credentials'), findsOneWidget);
    });
  });
}

Running on Both Platforms

# iOS simulator
flutter test integration_test/app_test.dart -d "iPhone 15"

# Android emulator
flutter test integration_test/app_test.dart -d emulator-5554

# All integration tests on both platforms (CI)
flutter test integration_test/ -d "iPhone 15"
flutter test integration_test/ -d emulator-5554

Flutter-Specific Patterns

// Custom test utilities for Flutter
extension IntegrationTestHelpers on WidgetTester {
  /// Scroll until widget is visible
  Future<void> scrollUntilVisible(
    Finder finder,
    Finder scrollable, {
    double delta = 300,
    int maxScrolls = 20,
  }) async {
    int scrolls = 0;
    while (finder.evaluate().isEmpty && scrolls < maxScrolls) {
      await drag(scrollable, Offset(0, -delta));
      await pumpAndSettle();
      scrolls++;
    }
  }

  /// Wait for network-dependent widget
  Future<void> waitForWidget(Finder finder, {Duration timeout = const Duration(seconds: 10)}) async {
    final end = DateTime.now().add(timeout);
    while (finder.evaluate().isEmpty && DateTime.now().isBefore(end)) {
      await pump(const Duration(milliseconds: 100));
    }
    expect(finder, findsOneWidget);
  }
}

---

Kotlin Multiplatform Test Sharing

Shared Test Module

// shared/src/commonTest/kotlin/AuthTests.kt
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue

class AuthValidationTest {
    private val validator = AuthValidator()

    @Test
    fun validEmailAccepted() {
        assertTrue(validator.isValidEmail("user@example.com"))
    }

    @Test
    fun invalidEmailRejected() {
        assertTrue(!validator.isValidEmail("not-an-email"))
    }

    @Test
    fun passwordStrengthCalculation() {
        assertEquals(PasswordStrength.WEAK, validator.checkStrength("abc"))
        assertEquals(PasswordStrength.STRONG, validator.checkStrength("C0mpl3x!Pass"))
    }
}

Platform-Specific Test Expectations

// shared/src/iosTest/kotlin/PlatformAuthTest.kt
import kotlin.test.Test
import kotlin.test.assertEquals

class PlatformAuthTest {
    @Test
    fun biometricTypeIsCorrect() {
        val biometric = BiometricHelper()
        // iOS returns FaceID or TouchID
        assertTrue(biometric.availableType in listOf(
            BiometricType.FACE_ID, BiometricType.TOUCH_ID, BiometricType.NONE
        ))
    }
}

// shared/src/androidTest/kotlin/PlatformAuthTest.kt
class PlatformAuthTest {
    @Test
    fun biometricTypeIsCorrect() {
        val biometric = BiometricHelper()
        // Android returns FINGERPRINT or FACE
        assertTrue(biometric.availableType in listOf(
            BiometricType.FINGERPRINT, BiometricType.FACE, BiometricType.NONE
        ))
    }
}

KMP Test Structure

shared/
├── src/
│   ├── commonMain/kotlin/        # Shared production code
│   ├── commonTest/kotlin/        # Shared tests (run on ALL platforms)
│   ├── androidMain/kotlin/       # Android-specific code
│   ├── androidUnitTest/kotlin/   # Android-specific tests
│   ├── iosMain/kotlin/           # iOS-specific code
│   └── iosTest/kotlin/           # iOS-specific tests

---

Appium Cross-Platform Patterns

Page Object Abstraction

// pages/LoginPage.ts - shared interface
export interface LoginPage {
  enterEmail(email: string): Promise<void>;
  enterPassword(password: string): Promise<void>;
  tapLogin(): Promise<void>;
  getErrorMessage(): Promise<string>;
}

// pages/LoginPage.ios.ts
export class IOSLoginPage implements LoginPage {
  constructor(private driver: WebdriverIO.Browser) {}

  async enterEmail(email: string) {
    const el = await this.driver.$('~email-input');
    await el.setValue(email);
  }

  async enterPassword(password: string) {
    const el = await this.driver.$('~password-input');
    await el.setValue(password);
  }

  async tapLogin() {
    const el = await this.driver.$('~login-button');
    await el.click();
  }

  async getErrorMessage() {
    const el = await this.driver.$('~error-message');
    return el.getText();
  }
}

// pages/LoginPage.android.ts
export class AndroidLoginPage implements LoginPage {
  constructor(private driver: WebdriverIO.Browser) {}

  async enterEmail(email: string) {
    const el = await this.driver.$('~email-input');
    await el.setValue(email);
  }
  // ... same interface, platform-specific selectors if needed
}

Platform-Aware Test Runner

// helpers/platform.ts
import { LoginPage } from '../pages/LoginPage';
import { IOSLoginPage } from '../pages/LoginPage.ios';
import { AndroidLoginPage } from '../pages/LoginPage.android';

export function getLoginPage(driver: WebdriverIO.Browser): LoginPage {
  const platformName = driver.capabilities.platformName;
  if (platformName === 'iOS') return new IOSLoginPage(driver);
  return new AndroidLoginPage(driver);
}

// tests/auth.test.ts - shared test, platform-agnostic
describe('Authentication', () => {
  let loginPage: LoginPage;

  before(async () => {
    loginPage = getLoginPage(driver);
  });

  it('should login successfully', async () => {
    await loginPage.enterEmail('user@example.com');
    await loginPage.enterPassword('password123');
    await loginPage.tapLogin();
    // Assert dashboard visible (cross-platform selector)
    await expect(driver.$('~dashboard-screen')).toBeDisplayed();
  });
});

---

Test Data Sharing Strategies

Shared Fixtures

// fixtures/test-data.ts - shared across platforms
export const TestUsers = {
  valid: { email: 'test-user@example.com', password: 'Test1234!' },
  invalid: { email: 'invalid@example.com', password: 'wrong' },
  admin: { email: 'admin@example.com', password: 'Admin1234!' },
} as const;

export const TestProducts = {
  basic: { id: 'prod-001', name: 'Basic Plan', price: 9.99 },
  premium: { id: 'prod-002', name: 'Premium Plan', price: 29.99 },
} as const;

API-Based Test Data Setup

// fixtures/api-setup.ts
export async function seedTestData(baseUrl: string) {
  // Create test user via API (skip UI)
  const response = await fetch(`${baseUrl}/api/test/seed`, {
    method: 'POST',
    headers: { 'X-Test-Key': process.env.TEST_API_KEY! },
    body: JSON.stringify({
      users: [TestUsers.valid, TestUsers.admin],
      products: [TestProducts.basic, TestProducts.premium],
    }),
  });
  return response.json();
}

export async function cleanupTestData(baseUrl: string) {
  await fetch(`${baseUrl}/api/test/cleanup`, {
    method: 'POST',
    headers: { 'X-Test-Key': process.env.TEST_API_KEY! },
  });
}

---

CI Matrix Configuration

GitHub Actions: Same Tests on Both Platforms

# .github/workflows/cross-platform-tests.yml
name: Cross-Platform E2E
on: [push, pull_request]

jobs:
  e2e-tests:
    strategy:
      fail-fast: false
      matrix:
        include:
          - platform: ios
            os: macos-latest
            device: "iPhone 15"
            command: "npx detox test --configuration ios.sim.debug"
          - platform: android
            os: ubuntu-latest
            device: "Pixel_8_API_34"
            command: "npx detox test --configuration android.emu.debug"
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci

      - name: Setup Android emulator
        if: matrix.platform == 'android'
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 34
          target: google_apis
          arch: x86_64
          profile: Pixel 8
          script: ${{ matrix.command }}

      - name: Run iOS tests
        if: matrix.platform == 'ios'
        run: |
          npx detox build --configuration ios.sim.debug
          ${{ matrix.command }}

      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: e2e-artifacts-${{ matrix.platform }}
          path: artifacts/

Flutter CI Matrix

jobs:
  integration-tests:
    strategy:
      fail-fast: false
      matrix:
        include:
          - platform: ios
            os: macos-latest
            device: "iPhone 15 Simulator"
          - platform: android
            os: ubuntu-latest
            device: "emulator-5554"
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: subosito/flutter-action@v2
        with: { flutter-version: '3.x' }
      - run: flutter test integration_test/ -d "${{ matrix.device }}"

---

When to Write Shared vs Platform-Specific Tests

Decision Rule

IF the behavior is:
  - identical on both platforms AND
  - uses the same selectors (testID/key) AND
  - doesn't touch OS-level APIs
THEN write a shared test.

OTHERWISE write platform-specific tests with a shared interface.

---

Migration Guide

From Platform-Native to Cross-Platform Testing

Phase 1: Audit existing tests (1 week)

1. Inventory all native tests (XCUITest, Espresso)
2. Categorize: shared-eligible vs platform-specific
3. Identify testID gaps (add testID/accessibilityIdentifier to shared components)
4. Choose cross-platform framework (Detox for RN, integration_test for Flutter, Appium for native)

Phase 2: Build abstraction layer (1-2 weeks)

1. Create page objects with platform interface
2. Set up shared test data / fixtures
3. Write first 3-5 cross-platform tests (login, navigation, core flow)
4. Validate on both platforms in CI

Phase 3: Migrate incrementally (ongoing)

1. New tests: write cross-platform by default
2. Existing tests: migrate during feature work (boy scout rule)
3. Keep platform-specific tests for OS-level behavior
4. Track migration progress: shared_tests / total_tests

Migration Tracking

Cross-Platform Test Migration - Sprint 12
------------------------------------------
Total E2E tests:        84
Shared (cross-plat):    52 (62%)
iOS-only:               16 (19%)
Android-only:           16 (19%)

Target: 75% shared by Q3

---

Related Resources

• framework-comparison.md -- full framework selection guide
• flake-management.md -- flake patterns specific to cross-platform tests
• device-farm-strategies.md -- running cross-platform suites on cloud devices
• SKILL.md -- parent mobile testing skill
• Detox Documentation
• Flutter Integration Testing
• Kotlin Multiplatform Testing
• Appium Documentation

Cloud Device Farm Strategies

Cloud device farm selection, cost optimization, and CI/CD integration for mobile testing at scale.

Contents

• Provider Comparison
• Pricing Models
• Device Selection Strategy
• Real Device vs Emulator Trade-Offs
• Test Sharding Across Devices
• Parallel Execution Optimization
• On-Premise Device Lab
• CI/CD Integration Patterns
• Cost Tracking and Budgeting
• Free Tier Optimization
• Device Availability and Queue Management
• Decision Checklist
• Related Resources

---

Provider Comparison

Framework Support Matrix

---

Pricing Models

Per-Minute Pricing

Pay only for active test time. Best for variable workloads.

Monthly estimate formula:
  Cost = (avg_test_minutes_per_run) × (runs_per_day) × (working_days) × (rate_per_minute)

Example (BrowserStack):
  15 min × 10 runs × 22 days × $0.20/min = $660/month

Concurrent Device Plans

Fixed number of parallel devices; unlimited minutes. Best for high-volume CI.

Example (AWS Device Farm - Unmetered):
  $250/device-slot/month
  5 slots = $1,250/month (unlimited test minutes)

Break-even vs per-minute:
  If usage > 6,250 min/month at $0.20/min → unmetered wins

Flat-Rate / Enterprise

Annual contracts with volume discounts. Negotiate when spending above $2,000/month.

---

Device Selection Strategy

Coverage vs Cost Matrix

Build device tiers from analytics data:

Tier 1 (Must-test, ~70% users):
  - Top 3-5 devices by active installs
  - Latest + previous OS version
  - Run on every PR

Tier 2 (Should-test, ~20% users):
  - Next 5-8 devices
  - Specific OEM variants (Samsung, Xiaomi, Pixel)
  - Run on merge to main / nightly

Tier 3 (Spot-check, ~10% users):
  - Older devices, budget phones
  - Edge OS versions (oldest supported)
  - Run weekly or pre-release

Device Selection Script

import json
from collections import Counter

def build_device_matrix(analytics_data: list[dict], tiers: dict) -> dict:
    """Build tiered device matrix from analytics data."""
    device_counts = Counter()
    for session in analytics_data:
        key = f"{session['device_model']}|{session['os_version']}"
        device_counts[key] += 1

    total = sum(device_counts.values())
    sorted_devices = device_counts.most_common()

    matrix = {"tier1": [], "tier2": [], "tier3": []}
    cumulative = 0
    for device, count in sorted_devices:
        pct = count / total
        cumulative += pct
        model, os_ver = device.split("|")
        entry = {"model": model, "os_version": os_ver, "user_share": f"{pct:.1%}"}

        if cumulative <= tiers.get("tier1_cutoff", 0.70):
            matrix["tier1"].append(entry)
        elif cumulative <= tiers.get("tier2_cutoff", 0.90):
            matrix["tier2"].append(entry)
        else:
            matrix["tier3"].append(entry)

    return matrix

# Usage
matrix = build_device_matrix(
    analytics_data=load_analytics(),
    tiers={"tier1_cutoff": 0.70, "tier2_cutoff": 0.90}
)
print(json.dumps(matrix, indent=2))

Minimum Viable Matrix

For teams with limited budget, start here:

---

Real Device vs Emulator Trade-Offs

When to Use Each

USE EMULATORS/SIMULATORS FOR:
  ✓ Unit test execution
  ✓ Integration tests
  ✓ Rapid iteration during development
  ✓ PR-level smoke checks
  ✓ Screenshot generation for docs

USE REAL DEVICES FOR:
  ✓ E2E / UI acceptance tests
  ✓ Performance benchmarking
  ✓ Camera / biometric / NFC flows
  ✓ Pre-release validation
  ✓ Network condition testing
  ✓ OEM-specific behavior verification

---

Test Sharding Across Devices

Sharding Strategies

By test suite (recommended for device farms):

# GitHub Actions: matrix strategy for device sharding
jobs:
  mobile-tests:
    strategy:
      fail-fast: false
      matrix:
        include:
          - device: "Google Pixel 8"
            os_version: "14.0"
            shard: "1/3"
          - device: "Samsung Galaxy S24"
            os_version: "14.0"
            shard: "2/3"
          - device: "Google Pixel 6"
            os_version: "13.0"
            shard: "3/3"
    steps:
      - name: Run sharded tests
        run: |
          ./gradlew connectedAndroidTest \
            -Pandroid.testInstrumentationRunnerArguments.numShards=3 \
            -Pandroid.testInstrumentationRunnerArguments.shardIndex=${{ matrix.shard }}

By test tag (functional grouping):

# Run smoke tests on all devices, regression on Tier 1 only
# Smoke (all devices)
./gradlew connectedAndroidTest \
  -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.Smoke

# Regression (Tier 1 only)
./gradlew connectedAndroidTest \
  -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.Regression

---

Parallel Execution Optimization

Optimal Parallelism Formula

Optimal parallel devices = ceil(total_test_minutes / target_wall_clock_minutes)

Example:
  Total suite: 120 test-minutes
  Target wall clock: 15 minutes
  Optimal devices: ceil(120 / 15) = 8 parallel devices

Parallelism Gotchas

---

On-Premise Device Lab

When It Makes Sense

• Testing >4 hours/day of real-device time
• Regulatory requirement for on-prem data processing
• Need devices not available in cloud (specialized hardware, carrier-specific)
• Break-even typically at 15-20 devices used daily

Setup Checklist

• [ ] USB hubs with independent power per port
• [ ] Dedicated Mac Mini / Linux host per 5-8 devices
• [ ] STF (Smartphone Test Farm) or similar orchestration
• [ ] Temperature-controlled environment
• [ ] Automated device health checks (battery, connectivity)
• [ ] Remote access for debugging (scrcpy for Android, Xcode wireless for iOS)
• [ ] Automatic device reboot schedule (daily)

# Smartphone Test Farm (STF) - open source device management
docker run -d --name stf \
  -p 7100:7100 \
  -p 7110:7110 \
  --link adb:adb \
  openstf/stf:latest \
  stf local --public-ip $(hostname -I | awk '{print $1}')

---

CI/CD Integration Patterns

BrowserStack + GitHub Actions

# .github/workflows/mobile-tests.yml
name: Mobile Tests
on: [push, pull_request]

jobs:
  android-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build APK
        run: ./gradlew assembleDebug assembleDebugAndroidTest

      - name: Upload and run on BrowserStack
        env:
          BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
          BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
        run: |
          # Upload app
          APP_URL=$(curl -u "$BROWSERSTACK_USERNAME:$BROWSERSTACK_ACCESS_KEY" \
            -X POST "https://api-cloud.browserstack.com/app-automate/upload" \
            -F "file=@app/build/outputs/apk/debug/app-debug.apk" \
            | jq -r '.app_url')

          # Upload test suite
          TEST_URL=$(curl -u "$BROWSERSTACK_USERNAME:$BROWSERSTACK_ACCESS_KEY" \
            -X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/test-suite" \
            -F "file=@app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk" \
            | jq -r '.test_suite_url')

          # Execute tests
          curl -u "$BROWSERSTACK_USERNAME:$BROWSERSTACK_ACCESS_KEY" \
            -X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
            -d "{\"app\": \"$APP_URL\", \"testSuite\": \"$TEST_URL\", \
                 \"devices\": [\"Google Pixel 8-14.0\", \"Samsung Galaxy S24-14.0\"]}"

Firebase Test Lab + Cloud Build

# cloudbuild.yaml
steps:
  - name: 'gcr.io/cloud-builders/gradle'
    args: ['assembleDebug', 'assembleDebugAndroidTest']

  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - 'firebase'
      - 'test'
      - 'android'
      - 'run'
      - '--type=instrumentation'
      - '--app=app/build/outputs/apk/debug/app-debug.apk'
      - '--test=app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk'
      - '--device=model=oriole,version=33'
      - '--device=model=redfin,version=31'
      - '--timeout=15m'
      - '--results-dir=test-results'

---

Cost Tracking and Budgeting

Monthly Cost Dashboard

def calculate_monthly_cost(
    provider: str,
    pricing_model: str,
    daily_runs: int,
    avg_minutes_per_run: float,
    parallel_devices: int,
    working_days: int = 22
) -> dict:
    """Estimate monthly device farm costs."""
    total_minutes = daily_runs * avg_minutes_per_run * working_days

    rates = {
        "browserstack": {"per_minute": 0.20, "concurrent_monthly": 199},
        "aws_device_farm": {"per_minute": 0.17, "concurrent_monthly": 250},
        "sauce_labs": {"per_minute": 0.16, "concurrent_monthly": 189},
        "lambdatest": {"per_minute": 0.10, "concurrent_monthly": 119},
    }

    rate = rates.get(provider, rates["browserstack"])

    if pricing_model == "per_minute":
        cost = total_minutes * parallel_devices * rate["per_minute"]
    else:
        cost = parallel_devices * rate["concurrent_monthly"]

    return {
        "provider": provider,
        "model": pricing_model,
        "total_minutes": total_minutes * parallel_devices,
        "monthly_cost": f"${cost:,.0f}",
        "cost_per_test_minute": f"${cost / (total_minutes * parallel_devices):.3f}",
    }

Cost Reduction Checklist

• [ ] Run Tier 2/3 devices only on nightly, not every PR
• [ ] Use emulators for unit/integration tests in CI
• [ ] Implement test sharding to reduce wall-clock (but not total minutes)
• [ ] Remove or quarantine flaky tests that waste rerun minutes
• [ ] Use free tiers for open-source or low-volume projects
• [ ] Cache device provisioning where providers support it
• [ ] Right-size parallel device count using timing data

---

Free Tier Optimization

# Firebase Test Lab: free tier usage
gcloud firebase test android run \
  --type=robo \
  --app=app-debug.apk \
  --device=model=Pixel2,version=30 \
  --timeout=120s \
  --results-bucket=gs://my-test-results
# Robo tests count toward free quota; use for exploratory coverage

---

Device Availability and Queue Management

Handling Queue Contention

Problem: Real devices may be in use by other customers.
  - Peak hours: 9am-5pm EST (US teams) and 9am-5pm IST (India teams)
  - Popular devices (iPhone 15, Pixel 8) have longest queues

Mitigation:
  1. Schedule heavy runs off-peak (midnight-6am local)
  2. Use equivalent device alternatives (Pixel 7 instead of Pixel 8)
  3. Set queue timeout + fallback to emulator
  4. Pre-reserve devices for release testing

Timeout and Fallback Pattern

# CI config with device fallback
env:
  DEVICE_TIMEOUT: 120  # seconds to wait for device

steps:
  - name: Run on real device
    id: real_device
    continue-on-error: true
    run: |
      timeout $DEVICE_TIMEOUT run-tests --device="Pixel 8" --real

  - name: Fallback to emulator
    if: steps.real_device.outcome == 'failure'
    run: |
      echo "Real device unavailable, falling back to emulator"
      run-tests --device="Pixel 8" --emulator

---

Decision Checklist

Before selecting a device farm provider:

• [ ] Identified top 10 devices from analytics
• [ ] Calculated monthly test minutes (current and projected)
• [ ] Compared per-minute vs concurrent pricing for your volume
• [ ] Verified framework support (Espresso, XCUITest, Appium, etc.)
• [ ] Tested API and CI integration with a trial run
• [ ] Confirmed data residency and compliance requirements
• [ ] Evaluated queue wait times for target devices
• [ ] Assessed on-prem vs cloud break-even for your scale

---

Related Resources

• framework-comparison.md -- automation framework selection guide
• flake-management.md -- managing flaky tests on device farms
• SKILL.md -- parent mobile testing skill
• BrowserStack App Automate
• Firebase Test Lab
• AWS Device Farm
• Sauce Labs Real Devices
• OpenSTF - Smartphone Test Farm

Mobile Test Flake Management

Strategies for identifying, tracking, and eliminating flaky tests in mobile automation.

Typical UI-suite flake rates can reach 5-30% without active controls; treat "flake budget" as a first-class quality SLI.

Contents

• Flake Detection
• Common Causes and Fixes
• Quarantine Strategy
• Rerun Policies
• Prevention Checklist
• Monitoring Dashboard
• Resources

Flake Detection

Metrics to Track

Artifacts and Reproducibility (REQUIRED)

Capture enough context to reproduce on the same device/OS and compare runs:

• iOS: .xcresult bundle, screenshots, screen recording (if enabled), device logs, simulator/device model + iOS version, test plan/config, app build SHA.
• Android: instrumentation output, logcat, screenshots/video (device farm), device model + API level + OEM build, test runner args, app build SHA.
• Network state: mocked vs real, throttling profile, backend environment, feature flags/experiments state.

Identifying Flaky Tests

# Track test history over time
# Look for tests that pass/fail inconsistently on same code

# Pattern: same test, same code, different results
Test: LoginFlow
Run 1: PASS
Run 2: FAIL (timeout)
Run 3: PASS
Run 4: PASS
Run 5: FAIL (element not found)

Test Analytics (Optional)

Use test analytics to correlate failures with device model/OS, infra, and recent changes.

Examples (non-exhaustive):

• Test analytics/visibility in CI (Datadog, Buildkite/Test Analytics, etc.)
• Device cloud observability (BrowserStack, LambdaTest, etc.)
• Quarantine workflows (Trunk.io, custom tags, etc.)

Common Causes and Fixes

1. Timing Issues (Most Common)

Symptom: Element not found, timeout errors

Bad:

// Fixed sleep - fragile
await driver.sleep(3000);
await driver.findElement(By.id('submit')).click();

Good:

// Explicit wait for condition
await driver.wait(
  until.elementIsVisible(driver.findElement(By.id('submit'))),
  10000
);
await driver.findElement(By.id('submit')).click();

Platform-specific:

// iOS: waitForExistence
let button = app.buttons["submitButton"]
XCTAssertTrue(button.waitForExistence(timeout: 10))
button.tap()

// Android Espresso: IdlingResource
IdlingRegistry.getInstance().register(networkIdlingResource)
onView(withId(R.id.submit)).perform(click())

2. Animation Interference

Symptom: Tap registers on wrong element, scroll fails

Fix: Disable animations in test builds

// iOS: In test setUp or launch arguments
app.launchArguments.append("--disable-animations")

// Android: Developer options or test rule
@get:Rule
val disableAnimationsRule = DisableAnimationsRule()

# Android: ADB command
adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0

3. Network Dependency

Symptom: Works locally, fails in CI

Fix: Mock network at test boundary

// iOS: URLProtocol stubbing
class MockURLProtocol: URLProtocol {
    static var mockResponses: [URL: Data] = [:]

    override class func canInit(with request: URLRequest) -> Bool { true }

    override func startLoading() {
        if let data = Self.mockResponses[request.url!] {
            client?.urlProtocol(self, didLoad: data)
        }
        client?.urlProtocolDidFinishLoading(self)
    }
}

// Android: OkHttp MockWebServer
@Before
fun setUp() {
    mockWebServer = MockWebServer()
    mockWebServer.start()

    mockWebServer.enqueue(MockResponse()
        .setBody("""{"user": "test"}""")
        .setResponseCode(200))
}

4. Shared State Between Tests

Symptom: Test passes alone, fails in suite

Fix: Reset state before each test

// iOS: Reset app state
override func setUp() {
    app = XCUIApplication()
    app.launchArguments = ["--reset-state", "--uitesting"]
    app.launch()
}

// Android: Clear app data
@Before
fun clearData() {
    InstrumentationRegistry.getInstrumentation()
        .targetContext.deleteDatabase("app.db")
    InstrumentationRegistry.getInstrumentation()
        .targetContext.getSharedPreferences("prefs", 0).edit().clear().apply()
}

5. Device-Specific Issues

Symptom: Passes on iPhone 15, fails on iPhone SE

Fix:

• Check screen size assumptions
• Verify element visibility before interaction
• Use scroll-to-element patterns

// Scroll to element before tap
let element = app.buttons["hiddenButton"]
while !element.isHittable {
    app.swipeUp()
}
element.tap()

6. Race Conditions

Symptom: Intermittent failures, non-deterministic order

Fix: Use synchronization primitives

// Android: Espresso IdlingResource for async operations
class NetworkIdlingResource : IdlingResource {
    private var callback: IdlingResource.ResourceCallback? = null
    private var isIdle = true

    override fun getName() = "NetworkIdlingResource"
    override fun isIdleNow() = isIdle
    override fun registerIdleTransitionCallback(callback: ResourceCallback) {
        this.callback = callback
    }

    fun setIdle(idle: Boolean) {
        isIdle = idle
        if (idle) callback?.onTransitionToIdle()
    }
}

Quarantine Strategy

Quarantine Workflow

1. Detect: Test fails 3+ times on same code within 24 hours 2. Quarantine: Mark as flaky and exclude from blocking CI 3. Assign: Assign owner with 1-week SLA 4. Fix or Delete: Either stabilize or remove permanently 5. Reinstate: Return to main suite after 10 consecutive passes

Implementation

For iOS UI automation (XCUITest), quarantine by test plan/selection rather than ad-hoc code flags:

• Put unstable tests in a separate class/target (for example FlakyTests) or a separate Xcode Test Plan configuration.
• Run stable suites on PR; run flaky suites in a non-blocking job.

For Android instrumentation, quarantine with annotations + runner arguments:

@Retention(AnnotationRetention.RUNTIME)
@Target(AnnotationTarget.FUNCTION)
annotation class Flaky(val bug: String)

@Flaky("JIRA-1234")
@Test
fun unstableTest() { }

# Stable suite (exclude flaky)
./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.notAnnotation=com.example.Flaky

# Flaky suite (run separately, non-blocking)
./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.Flaky

CI Configuration

# GitHub Actions: Separate flaky test job
jobs:
  stable-tests:
    runs-on: macos-latest
    steps:
      - run: xcodebuild test -skip-testing:MyAppTests/FlakyTests

  flaky-tests:
    runs-on: macos-latest
    continue-on-error: true  # Don't block PR
    steps:
      - run: xcodebuild test -only-testing:MyAppTests/FlakyTests

Rerun Policies

Recommended Limits

Implementation

# Fastlane
lane :test do
  scan(
    scheme: "MyApp",
    only_testing: ["MyAppUITests"],
    try_count: 2,  # Retries for failed tests (keep low)
    parallel_testing: true
  )
end

# xcodebuild: emit a result bundle for triage
xcodebuild test ... -resultBundlePath TestResults.xcresult
# Optional: rerun only failing tests via your CI orchestration (keep retries low)

Warning: High rerun rates mask underlying issues. If rerun rate >20%, stop and fix root causes.

Prevention Checklist

Before adding new UI tests:

• [ ] Uses explicit waits, not fixed sleeps
• [ ] Uses accessibilityIdentifier, not text labels
• [ ] Mocks network calls
• [ ] Resets app state in setUp
• [ ] Handles system alerts (permissions, notifications)
• [ ] Verified on smallest and largest device in matrix
• [ ] Runs 10x locally without failure

Monitoring Dashboard

Track these metrics weekly:

Flake Report - Week of 2026-01-18
---------------------------------
Total UI tests:           245
Flaky tests (>5% rate):   12 (4.9%)
Top offenders:
  1. testPaymentFlow      - 23% flake rate (network timing)
  2. testOnboarding       - 18% flake rate (animation)
  3. testDeepLink         - 15% flake rate (race condition)

Action items:
  - testPaymentFlow: Add network mock by 01/25
  - testOnboarding: Disable animations, verify by 01/22
  - testDeepLink: Review async handling by 01/24

Resources

• Bitrise: Why Flaky Tests Are Increasing
• AccelQ: Flaky Tests 2026
• TestDino: Flaky Test Detection Tools
• UI-Based Flaky Tests Research (ICSE)

Mobile Test Framework Comparison

Contents

• Defaults (2026)
• Decision Matrix
• When to Use Each
• Recommendation by App Type
• Hybrid Strategy
• CI Integration Patterns
• Cost Considerations

Defaults (2026)

• Prefer first-party frameworks (XCUITest + Espresso/Compose) for PR gates and smoke coverage.
• Add app-specific cross-platform frameworks (Detox for React Native; Flutter integration_test/Patrol for Flutter) when they reduce total maintenance.
• Use Appium when you truly need one codebase across iOS + Android (accept higher cost/flake).
• Use Maestro for fast-to-author, black-box smoke flows (treat as complementary, not a full replacement).
• Keep unit tests in native stacks (Swift Testing/XCTest; JUnit) and treat UI automation as a smaller, higher-cost layer.

Decision Matrix

When to Use Each

XCUITest (iOS Native UI)

Best for: iOS-only apps, teams with Swift expertise.

Pros:

• First-party Apple support, always up-to-date.
• Fast execution, runs in-process.
• Deep OS integration (accessibility, permissions).
• No external dependencies.

Cons:

• iOS only, no cross-platform.
• Parallelization is possible, but requires careful test isolation and infrastructure.

Espresso / Compose (Android Native UI)

Best for: Android-only apps, teams with Kotlin expertise.

Pros:

• First-party Google support.
• Synchronization with UI thread (fewer flakes).
• Fast execution.
• Excellent Android Studio integration.

Cons:

• Android only.
• Complex gestures and cross-app flows may need UIAutomator.

Maestro (Cross-Platform Smoke)

Best for: Fast-to-author smoke tests across iOS + Android, onboarding, and regression breadth.

Pros:

• Very fast authoring; readable YAML scenarios.
• Works well for "happy path" coverage and broad device/language coverage.

Cons:

• Limited for deep assertions and complex app-internal state.
• Still needs the same flake disciplines (stable selectors, determinism, controlled state).

Appium (Cross-Platform, Black-Box)

Best for: Cross-platform apps needing single test codebase, teams with existing Selenium expertise.

Pros:

• Write once, run on iOS and Android.
• Language-agnostic (Python, JS, Java, Ruby).
• Large community and ecosystem.
• Works with native, hybrid, and web apps.

Cons:

• Slower than native frameworks (uses WebDriver protocol).
• Higher flake rate due to indirection.
• Complex setup with multiple dependencies.
• Maintenance overhead keeping drivers in sync.

Detox (React Native Gray-Box)

Best for: React Native apps.

Pros:

• Gray-box testing with JS bridge access.
• Automatic synchronization with React Native.
• Fast execution.
• Good flake resistance.

Cons:

• React Native only.
• Requires Detox-specific app configuration.
• Smaller community than Appium.

Recommendation by App Type

Hybrid Strategy

Many teams use multiple frameworks:

1. Native frameworks for speed: XCUITest/Espresso for smoke tests in CI. 2. App-specific cross-platform: Detox (RN) or Patrol (Flutter) for shared coverage. 3. Black-box smoke: Maestro to expand breadth across devices/regions. 4. Cross-platform fallback: Appium for hard-to-reach areas and legacy coverage. 5. Manual for exploratory: Real devices for UX, edge cases, and "unknown unknowns".

CI Integration Patterns

Native (Fast Feedback)

# Run on every PR
- XCUITest smoke suite (5 min)
- Espresso smoke suite (5 min)

Cross-Platform (Nightly)

# Run nightly on device farm
- Appium full regression (30 min)
- Multiple device configurations

Cost Considerations

Mobile Performance Testing

Mobile app performance testing and benchmarking -- measuring startup time, frame rate, memory, battery, network, and app size with automated regression detection.

Contents

• Startup Time Measurement
• Frame Rate and Jank Detection
• Memory Profiling
• Battery Drain Testing
• Network Performance
• App Size Optimization
• Performance Budgets and CI Gates
• Automated Regression Detection
• Tools Comparison
• Performance Testing Checklist
• Related Resources

---

Startup Time Measurement

Startup Types

Android: Measuring Startup

# Displayed time (cold start) - simplest measurement
adb shell am start-activity -W -n com.example.app/.MainActivity
# Output: TotalTime: 1234  (milliseconds)

# Fully drawn time (report when app considers itself ready)
adb shell am start-activity -W -n com.example.app/.MainActivity
# Requires calling Activity.reportFullyDrawn() in app code

// Report fully drawn after data loads
class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        viewModel.data.observe(this) { data ->
            renderData(data)
            // Signal that the app is fully usable
            reportFullyDrawn()
        }
    }
}

Android: Macrobenchmark (Automated)

// benchmark/src/androidTest/java/StartupBenchmark.kt
@RunWith(AndroidJUnit4::class)
class StartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startupCold() = benchmarkRule.measureRepeated(
        packageName = "com.example.app",
        metrics = listOf(StartupTimingMetric()),
        iterations = 10,
        startupMode = StartupMode.COLD
    ) {
        pressHome()
        startActivityAndWait()
    }

    @Test
    fun startupWarm() = benchmarkRule.measureRepeated(
        packageName = "com.example.app",
        metrics = listOf(StartupTimingMetric()),
        iterations = 10,
        startupMode = StartupMode.WARM
    ) {
        pressHome()
        startActivityAndWait()
    }
}

iOS: Measuring Startup

// XCTest performance measurement
func testColdStartupTime() throws {
    let app = XCUIApplication()
    measure(metrics: [XCTApplicationLaunchMetric()]) {
        app.launch()
    }
}

// Custom metric: time to interactive
func testTimeToInteractive() throws {
    let app = XCUIApplication()
    let start = CFAbsoluteTimeGetCurrent()
    app.launch()

    // Wait for key element indicating app is usable
    let feed = app.collectionViews["mainFeed"]
    XCTAssertTrue(feed.waitForExistence(timeout: 5))

    let elapsed = CFAbsoluteTimeGetCurrent() - start
    XCTAssertLessThan(elapsed, 2.0, "Time to interactive exceeded 2s budget")
}

# Xcode Instruments: App Launch template
xcrun xctrace record --template "App Launch" \
  --device "iPhone 15" \
  --launch com.example.app \
  --output startup-trace.trace

---

Frame Rate and Jank Detection

Jank Thresholds

Android: Jank Detection

# systrace / Perfetto capture
adb shell perfetto --txt -c - --out /data/misc/perfetto-traces/trace.pb <<EOF
buffers: { size_kb: 63488 }
data_sources: { config { name: "linux.ftrace" ftrace_config {
  ftrace_events: "sched/sched_switch"
  ftrace_events: "power/suspend_resume"
  ftrace_events: "android_os/android_os_wait_for_vsync"
  atrace_categories: "gfx" atrace_categories: "view"
  atrace_categories: "wm" atrace_categories: "am"
} } }
duration_ms: 10000
EOF

adb pull /data/misc/perfetto-traces/trace.pb .
# Open at https://ui.perfetto.dev

# Quick jank stats via dumpsys
adb shell dumpsys gfxinfo com.example.app framestats

# Output includes:
# Total frames rendered: 1234
# Janky frames: 56 (4.54%)
# 50th percentile: 8ms
# 90th percentile: 14ms
# 95th percentile: 18ms
# 99th percentile: 28ms

iOS: Frame Rate Monitoring

// Instruments: Core Animation FPS
// Use Animation Hitches template in Instruments

// Programmatic monitoring (debug builds)
import QuartzCore

class FrameRateMonitor {
    private var displayLink: CADisplayLink?
    private var lastTimestamp: CFTimeInterval = 0
    private var frameCount = 0
    private var jankCount = 0

    func start() {
        displayLink = CADisplayLink(target: self, selector: #selector(tick))
        displayLink?.add(to: .main, forMode: .common)
    }

    @objc private func tick(link: CADisplayLink) {
        let frameDuration = link.timestamp - lastTimestamp
        frameCount += 1

        // Jank = frame took longer than 2x target (33ms for 60fps)
        if frameDuration > 0.033 && lastTimestamp > 0 {
            jankCount += 1
        }
        lastTimestamp = link.timestamp
    }

    var jankRate: Double {
        guard frameCount > 0 else { return 0 }
        return Double(jankCount) / Double(frameCount) * 100
    }
}

---

Memory Profiling

Key Metrics

Android Memory Profiling

# Quick memory snapshot
adb shell dumpsys meminfo com.example.app

# Key values to track:
#   TOTAL PSS (Proportional Set Size) - your app's real memory footprint
#   Java Heap
#   Native Heap
#   Graphics

# Detect leaks with LeakCanary (debug builds)
# build.gradle.kts
# debugImplementation("com.squareup.leakcanary:leakcanary-android:2.14")

// Automated memory test with Macrobenchmark
@Test
fun scrollMemoryUsage() = benchmarkRule.measureRepeated(
    packageName = "com.example.app",
    metrics = listOf(MemoryUsageMetric(MemoryUsageMetric.Mode.Max)),
    iterations = 5,
    startupMode = StartupMode.COLD
) {
    startActivityAndWait()
    // Scroll through content
    val list = device.findObject(By.res("com.example.app:id/recyclerView"))
    repeat(10) {
        list.fling(Direction.DOWN)
        device.waitForIdle()
    }
}

iOS Memory Profiling

# Xcode Instruments: Leaks template
xcrun xctrace record --template "Leaks" \
  --device "iPhone 15" \
  --attach com.example.app \
  --output leaks-trace.trace \
  --time-limit 60s

// Memory assertion in XCTest
func testMemoryAfterNavigation() throws {
    let app = XCUIApplication()
    app.launch()

    // Navigate through screens
    for _ in 0..<10 {
        app.buttons["detailButton"].tap()
        app.navigationBars.buttons.element(boundBy: 0).tap()
    }

    // Check memory via MetricKit or os_signpost
    let metrics = XCTMemoryMetric(application: app)
    measure(metrics: [metrics]) {
        app.buttons["detailButton"].tap()
        app.navigationBars.buttons.element(boundBy: 0).tap()
    }
}

---

Battery Drain Testing

Android: Battery Historian

# Reset battery stats
adb shell dumpsys batterystats --reset

# Run your test scenario (e.g., 30 min usage simulation)
# ...

# Capture bug report
adb bugreport bugreport.zip

# Analyze with Battery Historian
# Upload bugreport.zip to https://bathist.ef.lc/ or run locally:
docker run -p 9999:9999 gcr.io/android-battery-historian/stable:latest

iOS: Xcode Energy Diagnostics

# Xcode Instruments: Energy Log template
xcrun xctrace record --template "Energy Log" \
  --device "iPhone 15" \
  --attach com.example.app \
  --output energy-trace.trace \
  --time-limit 300s

Battery Budget Thresholds

---

Network Performance

Latency and Bandwidth Simulation

# Android: emulator network throttling
emulator -avd Pixel_8_API_34 -netdelay 3g -netspeed 3g

# Network condition presets:
#   none    : no delay
#   gprs    : 150-550ms delay, 20-40 kbps
#   edge    : 80-400ms delay, 120-240 kbps
#   umts    : 35-200ms delay, 384 kbps-2Mbps
#   hsdpa   : 0-100ms delay, 14.4 Mbps
#   lte     : 0-50ms delay, 100 Mbps

// iOS: Network Link Conditioner (Xcode)
// Settings > Developer > Network Link Conditioner

// Programmatic with URLSession configuration
let config = URLSessionConfiguration.default
config.timeoutIntervalForRequest = 10
config.waitsForConnectivity = true

// Test offline behavior
config.requestCachePolicy = .reloadIgnoringLocalCacheData

let session = URLSession(configuration: config)

API Response Time Validation

// Android: automated API performance test
@Test
fun apiResponseTimeBudget() = runBlocking {
    val client = OkHttpClient()
    val endpoints = listOf("/api/feed", "/api/profile", "/api/search")

    for (endpoint in endpoints) {
        val start = System.nanoTime()
        val response = client.newCall(
            Request.Builder().url("$BASE_URL$endpoint").build()
        ).execute()
        val elapsed = (System.nanoTime() - start) / 1_000_000 // ms

        assertTrue(
            "API $endpoint took ${elapsed}ms (budget: 500ms)",
            elapsed < 500
        )
        assertTrue("API $endpoint failed: ${response.code}", response.isSuccessful)
    }
}

---

App Size Optimization

Android: APK Analyzer

# Command-line APK analysis
$ANDROID_HOME/cmdline-tools/latest/bin/apkanalyzer apk summary app-release.apk
$ANDROID_HOME/cmdline-tools/latest/bin/apkanalyzer apk file-size app-release.apk
$ANDROID_HOME/cmdline-tools/latest/bin/apkanalyzer dex packages app-release.apk | head -20

# Size budget CI check
MAX_SIZE_MB=25
ACTUAL_SIZE=$(stat -f%z app-release.apk)
ACTUAL_MB=$((ACTUAL_SIZE / 1048576))
if [ "$ACTUAL_MB" -gt "$MAX_SIZE_MB" ]; then
  echo "FAIL: APK size ${ACTUAL_MB}MB exceeds budget ${MAX_SIZE_MB}MB"
  exit 1
fi

iOS: App Thinning Report

# Generate app thinning size report
xcodebuild -exportArchive \
  -archivePath MyApp.xcarchive \
  -exportOptionsPlist ExportOptions.plist \
  -exportPath output/ \
  -exportThinning '<thin-for-all-variants>'

# Parse App Thinning Size Report.txt for per-device sizes

Size Budgets

---

Performance Budgets and CI Gates

Budget Configuration

{
  "performance_budgets": {
    "startup_cold_ms": 1500,
    "startup_warm_ms": 1000,
    "jank_rate_pct": 5,
    "memory_peak_mb": 256,
    "apk_size_mb": 25,
    "ipa_size_mb": 30,
    "api_p95_ms": 500,
    "frozen_frames_pct": 0
  }
}

CI Gate Script

#!/usr/bin/env python3
"""Performance budget gate for CI pipelines."""
import json
import sys

def check_budgets(results_file: str, budgets_file: str) -> bool:
    with open(results_file) as f:
        results = json.load(f)
    with open(budgets_file) as f:
        budgets = json.load(f)["performance_budgets"]

    passed = True
    for metric, budget in budgets.items():
        actual = results.get(metric)
        if actual is None:
            print(f"  WARN: {metric} not measured")
            continue
        if actual > budget:
            print(f"  FAIL: {metric} = {actual} (budget: {budget})")
            passed = False
        else:
            print(f"  PASS: {metric} = {actual} (budget: {budget})")

    return passed

if __name__ == "__main__":
    ok = check_budgets(sys.argv[1], sys.argv[2])
    sys.exit(0 if ok else 1)

# GitHub Actions: performance gate
- name: Check performance budgets
  run: python scripts/check_perf_budgets.py results.json budgets.json

---

Automated Regression Detection

Statistical Comparison

def detect_regression(
    baseline: list[float],
    current: list[float],
    threshold_pct: float = 10.0
) -> dict:
    """Compare current metrics against baseline with statistical significance."""
    import statistics

    baseline_median = statistics.median(baseline)
    current_median = statistics.median(current)
    change_pct = ((current_median - baseline_median) / baseline_median) * 100

    # Mann-Whitney U test for significance (non-parametric)
    from scipy.stats import mannwhitneyu
    stat, p_value = mannwhitneyu(baseline, current, alternative='greater')

    regression = change_pct > threshold_pct and p_value < 0.05
    return {
        "baseline_median": f"{baseline_median:.1f}",
        "current_median": f"{current_median:.1f}",
        "change_pct": f"{change_pct:+.1f}%",
        "p_value": f"{p_value:.4f}",
        "regression_detected": regression,
    }

Trend Tracking

---

Tools Comparison

---

Performance Testing Checklist

Before release:

• [ ] Cold start measured on minimum-spec device (Tier 1)
• [ ] Frame rate profiled during key user journeys (scroll, transition)
• [ ] Memory profiled: no leaks after 10 navigation cycles
• [ ] Battery measured for 30-min active session
• [ ] App size within budget; delta from last release documented
• [ ] Network performance tested under 3G simulation
• [ ] Performance budgets gate passing in CI
• [ ] Regression comparison against last release baseline
• [ ] Results documented in release notes

---

Related Resources

• device-farm-strategies.md -- cloud device farm selection for perf testing at scale
• flake-management.md -- stability for performance test suites
• framework-comparison.md -- automation frameworks with performance support
• SKILL.md -- parent mobile testing skill
• Android Macrobenchmark
• Xcode Instruments
• Perfetto Trace Viewer
• Firebase Performance Monitoring