Senior Android Device Fleet Reliability Engineer — Self-Healing Device Lab

Senior Android Device Fleet Reliability Engineer — Build Self-Healing Device Lab Infrastructure Project overview We operate a small physical Android device lab used for browser automation over cellular connections. Our current environment is intentionally simple: Six Samsung Android phones Phones connected by USB to Ubuntu servers ADB Android Chrome Chrome DevTools Protocol Playwright-based automation Active cellular SIMs There is not an established device-management platform or mature infrastructure stack to review. Today, the phones are essentially plugged into Ubuntu hosts and accessed directly by Playwright workers. We have created a detailed architecture and phased project plan for the system we want to build. We are looking for an experienced engineer to validate the practical implementation choices and build the foundational infrastructure. This is not a standard Android app-development role and not primarily an Appium test-writing role. The work combines: Physical Android device-fleet reliability ADB and USB lifecycle management Linux services and process supervision Chrome DevTools Protocol Browser automation Device health state machines Lease-based resource allocation Automated recovery and incident logging The architecture should work well for six devices today and scale cleanly as the fleet grows. What we need built Each Ubuntu host should run one authoritative device supervisor responsible for the phones physically connected to that host. The supervisor will own: Device discovery Stable serial-number-to-CDP-port mappings The host’s ADB server ADB forwards Device health checks Recovery actions Device leases Per-device locks, retry limits, and cooldowns Cellular connection verification and IP rotation Structured incident logging Quarantine and escalation behavior Automation workers should request a healthy device through a defined interface. They should not independently scan ADB, assign ports, restart Chrome, rebuild forwards, reboot devices, or toggle cellular radios. The system should identify which layer is actually broken and apply the smallest safe recovery at that layer. For example: A missing page should not reboot a phone. A missing ADB forward should be recreated without restarting Chrome. A stopped Chrome process should be relaunched and verified. A USB-disconnected phone should not trigger recovery actions on unrelated phones. A shared USB-hub event should be handled as one correlated incident rather than several competing device recoveries. Every recovery must be followed by verification that the device is actually usable again. Initial engagement We would like to begin with a paid architecture-validation and implementation milestone. This is not a request for several weeks of open-ended research. The selected engineer should review the project plan, inspect the current Ubuntu and device environment, confirm or challenge specific implementation choices, and begin building a working vertical slice. The initial milestone should include: Confirm the proposed supervisor and worker boundaries. Define the implementation language and local RPC approach. Create a static device registry using serial numbers as identity. Implement fixed serial-to-CDP-port assignments. Implement non-disruptive device discovery. Establish one supervisor-owned ADB server per host. Create a basic supervisor service running under systemd. Expose liveness and readiness status. Implement a small lease-based worker interface. Demonstrate health checks and one or more safe recovery actions. Produce a realistic estimate for the remaining phases. The current plan proposes a supervisor-owned lease proxy so workers receive temporary CDP endpoints rather than permanent canonical ports. The proxy must be able to close existing CDP connections when a lease expires. We expect the engineer to understand and implement this requirement or propose a clearly safer alternative. Follow-on work Successful completion of the initial milestone may lead to implementation of the remaining phases. Phase 1: deterministic single-device recovery Likely recovery handlers include: Fresh-page or navigation retry Page and frame recovery Missing ADB-forward repair Chrome-stopped recovery Missing DevTools-socket recovery CDP endpoint repair Playwright reconnection Per-device recovery limits and cooldowns Phase 2: device and shared-infrastructure recovery Likely work includes: ADB device-offline recovery Individual USB disconnect recovery Shared USB-hub event correlation Cellular-data recovery Controlled mobile IP rotation Chrome memory and lifecycle management Device quarantine based on recurrence Higher-blast-radius actions such as restarting the shared ADB server, power-cycling an entire hub, or rebooting the host should be protected by strict conditions or human approval. Required experience Strong candidates will have hands-on experience with several of the following: Physical Android device fleets or device labs ADB daemon and transport behavior Android USB enumeration and disconnect failures Linux USB topology, hubs, udev, and autosuspend Chrome DevTools Protocol on Android Playwright, Appium, UiAutomator, or comparable automation Linux systemd services Python, Go, Rust, or another suitable systems/backend language Multi-process service design Unix sockets or local RPC Lease-based resource allocation State machines and deterministic recovery playbooks TCP or WebSocket proxying Structured logging and incident correlation Fault-injection testing Docker and Ubuntu operations Experience simply running Appium tests or using ADB manually is not enough. We need someone who can build the control layer that keeps physical devices available and prevents competing processes from interfering with them. Important engineering principles The implementation must preserve these invariants: The device serial number is its persistent identity. Exactly one supervisor is authoritative for a device at a time. Workers do not directly control ADB or canonical CDP ports. Device allocation and device health are separate states. Recovery starts at the lowest broken layer. Recovery attempts are bounded and rate-limited. Shared actions require stronger protections than device-local actions. Lease expiration must terminate existing access, not only block new access. Cellular-required jobs must verify cellular egress rather than assume it. Diagnostic collection must never block necessary recovery indefinitely. Healthy devices must not be disrupted while another device is recovering. Expected deliverables Depending on the agreed milestone scope: Production-quality source code Device registry Host-scoped supervisor Lease and worker API Lease-scoped CDP proxy systemd service configuration ADB ownership and key-handling configuration Device health state machine Recovery playbooks Structured incident logging Fault-injection tests Deployment documentation Operational runbooks Handoff documentation for our internal team All code, configuration, and documentation must be delivered to our repositories. The system should not depend on the freelancer remaining available for routine operation. How to apply Please include: The largest physical Android device fleet you have worked with What you personally designed or implemented Your experience diagnosing ADB and USB failures Your experience with Android Chrome or CDP An example of a self-healing or recovery system you built Your preferred implementation language and why Your availability during the next four to six weeks Your hourly rate A brief explanation of how you would approach the initial milestone Please do not send a generic Android-development proposal.