Security Engineer & SOC/IR Specialist (Wazuh, Prometheus, Ubuntu & macOS)

Overview We run a distributed fleet of remote Ubuntu servers, NVIDIA GPU workstations, and macOS machines connected over a Tailscale overlay network. Following a recent internal security review, we're hardening every endpoint and standing up continuous monitoring, alerting, and incident response. We're looking for a senior Security Engineer to lead this effort: harden and audit each machine, finalize a central Wazuh SIEM/XDR and Prometheus/Grafana monitoring stack, and bring the entire fleet under continuous security monitoring. A pilot is already deployed and proven. We now need an experienced professional to take it to production across the fleet. What You'll Do Phase 1 — Harden & Instrument the Fleet 1. Endpoint Hardening & Audit — Ubuntu and macOS Run baseline security audits on each machine. Review privileged accounts, SSH keys and authorized_keys, sudoers configuration, active network listeners, and persistence mechanisms. On Linux, inspect systemd units and timers, cron jobs, startup scripts, and other persistence mechanisms. On macOS, inspect LaunchDaemons, LaunchAgents, login items, remote-access tooling, and other persistence mechanisms. Identify anomalies and remediate unauthorized access, processes, services, and insecure configurations. 2. Central Monitoring Deploy, configure, and secure a central monitoring server running Wazuh SIEM/XDR and the Prometheus/Grafana monitoring stack. The stack includes: Wazuh Manager and Dashboard Prometheus Grafana Alertmanager Node Exporter NVIDIA DCGM Exporter for GPU systems 3. Secure Networking Ensure all agent-to-server and monitoring traffic flows exclusively over the Tailscale interface. Configure host firewalls to reject monitoring traffic from any unauthorized interface or source. Install and tune Wazuh agents across Ubuntu and macOS endpoints. Deploy Node Exporter on Linux systems and NVIDIA DCGM Exporter on applicable GPU hosts. Verify that monitoring services cannot be accessed through public or unintended network interfaces. 4. Resource Safety — Linux Implement and test systemd cgroup resource limits, including MemoryHigh and MemoryMax, to prevent heavy workloads from exhausting system memory or causing kernel-level instability. Phase 2 — SOC & Incident Response Monitor the Wazuh environment and investigate security alerts. Tune detection rules to reduce noise and prioritize actionable events. Perform threat hunting across endpoints. Investigate suspicious authentication activity, processes, persistence mechanisms, network connections, file changes, privilege escalation, and policy violations. Perform incident response, containment, remediation, and post-incident analysis when threats are identified. Maintain clear documentation of findings, changes, incidents, and remediation actions. Requirements Must Have 1. Deep hands-on experience with Ubuntu Server and macOS administration and security hardening. 2. Strong networking fundamentals, including overlay and mesh VPNs such as Tailscale and WireGuard, host firewalls including ufw, nftables/iptables, and macOS pf, NAT and port forwarding, and restricting services to specific interfaces or source addresses. 3. Production experience deploying and operating Wazuh and the Prometheus monitoring stack. 4. Real-world Incident Response and threat-hunting experience, not just monitoring infrastructure setup. 5. Disciplined, least-privilege handling of remote administrative access. 6. Clear written communication and documentation skills. You should be able to explain what you found, what you changed, why you changed it, and how the result was verified. Nice to Have 1. NVIDIA GPU host experience, including DCGM and driver/CUDA security and maintenance. 2. Knowledge of macOS internals, including TCC, MDM, system extensions, and virtualization hosts. 3. Detection engineering experience, including custom Wazuh rule development and tuning. Screening Questions Please answer all of the questions below and include one or two brief, sanitized examples from past work. Do not include client-sensitive or confidential information. Applications that skip the screening questions will not be reviewed. 1. Describe a recent engagement where you deployed a SIEM such as Wazuh alongside an observability stack such as Prometheus to monitor distributed Linux endpoints. What problems did you encounter, and how did you resolve them? 2. We route infrastructure traffic over Tailscale. How would you configure host firewalls and monitoring agents such as node_exporter so they accept traffic only from the central monitoring server over the overlay interface? How would you verify that the restrictions are working correctly? 3. macOS persistence differs significantly from Linux. Which specific locations, mechanisms, and native tools do you audit on a Mac when hunting for unauthorized persistence, backdoors, or unexpected remote-access agents? 4. Walk us through how you would triage a host you suspect is compromised while keeping it in service. What would you check, in what order, and how would you decide between remediation in place and rebuilding the host? 5. How do you tune Wazuh to reduce alert fatigue and ensure notifications are focused on actionable security threats?