Live Video Pipeline - RTSP Source, GPU Inference & PTZ, Live Video Output

We need a developer to help us design and build a pipeline that takes a panoramic RTSP feed, processes it on a GPU, and pushes a cropped region back out to another source at 30fps. The crop position comes from an object detector, so the effect is a virtual PTZ camera following the action. We have a working prototype. It runs, but preprocessing is still on the CPU and pegs a core, which caps us at one stream per box and keeps us from holding 30fps at full resolution. Reconnect handling is thin, and it's never been asked to run for three hours straight. This is what we're looking to solve. We'd prefer one process, everything as a command-line argument: tracker-pipeline --model ball.engine --tracker mytracker.CropTracker \ --input rtsp://camera/stream --output rtsp://localhost:8554/court1 Input and output each take an RTSP URL or a file path, and all four combinations need to work (file to RTSP, RTSP to file, etc). We test against recorded clips constantly, so file mode isn't an afterthought. Per frame: decode on GPU, convert to model input without touching the CPU, run a TensorRT engine, hand detections to a Python tracker module that returns a crop box, crop, encode with NVENC, push out. Audio comes in on the same stream and has to come out in sync — passthrough is fine, but the video path picks up variable latency, so audio has to be delayed to match. Sync that's fine for ten minutes and half a second off by the end of a game is a failure. This pipeline will run all day without breaks, so the audio should remain in sync. Requirements: The pipeline should sustain real-time (30 FPS) processing without accumulating latency. Under normal operation, dropped frames should be avoided, and when input interruptions occur, the pipeline should recover cleanly without falling behind real time. A few seconds of processing before generating the output is fine. Environment is an EC2 g6.xlarge with an L4, Deep Learning AMI, Ubuntu. Camera is an Axis panoramic at 5120x2560. Decode is PyNvVideoCodec today, but we're not attached to it if you have a better argument. Three things we care about: Model and tracker are arguments, not constants. We'll retrain with a different class count later and that shouldn't be a code change. The tracker loads by module path through a fixed interface we provide. The tracker itself is out of scope. We'll give you a stub that returns a crop box to build against. The real one is tuned against a lot of specific failure cases and we maintain it. If the crop looks wrong while you're testing, just let us know. It has to survive the real environment. Cameras drop, bad cable runs can cause drops, games run long. Input goes away, it reconnects and picks back up with nobody watching. Something genuinely unrecoverable, it exits non-zero and says why — we'd rather it die loudly than hang holding a GPU context. Structured logs to stdout, clean shutdown on SIGTERM. Happy to hear you out on component choices. Not looking for large scale architecture — batching, orchestration, and scaling will be handled elsewhere. One process that does one thing properly is what we are after for now. One number we want: how many copies run at once on a single g6.xlarge before something falls over, and what falls over first. Send the profiling, not an estimate. Preferred background: someone who has shipped hardware decode and encode in production — NVDEC/NVENC, PyNvVideoCodec, VPF, DeepStream, GStreamer nvcodec. TensorRT in something real-time, not a notebook. In your proposal, please include: A system you built with hardware decode/encode — throughput, what broke, how you fixed it. How you'd get NV12 into model input without a round trip through host memory. The camera drops the RTSP stream mid-game. Walk us through what your code does over the next sixty seconds. If you've also worked on automated camera framing for sports — deciding how the crop moves, not just moving it — mention it. That's separate work we'll be putting out later and we'd rather find one person for both. We'd want to break this into milestones, starting with something small — file in, file out against a stub tracker — before committing to the full scope. Propose a breakdown with your bid.