Build an LLM Pretraining Scaling & Reliability Lab

I am looking for an experienced ML Systems Engineer / Research Engineer to build a portfolio-quality project focused on LLM training operations, not chatbot development. The project should use LLM training framework and extend it into a small but professional training lab. The final system should include: Reproducible small LLM training run Training/validation loss tracking Tokens/sec and step-time metrics GPU/CPU memory monitoring Checkpoint save and resume Failure simulation and recovery Experiment comparison Simple dashboard Clear documentation and handover The goal is to demonstrate how LLM training runs can be started, monitored, debugged, resumed, compared, and documented. This is not a basic GPT clone or chatbot. I want clean ML engineering around pretraining reliability, observability, scaling experiments, and experiment tracking. Expected Deliverables: Framework recommendation Working baseline training run Metrics logging into run folders Simple dashboard for comparing runs Checkpoint/resume test At least 3–4 small experiments Failure/recovery report README and setup guide Short walkthrough/handover Screening Questions: Have you trained GPT/Transformer models before? Have you used PyTorch, LitGPT, nanoGPT, or similar? Can you explain tokens/sec, checkpointing, validation loss, and gradient accumulation? Have you built ML experiment dashboards or tracking systems? Can you provide clean documentation and a walkthrough? Important: I need clean code, clear documentation, and explanation, not just a black-box demo.