TraceML has been updated to enhance real-time observability during PyTorch training, particularly for long or remote runs. Key improvements include live monitoring of dataloader fetch times to identify input pipeline stalls, tracking GPU step time drift using non-blocking CUDA events, and monitoring CUDA memory to detect leaks before out-of-memory errors occur. Optional layer-wise timing and memory tracking are available for deeper debugging, and the tool is designed to complement existing profilers. Currently tested on single-GPU setups, with plans for multi-GPU support, TraceML aims to address common issues like step drift and memory creep across various training pipelines. Feedback is sought from users to refine signal detection. This matters because it helps optimize machine learning training processes by identifying and addressing runtime issues early.
