GPU usage
-
The Cost of Testing Every New AI Model
Read Full Article: The Cost of Testing Every New AI ModelDiscovering the ability to test every new AI model has led to a significant increase in electricity bills, as evidenced by a jump from $145 in February to $847 in March. The pursuit of optimizing model performance, such as experimenting with quantization settings for Llama 3.5 70B, results in intensive GPU usage, causing both financial strain and increased energy consumption. While there is a humorous nod to supporting renewable energy, the situation highlights the potential hidden costs of enthusiast-level AI experimentation. This matters because it underscores the environmental and financial implications of personal tech experimentation.
-
Script to Save Costs on Idle H100 Instances
Read Full Article: Script to Save Costs on Idle H100 InstancesIn the realm of machine learning research, the cost of running high-performance GPUs like the H100 can quickly add up, especially when instances are left idle. To address this, a simple yet effective daemon script was created to monitor GPU usage using nvidia-smi. The script detects when a training job has finished and, if the GPU remains idle for a configurable period (default is 20 minutes), it automatically shuts down the instance to prevent unnecessary costs. This solution, which is compatible with major cloud providers and open-sourced under the MIT license, offers a practical way to manage expenses by reducing idle time on expensive GPU resources. This matters because it helps researchers and developers save significant amounts of money on cloud computing costs.