# Using Hugging Face Jobs to train a NanoGPT model

## Executive summary

This video demonstrates how to execute large-scale machine learning training tasks, such as training a small LLM (NanoGPT), using Hugging Face Jobs. The process is designed to run on powerful remote GPUs (e.g., A100) without requiring the user to modify the existing training code or manage complex remote connections, relying instead on a generic shell script to handle environment setup, execution, and result persistence.

## Key takeaways

- Simplified Remote Training: Training can be moved to Hugging Face Compute in minutes for a low cost (e.g., $1 on an A100 GPU), eliminating the need for manual setup of virtual machines, SSH connections, or code transfers.
- Code Reusability: The training process can be executed using a simple, generic shell script that requires no changes to the original Python training code, provided the code uses `requirements.txt` and pulls data from the Hugging Face Hub.
- Guaranteed Result Persistence: Results (like intermediate checkpoints) are stored in a permanent Hugging Face bucket. This ensures that even if the job times out or the GPU machine shuts down, the results are not lost.

## Technical details

- Model Architecture: The example uses a small GPT-2 style model with 50 million parameters, featuring eight layers and an embedding dimension of 512, trained on the Tiny Stories dataset.
- Training Workflow: The local training code involves finding the available device (CUDA, MPS, or CPU), getting data from the Hugging Face Hub, instantiating the model/optimizer, and running a main loop with a cosine schedule and warm-up for the learning rate.
- Job Execution Mechanism: The shell script automates the process by: 1) Creating a persistent storage bucket; 2) Requesting a GPU machine (e.g., A100); 3) Mounting the bucket; 4) Copying the code; and 5) Running the Python script using `UV` in the remote environment.
- Job Management Parameters: The shell script allows setting a mandatory timeout (default is two hours) and ensures that the job automatically shuts down when complete or timed out, optimizing billing.

## Practical implications

- Build engineers can integrate large-scale ML training into CI/CD pipelines using Hugging Face Jobs, treating the remote GPU environment as a reliable, ephemeral compute resource.
- The use of a generic shell script minimizes the need for specialized cloud infrastructure knowledge, making the process highly portable across different ML projects.
- The built-in persistence of results in the dedicated bucket eliminates the risk of data loss associated with transient remote compute environments.

## Topics

Large Language Models (LLMs), Deep Learning Training, Cloud Computing, Build Automation, GPU Resource Management, NanoGPT Code, Hugging Face Compute

Source: https://www.youtube.com/watch?v=7Yx_7byZiFU
