diff --git a/dataflow/run-inference/requirements.txt b/dataflow/run-inference/requirements.txt index d0376ac202e..9b2b60247c7 100644 --- a/dataflow/run-inference/requirements.txt +++ b/dataflow/run-inference/requirements.txt @@ -1,3 +1,3 @@ apache-beam[gcp]==2.49.0 -torch==2.2.2 +torch==2.12.1 transformers==5.0.0rc3 diff --git a/people-and-planet-ai/weather-forecasting/notebooks/3-training.ipynb b/people-and-planet-ai/weather-forecasting/notebooks/3-training.ipynb index b8882b1d34d..be0cbe5b72a 100644 --- a/people-and-planet-ai/weather-forecasting/notebooks/3-training.ipynb +++ b/people-and-planet-ai/weather-forecasting/notebooks/3-training.ipynb @@ -3,6 +3,7 @@ { "cell_type": "code", "execution_count": null, + "id": "g4jtzXwEvW2-", "metadata": { "cellView": "form", "id": "g4jtzXwEvW2-" @@ -27,11 +28,11 @@ "# KIND, either express or implied. See the License for the\n", "# specific language governing permissions and limitations\n", "# under the License." - ], - "id": "g4jtzXwEvW2-" + ] }, { "cell_type": "markdown", + "id": "HtysPAVSvcMg", "metadata": { "id": "HtysPAVSvcMg" }, @@ -61,11 +62,11 @@ "* 💰 **Cost estimate**: [a few cents on Vertex AI](https://cloud.google.com/vertex-ai/pricing#custom-trained_models)\n", "\n", "💚 This is one of many **machine learning how-to samples** inspired from **real climate solutions** aired on the [People and Planet AI 🎥 series](https://www.youtube.com/playlist?list=PLIivdWyY5sqI-llB35Dcb187ZG155Rs_7)." - ], - "id": "HtysPAVSvcMg" + ] }, { "cell_type": "markdown", + "id": "RuFZck60B8t-", "metadata": { "id": "RuFZck60B8t-" }, @@ -73,36 +74,41 @@ "# 🎬 Before you begin\n", "\n", "Let's start by cloning the GitHub repository, and installing some dependencies." - ], - "id": "RuFZck60B8t-" + ] }, { "cell_type": "code", - "source": [ - "# Now let's get the code from GitHub and navigate to the sample.\n", - "!git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git\n", - "%cd python-docs-samples/people-and-planet-ai/weather-forecasting" - ], + "execution_count": null, + "id": "W-fPxkYD9FaP", "metadata": { "id": "W-fPxkYD9FaP" }, - "execution_count": null, "outputs": [], - "id": "W-fPxkYD9FaP" + "source": [ + "# Now let's get the code from GitHub and navigate to the sample.\n", + "!git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git\n", + "%cd python-docs-samples/people-and-planet-ai/weather-forecasting" + ] }, { "cell_type": "markdown", - "source": [ - "The [`weather-model`](../serving/weather-model) local package contains the model definition and the training script.\n", - "This ensures we use the same model definition for both training and predictions.\n" - ], + "id": "r5OijZcuInAe", "metadata": { "id": "r5OijZcuInAe" }, - "id": "r5OijZcuInAe" + "source": [ + "The [`weather-model`](../serving/weather-model) local package contains the model definition and the training script.\n", + "This ensures we use the same model definition for both training and predictions.\n" + ] }, { "cell_type": "code", + "execution_count": null, + "id": "AlcsK6pd-x0I", + "metadata": { + "id": "AlcsK6pd-x0I" + }, + "outputs": [], "source": [ "# Upgrade `setuptools` to install packages from pyproject.toml files.\n", "!pip install --quiet --upgrade --no-warn-conflicts pip setuptools\n", @@ -112,16 +118,11 @@ "\n", "# Install the `weather-model` local package.\n", "!pip install google-cloud-aiplatform serving/weather-model" - ], - "metadata": { - "id": "AlcsK6pd-x0I" - }, - "execution_count": null, - "outputs": [], - "id": "AlcsK6pd-x0I" + ] }, { "cell_type": "markdown", + "id": "G75Y6HszxBL8", "metadata": { "id": "G75Y6HszxBL8" }, @@ -132,12 +133,12 @@ "In order to ensure everything runs as expected, we **_must_ restart the runtime**. This allows Colab to load the latest versions of the libraries.\n", "\n", "![\"Runtime\" > \"Restart runtime\"](images/restart-runtime.png)" - ], - "id": "G75Y6HszxBL8" + ] }, { "cell_type": "code", "execution_count": null, + "id": "xGXRHJ9TFs24", "metadata": { "id": "xGXRHJ9TFs24" }, @@ -145,46 +146,46 @@ "source": [ "# Alternatively, restart the runtime by ending the process.\n", "exit()" - ], - "id": "xGXRHJ9TFs24" + ] }, { "cell_type": "markdown", - "source": [ - "After restarting the runtime, let's navigate back into the sample directory." - ], + "id": "WI_vvBpPD4tr", "metadata": { "id": "WI_vvBpPD4tr" }, - "id": "WI_vvBpPD4tr" + "source": [ + "After restarting the runtime, let's navigate back into the sample directory." + ] }, { "cell_type": "code", - "source": [ - "%cd python-docs-samples/people-and-planet-ai/weather-forecasting" - ], + "execution_count": null, + "id": "6fdyXMdlD3cz", "metadata": { - "id": "6fdyXMdlD3cz", "colab": { "base_uri": "https://localhost:8080/" }, + "id": "6fdyXMdlD3cz", "outputId": "457ec966-1f2a-4e76-df1b-62d7d9d77e60" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "[Errno 2] No such file or directory: 'python-docs-samples/people-and-planet-ai/weather-forecasting'\n", "/content/python-docs-samples/people-and-planet-ai/weather-forecasting/python-docs-samples/people-and-planet-ai/weather-forecasting\n" ] } ], - "id": "6fdyXMdlD3cz" + "source": [ + "%cd python-docs-samples/people-and-planet-ai/weather-forecasting" + ] }, { "cell_type": "markdown", + "id": "mHvEEW6oyFGV", "metadata": { "id": "mHvEEW6oyFGV" }, @@ -209,12 +210,12 @@ "\n", "Once you have everything ready, you can go ahead and fill in your Google Cloud resources in the following code cell.\n", "Make sure you run it!" - ], - "id": "mHvEEW6oyFGV" + ] }, { "cell_type": "code", "execution_count": null, + "id": "YMPNUR0pyRvy", "metadata": { "cellView": "form", "id": "YMPNUR0pyRvy" @@ -247,11 +248,11 @@ "\n", "# Set the gcloud project for other gcloud commands.\n", "!gcloud config set project {project}" - ], - "id": "YMPNUR0pyRvy" + ] }, { "cell_type": "markdown", + "id": "02b1b9dd", "metadata": { "id": "02b1b9dd" }, @@ -261,11 +262,11 @@ "We need our model for both training and for prediction.\n", "So we created the local [`weather-model`](../serving/weather-model) module.\n", "It contains [`weather/model.py`](../serving/weather-model/weather/model.py) where the model is defined, and [`weather/trainer.py`](../serving/weather-model/weather/trainer.py) where all the training code lives." - ], - "id": "02b1b9dd" + ] }, { "cell_type": "markdown", + "id": "PY5H3OMjfVAR", "metadata": { "id": "PY5H3OMjfVAR" }, @@ -276,26 +277,29 @@ "Fortunately, Vertex AI uses [Cloud Storage FUSE](https://cloud.google.com/blog/products/ai-machine-learning/cloud-storage-file-system-ai-training) to mount and access Cloud Storage files as if they were local files.\n", "\n", "For now, let's download the data files we created in the [🗄️ **Create the dataset**](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/notebooks/2-dataset.ipynb) notebook to have them locally." - ], - "id": "PY5H3OMjfVAR" + ] }, { "cell_type": "code", + "execution_count": null, + "id": "h_IUpnqvO-sa", + "metadata": { + "id": "h_IUpnqvO-sa" + }, + "outputs": [], "source": [ "data_path_gcs = f\"gs://{bucket}/weather/data\"\n", "\n", "!mkdir -p data-training\n", "!gcloud storage cp {data_path_gcs}/* data-training" - ], - "metadata": { - "id": "h_IUpnqvO-sa" - }, - "id": "h_IUpnqvO-sa", - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", + "id": "Pl3qbyggO7rR", + "metadata": { + "id": "Pl3qbyggO7rR" + }, "source": [ "First, we need to load the dataset to feed it to the model.\n", "To read a dataset in PyTorch, we could manually instantiate a subclass of `torch.utils.data.Dataset`, but we're going to use [Hugging Face 🤗 Datasets](https://huggingface.co/docs/datasets/main/en/index), which are a high-level interface to use datasets more easily.\n", @@ -307,14 +311,16 @@ "To split the our dataset into training and a testing/validation subsets, we use [`Dataset.train_test_split`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.train_test_split).\n", "\n", "In [`weather/trainer.py`](../serving/weather-model/weather/trainer.py) we defined the `read_dataset` function to load our data files, and returns us a 🤗 Dataset with train/test splits." - ], - "metadata": { - "id": "Pl3qbyggO7rR" - }, - "id": "Pl3qbyggO7rR" + ] }, { "cell_type": "code", + "execution_count": null, + "id": "rxwvw7ihacXy", + "metadata": { + "id": "rxwvw7ihacXy" + }, + "outputs": [], "source": [ "from weather.trainer import read_dataset\n", "\n", @@ -323,19 +329,12 @@ "\n", "# Read the dataset with train/test splits.\n", "dataset = read_dataset(data_path, train_test_ratio)" - ], - "metadata": { - "id": "rxwvw7ihacXy" - }, - "id": "rxwvw7ihacXy", - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", - "source": [ - "print(dataset)" - ], + "execution_count": null, + "id": "ItTBWR98dByh", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -343,12 +342,10 @@ "id": "ItTBWR98dByh", "outputId": "8c522562-9937-4dc5-e482-8bec3cdba277" }, - "id": "ItTBWR98dByh", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "DatasetDict({\n", " train: Dataset({\n", @@ -362,10 +359,17 @@ "})\n" ] } + ], + "source": [ + "print(dataset)" ] }, { "cell_type": "markdown", + "id": "jnlg80Tl4QLS", + "metadata": { + "id": "jnlg80Tl4QLS" + }, "source": [ "> 💡 For more information on loading data into a 🤗 Dataset, refer to the [Loading data](https://huggingface.co/docs/datasets/main/en/loading) guide.\n", "\n", @@ -374,23 +378,12 @@ "Let's see the shapes of the first training example from the `train` split.\n", "When we access an example, we get an `{'inputs': list, 'labels': list}` dictionary, where each value is a [Python list](https://docs.python.org/3/library/stdtypes.html#list).\n", "We can then convert them into [PyTorch tensors](https://pytorch.org/docs/stable/tensors.html) for further use." - ], - "metadata": { - "id": "jnlg80Tl4QLS" - }, - "id": "jnlg80Tl4QLS" + ] }, { "cell_type": "code", - "source": [ - "import torch\n", - "\n", - "train_dataset = dataset[\"train\"]\n", - "example = train_dataset[0] # random access the first element\n", - "\n", - "print(f\"inputs: {torch.as_tensor(example['inputs']).shape}\")\n", - "print(f\"labels: {torch.as_tensor(example['labels']).shape}\")" - ], + "execution_count": null, + "id": "Ji67MIKQ58Zr", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -398,34 +391,42 @@ "id": "Ji67MIKQ58Zr", "outputId": "7e98452a-2744-4a72-93e8-9a53e1f6b695" }, - "id": "Ji67MIKQ58Zr", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "inputs: torch.Size([5, 5, 52])\n", "labels: torch.Size([5, 5, 2])\n" ] } + ], + "source": [ + "import torch\n", + "\n", + "train_dataset = dataset[\"train\"]\n", + "example = train_dataset[0] # random access the first element\n", + "\n", + "print(f\"inputs: {torch.as_tensor(example['inputs']).shape}\")\n", + "print(f\"labels: {torch.as_tensor(example['labels']).shape}\")" ] }, { "cell_type": "markdown", + "id": "JWFJY1pv7T91", + "metadata": { + "id": "JWFJY1pv7T91" + }, "source": [ "The _inputs_ have the shape `(width, height, num_inputs)`, where each input is the value of an Earth Engine band.\n", "\n", "The _outputs_ have the shape `(width, height, num_outputs)`, where each output is a prediction.\n", "We're predicting for 2 and 6 hours into the future, so we get 2 outputs." - ], - "metadata": { - "id": "JWFJY1pv7T91" - }, - "id": "JWFJY1pv7T91" + ] }, { "cell_type": "markdown", + "id": "oQnLpK0OmutA", "metadata": { "id": "oQnLpK0OmutA" }, @@ -442,22 +443,12 @@ "\n", "We need to calculate the mean and standard deviation for each input, so each band is normalized within its own range.\n", "Both the mean and standard deviation must have the shape `(batch, width, height, num_inputs)`, which allows them to _broadcast_ to any batch size, width and height, as long as the `num_inputs` match." - ], - "id": "oQnLpK0OmutA" + ] }, { "cell_type": "code", - "source": [ - "import numpy as np\n", - "\n", - "# Let's get the mean and standard deviation.\n", - "data = np.array(dataset[\"train\"][\"inputs\"], np.float32)\n", - "mean = data.mean(axis=(0, 1, 2))[None, None, None, :]\n", - "std = data.std(axis=(0, 1, 2))[None, None, None, :]\n", - "\n", - "print(f\"mean: {mean.shape}\")\n", - "print(f\"std: {std.shape}\")" - ], + "execution_count": null, + "id": "YkOwsJBuYIHg", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -465,50 +456,42 @@ "id": "YkOwsJBuYIHg", "outputId": "4a3ed264-5dca-4169-bee9-9e4ecaea5409" }, - "id": "YkOwsJBuYIHg", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "mean: (1, 1, 1, 52)\n", "std: (1, 1, 1, 52)\n" ] } + ], + "source": [ + "import numpy as np\n", + "\n", + "# Let's get the mean and standard deviation.\n", + "data = np.array(dataset[\"train\"][\"inputs\"], np.float32)\n", + "mean = data.mean(axis=(0, 1, 2))[None, None, None, :]\n", + "std = data.std(axis=(0, 1, 2))[None, None, None, :]\n", + "\n", + "print(f\"mean: {mean.shape}\")\n", + "print(f\"std: {std.shape}\")" ] }, { "cell_type": "markdown", - "source": [ - "Let's see how the normalization works for a sample of an example's inputs." - ], + "id": "meHaHpxW-zt5", "metadata": { "id": "meHaHpxW-zt5" }, - "id": "meHaHpxW-zt5" + "source": [ + "Let's see how the normalization works for a sample of an example's inputs." + ] }, { "cell_type": "code", - "source": [ - "import torch\n", - "\n", - "from weather.model import Normalization\n", - "\n", - "normalization = Normalization(mean, std)\n", - "\n", - "sample = lambda x: x[0, 0, 0, 10:15].detach().numpy()\n", - "\n", - "print(f\"mean: {sample(normalization.mean)}\")\n", - "print(f\"std: {sample(normalization.std)}\")\n", - "print(\"-\" * 40)\n", - "\n", - "example = dataset[\"train\"][0]\n", - "example_inputs = torch.as_tensor([example[\"inputs\"]])\n", - "normalized_inputs = normalization(example_inputs)\n", - "print(f\"inputs: {sample(example_inputs)}\")\n", - "print(f\"normalized: {sample(normalized_inputs)}\")" - ], + "execution_count": null, + "id": "EUT8fowo-_Bv", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -516,12 +499,10 @@ "id": "EUT8fowo-_Bv", "outputId": "5fc2a64b-7dde-4803-9d21-34264bcf93f5" }, - "id": "EUT8fowo-_Bv", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "mean: [2202.3132 2355.514 2328.052 2470.9158 2687.0806]\n", "std: [256.82922 324.5936 332.1437 480.68338 351.21927]\n", @@ -530,10 +511,33 @@ "normalized: [0.36088872 0.48826003 0.6200569 0.6305278 0.76852113]\n" ] } + ], + "source": [ + "import torch\n", + "\n", + "from weather.model import Normalization\n", + "\n", + "normalization = Normalization(mean, std)\n", + "\n", + "sample = lambda x: x[0, 0, 0, 10:15].detach().numpy()\n", + "\n", + "print(f\"mean: {sample(normalization.mean)}\")\n", + "print(f\"std: {sample(normalization.std)}\")\n", + "print(\"-\" * 40)\n", + "\n", + "example = dataset[\"train\"][0]\n", + "example_inputs = torch.as_tensor([example[\"inputs\"]])\n", + "normalized_inputs = normalization(example_inputs)\n", + "print(f\"inputs: {sample(example_inputs)}\")\n", + "print(f\"normalized: {sample(normalized_inputs)}\")" ] }, { "cell_type": "markdown", + "id": "Idvef7Id49vE", + "metadata": { + "id": "Idvef7Id49vE" + }, "source": [ "After applying the `Normalization` layer, we get small numbers much closer to the range within -1 and 1, they don't have to be _exactly_ within the range, just close enough.\n", "\n", @@ -542,25 +546,12 @@ "We still want to pass our inputs in a channels-last format and want the predictions back as channels-last for convenience, but we must convert them to channels-first for PyTorch convolutional layers to work.\n", "\n", "In [`weather/model.py`](../serving/weather-model/weather/model.py) we define the `MoveDim` layer, which works similar to [`torch.movedim`](https://pytorch.org/docs/stable/generated/torch.movedim.html) so the model can move the channels dimension as needed.\n" - ], - "metadata": { - "id": "Idvef7Id49vE" - }, - "id": "Idvef7Id49vE" + ] }, { "cell_type": "code", - "source": [ - "from weather.model import MoveDim\n", - "\n", - "# We move the channels/last dimension (-1) to the second index (1),\n", - "# since the first (0) is for the batch dimension.\n", - "to_channels_first = MoveDim(-1, 1)\n", - "channels_first = to_channels_first(normalized_inputs)\n", - "\n", - "print(f\"normalized: {normalized_inputs.shape}\")\n", - "print(f\"channels-first: {channels_first.shape}\")" - ], + "execution_count": null, + "id": "AkrmnehOCuol", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -568,21 +559,34 @@ "id": "AkrmnehOCuol", "outputId": "0a41f8d0-8f61-4946-92e1-67e33e3eddf1" }, - "id": "AkrmnehOCuol", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "normalized: torch.Size([1, 5, 5, 52])\n", "channels-first: torch.Size([1, 52, 5, 5])\n" ] } + ], + "source": [ + "from weather.model import MoveDim\n", + "\n", + "# We move the channels/last dimension (-1) to the second index (1),\n", + "# since the first (0) is for the batch dimension.\n", + "to_channels_first = MoveDim(-1, 1)\n", + "channels_first = to_channels_first(normalized_inputs)\n", + "\n", + "print(f\"normalized: {normalized_inputs.shape}\")\n", + "print(f\"channels-first: {channels_first.shape}\")" ] }, { "cell_type": "markdown", + "id": "6JpbxntkEEtv", + "metadata": { + "id": "6JpbxntkEEtv" + }, "source": [ "The model then passes the data through a\n", "[2D Convolutional layer](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) for downsampling, and then through a\n", @@ -590,14 +594,28 @@ "We used a [`ReLU`](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) activation function inbetween all hidden layers since it's typically a good general purpose activation function.\n", "\n", "The Conv2D and DeConv2D layers form a very simple Fully Convolutional Network architecture, and since we're using the same _kernel size_ for both we get the same `(width, height)` as outputs." - ], - "metadata": { - "id": "6JpbxntkEEtv" - }, - "id": "6JpbxntkEEtv" + ] }, { "cell_type": "code", + "execution_count": null, + "id": "3Ima73TIEG1z", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "3Ima73TIEG1z", + "outputId": "4929396e-74c2-4cd3-9fdf-31e12117f064" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "FCN outputs: torch.Size([1, 128, 5, 5])\n" + ] + } + ], "source": [ "num_inputs = 52\n", "num_hidden1 = 64\n", @@ -613,44 +631,22 @@ "\n", "fcn_outputs = fully_convolutional_layers(channels_first)\n", "print(f\"FCN outputs: {fcn_outputs.shape}\")" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "3Ima73TIEG1z", - "outputId": "4929396e-74c2-4cd3-9fdf-31e12117f064" - }, - "id": "3Ima73TIEG1z", - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "FCN outputs: torch.Size([1, 128, 5, 5])\n" - ] - } ] }, { "cell_type": "markdown", - "source": [ - "Now, let's convert the results back into channels-last format with `MoveDim`." - ], + "id": "TkRDEANqFoLd", "metadata": { "id": "TkRDEANqFoLd" }, - "id": "TkRDEANqFoLd" + "source": [ + "Now, let's convert the results back into channels-last format with `MoveDim`." + ] }, { "cell_type": "code", - "source": [ - "to_channels_last = MoveDim(1, -1)\n", - "channels_last = to_channels_last(fcn_outputs)\n", - "\n", - "print(f\"channels-last: {channels_last.shape}\")" - ], + "execution_count": null, + "id": "-oAMnWtfFuzr", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -658,44 +654,37 @@ "id": "-oAMnWtfFuzr", "outputId": "7a436c24-4ab0-4040-dd6f-cbbf167d03ff" }, - "id": "-oAMnWtfFuzr", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "channels-last: torch.Size([1, 5, 5, 128])\n" ] } + ], + "source": [ + "to_channels_last = MoveDim(1, -1)\n", + "channels_last = to_channels_last(fcn_outputs)\n", + "\n", + "print(f\"channels-last: {channels_last.shape}\")" ] }, { "cell_type": "markdown", - "source": [ - "For the last layer, we use a [`Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer with the number of outputs we want.\n", - "Since we can't have negative precipitation, we passed the model's outputs through a final `ReLU` activation function." - ], + "id": "7x_OkkNvGabm", "metadata": { "id": "7x_OkkNvGabm" }, - "id": "7x_OkkNvGabm" + "source": [ + "For the last layer, we use a [`Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer with the number of outputs we want.\n", + "Since we can't have negative precipitation, we passed the model's outputs through a final `ReLU` activation function." + ] }, { "cell_type": "code", - "source": [ - "num_outputs = 2\n", - "\n", - "linear = torch.nn.Linear(num_hidden2, num_outputs)\n", - "relu = torch.nn.ReLU()\n", - "\n", - "with torch.no_grad():\n", - " raw_predictions = linear(channels_last)\n", - " predictions = relu(raw_predictions)\n", - "\n", - "print(f\"predictions: {predictions.shape}\")\n", - "print(predictions[0, 0, 0])" - ], + "execution_count": null, + "id": "q0T8CpEaGJew", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -703,21 +692,33 @@ "id": "q0T8CpEaGJew", "outputId": "227fe92d-2f86-4160-aad9-971d08032a51" }, - "id": "q0T8CpEaGJew", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "predictions: torch.Size([1, 5, 5, 2])\n", "tensor([0.0650, 0.0010])\n" ] } + ], + "source": [ + "num_outputs = 2\n", + "\n", + "linear = torch.nn.Linear(num_hidden2, num_outputs)\n", + "relu = torch.nn.ReLU()\n", + "\n", + "with torch.no_grad():\n", + " raw_predictions = linear(channels_last)\n", + " predictions = relu(raw_predictions)\n", + "\n", + "print(f\"predictions: {predictions.shape}\")\n", + "print(predictions[0, 0, 0])" ] }, { "cell_type": "markdown", + "id": "CDQIGsp24EX9", "metadata": { "id": "CDQIGsp24EX9" }, @@ -734,17 +735,12 @@ "To create a `WeatherModel`, we have to pass it a `WeatherConfig`.\n", "The `WeatherConfig` contains all the model's hyperparameters, and we must also pass the _mean_ and _standard deviation_ from the training dataset for the normalization layer.\n", "We defined `WeatherModel.create` which takes in the training dataset inputs and returns us a `WeatherModel` with the right `WeatherConfig`." - ], - "id": "CDQIGsp24EX9" + ] }, { "cell_type": "code", - "source": [ - "from weather.model import WeatherModel\n", - "\n", - "model = WeatherModel.create(dataset[\"train\"][\"inputs\"])\n", - "print(model)" - ], + "execution_count": null, + "id": "h0bzkGqwo-Ic", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -752,12 +748,10 @@ "id": "h0bzkGqwo-Ic", "outputId": "4a0f8622-e3fd-49cf-9eb2-c9e2777174b0" }, - "id": "h0bzkGqwo-Ic", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "WeatherModel(\n", " (layers): Sequential(\n", @@ -774,42 +768,31 @@ ")\n" ] } + ], + "source": [ + "from weather.model import WeatherModel\n", + "\n", + "model = WeatherModel.create(dataset[\"train\"][\"inputs\"])\n", + "print(model)" ] }, { "cell_type": "markdown", + "id": "6iS60sGCJczT", + "metadata": { + "id": "6iS60sGCJczT" + }, "source": [ "The model outputs a `{'loss': torch.Tensor, 'logits': torch.Tensor}` dictionary during training, and a `{'logits': torch.Tensor}` dictionary during predictions.\n", "This is what 🤗 Transformers expect for [model outputs](https://huggingface.co/docs/transformers/main/en/main_classes/output).\n", "\n", "Remember that we _must_ pass a _batch_ of inputs to the model, not a single input." - ], - "metadata": { - "id": "6iS60sGCJczT" - }, - "id": "6iS60sGCJczT" + ] }, { "cell_type": "code", - "source": [ - "example = dataset[\"test\"]\n", - "inputs_batch = torch.as_tensor(example[\"inputs\"][:1])\n", - "labels_batch = torch.as_tensor(example[\"labels\"][:1])\n", - "\n", - "# We pass the labels as well to get the loss, but it's optional.\n", - "# If we don't pass the labels, we simply won't get the loss.\n", - "# The predictions are under the 'logits' key.\n", - "with torch.no_grad():\n", - " predictions = model(inputs_batch, labels_batch)\n", - "\n", - "print(f\"inputs: {inputs_batch.shape}\")\n", - "print(f\"labels: {labels_batch.shape}\")\n", - "print(f\"loss: {predictions['loss']}\")\n", - "print(f\"predictions: {predictions['logits'].shape}\")\n", - "print(\"-\" * 40)\n", - "print(f\"sample labels: {labels_batch[0, 0, 0]}\")\n", - "print(f\"sample predictions: {predictions['logits'][0, 0, 0]}\")" - ], + "execution_count": null, + "id": "gKwdukOzJeNA", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -817,12 +800,10 @@ "id": "gKwdukOzJeNA", "outputId": "63287630-c21b-4b21-c6b0-168423fd2746" }, - "id": "gKwdukOzJeNA", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "inputs: torch.Size([1, 5, 5, 52])\n", "labels: torch.Size([1, 5, 5, 2])\n", @@ -833,21 +814,44 @@ "sample predictions: tensor([0.0797, 0.0000])\n" ] } + ], + "source": [ + "example = dataset[\"test\"]\n", + "inputs_batch = torch.as_tensor(example[\"inputs\"][:1])\n", + "labels_batch = torch.as_tensor(example[\"labels\"][:1])\n", + "\n", + "# We pass the labels as well to get the loss, but it's optional.\n", + "# If we don't pass the labels, we simply won't get the loss.\n", + "# The predictions are under the 'logits' key.\n", + "with torch.no_grad():\n", + " predictions = model(inputs_batch, labels_batch)\n", + "\n", + "print(f\"inputs: {inputs_batch.shape}\")\n", + "print(f\"labels: {labels_batch.shape}\")\n", + "print(f\"loss: {predictions['loss']}\")\n", + "print(f\"predictions: {predictions['logits'].shape}\")\n", + "print(\"-\" * 40)\n", + "print(f\"sample labels: {labels_batch[0, 0, 0]}\")\n", + "print(f\"sample predictions: {predictions['logits'][0, 0, 0]}\")" ] }, { "cell_type": "markdown", - "source": [ - "These predictions don't look great because we haven't trained our model.\n", - "Fortunately, since we've made our model compatible with 🤗 Transformers, we can simply use [`Trainer`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer), which takes care of all the training steps, automatically optimizes the whole process, and uses accelerators like GPUs if available." - ], + "id": "cxyoRnNlzsYu", "metadata": { "id": "cxyoRnNlzsYu" }, - "id": "cxyoRnNlzsYu" + "source": [ + "These predictions don't look great because we haven't trained our model.\n", + "Fortunately, since we've made our model compatible with 🤗 Transformers, we can simply use [`Trainer`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer), which takes care of all the training steps, automatically optimizes the whole process, and uses accelerators like GPUs if available." + ] }, { "cell_type": "markdown", + "id": "xG6PnXhfLzxO", + "metadata": { + "id": "xG6PnXhfLzxO" + }, "source": [ "## 👟 Train the model\n", "\n", @@ -857,53 +861,24 @@ "\n", "Then we pass the model, the `TrainingArguments`, and the training and testing datasets into the `Trainer`.\n", "Finally we can train the model with [`Trainer.train`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train)." - ], - "metadata": { - "id": "xG6PnXhfLzxO" - }, - "id": "xG6PnXhfLzxO" + ] }, { "cell_type": "code", - "source": [ - "from transformers import TrainingArguments, Trainer\n", - "\n", - "epochs = 5\n", - "batch_size = 512\n", - "\n", - "# Define our training job.\n", - "training_args = TrainingArguments(\n", - " output_dir=\"checkpoints\",\n", - " per_device_train_batch_size=batch_size,\n", - " per_device_eval_batch_size=batch_size,\n", - " num_train_epochs=epochs,\n", - " logging_strategy=\"epoch\",\n", - " evaluation_strategy=\"epoch\",\n", - ")\n", - "trainer = Trainer(\n", - " model,\n", - " training_args,\n", - " train_dataset=dataset[\"train\"],\n", - " eval_dataset=dataset[\"test\"],\n", - ")\n", - "\n", - "# Run the training job.\n", - "trainer.train()" - ], + "execution_count": null, + "id": "x4ta1oIsMveF", "metadata": { - "id": "x4ta1oIsMveF", "colab": { "base_uri": "https://localhost:8080/", "height": 825 }, + "id": "x4ta1oIsMveF", "outputId": "3a4e8674-dfa2-4cbf-8445-c3bcccfe4769" }, - "id": "x4ta1oIsMveF", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "PyTorch: setting up devices\n", "The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).\n", @@ -919,11 +894,7 @@ ] }, { - "output_type": "display_data", "data": { - "text/plain": [ - "" - ], "text/html": [ "\n", "
\n", @@ -967,13 +938,17 @@ " \n", " \n", "

" + ], + "text/plain": [ + "" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "***** Running Evaluation *****\n", " Num examples = 341\n", @@ -998,48 +973,70 @@ ] }, { - "output_type": "execute_result", "data": { "text/plain": [ "TrainOutput(global_step=30, training_loss=1.2740394274393718, metrics={'train_runtime': 23.7216, 'train_samples_per_second': 646.878, 'train_steps_per_second': 1.265, 'total_flos': 0.0, 'train_loss': 1.2740394274393718, 'epoch': 5.0})" ] }, + "execution_count": 31, "metadata": {}, - "execution_count": 31 + "output_type": "execute_result" } + ], + "source": [ + "from transformers import TrainingArguments, Trainer\n", + "\n", + "epochs = 5\n", + "batch_size = 512\n", + "\n", + "# Define our training job.\n", + "training_args = TrainingArguments(\n", + " output_dir=\"checkpoints\",\n", + " per_device_train_batch_size=batch_size,\n", + " per_device_eval_batch_size=batch_size,\n", + " num_train_epochs=epochs,\n", + " logging_strategy=\"epoch\",\n", + " evaluation_strategy=\"epoch\",\n", + ")\n", + "trainer = Trainer(\n", + " model,\n", + " training_args,\n", + " train_dataset=dataset[\"train\"],\n", + " eval_dataset=dataset[\"test\"],\n", + ")\n", + "\n", + "# Run the training job.\n", + "trainer.train()" ] }, { "cell_type": "markdown", - "source": [ - "> 💡 Both losses should go down every epoch, and they should be roughly similar.\n", - "> If the training loss goes down, but the testing loss stays flat or goes up, it might be a sign that the model is [overfitting](https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting), meaning that it's memorizing the training dataset instead of learning to generalize." - ], + "id": "jPFCmhruOvjB", "metadata": { "id": "jPFCmhruOvjB" }, - "id": "jPFCmhruOvjB" + "source": [ + "> 💡 Both losses should go down every epoch, and they should be roughly similar.\n", + "> If the training loss goes down, but the testing loss stays flat or goes up, it might be a sign that the model is [overfitting](https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting), meaning that it's memorizing the training dataset instead of learning to generalize." + ] }, { "cell_type": "markdown", + "id": "_AxB_p2-z4UH", + "metadata": { + "id": "_AxB_p2-z4UH" + }, "source": [ "## 💾 Save and load the model\n", "\n", "After the model has finished training, we can save it with [`Trainer.save_model`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.save_model).\n", "\n" - ], - "metadata": { - "id": "_AxB_p2-z4UH" - }, - "id": "_AxB_p2-z4UH" + ] }, { "cell_type": "code", - "source": [ - "trainer.save_model(\"model\")\n", - "\n", - "!ls -lh model" - ], + "execution_count": null, + "id": "NPLnvRydOik0", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -1047,12 +1044,10 @@ "id": "NPLnvRydOik0", "outputId": "c788cf33-ec67-4612-9f44-1262dc872625" }, - "id": "NPLnvRydOik0", - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "Saving model checkpoint to model\n", "Configuration saved in model/config.json\n", @@ -1060,8 +1055,8 @@ ] }, { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "total 420K\n", "-rw-r--r-- 1 root root 3.4K Jan 11 21:33 config.json\n", @@ -1069,10 +1064,16 @@ "-rw-r--r-- 1 root root 3.4K Jan 11 21:33 training_args.bin\n" ] } + ], + "source": [ + "trainer.save_model(\"model\")\n", + "\n", + "!ls -lh model" ] }, { "cell_type": "markdown", + "id": "icxhkboQA_o5", "metadata": { "id": "icxhkboQA_o5" }, @@ -1080,12 +1081,12 @@ "Now that we have a trained model, we can save it and load it anywhere else.\n", "We can load a 🤗 Transformers model with [`PreTrainedModel.from_pretrained`](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained), in our case with `WeatherModel.from_pretrained`.\n", "This loads all the model's hyperparameters as well as the _mean_ and _standard deviation_ for the normalization layer." - ], - "id": "icxhkboQA_o5" + ] }, { "cell_type": "code", "execution_count": null, + "id": "xsxX2Mb-CwWk", "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -1095,8 +1096,8 @@ }, "outputs": [ { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "loading configuration file model/config.json\n", "Model config WeatherConfig {\n", @@ -1244,8 +1245,8 @@ ] }, { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "WeatherModel(\n", " (layers): Sequential(\n", @@ -1268,11 +1269,11 @@ "\n", "model = WeatherModel.from_pretrained(\"model\")\n", "print(model)" - ], - "id": "xsxX2Mb-CwWk" + ] }, { "cell_type": "markdown", + "id": "IO73AYtsCIQ_", "metadata": { "id": "IO73AYtsCIQ_" }, @@ -1290,39 +1291,36 @@ "\n", "The model and trainer are defined in the [`serving/weather-model`](../serving/weather-model) module.\n", "To run it in Vertex AI, we must build the package, copy it to Cloud Storage, and launch a custom training job with [`CustomPythonPackageTrainingJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.CustomPythonPackageTrainingJob)." - ], - "id": "IO73AYtsCIQ_" + ] }, { "cell_type": "code", - "source": [ - "# Build the `weather-model` package.\n", - "!python -m build serving/weather-model" - ], + "execution_count": null, + "id": "v1SZt1iA2Wrh", "metadata": { "id": "v1SZt1iA2Wrh" }, - "execution_count": null, "outputs": [], - "id": "v1SZt1iA2Wrh" + "source": [ + "# Build the `weather-model` package.\n", + "!python -m build serving/weather-model" + ] }, { "cell_type": "code", - "source": [ - "!ls -lh serving/weather-model/dist" - ], + "execution_count": null, + "id": "y4F1_eA32Wrh", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, - "outputId": "2f2b4dc2-a287-4822-caed-7f8115246d7d", - "id": "y4F1_eA32Wrh" + "id": "y4F1_eA32Wrh", + "outputId": "2f2b4dc2-a287-4822-caed-7f8115246d7d" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "total 16K\n", "-rw-r--r-- 1 root root 5.9K Jan 11 18:29 weather_model-1.0.0-py3-none-any.whl\n", @@ -1330,36 +1328,39 @@ ] } ], - "id": "y4F1_eA32Wrh" + "source": [ + "!ls -lh serving/weather-model/dist" + ] }, { "cell_type": "code", - "source": [ - "# Stage the `weather-model` package in Cloud Storage.\n", - "!gcloud storage cp serving/weather-model/dist/weather-model-1.0.0.tar.gz gs://{bucket}/weather/" - ], + "execution_count": null, + "id": "JA1k9ky02dsx", "metadata": { "id": "JA1k9ky02dsx" }, - "id": "JA1k9ky02dsx", - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "# Stage the `weather-model` package in Cloud Storage.\n", + "!gcloud storage cp serving/weather-model/dist/weather-model-1.0.0.tar.gz gs://{bucket}/weather/" + ] }, { "cell_type": "markdown", + "id": "yk9X4YQcDPpR", + "metadata": { + "id": "yk9X4YQcDPpR" + }, "source": [ "In Vertex AI, we can access Cloud Storage files directly as if they were local files via Cloud Storage FUSE.\n", "Cloud Storage files are available under `/gcs` followed by your bucket and file path.\n", "To learn more, see the [Cloud Storage as a File System in AI Training](https://cloud.google.com/blog/products/ai-machine-learning/cloud-storage-file-system-ai-training) blog post." - ], - "metadata": { - "id": "yk9X4YQcDPpR" - }, - "id": "yk9X4YQcDPpR" + ] }, { "cell_type": "code", "execution_count": null, + "id": "Ny4x99GiS2Lm", "metadata": { "id": "Ny4x99GiS2Lm" }, @@ -1381,7 +1382,7 @@ " display_name=\"weather-forecasting\",\n", " python_package_gcs_uri=f\"gs://{bucket}/weather/weather-model-1.0.0.tar.gz\",\n", " python_module_name=\"weather.trainer\",\n", - " container_uri=\"us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.2-8.py310:latest\",\n", + " container_uri=\"us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.2-4.py310:latest\",\n", ")\n", "job.run(\n", " machine_type=\"n1-highmem-8\",\n", @@ -1394,31 +1395,30 @@ " ],\n", " timeout=timeout_min * 60, # in seconds\n", ")" - ], - "id": "Ny4x99GiS2Lm" + ] }, { "cell_type": "markdown", + "id": "zw_kcyw4gOLF", "metadata": { "id": "zw_kcyw4gOLF" }, "source": [ "> 💡 Look at your Vertex AI training jobs: https://console.cloud.google.com/vertex-ai/training/custom-jobs" - ], - "id": "zw_kcyw4gOLF" + ] }, { "cell_type": "markdown", + "id": "79RnF-lYBRTS", + "metadata": { + "id": "79RnF-lYBRTS" + }, "source": [ "# ⛳️ What's next?\n", "\n", "* [![Open in Colab](https://github.com/googlecolab/open_in_colab/raw/main/images/icon16.png) **🔮 Model predictions**](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/notebooks/4-predictions.ipynb):\n", " Get predictions from the model with data it has never seen before." - ], - "metadata": { - "id": "79RnF-lYBRTS" - }, - "id": "79RnF-lYBRTS" + ] } ], "metadata": { diff --git a/people-and-planet-ai/weather-forecasting/serving/weather-model/pyproject.toml b/people-and-planet-ai/weather-forecasting/serving/weather-model/pyproject.toml index 43c03683ccd..32c96cbb4f2 100644 --- a/people-and-planet-ai/weather-forecasting/serving/weather-model/pyproject.toml +++ b/people-and-planet-ai/weather-forecasting/serving/weather-model/pyproject.toml @@ -18,7 +18,7 @@ name = "weather-model" version = "1.0.0" dependencies = [ "datasets==4.0.0", - "torch==2.8.0", # make sure this matches the `container_uri` in `notebooks/3-training.ipynb` + "torch==2.4.0", # make sure this matches the `container_uri` in `notebooks/3-training.ipynb` "transformers==5.0.0", ]