{ "cells": [ { "cell_type": "markdown", "id": "17ebc6cd", "metadata": {}, "source": [ "# Assignment 3: worked solution\n", "\n", "This solution compares model complexity and regularization using training data only.\n", "\n", "1. evaluate one polynomial degree and penalty strength with five-fold cross-validation;\n", "2. compare the five supplied settings;\n", "3. diagnose underfitting, overfitting, and the effect of regularization.\n", "\n", "Your code may look different and still be correct." ] }, { "cell_type": "code", "execution_count": 1, "id": "12f42073", "metadata": { "execution": { "iopub.execute_input": "2026-08-21T13:11:22.448663Z", "iopub.status.busy": "2026-08-21T13:11:22.448473Z", "iopub.status.idle": "2026-08-21T13:11:24.616806Z", "shell.execute_reply": "2026-08-21T13:11:24.615816Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "training rows: 52\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "from sklearn.linear_model import LinearRegression, Ridge\n", "from sklearn.model_selection import KFold, cross_validate, train_test_split\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import PolynomialFeatures, StandardScaler\n", "\n", "# Recreate the same demonstration data and training split as the Day 3 notebook.\n", "rng = np.random.default_rng(12)\n", "\n", "\n", "def true_function(x):\n", " \"\"\"The pattern used to create the demonstration data.\"\"\"\n", " return 2 + 0.8 * x - 0.7 * x**2 + 0.15 * x**3\n", "\n", "\n", "x = np.linspace(-3, 3, 70)\n", "y = true_function(x) + rng.normal(loc=0, scale=1.4, size=len(x))\n", "X = x.reshape(-1, 1)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.25, random_state=42\n", ")\n", "cv = KFold(n_splits=5, shuffle=True, random_state=42)\n", "\n", "\n", "def make_polynomial_model(degree, alpha=0.0):\n", " \"\"\"Create power columns, standardize them, and fit linear regression.\"\"\"\n", " regression = LinearRegression() if alpha == 0 else Ridge(alpha=alpha)\n", " return Pipeline([\n", " (\"powers\", PolynomialFeatures(degree=degree, include_bias=False)),\n", " (\"scale\", StandardScaler()),\n", " (\"regression\", regression),\n", " ])\n", "\n", "\n", "assignment_settings = [\n", " {\"degree\": 1, \"alpha\": 0.0},\n", " {\"degree\": 3, \"alpha\": 0.0},\n", " {\"degree\": 12, \"alpha\": 0.0},\n", " {\"degree\": 12, \"alpha\": 1.0},\n", " {\"degree\": 12, \"alpha\": 100.0},\n", "]\n", "\n", "print(\"training rows:\", len(X_train))" ] }, { "cell_type": "markdown", "id": "24875c5d", "metadata": {}, "source": [ "## Task 1: evaluate one setting\n", "\n", "`cross_validate` returns negative RMSE because scikit-learn expects a larger score to be better.\n", "Negating those values gives the usual positive RMSE, where lower is better." ] }, { "cell_type": "code", "execution_count": 2, "id": "a3dc5702", "metadata": { "execution": { "iopub.execute_input": "2026-08-21T13:11:24.618931Z", "iopub.status.busy": "2026-08-21T13:11:24.618594Z", "iopub.status.idle": "2026-08-21T13:11:24.644011Z", "shell.execute_reply": "2026-08-21T13:11:24.643255Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'degree': 1, 'alpha': 0.0, 'training_RMSE': np.float64(2.2952313935945274), 'CV_RMSE': np.float64(2.4017266820862364), 'generalization_gap': np.float64(0.10649528849170897)}\n" ] } ], "source": [ "def evaluate_setting(degree, alpha):\n", " \"\"\"Return training and CV errors for one model setting.\"\"\"\n", " model = make_polynomial_model(degree, alpha)\n", " scores = cross_validate(\n", " model,\n", " X_train,\n", " y_train,\n", " cv=cv,\n", " scoring=\"neg_root_mean_squared_error\",\n", " return_train_score=True,\n", " )\n", "\n", " training_rmse = -scores[\"train_score\"].mean()\n", " cv_rmse = -scores[\"test_score\"].mean()\n", "\n", " return {\n", " \"degree\": degree,\n", " \"alpha\": alpha,\n", " \"training_RMSE\": training_rmse,\n", " \"CV_RMSE\": cv_rmse,\n", " \"generalization_gap\": cv_rmse - training_rmse,\n", " }\n", "\n", "\n", "print(evaluate_setting(degree=1, alpha=0.0))" ] }, { "cell_type": "markdown", "id": "3a6d9aaa", "metadata": {}, "source": [ "## Task 2: compare the supplied settings\n", "\n", "Each settings dictionary can be passed into the function with `**setting`. We collect the\n", "result dictionaries, create a DataFrame, and sort by validation error rather than training\n", "error." ] }, { "cell_type": "code", "execution_count": 3, "id": "c74d247e", "metadata": { "execution": { "iopub.execute_input": "2026-08-21T13:11:24.645639Z", "iopub.status.busy": "2026-08-21T13:11:24.645442Z", "iopub.status.idle": "2026-08-21T13:11:24.762953Z", "shell.execute_reply": "2026-08-21T13:11:24.762257Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
degreealphatraining_RMSECV_RMSEgeneralization_gap
0121.01.1241.2000.076
130.01.2361.2990.064
2120.00.8661.5110.645
312100.01.9151.9260.011
410.02.2952.4020.106
\n", "
" ], "text/plain": [ " degree alpha training_RMSE CV_RMSE generalization_gap\n", "0 12 1.0 1.124 1.200 0.076\n", "1 3 0.0 1.236 1.299 0.064\n", "2 12 0.0 0.866 1.511 0.645\n", "3 12 100.0 1.915 1.926 0.011\n", "4 1 0.0 2.295 2.402 0.106" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "assignment_rows = [\n", " evaluate_setting(**setting)\n", " for setting in assignment_settings\n", "]\n", "\n", "assignment_results = (\n", " pd.DataFrame(assignment_rows)\n", " .sort_values(\"CV_RMSE\")\n", " .reset_index(drop=True)\n", ")\n", "\n", "display(assignment_results.round(3))" ] }, { "cell_type": "markdown", "id": "69c9ba71", "metadata": {}, "source": [ "## Check the solution\n", "\n", "The table should contain one row per candidate and place the lowest cross-validation RMSE first.\n", "The test set is not needed for this assignment." ] }, { "cell_type": "code", "execution_count": 4, "id": "42a1d3bf", "metadata": { "execution": { "iopub.execute_input": "2026-08-21T13:11:24.764574Z", "iopub.status.busy": "2026-08-21T13:11:24.764362Z", "iopub.status.idle": "2026-08-21T13:11:24.768241Z", "shell.execute_reply": "2026-08-21T13:11:24.767530Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "OK: the comparison table has the expected structure\n" ] } ], "source": [ "assert list(assignment_results.columns) == [\n", " \"degree\",\n", " \"alpha\",\n", " \"training_RMSE\",\n", " \"CV_RMSE\",\n", " \"generalization_gap\",\n", "]\n", "assert len(assignment_results) == len(assignment_settings)\n", "assert assignment_results[\"CV_RMSE\"].is_monotonic_increasing\n", "print(\"OK: the comparison table has the expected structure\")" ] }, { "cell_type": "markdown", "id": "38a25bab", "metadata": {}, "source": [ "## Reflection answer\n", "\n", "Degree 1 shows the clearest underfitting because both its training and cross-validation RMSE are\n", "high: the straight line cannot represent the curved pattern. Degree 12 without a penalty shows\n", "overfitting because its training RMSE is low but its cross-validation RMSE and generalization\n", "gap are much larger. Adding the moderate penalty $\\alpha=1$ increases training RMSE slightly\n", "but lowers cross-validation RMSE and makes the flexible model generalize better. The very strong\n", "penalty $\\alpha=100$ shrinks the model too much and returns it toward underfitting. We choose\n", "with cross-validation RMSE because training RMSE rewards models for fitting rows they have\n", "already seen." ] }, { "cell_type": "markdown", "id": "1a89579d", "metadata": {}, "source": [ "## Main ideas\n", "\n", "- Diagnose underfitting from high training and validation error together.\n", "- Diagnose overfitting from low training error and a larger validation error.\n", "- A moderate penalty can reduce variance in a flexible model.\n", "- Too much penalty increases bias.\n", "- Hyperparameters are chosen with validation data, not training error or repeated test checks." ] } ], "metadata": { "kernelspec": { "display_name": "ML Workshop (Python 3.11)", "language": "python", "name": "mlworkshop" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }