+ +

Polynomial Regression Using TensorFlow 2.x

+ +

I have a similar post titled Polynomial Regression Using Tensorflow that used tensorflow.compat.v1 (Which still works as of TF 2.16). But, I thought it would be nicer to redo it with newer TF versions.

+ +

I will be skipping all the introductions about polynomial regression and jumping straight to the code. Personally, I prefer using scikit-learn for this task.

+ +

Position vs Salary Dataset

+ +

Again, we will be using https://drive.google.com/file/d/1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9/view (Salary vs Position Dataset)

+ +

If you are in a Python Notebook environment like Kaggle or Google Colaboratory, you can simply run:

+ +

!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9' -O data.csv
+

+ +

Code

+ +

If you just want to copy-paste the code, scroll to the bottom for the entire snippet. Here I will try and walk through setting up code for a 3rd-degree (cubic) polynomial

+ +

Imports

+ +

import pandas as pd
+import tensorflow as tf
+import matplotlib.pyplot as plt
+import numpy as np
+

+ +

Reading the Dataset

+ +

df = pd.read_csv("data.csv")
+

+ +

Variables and Constants

+ +

Here, we initialize the X and Y values as constants, since they are not going to change. The coefficients are defined as variables.

+ +

X = tf.constant(df["Level"], dtype=tf.float32)
+Y = tf.constant(df["Salary"], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(4)]
+

+ +

Here, X and Y are the values from our dataset. We initialize the coefficients for the equations as small random values.

+ +

These coefficients are evaluated by Tensorflow's tf.math.poyval function which returns the n-th order polynomial based on how many coefficients are passed. Since our list of coefficients contains 4 different variables, it will be evaluated as:

+ +

y = (x**3)*coefficients[3] + (x**2)*coefficients[2] + (x**1)*coefficients[1] (x**0)*coefficients[0]
+

+ +

Which is equivalent to the general cubic equation:

+ +

+ +$$ +y = ax^3 + bx^2 + cx + d +$$ + +### Optimizer Selection & Training +

+ +

optimizer = tf.keras.optimizers.Adam(learning_rate=0.3)
+num_epochs = 10_000
+
+for epoch in range(num_epochs):
+    with tf.GradientTape() as tape:
+        y_pred = tf.math.polyval(coefficients, X)
+        loss = tf.reduce_mean(tf.square(y - y_pred))
+    grads = tape.gradient(loss, coefficients)
+    optimizer.apply_gradients(zip(grads, coefficients))
+    if (epoch+1) % 1000 == 0:
+        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}"
+

+ +

+ + +In TensorFlow 1, we would have been using `tf.Session` instead. + +Here we are using `GradientTape()` instead, to keep track of the loss evaluation and coefficients. This is crucial, as our optimizer needs these gradients to be able to optimize our coefficients. + +Our loss function is Mean Squared Error (MSE) + +$$ += \frac{1}{n}\sum_{i=1}^{n} (Y_i - \^{Y_i}) +$$ + +Where $\^{Y_i}$ is the predicted value and $Y_i$ is the actual value + +### Plotting Final Coefficients +

+ +

final_coefficients = [c.numpy() for c in coefficients]
+print("Final Coefficients:", final_coefficients)
+
+plt.plot(df["Level"], df["Salary"], label="Original Data")
+plt.plot(df["Level"],[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df["Level"]])
+plt.ylabel('Salary')
+plt.xlabel('Position')
+plt.title("Salary vs Position")
+plt.show()
+

+ +

+ + + +## Code Snippet for a Polynomial of Degree N + +### Using Gradient Tape + +This should work regardless of the Keras backend version (2 or 3) +

+ +

import tensorflow as tf
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+
+df = pd.read_csv("data.csv")
+
+############################
+## Change Parameters Here ##
+############################
+x_column = "Level"         #
+y_column = "Salary"        #
+degree = 2                 #
+learning_rate = 0.3        #
+num_epochs = 25_000        #
+############################
+
+X = tf.constant(df[x_column], dtype=tf.float32)
+Y = tf.constant(df[y_column], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]
+
+optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+
+for epoch in range(num_epochs):
+    with tf.GradientTape() as tape:
+        y_pred = tf.math.polyval(coefficients, X)
+        loss = tf.reduce_mean(tf.square(Y - y_pred))
+    grads = tape.gradient(loss, coefficients)
+    optimizer.apply_gradients(zip(grads, coefficients))
+    if (epoch+1) % 1000 == 0:
+        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}")
+
+final_coefficients = [c.numpy() for c in coefficients]
+print("Final Coefficients:", final_coefficients)
+
+print("Final Equation:", end=" ")
+for i in range(degree+1):
+  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")
+
+plt.plot(X, Y, label="Original Data")
+plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]]), label="Our Poynomial"
+plt.ylabel(y_column)
+plt.xlabel(x_column)
+plt.title(f"{x_column} vs {y_column}")
+plt.legend()
+plt.show()
+

+ +

+ + +### Without Gradient Tape + +This relies on the Optimizer's `minimize` function and uses the `var_list` parameter to update the variables. + +This will not work with Keras 3 backend in TF 2.16.0 and above unless you switch to the legacy backend. +

+ +

import tensorflow as tf
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+
+df = pd.read_csv("data.csv")
+
+############################
+## Change Parameters Here ##
+############################
+x_column = "Level"         #
+y_column = "Salary"        #
+degree = 2                 #
+learning_rate = 0.3        #
+num_epochs = 25_000        #
+############################
+
+X = tf.constant(df[x_column], dtype=tf.float32)
+Y = tf.constant(df[y_column], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]
+
+optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+
+def loss_function():
+  pred_y = tf.math.polyval(coefficients, X)
+  return tf.reduce_mean(tf.square(pred_y - Y))
+
+for epoch in range(num_epochs):
+    optimizer.minimize(loss_function, var_list=coefficients)
+    if (epoch+1) % 1000 == 0:
+        current_loss = loss_function().numpy()
+        print(f"Epoch {epoch+1}: Training Loss: {current_loss}")
+
+final_coefficients = coefficients.numpy()
+print("Final Coefficients:", final_coefficients)
+
+print("Final Equation:", end=" ")
+for i in range(degree+1):
+  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")
+
+plt.plot(X, Y, label="Original Data")
+plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]], label="Our Polynomial")
+plt.ylabel(y_column)
+plt.xlabel(x_column)
+plt.legend()
+plt.title(f"{x_column} vs {y_column}")
+plt.show()
+

+ +

+ + + +As always, remember to tweak the parameters and choose the correct model for the job. A polynomial regression model might not even be the best model for this particular dataset. + +## Further Programming + +How would you modify this code to use another type of nonlinear regression? Say, $ y = ab^x $ + +Hint: Your loss calculation would be similar to: +

+ +

bx = tf.pow(coefficients[1], X)
+pred_y = tf.math.multiply(coefficients[0], bx)
+loss = tf.reduce_mean(tf.square(pred_y - Y))
+

+ +

If you have scrolled this far, consider subscribing to my mailing list here. You can subscribe to either a specific type of post you are interested in, or subscribe to everything with the "Everything" list.

+ +

Position vs Salary Dataset

Code

Imports

Reading the Dataset

Variables and Constants

Recent Posts

Polynomial Regression Using TensorFlow 2.x

Position vs Salary Dataset

Code

Imports

Reading the Dataset

Variables and Constants