1 files changed, 242 insertions, 0 deletions
diff --git a/Content/posts/2024-03-21-Polynomial-Regression-in-TensorFlow-2.md b/Content/posts/2024-03-21-Polynomial-Regression-in-TensorFlow-2.md
new file mode 100644
index 0000000..6317175
--- /dev/null
+++ b/Content/posts/2024-03-21-Polynomial-Regression-in-TensorFlow-2.md
@@ -0,0 +1,242 @@
+---
+date: 2024-03-21 12:46
+description: Predicting n-th degree polynomials using TensorFlow 2.x
+tags: Tutorial, Tensorflow, Colab
+---
+
+# Polynomial Regression Using TensorFlow 2.x
+
+I have a similar post titled [Polynomial Regression Using Tensorflow](/posts/2019-12-16-TensorFlow-Polynomial-Regression.html) that used `tensorflow.compat.v1` (Which still works as of TF 2.16). But, I thought it would be nicer to redo it with newer TF versions. 
+
+I will be skipping all the introductions about polynomial regression and jumping straight to the code. Personally, I prefer using `scikit-learn` for this task.
+
+## Position vs Salary Dataset
+
+Again, we will be using https://drive.google.com/file/d/1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9/view (Salary vs Position Dataset)
+
+If you are in a Python Notebook environment like Kaggle or Google Colaboratory, you can simply run:
+```Termcap
+!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9' -O data.csv
+```
+
+## Code
+
+If you just want to copy-paste the code, scroll to the bottom for the entire snippet. Here I will try and walk through setting up code for a 3rd-degree (cubic) polynomial
+
+### Imports
+
+```python
+import pandas as pd
+import tensorflow as tf
+import matplotlib.pyplot as plt
+import numpy as np
+```
+
+### Reading the Dataset
+
+```python
+df = pd.read_csv("data.csv")
+```
+
+### Variables and Constants
+
+Here, we initialize the X and Y values as constants, since they are not going to change. The coefficients are defined as variables.
+
+```python
+X = tf.constant(df["Level"], dtype=tf.float32)
+Y = tf.constant(df["Salary"], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(4)]
+```
+
+Here, `X` and `Y` are the values from our dataset. We initialize the coefficients for the equations as small random values.
+
+These coefficients are evaluated by Tensorflow's `tf.math.poyval` function which returns the n-th order polynomial based on how many coefficients are passed. Since our list of coefficients contains 4 different variables, it will be evaluated as:
+
+```
+y = (x**3)*coefficients[3] + (x**2)*coefficients[2] + (x**1)*coefficients[1] (x**0)*coefficients[0]
+``` 
+
+Which is equivalent to the general cubic equation:
+
+<script src="https://cdn.jsdelivr.net/npm/mathjax@4.0.0-beta.4/tex-mml-chtml.js" id="MathJax-script"></script>
+<script src="https://cdn.jsdelivr.net/npm/mathjax@4.0.0-beta.4/input/tex/extensions/noerrors.js" charset="UTF-8"></script>
+
+$$
+y = ax^3 + bx^2 + cx + d
+$$
+
+### Optimizer Selection & Training
+
+```python
+optimizer = tf.keras.optimizers.Adam(learning_rate=0.3)
+num_epochs = 10_000
+
+for epoch in range(num_epochs):
+    with tf.GradientTape() as tape:
+        y_pred = tf.math.polyval(coefficients, X)
+        loss = tf.reduce_mean(tf.square(y - y_pred))
+    grads = tape.gradient(loss, coefficients)
+    optimizer.apply_gradients(zip(grads, coefficients))
+    if (epoch+1) % 1000 == 0:
+        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}"
+```
+
+In TensorFlow 1, we would have been using `tf.Session` instead. 
+
+Here we are using `GradientTape()` instead, to keep track of the loss evaluation and coefficients. This is crucial, as our optimizer needs these gradients to be able to optimize our coefficients. 
+
+Our loss function is Mean Squared Error (MSE):
+
+$$
+= \frac{1}{n} \sum_{i=1}^{n}{(Y\_i - \hat{Y\_i})^2}
+$$
+
+Where <math xmlns="http://www.w3.org/1998/Math/MathML"><mover><msub><mi>Y</mi><mi>i</mi></msub><mo stretchy="false" style="math-style:normal;math-depth:0;">^</mo></mover></math> is the predicted value and <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>Y</mi><mi>i</mi></msub></math> is the actual value
+
+### Plotting Final Coefficients
+
+```python
+final_coefficients = [c.numpy() for c in coefficients]
+print("Final Coefficients:", final_coefficients)
+
+plt.plot(df["Level"], df["Salary"], label="Original Data")
+plt.plot(df["Level"],[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df["Level"]])
+plt.ylabel('Salary')
+plt.xlabel('Position')
+plt.title("Salary vs Position")
+plt.show()
+```
+
+
+## Code Snippet for a Polynomial of Degree N
+
+### Using Gradient Tape
+
+This should work regardless of the Keras backend version (2 or 3)
+
+```python
+import tensorflow as tf
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+
+df = pd.read_csv("data.csv")
+
+############################
+## Change Parameters Here ##
+############################
+x_column = "Level"         #
+y_column = "Salary"        #
+degree = 2                 #
+learning_rate = 0.3        #
+num_epochs = 25_000        #
+############################
+
+X = tf.constant(df[x_column], dtype=tf.float32)
+Y = tf.constant(df[y_column], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]
+
+optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+
+for epoch in range(num_epochs):
+    with tf.GradientTape() as tape:
+        y_pred = tf.math.polyval(coefficients, X)
+        loss = tf.reduce_mean(tf.square(Y - y_pred))
+    grads = tape.gradient(loss, coefficients)
+    optimizer.apply_gradients(zip(grads, coefficients))
+    if (epoch+1) % 1000 == 0:
+        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}")
+
+final_coefficients = [c.numpy() for c in coefficients]
+print("Final Coefficients:", final_coefficients)
+
+print("Final Equation:", end=" ")
+for i in range(degree+1):
+  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")
+
+plt.plot(X, Y, label="Original Data")
+plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]]), label="Our Poynomial"
+plt.ylabel(y_column)
+plt.xlabel(x_column)
+plt.title(f"{x_column} vs {y_column}")
+plt.legend()
+plt.show()
+```
+
+### Without Gradient Tape
+
+This relies on the Optimizer's `minimize` function and uses the `var_list` parameter to update the variables.
+
+This will not work with Keras 3 backend in TF 2.16.0 and above unless you switch to the legacy backend.
+
+```python
+import tensorflow as tf
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+
+df = pd.read_csv("data.csv")
+
+############################
+## Change Parameters Here ##
+############################
+x_column = "Level"         #
+y_column = "Salary"        #
+degree = 2                 #
+learning_rate = 0.3        #
+num_epochs = 25_000        #
+############################
+
+X = tf.constant(df[x_column], dtype=tf.float32)
+Y = tf.constant(df[y_column], dtype=tf.float32)
+
+coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]
+
+optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
+
+def loss_function():
+  pred_y = tf.math.polyval(coefficients, X)
+  return tf.reduce_mean(tf.square(pred_y - Y))
+
+for epoch in range(num_epochs):
+    optimizer.minimize(loss_function, var_list=coefficients)
+    if (epoch+1) % 1000 == 0:
+        current_loss = loss_function().numpy()
+        print(f"Epoch {epoch+1}: Training Loss: {current_loss}")
+
+final_coefficients = coefficients.numpy()
+print("Final Coefficients:", final_coefficients)
+
+print("Final Equation:", end=" ")
+for i in range(degree+1):
+  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")
+
+plt.plot(X, Y, label="Original Data")
+plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]], label="Our Polynomial")
+plt.ylabel(y_column)
+plt.xlabel(x_column)
+plt.legend()
+plt.title(f"{x_column} vs {y_column}")
+plt.show()
+```
+
+
+As always, remember to tweak the parameters and choose the correct model for the job. A polynomial regression model might not even be the best model for this particular dataset.
+
+## Further Programming
+
+How would you modify this code to use another type of nonlinear regression? Say, 
+
+$$ y = ab^x $$
+
+Hint: Your loss calculation would be similar to:
+
+```python
+bx = tf.pow(coefficients[1], X)
+pred_y = tf.math.multiply(coefficients[0], bx)
+loss = tf.reduce_mean(tf.square(pred_y - Y))
+```
+
+