1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
|
<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"/><meta name="og:site_name" content="Navan Chauhan"/><link rel="canonical" href="https://navanchauhan.github.io/posts/2020-08-01-Natural-Feature-Tracking-ARJS"/><meta name="twitter:url" content="https://navanchauhan.github.io/posts/2020-08-01-Natural-Feature-Tracking-ARJS"/><meta name="og:url" content="https://navanchauhan.github.io/posts/2020-08-01-Natural-Feature-Tracking-ARJS"/><title>Introduction to AR.js and Natural Feature Tracking | Navan Chauhan</title><meta name="twitter:title" content="Introduction to AR.js and Natural Feature Tracking | Navan Chauhan"/><meta name="og:title" content="Introduction to AR.js and Natural Feature Tracking | Navan Chauhan"/><meta name="description" content="An introduction to AR.js and NFT"/><meta name="twitter:description" content="An introduction to AR.js and NFT"/><meta name="og:description" content="An introduction to AR.js and NFT"/><meta name="twitter:card" content="summary"/><link rel="stylesheet" href="/styles.css" type="text/css"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><link rel="shortcut icon" href="/images/favicon.png" type="image/png"/><link rel="alternate" href="/feed.rss" type="application/rss+xml" title="Subscribe to Navan Chauhan"/><meta name="twitter:image" content="https://navanchauhan.github.io/images/logo.png"/><meta name="og:image" content="https://navanchauhan.github.io/images/logo.png"/></head><head><script async src="//gc.zgo.at/count.js" data-goatcounter="https://navanchauhan.goatcounter.com/count"></script></head><body class="item-page"><header><div class="wrapper"><a class="site-name" href="/">Navan Chauhan</a><nav><ul><li><a href="/about">About Me</a></li><li><a class="selected" href="/posts">Posts</a></li><li><a href="/publications">Publications</a></li><li><a href="/assets/résumé.pdf">Résumé</a></li><li><a href="https://navanchauhan.github.io/repo">Repo</a></li><li><a href="/feed.rss">RSS</a></li></ul></nav></div></header><div class="wrapper"><article><div class="content"><span class="reading-time">7 minute read</span><span class="reading-time">Created on August 1, 2020</span><span class="reading-time">Last modified on September 15, 2020</span><h1>Introduction to AR.js and Natural Feature Tracking</h1><h2>AR.js</h2><p>AR.js is a lightweight library for Augmented Reality on the Web, coming with features like Image Tracking, Location based AR and Marker tracking. It is the easiest option for cross-browser augmented reality.</p><p>The same code works for iOS, Android, Desktops and even VR Browsers!</p><p>It was initially created by Jerome Etienne and is now maintained by Nicolo Carpignoli and the AR-js Organisation</p><h2>NFT</h2><p>Usually for augmented reality you need specialised markers, like this Hiro marker (notice the thick non-aesthetic borders 🤢)</p><img src="https://upload.wikimedia.org/wikipedia/commons/4/48/Hiro_marker_ARjs.png"/><p>This is called marker based tracking where the code knows what to look for. NFT or Natural Feature Tracing converts normal images into markers by extracting 'features' from it, this way you can use any image of your liking!</p><p>I'll be using my GitHub profile picture</p><img src="/images/me.jpeg"/><h2>Creating the Marker!</h2><p>First we need to create the marker files required by AR.js for NFT. For this we use Carnaux's repository 'NFT-Marker-Creator'.</p><pre><code><div class="highlight"><span></span>$ git clone https://github.com/Carnaux/NFT-Marker-Creator
Cloning into <span class="s1">'NFT-Marker-Creator'</span>...
remote: Enumerating objects: <span class="m">79</span>, <span class="k">done</span>.
remote: Counting objects: <span class="m">100</span>% <span class="o">(</span><span class="m">79</span>/79<span class="o">)</span>, <span class="k">done</span>.
remote: Compressing objects: <span class="m">100</span>% <span class="o">(</span><span class="m">72</span>/72<span class="o">)</span>, <span class="k">done</span>.
remote: Total <span class="m">580</span> <span class="o">(</span>delta <span class="m">10</span><span class="o">)</span>, reused <span class="m">59</span> <span class="o">(</span>delta <span class="m">7</span><span class="o">)</span>, pack-reused <span class="m">501</span>
Receiving objects: <span class="m">100</span>% <span class="o">(</span><span class="m">580</span>/580<span class="o">)</span>, <span class="m">9</span>.88 MiB <span class="p">|</span> <span class="m">282</span>.00 KiB/s, <span class="k">done</span>.
Resolving deltas: <span class="m">100</span>% <span class="o">(</span><span class="m">262</span>/262<span class="o">)</span>, <span class="k">done</span>.
$ <span class="nb">cd</span> NFT-Makrer-Creator
</div></code></pre><h3>Install the dependencies</h3><pre><code><div class="highlight"><span></span>$ npm install
npm WARN nodegenerator@1.0.0 No repository field.
added <span class="m">67</span> packages from <span class="m">56</span> contributors and audited <span class="m">67</span> packages in <span class="m">2</span>.96s
<span class="m">1</span> package is looking <span class="k">for</span> funding
run <span class="sb">`</span>npm fund<span class="sb">`</span> <span class="k">for</span> details
found <span class="m">0</span> vulnerabilities
╭────────────────────────────────────────────────────────────────╮
│ │
│ New patch version of npm available! <span class="m">6</span>.14.5 → <span class="m">6</span>.14.7 │
│ Changelog: https://github.com/npm/cli/releases/tag/v6.14.7 │
│ Run npm install -g npm to update! │
│ │
╰────────────────────────────────────────────────────────────────╯
</div></code></pre><h3>Copy the target marker to the folder</h3><pre><code><div class="highlight"><span></span>$ cp ~/CodingAndStuff/ARjs/me.png .
</div></code></pre><h3>Generate Marker</h3><pre><code><div class="highlight"><span></span>$ node app.js -i me.png
Confidence level: <span class="o">[</span> * * * * * <span class="o">]</span> <span class="m">5</span>/5 <span class="o">||</span> Entropy: <span class="m">5</span>.24 <span class="o">||</span> Current max: <span class="m">5</span>.17 min: <span class="m">4</span>.6
Do you want to <span class="k">continue</span>? <span class="o">(</span>Y/N<span class="o">)</span>
y
writeStringToMemory is deprecated and should not be called! Use stringToUTF8<span class="o">()</span> instead!
<span class="o">[</span>info<span class="o">]</span>
Commands:
<span class="o">[</span>info<span class="o">]</span> --
Generator started at <span class="m">2020</span>-08-01 <span class="m">16</span>:01:41 +0580
<span class="o">[</span>info<span class="o">]</span> Tracking Extraction <span class="nv">Level</span> <span class="o">=</span> <span class="m">2</span>
<span class="o">[</span>info<span class="o">]</span> <span class="nv">MAX_THRESH</span> <span class="o">=</span> <span class="m">0</span>.900000
<span class="o">[</span>info<span class="o">]</span> <span class="nv">MIN_THRESH</span> <span class="o">=</span> <span class="m">0</span>.550000
<span class="o">[</span>info<span class="o">]</span> <span class="nv">SD_THRESH</span> <span class="o">=</span> <span class="m">8</span>.000000
<span class="o">[</span>info<span class="o">]</span> Initialization Extraction <span class="nv">Level</span> <span class="o">=</span> <span class="m">1</span>
<span class="o">[</span>info<span class="o">]</span> <span class="nv">SURF_FEATURE</span> <span class="o">=</span> <span class="m">100</span>
<span class="o">[</span>info<span class="o">]</span> min allow <span class="m">3</span>.699000.
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">1</span><span class="o">)</span>: <span class="m">3</span>.699000
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">2</span><span class="o">)</span>: <span class="m">4</span>.660448
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">3</span><span class="o">)</span>: <span class="m">5</span>.871797
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">4</span><span class="o">)</span>: <span class="m">7</span>.398000
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">5</span><span class="o">)</span>: <span class="m">9</span>.320896
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">6</span><span class="o">)</span>: <span class="m">11</span>.743593
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">7</span><span class="o">)</span>: <span class="m">14</span>.796000
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">8</span><span class="o">)</span>: <span class="m">18</span>.641792
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">9</span><span class="o">)</span>: <span class="m">23</span>.487186
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">10</span><span class="o">)</span>: <span class="m">29</span>.592001
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">11</span><span class="o">)</span>: <span class="m">37</span>.283585
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">12</span><span class="o">)</span>: <span class="m">46</span>.974373
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">13</span><span class="o">)</span>: <span class="m">59</span>.184002
<span class="o">[</span>info<span class="o">]</span> Image DPI <span class="o">(</span><span class="m">14</span><span class="o">)</span>: <span class="m">72</span>.000000
<span class="o">[</span>info<span class="o">]</span> Generating ImageSet...
<span class="o">[</span>info<span class="o">]</span> <span class="o">(</span>Source image <span class="nv">xsize</span><span class="o">=</span><span class="m">568</span>, <span class="nv">ysize</span><span class="o">=</span><span class="m">545</span>, <span class="nv">channels</span><span class="o">=</span><span class="m">3</span>, <span class="nv">dpi</span><span class="o">=</span><span class="m">72</span>.0<span class="o">)</span>.
<span class="o">[</span>info<span class="o">]</span> Done.
<span class="o">[</span>info<span class="o">]</span> Saving to asa.iset...
<span class="o">[</span>info<span class="o">]</span> Done.
<span class="o">[</span>info<span class="o">]</span> Generating FeatureList...
...
<span class="o">[</span>info<span class="o">]</span> <span class="o">(</span><span class="m">46</span>, <span class="m">44</span><span class="o">)</span> <span class="m">5</span>.871797<span class="o">[</span>dpi<span class="o">]</span>
<span class="o">[</span>info<span class="o">]</span> Freak features - <span class="m">23</span><span class="o">[</span>info<span class="o">]</span> <span class="o">=========</span> <span class="nv">23</span> <span class="o">===========</span>
<span class="o">[</span>info<span class="o">]</span> <span class="o">(</span><span class="m">37</span>, <span class="m">35</span><span class="o">)</span> <span class="m">4</span>.660448<span class="o">[</span>dpi<span class="o">]</span>
<span class="o">[</span>info<span class="o">]</span> Freak features - <span class="m">19</span><span class="o">[</span>info<span class="o">]</span> <span class="o">=========</span> <span class="nv">19</span> <span class="o">===========</span>
<span class="o">[</span>info<span class="o">]</span> <span class="o">(</span><span class="m">29</span>, <span class="m">28</span><span class="o">)</span> <span class="m">3</span>.699000<span class="o">[</span>dpi<span class="o">]</span>
<span class="o">[</span>info<span class="o">]</span> Freak features - <span class="m">9</span><span class="o">[</span>info<span class="o">]</span> <span class="o">=========</span> <span class="nv">9</span> <span class="o">===========</span>
<span class="o">[</span>info<span class="o">]</span> Done.
<span class="o">[</span>info<span class="o">]</span> Saving FeatureSet3...
<span class="o">[</span>info<span class="o">]</span> Done.
<span class="o">[</span>info<span class="o">]</span> Generator finished at <span class="m">2020</span>-08-01 <span class="m">16</span>:02:02 +0580
--
Finished marker creation!
Now configuring demo!
Finished!
To run demo use: <span class="s1">'npm run demo'</span>
</div></code></pre><p>Now we have the required files in the output folder</p><pre><code><div class="highlight"><span></span>$ ls output
me.fset me.fset3 me.iset
</div></code></pre><h2>Creating the HTML Page</h2><p>Create a new file called <code>index.html</code> in your project folder. This is the basic template we are going to use. Replace <code>me</code> with the root filename of your image, for example <code>NeverGonnaGiveYouUp.png</code> will become <code>NeverGonnaGiveYouUp</code>. Make sure you have copied all three files from the output folder in the previous step to the root of your project folder.</p><pre><code><div class="highlight"><span></span><span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"https://cdn.jsdelivr.net/gh/aframevr/aframe@1c2407b26c61958baa93967b5412487cd94b290b/dist/aframe-master.min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar-nft.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">style</span><span class="p">></span>
<span class="p">.</span><span class="nc">arjs-loader</span> <span class="p">{</span>
<span class="k">height</span><span class="p">:</span> <span class="mi">100</span><span class="kt">%</span><span class="p">;</span>
<span class="k">width</span><span class="p">:</span> <span class="mi">100</span><span class="kt">%</span><span class="p">;</span>
<span class="k">position</span><span class="p">:</span> <span class="kc">absolute</span><span class="p">;</span>
<span class="k">top</span><span class="p">:</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">left</span><span class="p">:</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">background-color</span><span class="p">:</span> <span class="nb">rgba</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mf">0.8</span><span class="p">);</span>
<span class="k">z-index</span><span class="p">:</span> <span class="mi">9999</span><span class="p">;</span>
<span class="k">display</span><span class="p">:</span> <span class="kc">flex</span><span class="p">;</span>
<span class="k">justify-content</span><span class="p">:</span> <span class="kc">center</span><span class="p">;</span>
<span class="k">align-items</span><span class="p">:</span> <span class="kc">center</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">.</span><span class="nc">arjs-loader</span> <span class="nt">div</span> <span class="p">{</span>
<span class="k">text-align</span><span class="p">:</span> <span class="kc">center</span><span class="p">;</span>
<span class="k">font-size</span><span class="p">:</span> <span class="mf">1.25</span><span class="kt">em</span><span class="p">;</span>
<span class="k">color</span><span class="p">:</span> <span class="kc">white</span><span class="p">;</span>
<span class="p">}</span>
<span class="p"></</span><span class="nt">style</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span> <span class="na">style</span><span class="o">=</span><span class="s">"margin : 0px; overflow: hidden;"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"arjs-loader"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span><span class="p">></span>Calculating Image Descriptors....<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">a-scene</span>
<span class="na">vr-mode-ui</span><span class="o">=</span><span class="s">"enabled: false;"</span>
<span class="na">renderer</span><span class="o">=</span><span class="s">"logarithmicDepthBuffer: true;"</span>
<span class="na">embedded</span>
<span class="na">arjs</span><span class="o">=</span><span class="s">"trackingMethod: best; sourceType: webcam;debugUIEnabled: false;"</span>
<span class="p">></span>
<span class="p"><</span><span class="nt">a-nft</span>
<span class="na">type</span><span class="o">=</span><span class="s">"nft"</span>
<span class="na">url</span><span class="o">=</span><span class="s">"./me"</span>
<span class="na">smooth</span><span class="o">=</span><span class="s">"true"</span>
<span class="na">smoothCount</span><span class="o">=</span><span class="s">"10"</span>
<span class="na">smoothTolerance</span><span class="o">=</span><span class="s">".01"</span>
<span class="na">smoothThreshold</span><span class="o">=</span><span class="s">"5"</span>
<span class="p">></span>
<span class="p"></</span><span class="nt">a-nft</span><span class="p">></span>
<span class="p"><</span><span class="nt">a-entity</span> <span class="na">camera</span><span class="p">></</span><span class="nt">a-entity</span><span class="p">></span>
<span class="p"></</span><span class="nt">a-scene</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
</div></code></pre><p>In this we are creating a AFrame scene and we are telling it that we want to use NFT Tracking. The amazing part about using AFrame is that we are able to use all AFrame objects!</p><h2>Adding a simple box</h2><p>Let us add a simple box!</p><pre><code><div class="highlight"><span></span><span class="p"><</span><span class="nt">a-nft</span> <span class="err">.....</span><span class="p">></span>
<span class="p"><</span><span class="nt">a-box</span> <span class="na">position</span><span class="o">=</span><span class="s">'100 0.5 -180'</span> <span class="na">material</span><span class="o">=</span><span class="s">'opacity: 0.5; side: double'</span> <span class="na">scale</span><span class="o">=</span><span class="s">"100 100 100"</span><span class="p">></</span><span class="nt">a-box</span><span class="p">></span>
<span class="p"></</span><span class="nt">a-nft</span><span class="p">></span>
</div></code></pre><p>Now to test it out we will need to create a simple server, I use Python's inbuilt <code>SimpleHTTPServer</code> alongside <code>ngrok</code></p><p>In one terminal window, <code>cd</code> to the project directory. Currently your project folder should have 4 files, <code>index.html</code>, <code>me.fset3</code>, <code>me.fset</code> and <code>me.iset</code></p><p>Open up two terminal windows and <code>cd</code> into your project folder then run the following commands to start up your server.</p><p>In the first terminal window start the Python Server</p><pre><code><div class="highlight"><span></span>$ <span class="nb">cd</span> ~/CodingAndStuff/ARjs
$ python2 -m SimpleHTTPServer
Serving HTTP on <span class="m">0</span>.0.0.0 port <span class="m">8000</span> ...
</div></code></pre><p>In the other window run <code>ngrok</code> ( Make sure you have installed it prior to running this step )</p><pre><code><div class="highlight"><span></span>$ ngrok http <span class="m">8000</span>
</div></code></pre><img src="/assets/posts/arjs/01-ngrok.png"/><p>Now copy the url to your phone and try running the example</p><img src="/assets/posts/arjs/02-box-demo.gif"/><p>👏 Congratulations! You just built an Augmented Reality experience using AR.js and AFrame</p><h2>Adding a Torus-Knot in the box</h2><p>Edit your <code>index.html</code></p><pre><code><div class="highlight"><span></span><span class="p"><</span><span class="nt">a-nft</span> <span class="err">..</span><span class="p">></span>
<span class="p"><</span><span class="nt">a-box</span> <span class="err">..</span><span class="p">></span>
<span class="p"><</span><span class="nt">a-torus-knot</span> <span class="na">radius</span><span class="o">=</span><span class="s">'0.26'</span> <span class="na">radius-tubular</span><span class="o">=</span><span class="s">'0.05'</span> <span class="p">></</span><span class="nt">a-torus-knot</span><span class="p">></span>
<span class="p"></</span> <span class="nt">a-box</span><span class="p">></span>
<span class="p"></</span> <span class="nt">a-nft</span><span class="p">></span>
</div></code></pre><img src="/assets/posts/arjs/03-knot.png"/><h2>Where are the GIFs?</h2><p>Now that we know how to place a box in the scene and add a torus knot in it, what do we do next? We bring the classic internet back!</p><p><code>AFrame GIF Shader</code> is a gif shader for A-Frame created by mayognaise.</p><h3>First things first</h3><p>Add <code><script src="https://rawgit.com/mayognaise/aframe-gif-shader/master/dist/aframe-gif-shader.min.js"></script> </code> to <code><head></code></p><p>Change the box's material to add the GIF shader</p><pre><code><div class="highlight"><span></span>...
<span class="p"><</span><span class="nt">a-box</span> <span class="na">position</span><span class="o">=</span><span class="s">'100 0.5 -180'</span> <span class="na">material</span><span class="o">=</span><span class="s">"shader:gif;src:url(https://media.tenor.com/images/412b1aa9149d98d561df62db221e0789/tenor.gif);opacity:.5"</span> <span class="err">.....</span><span class="p">></span>
</div></code></pre><img src="/assets/posts/arjs/04-nyan.gif"/><h2>Bonus Idea: Integrate it with GitHub's new profile Readme Feature!</h2><h3>1) Host the code using GitHub Pages</h3><h3>2) Create a new repository ( the name should be your GitHub username )</h3><h3>3) Add QR Code to the page and tell the users to scan your profile picture</h3><h3>??) Profit 💸</h3><p>Here is a screenshot of me scanning a rounded version of my profile picture ( It still works! Even though the image is cropped and I haven't changed any line of code )</p><img src="/assets/posts/arjs/05-GitHub.jpg"/></div><span>Tagged with: </span><ul class="tag-list"><li><a href="/tags/tutorial">Tutorial</a></li><li><a href="/tags/arjs">AR.js</a></li><li><a href="/tags/javascript">JavaScript</a></li><li><a href="/tags/augmentedreality">Augmented-Reality</a></li></ul></article></div><footer><p>Made with ❤️ using <a href="https://github.com/johnsundell/publish">Publish</a></p><p><a href="/feed.rss">RSS feed</a></p></footer></body></html>
|
