diff options
| author | navanchauhan <navanchauhan@gmail.com> | 2023-02-08 17:40:27 -0700 |
|---|---|---|
| committer | navanchauhan <navanchauhan@gmail.com> | 2023-02-08 17:40:27 -0700 |
| commit | e3c2fac4f49859268d2f337ecaa64c41e3a6bd1d (patch) | |
| tree | d336786cbcd3f8d39f4914b734741378126d22df /docs/feed.rss | |
| parent | 0d34e3633ba36edd049ce3ba52908e061b7273d5 (diff) | |
new post
Diffstat (limited to 'docs/feed.rss')
| -rw-r--r-- | docs/feed.rss | 249 |
1 files changed, 247 insertions, 2 deletions
diff --git a/docs/feed.rss b/docs/feed.rss index d81a885..22944f0 100644 --- a/docs/feed.rss +++ b/docs/feed.rss @@ -4,8 +4,8 @@ <title>Navan's Archive</title> <description>Rare Tips, Tricks and Posts</description> <link>https://web.navan.dev/</link><language>en</language> - <lastBuildDate>Wed, 01 Feb 2023 12:29:22 -0000</lastBuildDate> - <pubDate>Wed, 01 Feb 2023 12:29:22 -0000</pubDate> + <lastBuildDate>Wed, 08 Feb 2023 17:38:18 -0000</lastBuildDate> + <pubDate>Wed, 08 Feb 2023 17:38:18 -0000</pubDate> <ttl>250</ttl> <atom:link href="https://web.navan.dev/feed.rss" rel="self" type="application/rss+xml"/> @@ -4270,6 +4270,251 @@ return path(str, boost::filesystem::native); <item> <guid isPermaLink="true"> + https://web.navan.dev/posts/2023-02-08-Interact-with-siri-from-the-terminal.html + </guid> + <title> + Interacting with Siri using the command line + </title> + <description> + Code snippet to interact with Siri by issuing commands from the command-line. + </description> + <link>https://web.navan.dev/posts/2023-02-08-Interact-with-siri-from-the-terminal.html</link> + <pubDate>Wed, 08 Feb 2023 17:21:00 -0000</pubDate> + <content:encoded><![CDATA[<h1>Interacting with Siri using the command line</h1> + +<p>My main objective was to see if I could issue multi-intent commands in one go. Obviously, Siri cannot do that (neither can Alexa, Cortana, or Google Assistant). The script here can issue either a single command, or use the help of OpenAI's DaVinci model to extract multiple commands and pass them onto siri.</p> + +<h2>Prerequisites</h2> + +<ul> +<li>Run macOS</li> +<li>Enable type to Siri (Settings > Accessibility -> Type to Siri)</li> +<li>Enable the Terminal to control System Events (The first time you run the script, it will prompt you to enable it)</li> +</ul> + +<h2>Show me ze code</h2> + +<p>If you are here just for the code:</p> + +<div class="codehilite"> +<pre><span></span><code><span class="kn">import</span> <span class="nn">argparse</span> +<span class="kn">import</span> <span class="nn">applescript</span> +<span class="kn">import</span> <span class="nn">openai</span> + +<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span> + +<span class="n">openai</span><span class="o">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">getenv</span><span class="p">(</span><span class="s2">"OPENAI_KEY"</span><span class="p">)</span> +<span class="n">engine</span> <span class="o">=</span> <span class="s2">"text-davinci-003"</span> + +<span class="k">def</span> <span class="nf">execute_with_llm</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> + <span class="n">llm_prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.</span> + +<span class="s2"> Example:</span> +<span class="s2"> Q: "Turn on the lights and turn off the lights"</span> +<span class="s2"> A: ["Turn on the lights", "Turn off the lights"]</span> + +<span class="s2"> Q: "Switch off the lights and then play some music"</span> +<span class="s2"> A: ["Switch off the lights", "Play some music"]</span> + +<span class="s2"> Q: "I am feeling sad today, play some music"</span> +<span class="s2"> A: ["Play some cheerful music"]</span> + +<span class="s2"> Q: "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span> +<span class="s2"> A: </span> +<span class="s2"> """</span> + + <span class="n">completion</span> <span class="o">=</span> <span class="n">openai</span><span class="o">.</span><span class="n">Completion</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">engine</span><span class="o">=</span><span class="n">engine</span><span class="p">,</span> <span class="n">prompt</span><span class="o">=</span><span class="n">llm_prompt</span><span class="p">,</span> <span class="n">max_tokens</span><span class="o">=</span><span class="nb">len</span><span class="p">(</span><span class="n">command_text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> + + <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="nb">eval</span><span class="p">(</span><span class="n">completion</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">):</span> + <span class="n">execute_command</span><span class="p">(</span><span class="n">task</span><span class="p">)</span> + + +<span class="k">def</span> <span class="nf">execute_command</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> + <span class="sd">"""Execute a Siri command."""</span> + + <span class="n">script</span> <span class="o">=</span> <span class="n">applescript</span><span class="o">.</span><span class="n">AppleScript</span><span class="p">(</span><span class="sa">f</span><span class="s2">"""</span> +<span class="s2"> tell application "System Events" to tell the front menu bar of process "SystemUIServer"</span> +<span class="s2"> tell (first menu bar item whose description is "Siri")</span> +<span class="s2"> perform action "AXPress"</span> +<span class="s2"> end tell</span> +<span class="s2"> end tell</span> + +<span class="s2"> delay 2</span> + +<span class="s2"> tell application "System Events"</span> +<span class="s2"> set textToType to "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span> +<span class="s2"> keystroke textToType</span> +<span class="s2"> key code 36</span> +<span class="s2"> end tell</span> +<span class="s2"> """</span><span class="p">)</span> + + <span class="n">script</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> + + +<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span> + <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> + <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">"command"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=</span><span class="s2">"?"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"The command to pass to Siri"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"What time is it?"</span><span class="p">)</span> + <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'--openai'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="n">argparse</span><span class="o">.</span><span class="n">BooleanOptionalAction</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"Use OpenAI to detect multiple intents"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> + <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> + + <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">openai</span><span class="p">:</span> + <span class="n">execute_with_llm</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span> + <span class="k">else</span><span class="p">:</span> + <span class="n">execute_command</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span> +</code></pre> +</div> + +<p>Usage:</p> + +<div class="codehilite"> +<pre><span></span><code>python3 main.py <span class="s2">"play some taylor swift"</span> +python3 main.py <span class="s2">"turn off the lights and play some music"</span> --openai +</code></pre> +</div> + +<h2>ELI5</h2> + +<p>I am not actually going to explain it as if I am explaining to a five-year old kid.</p> + +<h3>AppleScript</h3> + +<p>In the age of Siri Shortcuts, AppleScript can still do more. It is a scripting language created by Apple that can help you automate pretty much anything you see on your screen.</p> + +<p>We use the following AppleScript to trigger Siri and then type in our command:</p> + +<div class="codehilite"> +<pre><span></span><code><span class="k">tell</span> <span class="nb">application</span> <span class="s2">"System Events"</span> <span class="k">to</span> <span class="k">tell</span> <span class="nb">the</span> <span class="nb">front</span> <span class="na">menu</span> <span class="nv">bar</span> <span class="k">of</span> <span class="nv">process</span> <span class="s2">"SystemUIServer"</span> + <span class="k">tell</span> <span class="p">(</span><span class="nb">first</span> <span class="na">menu</span> <span class="nv">bar</span> <span class="nb">item</span> <span class="nb">whose</span> <span class="nv">description</span> <span class="ow">is</span> <span class="s2">"Siri"</span><span class="p">)</span> + <span class="nb">perform action</span> <span class="s2">"AXPress"</span> + <span class="k">end</span> <span class="k">tell</span> +<span class="k">end</span> <span class="k">tell</span> + +<span class="nb">delay</span> <span class="mi">2</span> + +<span class="k">tell</span> <span class="nb">application</span> <span class="s2">"System Events"</span> + <span class="k">set</span> <span class="nv">textToType</span> <span class="k">to</span> <span class="s2">"Play some rock music"</span> + <span class="nv">keystroke</span> <span class="nv">textToType</span> + <span class="na">key code</span> <span class="mi">36</span> +<span class="k">end</span> <span class="k">tell</span> +</code></pre> +</div> + +<p>This first triggers Siri, waits for a couple of seconds, and then types in our command. In the script, this functionality is handled by the <code>execute_command</code> function.</p> + +<div class="codehilite"> +<pre><span></span><code><span class="kn">import</span> <span class="nn">applescript</span> + +<span class="k">def</span> <span class="nf">execute_command</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> + <span class="sd">"""Execute a Siri command."""</span> + + <span class="n">script</span> <span class="o">=</span> <span class="n">applescript</span><span class="o">.</span><span class="n">AppleScript</span><span class="p">(</span><span class="sa">f</span><span class="s2">"""</span> +<span class="s2"> tell application "System Events" to tell the front menu bar of process "SystemUIServer"</span> +<span class="s2"> tell (first menu bar item whose description is "Siri")</span> +<span class="s2"> perform action "AXPress"</span> +<span class="s2"> end tell</span> +<span class="s2"> end tell</span> + +<span class="s2"> delay 2</span> + +<span class="s2"> tell application "System Events"</span> +<span class="s2"> set textToType to "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span> +<span class="s2"> keystroke textToType</span> +<span class="s2"> key code 36</span> +<span class="s2"> end tell</span> +<span class="s2"> """</span><span class="p">)</span> + + <span class="n">script</span><span class="o">.</span><span class="n">run</span><span class="p">()</span> +</code></pre> +</div> + +<h3>Multi-Intent Commands</h3> + +<p>We can call OpenAI's API to autocomplete our prompt and extract multiple commands. We don't need to use OpenAI's API, and can also simply use Google's Flan-T5 model using HuggingFace's transformers library. </p> + +<h4>Ze Prompt</h4> + +<div class="codehilite"> +<pre><span></span><code>You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command. + + Example: + Q: "Turn on the lights and turn off the lights" + A: ["Turn on the lights", "Turn off the lights"] + + Q: "Switch off the lights and then play some music" + A: ["Switch off the lights", "Play some music"] + + Q: "I am feeling sad today, play some music" + A: ["Play some cheerful music"] + + Q: "{command_text}" + A: +</code></pre> +</div> + +<p>This prompt gives the model a few examples to increase the generation accuracy, along with instructing it to return a Python list. </p> + +<h4>Ze Code</h4> + +<div class="codehilite"> +<pre><span></span><code><span class="kn">import</span> <span class="nn">openai</span> + +<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span> + +<span class="n">openai</span><span class="o">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">getenv</span><span class="p">(</span><span class="s2">"OPENAI_KEY"</span><span class="p">)</span> +<span class="n">engine</span> <span class="o">=</span> <span class="s2">"text-davinci-003"</span> + +<span class="k">def</span> <span class="nf">execute_with_llm</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span> + <span class="n">llm_prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.</span> + +<span class="s2"> Example:</span> +<span class="s2"> Q: "Turn on the lights and turn off the lights"</span> +<span class="s2"> A: ["Turn on the lights", "Turn off the lights"]</span> + +<span class="s2"> Q: "Switch off the lights and then play some music"</span> +<span class="s2"> A: ["Switch off the lights", "Play some music"]</span> + +<span class="s2"> Q: "I am feeling sad today, play some music"</span> +<span class="s2"> A: ["Play some cheerful music"]</span> + +<span class="s2"> Q: "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span> +<span class="s2"> A: </span> +<span class="s2"> """</span> + + <span class="n">completion</span> <span class="o">=</span> <span class="n">openai</span><span class="o">.</span><span class="n">Completion</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">engine</span><span class="o">=</span><span class="n">engine</span><span class="p">,</span> <span class="n">prompt</span><span class="o">=</span><span class="n">llm_prompt</span><span class="p">,</span> <span class="n">max_tokens</span><span class="o">=</span><span class="nb">len</span><span class="p">(</span><span class="n">command_text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> + + <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="nb">eval</span><span class="p">(</span><span class="n">completion</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">):</span> <span class="c1"># NEVER EVAL IN PROD RIGHT LIKE THIS</span> + <span class="n">execute_command</span><span class="p">(</span><span class="n">task</span><span class="p">)</span> +</code></pre> +</div> + +<h3>Gluing together code</h3> + +<p>To finish it all off, we can use argparse to only send the input command to OpenAI when asked to do so.</p> + +<div class="codehilite"> +<pre><span></span><code><span class="kn">import</span> <span class="nn">argparse</span> + +<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span> + <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> + <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">"command"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=</span><span class="s2">"?"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"The command to pass to Siri"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"What time is it?"</span><span class="p">)</span> + <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'--openai'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="n">argparse</span><span class="o">.</span><span class="n">BooleanOptionalAction</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"Use OpenAI to detect multiple intents"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> + <span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span> + + <span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">openai</span><span class="p">:</span> + <span class="n">execute_with_llm</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span> + <span class="k">else</span><span class="p">:</span> + <span class="n">execute_command</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span> +</code></pre> +</div> + +<h2>Conclusion</h2> + +<p>Siri is still dumb. When I ask it to <code>Switch off the lights</code>, it default to the home thousands of miles away. But, this code snippet definitely does work!</p> +]]></content:encoded> + </item> + + <item> + <guid isPermaLink="true"> https://web.navan.dev/posts/2020-01-16-Image-Classifier-Using-Turicreate.html </guid> <title> |