<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" href="/assets/main.css" />
<link rel="stylesheet" href="/assets/sakura.css" />
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Interacting with Siri using the command line</title>
<meta name="og:site_name" content="Navan Chauhan" />
<link rel="canonical" href="https://web.navan.dev/" />
<meta name="twitter:url" content="https://web.navan.dev/" />
<meta name="og:url" content="https://web.navan.dev/" />
<meta name="twitter:title" content="Interacting with Siri using the command line" />
<meta name="og:title" content="Interacting with Siri using the command line" />
<meta name="description" content="Code snippet to interact with Siri by issuing commands from the command-line." />
<meta name="twitter:description" content="Code snippet to interact with Siri by issuing commands from the command-line." />
<meta name="og:description" content="Code snippet to interact with Siri by issuing commands from the command-line." />
<meta name="twitter:card" content="summary_large_image" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="shortcut icon" href="/images/favicon.png" type="image/png" />
<link rel="alternate" href="/feed.rss" type="application/rss+xml" title="Subscribe to Navan Chauhan" />
<meta name="twitter:image" content="https://web.navan.dev/images/opengraph/posts/2023-02-08-Interact-with-siri-from-the-terminal.png" />
<meta name="og:image" content="https://web.navan.dev/images/opengraph/posts/2023-02-08-Interact-with-siri-from-the-terminal.png" />
<link rel="manifest" href="manifest.json" />
<meta name="google-site-verification" content="LVeSZxz-QskhbEjHxOi7-BM5dDxTg53x2TwrjFxfL0k" />
<script data-goatcounter="https://navanchauhan.goatcounter.com/count"
async src="//gc.zgo.at/count.js"></script>
<script defer data-domain="web.navan.dev" src="https://plausible.io/js/plausible.js"></script>
<script defer data-domain="web.navan.dev" src="https://plausible.navan.dev/js/plausible.js"></script>
<!-- Begin Inspectlet Asynchronous Code. Only for some testing, will be removed soon -->
<script type="text/javascript">
(function() {
window.__insp = window.__insp || [];
__insp.push(['wid', 1038401947]);
var ldinsp = function(){
if(typeof window.__inspld != "undefined") return; window.__inspld = 1; var insp = document.createElement('script'); insp.type = 'text/javascript'; insp.async = true; insp.id = "inspsync"; insp.src = ('https:' == document.location.protocol ? 'https' : 'http') + '://cdn.inspectlet.com/inspectlet.js?wid=1038401947&r=' + Math.floor(new Date().getTime()/3600000); var x = document.getElementsByTagName('script')[0]; x.parentNode.insertBefore(insp, x); };
setTimeout(ldinsp, 0);
})();
</script>
<!-- End Inspectlet Asynchronous Code -->
</head>
<body>
<nav style="display: block;">
|
<a href="/">home</a> |
<a href="/about/">about/links</a> |
<a href="/posts/">posts</a> |
<a href="/publications/">publications</a> |
<a href="/repo/">iOS repo</a> |
<a href="/feed.rss">RSS Feed</a> |
</nav>
<main>
<h1>Interacting with Siri using the command line</h1>
<p>My main objective was to see if I could issue multi-intent commands in one go. Obviously, Siri cannot do that (neither can Alexa, Cortana, or Google Assistant). The script here can issue either a single command, or use the help of OpenAI's DaVinci model to extract multiple commands and pass them onto siri.</p>
<h2>Prerequisites</h2>
<ul>
<li>Run macOS</li>
<li>Enable type to Siri (Settings > Accessibility -> Type to Siri)</li>
<li>Enable the Terminal to control System Events (The first time you run the script, it will prompt you to enable it)</li>
</ul>
<h2>Show me ze code</h2>
<p>If you are here just for the code:</p>
<div class="codehilite">
<pre><span></span><code><span class="kn">import</span> <span class="nn">argparse</span>
<span class="kn">import</span> <span class="nn">applescript</span>
<span class="kn">import</span> <span class="nn">openai</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span>
<span class="n">openai</span><span class="o">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">getenv</span><span class="p">(</span><span class="s2">"OPENAI_KEY"</span><span class="p">)</span>
<span class="n">engine</span> <span class="o">=</span> <span class="s2">"text-davinci-003"</span>
<span class="k">def</span> <span class="nf">execute_with_llm</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="n">llm_prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.</span>
<span class="s2"> Example:</span>
<span class="s2"> Q: "Turn on the lights and turn off the lights"</span>
<span class="s2"> A: ["Turn on the lights", "Turn off the lights"]</span>
<span class="s2"> Q: "Switch off the lights and then play some music"</span>
<span class="s2"> A: ["Switch off the lights", "Play some music"]</span>
<span class="s2"> Q: "I am feeling sad today, play some music"</span>
<span class="s2"> A: ["Play some cheerful music"]</span>
<span class="s2"> Q: "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span>
<span class="s2"> A: </span>
<span class="s2"> """</span>
<span class="n">completion</span> <span class="o">=</span> <span class="n">openai</span><span class="o">.</span><span class="n">Completion</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">engine</span><span class="o">=</span><span class="n">engine</span><span class="p">,</span> <span class="n">prompt</span><span class="o">=</span><span class="n">llm_prompt</span><span class="p">,</span> <span class="n">max_tokens</span><span class="o">=</span><span class="nb">len</span><span class="p">(</span><span class="n">command_text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="nb">eval</span><span class="p">(</span><span class="n">completion</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">):</span>
<span class="n">execute_command</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">execute_command</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="sd">"""Execute a Siri command."""</span>
<span class="n">script</span> <span class="o">=</span> <span class="n">applescript</span><span class="o">.</span><span class="n">AppleScript</span><span class="p">(</span><span class="sa">f</span><span class="s2">"""</span>
<span class="s2"> tell application "System Events" to tell the front menu bar of process "SystemUIServer"</span>
<span class="s2"> tell (first menu bar item whose description is "Siri")</span>
<span class="s2"> perform action "AXPress"</span>
<span class="s2"> end tell</span>
<span class="s2"> end tell</span>
<span class="s2"> delay 2</span>
<span class="s2"> tell application "System Events"</span>
<span class="s2"> set textToType to "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span>
<span class="s2"> keystroke textToType</span>
<span class="s2"> key code 36</span>
<span class="s2"> end tell</span>
<span class="s2"> """</span><span class="p">)</span>
<span class="n">script</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">"command"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=</span><span class="s2">"?"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"The command to pass to Siri"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"What time is it?"</span><span class="p">)</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'--openai'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="n">argparse</span><span class="o">.</span><span class="n">BooleanOptionalAction</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"Use OpenAI to detect multiple intents"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
<span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">openai</span><span class="p">:</span>
<span class="n">execute_with_llm</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">execute_command</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
</code></pre>
</div>
<p>Usage:</p>
<div class="codehilite">
<pre><span></span><code>python3 main.py <span class="s2">"play some taylor swift"</span>
python3 main.py <span class="s2">"turn off the lights and play some music"</span> --openai
</code></pre>
</div>
<h2>ELI5</h2>
<p>I am not actually going to explain it as if I am explaining to a five-year old kid.</p>
<h3>AppleScript</h3>
<p>In the age of Siri Shortcuts, AppleScript can still do more. It is a scripting language created by Apple that can help you automate pretty much anything you see on your screen.</p>
<p>We use the following AppleScript to trigger Siri and then type in our command:</p>
<div class="codehilite">
<pre><span></span><code><span class="k">tell</span> <span class="nb">application</span> <span class="s2">"System Events"</span> <span class="k">to</span> <span class="k">tell</span> <span class="nb">the</span> <span class="nb">front</span> <span class="na">menu</span> <span class="nv">bar</span> <span class="k">of</span> <span class="nv">process</span> <span class="s2">"SystemUIServer"</span>
<span class="k">tell</span> <span class="p">(</span><span class="nb">first</span> <span class="na">menu</span> <span class="nv">bar</span> <span class="nb">item</span> <span class="nb">whose</span> <span class="nv">description</span> <span class="ow">is</span> <span class="s2">"Siri"</span><span class="p">)</span>
<span class="nb">perform action</span> <span class="s2">"AXPress"</span>
<span class="k">end</span> <span class="k">tell</span>
<span class="k">end</span> <span class="k">tell</span>
<span class="nb">delay</span> <span class="mi">2</span>
<span class="k">tell</span> <span class="nb">application</span> <span class="s2">"System Events"</span>
<span class="k">set</span> <span class="nv">textToType</span> <span class="k">to</span> <span class="s2">"Play some rock music"</span>
<span class="nv">keystroke</span> <span class="nv">textToType</span>
<span class="na">key code</span> <span class="mi">36</span>
<span class="k">end</span> <span class="k">tell</span>
</code></pre>
</div>
<p>This first triggers Siri, waits for a couple of seconds, and then types in our command. In the script, this functionality is handled by the <code>execute_command</code> function.</p>
<div class="codehilite">
<pre><span></span><code><span class="kn">import</span> <span class="nn">applescript</span>
<span class="k">def</span> <span class="nf">execute_command</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="sd">"""Execute a Siri command."""</span>
<span class="n">script</span> <span class="o">=</span> <span class="n">applescript</span><span class="o">.</span><span class="n">AppleScript</span><span class="p">(</span><span class="sa">f</span><span class="s2">"""</span>
<span class="s2"> tell application "System Events" to tell the front menu bar of process "SystemUIServer"</span>
<span class="s2"> tell (first menu bar item whose description is "Siri")</span>
<span class="s2"> perform action "AXPress"</span>
<span class="s2"> end tell</span>
<span class="s2"> end tell</span>
<span class="s2"> delay 2</span>
<span class="s2"> tell application "System Events"</span>
<span class="s2"> set textToType to "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span>
<span class="s2"> keystroke textToType</span>
<span class="s2"> key code 36</span>
<span class="s2"> end tell</span>
<span class="s2"> """</span><span class="p">)</span>
<span class="n">script</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</code></pre>
</div>
<h3>Multi-Intent Commands</h3>
<p>We can call OpenAI's API to autocomplete our prompt and extract multiple commands. We don't need to use OpenAI's API, and can also simply use Google's Flan-T5 model using HuggingFace's transformers library. </p>
<h4>Ze Prompt</h4>
<div class="codehilite">
<pre><span></span><code>You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.
Example:
Q: "Turn on the lights and turn off the lights"
A: ["Turn on the lights", "Turn off the lights"]
Q: "Switch off the lights and then play some music"
A: ["Switch off the lights", "Play some music"]
Q: "I am feeling sad today, play some music"
A: ["Play some cheerful music"]
Q: "{command_text}"
A:
</code></pre>
</div>
<p>This prompt gives the model a few examples to increase the generation accuracy, along with instructing it to return a Python list. </p>
<h4>Ze Code</h4>
<div class="codehilite">
<pre><span></span><code><span class="kn">import</span> <span class="nn">openai</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">getenv</span>
<span class="n">openai</span><span class="o">.</span><span class="n">api_key</span> <span class="o">=</span> <span class="n">getenv</span><span class="p">(</span><span class="s2">"OPENAI_KEY"</span><span class="p">)</span>
<span class="n">engine</span> <span class="o">=</span> <span class="s2">"text-davinci-003"</span>
<span class="k">def</span> <span class="nf">execute_with_llm</span><span class="p">(</span><span class="n">command_text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="n">llm_prompt</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"""You are provided with multiple commands as a single command. Break down all the commands and return them in a list of strings. If you are provided with a single command, return a list with a single string, trying your best to understand the command.</span>
<span class="s2"> Example:</span>
<span class="s2"> Q: "Turn on the lights and turn off the lights"</span>
<span class="s2"> A: ["Turn on the lights", "Turn off the lights"]</span>
<span class="s2"> Q: "Switch off the lights and then play some music"</span>
<span class="s2"> A: ["Switch off the lights", "Play some music"]</span>
<span class="s2"> Q: "I am feeling sad today, play some music"</span>
<span class="s2"> A: ["Play some cheerful music"]</span>
<span class="s2"> Q: "</span><span class="si">{</span><span class="n">command_text</span><span class="si">}</span><span class="s2">"</span>
<span class="s2"> A: </span>
<span class="s2"> """</span>
<span class="n">completion</span> <span class="o">=</span> <span class="n">openai</span><span class="o">.</span><span class="n">Completion</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">engine</span><span class="o">=</span><span class="n">engine</span><span class="p">,</span> <span class="n">prompt</span><span class="o">=</span><span class="n">llm_prompt</span><span class="p">,</span> <span class="n">max_tokens</span><span class="o">=</span><span class="nb">len</span><span class="p">(</span><span class="n">command_text</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">" "</span><span class="p">))</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="nb">eval</span><span class="p">(</span><span class="n">completion</span><span class="o">.</span><span class="n">choices</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="p">):</span> <span class="c1"># NEVER EVAL IN PROD RIGHT LIKE THIS</span>
<span class="n">execute_command</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
</code></pre>
</div>
<h3>Gluing together code</h3>
<p>To finish it all off, we can use argparse to only send the input command to OpenAI when asked to do so.</p>
<div class="codehilite">
<pre><span></span><code><span class="kn">import</span> <span class="nn">argparse</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">"command"</span><span class="p">,</span> <span class="n">nargs</span><span class="o">=</span><span class="s2">"?"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"The command to pass to Siri"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"What time is it?"</span><span class="p">)</span>
<span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'--openai'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="n">argparse</span><span class="o">.</span><span class="n">BooleanOptionalAction</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">"Use OpenAI to detect multiple intents"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
<span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">openai</span><span class="p">:</span>
<span class="n">execute_with_llm</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">execute_command</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
</code></pre>
</div>
<h2>Conclusion</h2>
<p>Siri is still dumb. When I ask it to <code>Switch off the lights</code>, it default to the home thousands of miles away. But, this code snippet definitely does work!</p>
<blockquote>If you have scrolled this far, consider subscribing to my mailing list <a href="https://listmonk.navan.dev/subscription/form">here.</a> You can subscribe to either a specific type of post you are interested in, or subscribe to everything with the "Everything" list.</blockquote>
<script data-isso="//comments.navan.dev/"
src="//comments.navan.dev/js/embed.min.js"></script>
<section id="isso-thread">
<noscript>Javascript needs to be activated to view comments.</noscript>
</section>
</main>
<script src="assets/manup.min.js"></script>
<script src="/pwabuilder-sw-register.js"></script>
</body>
</html>