What are the Apple Voice First announcements from WWDC 2019 and why are they important?
What are the Apple Voice First announcements from WWDC 219 and why are they important? Today June 3rd 219 Apple announced two fundamental aspects of their Voice First roadmap to the future Voice Control OS Accessibility control Siri Shortcuts Conversational Siri Shortcuts + Siri Suggested Automation Both aspects are a massive step forward for Apple and create a robust foundation for the potential of innovation. Surprisingly enough both voice features do not connect directly. They are separated by use cases and design goals and this aspect may not be very clear today. Yet they form a powerful foundation to a Voice First future for Apple. Voice Control has been part of the iPhone and iPod touch since iOS 3.. It's similar to Siri but more limited in scope and aim. Yet on the other side Siri simply can not do some things Voice Control has been able to do for years. Voice Control only lets you control certain parts of iOS and MacOS. It is rightly classified as an accessibility feature. In fact Apple even has a user ge to choose from Voice Control or Siri 1 as both can not operate at the same time. Use Voice Control instead of Siri on iPhone (iOS 12) Instead of using Siri you can use Voice Control to make calls and control music playback. For example if you can use Siri because youre not connected to the Internet you can use Voice Control for music playback. Note When you turn on Voice Control you can talk to Siri. Voice Control and Siri are similar in the aspect that they both use speech recognition as a user interface. However Voice Control has a family history reaching back to July 29 1993 when Apple releases the Macintosh Quadra 66avputers with a full AT&T DSP Chip the first DSP built into a consumerputer. Specimen of a 1993 advertisement for Apple PlainTalk. italic The PlainTalk Of Apple Voice Control Coming 18 years before Siri PlainTalk was the first Apple system software to recognize and synthesize speech 2. Apple Macintalk was the first synthesize speech system on the Macintosh system on January 24th 1984. The original premise of PlainTalk was to control theputer as a proxy to what you would have d or clicked on. Although PlainTalk was tone of the most advanced system to recognize and synthesize speech in a consumerputer. PlainTalk became part of Macintosh System and continues on in modified forms up to the latest MacOS and iOS. By the late 199s PlainTalk became more of an accessibility product then a general purpose consumer product. This is because there was no intelligence in the system and the user was simply speakingmands that replaced keyboard input and mouse clicks. It was cumbersome and frustrated to many users that did not have a direct accessibility need. In Mac OS X 1.7 Lion and earlier PlainTalk speech recognition was voicemand oriented only not intended for dictation. It could be configured to listen formands when a hot key is pressed after being addressed with an activation phrase such as Computer or Macintosh or without prompt with visual and ual feedback about listening status availablemands and actions. It can alsomunicate back with the user using speech synthesis. The early versions of the speech recognition provided full access to the menus. This support was soon removed as it required too many system resources and recognition performance suffered. It was renamed in Mac OS X 1.3 as universal access technology. The user could launch items located in a special folder called Speakable Items by speaking the item name while the system is in listening mode. Apple shipped a number of AppleScripts in this folder. Additional functionality was also provided by individual apps that supported the system. There was also an API that let programs define and modify an available vocabulary. For example the Finder provides a vocabulary for manipulating files and windows. By OS X 1.8 Mountain Lion Apple introduced Dictation under Accessibility and Voice Control and it was intended for general input and transcriptions. Originally it required sending the audio data to Apple servers for processing by OS X 1.9 Mavericks Apple renamed this Voice Control and added the option to download support for dictation without an Internet connection for enhanced privacy. From Siri To SiriOS Seriously Siri has apletely different family history. With Siri Apple is using the results of over 4 years of research funded by DARPA via SRI International Artificial Intelligence Center. Siri Inc. was a spin-off of SRI Intentional through the Personalized Assistant That Learns Program (PAL) and Cognitive Agent that Learns and Organizes (CALO)Program 3. Siri has speech recognition and speech synthesis like Voice Control but is built on AI platforms that determine intents that may not be directly related to control of theputer system. Siri was one of the first AI Intelligent Assistants. Announced on October 4th 211 Siri was Steve Jobs last acquisition and at the time of his passing he thought Siri would be a very large part of Apple future. By March 22nd 217 Apple announced it acquired the 3 personpany that built Workflow 4. Workflow was an automation app for iOS that let you create workflows. A workflow is made of a series of actions executed in a single flow from top to bottom similar to AppleScript and HyperCard. You would press the Play button at the top and Actions will execute one after the other; once it done the output of a workflow the result will be displayed at the bottom of the chain of actions ormunicated visually or using Siri voice. Apple has since renamed Workflow to Siri Shortcuts. Interestingly enough PlainTalk in the 199s had AppleScript execution ability and it was possible to create veryplex actions controlling theputer and just about all software menus and actions via PlainTalk AppleScripts. Two Paths Diverge In Voice Today Voice Control and Siri with Siri Shortcuts are not connected. They are twopletely different projects and products with-in Apple. This is a fundamentally large flaw in the uniformity peoplee to expect from Apple. From the user perspective they see this two systems as voice interaction with theputer and do not make a distinction that one is primarily for Accessibility functionality and the other is a Voice First assistant. Indeed Apple took great effort to present Voice Control in this manner it still is a glaring inconsistency that this is not one system with Accessibility features that reveal their functionality to the user as needed or requested. I have little doubt that in future versions these two systems willbine. However I feel strongly it should have happened in 219. s Specimen video showing Apple Voice Control in use. italic Voice Control allows any user to bypass the keyboard mouse and touch screen with their voice on iOS and MacOS to control the functions of the OS and apps running on the OS. In a fundamental way it is a replacement of the keyboard mouse and touch screen for those that may not be able to control these as parts of the user interface in a traditional manner. The new version of Voice Control in the new versions of iOS and MacOS does use many AI aspects but this is more weighted to understanding the intent of dictation like Add heart Emoji. Voice control allows for rich editing features that let you make corrections as you dictate. It also allows you to navigateprehensively using voicemands and voice gestures. There are developer APIs that can deeply connect to Voice Control but the system places numbers or names next to everything you can tap and superimpose a grid onscreen to Voice tap or Voice click when there aren direct controls or buttons accessible. Voice Control does not use Siri directly but can use some Siri Shortcuts. It is also very difficult to allow for Siri and Voice Control to coexist on iOS and MacOS simultaneously. Currently the user will need to choose one over the other. Clearly Apple has made Voice Control decidedly an accessibility feature. Although no longer deeply hidden they moved the Accessibility menus to the top of iOS and MacOS Settings for easier discovery. Apple also allows new button and swipe access to Voice Control on iOS. Voice Control will use the camera on the device to detect if the user is actually speaking to the system by detecting the user's attention focus. It will also detect if the user is authorized to access the device. This is a simple but uniquely powerful feature. Much like FaceID this feature shows a path for how Voice First systems can understand who the user is talking to and determine a course of action. The New Siri Shortcuts OS Integrated App In the first version of Siri Shortcuts users could create their own voicemands to launch a favorite app or a specific action. For example order pizza could trigger a mobile order on the Dominos pizza app. With the updated Siri Shortcuts developers can suggest a voicemand and users can opt to instal it with a tap of a button to be available by Siri voice interaction in the future. The Siri Shortcuts app is now built into iOS 13. It features an enhanced new gallery view that lets users install preconfigured shortcuts from the developers of your installed Apps and from Apple. Shortcuts can also be paired with actions from other apps with information passing between apps. And with the new Automations tab users can set up shortcuts to run automatically based on events and triggers. The new Siri Shortcuts now allows parameters to make the flow of interaction more conversational. Called Conversational Shortcuts in the WWDC demo a user wanted to find something to cook from their preferred recipes app launched via a Shortcut that displayed a list of their favorite recipes. When they picked a recipe from the list it would then take the user to that recipe and start playing the cooking instructions. This is allows Siri to ask follow-up questions with full continuity. The Conversational Shortcuts allow Siri can to ask follow-up questions which allows your shortcuts to pass parameters along the way to any app and Siri itself. For example when a user says Order takeout Siri can ask Which order would you like? and present a list of favorite orders to choose from a food ordering app. Conversational Shortcuts are a profoundly powerful feature. As it is a new pipeline ofmunication passing information and personalization. In effect Siri is writing Siri mini apps in real-time . Although currently this is limited to just the Shortcuts framework it is an extraordinarily powerful feature that is accessible to non programmers in a simple scripting environment. Siri Suggestions and Suggested Events allows Siri to learn the user routines and to suggest relevant shortcuts from your app at just the right time on iOS and watchOS. Siri can also detect events in your apps and create helpful suggestions to add them to Calendar or reminders to check in with your app and directions when needed. This will help those who've never used Siri Shortcuts before their habits and personalize them adding AI-assisted templates for things like heading home to going to the gym. The Automations tab in the Shortcuts app allows users to set when to run any Shortcut when creating a custommand in the Siri Shortcuts apps. For instance you could configure a Shortcut to run based on time of day when you start a workout on your Apple Watch when you connect to CarPlay. Automator workflows can be cascaded into very intricate actions that when paired with parameter passing. The editor in the updated Shortcuts app now enables full configuration of an app actions including the ability to pass information in or out of the action through parameters. This allows an app actions to bebined with others for more robust multi-step shortcuts. In practice this would allow users to do something like run a Shortcut for a restaurant delivery app choose their meals place the order then the whole family what for dinner and when it will arrive. Siri Suggestions and Suggested Events helps users quickly aplish tasks related to apps using their voice. Siri intelligently pairs users daily routines with apps to suggest convenient shortcuts right when theyre needed on the lock screen Search bar or from the Siri watch face. Siri also seems to be aligning to the new Apple Cards feature in iOS 13. Apple Cards allow small informational elements that are easily dismissed and can form the basis of a Voice First environment where ephemeral and conual visual information can be presented as an overlay to the existing screen or even the lock screen. This feature is a requirement for a Voice First OS like SiriOS. Because of Apple privacy centric approach Siri Suggestions are held entirely on the device and is not on a cloud to be mined for adverting targeting or worse. Thus as Siri gets to know you at the deepest level yet there is somefort in knowing the system was designer to maintain privacy and security. Siri learns locally on the device to maintain privacy. Apple on-device CoreML will allow for BERT DeepSpeech Open AI GPT and many other AI based systems to run locally and not in the cloud. This is a dramatic contrast to any other Voice First platform. It is part of Apple big privacymitment. Siri intelligence is a system wide enhanced feature that offers personalized suggestions in Apple Podcasts Safari and Maps. Siri can even detect reminders in messages and events in third-party apps. As you longer more descriptive sentences Siri can Conner to reminders that will automatically understand and provide relevant suggestions. For example when youre chatting with someone in Messages Siri recognizes possible reminders and makes suggestions for you to create them. Apple still has limited Siri to 12 domains and intents like fitness messaging payments or ride booking integrated to let the user perform certain actions or intents in response to spokenmands and questions. The new Siri Shortcuts app allows for a work around but at some point domains and intents need to be deeply indigenous to the OS to be effective and to monetize for the App developer in a conclusive manner. Siri Shortcuts do not currently work on MacOS directly. There is of course the ability to run Siri Shortcuts inside of Marzipan which is essentially an iPadOS simulator. It will not be indigenous to MacOS at this point. A Big Step For Apple Voice First But This Is Not SiriOS Today Apple made great strides and fixed many elements they have lacked with Siri over the years. In a vacuum without the Voice First revolution of the last 5 years what Apple announced would have been stunning. However the future is Voice First as much as it was mobile and gesture based ten years ago and hypered Internet twenty years ago and graphical and mouse driven thirty years ago. The primary user interface has changed about every 1 years since the dawn of theputer age 5. Specimen from the author demonstrating UI transitions form 1967 forward. italic Apple invented the talkingputer in 1984 the talking and listeningputer in 1993 and the talking and listening Voice First Intelligent Assistant in 211. Over the last eight years I think just about every informed and astute observer inside of and outside of Apple understand that Apple let Siri languish with benign neglect. This came about through a confluence of internal biases that always form in even the best legacypanies. Apple simply did not take the shift to Voice First ethos seriously. This became abundantly clear by 215 as Amazon Echo products featuring Alexa became part of the a class of devices Voice First devices that are the fastest adoption of any technology in history. I wrote the definitive article in 215 that predicted Voice First and the explosions adoption that followed 6. Specimen chart showing Voice First as the fastest adopted technology in history. italic Like all disruptions and paradigm shifts the current leaders of technology simply find it impossible to fathom the changes. Even when it is presented with the indisputable facts of adoption on their own platforms. They see it as a feature and not a product. Compounding the problem is the industry observers and supporters that also deny there is a change. It bes an echo chamber of Voice First is not the next thing. Yet the reality can be found empirically. It is user driven to a level never seen in the tech world. The youngest cohorts and the oldest cohorts are driving Voice First approaches on iOS Google searches millions of Alexa devices. I have been on record for over 1 years that Apple needs a VoiceOS a SiriOS. Just as much as they need WatchOS TVOS iPadOS iOS and MacOS. The only thing different is SiriOS can live independently on a VoiceFirst device or co-habituate with the other OSs to build a layer of AI based Voice First Intelligence on the system. Apple could allow access to SiriOS with security via any device even an Alexa device. This is how Apple Music should have been delivered to Alexa devices through the intelligent channel of SiriOS connected through Alexa Google Assistant or any other system. This is an Apple Service that is worthy of Apple as it can bring uniquely powerful Siri technology along with music podcasts and video. Not only to control the functions but to conually interact with the user. This can be an Apple subscription that is not equal to Spotify Google Play Amazon Music. It would be an intelligent delivery channel. The Voice First future is not just about voice it is not Voice Only and it is not about the direct accessibility of controlling the OS like Voice Control open photos select number 8 copy. It is about the deep con Siri can form to actually get work done for you. This does not require General Artificial Intelligence as some folks in the tech world embarrassingly pronounce. It requires the system to know you your goals and intents with deep con. And this can be done with no major advances in AI or chips. It just requires vision leadership and a will to act. This is quintessential Apple. I wrote about this in early late 218 6 I said For Apple to be successful with any Voice First move it is incumbent they build the best experience for the user and for the developer. This means a game changing way to monetize developers with a marketing system like the App Store but not an app store and a value proposition to the user. None of these options will take place if Siri is just an app or an appendage to iOS. It needs to have a clear path as a stand-alone OS that can be ambient. The first candidate device to have SiriOS would be a $99 Voice First device designed by Apple. SiriOS would be a co-OS on the iPhone iPad Apple Watch (limited) and Mac platform. It would run independently of the host OS yet interact with it on a global level. This sounds moreplex then it is SiriOS would coexist on top of Darwin. Apple Watch has an OS Siri currently does not. It is foolish to try to retrofit this modern AI based SiriOS on top of an existing iOS or MacOS system. Clearly just like iOS is based on MacOS SiriOS would also share this heritage with notable user gelines function and abilities that a Voice First OS requires. There is much more to this but the rest would be best delivered to Apple directly and not in a public environment. Apple still has a window to make this vision of a SiriOS viable. I assert the window is about 24 months from the time of this writing. Apple can accept the market demand of millions of Voice First devices and shift their thinking or they can continue to lose ground. The first step is to find a Collaborate with the leading authority on the Voice First revolution. that has identified the future with a track record or they can try to find folks that are guessing about it. Either way Apple will need to find all the talent possible both technical and visionary to surpass the 11 person Alexa Army that is just about hiring everyone possible. s html