Learning from Living with AI Assistants in Smart Home

We humans are quite interesting species. A combination of chemistry-driven emotions with the logic and amazing capability to build incredible models of the world that all are wrong but some are useful is what we feel makes us special.

Human’s brain is an infinite source of wonders. We want to reach for the stars, and we want to build an AI that is for far from where we are today, our ambitions are pretty much unlimited.

But getting there is hard. So hard that we’ve used to a modern tactic: fake it until you make it.

Take, for example, AI assistants of today. We love them so much, and we are so proud of what we’ve achieved, we can’t even agree how we call them. Personal Digital Assistants? Personal Intelligent Assistants? Personal AI Assistants? AI Assistants? Enterprise Intelligent Assistants? PDA, PIA, PAIA, AIA, EIA — lots of acronyms. But except the first one all pretend to be “intelligent”.

It’s a nice thing. As long as nobody dives deep into what “Intelligence” means it’s fine. I mean, sure, there’s some intelligence built into our assistants. Sure. But what can they really do?

After living with Cortana between 2014 and 2018, and spending much of 2018 building a couple of scenarios for Yandex’s Assistant called Alice, I wondered what has been achieved by the biggest player in the market, Amazon. For sure, Yandex got a few cool things like Chit-Chat, but I was pretty sure Alexa was way ahead. It simply had to be more intelligent.

To experience this life with AI assistant everywhere, I’ve made several things: downloaded and started using Alexa as my primary assistant on my phone, installed it as my desktop assistant, purchased big Echo (2nd gen), and Echo Dot (3rd gen). Later I’ve got Jabra Elite 65t with Alexa inside, as well as installed “Ultimate Alexa” app on my Android Wear smartwatch, and got Vuzix AR Blade with Alexa in it. Finally, I’ve brought lots of smart home equipment from US and installed all of it in our Kornevs Country Residence in Russia, in an effort to make it smart.

In this note, I’ll share my experiences of living in a Smart Home enabled by Alexa (and Yandex Alice), half a year later.

What Should an AI Assistant Do?

Before Alexa I had experience with Cortana (on my Windows-based and Android-based phones, my smart band, and my Windows PC), as well as Yandex Alice (on my Android phones, and my Windows PC). I’ve shared some of my thoughts on the subject before ( UX Issues, Can Facebook Build A Better Digital Assistant, Few Thoughts on AI Assistants). This experience, Nielsen Norman’s group article on Intelligent Assistants’ Usability, along with a my decade-long passion for ubiquitous computing, allowed me to formulate some of the expectations:

• Voice input: AI Assistant should support efficient and low-error speech recognition.
• Natural Language Understanding: AI Assistant should understand complex phrases (e.g., multiclause), produce same results for equivalent queries, better understand complex intents in the same phrase.
• Pervasiveness: AI Assistant should be available everywhere.
• Multivariate User Input: AI assistant should support text input, voice input, more complex UI input (touch, mouse, gestures, etc.)
• Extensibility: AI Assistant should provide extensibility on the scenarios level
• Cross-Device Experiences: AI assistant should provide a coherent cross-device experience
• Powerful Smartphone App w/ Deep OS Integration: AI assistant should have a fast and powerful app for your phone (unlike Alexa) and integrate as deeply into the phone as possible (like Cortana)
• Reliability and Offline Support: AI assistant should be reliable and be capable of working offline
• Social Graph Awareness: AI assistant should be knowledgeable of your social graph (not exactly your Facebook graph, btw), so that you could ask your assistant to do something to those whom you care about
• Knowledge Graph (World | User): AI assistant should be able to know what you care about, remember things that are important to you, and let you get back to them
• Agency: AI assistant should be able to operate on your behalf
• Emotional Connection: AI assistant should be able to create and maintain emotional connection with its user
• Cross-AI Assistant Collaboration: AI assistant should be able to talk to other assistants to solve it’s user’s problems

While these focus areas touch AI Assistant user experience in multiple areas, in this note I’d like to highlight my experiences with AI assistants at smart home.

Our Smart Home, Circa July 2019

Before I can share our experiences, let me show you how (logically) smart home equipment is organized in our home.

Most of the smart home equipment had been purchased in USA during my last stay there last Summer, while the rest has been either purchased and delivered from overseas, or purchased locally (mostly Smart Home devices by Rubetek).

We have smart home equipment from several OEMs: Ring, Blink, Rubetek, TP-Link, Smart Life.

My original goal was to make sure that I can control all of the home equipment using Alexa, however, some of the Rubetek devices work quite poorly with it, and thus we’ve added a few Yandex.Mini stations to the mix. We also use Yandex.Station to watch movies through our KinoPoisk subscription (similar to Netflix).

What Can Alexa in Smart Home Actually Do?

In short, Alexa was our both biggest source of excitement and biggest source of disappointment. It’s quite sophisticated — things like Drop-In that allow you to use Echos like intercom are amazing, it is quite intelligent in understanding which device is the closest to you to listen to you, it control our smart home equipment.

However, as we call it now, we wanted to get a Smart Home, but after spending about $3K on all this equipment, and after living with it, we call our home a Mad House.

Let’s look at how it actually works, based on the list of the core expectations.

• Voice Input — given that both of us (as well as our rare guests) are not native speakers, we do realize that Alexa might not be able to support efficient and low-error speech recognition; empirical feeling is, Alexa understands me in 70% of the cases, and Alexa understands my Mom in 30% of the cases (but there’s a stupid reason, see below)
• Natural Language Understanding — Alexa sometimes is able to execute several commands in a row (e.g., switch off Master Bedroom Light, switch on Living Room Light, switch off Pump), but you can’t rely upon this functionality. However, it often fails to understand my Mom: we’ve named all Lights by their location (Mom Bedroom Light, Master Bedroom Light, Living Room Light, Dining Room Light, Kitchen Light etc.); Alexa often fails to understand which room we mean. Even though each room with lights has one Echo device in it, Alexa might ask you again which of the lights you meant. It’s very annoying.
• Also, from product perspective, there’s a disappointing approach to support multiple users under one roof; you’re supposed to add each household member to the so called “Amazon Household”, to allow user to command your equipment. However, if you do that, and enable voice identification, system will tell to other members that they are not allowed to use “Daniel’s smart home equipment”. We had to disable voice identification, but even after that, system can recognize my Mom, and then reject her asks to switch on/off lights.
• Pervasiveness — this was very easy (though quite costly) to implement; while “Her” movie expects you to wear earbuds, at home it’s quite uncomfortable and impractical. Instead, we’ve placed Echo units in all places we might need to ask Alexa to help us.
• Multivariate User Input — this is a problem, even for Smart Home. In theory, you can use your Alexa app on your phone to control your smart home equipment. In practice, it’s quite problematic. All of the equipment is available through their respective apps, but it’s impractical to go to each of them. However, Alexa’s app is too often slow or simply unresponsive, and it’s very painful to use it for smart home control. Thus, I have to resort to apps (and remember which app to use when). Oh, and my Mom or guests can’t use this at all. I’ve also wanted to use our Fire tablet as a visual UI for smart home, but unfortunately exactly the model we have doesn’t support rich Alexa UI.
• Extensibility — well, smart home scenarios are extensible by design.
• Cross-Device Experiences — here, I’d say we have a few scenarios: Reminders, Smart Home control, Music, Calls.
• Reminders — if you set a Reminder on a device in Dining Room, it will show a notification on your phone when you’re in office, but if app is dead, no notification will appear. App is often dead, so I can’t rely on Alexa Reminders.
• Smart Home control — so-so; when Alexa app works, and all of the OEM skills work fine, and sun is in the right position, yes, you can control your smart home equipment both through Alexa app’s UI and through voice commands on Echo devices.
• Music — so-so; when Alexa app works, you can control music that is playing on Echo device(s) from your phone.
• Calls — so-so; when Alexa app works, you can use Drop-Ins and Announcements to send them to your Echo devices. Oh, and starting with November, Drop-Ins often require us to spend a minute or more to get a connection between devices. It might take up to 5 mins (!).
• Powerful Smartphone App w/ Deep OS Integration — this is funny. In theory, Alexa’s app is super powerful these days, it can do lots of things. But given how often it is slow to respond or unresponsive, I’d say using Alexa’s app is an experience you won’t love.
• Reliability and Offline Support — given that Alexa often misunderstands us; often recognizes Mom and denies her asks to control smart home equipment; its app on Android is slow or unresponsive; I’d say, Alexa doesn’t provide a reliable UX. Oh, and forget about offline with Alexa.
• Social Graph Awareness — while Alexa knows about my Mom, I couldn’t find a way to teach it about my friend (who just got Echo Dot 3 for his Birthday a couple weeks ago). Alexa doesn’t know anything else though it can see my contacts.
• Knowledge Graph (World | User) — most of my general knowledge questions to Alexa turned out to be too specific for it to correctly translate them to its underlying Evi-based knowledge graph. As of User Graph, Alexa can keep your Lists, Reminders, and Remember things for you. Lists are a pain in the ass (they are a part of Alexa app, and I don’t want to start about inefficiency of using voice to control your Lists like Shopping). Reminders are a pain for similar reason. Finally, while Alexa can remember things for you, there is no efficient way to edit them, or simply retrieve them. I’d say, this is quite an underinvested area, too.
• Agency — well, Routines in Alexa are quite good. We use them for smart home equipment control. Mostly they provide IFTTT-like experience, and they have lots of integrations. But they are as good as integrated services are, and it turned out, most of the Rubetek hardware has awful software integration with Alexa, so using it with Alexa is a pain. Oh, and TP-Link lamps often stop responding to Alexa. Other than Routines and Reminders, I won’t say Alexa is good at anything else in this regard.
• Emotional Connection — well, full failure on Alexa side. It is an emotionless thing. It can’t understand emotions. It can’t support a chit-chat (“Let’s Chat” thing allows you to chat to Alexa Prize socialbots; while that’s cool, it’s not a chit-chat like one in Yandex’s Alice).
• Cross-AI Assistant Collaboration — well, formally, you can use Cortana together with Alexa. In practice, I’d rather prefer if Alexa could use Yandex’s Alice as proxy to control equipment it can’t control efficiently directly (due to a failure to deliver solid UX on Rubetek’s side. This is not an option (perhaps too early; or, well, mostly a low priority for Yandex or Amazon).

Ending Thoughts

Alexa is super promising, and there’s a lot of good work in it. However, in all of these areas except Pervasiveness, it is mostly mediocre.

It is unreliable, it doesn’t provide any offline support, it doesn’t understand emotions, it fails to understand most of the knowledge graph queries, it’s awful for remembering and retrieving important stuff, its cross-device experiences lack quality and stability, its NLU and voice input quality is far from being adequate for the everyday life.

We wanted to get a Smart Home.

Instead, we’ve got a Mad House. We constantly argue with Alexa (and Yandex’s Alice). Shout out loud on them. We have to be super patient to use voice control. Often failures are really painful.

We will continue using Alexa (and Yandex’s Alice) in our home, but all in all, as of today I wouldn’t recommend modern AI assistants and supported Smart Home equipment for everyday users. Too much of headache, too silly NLU, too many product-level errors, and lots of problems that make the whole experience too painful.

In one of the upcoming posts I’ll share some of the thoughts on what could realistically be done to make modern AI assistants be better (if not good enough) for everyday use.

Originally published at https://www.facebook.com/notes/356584012116581/ on January 6, 2020.