Amazon Echo

derrick.thames

Hi,

I'm trying to integrate the Amazon Echo to my Netlinx master. I've seen snippets of code on the internet that are using a Raspberry Pi, so I'm assuming my Netlinx master can emulate the same function?
http://www.makermusings.com/2015/07/13/amazon-echo-and-home-automation/

Initially I'm simply trying to see the UPnP request the Echo sends out when you say "Alexa, find devices". In order to do that I:
1. Used Netscan to see what IP address it is set to (either .14 or .44)
2. Inserted the below code and watched in diagnostics. So far I don't see anything coming in.

Anybody have any experience yet?

Thanks so much - Derrick

DEFINE_DEVICE
dvEcho1 = 0:21:0; //Set the IP Address below
dvEcho2 = 0:22:0; //Set the IP Address below
dvEcho3 = 0:23:0; //Set the IP Address below
dvEcho4 = 0:24:0; //Set the IP Address below
dvEcho5 = 0:25:0; //Set the IP Address below
dvEcho6 = 0:26:0; //Set the IP Address below


DEFINE_START //Wasn't sure what port to open so defined multiple ones on different ports
WAIT 200
{
IP_CLIENT_OPEN (3,'192.168.0.151',3663,1)
IP_CLIENT_OPEN (21,'192.168.0.14',1900,3)
IP_CLIENT_OPEN (22,'192.168.0.14',49153,3)
IP_CLIENT_OPEN (23,'192.168.0.14',50000,3)
IP_CLIENT_OPEN (24,'192.168.0.44',1900,3)
IP_CLIENT_OPEN (25,'192.168.0.44',49153,3)
IP_CLIENT_OPEN (26,'192.168.0.44',50000,3)
}

DEFINE_EVENT
DATA_EVENT [dvEcho1]
{
ONLINE:
{
SEND_STRING 0, "'Echo1 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo1 has gone OFFLINE'"
IP_CLIENT_OPEN (21,'192.168.0.14',1900,3)
}
}

DATA_EVENT [dvEcho2]
{
ONLINE:
{
SEND_STRING 0, "'Echo2 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo2 has gone OFFLINE'"
IP_CLIENT_OPEN (22,'192.168.0.14',49153,3)
}
}

DATA_EVENT [dvEcho3]
{
ONLINE:
{
SEND_STRING 0, "'Echo3 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo3 has gone OFFLINE'"
IP_CLIENT_OPEN (23,'192.168.0.14',50000,3)
}
}

DATA_EVENT [dvEcho4]
{
ONLINE:
{
SEND_STRING 0, "'Echo4 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo4 has gone OFFLINE'"
IP_CLIENT_OPEN (24,'192.168.0.44',1900,3)
}
}

DATA_EVENT [dvEcho5]
{
ONLINE:
{
SEND_STRING 0, "'Echo5 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo5 has gone OFFLINE'"
IP_CLIENT_OPEN (25,'192.168.0.44',49153,3)
}
}

DATA_EVENT [dvEcho6]
{
ONLINE:
{
SEND_STRING 0, "'Echo6 - ',DATA.TEXT"
}
OFFLINE:
{
SEND_STRING 0, "'Echo6 has gone OFFLINE'"
IP_CLIENT_OPEN (26,'192.168.0.44',50000,3)
}
}

ericmedley

In the code:

DATA_EVENT [dvEcho1]
{
ONLINE:
{
SEND_STRING 0, "'Echo1 - ',DATA.TEXT"

you don't really do anything apart from opening the socket. You do send yourself the message to your termijnal by the "send_string 0"

conversation methods (APIs) are all different. But, typically upon making a connection, something along the lines of either the device says "hello" or it sits there waiting for you to initiate the conversation.

If the device is sitting there waiting for you; then you will probably put something in the ONLINE: event as that is how your code knows that a connection has been established. So you might put a Send_String dvEcho1, 'This is my hello to you' or whatever the conversation starter is.

If on the other hand, the remote device (server) does indeed say howdy first, you will need to watch in the DATA_EVNET>STRING: event and parse for whatever it sends first - prompting you to send your command.

DATA_EVENT [dvEcho1]
{
STRING:
{
if(find_string(data.text,"Hello - I am an Echo - what is your request?",1)){
    send_string dvEcho1,'this is my first command';
  }

I'm guessing your conversation will go one of those two ways.

GregG

I registered as a developer for the echo as soon as it came out, and so far there is no API for local connections into the echo on any port. It integrates to other devices using outgoing connections only.

You can either do what the guy in that link you shared does and emulate a UPNP responder and multiple socket listener on the AMX master (possibly hard/slow).
Or use the alexa skills kit and a public web server to get hits for data in a chain like echo->amazon->your-public-server->amx.

Either way it's not quite as flexible as I'd like for custom home automation at this point.

derrick.thames

Thanks for the feedback guys, and thank you for the code snippets!

When I open the port I get "ONERROR". I'm assuming this is because the Echo doesn't open it's port until you initiate the search for devices?

As this is my first time connecting with a device of this type, I'm assuming systems like Lutron homeworks simply have an open port at all times, and then you connect to it with the login/password?

Initially it would be nice see the string coming from the Echo when you tell it to search for devices. Should I open the port "IP_CLIENT_OPEN (21,'192.168.0.14',1900,3)" right after the Echo starts its search?

GregG

UPNP requests go out using UDP broadcast, so you have to open a UDP socket listener on the amx master using the IP_MC_SERVER_OPEN() function. After the AMX is listening, then you trigger the echo to search.

derrick.thames

Thanks alot!

I added in this code and got error messages, however at least it seemed like something was trying to come back:


I'm not sure what port the UDP messages are coming in on so I defined 4 ports for 4 different settings:
DEFINE_DEVICE
dvEcho1 = 0:21:0; //Set the IP Address below
dvEcho2 = 0:22:0; //Set the IP Address below
dvEcho3 = 0:23:0; //Set the IP Address below
dvEcho4 = 0:24:0; //Set the IP Address below


Then I opened ports:
DEFINE_START
WAIT 400
{
IP_MC_SERVER_OPEN (21,'192.168.0.17',1900)
IP_MC_SERVER_OPEN (22,'192.168.0.17',50000)
IP_MC_SERVER_OPEN (23,'192.168.0.17',49153)
IP_MC_SERVER_OPEN (24,'192.168.0.17',80)
}



And had string events for seeing what comes back:
DEFINE_EVENT
DATA_EVENT [dvEcho1]
{
ONLINE:
{
SEND_STRING 0, "'Echo1 - ',DATA.TEXT"
}
}

DATA_EVENT [dvEcho2]
{
ONLINE:
{
SEND_STRING 0, "'Echo2 - ',DATA.TEXT"
}
}

DATA_EVENT [dvEcho3]
{
ONLINE:
{
SEND_STRING 0, "'Echo3 - ',DATA.TEXT"
}
}

DATA_EVENT [dvEcho4]
{
ONLINE:
{
SEND_STRING 0, "'Echo4 - ',DATA.TEXT"
}
}



Watching Output diagnostics I got this:
Line 4 (09:41:51):: MCServerOpen IP_ADD_MEMBERSHIP error 0x16
Line 5 (09:41:51):: Memory Available = 32830088 <21456>
Line 6 (09:41:51):: MCServerOpen IP_ADD_MEMBERSHIP error 0x16
Line 7 (09:41:51):: Memory Available = 32814088 <16000>
Line 8 (09:41:51):: MCServerOpen IP_ADD_MEMBERSHIP error 0x16
Line 9 (09:41:51):: Memory Available = 32798088 <16000>
Line 10 (09:41:51):: MCServerOpen IP_ADD_MEMBERSHIP error 0x16
Line 11 (09:41:51):: Memory Available = 32782088 <16000>
Line 12 (09:41:51):: CIpEvent::OnLine 0:21:1
Line 14 (09:41:51):: Echo1 -
Line 16 (09:41:51):: CIpEvent::OnLine 0:22:1
Line 17 (09:41:51):: Echo2 -
Line 18 (09:41:51):: CIpEvent::OnLine 0:23:1
Line 19 (09:41:51):: Echo3 -
Line 20 (09:41:51):: CIpEvent::OnLine 0:24:1
Line 21 (09:41:51):: Echo4 -

ericmedley

You need to put your send_stirng 0 statements in the STRING: event. not ONLINE: Or you can look at the ONERROR: event to see if there was some error.

GregG

Also, the IP address that you open for multicast has nothing to do with the IPs of the devices in use, try: IP_MC_SERVER_OPEN (21,'239.255.255.250',1900)

derrick.thames

Thanks Greg! Rookie mistake on the string event. I now saw the search come in:
Line 15 (10:33:14):: Echo1 - M-SEARCH * HTTP/1.1$0D$0AHOST: 239.255.255.250:1900$0D$0AMAN: "ssdp:discover"$0D$0AMX: 15$0D$0AST: urn:Belkin:device:**$0D$0A$0D$0A
Line 16 (10:33:14):: Echo1 - M-SEARCH * HTTP/1.1$0D$0AHOST: 239.255.255.250:1900$0D$0AMAN: "ssdp:discover"$0D$0AMX: 15$0D$0AST: urn:Belkin:device:**$0D$0A$0D$0A
Line 17 (10:33:17):: Echo1 - M-SEARCH * HTTP/1.1$0D$0AHost:239.255.255.250:1900$0D$0AST:urn:schemas-upnp-org:device:InternetGatewayDevice:1$0D$0AMan:"ssdp:discover"

In order to respond do I now need to open a port via IP_CLIENT_OPEN (dvIPClient.Port,'239.255.255.250',1900,IP_UDP)? I'm not clear on what the (dvIPClient.Port) is on this call? The port where I previously received the broadcast (21)?

Best,

Derrick

ericmedley

derrick.thames wrote: »

Thanks Greg! Rookie mistake on the string event.

you're welcome.

derrick.thames

Sorry Eric. Thank you too!

GregG

For what you are trying to do here the UDP port is the same place where you send the initial reply, the reply you send will then include the IP address and port on the AMX master where your code is listening for switch controls to come in from the echo.

So in your string event for port 1900 UDP you would put something like:

Send_String dvEcho1,"'HTTP/1.1 200 OK',13,10,
'CACHE-CONTROL: max-age=86400',13,10,
'DATE: Thu, 18 Jun 2016 18:07:01 GMT',13,10,
'EXT: ',13,10,
'LOCATION: http://192.168.0.150:6000/setup.xml',13,10,
'OPT: "http://schemas.upnp.org/upnp/1/0/"; ns=01',13,10,
'01-NLS: 905bfa3c-1dd2-11b2-8928-fd8aebaf491c',13,10,
'SERVER: Unspecified, UPnP/1.0, Unspecified',13,10,
'X-User-Agent: redsonic',13,10,
'ST: urn:Belkin:device:**',13,10,
'USN: uuid:Socket-1_0-221517K0101769::urn:Belkin:device:**'13,10,10"

This would tell the echo that you have a wemo switch at IP 192.168.0.150 (substitute the ip of your master) on TCP port 6000 (need a unique port for every virtual switch)

The code you write to listen for switch on/off commands would now need to be written to listen on port 6000 TCP using IP_SERVER_OPEN()

You will have to send one of those reply strings, each with a different port listening on the master for each virtual wemo switch you want to pretend to be.

The code you have listening on port 6000 (or whatever you choose) will initially have to serve the file "/setup.xml" with more details about your fake switch, and then it will need to handle all the other POST requests that the echo will send to that switch --- Congratulations, you're well on your way to writing a web server in NetLinx code.

derrick.thames

Thank you so much Greg. This is completely clear to me. It will be so cool once I get it into the Netlinx master so I can control lights, hvac, music everything on my house system.
I think it holds 30 wemo switches and I doubt I'll go over 30 commands.

Thanks again!

Best,

Derrick

derrick.thames

Hopefully last question Greg. With IP_SERVER_OPEN (PORT1, SvPort, IP_TCP), is the PORT1 a local port I define to use in netlinx and then the SvPort Server port the 6000?

GregG

What I really wanted from the echo was one of two things, either:

1) The ability to pass the raw text from the amazon server speech recognition engine to my server.
The ASK (Amazon skills kit) forces you to spell out every possible combination of words you want to be able to recognize, and if the user triggers one, then your server gets a message. This is far too limiting. Plus it makes you preface all the commands with the tag name of your "skill". Like "Alexa, tell Jarvis to turn on the TV."

or
2) The ability to write an internal app that can get the above data and do things with it inside the local network, like opening a single port to AMX sending requests.
Unfortunately, they seem to be playing favorites with the large manufacturers on this one and they are also probably charging an arm and a leg to get into the platform as an integrated application. Someone like Harmon might be able to get them to allow in integrated application, but independent programmers aren't getting anything in there, yet.

GregG

derrick.thames wrote: »

Hopefully last question Greg. With IP_SERVER_OPEN (PORT1, SvPort, IP_TCP), is the PORT1 a local port I define to use in netlinx and then the SvPort Server port the 6000?

PORT1 is the netlinx code's device port number, and svport would be the 6000, yes.

John Nagy

GregG wrote: »

What I really wanted from the echo was one of two things, either:

1) The ability to pass the raw text from the amazon server speech recognition engine to my server.
The ASK (Amazon skills kit) forces you to spell out every possible combination of words you want to be able to recognize, and if the user triggers one, then your server gets a message. This is far too limiting. Plus it makes you preface all the commands with the tag name of your "skill". Like "Alexa, tell Jarvis to turn on the TV.".

I hear you (no pun intended). Getting the text of the statement would allow us to use our existing speech recognition system exactly as we do for TPControl and Voice Viper, which scans an editable 2,000 item phrase library within the NetLinx in about a second. The size of the library allows 10 versions of every command, so exact phrasing can vary by user or time, and still work. But such a text hand-off from Echo looks unlikely.

However, we are looking at making a single skill that just triggers voice recognition on TPControl to begin, for the sole purpose of not having to press a button to start it. "Alexa, get Cindy," Once we know we should listen, we can start up our CINDY on a nearby Android or iThing with TPControl, and take it from there. Not ideal, but worth the extra hardware if the user really can't press the button... or doesn't want to.

There are several apps for Android that accomplish a hands-free activation, but the most promising of them (UTTER, with offline activation) has had the developer go silent for over a year now. The others seem to be massive battery suckers and likely privacy violation listening and passing everything to Google 24/7. Unacceptable!

Keep in touch if you want to collaborate or share thoughts.

derrick.thames

Thank you Greg!

I agree, being able to utilize the voice recognition capability of the Echo and have the raw data would be optimum.

[Deleted User]

Derrick and Greg, does the newly announced smart home skill kit help in your efforts? https://developer.amazon.com/public/community/post/Tx1KIRDSNFDHEA4/Amazon-Enables-Developers-to-Extend-Alexa-s-Smart-Home-Capabilities-a-New-Additi

GregG

AMX_Chris wrote: »

Derrick and Greg, does the newly announced smart home skill kit help in your efforts? https://developer.amazon.com/public/...s-a-New-Additi

It helps to eliminate the extra invocation to the echo, so you can now set it up to use "Alexa, turn on the lights" instead of "Alexa, tell <myappname> to turn on the lights". This levels the playing field with those other previously "built-in" home control manufacturers who were picked for the beta.

Unfortunately it does not eliminate the need for an "outside the home" server presence to which amazon will relay the user requests.

If you notice in this example diagram there is still no direct control link from the echo right over to the in-home devices:

https://developer.amazon.com/public/...home-skill-api

mixu

This would be such an interesting connection to have, from Echo to my AMX. Now Amazon is adding control for Sonos early 2017 and if I could control my AMX system from Echo as well, it would be quite something.

mixu

I am very surprised there is no more interest here to work on getting Echo talking to AMX. I know the other manufacturer has chosen Amazon as their partner and Harman is going with Watson from IBM. But who knows what the price for that will be. No news on a release date either.

ericmedley

mixu wrote: »

I am very surprised there is no more interest here to work on getting Echo talking to AMX. I know the other manufacturer has chosen Amazon as their partner and Harman is going with Watson from IBM. But who knows what the price for that will be. No news on a release date either.

It might have something to do with where that product lives in the world and where AMX lives. The Echo is kind of a "Resi" product, although not totally. AMX's presence in Resi is pretty muted. I would also add that this kind of product kind of lives in a world all its own too in that there is a trend for manufacturers to not necessarily partner with other things. You can spend a lot of time developing for this platform only to have it totally change on you the next product cycle in 12 months.

In my own time I've spent many hundreds of hours working on things like an IP module for Apple TV/iTunes players where there was no published API but also no one at Apple stopping us from doing it. You'd get it done and working only to have the protocol change one day, thus breaking your code. Another example was with my Autonomics media player. In that case it wasn't the actual manufacturer that broke the module but one of the streaming services served from it. The programmer of the Autonomics software was given resources to fix their web UI but not their control system API. His response was to shrug his shoulders and say, "sucks to be you"

It's a little hard to hang out at the playground when none of the kids really want to play with you.

[Deleted User]

While not really covered by Pro AV industry press, Harman is leading the way in this space and our efforts are being highlighted by mainstream tech press. The IBM Watson partnership is the first in a number of partnerships in this space between Harman and makers of Voice Control/Assistant platforms. Just as we have seen with BYOD, AV/IT Managers are pushed to offer solutions in their environments that started in mass market consumer electronics, but were not designed for enterprise IT environments. As a hardware partner, we have the ability to construct products that will conform with corporate IT security requirements while delivering the pro-level reliability and features you need as an integration professional. The IBM experience is fully customizable and connect into client data systems in a way that Amazon or Google Home would be unable to do. Because of this, Watson is the voice control product for the Pro market.

Cortana will be interesting given the Microsoft connection to corporate IT, but this platform and API is not as mature as the IBM offering and it will be interesting to see it progress. Just before CES, we were announced as the hardware partner for the first consumer Microsoft Cortana device:http://www.zdnet.com/article/harmankardon-readies-a-cortana-powered-speaker/

At CES, we were announced as the hardware partner of the first consumer IBM Watson device: https://www.wired.com/2017/01/lenovos-smart-speaker-marries-alexa-smarts-hardon-karman-sounds/

Additionally, our engineering teams have released speakers with chromecast built in to work seamlessly in a Google Home environment as a wireless audio end point within that eco system. The Harman Connected services team is a core software development partner for Google and I look forward to how this partnership will grow.

There are a number of synergies happening within the Harman engineering community and our trajectory in this space is clear. The JBL Horizon speaker for IBM Watson was designed by the Pro team and the AMX code development efforts behind the current IBM Watson Cognitive Room Experience will serve as a framework for other platforms going forward. I cannot comment on current projects, but I will say that I am excited about the voice control space and what Harman is doing to be a leader in this space for both the Pro and Resi markets.

mixu

I've solved the problem by using this in between http://www.hcatech.com/ it now translates the commands to serial to my NI-3100. Works great.

John Nagy

Can you explain what you mean by that... which problem this has solved, and how? I looked at the page you linked and don't yet see anything that implies "translating the commands to serial", but if you mean that you can get literal text strings that are the translation of speech it has heard, that's important.

If HCA only can deliver trigger information based on Alexa's interpretation of the phrases it heard, processed through the skills it has, it's still disappointing... we have a very capable text-parsing engine that runs in the Netlinx and can do a keyword search and match of three input text samples simultaneously through a 1,500 item database match list of phrases and related actions... in under 1 second. This works really well with TPControl's speech-to-text, as well as with Voice Viper. If Alexa could just send us text strings, it would work too.

Collaboration, anyone?

mixu

John Nagy wrote: »

Can you explain what you mean by that... which problem this has solved, and how?

This little software solved my issue of getting Alexa somehow talking to my AMX controller. I agree it is not the most clean way of doing it, but it works. I apologize for not being clear. You can define in the software what word Alexa responds to. You can turn it on or off. So you will always have to start by saying "Alexa, turn on..." or turn off.

I am then sending serial commands out to my NI-3100 from a PC (which I have always running anyways). You can define what you send out, I only send the name of the command in text, like "projector_on". It is quite a bit of work to configure every command, and sure would be easier to get text straight in to AMX without another layer to program.

But I am quite happy how it works for me. It has been very reliable and fast. The light scenes change right before Alexa responds "OK".

John Nagy

Thanks for the reply. It's what I figured... you are leveraging the trigger that Alexa decides to give, not the text. The need for yet another computer in the middle is also disappointing.

Ideally, we hope for a way with Alexa or the Google unit that will be a smart voice gateway with the following capabilities:
1. Hands-free initiation of commands
2. Independent value-add responses (Google search, information, perhaps control of some devices outside the AMX system's control)
3. Ability to transfer AV and AMX-controlled commands to the AMX as text strings, verbatim or ideally simplified (remove "please" and "can you" and "the" etc.) The transfer could be on literal command, such as "Alexa, tell CineTouch...." but better if a different key start word could be used... and/or if whatever command Alexa hears can't be figured out by Alexa, it would be passed to the AMX as text for another look instead of bounced as not understood.

mixu

There is a way not to have a local computer do the triggering, but it would involve some heavier programming. I used this info to build a control for my media player software Kodi from Alexa.

https://github.com/m0ngr31/kodi-alexa

The actual "server" part is stored on the Amazon AWS service. You do need a developer account for it, what [USER="644"]GregG[/USER] is referring to on his earlier posts. I don't know how much better this would be, at least you can have feedback from the object getting controlled. For example I can say "Alexa, ask Kodi which are the latest movies on my list" and I get a reply. Would be interesting to hear how difficult this would be to implement with AMX. Probably too much work because every system would be end up being completely different.

amdpower

I've had mine working for about a year. The skill is really just a passthrough. I send it to my home service which runs stunnel to translate https to http so that the AMX can accept the connection. Everything is just JSON back and forth and it is quite easy. If I get time I'l post how I did it with examples. I call the skill Max so you just say "Tell Max to arm the security system" or similar. I've got full control of everything and there isn't much to do other than parse the text which comes back. Very reliable.

MLaletas

Is that still required with the NX series TLS connection?