Mac Home Automation

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====Raw Protocol Docs APIv2====
XTension talks to it’s plugins via a socket connection. Currently most plugins run on the same machine as XTension but there is support for allowing the plugins to connect from external machines as well. If you’re writing your plugin in Python or other future supported scripting systems then all this work will be taken care of by the XTension object that is provided by the included scripts. This information is provided for those that might wish to implement a connection from an unsupported scripting system or a binary compiled application. Note that at this moment all communication at this level with XTension is unencrypted. No SSL support is available for this channel. The reason being that most or all of the communication at this level is between apps running on the same machine and so encrypting it doesn’t make any sense. If remote connections are necessary over the internet I would recommend using a VPN or other encrypted channel to forward the data such as an SSH tunnel.

===Supported Languages===
At this moment the only examples are for connecting with Python. Since the connection to XTension is through a socket and the protocols are documented any language capable of implementing these protocols would work just fine though you might have to implemented these protocols yourself. I currently have release level plugin implementations in Python and [[http://xojo.com/|Xojo]], I also have an alpha level implementation written in Node JS for Javascript. That is a problem in releasing a plugin as you would either have to embed the NodeJS interpreter into your plugin or require that the user install it and it’s other dependencies before installing your plugin. At this moment I don’t recommend using that though there are other possibilities in the future for how I might handle javascript plugins. 

A Xojo plugin is an excellent option if you require a binary application. The downsides are that the application is larger but otherwise those plugin libraries are well debugged as I use them myself quite a bit.

Another option is to wrap a command line utility in some simple python code rather than start from scratch. If the command line utility is not already installed on MacOS and licensing permits it you can include the binary utility in the plugin itself so that no other installations are necessary. This is the tack I took with the [[supported_hardware:switchbot|Switchbot]] plugin. Since controlling BluetoothLE would have required new python libraries be included and I wanted to provide a command line utility for it anyway I wrote the low level bluetooth handlers into a command line utility and included it inside some relatively simple Python code. You can examine the code as an example by downloading the switchbot plugin linked to above.

===What is a plugin===
In XTension, a plugin is a separate command line application that communicates to XTension via a socket. The plugins do not share memory or CPU space with XTension itself so a misbehaving plugin is less likely to be able to degrade the performance of XTension itself. This also means that plugins will better be able to take advantage of multiple processors or other resources available on the server. Plugins which lose their communications with the device they are meant to be talking to or otherwise have errors can either recover from those themselves or they can simply quit. XTension will restart the plugin application after a short timeout and will continue to try to restart the plugin until it’s retry count is used up or until the connection is working again. This is especially useful for some devices which may not handle errors gracefully like USB devices. Due to limitations with some of their drivers it is simply not possible to gracefully recover from a USB bus hiccup or other error without quitting and restarting the host process. XTension will take care of that for you automatically if you are forced to deal with a device like that.


===Command Line Parameters===
XTension launches your script or binary plugin based on the data in the “info.json” file. XTension will run your plugin with 3 command line parameters. No command line argument parsing is necessary, these parameters and their locations will not change for APIv2 though some plugins may require extra parameters added in the future to the end of the list, the first 3 will always be available. Note that most systems will pass you the link or name of your executable as the first parameter, so these would start at the second actual argument.

^Index^Value^
|1| The DNS name or IP address of the XTension instance you should connect to for receiving your data. If your plugin is started normally by XTension this will always hold either 127.0.0.1 or just “localhost”. If you are running your plugin on a separate machine then this should contain the IP address of the XTension machine you wish to connect to. In that case you will have to provide your own mechanism to launch the process on the remote machine.|
|2| The Port that the XTension plugin server is listening on. This defaults to 52301 but can be changed by the user in the XTension Preferences if there is a conflict with something else running on the server. |
|3| The connection ID number. This number is sent as part of the initial handshake to XTension. This lets XTension match up the incoming connection from your plugin to the interface object that launched the process. In the case of normally started plugins this will be auto-assigned a number between 1 and 99. This number is dynamic and should not be saved but always taken from the command line arguments. It simply increments with each plugin instances that is started and then rolls over if we reach 99. For remote plugins the ID number can be specified ahead of time in the interface setup dialog and will not change. That will allow you to startup remote plugins with a known ID number that will always be connected to the proper endpoint in XTension. A remote ID number should be a pseudo random 4 digit number.|

===Initial Handshake===
Once your plugin is launched and has found the command line params listed above it should open a TCP socket to the passed address and port. Once the connection is accepted you must send a short handshake string that contains a hello message and the connection ID number. The format is:\\
\\
  plugin hello message:IDNumber\r
\\

The format of the plugin hello message is not important and is only used if there are problems in the connection. Otherwise XTension ignores this value. The string must contain only a single colon, and after the colon must be the ID number passed from the command line parameters followed by a carriage return.\\
\\
Once you’ve sent that your pipe to XTension is open and you may start to receive commands if the units assigned to your interface change state.

===XTension Commands===
XTension commands are sent via the pipe you just opened. All communications with XTension is through these command objects. They contain a header character that will always be “J” or “K” which controls the integer size that contains the size of the packet. You have to read the header of the command to properly parse the rest of the command. The rest of the command are key/value pairs. In the case of the command packet all keys are 4 byte strings. You can see them all in the python plugin include file “xtension_constants.py” included in the demo plugin. The value at any of those keys may be any length as each value as a length integer, either 2 or 4 bytes depending on the header,  In order to parse these packets from a stream it is necessary to read the header and then either 2 or 4 more bytes for the size depending on the header. Then read the correct number of bytes and pass them off to an XTensionCommand class for parsing and creating of an XTension Command object that you can use to get and set values as well as send commands back to XTension. A useful implementation of the command would be a wrapper around a dictionary class. Received values are placed into the dictionary and to send the packet values are read out from the dictionary. Specific keys may appear in any order and are different based on the specific needs of the command being sent. You should save but ignore keys present that you are not using but not log errors if more data than necessary is present. 


==J Header Packets==
For packets that start with a J header all lengths are 2 byte unsigned integers. The K header is only used if the size of the packet is too large to be represented with the 2 bytes. You should be prepared to parse either packet however as larger database dumps or requested images will require the larger size. 
^Size^Type^Function^
|1 byte| char | Header “J” in this case|
|2 bytes| unsigned int | the length of the packet including the header and packet size bytes|
|1 byte| flags | unused at the moment, originally a bit was set to indicate that the server expected it’s lengths in little or big endian format. This was because we were potentially talking between PPC and PowerPC processors. There are no more PowerPC processors capable of running the app so this can be ignored. All numbers will be standard CPU endian.|
|4 Bytes| char | A string that contains the 4 character constant name of the value. This string is not null terminated, just 4 bytes. On MacOS this is akin to the OSType but does not use any of those constants. See the xtension_constants.py file for the values you can expect to receive or will need to send.|
|2 bytes| unsigned int | The length of the data|
|variable| char | the string representing the data that should be assigned to the above key in the dictionary. This string is not null terminated. All data received from a command is in string format including other numbers. Only the length bytes are low level binary values. |
|...|...|repeat reading until EOF or you reach the length in the original packet length.|


==K Header Packets==
For packets that start with a K header all lengths are 4 byte unsigned integers. It is identical to the J header packet above except for the longer lengths. You should be prepared to parse both as the K header packets will be automatically sent by XTension if a packet is too large to use the smaller sizes. For sending commands to XTension either command can be used however it’s only necessary to implement the K header for sending. All commands can be sent that way regardless of size and should you have to send a large amount of data further processing or checking the size is not necessary.

^Size^Type^Function^
|1 byte| char | Header “J” in this case|
|4 bytes| unsigned int | the length of the packet including the header and packet size bytes|
|1 byte| flags | unused at the moment, originally a bit was set to indicate that the server expected it’s lengths in little or big endian format. This was because we were potentially talking between PPC and PowerPC processors. There are no more PowerPC processors capable of running the app so this can be ignored. All numbers will be standard CPU endian.|
|4 Bytes| char | A string that contains the 4 character constant name of the value. This string is not null terminated, just 4 bytes. On MacOS this is akin to the OSType but does not use any of those constants. See the xtension_constants.py file for the values you can expect to receive or will need to send.|
|4 bytes| unsigned int | The length of the data|
|variable| char | the string representing the data that should be assigned to the above key in the dictionary. This string is not null terminated. All data received from a command is in string format including other numbers. Only the length bytes are low level binary values. |
|...|...|repeat reading until EOF or you reach the length in the original packet length.|


===xtData objects===
Most non-command data in XTension is stored and sent to you as xtData objects. These are basically a flattening of a dictionary of key/value pairs similar to the command stream above however the key and the value can both be any length and additional dictionaries can be embedded as well requiring a recursive parsing of embedded dictionaries. This is similar to how a JSON object might work but the xtData object in XTension predates the easy availability of JSON parsing utilities in the system so this structure is still used. I do have some work done internally to present this data in JSON format and if implementing this protocol is too complicated in your chosen system then please let me know and I will move making that available up the to do list. 

xtData objects are sent flattened in a key in a command. For example if you send a command to XTension asking it to tell you about all the units that it currently has assigned to your interface they will be sent back in another Command object with a flattened xtData object stored in the key xtKeyData (as defined in the xtension_constants.py file) you can then extract that field from the command and pass it to a new xtData object to be parsed or however else you are going to parse the flattened data to your application. You should also be prepared to flatten your own xtData objects as some commands may require flattened xtData streams as part of the needed info.

Once you have requested that dump of all units assigned to your interface (or all units if you are requesting access to the entire database) you are effectively subscribed to changes to any of the internal values of those units. If anything changes in the unit a few ms seconds later an update will be generated and sent to your plugin. It will be the same layout as the initial database dump but will contain only the changed values. Your class should be able to receive these update packets and change only the included data in order to keep your plugins database of it’s units up to date. The xtData class in the xtension_plugin.py code also has a subscription method so that you can subscribe to specific keys in the database and receive a callback when they are changed in XTension. For example the demo dimmer plugin uses this to watch the status of the “dimmable” flag in the database. If a user changes a unit in XTension from dimmable to non-dimmable the database in the plugin process will be updated and the proper command will be sent to the hardware to conform the output channel to be the same. For some simpler plugins implementing this callback system may not be necessary but it is extremely handy for more full featured plugins.

Each xtData class will also contain a UUID. This is used when merging data so that embedded xtData classes can be found and the merging data routed to the proper embedded data class. The example xtData class contains a getContainerByUUID class that is used in the merge function to make sure the new data is applied to the proper container class. Embedded xtData classes are called containers in the original code and I may refer to them that way below. The UUID key will always be “_typeuuid_"

You can read the commented python code that parses and flattens these objects as well as handles the subscription to changes in the xtension_plugin.py demo code file.

==Data Representation==
All data in the xtData stream are strings. They can represent dates, colors, integer, floats or files. The type is stored as a separate entry in the dictionary with the kNamedTypePrefix appended to the key. For example if you save a string into the database with the name “myKey” and the value “myValue” it will generate 2 entries into the dictionary. The first will be the passed key and the passed value “myKey=myValue” the second will prepend the named type prefix onto the key and contain the type as a second 3 byte string. The second entry might look like “_typ_myKey=str” you can either ignore these type keys or use them to return the strings converted to the proper data types. When a value is set in a dictionary you should convert it to the proper string format and set a type entry to help the receiver know how to handle the value when it is received.

Data Type Constants:
^Data Type^Constant Value^info^
|Binary Data|bin|treated as a string, but may have unknown encoding or contain characters incompatible with normal string handling|
|Boolean|boo|will contain the string ‘True’ or ‘False’ note that in case sensitive languages they are capitalized|
|Color|col|a comma separated list of the decimal RGB values potentially also containing a 4th value for alpha “45,255,75”|
|Date|dte|Dates are represented as human readable strings that can be easily split and turned into local date objects in whatever system you are using. “month/day/year hour:minute:second” first split on the only space which will give you the date and time portions separate. The Date portion can be split into month/day/year by splitting on the back slash. The year will always be a 4 digit year. The time portion can be split into hour:minute:second by splitting on the colon. The hour is always a 24 hour number from 0 to 23 here. The format of this field is independent from any local machine time formatting or international time/date formatting.|
|Float|dou|A float or double precision number depending on your language. As a string this will be represented as a number followed by a decimal point and optionally any decimal values needed. The decimal point will always be present even if there is no decimal portion. Any val() type processor should be able to handle this but keep in mind that the decimal point used will always be a period even if the local number formatting options of your machine expect a comma as in Europe|
|File|fil|this will be the full path or shell path to the file being referenced. This is rarely used except for things like references to specific video recording folders or such. No file references are in any standard data structures that are routinely shared with a plugin.|
|Integer|int|the number as a string. It may also include a decimal at the end like the Float does but will never contain anything after it.|
|String|str|The string as set. Encoding should always be compatible with the UTF8 encoding system even though many strings passed are simple ascii. This is not a null terminated string.|
|Picture|pic|this is no longer used. If you’re building a video interface plugin or other interface that wishes to make image resources available the raw picture data should be sent as a keyed data entry in the Command separate from the description information which can be in an xtData object.|

==xtData Format==
The flattened xtData object requires recursive parsing in order to support the embedded xtData objects. See the xtData object in the xtension_plugin.py file for more info on how this might be done properly.

The initial header that identifies this packet as a flattened xtData object is 4 bytes “Xbdb” all other headers included in the stream are 1 byte long.

^Name^Size^Value^Usage^
|Packet Header| 4 bytes | Xbdb | The first 4 bytes of each flattened xtData objet will contain this string for validation. Also the first 4 bytes of an embedded xtData object will also be these 4 bytes.|
|Protocol Version | 1 byte | 0x45 | verify this byte in the packet so that you know you’re reading a version of the flattened object you know how to read.|
|Value Header| 1 byte | 0x56 | will precede a standard key=value data pair.|
|Object Header| 1 byte | 0x4F | will precede an embedded xtData object. At this point you should read the name of the object and then create a new xtData object and pass the stream off to it for reading. Then add the embedded object to whatever arrays you store them in inside your xtData class.|
|Picture Header| 1 byte | 0x46 | this is no longer used.|
|End Of Object| 1 byte | 0x45 | if you read this byte then you should stop reading from the stream and return. Your object is done. If it’s an embedded xtData object being loaded recursively then returning after reading this will return control of the stream to the parent object so it can continue to create more if necessary.|

==Parsing flattened xtData objects==
  * Pass the stream or string or buffer to a method in the xtData class.
  * Read 4 bytes and verify the Packet Header
  * Read 1 byte and verify the format version
  * Read 3 unused flag bytes
  * begin a loop that will read until we receive the End Of Object header.
    * read 1 byte
      * if you’ve reached EOF then return
      * if it’s an End Of Object header then return
      * if it’s a Value Header then:
        * read 4 bytes of key length
        * read the above number of bytes into the Key string
        * read 4 bytes of Value length
        * read the above number of bytes into the Value
        * store the key/value pair in a dictionary or associative array or something simlar
      * if it’s an Object Header then:
        * create a new xtData object and prepare it for parsing
        * read 4 bytes that are the length of the name of the embedded object
        * read the number of bytes above and assign to a local name variable in the new xtData object
        * append the new xtData class to an array of embedded objects or other method of storing a reference to them
        * pass the stream off to the new xtData class and let it parse itself until it reaches an End Of Object header at which point it will return to you and the you can read the next header byte off the stream where it left off.


===Getting Your Configuration===
Once you’re connected you can ask XTension for your configuration info. This will include all the port names or internet addresses or any other data that is setup in the interface setup dialog or your dynamic interface embedded in the same. Sending this command also subscribes you to any changes in the settings xtData object in XTension. So if any value is edited by the user the new values will be sent as an update packet which you should merge with the existing xtData object thereby generating subscriber events for anyone that needs to know if those values change.

^Key Constant^Value^ ^
|xtKeyCommand|xtCommandGetKeyedData| Required. All xtCommand objects must at least have a command key specified|
|xtKeyAddress|”all”|Required. You can save other keyed data object into the preferences if necessary, requesting the one with the constant string “all” (without the quotes) will send you the base data and all settings on the edit interface dialog window.|

XTension will then respond with another command that will have the xtData object with all that information flattened into the command under the xtKeyData key.

Response:
^Key Constant^Value^ ^
|xtKeyCommand|xtCommandSetKeyedData| a reply with an xtData object stored in the xtKeyData key. Use the xtKeyAddress key to find the name of the object.|
|xtKeyAddress|”all”| (without the quotes)|
|xtKeyData| (flattened xtData information)| pass to the parse method of a new xtData object|





If you have not yet received any base configuration then unflatten the xtData as sent and store it to a global or class variable. If you have already received this data and receive this command with this address again then it will contain only the changes that have been made to the settings by the user. Unflatten the changes into a new xtData object and merge it with the existing xtData class of your settings so that any changes can be sent to subscribers interested in those values.





===Controlling Units===
For more detail see the plugin examples. To tell XTension that you have received a value change for a unit you would create an xtData object, insert the following key/value pairs, flatten the command and send it up the pipe. Note that all values stored in the xtCommand object are strings.

You can send a command to a Unit in XTension in 1 of 2 ways. A plugin that has received access to the entire database should include the unitqueID key in the command with the number set to the ID string from the Unit they want to control. A normal plugin with access only to it’s own units should include both the device type “tag” as specified in the info.json file as well as the address key. The command will be routed to and handled by the proper unit in XTension either way.


^Key Constant^Value^ ^
|xtKeyCommand|xtCommandSetValue| Required. Every command must always include a command key|
|xtKeyAddress|(any string)| Required. The same string as set in the Unit’s address field in XTension|
|xtKeyTag|(any string)| Required. Each device type requires a tag that tells XTension which kind of unit the command is destined for. See the info.json docs for more info.|
|xtKeyValue|(string representation of the new numerical value)| Required for a set value command. XTension can hold float number values for Units so it’s fine to send 17.5 or just 0 or -400 or whatever makes sense for this unit|
|xtKeyUpdateOnly|(boolean)|optional. If you include this value and it’s set to True then the unit will only update itself if the new value is different from it’s current value. If this is absent or set to false then any value will cause the Unit to update, run it’s unit scripts and update it’s Last Activity date. This is useful for things like temperature sensors that might repeat their values fairly often without changing. By setting this flag to true you can send each reading to XTension but not have it spam the log with new values unless something changes. Even if the Unit Scripts do not run the last message received date (not the Last Activity date) will be updated. This can be used to display errors for sensors that stopped responding if you know you can expect a message from them fairly often. So it’s generally a good idea to send updates as often as they come in but with this flag set to true.|
Mac Home Automation

User Tools

Site Tools

Page Tools
