Mac Home Automation

So many devices and interfaces now are ethernet or WiFi enabled that it makes sense to do a tutorial on networking for home automation. Just to discuss and define the terms and the things you’ll need to know about your network and router to make sure you can keep everything connected. While it’s not that complicated overall there are several common ways that these setups can go wrong and cause problems.

TCP/IP is the system that makes the internet work. It’s how the packet of data I send to something finds that machine anywhere in the world and how the response finds it’s way back to me. We’ve pretty much run out of IP version 4 addresses, and so the internet is moving to IP version 6 with much more address space and other nice features, but that doesn’t matter much for connecting to devices on your local network, so for this document we’re going to stick to discussing IPv4 which is supported by every router and every switch and machine out there.

Any machine needs to know 4 pieces of information to be on the network:

• The IP address
• The Netmask
• The Router address
• at least one DNS server address

We will discuss them in detail further down.

Your connection to the internet is likely through a cable modem, a DSL modem or some other kind of modem. (If you’re lucky enough to have a fiber connection don’t even talk to me because I hate you) That modem will get an IPv4 address from your ISP as part of it’s power up process. That is your external internet address and you only ever get one of them at a time. Your ISP will change it periodically on you too so you can’t rely on it to stay the same unless you pay extra, but luckily there are several ways around that problem. You normally don’t see this address at all on any of the machines inside your network. Your modem will also be running a “DHCP” or Dynamic Host Configuration Protocol server. This is what every machine on the network asks for an IP address and how each machine manages to get a valid setup when you have it just set to “DHCP” mode. This is good in that it keeps you from having to enter anything at all, but you need to know something about the setup of that server in order to configure backwards connections for a web interface.

You only have one internet address, but you can connect any number of machines on the inside network of that modem because it shares that single IP address to all the machines on the inside of your network. This is true if you have wifi or ethernet network or some combination of both. This ability to share the connection is called NAT or “Network Address Translation”. To the internet your whole house looks like just one machine, your router/modem and it figures out where the incoming packets should be sent within your network.

This is the default setup of every router, and where things can begin to go horribly wrong when you want to set up an incoming connection for a web interface or a JSON request or something like that. While your router has to have an Internet visible and routable IP address, your internal machines will be given an address automatically by the modem that is within one or 2 special ranges for these kind of shared networks. The two most commonly used internal network shared address ranges are 192.168.0.??? and 10.0.1.??? you may see other numbers in the 3rd place of that 10.0.???.??? address too and thats OK.

The first problem that you may run into setting up a properly functional home network is when you have more than one device that is running a DHCP server and a NAT internet sharing connection.

The default setting for any modem is going to be to share the connection and run DHCP and NAT. If you plug an Airport or other wifi router into that modem the default setup for it is to run ANOTHER DHCP server and re-share the already shared IP address to your machines via yet another NAT layer. The modem shares it’s one IP address via NAT, the AirPort will reshare that again and create a second level of NAT. This actually works, for the most part, as long as all your traffic is only outgoing, web surfing and email or things like that. As soon as you need to connect to something that wants to make an incoming connection like any VOIP telephone system or back to my mac, or video chatting then you’re going to start to have problems with this. It’s impossible to properly setup an incoming connection for a web interface through a double NAT setup.

The cable or DSL companies would like you to use their modems which often have wifi built in now and nothing else. I personally did not care for the new Comcast modem with the “xfinity” wifi guest network that I couldn’t turn off. Their wifi strength was horrible and I don’t fully trust that the setup on the modem will stick through their own resets, nor do I necessarily want them able to change my NAT configuration. So in my case it was necessary to set up their modem into “Bridge” mode and use my original Airport as both my wifi router and my NAT/Sharing router. (NOTE: if you’ve got one of the new xfinity modems you can’t actually put it into bridge mode yourself, you must call comcast support and have them do it remotely. They will happily do this for you, but it may not stick through their resets so you may sometimes lose your connection and have to call them back. I would recommend purchasing your own modem rather than renting one of theirs so that it’s yours and you control it completely.)

While it’s certainly possible to make ludicrously complicated network topologies with many routers and NAT settings and set things up to work, you don’t want to do this (unless you don’t actually need to read this article).

As a strict rule, that you violate at risk to your sanity, your network should only ever have one device running DHCP/NAT and sharing your connection. You may have multiple airports or other wifi routers, wifi extenders, ethernet routers and multiple switches and hubs, but any other device downstream of the one providing DHCP/NAT sharing MUST BE SET TO BRIDGE MODE.

Bridge mode means that the device is simply providing a bridge between the 2 kinds of connections. It is not going to do ANY processing on it’s own, it’s not going to run a DHCP server or do any NAT sharing or any routing or any thinking about the traffic at all. It’s just an adaptor between ethernet and wifi, or cable and ethernet, or DSL and ethernet.

In my application the modem is set to bridge mode. It is providing only an ethernet to cable bridge connection. Instead of it getting my internet routable address, it’s my Airport that gets the internet routable address and handles the DHCP and NAT sharing. The Airport then provides services for all the shared addresses on my network both wired and wifi.

Any wifi extender or second wifi access point you setup must be set to bridge mode also. (yes, technically if you setup a wifi extender it will already be in bridge mode, but many can also be separate access points and many support other strange services that might break things, more discussion on extending a wifi network further down) Remember the default setup of any wifi access point is to take an IP addres via it’s “WAN” (wide area network) and reshare that with DHCP and NAT over the wifi and if it has more ethernet ports the “LAN” (Local Area Network) ports. You can add as many Airports as you want (until you use up all the wifi channels) to your wired network, but if you want the devices that connect to them to be able to see each other then they must be set up as bridges, and not running DHCP/NAT sharing. For example, I have a wired printer on the ethernet network. If I put another Airport on my ethernet in sharing mode then I would not be able to see that printer from my wifi devices. This is a very common state of network headache. You fix it by making sure there is only one wifi router in sharing mode, and that all others are connected to one of it’s LAN ports and set to Bridge Mode.

Bridge mode make sure that whats on both sides of the device are really on the same local network. If the device is sharing then the network it creates is totally separate. It may still be able to route traffic to the internet, but it will not be able to route traffic to anything that is on a different shared network.

There are 2 ways to extend a wifi network. Which one you use depends on the possibility of getting an ethernet cable to the second location.

Wifi Extenders do not require a cable back to the network which makes them invaluable in situations where you can’t get one. They are available inexpensively from many companies and I hate them all. Since they have to receive each packet and then rebroadcast it on the same network they basically cut the speed of any device talking through them in half. While it’s possible that there are ones out there that are well thought out and work well I have never had the pleasure of owning one that was. They have all been difficult to configure, painful to keep running, they randomly restart and sometimes forget your network settings after a power failure. I do not recommend using them at all. If you are stuck with this as your only option then make sure that your Airport or other wifi router has the checkbox somewhere in it’s configuration that says “allow this network to be extended” turned on. The latest Airports I believe have this as the default and may not even provide a checkbox to turn it off.

Adding more access points is my own preferred way to extend a network but this only works if you can get an ethernet cable to the location of the new access point. Such is my faith in this as the proper way to do things that I actually crawled through my crawlspace with the cable in order to get it to the new access point.

There are some very inexpensive wifi extenders that can also operate as an access point, or in “AP” mode. These are fine. They do not all need to be the same make and model or all Airport Expresses or anything like that.

The ethernet cable you ran must be connected to the LAN ports of whatever device is doing your DHCP/NAT sharing.

Set the new access point to “create a wifi network” let it pick its channel automatically as it will need to be different than the main wifi network.

Set the new access point with the exact same wifi network name, password and password type as the main network. This is the trick that allows it to extend the network rather than just create a different one that you’d have to switch between. As long as the network has the same name and password any device will happily switch between them as one becomes stronger than the other. Treating it as the same network. You’ll never need to select the different one or do anything differently on your devices, they will just work as you move around.

Lastly as already discussed make sure the new access point is set to BRIDGE MODE.

You can setup as many access points as you need for as wide a network as you wish to create this way. As long as all are set to bridge mode all your devices will be on the same local network and therefore able to see each other and share resources like printers.

A DHCP server hands out network configuration information including an IP address for a machine on the network to use. It’s because of this that you can just plug the ethernet cable into your computer or log onto the wifi network and have things work without having to enter a bunch of stuff in the TCP/IP setup control panel each time you want to connect to a different network.

For just using the internet your machines or devices IP address doesn’t matter. As long as it was given to you by the DHCP server and has the proper other settings it will work for you to be on the internet and to access other machines or printers on your local network.

The problem comes when you want to reach the Web Remote on the XTension machine, or when XTension wants to connect to your Vera, or when XTension wants to connect to the Phillips Hue hub on the network to control a lamp. In those cases it does matter what the address is and we have to know them ahead of time. For the machines that will need to be reached by other machines it’s necessary that they have a static IP address. A DHCP Address WILL eventually change. It won’t work to simply look at the network config and see that oh, it’s 192.168.0.103 and enter that. That is from a pool of addresses that will change and get passed around, you cannot use this for a reliable connection.

As I’ve already said, only 1 DHCP server may be running on the network at a time. There are many devices that CAN run one if you want them to, but only 1 can be on at a time. Choose the device that you’re most comfortable working with. For me that is an Airport device but the settings will be similar for any DHCP server out there. Whatever wifi router or cable modem that you’ve decided to use as your router/sharing device it will have a tab or a page for configuring the DHCP server.

Here’s an example of the DHCP setup window from an Airport. You get to this by clicking on the “Network” tab of the setup window and then clicking the “Network Options” button in that window. Other devices will have different looking windows and may have different names for things, but all these settings will be there somewhere.

This setting isn’t really that critical on the average home network. This is how long an IP address reservation will last. If a machine connects to the server and gets an IP address it also receives this timeout. If it uses an IP address for longer than this it will need to request a new address to use as this one may then be recycled by the server. Depending on the server it may allow them to continually refresh the timeout and keep the same address or it may send them a new one every day. This is also the amount of time that a machines address will stay reserved while offline or sleeping. So if you have hundreds of people coming onto your network for a few minutes a day then you need to set this to a smaller number than the average home user.

An IPv4 address has 2 parts. The “NetMask” setting of the TCP/IP control panel preferences tells the system how much of the address is used for the local network, and how much is used for remote routing to the internet. The NetMask tells you how many addresses you actually have available on your local network. It is possible to set this so that you could have thousands of machines on your local network, but for the scope of this article we’re going to assume that your netmask is set to “255.255.255.0” which is the default for the 192.168.0.??? network that is the most standard for DHCP servers.

What this means is that the 255.255.255 part will always be the same for all machines on your local network. Any address starting with 192.168.0 will be considered to be local. Any address that has a different number in those first 3 numbers will be considered to be out on the internet somewhere and will be routed to the internet. So on this network all our addresses will begin with 192.168.0.??? and we can use the numbers from 192.158.0.1 to 192.168.0.255 as valid local IP addresses. This local network may have 255 machines on it.

If you’re using the 10.0.???.??? addresses then I’ve seen some routers that treat this as having a netmask of 255.255.0.0 which means that anything that starts with 10.0 is on the local network. In this case you have 2 numbers to work with, the valid local addresses are 10.0.0.1 to 10.0.255.255 so on this network you have 65,025 potential addresses for lots more machines. This isn’t an important difference as most DHCP servers on routers or Airports will not handle that anyway. No matter if you’re using 192.168 or 10.0 the rest of this information will be the same.

This is the important setting for us in figuring out a valid IP range in which to assign our static IP’s.

You can see in mine that the range is from 192.168.0.100 to 200

That means that the DHCP server will give out addresses from that pool of 100 addresses from 192.168.0.100 to 192.168.0.200. That means that any other valid address on the network is safe to use for a static IP.

With the exception of 192.168.0.1 and possibly 192.168.0.255 which are special addresses. The router address is almost always .1 so don’t use 192.168.0.1.

Pick a range of numbers to use for your static IP devices and start a document in text edit on the server to write them down do you don’t forget what is where. In my case I started allocating my static IP addresses at 192.168.0.50 and just count up from there as I add more devices. If I ever get to 60 devices on my network I can start counting down from there or I can begin again at 192.168.0.201.

The default of every DHCP server I’ve seen in a home device is about 100 addresses leaving the rest free. It’s possible though that you will encounter one with a setup that reserves the entire valid range for the DHCP server. This is not necessary and you should change that config to a similar range to the above to free up some addresses for static IPs. If there are devices already using addresses outside the new range you may need to force them to get a new IP from the server either by power cycling them or pressing the “renew DHCP Lease” button in their network configuration control panel.

In order to have things like the web remote or other services from the internet talking to XTension work, you must also click the checkbox for “Enable NAT Port Mapping Protocol” or its similar setting on a non-apple device.

There is another way that you can reserve a static IP for a device just called “DHCP Reservations” on Apple machines. When a machine requests the local network configuration from the DHCP server it also sends 2 bits of identifying information. The first is a short string of text called the “DHCP Client ID” you’ll see this on your computer's network setup panel. The second is the low level “MAC” address (not Apple Mac, but Media Access Control address) this is unique for all network interfaces on the local network and it hard coded into the network interface. You can reserve an address within the range of your DHCP server by using one of these. However, if you reset your Airport or you lose your configuration for other reasons you’ll also lose these settings. It has the advantage of storing the numbers reserved for you along with a label as to what it is and the IP address so you can just go to your router setup pages to see what is reserved, you don’t risk losing the file you created where you are keeping track of things. And you can change them if you needed to for some reason. In general though I would recommend doing it the simple way. Just keep track yourself in a document stored on your server of what IP’s outside the DHCP range you have given to what.

Once your network is properly extended and setup and your XTEnsion machine has a static IP address you can configure your wifi/modem/router to send traffic through to the XTension machine for specific services like the Web Remote.

All internet traffic has as it’s target the address of the machine it’s bound for, but it also has a “port” number which tells it to what program or server on that machine it should be sent to. Without that each machine will only be able to listen for one kind of thing. This is called the “port” and is a number between 1 and 65,000. When you go to a web page that is port 80. Thats whats known as a “well known port” the first thousand or so are reserved for specific server usage and you can’t run Web Remote on them anyway as MacOS reserves the first 1024 ports for server processes running as root only. So for your Web Remote you’ll have to remember your address as well as a port number.

You can setup as many Web Remote and Mobile Web Remote instances as you want different interfaces, but each one must have a separate port number and a separate entry in the NAT Passthrough section of the routers configuration. The ports must be higher than 1024 and less than 65,000. In general it doesn’t matter what else you set them to after that unless you are running other things on your server that are listening on other ports. XTension itself uses port 52301 to talk to it’s various components so you shouldn’t use that port. For a secondary web server port 8080 is traditional and easy to remember. If you have multiple interfaces you can count up from there. My main interface might be 8080 and I might have one specifically for my wife to check the perimeter status at 8081 and the mobile one at 8082 and so forth.

There is a chance that you’ll hit something else when you go to set these up, but if so the Web Remote will fail to launch after you set the port in it’s configuration in XTension and you’ll get a message in the log telling you that it was unable to bind to that port meaning that port is already in use. If you don’t get that message then it’s likely OK to use that number for your interface.

All listening services in XTension, the Web Remotes and the JSON interface, have a configuration option where you can enter the port number. Set this to the port you’ve chosen above for the Web Remote or other service that you are configuring and enable the interface. If it starts without errors then you have reserved that port number for that interface. If you wish to create more they must have a unique port number.

On an Airport configuration the NAT passthrough is just called “Port Settings” and is on the “Network” panel of the setup window. The same place we looked at above for DHCP range info. Click the “+” button to add a new one:

The description is also a popup and if you’re passing through file sharing or VNC to your XTension machine you can just select it from the popup and all the values will be setup for you. If you’re creating your own as for the Web Remote you’ll need to enter your own description, in the example above I’ve called it “Web Remote Passthrough 8080” to remind me which one of many I might passthrough.

There are “public” and “private” settings for each of the entries. The Public ones are what it will expose to the outside world, the private ones are what it will re-route them to on the inside.

The first selection is for Public UDP ports. This is a lower level type of networking and XTension doesn’t use it at all. Leave the UDP fields blank. (UDP does not have the error checking and guaranteed delivery of the higher level TCP packets, it also doesn’t promise that your data will arrive in order. However it can be faster to send data where absolute integrity doesn’t matter so much, stuff like video chat where dropping a packet or 2 now and again is less annoying than having the stream stop while TCP retries the transmission till it gets through)

In the Example we decided to use port 8080 for our Web Remote. We are going to make both the Public TCP Port and the Private TCP Port the same value. You can enter different values into these fields and the router will redirect to the other port. This is very useful for sharing something like your VNC connection from multiple computers to the internet. VNC you can’t easily make run on any other port, so each computer inside your network will be running it on the same port. If you couldn’t redirect the ports you would have to choose only one machine to remote control. Since you can change the ports you can redirect port 5900 to the server, 5901 to another machine but with the internet port set to 5900 and so forth. Connecting your client to the higher port will access a different machine inside the network.

The Private IP address is the static address we have assigned to the XTension server. You can see here again that if the DHCP server gave the server a different IP address, then you’d not be able to reach it from the internet randomly after it updated its reservation and changed its IP. Some routers do have more complex settings where you can set these passthroughs to use the DHCP ID or MAC address but I do not recommend using this as it just adds complexity and another layer for things not to be updated and not to be showing you where things are really going.

Your modem, be it in Bridge mode or Sharing mode, will get an IP address from your provider. This will be a DHCP address just like we have discussed above with all the problems associated with having an address that changes. Some cable providers don’t change IP’s but once a year, some might do it every 2 days and every time they have a network hiccup or you take a power hit. You can’t just look at your external IP address in your router setup pages and make all your links to connect to that, it’s going to change.

You can’t use a regular DNS service either as most of the main internet services that offer such services make their minimum timeout 48 hours. It would take 2 days after your IP address changed in order for the new IP address to be propagated across the internet for you to connect again.

There are 2 (at least 2 that I have used) excellent services to provide “Dynamic DNS” services. These companies give you a piece of client software that you run on your XTension server. Every few hours it connects to their server and checks what IP address the connection came from. This will be your external address as visible to the internet. If it’s different than the last time it checked in the record in their DNS database will be updated. So you can continue to keep your link to “JamesHouseRules.no-ip.org” or “myAutomatrix.dyndns.org” and that name will always point to your home.

The 2 services I have used that have good Mac support are no-ip.com and dyndns.org they both have excellent Mac client software and have proven very reliable in my use. No-ip still has a free service I believe with the limitation that you have to manually update your IP at least once a month or it will be removed from the database. They just aren’t very expensive and it’s well worth the few dollars to have a reliable link pointing to your web remote interfaces.

Long ethernet cords are unusually expensive for what they are. They are hard to pull through holes in the wall with the jack already attached. You can easily make your own ethernet cords for long runs through the house to those extra access points you setup to extend your network and buying a box of bulk ethernet cable, some cable ends and the crimper can be cheaper than buying several long ethernet cords and the connectors to put them together. It’s not difficult to do, but there is a trick to it.

If you use your continuity tester on the pins of an ethernet jack you’ll find that they are just straight through. There is no reversing of them top to bottom or any null modem confusion. They just go straight through. Pin 1 to pin 1, pin 2 to pin 2 and so forth.

This isn’t the whole story though, it actually matters which pairs of wire are twisted together! For a short cord it won’t be obvious that there is any problem, but if you pull a long one through the attic or basement and add the pairs in order 1&2, 3&4, 5&6, 7&8 it won’t work! The reason is historical. Initially people wanted ethernet to live alongside with the phone system. If you plug a phone jack into the middle of an ethernet jack you get connected to pins 3, 4, 5 and 6. So Ethernet originally could live on the first and last pair in the jack, leaving the middle 2 for telephone.

So the pairs must be twisted together and wired in this order:

1 & 2, 3 & 6, 4 & 5, 7 & 8

Here is a screenshot of the wikipedia article about cat-5 and 6 wiring with the colors and pairs shown.

as you can see the blue pair is the middle, the orange pair has one leg on either side of the middle blue pair. And then the 2 outside pairs are just in order. There may be a standard for which color needs to go where, or if the white/color version goes above or below but I don’t believe that matters very much. As long as you’re pairs are twisted together in this order it doesn’t matter if you put the blue one first, only that each connector be done the same way on your cable. That being said, if you do them differently on both ends then it won’t work so why not use the officially documented version.

An ethernet jack is called an “RJ45” jack. There are 2 kinds regularly available that you can use to make connections.

The cheapest and easiest to get are standard crimp connectors. This is what I generally use.

The second type of jack is called a pull through ethernet jack. This almost identical to the first type except that you can pull the wires all the way through the front of the jack to verify that you’ve got them in the right order. You can also pull the slack out of the wire getting the twisted portion as close to the connectors as possible. It’s possible byt very difficult not to have to untwist a few inches in order to properly get it into the first type. There is a limit to how much you can untwist before it starts to cause a reduction in speed on the network or makes it susceptible to other interference. Once you’ve got it pulled through you use the same crimp tool to push in the connectors and then snip the extra wire off flush with the end.

There are several kinds of ethernet cable you can push into these connectors. First it comes in “plenum rated” or not. This is an indication of the kind of plastic that it’s made of. If the wire is to be pulled through “plenums” or sneaked through your cold air return down to the basement or something it should be plenum rated. This means that if there is a fire the plastic it’s made from when burnt releases fewer toxic fumes than the non-rated kind.

You can buy solid or stranded. I tend to use solid wire but thats just because it’s easier to punch down into blocks and seems stronger. It will not bend as many times before breaking however. Each jack type is rated for either solid or stranded. Generally any jack will work, but there are different forms of the inside of the pins where the crimp tool pushes them down into the wire if they are for stranded or for solid. Get the appropriate one for the best connection, but in a pinch use what you’ve got.

Cat-3 wire is only 4 pairs and is suitable for alarm system wiring or telephone use only.

Cat-5 wire will still work for 10bt and probably 100bt networks if the cable is short. This is less expensive but has been replaced pretty much everywhere with cat-5e.

Cat-5e (enhanced) is the choice for 100bt networking.

Cat-6 is necessary for the 1000bt (gigabit) network wiring. It is more expensive and has a plastic separator in the middle that keeps the wires properly aligned for less crosstalk. It’s usually a bit more expensive though.

For wiring gigabit ethernet there is no shortcut. If you want it to work properly you must use the proper pairs as talked about above and you need to get the twisted portion of the as close to the jack as possible. If it sticks out without all the twisting for an inch or so don’t worry, it will still work except in the most difficult of noisy environments. Gigabit ethernet uses all 4 pairs in the cable.

100bt and 10bt ethernet which are used by most non-computer devices like wired cameras and other similar devices are more forgiving. Each needs only 2 pairs. It’s actually possible to have 2 separate 100bt ethernet connections through the same cat5e cable. Since the WAN port on my airport and the output from my cable modem are only 100bt (technically the modem is a gigabit port, but the upstream bandwidth wouldn’t even begin to tax a 100bt network so I don’t care) I have a single cable that runs from my modem down to the airport in another part of the house where it’s connect to the WAN port, and then a plug from the LAN side runs up the other 2 pairs back to the data closet where it plugs into the main switch. That way I can put my wireless router where it makes the most sense, but still have a good connection to the modem and everything else in the data close over only 1 wire.

Depending on the device you may be able to do a poor mans Power Over Ethernet. If the current draw of the camera or device isn’t that great 100bt ethernet leaves 2 pairs free in the cable. To power a camera use 2 pairs for ethernet. (the middle pair on the plug and the pair directly outside it) and connect the 2 remaining pairs together so that you have 2 current carrying wires of double the amount of copper. So if you used the blue/white and orange/white pairs for ethernet, that would let you twist green and green/white together for the ground, and brown/brown white for the positive. I’ve done this for many devices with great success. Though it depends on their tolerance for noise on the network and how noisy their power supplies are and how much current they draw and how tolerant they are of voltage drops.