TL;DR : Outdoor Distributed Wireless from Cell Service is possible!

Summary

Every year, I goto the Grandfather Mountain Highland Games for a wonderous 10 day vacation in a remote-ish part of the NC Blueridge Mountains.  Because of the location, getting reliable internet service in and around the hills where the games are held and campers camp is a bit difficult.  There are a few local carriers that can reach various parts of the hill, but no service is 100% available everywhere.  So this year, I decided I was going to address that. I took some spare parts and devices I had lying around and created a working “mesh” network that extended “wifi” service around the campground.  I ran into some unique challenges and successes that will contribute to an improved service for next year.

Pros

• Wifi mesh networks work reasonably well to move data around an environment, even when multiple hops are involved
• Well configured routers and load-balanced hotspots make a reasonably fast network. \
• Signal strength to user-devices wasn’t as much of an issue as I thought, the distribution of AP’s covered the spaces adequately; however signal strength between AP’s was.
• Using weather-proofed Wifi devices and Power over Ethernet Cable (POE)  are easy to cable and give much leeway in where devices can be placed.

Cons

• The mesh networks take a while to “stabilize” when the power up.  EG.. Nodes that are further away from the gateway take the longest to take up as each node in line has to establish connectivity and configuration.
• The connected device count can significantly impact operation of the system.  End-point nodes can be so constrained as to not even get DHCP service and therefore no operational capability.
• Mesh network layout and configuration can also significantly impact the operation of the system as you lose 1/2 your bandwidth with every “hop” of an access point.
• There are *way* too many ways users and programs can find to “stream” data, and thereby it’s difficult to constrain network usage.
• When using Cell Service for bandwidth you’re constrained by cell tower signal strength, and the other users on the cell.

While, the effort wasn’t 100% successful, I learned a lot and will be able to apply my lessons to make it bigger and better for next year.

Concept

The initial idea was driven by the fact that I had almost no signal near where I camp on the mountain, however going up the hill about 200 yards, there was ample signal ( this has since changed.. as T-Mobile is extending services ) and I wanted to have simple internet in my camp.  So I started exploring ways to solve that.  Last year, I had an AT&T MiFi “hotspot” setup up the hill at the Pack-Rats camp store.  This allowed me to help them with credit-card sales ( as they didn’t have a carrier with reliable service ) and was going to get me signal in camp.  Alas .. the hotspot didn’t quite reach and I had iffy service at best.

I decided this year to improve the concept by extending Wifi down the hill to my camp using a 802.11ac “mesh” network where the access-points(APs) would talk to each other and relay data where needed.  I have been doing alot of work with the OpenWRT system as well as playing around with Open-Mesh 802.11ac routers and I had a few hotspot LTE based wireless access points and decided to put it all together and create a service that I could use and extend to others.

Design

To create the network I needed a smart gateway to route and manage data.  I do my own builds of OpenWRT ( an open-source Linux operating system ) .. and built a version of the latest OS with the features I wanted:

• Robust DNS/DHCP (dnsmasq)
• Multi-WAN load-balancing (mwan3)
• Squid Proxy to alleviate some non-SSL traffic
• Extra logging and statistics

To connect to the Internet, I have an AT&T branded Netgear AirCard 770S.  I have used this particular device a few times, and had come to rely on it for routing and decent service ( where AT&T will reach ). I also have T-Mobile on my personal devices and thought I might use my Apple iPad2 as a routable point and/or any other devices with reasonable amounts of bandwidth available. Initially, I had explored routing to the LTE Wifi via the USB port, however this proved very inconsistent between the implementations of the devices and decided to instead use the native wireless capabilities of the router.

I grabbed a spare TP-Link Archer C7 1750 router and applied the configuration and added a few features of my own:

• Setup the LEDs to let me know when connectivity was established with the different ‘wan’ interfaces configured.
• Setup squid to work as a transparent proxy on non-SSL web traffic.
• Setup logging to a plugged in MicroSD card on the USB port.

So my initial configuration looked like this:

• iPad setup in access-point mode
• AT&T Hotspot on LTE
• TP-Link as gateway
• Wired connection to OpenMesh router as the mesh gateway

I had hoped the iPad would do 802.11a ( 5GHz ) to allow me to leverage both radios in the router, but it didn’t. I learned that both the iPad and Hotspot needed to transmit on the same channel to allow for the TP-Link’s 802.11b/n (2.6GHz) radio to see them concurrently ( and there’s no way to set the channel on an iPad.. grrrr ).

I tested this at home and it worked reasonably well.  I could see packets moving, almost cable-like speeds and the proxy was effective for non-SSL sites.  I expanded the configuration with a few more “mesh” AP’s I had around .. and tested using my laptop to beat on the network from various locations around my house and neighborhood.  This again yielded decent results.

Deployment Plan

Hardware

I already had 3 of the OpenMesh AP’s so I grabbed a few more and water proof cases.  To power them I decided to leverage CAT-5 vs trying to use standard power supplies.  The AP’s have several different power requirements ( 18-24v and 48v ) but I found single plug “bricks” that allowed me to just plugin to the outlet and run cat-5 from there.

Networks

So I wanted to allow camping users to also share this hard work, and also still wanted to be able to support PackRats, so I created a few separate wireless SSID networks (in order of bandwidth availability:

• Matheson – my network for monitoring and troubleshooting. (And I wanted to say that Clan Matheson was all over the Mountain 😉 ) This network required a WPA2 password to access and was setup directly off the native LAN ( 172.16.0.0/16 ) and had unrestricted bandwidth.
• Vendor – this was setup to support any vendors ( we only had 1 ) and required a voucher code to connect to. This was also setup to the native LAN, but LAN access was disabled, only internet bound routing allowed.  This too had unrestricted bandwidth per connected device.
• Camping – this was for the local people to connect to to be able to get general access.  This was setup on it’s own VLAN (100) and designed for 1000+ devices (172.18.0.0/22) mainly to alleviate any DHCP spamming.  This was initially restricted to 10Mb/s .. and reduced to 5 Mb/s to reduce congestion.
• Open – this was for anyone not camping, and was dampened down with very little bandwidth.  This was also setup for 1000+ devices ( 172.18.64.0/22 )  (the numbers of users on this network later proved untenable and the network was disabled)

Software

I didn’t want devices to just connect, I wanted controlled access. Luckily Open-Mesh had the Cloudtrax software integrated, so I was able to setup a “splash” page which required users to accept usage terms and conditions as well as a PayPal button for donations for bandwidth.  For the vendor network, Cloudtrax supports ‘vouchers’ which allowed me to control, track and assign each vendor a single code for a controlled number of devices and bandwidth.

I considered using a ‘walled garden’ approach for these configurations, but I still needed them to approve access before getting access so that tool wasn’t as useful as I’d have liked. I setup dnsmasq to be the DHCP/DNS servers to  control some access to the network via DNS blackhole .. however also proved untenable as many users just used external DNS, and dnsmasq wasn’t as responsive as it needed to be under heavy loads. I also setup some tools to detect and kill streaming.

Deployment

So after arriving and setting up camp, I grabbed the router, hotspots, wire ties and some AP’s and started putting them up.  I decided initially to put the base station at Packrats up near the road as they had someone there to watch it and the signal strength up there was good.  As soon as I booted everything, configured the iPad and hotspot to both use the same channel, and gave it a little time for the first AP to check in.. I had internet! Fantastic.

I used an android tool called Wifi-Analyzer to track signal strengths and channel usage and started walking down the roads and finding power poles with an outlet w/in 15 feet ( that was the CAT-5 Cable length I was working with) and put them up.  My network steadily grew to 4 AP’s.  I started also noting users were finding it before I even announced, 35 were connected w/in 1 hour.

Growth

So my little network that could .. started working .. with the low numbers of users I didn’t have any real problems, and bandwidth was reasonable given the circumstances.  As the next few days moved ahead, and we got more and more users on the campground, things changed:

• By July 3 I had 112 concurrent devices and using up to 2.8 Mb/s .. at that rate I burned through the data I had, but I also had a few donations come in, so I bought some more data on the AP and extended my T-Mobile account to handle things. I also noted the numbers of devices per AP was starting to approach their capabilities so I ordered a few more AP’s.. they would arrive Wednesday July 6.
• By July 4 I realized that the T-Mobile didn’t count data for BingeOn services  AND was bw-limited by T-Mobile, so I reconfigured the firewall/DNS settings to force supported streams that way.  I saw an improvement in my data-usage as tools and services started using Youtube via BingeON vs the other services.
• I had one major outage on July 4 as data ran out on both AP’s concurrently in the middle of the night.  I was able to resolve in the morning and get things back up reasonably quickly
• On July 6, the new AP’s came in .. and specifically got the biggest baddest one they had the MC1750 which could handle way more users and was more powerful.  I added and extended the network, keeping the campground the strongest group. Once the mesh reconfigured, I found the AP’s waayyy up on the field could see it directly.  This was great as it gave us way more capability.
• I had the second major outage on July 8 during a severe thunderstorm that had 87 mph winds and caused some carnage.  Most notably, the signal patterns for LTE changed around camp, and I couldn’t get reliable service inside PackRats anymore; however AT&T was usable .. so I added more data on that service and left it up and went to cleanup my camp.
• On July 9, after discovering that LTE was now stronger in my camper than up the hill ( bent antennas perhaps ) .. I reconfigured the network to have the gateway in my camper and left the other AP’s to re-mesh ( This took about 1.5 hours to settle )
• I removed the field AP’s on Sunday at 4 when the games officially closed, and removed the camping ones Monday morning.

Review

So I gathered lots of data to share:

Usage

• The system serviced a total of 1135 devices, with 259 active at one point in time.
• Users used 68.4 Gb of data, which translated into about $250 in data costs with LTE service.
• Usage patterns matched the weather, if it was raining I saw way more internet usage 😉
• Usage patterns also matched the locations of the access points,  the coverage of camping had significantly more users.

Top Applications

• SSL was the #1 usage, which is a good thing for users as your communication was protected, but bad for me as I had way to meter the usage.  This likely covered Mail and some internet usage.
• HTTP was the next, which means the proxy was likely worth setting up.
• Streaming was more popular than Facebook

Top Devices

• Apple is still cooler than Microsoft it seems … tho in the phone world it’s close.
• Android devices use more data than iOS.  That may be due to being cheaper, and more “heavy” users use them to do streamy stuff ( looking at the younger generation .. but no data to support it )

Heavy Users

• Some users really use the network hard no matter what you do to mitigate it.
• I played some whack-a-mole for a while trying to restrict heavy usage, but as user counts went up it became a fools errand.

Access Points

• The distribution of the AP’s was based on available power and location to attach.
• As predicted, the higher an AP was placed, the better it operated.
• More powerful, higher-density AP’s worked significantly better and lowered the number of hops between devices.

Lessons Learned

So many good things I learned:

1. There is a need and desire for Internet accessibility at the event. There were a total of 1135 devices that attached to the service, and even with the degradation we had due to the too-many-device disconnects and slow bandwidth.. users still were very active until the last moment ( there were 126 active devices on Sunday at 3pm, and 58 Monday morning when I had to shutdown. )  Usage patterns indicate people were using social media services such as Facebook, Instagram, Twitter and Youtube and using it consistently.  I also heard from many that were outside the range of the access points (McRowdy) that they would really like to have service.
2. Outside influences make large impacts. After the storm, when the service patterns on the mountain changed caused accessibility issues.  Vendors whom accepted credit cards mostly used their own service; however after the damage to the cell tower, many were unable to continue to do that as they couldn’t get signal.  Because i had the flexibility to move my LTE reception point, I was able to re-establish service quickly and effectively. The other thing i noticed is the saturation of the AT&T LTE cell tower impacted my services directly.. e.g. when 10k people were at the event with their phones banging on the tower. The T-Mobile side worked better, perhaps because it’s newer or it’s less prevalent in that area of the state and so fewer users.
3. LTE service can work in this environment, with caveats. Having 2 or more hotspots really made a big difference in throughput, especially proxying the non-SSL traffic.  Having a higher bandwidth gateway will make a big difference, and deploying multiple gateways would be even better. The meshed AP’s support having multiple gateways at different points, however has it’s own challenge with NAT/Routing. Something to debate.
4. Internet Usage is hard to control and more prevalent than expected. More and more sites are going to SSL and providing higher and higher bandwidth services. This translates into more usage for carriers ( which makes them happy for charging and also presents the challenges for supporting it ) so finding ways to not have to restrict access would be advantageous as it’s an uphill battle.
5. WIFI Access points need to handle higher densities to be effective in  this environment. The numbers of devices that attached far exceeded my expectations and even base usage by persistent apps on devices can saturate the capabilities of meshed AP’s when more than 1 or 2 hops from the gateway.
6. Funding a donation service is possible. I did this year on donations and didn’t quite have my data expenses met, however many said they would happily donate to have access.  There were a few impediments:
   • Many people don’t like PayPal and won’t use it
   • Many people would rather give cash ( or Scotch! )
   • The Donate section was missed ( though it was above the Free Access [ Continue ] button )
7. Campers block signal. I had several individuals that noted they could not get signal in their campers, including one particular one where the access point was less than 10 feet from the camper; but could get LTE off a hotspot.  I believe the problem is related to the radio frequency as WIFI is generally 2.6 GHz and LTE is generally lower; which means it can penetrate walls much better.

Next Time

Some things I am already planning for the next iteration of the Camping Wireless

• More access points! – My objective will be to get many more of the MC1750’s deployed in strategic locations so they can mesh more effectively.  I have already talked to OpenMesh and they are working on some bigger and better devices with higher density. I also hope to cover more of the area.. including McRowdy and Happy Valley.
• Fiber Connectivity – I have a vendor that was willing to donate 100Mb/s to the field from the maintenance building for Grandfather Mountain, however we weren’t able to make it happen this year.  I will pursue this for next year and get it linked up to the field.  I will continue to have the hotspot configuration available as fail-back in case fiber goes down.
• Larger Infrastructure –  I plan to deploy multiple routers in a fail-over configuration and move the proxy to higher speed hardware with redundancy.  I will have UPS setup for the power for the devices ( so 87 mph winds can’t knock us down ). I also hope to setup some point-to-point wireless to extend w/o having to mesh to improve the device density per access-point per hop.
• Assistance – I’ll be looking for a few volunteers with tech experience to assist me with getting things setup and monitoring the system.  I did this mostly on my own this year and loved it .. but I can see this becoming bigger.  I also hope to propose this as a service that GMHG can use as well to improve services to the Games in general.