Wireless at GMHG

TL;DR : Outdoor Distributed Wireless from Cell Service is possible!

Summary

13592439_10209393081521754_3545192990541101721_nEvery year, I goto the Grandfather Mountain Highland Games for a wonderous 10 day vacation in a remote-ish part of the NC Blueridge Mountains.  Because of the location, getting reliable internet service in and around the hills where the games are held and campers camp is a bit difficult.  There are a few local carriers that can reach various parts of the hill, but no service is 100% available everywhere.  So this year, I decided I was going to address that. I took some spare parts and devices I had lying around and created a working “mesh” network that extended “wifi” service around the campground.  I ran into some unique challenges and successes that will contribute to an improved service for next year.

Pros

  • Wifi mesh networks work reasonably well to move data around an environment, even when multiple hops are involved
  • Well configured routers and load-balanced hotspots make a reasonably fast network. \
  • Signal strength to user-devices wasn’t as much of an issue as I thought, the distribution of AP’s covered the spaces adequately; however signal strength between AP’s was.
  • Using weather-proofed Wifi devices and Power over Ethernet Cable (POE)  are easy to cable and give much leeway in where devices can be placed.

Cons

  • The mesh networks take a while to “stabilize” when the power up.  EG.. Nodes that are further away from the gateway take the longest to take up as each node in line has to establish connectivity and configuration.
  • The connected device count can significantly impact operation of the system.  End-point nodes can be so constrained as to not even get DHCP service and therefore no operational capability.
  • Mesh network layout and configuration can also significantly impact the operation of the system as you lose 1/2 your bandwidth with every “hop” of an access point.
  • There are *way* too many ways users and programs can find to “stream” data, and thereby it’s difficult to constrain network usage.
  • When using Cell Service for bandwidth you’re constrained by cell tower signal strength, and the other users on the cell.

While, the effort wasn’t 100% successful, I learned a lot and will be able to apply my lessons to make it bigger and better for next year.

Concept

The initial idea was driven by the fact that I had almost no signal near where I camp on the mountain, however going up the hill about 200 yards, there was ample signal ( this has since changed.. as T-Mobile is extending services ) and I wanted to have simple internet in my camp.  So I started exploring ways to solve that.  Last year, I had an AT&T MiFi “hotspot” setup up the hill at the Pack-Rats camp store.  This allowed me to help them with credit-card sales ( as they didn’t have a carrier with reliable service ) and was going to get me signal in camp.  Alas .. the hotspot didn’t quite reach and I had iffy service at best.

I decided this year to improve the concept by extending Wifi down the hill to my camp using a 802.11ac “mesh” network where the access-points(APs) would talk to each other and relay data where needed.  I have been doing alot of work with the OpenWRT system as well as playing around with Open-Mesh 802.11ac routers and I had a few hotspot LTE based wireless access points and decided to put it all together and create a service that I could use and extend to others.

Design

To create the network I needed a smart gateway to route and manage data.  I do my own builds of OpenWRT ( an open-source Linux operating system ) .. and built a version of the latest OS with the features I wanted:

To connect to the Internet, I have an AT&T branded Netgear AirCard 770S.  I have used this particular device a few times, and had come to rely on it for routing and decent service ( where AT&T will reach ). I also have T-Mobile on my personal devices and thought I might use my Apple iPad2 as a routable point and/or any other devices with reasonable amounts of bandwidth available. Initially, I had explored routing to the LTE Wifi via the USB port, however this proved very inconsistent between the implementations of the devices and decided to instead use the native wireless capabilities of the router.

I grabbed a spare TP-Link Archer C7 1750 router and applied the configuration and added a few features of my own:

  • Setup the LEDs to let me know when connectivity was established with the different ‘wan’ interfaces configured.
  • Setup squid to work as a transparent proxy on non-SSL web traffic.
  • Setup logging to a plugged in MicroSD card on the USB port.

So my initial configuration looked like this:GMHG-Base

  • iPad setup in access-point mode
  • AT&T Hotspot on LTE
  • TP-Link as gateway
  • Wired connection to OpenMesh router as the mesh gateway

I had hoped the iPad would do 802.11a ( 5GHz ) to allow me to leverage both radios in the router, but it didn’t. I learned that both the iPad and Hotspot needed to transmit on the same channel to allow for the TP-Link’s 802.11b/n (2.6GHz) radio to see them concurrently ( and there’s no way to set the channel on an iPad.. grrrr ).

I tested this at home and it worked reasonably well.  I could see packets moving, almost cable-like speeds and the proxy was effective for non-SSL sites.  I expanded the configuration with a few more “mesh” AP’s I had around .. and tested using my laptop to beat on the network from various locations around my house and neighborhood.  This again yielded decent results.

Deployment Plan

HardwareIMG_2883

I already had 3 of the OpenMesh AP’s so I grabbed a few more and water proof cases.  To power them I decided to leverage CAT-5 vs trying to use standard power supplies.  The AP’s have several different power requirements ( 18-24v and 48v ) but I found single plug “bricks” that allowed me to just plugin to the outlet and run cat-5 from there.

Networks

So I wanted to allow camping users to also share this hard work, and also still wanted to be able to support PackRats, so I created a few separate wireless SSID networks (in order of bandwidth availability:

    • Matheson – my network for monitoring and troubleshooting. (And I wanted to say that Clan Matheson was all over the Mountain 😉 ) This network required a WPA2 password to access and was setup directly off the native LAN ( 172.16.0.0/16 ) and had unrestricted bandwidth.
    • Vendor – this was setup to support any vendors ( we only had 1 ) and required a voucher code to connect to. This was also setup to the native LAN, but LAN access was disabled, only internet bound routing allowed.  This too had unrestricted bandwidth per connected device.
    • Camping – this was for the local people to connect to to be able to get general access.  This was setup on it’s own VLAN (100) and designed for 1000+ devices (172.18.0.0/22) mainly to alleviate any DHCP spamming.  This was initially restricted to 10Mb/s .. and reduced to 5 Mb/s to reduce congestion.
    • Open – this was for anyone not camping, and was dampened down with very little bandwidth.  This was also setup for 1000+ devices ( 172.18.64.0/22 )  (the numbers of users on this network later proved untenable and the network was disabled)

Software

I didn’t want devices to just connect, I wanted controlled access. Luckily Open-Mesh had the Cloudtrax software integrated, so I was able to setup a “splash” page which required users to accept usage terms and conditions as well as a PayPal button for donations for bandwidth.  For the vendor network, Cloudtrax supports ‘vouchers’ which allowed me to control, track and assign each vendor a single code for a controlled number of devices and bandwidth.

I considered using a ‘walled garden’ approach for these configurations, but I still needed them to approve access before getting access so that tool wasn’t as useful as I’d have liked. I setup dnsmasq to be the DHCP/DNS servers to  control some access to the network via DNS blackhole .. however also proved untenable as many users just used external DNS, and dnsmasq wasn’t as responsive as it needed to be under heavy loads. I also setup some tools to detect and kill streaming.

Deployment

13439112_514859712042049_724458573276327631_nSo after arriving and setting up camp, I grabbed the router, hotspots, wire ties and some AP’s and started putting them up.  I decided initially to put the base station at Packrats up near the road as they had someone there to watch it and the signal strength up there was good.  As soon as I booted everything, configured the iPad and hotspot to both use the same channel, and gave it a little time for the first AP to check in.. I had internet! Fantastic.

Screen Shot 2016-07-12 at 11.27.06 AM I used an android tool called Wifi-Analyzer to track signal strengths and channel usage and started walking down the roads and finding power poles with an outlet w/in 15 feet ( that was the CAT-5 Cable length I was working with) and put them up.  My network steadily grew to 4 AP’s.  I started also noting users were finding it before I even announced, 35 were connected w/in 1 hour.

Growth

So my little network that could .. started working .. with the low numbers of users I didn’t have any real problems, and bandwidth was reasonable given the circumstances.  As the next few days moved ahead, and we got more and more users on the campground, things changed:

  • By July 3 I had 112 concurrent devices and using up to 2.8 Mb/s .. at that rate I burned through the data I had, but I also had a few donations come in, so I bought some more data on the AP and extended my T-Mobile account to handle things. I also noted the numbers of devices per AP was starting to approach their capabilities so I ordered a few more AP’s.. they would arrive Wednesday July 6.
  • By July 4 I realized that the T-Mobile didn’t count data for BingeOn services  AND was bw-limited by T-Mobile, so I reconfigured the firewall/DNS settings to force supported streams that way.  I saw an improvement in my data-usage as tools and services started using Youtube via BingeON vs the other services.
  • I had one major outage on July 4 as data ran out on both AP’s concurrently in the middle of the night.  I was able to resolve in the morning and get things back up reasonably quicklyScreen Shot 2016-07-12 at 11.40.16 AM
  • On July 6, the new AP’s came in .. and specifically got the biggest baddest one they had the MC1750 which could handle way more users and was more powerful.  I added and extended the network, keeping the campground the strongest group. Once the mesh reconfigured, I found the AP’s waayyy up on the field could see it directly.  This was great as it gave us way more capability.
  • I had the second major outage on July 8 during a severe thunderstorm that had 87 mph winds and caused some carnage.  Most notably, the signal patterns for LTE changed around camp, and I couldn’t get reliable service inside PackRats anymore; however AT&T was usable .. so I added more data on that service and left it up and went to cleanup my camp.
  • On July 9, after discovering that LTE was now stronger in my camper than up the hill ( bent antennas perhaps ) .. I reconfigured the network to have the gateway in my camper and left the other AP’s to re-mesh ( This took about 1.5 hours to settle )
  • I removed the field AP’s on Sunday at 4 when the games officially closed, and removed the camping ones Monday morning.

Review

So I gathered lots of data to share:

Usage

  • Screen Shot 2016-07-12 at 11.44.37 AMThe system serviced a total of 1135 devices, with 259 active at one point in time.
  • Users used 68.4 Gb of data, which translated into about $250 in data costs with LTE service.
  • Usage patterns matched the weather, if it was raining I saw way more internet usage 😉
  • Usage patterns also matched the locations of the access points,  the coverage of camping had significantly more users.

Top Applications

  • Screen Shot 2016-07-12 at 11.47.47 AMSSL was the #1 usage, which is a good thing for users as your communication was protected, but bad for me as I had way to meter the usage.  This likely covered Mail and some internet usage.
  • HTTP was the next, which means the proxy was likely worth setting up.
  • Streaming was more popular than Facebook

Top Devices

  • Screen Shot 2016-07-12 at 11.51.15 AMApple is still cooler than Microsoft it seems … tho in the phone world it’s close.
  • Android devices use more data than iOS.  That may be due to being cheaper, and more “heavy” users use them to do streamy stuff ( looking at the younger generation .. but no data to support it )

Heavy UsersScreen Shot 2016-07-13 at 8.29.08 AM

  • Some users really use the network hard no matter what you do to mitigate it.
  • I played some whack-a-mole for a while trying to restrict heavy usage, but as user counts went up it became a fools errand.

Access Points

  • The distribution of the AP’s was based on available power and location to attach.
  • As predicted, the higher an AP was placed, the better it operated.
  • More powerful, higher-density AP’s worked significantly better and lowered the number of hops between devices.

Lessons Learned

So many good things I learned:

      1. There is a need and desire for Internet accessibility at the event. There were a total of 1135 devices that attached to the service, and even with the degradation we had due to the too-many-device disconnects and slow bandwidth.. users still were very active until the last moment ( there were 126 active devices on Sunday at 3pm, and 58 Monday morning when I had to shutdown. )  Usage patterns indicate people were using social media services such as Facebook, Instagram, Twitter and Youtube and using it consistently.  I also heard from many that were outside the range of the access points (McRowdy) that they would really like to have service.
      2. Outside influences make large impacts. After the storm, when the service patterns on the mountain changed caused accessibility issues.  Vendors whom accepted credit cards mostly used their own service; however after the damage to the cell tower, many were unable to continue to do that as they couldn’t get signal.  Because i had the flexibility to move my LTE reception point, I was able to re-establish service quickly and effectively. The other thing i noticed is the saturation of the AT&T LTE cell tower impacted my services directly.. e.g. when 10k people were at the event with their phones banging on the tower. The T-Mobile side worked better, perhaps because it’s newer or it’s less prevalent in that area of the state and so fewer users.
      3. LTE service can work in this environment, with caveats. Having 2 or more hotspots really made a big difference in throughput, especially proxying the non-SSL traffic.  Having a higher bandwidth gateway will make a big difference, and deploying multiple gateways would be even better. The meshed AP’s support having multiple gateways at different points, however has it’s own challenge with NAT/Routing. Something to debate.
      4. Internet Usage is hard to control and more prevalent than expected. More and more sites are going to SSL and providing higher and higher bandwidth services. This translates into more usage for carriers ( which makes them happy for charging and also presents the challenges for supporting it ) so finding ways to not have to restrict access would be advantageous as it’s an uphill battle.
      5. WIFI Access points need to handle higher densities to be effective in  this environment. The numbers of devices that attached far exceeded my expectations and even base usage by persistent apps on devices can saturate the capabilities of meshed AP’s when more than 1 or 2 hops from the gateway.
      6. Funding a donation service is possible. I did this year on donations and didn’t quite have my data expenses met, however many said they would happily donate to have access.  There were a few impediments:
        • Many people don’t like PayPal and won’t use it
        • Many people would rather give cash ( or Scotch! )
        • The Donate section was missed ( though it was above the Free Access [ Continue ] button )
      7. Campers block signal. I had several individuals that noted they could not get signal in their campers, including one particular one where the access point was less than 10 feet from the camper; but could get LTE off a hotspot.  I believe the problem is related to the radio frequency as WIFI is generally 2.6 GHz and LTE is generally lower; which means it can penetrate walls much better.

Next Time

Some things I am already planning for the next iteration of the Camping Wireless

    • More access points! – My objective will be to get many more of the MC1750’s deployed in strategic locations so they can mesh more effectively.  I have already talked to OpenMesh and they are working on some bigger and better devices with higher density. I also hope to cover more of the area.. including McRowdy and Happy Valley.
    • Fiber Connectivity – I have a vendor that was willing to donate 100Mb/s to the field from the maintenance building for Grandfather Mountain, however we weren’t able to make it happen this year.  I will pursue this for next year and get it linked up to the field.  I will continue to have the hotspot configuration available as fail-back in case fiber goes down.
    • Larger Infrastructure –  I plan to deploy multiple routers in a fail-over configuration and move the proxy to higher speed hardware with redundancy.  I will have UPS setup for the power for the devices ( so 87 mph winds can’t knock us down ). I also hope to setup some point-to-point wireless to extend w/o having to mesh to improve the device density per access-point per hop.
    • Assistance – I’ll be looking for a few volunteers with tech experience to assist me with getting things setup and monitoring the system.  I did this mostly on my own this year and loved it .. but I can see this becoming bigger.  I also hope to propose this as a service that GMHG can use as well to improve services to the Games in general.
Posted in Uncategorized, WiFi | Tagged , , , , , | Leave a comment

Partners in Business at Utah State

So I was asked to speak at Partners in Business Information Technology Conference at Utah State University.  I was honored to be the closing Keynote speaker.  I had an amazing time and met some wonderful people. This trip was amazing.

On my flight out I was met by the Conference Coordinator, Brandon Layne.  We hit it right off and had a grand time just riding in the car to-from Salt Lake City airport. Talking all kinds of IT stuff and geek culture.  Went to dinner at a local’s place that had some good barbecue, and I crashed at the University Inn.

Sunday, I went and found Cafe Ibis in town, what an amazing place. The coffee was awesome and the atmosphere was great.  Met some great people there asking about my stickers on my lappy 😉 and talked about hacking and NASA in general.  Brandon came back and picked me up and we had another 1.5 hour trip to the airport to chat some more on geeky stuff.  Picked up Pat and made our way back to Logan where I caught up with my friend Dr. Nicole Velazquez.  We had lunch with her and then went back to her new place and hung out with her husband… we closed the evening watching The Matrix over good scotch.

Next day I got to speak on the local NPR affiliate Access Utah program with David Thaw on computer security.  That was a great time and we got some fun phone calls to answer.  The conversations were good and our host was awesome.  After that, David and I went and visited the Utah University IT Sec Department and got to see some neat tools they use to monitor their network.  We also had some good discussion about phishing and management.  They gave me access to the tools, so I am gonna take a look at em when I get home.  I then got to meet and hangout with my Student Host, Russ.  He was a great guy and we had some awesome discussion about life, computers, tc.. We closed the day with a really nice dinner sponsored by the university and then Dave, Pat and I sat round my room talking about stuff.

Tuesday we had a fantastic time at the conference.  The opening keynote was about 3D printing and well presented.  Pat and David’s talks were both great and well received.  At lunch we had a security panel and got to debate all sorts of interesting stuff, from Password vaults to what Students should do as they approach graduation to get into IT Security.  I spent some time ‘judging’ some great business IT proposals and got to talk to some students who had really interesting ideas in the areas of business information management and even cloud-based authentication.  At the end of the day, I got to give my ‘Hacking your Mind and Emotions’ talk.  It’s one of my favorite and was well received by everyone there.  To close the day, we had a thank-you dinner, and I got to sit with students and talk to them about security and life in general.

I am on the way home now and fondly remembering my experience.  Utah was surprising in how amazingly nice everyone was and how much I enjoyed the discussions.  I will always remember all the really cool people I got to meet: Brandon, Tim, Russ, Kristen and even Taylor who was the young lady at the hotel who told me where to get some really neat things to take back home ( I got cheese!! ).

My thanks to everyone at Utah State and the Business school for the wonderful time!

 

 

 

Posted in IT Security, Uncategorized | Tagged | Leave a comment

LISA 13

So I had a grand time teaching ( and speaking ) at LISA this year.  Made some new friends, saw some old ones and really enjoyed being a part of such a wonderful group and idea.  I have been invited to be on the Program Committee for next year, so now I have a chance to give back.  Many thanks to everyone who came to class and for the kind words about it and my talk, I have been quite humbled at the response.  I will endeavor to create an Advanced version for next year.

Posted in Uncategorized | Leave a comment

Shootout is ON!

Please meet at the hotel at 10am. We’ll be leaving from there!

 

Posted in Uncategorized | Leave a comment

I’m christian unless you’re gay

So I read this fantastic response to this original post called “I’m Christian unless you’re gay”.  What a fantastic read.  Earlier in my life, I was in a christian band.  Somewhat surprising if you know that I was at the least an agnostic, and border-line atheist at that point in my life.  I always held that music was the reason I did that, but I also felt it was a good exploration of faith for me.  And I found it lacking.  Specifically after two events that touched me.

First, we had a fan.  Well we had many 😉 but one in particular was a young girl, who was very very strong in her faith, and probably one of the nicest persons I have known. I got to know her very well, invited her and her bf on a mini vacation to a country house we had access to. Etc. A perfect example of someone who lived her life for everyone else. She always had a smile, always wanted to help.

She died at a very early age of 17 coming back from work.  A drunk driver entered the interstate going the wrong way and had a head on with her. She didn’t even get the dignity of passing quick, she ended up in the hospital in pain, for months, contracted staph and slowly faded away.  Between that and losing my sister to SIDS early, it helped define my thought that faith was misplaced at the very least.

But to the point of this article, during this same period, we had a band we opened for routinely.  They were better than we were, had albums, etc.  The lead singer, had an amazing voice and was one of my favorite people.  One day, he didn’t show up with the band to play with us.  The rest of the band was angry and upset.  I found out, that the singer “came out” to them that he was gay.  They immediately fired him.  This action too struck me as particularly un-christian like.  Love the sinner and all that.

I have since pretty much solidified my unbelief (aka thomas covenant) and hearing of persecution of gay or ‘different’ viewers makes me at least as angry as the proud mother was in that article.  I am happy that she has changed her views, however I think it’s tragic that it takes a personal matter to make that happen.  Many christians are the ultimate hypocrites in my view.  They pick and choose the parts of their faith that are the most hateful, while blithely ignoring (For the most part, there are exceptions) the ‘love thy neighbor’ parts and the parts that contradict.  Sadly, it’s not even the ‘fringe’ but core to many parts of the tenants of the faith.

I can rant on this all day, but I felt that if I can make others aware of these articles, perhaps they too can have a personal change in how they approach ‘different’ people.

 

Posted in Personal | Tagged | Leave a comment

Legend of Korra

So the first 2 episodes are up http://www.followthesignal.com/the-legend-of-korra/ and so far it looks great!! if you were a fan of Aang… this seems like it will be just as good.

Posted in Uncategorized | Leave a comment

Welcome!!!

I have this post up to welcome everyone to the site who are considering coming to the shootout this year!! You can read below for the evolution of this years event, but we will be going to the excellent Range37 for our annual weapons extravaganza! I encourage you to look around for yourself, I anticipate this to be much fun.

I have created a mailing list for notices and discussion for the event.  Please go subscribe here to be informed of carpools etc.  I’ll be updating this blog as well if you want to RSS.

I am looking for fun ideas for contests this year.  The balloon board was a hit (pun intended) last year, and we’ll probably do something similar.  If you have thoughts, mail me so we can start to put things together.

Lastly, this year we WILL have an onsite grill ( Big enough for a pig .. working on that 😉 and picnic tables and .. wow! .. bathrooms!

I look forward to hearing from everyone, and can’t wait to see ya for the fun.

Posted in CarolinaCon, Shootout2012 | Leave a comment

TSA Scanners found to be easily penetrated

https://tsaoutofourpants.wordpress.com/2012/03/06/1b-of-nude-body-scanners-made-worthless-by-blog-how-anyone-can-get-anything-past-the-tsas-nude-body-scanners/

This is a very interesting read on how the scanners can be easily subverted. If you fly, this is worth your time to read.

Posted in Flying | Tagged | Leave a comment

Flying a 737 in a demo today

Well .. flying a simulator. We have high-fidelity plane modeling and medium fidelity simulator. I’ll have someone take pics of me and post em here.. but lots of fun. I have learned that flying a Jet is way different than flying a GA plane. The autopilot really makes things much easier. One thing about this system is the yoke has no trim .. when you come out of autopilot .. where the yoke is .. is neutral. So it’s good to get on final and glide-slope to make sure you can land it.

Posted in Flying, NASA | Leave a comment

#CCShootout is a GO!

I have made arrangements with the guys at Range37 ! Sounds like it’s perfect for us:

  • we start at 11am .. and can shoot till close ( Tho we’ll leave around 4ish to make it to the ‘con )
  • $200 to rent a bay ( 20-25 shooters on the line at a time )
  • A Bay is 35 Meters deep and we can put most any target out there we want
  • They have a grill and picnic tables for lunching.. so we can plan some cookin!
  • Range officer on site at all times to manage the shoot
  • they have some other ranges available ( up to 400 yards )
  • they have a shop on site with:
    • ammo
    • targets
    • rental for all kinds of weapons

    So sounds like we’re set. I am working with the #Carolinacon guys now to get the announcement ready and start working on extra things:

  • Registration?
  • Lunch?
  • T-Shirts?
  • Contests?
  • Carpool?
    More on this here as I move forward.

  • Posted in CarolinaCon, Shootout2012 | Leave a comment