WISP Design – Migrating from Bridged to Routed

TFW adding the 301st subscriber to your bridged WISP….

 

 

Why are bridged networks so popular?

  • Getting an ISP network started can be a daunting task. Especially, if you don’t have a networking background.
  • Understanding L1/L2/L3 is not easy – I spent a number of years working in IT before I really started to grasp concepts like subnetting, the OSI model and Layer 2 vs. Layer 3. It takes a while.
  • Bridged networks are very attractive when first starting out. No subnetting is required and the entire network can be NATted out an upstream router with minimal configuration.

 

What does a “bridged” network look like?

  • Bridged networks use a single Layer 3 subnet across the same Layer 2 broadcast domain (typically over switches and software/hardware bridges) which is extended to all towers in the WISP
  • Bridging can be done with or without VLANs but they are most commonly untagged.
  • The diagram below is a very common example of a bridged WISP network.

 

What is the difference between switching and bridging?

These days, there isn’t much difference between the two terms, switch is a marketing term for a multiport hardware-accelerated bridge that became popular in the 1990s to distinguish it from hubs which did not separate collision domains. Both types can use VLANs, spanning tree and forward Layer-2 frames to multiple ports

  • Bridging (Software) – Most radios use some variant of linux and a bridge that is dependent upon the CPU to forward frames. Most routers will also allow ports to be bridged together in software and the speed is dependent upon system resources and load
  • Bridging (Hardware) – Most commonly, you’ll find this in vendors like MikroTik and Juniper. Certain hardware models allow the bridge to be offloaded into hardware instead of CPU so that frames can be forwarded at wire speed
  • Switching (Hardware) – This category includes most all ethernet switches. Frames are forwarded using ASICs that depend on a CAM table to hold MAC addresses.

 

What are some of the issues with bridged Networks?

 

  • Broadcast – One L2 broadcast will go through every radio and backhaul in a WISP network
  • ARP Traffic -Also an L2 broadcast type, ARP storms and heavy ARP traffic can easily cripple a bridged network
  • MAC Table size limitations – some equipment types have limited MAC table sizes. Often in CPU based equipment, the limitation is either RAM or default settings in the linux bridge which is typically limited to 1024 entries per bridge.
  • Scale – Typically anything beyond a /24 (254 hosts) will start to have issues without a number of L2 enhancements like client isolation, MAC filtering, etc. At some point those solutions don’t scale
  • Subnetting – ideally you don’t want multiple subnets on the same broadcast domain for security and isolation of failure domains
  • Performance – most routers are more efficient in routing packets vs. bridging packets in a larger network.
  • Security – Implementing security policies and isolating customers and protected infrastructure is much easier at L3

 

How does routing help?

Routing separates broadcast domains

Here is what a single broadcast domain looks like in a bridged network

 

 

And to compare, here is the same network but routed

 

How do I  Migrate?

 

Answer: Patience and planning (and VLANs)

The question you’ve probably been waiting for….how do I migrate? So the dirty little secret is that you don’t have to migrate all at once.

There are a few different ways you can use VLANs to migrate the network one tower at a time.

 

Where do I start?

 

  • Prep work – be sure to back up all configs and if possible, put together a diagram of your network that includes physical connections and VLANs / Subnets where applicable – this can be done with Visio, LucidChart or using mapping software
  • Pick a good time – Look at your monitoring software and pick a day and time that represents your lowest volume of traffic
  • Be realistic about time – If you think it will take 1 hour, plan for 4 – you’ll be amazed how often 1 hour turns into 4 ūüôā
  • Have a rollback plan – Understand what steps you need to take to roll back – even better, write it down!

 

Types of migration

 

Type 1 – Last mile back to the core – start at the very end of a chain of towers and work your way back in – one tower at a time

  • Benefits
    • Lower risk, only affecting one or two towers at a time at the end of a chain of towers.
    • Doesn’t specifically require VLANs in some network topologies but they are still recommended
    • Easy rollback, if it doesn’t work, replace the original config and analyze what went wrong
    • If you’re successful, you can move to the next closest tower and repeat the process which continues to shrink the broadcast domain – this also has the side benefit of helping to stabilize your bridged network as you migrate by making it smaller.
  • Drawbacks
    • Distance – in some networks, getting all the way out to the edge during a late night maintenance window can be a challenge

 

 

Type 2 – Core out to the last mile – start at the core or where your bandwidth comes in (often the same place) and work your way out – one tower at a time

  • Benefits
    • Physically closer to migrate the first hop
    • Can use VLANs to keep the existing bridged topology but route to towers that are converted
    • Same as the first, if you’re successful, you can move to the next closest tower and repeat the process which continues to shrink the broadcast domain – this also has the side benefit of helping to stabilize your bridged network as you migrate by making it smaller.
  • Drawbacks
    • Risk – if you have an issue with the first hop, you may take down a larger number of towers than 1 or 2 as compared to the first method.
    • Requires more config – you need to preserve the legacy broadcast domain through converted towers and then go back and clean it up

 

 

Type 3 – Build L3 to a new tower from the core –¬†If you happen to have a new tower build on deck that will directly connect back to the core (where the gateway for the bridged L3 network is), then you can build a new tower as L3.¬† This helps to understand what’s involved and then use one of the previous two methods to migrate the rest of the network.¬†

  • Benefits
    • Lowest risk option – building out new sites in a different design is one of the lowest risk ways to migrate
    • No need for rollback – it’s new and not in service, so if you don’t get it right on the first try, you can keep working on it
  • Drawbacks
    • There aren’t any major drawbacks to this approach, except the rest of the network must still be migrated using one of the previous two methods

 

 

Using switch-centric design to assist with migration

In order to pass VLANs and the legacy broadcast domain through the network easier, consider putting all physical links into a switch at the core and the tower instead of directly into the router.

This type of design makes operation of the WISP significantly easier as new subnets and services that are needed don’t always need a trip to the tower to add cabling.

It also makes config migration easier when upgrading the tower router by putting most of the interface references into VLANs instead of physical interfaces.

Example of a switch-centric tower design

 

 

Closing thoughts…

This will help to get you started down the road to migration. Using a virtual lab like EVE-NG or GNS3 will help to understand the concepts before you deploy it in prod and is a good addition to the process.

Take your time and think through what you want to do and write down your plan – often you’ll find gaps when you create a list of steps which you can correct before migration and save time.

Good luck!

 

Need help with your Migration? Call the WISP experts at IP ArchiTechs

https://iparchitechs.com/contact-us/

Cisco to MikroTik – Switching and VLANs

 

 

About the Cisco to MikroTik series

 

One of the most difficult configuration challenges for MikroTik equipment seems to be switching and VLANs in the CRS series. Admittedly, the revamp of VLAN configuration for MikroTik CRS switches in early 2018 made things a lot easier. But, sometimes there is still confusion on how to configure VLANs and IP addresses in VLANs with MikroTik RouterOS operating on a switch.

This will only cover VLAN configuration for CRS 3xx series switches in RouterOS as SwitchOS is not nearly as common in operational deployments.

CRS 1xx/2xx series use an older style of configuration and seem to be on the way out so I’m not 100% sure whether or not i’ll write a similar guide on that series.

If you’ve been in networking for a while, you probably started with learning the Cisco CLI. Therefore, it is helpful to¬†compare the commands if you¬†want¬†to¬†implement a network with a MikroTik and Cisco switches.

This is the fourth post in a series that creates a Rosetta stone between IOS and RouterOS. Here are some of the others:

Click¬†here¬†for the first article in this series – “Cisco to MikroTik BGP command translation”
Click¬†here¬†for the second article in this series – “Cisco to MikroTik¬†OSPF command translation”
Click here¬†for the third article in the series – “Cisco to MikroTik MPLS command translation”

While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.

Hardware for testing

In the last article, we began using EVE-NG instead of GNS3 to emulate both Cisco IOS and RouterOS so we could compare the different commands and ensure the translation was as close as possible. However in switching, we still have to use real hardware at least in the realm of MikroTik – Cisco has IOSvL2 images that can be used in EVE-NG for switching.

 

Notes on hardware bridging in the CRS series

Bridging is a very confusing topic within the realm of MikroTik equipment. It is often associated with CPU forwarding and is generally seen as something to be avoided if at all possible.

There are a few reasons for this…

1. Within routers, bridging generally does rely on the CPU for forwarding and the throughput is limited to the size of the CPU.

2. In the previous generation of CRS configuration, bridging was not the best way to configure the switch – using the port master/slave option would trigger hardware forwarding.

After MikroTik revamped the switch config for VLANs in 2018 to utilize the bridge, it more closely resembles the style of configuration for Metro Ethernet Layer 2 as well as vendors like Juniper that use the ‘bridge-domain’ style of config.

Using the bridge for hardware offload of L2 traffic

Note the Hw. Offload verification under this bridge port in the CRS317

It is important to realize that bridging in the CRS, when used for VLAN configuration is actually using the switch ASIC to forward traffic and not the CPU.

In this instance, the bridge is merely used as a familiar configuration tool to tie ports and VLANs together but does in fact allow for the forwarding of traffic in hardware at wirespeed.

 

Cisco to MikroTik – command translation

Cisco commandMikroTik Command
interface FastEthernet5/0/47
switchport access vlan 100
switchport mode access
end
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1 pvid=100
interface GigabitEthernet5/0/4
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 200
switchport mode trunk
end
/interface bridge vlan
add bridge=bridge1 tagged=sfp-sfpplus1 vlan-ids=200
interface Vlan200
ip address 172.16.1.254 255.255.255.0
end
/interface vlan
add interface=bridge1 name=vlan200 vlan-id=200
/interface bridge vlan
add bridge=bridge1 tagged=sfp-sfpplus1,bridge1 vlan-ids=200
/ip address
add address=172.16.1.254/24 interface=vlan200 network=172.16.1.0
spanning-tree mode mst
/interface bridge
add fast-forward=no name=bridge1 priority=0 protocol-mode=mstp region-name=main vlan-filtering=yes
interface FastEthernet5/0/47
switchport access vlan 200
switchport mode access
spanning-tree portfast
end
interface bridge port set edge=yes-discover
interface GigabitEthernet5/0/4
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 200
switchport mode trunk
channel-group 1 mode active
end

interface Port-channel1
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 200
switchport mode trunk
end
interface bonding
add mode=802.3ad name=Po1 slaves=sfp-sfpplus1,sfp-sfpplus3 \
transmit-hash-policy=layer-2-and-3

/interface bridge vlan
add bridge=bridge1 tagged=Po1,bridge1 vlan-ids=200
show mac address-tableinterface bridge host print
show mac address-table vlan 200interface bridge host print where vid=200
show mac address-table interface Gi5/0/4interface bridge host print where interface=sfp-sfpplus1
show interfaces trunk
show vlan
interface bridge vlan print
show spanning-tree
interface bridge monitor
show etherchannel summaryinterface bonding print detail

 


Examples of the MikroTik RouterOS commands from the table above



Untagged switch port

This command will create an untagged or “access” switch port on VLAN 100

 

Tagged switch port

This command will create a tagged or “trunk” switch port on VLAN 200. Additional VLANs can be tagged on a port by using the same syntax and adding a new VLAN number.

 

Layer 3 VLAN Interface

Similar to a Cisco SVI (but dependent on the CPU and not an ASIC) this command will create a layer 3 interface on VLAN 200

 

Multiple STP

This command will set the bridge loop prevention protocol to Multiple Spanning Tree. As a general observation, MSTP tends to be the most compatible across vendors as some vendors like Cisco use a proprietary version of Rapid STP.

 

STP Edge port

This is referred to as “portfast” in the Cisco world and allows a port facing a device that isn’t a bridge or a switch to transition immediately to forwarding but if it detects a BPDU, it will revert to normal STP operation. (this is the difference between edge=yes and edge=yes-discover)

 

LACP Bonding

This command will create a bonding interface which is similar to a Port Channel in Cisco’s switches. Two or more physical interfaces can be selected to bond together and then the 802.3ad mode is selected which is LACP. You can also select the hashing policy and ideally it should match what the device on the other end is set for to get the best distribution of traffic and avoid interoperability issues.

 

View the MAC table of the switch

This print command will show all learned MAC addresses and associated VLANs in the CAM table of the switch

 

View the MAC table for VLAN 200 in the switch

This print command will show all learned MAC addresses in VLAN 200.

 

View the MAC table for bonding interface Po1 in the switch

This print command will show all learned MAC addresses on port Po1.

 

View the current VLANs configured in the switch 

The bridge vlan print command will show all configured VLANs in the switch.

 

View Bridge Spanning Tree information 

The bridge monitor command will show the configuration details and current state of spanning tree including the root bridge and root port

 

LACP Bonding information

This command will show the details of the LACP configuration and whether the bonding interface is running which indicates a valid LACP neighbor.