• Home

Items filtered by date: December 2015

Tuesday, 05 April 2022 13:22

My Server Rack Mount Cabinet Notes

This is my research into Server Cabinets

My Kit

  • Cabinet: Excel Environ CR600 - 600mm Wide Comms Rack (542-2468-GSBN-BK) (24U 600w x 800d)
    • 24U allows me to have:
      • a shelf for my UPS
      • a shelf for 2 x Desktop PCS
      • a shelf for Low profile PC that I will use for my router
      • a rack mount KVM switch
      • a rack mount switch
      • a rack mount patch panel
      • Some cable tidies
    • Not too tall so I can put a monitor on the top and a Wifi keyboard attached to my KVM for occasional use.
    • Max Load: 800KG
  • 3 x StarTech 1U 19 inch Adjustable Vented Rack Mount Shelf Heavy Duty Fixed Server Rack Cabinet Shelf (ADJSHELFHDV)
    • They fill the full depth of my cabinet (from post to post)
    • Strong
    • Can be used for whatever I want
    • Vented to keep air flowing and I can use these vents to secure equipment with
  • Penn Elcom Delux Cage Nut Tool CN01 | PennElcomOnline.com
    • Good for removing nuts easily and can be used for putting them in when plenty of space.
  • Penn Elcom Economy Cage nut tool SX1100 | PennElcomOnline.com
    • Easy for putting tools in. can be done from the front so not much space in the rack is needed.

Cabinets / Racks

  • 4 posts
    • Cabinets which have 4 posts can usually have their posts moved closer together if required (i.e. they are adjustable)
    • 21inch (53.34cm) is the distance between the forward and rear facing parts of the posts (usually)
  • 2 post
    • This configuration is usually found in the smaller cabinets.
  • Half
  • Different widths and heights
  • Most 19" cabinets are 600mm wide. This allows some space either side of the racks.
  • Rack Unit Calculator from Penn Elcom Online
    • A rack unit (or U) is the standard terminology for describing sizes associated with 19 inch racking. Knowing how much rack unit space you require is incredibly important when installing 19 inch rack cabinets and enclosures. The unit calculator below can convert rack U's into cm, inches and feet, which makes it a very useful tool for any installer or musician who needs to know exactly what equipment to buy when building a 19 inch rack. Scroll down for a complete table of rack U-to-inch/feet/cm values.
  • Server Rack Buying Guide | Server Rack Ordering Info
  • How to Plan A Server Rack Installation | Cabinet Layout Guide | Home Network Rack
  • Before You Buy - Server Racks | StarTech
    • This video covers things you need to know before you purchase a server rack, including what a rack unit is, the difference between 2 and 4 post racks, open and closed racks, and minimum and maxmimum mounting depths, as well as to consider mounting devices on both sides of a rack, whether you need casters, want to bolt your rack to the ground, or leave it free standing.

Shelves / Drawers

There are different Types of Shelves

  • Cantilever
    • This are fixed by 2 posts. A front or rear pair.
    • When describing the size of these shelves i.e. 350mm/450mm/500mm this refers to the depth of the shelf.
    • They are all the same width because they are for 19" racks.
  • Clamping
    • This is similar to a cantilever shelf except the sides are level all the way to the back.
    • These level sides allow clamps to be attached to the sides to help attach things.
  • Fixed
    • These require a 4 post cabinet and are fixed at both ends.
    • Tend to have higher load capacities
    • When describing the size it refers to the overall depth of the cabinet usually. So a 800mm deep cabinet would require a 550mm (21inch/53.34cm + the M6 bolts) so it would fit inside the cabinet. Double check this before purchase as sometimes sellers get mixed up.
    • They are all the same width because they are for 19" racks.
  • Fixed (Adjustable)
    • Same properties as a Fixed shelf but with the ability to configure for a range of cabinet depths.
  • Sliding
    • These are used for things like keyboards or light equipment you need to pull out easily.
  • Drawers
    • These are just like your normal clothes drawers except made out of metal.
    • Usually have locks on them

They can come with Different options

  • Vented
    • Holes are milled into the metal to provide better air circulation.
  • Heavy Duty
    • Thicker or better quality metal that can take more weight.
    • These units are usually heavier that normal shelves.
  • Different colours
    • Personal choice, but I always will pick black.

Where to Buy

Cabinets

Shelves

Published in Networks
Read more...
Sunday, 03 April 2022 12:18

My Treadmill Notes

General Treadmill Questions

Slippery Treadmill Belt / Why are my trainers slipping on the Running Belt?

This is dealing with you and your trainers slipping on the treadmill belt, not about the belt slipping on the motor.

Scenarios

  • Visually it looks like it should still have traction. when I clean it, it looks pretty gross and seems to help a little bit but gets slippery again soon after.
  • When I try to google anything the results are mostly people talking about "slipping" as in the belt itself is slipping.
  • New trainers don't make a difference
  • The belt feels slippy underfoot
  • My trainers started to slip while walking, running is no longer safe.
  • Cannot push and pull the belt while the Treadmill is off with just my feet and trainers (Belt surface is very slippy).

Causes

  • The belt is smooth because it is covered in dirt.
  • The belt is too smooth to provide grip.
  • The belt is now porous and is letting the lube through from the underside of the belt to the top layer which you walk on.
    • When I wipe the belt down, black grease comes off.
    • It sounds like you are slipping on the lube for the belt. The lube is on the underside of the belt to keep it from dragging on the walk platform but sometimes some of it can work its way out of the edges and might have gotten onto the top of the belt. In that case you can wet a cloth in warm water and mild soap and use it to remove the lube from the walking surface.
    • Since you already tried wiping away the lube with no success it might be that your belt has worn to the point where the lube can soak through the belt itself to the walking surface. Unfortunately, in that case the lube will continue to work through every time that you clean the belt and the only way to get your treadmill back to normal will be to replace the walk belt.
  • The belt has completely worn out.

Solutions

  1. Take a bucket of warm soapy water and bristled scrub brush and scrubbed the whole thing, then towel it off. This will remove any lube that has come trough the mat.
  2. Roughen up the belt with sandpaper.
    • Hold down sand paper on the belt while it's running.
    • It seemed to help a little but I think the sandpaper I used was to fine grit.
    • Don't use too abrasive sandpaper or you will ruin your belt.
  3. Buy a new treadmill belt (because it is worn out)

Example of Bad Running Belt

The images below show the underside of my failed Running belt.

 

Treadmill Running Belt is slipping on the motor

This section deals with issue relating to the belt not being correctly moved by the motor.

The Running Belt might need oiling/Lubrication?

  • If the belt slips on the motor.
  • The treadmill is making a droning noise.
  • The speed is inconsistent, i.e. when your feet touch the belt, it slows down.

How to lubricate your Treadmill Running Belt

Carl Lewis MOT25 - General

Description

The Carl Lewis MOT25 Foldable Motorised Treadmill comes with a 0-9mph speed range and a full 15% maximum power incline, adding more variation to your workouts. Workout programmes and pulse monitor provides motivating and safe exercise. The running deck folds up at the press of a button and can be wheeled away for easy storage.

Features

Walking and running are some of the best forms of exercise for improved general fitness. Using a treadmill puts less impact on your joints and also offers a more varied workout with performance feedback.

  • Power incline 0 - 15%
  • 3.0 HP peak
  • 0 - 10 mph speed range
  • Cut off key
  • Power folds for storage
  • 41 x 132cm running area
  • Computer Displays; speed, distance, time and pulse
  • Product dimensions: (H)117, (W)160, (D)80cm
  • Max body weight: 115kg

Links

Servicing Carl Lewis MOT25 Treadmill

Servicing the Console

On my console I found I had the following issues

  • Buttons did not work properly, and would stick often.
  • Console was very loose and so would not stay upright
  • LCD displays were not always working properly

Follow these instructions and you should be able to fix both.

Obviously make sure the unit is not plugged in

  • Sellotape the console to the handles as shown below. This prevents the unit falling while you are taking it apart.
  • Unscrew the 15 screws at the BACK of the unit. There are 4 screws under the sticker on the front which would allow access but it would make a mess of your console so i decided not to go this route.
  • Carefully pull the back casing away from the unit and you will find some wires fall out. Make a note of these wires and how they look for easier re-installations, a photo with your phone is probably best.
  • Put the case back to one side. I would put the screws in this to keep them safe.
  • Reposition the case for easier access to the PCB.
    • Remove the Sellotape holding the case in place
    • Push the case forwards and it will come off the main crossbar
    • place the console case flat on the arms of your treadmill
  • Remove the 2 rubber grommets making not of their position

    • the flat bit goes at the bottom and allows the screw cylinders to meet each other when you put the case back together
  • Clean the rubbers and crossbar (this fixes the console not standing up straight)
    • simply was the rubber grommets in warm water (no soap) with your hands to remove all of the dirt and then let them air dry some where warm in your house.
    • If the rubber has perished, use a fine grain sandpaper to roughen them up a little.
    • Clean the cross bar with a lint free cloth and some Isopropanol Alcohol
  • If you want to remove the wires from the PC to make things easier, make notes of what went where or take several photos.
    • I personal left all of the mains wires connected and just disconnected the heart monitor sensors after putting labels on them and then motor control plug.
  • Remove the 2 screws from the heart rate monitor module.
  • Remove the other black screws from the main PCB.

    • Note how tight them were.
    • Also the image above still has the Heart Rate monitor attached. There are 2 more screws underneath it for the main PCB.
  • You can now remove the PCB from the case.
    • Some of the LCDs and their rubber connectors will still be attached and some left on the PCB, so make note of what went where
  • Pull the rubber buttons out and clean them
    • Warm water in your sink (no soap) and wash with your hands gently.
    • Shake of the majority of the water and then place the rubber buttons somewhere warm in your house to air dry. You should make sure both sides get dried.
    • Do not use any cloths to clean these as the fibres will get stuck on the button and you will have to start again.
  • The button pads on the PCB need cleaning.
    • Using a lint free cloth and some Isopropanol Alcohol just rub on the pads gently until they are clean
  • Cleaning the display feed PCB pads - I have not done this bit.
    • Using the same technique as the cleaning the button pads, clean the display PCB pads
  • Cleaning the display rubber signal transfer pads - I have not done this as I am not sure if this will damage them
    • Clean them with Isopropanol Alcohol
  • Re-install the dry and clean buttons
  • Re-install the main PCB
    • Correctly locate the display transfer pads either on the PCB on on in the main case ready for mating.
    • Place the PCB back in the main case
    • Screw the PCB down making sure that you lightly press the board down while doing this to get a good compression of pads.
  • Re-install the Heart Rate monitor daughter board.
    • place down correctly and screw in
  • Re-install the dry and clean rubber grommets (in the correct direction)
  • Push the case back on
    • Correctly locate the wires (as per photo you took earlier)
    • push the case back on to the crossbar with the rubber grommets going into their appropriate slots.
    • The 2 metal stops should be push against the black blocks on the cross bar. This should put the console in the upright position.
    • Sellotape the case in position (as before) so we can work on it easier.
  • Reconnect the heart rate monitor

Now is a good time to check if all of the buttons, displays and functions work before continuing. Be safe when messing with mains voltage. Fix any issues before going any further.

  • Change to/from Miles/Kilometres if required.
    • see notes below
  • Push the rear case back onto the main case
    • When you do this make sure no wires are trapped where they should not be as this will stop the case going back together.
    • If in doubt consult your photo from earlier on how the wires should go back in.
  • Done

Your buttons and displays should now work with no issues for years to come.

Changing from Miles to Kilometres

Assuming you have the console opened as described above, the change from Miles to Kilometers or the other way around is easy.

There is a jumper on the PCB that controls this feature and only has 2 positions:

  1. Closed (Jumper present) = Miles
  2. Open (Jumper missing) = Kilometres

The location is shown in the image below (currently closed), set to your preference as required:

This option is set in the factory due to the region it was going to get sold in.

Removing the Running Belt

  • Put the treadmill flat with no incline
  • Because there is a large capacitor in the motor housing you should leave the unit to discharge over night or be very careful.
  • Make sure the treadmill is off and unplugged
  • Remove the Rear Roller Adjustment covers by removing the 2 screws that hold each of them in.
  • Remove the side rails by pulling sliding them off the end.
     

     
  • Your treadmill should now look like this with the Main Motor Cowling still present at the back
  • Remove the Power Supply and Main Motor area cowling.

    • removing the 4 holding screws
    • lift the covering straight up. It should move with relative ease.
    • The will now expose the power supply and main motor
  • Remove the Rear Roller (not the motor end)

    • Unscrew the adjustment bolts a little bit on one side and then the other, so the bolts are released evenly otherwise the roller could become stuck or damaged. Keep doing this until the bolts have been removed.
    • Put the bolts to one side
    • Take a photo of the roller so you know how it was installed
    • The roller is now loose and can be removed by pushing it towards the motor


      Make sure you do not loose the metal washers
    • Place the roller on a piece of clean plastic or surface. A bin/Trash bag should do.
    • Put one of the adjustment bolts back in the roller and the other placed back in the treadmill. This is so you can identify which way around the roller fit in the treadmill.


  • Remove the front motor roller
    • Before removing the bolt, put some Tipex on the shaft so when you take it out you know the insertion depth it was set at.
    • The adjustment bolt should be unscrewed all the way.
    • Remove the motor belt whilst making a note of what direction it goes in
    • Remove the roller and place next to your other roller.
  • Remove the the Deck (and belt)
    • Unscrew the anti-static plates from the Deck and then put the screws back in the board for safe keeping
    • Unscrew the Deck fixing screws in the order as shown below but only partway before you move to the next screw.

      • They have to be removed in this way because the screws sometime unscrew the rubber mounting below instead of coming out on their own and this method prevents the Deck from getting damaged by lifting the Deck evenly irrespective of which screws fail to come out of their rubber mounting on their own.



      • Take your time doing this section and make sure you use a the correct screw fitting (low profile fat cross)
      • When you think you have undone the all of the a screws, lift the Deck and see if it is free.
      • When free lift both the Deck and the belt away from the treadmill.
    • Separate the belt from the Deck
    • The running belt is removed.
    • Remove any rubber mountings still present on the underside of the deck.
    • Your treadmill should now look like this
    • Replace any rubber mounts that came off with the Deck back onto the frame of the Treadmill.

Cleaning the Running Belt area

This assumes you have just removed the old one and your treadmill following the instructions above.

This is optional but highly recommend because you don't swap your running Belt everyday.

While your treadmill is open, clean the following parts with warm soapy water:

  • The rollers
    • Clean only with a lint free cloth by wiping the rollers clean of any dirt.
    • They do not need to be oiled
    • So dirt might just need some encouragement with your finger nails to come off, again be gentle.
  • Deck
    • avoid the slippery running surface with soapy water, this should be cleaned with a clean lint free cloth only
  • Anti-Static plates
  • The side rails
  • The corner covers
  • The Motor Housing/Cowling
  • The Motor area

Installing a new Running Belt

This assumes you have just removed the old one and your treadmill following the instructions above.

  • Follow the instructions above in reverse with the following exceptions:
    • Before running or configuring the belt, Check to see if your Running Belt needs Lubrcation. If it does, apply some lubrication sparingly to the slide surface of the Deck. This is important to prevent damage.
  • Done

Front Roller Angle Configuration

Not all Treadmills have this adjustable front roller

This is required otherwise your Running Belt will always track left and no amount of adjustment will fix it.

Method 1

  • Screw the thread in to the Tipex mark you made earlier on the adjustment bolt

Method 2

  • Pull the belt tight against the Front roller
    • With your hands evenly placed on the belt (so both sides are pulled the same)
    • Palm down and pulling the running belt towards you
    • Not much force is required
    • The motor will make a bit of a weird noise, but this is normal
  • You should now see that the running belt is probably on an angle
  • Using the drive roller adjustment bolt
    • Tighten this slowly until the Front Roller and therefore the Running Belt are in alignment (parallel) with the Deck. Check with a ruler as shown below.
    • The belt should be in the middle of the slide surface.
    • Every few turns you might need to re-pull the belt with you hands to make sure it is sitting right and the adjustment is applied correctly.

How to configure the Running Belt's tension (Euro Treadmill Belts Method)

Video: Replacement treadmill belt - how to tension your belt | EuroTreadmillBelts.com

These are my notes for this method which I have used successfully.

You can use masking or electrical tape instead of Tipex

  • This method creates a 0.5% pre-tension on the belt.
  • This is a precise method and so should your measurements
  • Before starting, the belt should be installed but not tensioned
  • Tensioning your belt
    • Make 2 marks on each side, exactly 1 meter apart. It does not really matter where these marks are as long as they are 1m apart.
    • Leaving the tape measure on the treadmill (not on the Running Belt), tension one side untill you have stretched it my 5mm. You will see the Running Belt stretch against the tape measure.
    • Now do the other side.
    • See if the belt is now gripping, if not, tension each side up a little more (not too much)
  • Dynamically Track
    • If the belt is gripping, you can make adjustments using the tension screws independently to correct any issue with the left/right tracking.
  • After Fitting
    • Ideally the belt should allow enough room to fit the belt and give about 50-60% of the adjustment potential to tension the bel. This allows for further tensioning during the belts lifetime.

Stomp Test

After you believe you have configure/tensioned your Running belt correctly as per the method above, we now need to do a final test.

  • How To Adjust A Treadmill Belt | Treadmill Maintenance
    • This is best to be done with the cowling of so you can see the actually motor, but is not required.
    • Video time is set to the part where the stomp test is demonstrated
    • This video also shows you how to track a Treadmill but not pre-tension it.

Re-assemble the Treadmill

This is pretty much the reverse as above (after installing your new belt), but here is an overview:

  • Replace the motor cowling
  • Put the side rails back on
  • Put the Rear Roller Adjustment covers back on.

Cleaning the Incline and Vertical Store motor areas

This area is only possible to access if you put the treadmill into it's vertical storage mode.

  • Put the treadmill in to it's vertical storage mode
  • Because there is a large capacitor in the motor housing you should leave the unit to discharge over night or be very careful.
  • Make sure the treadmill is off and unplugged
  • Remove the feet protectors
  • Remove the Vertical Storage and Incline Motor Cowling

Parts Close Up Pictures

  • Power Supply and Main Motor Area (Front)
  • Power Supply and Main Motor Area (Rear)
  • Main Motor
  • Main Motor Belt
  • Power Supply
  • Power Supply Switches
  • as
Published in Health
Read more...
Tuesday, 01 March 2022 14:11

cPanel - Global Email Filter not working

I wanted to filter emails by the From field using the 'Global Email Filters' feature in my cPanel Account, but this is not working.

Below is an example rule called 'Dodgy SPAM (PCP)'.

And this is an example email I wanted to block:

From:	PCP Claims <pcpclaims@somespammer.com>
Sent:	13 February 2022 17:35
To:	iamspam@example.com
Subject:	If you took finance for your car, you may have been overcharged
 
Did you take finance for your car? 

There's a chance that you were overcharged 
  
If you have a current PCP agreement, or it has ended within the last 5 
years, then the likelihood is that you were the victim of mis-selling.  
 
Whether that's undisclosed commission, hidden APR's, failure to offer a 
more suitable product or one of many other reasons. 
 
Get A Free Assessment 

If you no longer wish to receive these let us know here

When i tested the email it says "Error: You do not own an email filter that matches the given parameters."



So my rules are in place and I have a suitable email text to test and it should be working.

NB: It should be noted that blocking SPAM using the individual email account filters works fine, just not the global one.

Solution

After some back and forth tickets with my provider the following solution was reached and my issue disappeared.

There is some problem with 'Global email filter' option from the cpanel itself due to which is not working. Now we have forwarded this issue to the Cpanel team. They will look into it and once we received any update from them then we will update you on the same ticket accordingly.
Hosting Provider

So they then followed this up with the following resolution:

I found that the vfilters file for this domain was missing. This is where the global filters are normally stored and read from when Exim processes them.

[01:46:26 server77 root@xxxxxxxx ~]cPs# cat /etc/vfilters/quantumwarp.com
cat: /etc/vfilters/quantumwarp.com: No such file or directory

I created this file and corrected permissions.

[01:47:00 server77 root@xxxxxxxx ~]cPs# touch /etc/vfilters/quantumwarp.com && /scripts/mailperm abcdefg --verbose
Checking mx configuration for demo.quantumwarp.com (abcdefg)...[auto]...Done
Checking mx configuration for quantumwarp.com (abcdefg)...[local]...Done
Checking mx configuration for search.quantumwarp.com (abcdefg)...[auto]...Done
Fixed permissions on /etc/vfilters/quantumwarp.com : was (0644), now (0640)
Fixed ownership of /etc/vfilters/quantumwarp.com : was (uid=0,gid=0), now (uid=1481,gid=12)
Fixed permissions on /home/abcdefg/mail/quantumwarp.com/hosting/dovecot-quota: was (0600), now (0640)

The custom filters are now there as normally expected:

[01:50:28 server77 root@xxxxxxxx ~]cPs# wc -l /etc/vfilters/quantumwarp.com
65 /etc/vfilters/quantumwarp.com

With this, the filters should proceed to function as expected. You may find this helpful for testing the filters going forward:

As per their update (cPanel), there was issue with the vfilter configuration file which was missing for the site (quantumwarp.com). Now they have rectified the issue and  'Global Email Filter' option is working fine now.

 

Published in cPanel
Read more...
Sunday, 13 February 2022 09:42

Commodore Tape Transfers

I struggled to find instructions on how someone actually transferred a real Commodore Plus4 or C16 tape into a TAP file written for newbies. I hope to address that and once you have done a couple everything will make sense. I always find once I have got something to work everything else gets easier.

Prerequisites

Hardware

  • 2 x Pendrives
    1. One for booting Dos
    2. One for holding all of the software we will run (Commodore and PC) and to hold TAP of captures tapes.
  • 1 x Transfer cable (X1531 / XE1541)
    • X1531
      • Used for connecting 1531 directly to the PC's parallel port.
    • XE1541
      • Is better for newer PCs.
      • is compatible with VC1541 and MTAP so I will use this one.
      • This is the same as X1541 but with diodes for newer types of parallel ports.
    • Other cables such as can be used, but for the purposes of this tutorial this is what I am going to use becasue it is what I have
  • 1 x Old PC with a compatible Parallel port (LPT) (DOS PC)
  • 1 x Modern Windows 10 PC
  • 1 x Commodore Computer (C16 / Plus4 / C64)

Software

  • Rufus - For creating bootable USB Pendrives
  • Windows 98 SE Boot Disk Image
  • Enhanced DOSKEY
    • Enhanced DosKey with autocompletion
    • You can probably use DOSKEY from MS-DOS if you can find it.
  • CWSDPMI - (Is included in MTAP and VC1541)
  • MTAP
  • TAPSERV (included in MTAP)
    • Software that runs on a C64 to aid in generating digital backups of tapes (TAP) of the X1541 cable range.
    • Only Wholewave (V1 TAP) format is supported.
  • PTAP
  • VC1541
    • 1541 Drive emulator you run on your PC (supports X1541 or XE1541)
    • This works with any Commodore that has the serial port.
  • TTAPSERV
    • By Ulysses777
    • Software that runs on a C16/Plus4 to aid in generating digital backups of tapes (TAP) of the X1541 cable range.
    • Both Wholewave (V1 TAP) and Halfwave (V2 TAP)  formats are supported.
  • Tape Wave Checker (formerly WAVCHECK)  (will add link later)
    • By Ulysses777
    • A VERY experimental program for checking the waveform of a tape.
  • PTAP
    • Temporary download link
    • By Ulysses777
    • This is an experimental modification of PTAP which will also work with halfwave tapes. There are no extra command line switches needed.
  • TAPClean Front End
    • A fork of FinalTAP with a GUI
    • Maintained by Luigi Di Fraia (and is maintained)
  • Archiver-Copy
    • Plus4
    • Written by Pigmy, a legendary member of the early Plus/4 scene
    • Should handle reading halfwave.
    • Copy between any combination of tape and disk.

Batch Files and Commands

We now need to work out what commands we are going to need and then build the relevant batch files.

Bach files are optional because they just run the command we program them to do but they will save you a lot of time and heartache when you get the command wrong and have to type it again.

MTAP

Most of the transfer methods below use MTAP so it is important we deal with the MTAP commands now and while we are at it, we will make some batch files to make life easier.

The Commands

mtap - Commodore TAP file Generator v0.36

usage: mtap [-lpt] [-x[e]] [-buffer <size>] [-v] <tap output file>
  -lpt<x>:  use parallel port x (default: lpt1)
  -x:       use X1541  cable for transfer
  -xa:      use XA1541 cable for transfer
  -xe:      use XE1541 cable for transfer
  -b:       increase buffer size (default: 4 MB)
  -h:       halfwaves
  -v:       record Version 0 TAP
  -vicntsc: record VIC-20 NTSC tape
  -vicpal:  record VIC-20 PAL  tape
  -c16ntsc: record C16 NTSC tape
  -c16pal:  record C16 PAL  tape
  -c64ntsc: record C64 NTSC tape
  -c64pal:  record C64 PAL  tape

Once you understand these you can write your own as required.

Explained Examples

mtap -lpt1 -b 10 -h -c16pal C:\TAPS\OUTPUT.TAP
  • -lpt1 - MTAP only works on the parallel port so what is this. This is X1531 mode and is the original method to grab tapes using the X1531 cable (or C64S adapter)
  • -b 10 - Use a buffer of 10MB. MTAP needs to store the whole tape in memory otherwise it will fail. The buffer default is 4MB but some tapes are bigger (not many) so it does not harm to set it to 10MB.
  • -h - Sample the tape in halfwave mode. This sets MTAP to sample in halfwaves which is specific to C16 and Plus4 computers
  • -c16pal - There is a need to set the frequency of the sampling because it is different in the NTSC and PAL regions and also across the different devices.
  • C:\TAPS\OUTPUT.TAP - This is obviously the output location of the TAP file. Keep the filepath and filename all in CAPS so VC1541 can handle the TAPs.
mtap -xa -b 10 -c16pal C:\TAPS\OUTPUT.TAP
  • -xa - use XA1541 cable for transfer. This requires you are running either TTAPSERV.PRG (C16/Plus4) or TAPSERV.PRG (C64) on your Commodore of choice. These are called Tape Servers and aid in getting full digital copies of tapes via real hardware.

Batch Files

In this section you build your own MTAP commands and then move them into batch files (makew.bat and makeh.bat).

These batch files will go in the root of the Software pendrive and can be called instead of typing out the whole command every time (unless you install DOSKEY)

  • Create your MTAP commands.
    • We are going to make one for Wholewave and one for Halfwave tapes.
    • There are some examples below.
      mtap -lpt1 -b 10 -h -c16pal C:\TAPS\OUTPUT.TAP
mtap -lpt1 -b 10 -c16pal C:\TAPS\OUTPUT.TAP
  • Make the MTAP Batch commands.
    • We are going to make one for Halfwave and one for Wholewave tapes.
    • Edit the file makeh.bat (Halfwave) and put your command in it with the format: 
      mtap -lpt1 -b 10 -h -c16pal C:\TAPS\%1.TAP
    • Create a file called makew.bat (Wholewave) and put your command in it with the format: 
      mtap -lpt1 -b 10 -c16pal C:\TAPS\%1.TAP
    • To run the command from DOS, enter the relevant command below to which will create a TAP with the name OUTPUT.TAP
      makew OUTPUT
makeh OUTPUT
      • When the batch file is run, the batch variable %1 will be replaced by the name, OUTPUT.
      • So from this you can see you only have to type in the new name for the TAP file whether you are sampling a Wholewave or Halfwave tape.
    • Done

VC1541 Batch file (Virtual Disk Drive)

  • Create a file called diskemu.bat and its contents should be this command line 
    vc1541 -dir C:\DISKS -lpt 1 -cable XE1541 
  • Move this batch file to the the root of the Software pendrive
  • Done

Setup Windows PC

We all need one of these.

Setup DOS PC

This is your old computer with a compatible parallel port.

Build DOS

Now we need to configure the DOS environment.

  • Bootable pendrive
    • Install and run Rufus on your Windows PC
    • Take one of the pendrives and make it bootable with Rufus using the Win98SE boot disk image.
  • Software Pendrive
    • Take the other pendrive and format it with FAT32
    • Put the software on the pendrive as shown below:
      /DISKS/AZIMUTH.D64
/DISKS/TAPSERV.PRG
/DISKS/TTAPSERV.PRG
/DISKS/TAPE_WAVE_CHECKE.PRG
/TAPS/
CWSDPMI.EXE
diskemu.bat
doskey.com
makeh.bat
makew.bat
MTAP.EXE
PTAP
VC1541.EXE
    • The folders 'DISKS' and 'TAPS' must be in capitals for VC1541 to use them.
    • Yes I know they are not all disks !!! but is a nice easy name for a folder.
  • You might not use all this software but it makes it easier for these instructions.

Boot DOS

We need to boot from the DOS pendrive (on the DOS PC) you have just built and follow these instructions to complete your DOS environment preparation.

Connect your Commodore cables to the PC before turning on either the PC or the Commodore on to prevent damage.

  • Place both pendrives in your DOS PC
  • Ensure in the BIOS that the following are set
    • LPT1 = ECP (this is a good default but this depend on your cable)
    • Boot from USB is enabled
    • Legacy USB Keyboard/Mouse
    • Check other settings are suitable.
  • Turn the DOS PC on
  • Window 98SE Boot disk will load
  • Select "Start computer without CD-ROM support." as this prevents any junk being loaded
    The bootable pendrive will map to the drive letter A:
    The Software pendrive will map to the drive letter C:
  • Run the following commands 
    c:
doskey
CWSDPMI   (optional, see below)
    • C: just puts us where we should be.
    • DOSKEY just makes life easier because you can use the up and down arrows to save re-typing commands.
    • CWSDPMI
      • Is not always needed.
      • To check, run MTAP.EXE and if it runs without error, installing CWSDPMI can be skipped.
      • I had one motherboard that needed this another that ran with the need. This might be the difference between AMD and Intel but I am not sure.
      • If CWSDPMI is required, the error will be "Load error: no DPMI - Get csdpmi*b.zip"
      • In Windows you don't need the CWS*.EXE files at all, since Windows uses DPMI already. But don't use MTAP in Windows, this just helps explain the issue.
      • DPMI = (DOS Protected-Mode Interface) server
      • The sed (Stream Editor) FAQ - 5.5. What is CSDPMI*B.ZIP and why do I need it?
        • This explains the issue well.
        • binaries and sources are available here.
      • Alt Downloads
      • HDPMI - DPMI Server (Version 3.17) - An alternative to CWSDMPI but I have not used it or had the need. This is for reference only.
  • Your DOS environment is now setup and ready for use.
  • Proceed to 'Transfer Methods' section and pick which options suits you and your equipment.

Notes

  • This is my setup which works well but you can
    • Try other DOS disks such as FreeDos which will only require 1 pendrive
    • Different software for creating bootable USB
    • Be crazy and boot from a real floppy disk
    • Use a Win98SE disk with the bare minimum of files on it. I just used the default one because I did not want to spend the time removing all of the unneeded stuff.
  • if you use Rufus and to create bootable USB pendrive then the partition on it will only be 1.44MB which is normal.
  • MS-DOS 6.22 does not support FAT32 so that is why we use Win98SE boot diisk which supports it.
  • You might need to try different LPT modes on your mother board (SPP/ECP/EPP/ECP+EPP) to see which one works. bear in mind your adapter cable has to be compatible with that LPT mode.
  • Parallel Port (additional)
    • Parallel port - Wikipedia - An article on Parallel Ports and their different modes.
    • Downtown Doug Brown » Parallel Port Tester
      • After playing around with a ton of parallel port cards in an attempt to figure out Willem programmer compatibility, I decided it would be useful to write a parallel port tester program. Just a simple utility where you can set the output value of each output pin and read the value of each input pin.
      • An easy to use Windows utility
    • Parallel port output - Simple circuits and examples to describe how to use PC parallel port as general purpose output port.
    • How to Use a PC's Parallel Port to Commun | Maxim Integrated - This tutorial shows how to build a quick, simple, and cost-effective 2-wire (I2C-compatible) interface using a PC's parallel port. No difficult to procure ICs, microcontrollers, or firmware needed.
    • DOS Utilities Collection
      • Many DOS utilities
      • LPTstat v1.0 (LPTSTAT.EXE) - The PC parallel port uses a practically direct I/O port to pin connection. LPTstat shows each bit's status in real-time, and includes the bit's location in the PC address space and its corresponding pin number on a DB-25 connector. You can use the left and right arrow keys to switch LPT ports, and the up and down arrow keys followed by ENTER to toggle a bit.
    • PCI Parallel port doesn't work under DOS 6.22 | PC Review
      • Hi all,I tried to get a PCI parallel port (LPT2) to work under DOS 6.22, but without luck. When I connect the printer to LPT1 it is immediately online even if the computer is turned off, whereas for LPT2 the card isn't recognized, and the printer never gets online.
      • Some PCI parallel cards just will not work under as a DOS parallel port. Under DOS, there were only 3 recognized addresses ( 0x3bc, 0x378, and 0x278) and 2 recognized IRQs 7 & 5. These are not normally assigned to a PCI device because of possible conflicts with existing devices. Any PCI card that will work with DOS programs must be settable to these settings, either through a utility program/driver or jumpers. I know there are some out there, just don't remember who makes or sells them. You need to search for a PCI card that specifically says it will work with DOS programs. Or rewrite the DOS program to use the addresses and irqs the PCI card is being assigned.
    • Check if printer is on LPT from DOS batch file - 'm searching for a method to check from a batch file if a printer is connected to LPT port.
    • How can I make a test lpt port in MS-DOS ? - Techyv.com - Hi everyone, I think my lpt port is damaged, please, how to proceed to check this error? And how to make a dos lpt test? thank you in advance!

 

Transfer Methods (Commodore Tape to PC)

There are several methods on transferring tapes to .TAP files on a PC but are mostly the same for Vic20, C16, Plus4, C64, C128, PET tapes.

  • Plus4 and C16 tapes should be always sampled in Halfwave mode (TAP V2)
  • C64, C128, VIC20 and other Commodore tapes should be always sampled in Wholewave mode (TAP V1)

Connect Parallel Port direct to a Commodore Cassette drive (X1531 Method)

There are 2 adaptors that can be used to connect your tape drive directly to your LPT port on the PC,  this just depends on whether it is a 1530 or a 1531 drive. These two drives are identical inside and just differ in their connectors. The 1531 has a mini DIN connector whereas the 1530 (C2N) has a female edge connector. When people refer to the X1531 Cable this can mean either drive setup with their appropriate adapter.

  • C64S tape adapter
    • In this link you can see a modified X1541 connector with a tape port added. You can make an adapter without the serial port.
    • Don't connect the tape drive and a serial device at the same time
    • Only connect the power when using the tape drive.
  • X1531 tape connector - A simple tutorial on how to make the adapter for the 1531 Tape Drive.
    • Will only work on a SPP parallel port on older PCs.
  • More notes on cables below in the 'Cables' section

Some people online say that the 'Game Port' might not be able to supply enough power to the Datassette when doing long tapes. I am unable to confirm this but perhaps should be something to watch out for.

Instructions

  • (Optional) Edit your batch files on the Software pendrive
  • Make sure the PC is turned off.
  • Connect the X1531 cable to the PC and the 1531 Datassette.
  • Make sure the parallel port is set to SPP
  • Load your DOS environment on the DOS PC
  • Turn the Commodore computer on.
  • Place a cassette into the 1530/1531 the drive that you want to sample.
  • Make sure the tape is fully rewound before you start.
  • Now sample the tape with MTAP by running one of these commands on the DOS PC, select the appropriate one or use your own:
    mtap -lpt1 -b 10 -h -c16pal C:\TAPS\OUTPUT.TAP
mtap -lpt1 -b 10 -c16pal C:\TAPS\OUTPUT.TAP
makeh OUTPUT
makew OUTPUT
    • The batch commands assume you have set your batch files as follows (to match the examples above): 
      makeh.bat
mtap -lpt1 -b 10 -h -c16pal C:\TAPS\%1.TAP

makew.bat
mtap -lpt1 -b 10 -c16pal C:\TAPS\%1.TAP
  • "Press <PLAY> on tape!" will appear, now press play on your Commodore Datassette.
  • The tape will play to the end and then play button will pop up as normal.
  • (optional) Re-run to get a second copy
    • Binary compare the 2 tapes (or quickly by size with DOS dir command), they should be the same. same size or very close.
    • If they don't match it could be because 'the tape is damaged or dirty'/head needs aligning/tape players is defective/heads are dirty/rubber band faulty
    • Regular cleaning of the tape heads when using old tapes is important
  • Turn off the DOS PC and move the Software pendrive to the Windows PC
  • Load the TAP in an emulator and see if it works.
  • Done

TTAPSERV (Preferred Method)

This will do Wholewave and Halfwave tapes (i.e. All Commodore tapes)

This is the most reliable method of reading taps because it is using real Commodore C16 or Plus4 that supports both Wholewave and Halfwave tapes, and this method ensures the Datassette will get enough power for it's motors when dealing with long tapes.

This is what I do to transfer my commodore tapes to the PC.

Instructions

  • (Optional) Edit your batch files on the Software pendrive
  • Make sure the PC is turned off.
  • Connect the Plus4 to the parallel port of the PC using the XE1541 cable
  • Make sure the parallel port is set to ECP
    • This is recommended in the VC1541 instructions because of auto-detection of EPP/DMA is dodgy.
  • Load your DOS environment.
  • Turn the Commodore computer on now otherwise you will might get a cable detection error and file transfers will be disabled.
  • Run the VC1541 software on the DOS PC. (CWSDPMI.EXE might not be needed on your system, see notes above) 
    VC1541.EXE -dir C:\DISKS -lpt 1 -cable XE1541
or
diskemu.bat
  • The 'DISKS' folder should already be selected because we defined it in the commands above and this folder has the TTAPSERV.PRG in it, if not, just browse for the file with your cursor keys.
    • The directory shown on the left is what is presented as the directory to the Commodore computer.
  • Load TTAPSERV.PRG on your Plus4 by typing in the following command:
    LOAD "TTAPSERV.PRG",8
  • VC1541 now will transfer the file from the PC to the Plus4 as if it were a real 1541 disk drive.
  • Type RUN on the Plus4 to launch TTAPSERV.
  • Set TTAPSERV to the appropriate mode to match your MTAP command.
    • F1 = Wholewave
    • F2 = Halfwave
  • Exit VC1541 on the DOS PC
  • Place a cassette into the 1530/1531 the drive that you want to sample.
  • Make sure the tape is fully rewound before you start.
  • Now sample the tape with MTAP by running one of these commands on the DOS PC, select the appropriate one or use your own:
    mtap -xe -b 10 -h -c16pal C:\TAPS\OUTPUT.TAP
mtap -xe -b 10 -c16pal C:\TAPS\OUTPUT.TAP
makeh OUTPUT
makew OUTPUT
    • The batch commands assume you have set your batch files as follows (to match the examples above): 
      makeh.bat
mtap -xe -b 10 -h -c16pal C:\TAPS\%1.TAP

makew.bat
mtap -xe -b 10 -c16pal C:\TAPS\%1.TAP
  • "Press <PLAY> on tape!" will appear, now press play on your Commodore Datasette.
  • The tape will play to the end and then play button will pop up as normal.
  • (optional) Re-run to get a second copy
    • Binary compare the 2 tapes (or quickly by size with DOS dir command), they should be the same. same size or very close.
    • If they don't match it could be because 'the tape is damaged or dirty'/head needs aligning/tape players is defective/heads are dirty/rubber band faulty
    • Regular cleaning of the tape heads when using old tapes is important
  • Turn off the DOS PC and move the Software pendrive to the Windows PC
  • Load the TAP in an emulator and see if it works.
  • Done

These instructions were adapted from the forum thread Best or easiest way to transfer from tape - Commodore 64 (C64) Forum

TAPSERV

This will do Wholewave tapes only. (Cannot sample some C16 and Plus4 tapes)

  • Same as for TTAPSERV.PRG except
    • You use a Commodore C64
    • You use TAPSERV.PRG as the tape server
    • There is no option to select Wholewave or Halfwave tapes.
  • This is the original tape server that is designed to run on the C64 but is only capable of capturing Wholewave tapes.
  • TAPSERV for the C64 cannot read halfwave tapes, as the C64 is unable to read them itself.

1530USB (Audio and WAV)

I have not used this method

This is a little more around the houses but might be useful when the tape is really dirty.

DC2N (DMP Files)

I have not used this method, hence the lack of detailed instructions

DC2N is a piece of hardware that can do high resolutions dumps of tapes to it's own DMP format which can be converted to TAP but requires a special piece of hardware

This seems to be the preferred way of making tape backups in the scene because they are high res and you can make TAP files from them.

OpenCBM - ZoomTape/ZoomTAP/TapeXUM/tapread.exe

I don't know much about this, but I discovered TAP utilities in the OpenCBM package and references to a ZoomTape device.

  • TapeXUM - c64scene.pl
    • TapeXUM is an adapter that connects a tape recorder (Datasette) to a PC via USB. With its help, you can save and archive data on a cassette from/to TAP files. TapeXUM is based on the ZoomTape project.
    • ZoomTape was an appetizer for ZoomFloppy whose audience is mainly Commodore fans from overseas. In the states, the purchase of a computer with a station in the 1980s was not a utopia and the cassette player was forgotten very quickly and probably often never even found its way into homes, which results in their lack of sentiment towards the cassette medium. The story was different with us, the tape recorder was the basic accessory.
    • I decided to resurrect ZoomTape on a new board using DIY-friendly electronic components, such as the Arduino Pro Micro, which, combined with appropriate modifications in the code, contributed to positive results. I called the whole thing TapeXUM and I published all the files related to the project free of charge. Everyone can assemble and test the adapter with the cassette on their own. Currently, the device will only work in Windows 7 and 10 64-bit, and communication is carried out using tools in the command line.
  • GitHub - r1me/TapeXUM: - Capture and write Commodore tapes via USB device. The project's home
  • TapeXUM - Capture and write Commodore tapes via USB device - Share Project - PCBWay
    • What's TapeXUM ?TapeXUM is a USB device that can read and write to cassettes of Commodore 16/64/VIC-20 (read and write) and ZX Spectrum (read only) format. It requires a Commodore datasette connected to it. This project is based on ZoomTape by Arnd Menge, a great idea and originally a part of OpenCBM package. TapeXUM uses Arduino Pro Micro and through hole components so it's a DIY friendly design.
    • Explains a bit of the history aswell.
  • ZoomTape daughterboard for the ZoomFloppy, when available? - Commodore 64 (C64) Forum
  • ZoomFloppy - CBM 1530/1531 - Tape Drive Connection | jonnz.de 
    • This might of been an original concept but schematics were never released.
    • It gives some screenshots of the software in use.
    • Advises to use the Google Group here - https://googlegroups.com/group/zoomfloppy-users 
    • zoomtape.de.vu - This seems to be the same site. I have used Wayback Machine to get this site
  • Zoom Floppy and Cassette | Google Groups
    • This is the part of the discussion showing a schematic of a ZoomTape/TapeXUM
    • RetroInnovations has made a schematic and mentions Arnd Menge has made an imroved version.
  • CRX 2018 - Nate Lawson - ZoomFloppy | YouTube | Commodore Retro eXpo
    • Presentation by Nate Lawson about the history of the ZoomFloppy and the latest update in the works.
    • This is an excellent talk going ito some of the technical details on how the device and the various Commodore driver work
    • ZoomTape Daughter board mentioned at this time - Arnd Menge designed the daughter board but does not seem to of released it.

Transfer Methods (Other)

TAP to WAV

see 1530USB

Manual Copy PRGs to Emulated Disk drive

  • Setup a virtual Commodore Disk Drive on a PC (VC1541/viceEMU/etc...)
  • Run manual BASIC commands or other PRG to copy files from the tape to the disk drive.
  • What command did I use for King Size to Disk. asked here http://plus4world.powweb.com/forum/44433 
  • The emulated disk drive can be supplied from where ever but the VC1541 over the XE1541 cable is a good starting point.
  • From tape to disk? - Commodore 64 (C64) Forum
    • Just using BASIC commands to copy programs from Tape to Disk will not work.
    • The reason is that when you use the SAVE command it will only copy the BASIC portion of the game.
    • You must use a copier program which defeats the object of this method.
    • Some tapes will use their own Turbo Loaders to load the game and this could cause issues.
    • This might only be true if you use SAVE"FILENAME" / SAVE"FILENAME",1 / SAVE"FILENAME",1,0 / SAVE"FILENAME",8 / SAVE"FILENAME",8,0 as these statements only save the BASIC part.
  • SAVE - C64-Wiki
  • LOAD - C64-Wiki
  • When a program autoruns it is because the code that has just been loaded has overwritten the 'Ready' prompt call so when the 'Ready' prompt is called after the program has been loaded, the PRGs code is executed instead of the 'Ready' prompt and thus the program runs.

I have used the following method to get games of the King Size Volume 1 & 2 tapes to a Disk. This will not work with games that have Turbo Loaders.

LOAD"FILENAME"       (Load as normal)
Press Run/Stop       (You must be able to stop the program with this key)

SAVE"FILENAME",8     (Save the file to disk as a BASIC program)
or
SAVE"FILENAME",8,1   (Save the file to disk as Machine Code, If above save does not work try this one)
  • Just to state the obvious
    • You do this procedure on a real Commodore computer whilst running VC1541 Disk emulator on your DOS PC and then the PRG saves will be on your PC.
    • Check the Saves work

Creating TAP files from PC/PRG/Disk (not real tapes) - TAP to Tape

Access real drives in emulators

  • VICE - Connect Real Disk Drives
    • VICE uses OpenCBM as a proxy to communicate with the real drives so it is only limited by this software. 
    • You need a compatible cable to connect your Disk Drive: XM1541, XA1541, XU1541, XUM1541 (a.k.a ZoomFloppy)
      • The following addon cables are supported: XP1541 or XP1571
    • OpenCBM - Supported Disk Drives
    • VICE communicates with the drive, but the uses the CBM4WIN driver which only implements basic access to the drive.
    • Also a lot of custom loaders used in games etc will simply fail when using Vice with a real drive. It's much better to use the XM1541-cable only for creating disk images and using those in VICE.
    • OpenCBM - Supported Drives
    • For saftey only connect your Commodore equiment when you PC is off. I have heard that connecting these devices while on can damage the Commodore kit.
    • In VICE:
      • Settings --> Peripheral devices --> Drive Type = the drive you are adding. I am not sure how important this is, but it cant harm to have this match.
      • Settings --> Peripheral devices --> IEC-Device , enable and set 'Device type' to "Real device (OpenCBM)".
  • YAPE (Parallel 1541 emulation)
    • In YAPE
      • Settings --> Drive X setup --> Drive Emulation = Parallel 1541 emulation [CPU level/compatibility mode]

 I have not found any support for directly using the 1530/1531 to Datassettes in any emulator. If anyone knows different please let me know.

Capture PET Tapes to Disk using a 1530/1531 Datassette attached to a C64

I got this from How can I get PET Tape programs onto disk? | Port Commodore

  • I have not tried this and I don't fully understand this code but I wanted to add this to my site as well so it does not get lost.
  • This involves using a Commodore 64 but with the code being BASIC it might work on a C16/Plus4
  • I am guessing this changes the tape sample frequency.

You could load/save programs (which relocates the start address which you have to fix later) or you can use this program I discovered for the 64. This program will automatically LOAD each file off of tape and subsequently SAVE it to disk (device number 8) and continue until you stop it (or a disk error occurs when it attempts to save a file with the same name as one already on disk, disk full, or end of tape.)

Here is the listing (remember this is written for the Commodore 64):

5 rem tape2disk
10 for i=53181 to 53247:read a:poke i,a:next i
20 print "[clr][down][down][down]tape to disk transfer program!"
30 print"[down]insert formatted disk in drive 8."
40 print"[down]insert tape, rewind and then press play.[down]"
50 sys 53181
100 data 169,1,162,1,160,1,32,186,255,169,0,162,65,160,3,32,189,255
110 data 169,0,32,213,255,169,8,162,8,160,255,32,186,255,169,20,162
120 data 65,160,3,32,189,255,173,61,3,141,251,0,173,62,3,141,252,0
130 data 169,251,174,63,3,172,64,3,32,216,255,76,189,207
  • Type this in, SAVE it (don't want to type it in again, right?),
  • put a blank formatted disk in drive 8, type RUN.
  • If everything is working the message will display and the computer will now ask you to press PLAY on tape, put in a cassette,
  • rewind if necessary, and press PLAY.
  • Let it run through the tape (this could take an hour or more if it is a long tape).
  • Reset the computer and load the disk directory.
  • You should now have the tape programs on the disk
  • NB: that the filenames are all padded with extra spaces, you will need to use a disk or directory editor to “fix” the file names.

Notes

What Happens?

This is a table that will just let you know what happens when you sample tapes using a Datasette 1531 and various setups and tapes. This will help you identify what is normal behaviour and what sort of tape format you are sampling.

Software / Hardware / MTAP Mode Fullwave Tape
(Number Builder)		 Halfwave Tape
(Bridgehead +4)
  • (Fullwave)
  • MTAP Only
  • X1531 Cable / Plus4
  • mtap -lpt1 -b 10 -c16pal %1.tap
  • File Size: 195KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  • File Size: 247KB
  • The program name is visible in VICE TAP browser
  • The program (header) is found and the Plus4 tries to load the PRG but it never succeeds and the tape runs to the end.
  • (Halfwave)
  • MTAP Only
  • X1531 Cable / Plus4
  • mtap -lpt1 -b 10 -h -c16pal %1.tap
  • File Size: 390KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  • File Size: 494KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  • (Fullwave)
  • MTAP + TTAPSERV
  • XE Cable / Plus4
  • mtap -xe -b 10 -c16pal %1.tap
  
  • File Size: 195KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  
  • File Size: 247KB
  • The program name is visible in VICE TAP browser
  • The program (header) is found and the Plus4 tries to load the PRG but it never succeeds and the tape runs to the end.
  • (Halfwave)
  • MTAP + TTAPSERV
  • XE Cable / Plus4
  • mtap -xe -b 10 -h -c16pal %1.tap
  
  • File Size: 390KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  
  • File Size: 194KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  • (Fullwave)
  • MTAP + TAPSERV
  • XE Cable / C64S
  • mtap -xe -b 10 -c16pal %1.tap
  
  • File Size: 195KB
  • The program name is visible in VICE TAP browser
  • The program loads correctly
  
  • File Size: 251KB
  • The program name is NOT visible in VICE TAP browser
  • ?BREAK  ERROR - The tape does not get to the end.
  • (Halfwave)
  • MTAP + TAPSERV
  • XE Cable / C64S
  • mtap -xe -b 10 -h -c16pal %1.tap
  
  • File Size: 195KB
  • The program name is NOT visible in VICE TAP browser
  • Blank screen and the tape just runs to the end.
  
  • File Size: 250KB
  • The program name is NOT visible in VICE TAP browser
  • Blank screen and the tape just runs to the end.

NB: replace %1 for output.tap or whatever you want your tape to be called.

X Series Adapters Types

XA1541 is the most advanced of the legacy X series adapters. ZoomFloppy/XUM1541 is now recommended for standard disk drive operations.

I have found some good explanations of the different cable types and I will put theme here for you to look through

Imaging Tapes

  • When imaging tapes
    • When imaging tapes, keep a constant eye on the tape head, it will get covered in crap from the old taps.
    • Pressing pay will expose the heads making them easier clean. regular cleaning when dealing with old tapes get consistent results
    • I recommend getting a proper tape cassette cleaner
    • If you find you transfers getting smaller or inconsistent and then the TAPs not working this is most likely becasue your tape player heads are dirty. This does not preclude the other basic stuff ie belts, motor and a knackered head. this is caused by tape player working and then it stops working because of use.
    • If the tape keeps causing the play button to trigger (like when you get to the end of the tape) the fastfoward the tape and rewind it and this will loosen it off.
    • The TAP file is created as the tape is played and the size is dependent on the length of the tape, the contents do not change the size.
    • When you transfer a tape to TAP some times there will be a slight difference in the file sizes but this is normal. They should be around the same size.
    • If you start getting a lot of tap grabs faiing using TTAPSERV.PRG then it could be the serial port on your Commodore failing or the serial interface.
    • When using MTAP, if you find you start getting some bad results, try rebooting your DOS PC. I am sure MTAP is stable but it does not harm.
    • If the tape does not read first time, keep trying and trying as it seems to clean crap off the tape.
    • Clean the tape head regularly as tapes deposity crap on the heads, especially old ones.
    • Try at a different part of the day i.e. cold in morning, warm in the afternoon might make data capture better (not 100% abut this one)
    • Dont run your tape player too long (i.e. loads of tapes).
      • These datassettes are old.
      • Let them cool down between batches.
      • When the motors are warm as they do not provide as much power/torque.
    • Tapes that dont read properly on the first time more than likely will have deposited crap on the tape head so after a dogy read, inspect clean the head.
    • I tend to do tapes in small baches of ten and then check/clean heads. i would rather have 10 failed tapes than 50.
    • After imaging long tapes, check the head because it is like doing several small tapes
    • On bad tapes always do A and B sides as one might work
    • Some tapes have different stuff on side A and side B. I generally read the tape insert to assess this. If in doubt do both sides as they might be different anyway (i.e. Winter Games)
    • If you image the tape the several times on the same side and the file size is the same then it is most likely to be a good digitisation.
    • The same program on different sides of the tape can generate slightly different file sizes but should be within 1KB
    • Some tapes are not recorded on both sides (this is rare)
    • Dont leave TTAPSERV running when you are not imaging tapes because it keeps the tape player powered up and spinning (i.e. the capston)
    • If your datassette jams
      • The XE1541 LED will probably go off if running MTAP
      • It will be permanently powered off for safety (i think)
      • You must power cycle your Commodore to bring it back online
  • Check on Plus4 World
    • Always check on plus4world to see if your tape, inlay and any other information such as stickers are present and correct.
    • Sometimes you might have a slightly different version.
    • I would recommend to always image your tapes (in V2/halfwave) just incase this is not on plus4world.
  • If your TAP files are very small (<1KB) when using TTAPSERV or TAPSERV
    • The connection from your PC to the datasette is faulty.
    • check you XEM1541 for blown diodes, i did and I found one had gone. Teseting them is outside the scope of this article. But it is not hard and just requires a multimeter.
  • TAP doesn't work - When you check a TAP image it is not working, these are some reasons:
    • Doesn't Work: potentially the tape is halfwave and you have not added the switch
    • You can check against the same tape on Plus4world and see if it has the same number of parts/files
    • Check it runs in VICE and then if not check in YAPE as some times it will run in one but not the other.
    • Always make sure the all program runs. I also use a cheat when the game has mulitple parts on the tape that are loaded after you complete one section so I can check each part (i.e. Ye Are Kung fu).
    • Getting garbled graphics on the screen
      • Does not (always) mean the TAP image has failed.
      • Some rare games will only play on a C16 (or Plus 4 with 16k ram)
      • Does not like that does not like that particular emulator. Try the game in another emulator.
    • Not all games work with LOAD"" - some need LOAD"",1,1 and some a SYS command. Instructions are usually found on the inlay if required.#
    • Check for notes or loading instructions on plus4world, someone might have already found and fixed the issue
    • The datasette needs cleaning.
    • Set the Tape player Head Azimuth
    • When using VICE use 'Attach/Load' the 'Autostart' does not always work. i.e. Space 2 Pilot will not load
    • The tape has perished beyond recovery.
  • VICE does display TAP file directory for halfwaves because the headers are stored in fullwave
  • c64 use - TAPSERV or COPY 235 (disk and tape)
  • Finaltap/CleanTAP will check tap image validity and give you loads of information on it - This only works on wholewave tapes (i.e. C64)
  • is there a command to copy ech file on tape to the disk
  • Tape Transfer | Commodore Info Page - A great overview of the process.
  • Tapes usually have the game recorded on both sides of the tape.
  • Commodore 16 tape imaging | offog.org - A real persons instructions on how he imaged his tapes.
  • Archiving C64 Tapes Correctly – pagetable.com - A article how this guy on his journey figured out how to image C64 tapes. It does have some technical aspects to it but is an easy read.
  • Accurately dump Commodore 64 tapes | PythonRepo - This includes a full process from research, to programming in Python and building a custom adapter. Diagrams and schematics are available.
  • Commodore 64 C64S Tape Adapter | YouTube
    • This homemade C64S Tape Adapter allows You to connect a C2N datasette to a PC running DOS and transfer data from or to Your C64 tape deck.
    • This video shows the guy using this nice looking adapter to capture TAP images and using tapeio.exe to check head azimuth.
    • I have tapeio.exe and although it is a C64 program it will allow you to set the head azimuth of your Commodore Datasette from a PC, see the video.
    • Links to all the required software
    • A great video.
  • Best or easiest way to transfer from tape - Commodore 64 (C64) Forum - I based my instructions on this.
  • RAW Audio and WAV
  • GitHub - francescovannini/truetape64:  - Accurately dump Commodore 64 tapes by building an inexpensive hardware adapter
  • awsm — Build your own 1530 to C16,C116,Plus/4 adapter and make an C64 SD2IEC work on your 264 computer
    • Build your own 1530 to C16,C116,Plus/4 adapter and make an C64 SD2IEC work on your 264 computer

Tape Formats

  • VICE Manual - 17 The emulator file formats - Very technical and includes the frequencies at which the various Commodore computer tapes were recorded at.
  • Tap file - Just Solve the File Format Problem - A basic description of a TAP file
  • The C64 file formats list | ist.uwaterloo.ca - A lot of detailed information.
  • How did the Cbm64/vic20 tapes work technically - Markus Brenner described the actually reading/writing process of Commodore tapes drives to help with the author of YAPE  implement TAP support in his Plus/4 emulator.
  • C64 RAW TAPE (.TAP) FILE FORMAT. - A simple breakdown.
  • The DC2N Project Homepage - Technical information - The DC2N format and other technical data.
  • zinc64/Analyzing C64 tape loaders.txt at master · binaryfields/zinc64 · GitHub
    • This explains Turbo Loaders and other technical details about the tape process.
    • A Turbo Loader is a piece of software that is loaded by the normal Commodore tape procedures and is then executed as any normal program. The function of the Turbo Loader is to read non-standard pulses on the tape which are put together a lot closer that normal tapes allowing them to be loaded up quicker and have a higher density of data on a tape.
  • How Commodore tapes work | wav-prg
  • Forget T64
    • it's not really a tape archive but more something like a file container. If you want real backups of tapes then use the TAP format.
  • TAP filesize depends on the amount of data not tape length, but it is a good indicator (longer tape, bigger TAP). Halfwave files are twice the size of a Wholewave tape.
  • TAP files come in the following flavours:
    • V0 - The intial format and is not used anymore.
    • V1 - This revision only supports Wholewaves and it referred to as Wholewave mode.
    • V2 - This supports Wholewaves and Halfwaves but is also referred to as Halfwave mode.
  • VICE (C64) will not load TAP V2 files
    • Rather than it being impossible it probably is a choice of the developers not to use TAP V2 for C64 as it is never needed.
    • You cannot see filenames in the files either as it porbably doe snot know how to search for headers in them.

Halfwave / Wholewave

  • There are 2 ways of recording tapes in the Commodore range and they vary how the signal is recorded on the tap.
    • Wholewave
      • All Commodore computers in the 264 range can read this format.
      • Example Wholewave Games: Number Builder
    • Halfwave
      • This format is only compatible with the C16 and Plus4
      • For the same tape, halfwaves files are twice as big
      • halfwaves TAPs are twice the size for the same length (time) of tape.
      • For the same length of tape, the halfwave files are twice as big
    • Example Halfwave Games: Aardvark, ACE, ACE2, Airwolf2, Beach Head, Bongo, Bridgehead +4, Demolition, Finders Keepers, Frank Brunos Boxing, Ikari Warriors, Legionnaire, Molecule Man, Paper Boy, Pharaohs Tomb, Sea Strike, Sky Hawk, Space Pilot, Steve davis Snooker, Strip Poker, Strip Poker 2, Summer Events, Task Force, Thrust, Tomcat, Who Dares Wins 2 and Winter Events.
    • Transfering C16 taps to real tapes - the horror, the pain... - Commodore 64 (C64) Forum
      • PTAP does not support halfwaves
      • PTAP options have vic20ntsc / vic20pal, but there's no option for c16, so i just used "ptap nameofthegame.tap"
      • The C16 and Plus/4 can use either half or whole (C64) waves
      • One can convert a half-wave TAP to a whole-wave TAP and record it at C16 frequency?
        • Short answer is that "halfwaves" can not be converted to "wholewaves". It's because the C64 tape input is level-triggered whereas the plus/4's is edge-triggered.
        • The long answer would be that some TAP images could indeed be converted to C64's native "wholewave" format (where an entire square wave pulse is encoded per TAP "unit"). Those loaders that do not invert (usually incidentally) the phase of the signal. This is depending on the actual loader: KERNAL and Novaload are generally fine, but later tape turbos will not work as wholewave TAP whatever we'd try. Most TAP images on Plus4world are in fact encoded in wholewaves, especially the old "classic" games, mostly because a lot of the archiving was already done when Markus figured that the plus/4 is just different in this respect.
    • "Half-wave" means that by one TAP byte we measure the length of one half of a square wave pulse, and the initial phase is "low" (for the same C64 TAP you would store the sum of the two subsequenting bytes on one TAP byte). While the C64 tape input is edge triggered (IIRC) the plus/4 does not have this and many custom tape loaders invert the phase of the tape pulse (probably incidentally). 
    • Captured King Size Volume 1 and 2 tapes with MTAP and TAPSERV but the programs will not load - Plus/4 World
      • My discovery on imaging tapes and finding out what Halfwaves were.
      • Lots of information here and a good thread to read if you are stuck.
      • The size difference for your Number Builder TAPs is normal. The reason for this is explained by the differences between the two TAP formats:
        • The fullwave/wholewave TAP format is the original format, and was originally designed for C64 emulators. Each byte in the file represents the duration of a complete low-high wave, which is how the C64 responds to tape pulses.
        • However, because the fullwave TAP format is designed to assume a low-high wave, it will not work for turbos which use inverted (high-low) waves. That's where the halfwave TAP format comes in. Each byte in a halfwave TAP file represents the duration of each half of each wave cycle independently.
        • Basically, the halfwave TAP format is simply a more precise method of dumping tapes. Any tape that can be dumped as a fullwave TAP can also be dumped as a halfwave TAP, and there are some tapes that can only be dumped as a halfwave TAP.
    • TAP -> WAV -> PC sound card -> cassette recorder -> datasette -> c16/+4 - Plus/4 World
      • this discusses halwaves
      • sampling from an audio sort
      • Physically modifying your Plus4 to play loading sound from the tape player and potential using this sound to set the azimuth of the heads.
    • Are turbo loaders required to use halfwave formats? = I think so.
    • Is halwave a type of turbo? = no, halwave wave is how the signal is stored on the tape.
    • How can I tell which tapes are using halfwaves? = Use 'Tape Wave Checher'
    • Does TAPSERV.PRG handle halfwaves? = no
    • Will TTAPSERV image C64 tapes, will it do PET tapes? = It will do C64 tapes, but might not do PET tapes
  • What is the default Tape wave format for the Plus4 - Forum (#44961) - Plus/4 World
    • The normal kernal save routine is wholewave. Anything that does not use a Turbo is wholewave.
    • Halfwave is only found with Turbo Loaders on C16/Plus4, but maybe only 75% of the Turbo Loaders out there actually use it.
    • Novaload is a wholewave for example, but stuff from say Anco are ALL halfwave. Something like that commercial Kingsize Turbo saving program was halfwave.
    • Halfwave was used to prevent copying or make it more difficult, but that just turned out to be puny. All halfwave did was make some prog's a bit more difficult to load if your tape deck was not aligned properly.

Hardware

Datassette Maintenance

You need to make sure your Datassette is running well so I will outline the things your should do.

Squeaking Tapes

  • If your tape makes a squeaking noise, stop what you are doing and read this section

I swapped my belts in my commodore 1531 drive and on a few cassettes I get a squeaking noise, the tapes then either really slow down or the tape player just stops turning but the play button does not pop up. I have cleaned the pinch roller with isopropanol alcohol.

This is a common issue with older tapes, and it is usually the tapes that cause this. These instructions assume your Datassette is in perfect working order (i.e. clean heads, working drive belt).

Cause

  • The tape inside the cassettes when they are made are impregnated with a lubricated (not visibly to naked eye etc..) to aid the tape running smoothly through tape players. Over time this lubricate dissipates and this lack of lubricant that causes the squeak.
  • In particular, the squeak is caused by the tape on the donour reel creating too much friction preventing the 'Pinch Roller' from pulling the tape through smoothly. When this happens the 'Pinch Roller' slips on the tape creating the distinctive squeak.
  • When a tape squeaks, where does the noise come actually from?
    • The tape between the sponge and the head? = No.
    • The pinch roller slipping on the tape? = Usually, as it’s trying to pull the tape through, but can’t get it off the reel.
    • Other = Can also be the cassette drive belt slipping on the pulley / capstan.

Solutions

  • Fast forward and rewind the tape a few times. This does not always work.
  • Turn the cassette player so the receiving reel is towards the ground.
    • This is not the same as turning the tape over.
    • This changes the abient fall direction of the tape and might re-wrap it better ad/or make less friction.
    • This will only make a small difference so don't expect miracles.
  • Try a different drive. Sometimes a drive might have a slightly more powerful motor which is just enough.
  • How to relubricate your stuck cassette tapes | YouTube | ANA[DIA]LOG
    • This explains squeaky tapes and potentially how to fix them.
    • In this video we learn how to restore the lubricant in audio cassette tapes and obtain perfect transportation free of wow and flutter issues and squeaky sounds!
    • This is not really a permanent solution, it is more to allow you to recover the data on the tapes.
  • Cassette tape cleaner - What, how & why? | YouTube | Techmoan - This video shows a tape cleaning machine. Perhaps build your own.
  • (Thanks to Dave @ www.dataserve-retro.co.uk)
    • Lubricate the tapes manually  
      • Me
        • I swapped my belts in my commodore 1531 drive and on a few cassettes I get a squeaking noise, the tapes then either really slow down or the tape player just stops turning but the play button does not pop up. I have cleaned the pinch roller with isopropanol alcohol.
        • Do you know what this could be?
      • Dave
        • That’s a very common problem.
        • It’s not a Datassette issue, it’s the tapes themselves (unless you know for certain that they were OK before the belt change).
        • For some reason, some tapes (a lot) tighten up with age to the point where cassette players struggle to pull the tape through and audibly complain.
        • I have tried quite a few ways of sorting this out over the years, lubricating the cassettes, cleaning them with head cleaner and lubricating with silicone, WD40 and PTFE lubricant sprays.
        • I did get some success with silicone lubricant which didn’t damage the tape and I thought all was well. However, after a few weeks on the shelf, I tested them again but they were tight again and wouldn’t load. It seems to be a terminal illness for them.
        • Will this spray be good for fixing tapes (temporarily of course)
    • Use a tighter belt?
      • Me
        • I am still playing around the squeaking tapes. I think the squeak is the drive belt slipping probably caused by the tape and in particular the clamping of the tape by the head into the spring loaded sponge. I will let you know. I thought it might of been the pinch roller but I am not sure. I am also going to try a tighter belt as this will be able to generate more friction on the spools and see if that solves my issue. Fastforward and rewind work fine. My goal is to sample my tapes not really to keep them as a going concern.
        • How tight should these rubber bands be on the tape player. I am just wondering if they are actually tight enough because if I hold one of the rollers (spinny things inside) with my finger and spin another, it is quite free. I am just wondering if I got a bum band or your suppliers are sending you some out of tolerance. Some on the internet say 80mm and another advert says 75mm band, maybe this is the issue. My tapes all fast forward and reverse with no issue.
        • I am also going to try a tighter belt as this will be able to generate more friction on the spools and see if that solves my issue. Fastforward and rewind work fine.
      • Dave
        • Be careful with a tighter belt, it will make the tape run at the wrong speed with the increased tension, the motor will struggle and they fail quite quickly when too tight.
    • Mould is a real problem on floppy disks, but not seen it very often on cassettes though.

Drive Emulators

  • Commodore 64 disk / tape emulation - Wikipedia - A big list of emulators of which most will work with the C16 and Plus4.
  • Partial Emulators (Parallel Port)
    • VC1541
      • Can run with a X1541 or XE1541 cable
      • Easy to setup
      • supports read and write
      • VC1541 disk drive not recognised/not found
        • Start order: Power PC on to DOS, turn Commodore Computer on, run VC1541
        • You might have to reboot both PC and the Commodore sometimes
        • Check physical connections. unplug and re-plug them, especialy with old cables the connections might not be perfect
      • If the XE adapter LED goes off upon accessing VC1541 with the Commodore computer
        • There is probably a bad connection.
        • I redid my solder joints on the adapter (cold joints) but also pay attention to the pin connections of the parallel port onto the board as these could be liable for failure.
    • vice1541EMU
      • Might need a special cable The 1541EMU cable (type 1) | Star Commander
      • I think it can use a XE1541 cable??
      • 1541EMU - C64-Wiki
        • The 1541EMU software emulates the internal hardware of the Commodore 1541 disk drive on a PC connected to the C64 by a designated cable. Because the actual 1541 hardware is emulated, even fastloaders can be supported.[1]
        • The 1541EMU software can run successfully only on PCs with real-time capability and single task operating systems. The requirements for emulating the 1541 disk drive are exceptionally hard. The software was developed on Pentium PCs, and timing can be an issue on newer computers.
      • 1541EmU: Demo | YouTube | Electronics Tinkerer
      • GitHub - ElectronicsTinkerer/1541EmU: Source code and eagle files for the 1541 Emulator
      • With this software you can use your PC computer as a disk drive for those 8-bit Commodore home computers that are equipped with serial bus (this includes for example C-64, C-128, VIC-20, Plus/4 and C-16). Instead of recognizing just the stock serial bus protocol and some of the drive commands, 1541EMU completely emulates the internal hardware of the Commodore 1541 disk drive. This means that even fastloaders are supported.
    • 64HDD
      • A serial bus and disk drive emulator
      • Will run with either X1541 or XE1541 cables
    • CBM-HD
      • CBM-HD is a project where a PC simulates one or more IEEE devices. If you are familiar with 64HDD, a PC simulating a 1541 drive, then you can consider CBM-HD as its IEEE equivalent. 
      • Emulates PET drives (IEEE): 3040, 4040, 8050, 8250
      • This requires a special interface
  • Hardware Drive Emulators
    • Pi1541
      • Pi1541 is a real-time, cycle exact, Commodore 1541 disk drive emulator that can run on a Raspberry Pi 3B, 3B+ or 3A+.
      • A powerful setup but can be pricey.
      • Can get them in mini 1541 drive cases
      • Unlike SD2IEC, Pi1541 emulates a 6502 and the two 6522s. Any code it is asked to run is run in a cycle exact way.
      • As Pi1541 can execute code on its emulated 6502 core it supports a vast range of fast loaders (games and demo scene) even copy protected originals.
    • SD2IEC
      • Very popular on eBay and you can get them in mini 1541 3D printed cases
      • SD2IEC supports a limited set of fast loaders by attempting to guess the fast loader from the code sent to it. SD2IEC will not, and cannot, execute the code, it just simulates the communication protocols. As a consequence only a small amount of popular fast loaders are supported.

Emulator

  • VICE
    • When you reset the emulator, the tape counter is reset, so to disable it:
      • If using the older Win32 interface, it's in Settings -> Datasette settings -> uncheck 'Reset Datasette with CPU'
      • If using the GTK interface, it's in Settings drop menu -> Settings -> I/O Extensions -> Tape port devices -> uncheck 'Reset datasette with CPU'

Tape Wave Checker (TAPE_WAVE_CHECKE.PRG) (a.k.a WAVECHECK)

  • Download here
  • Yes, there is an 'R' missing at the end because the filename exceeded the maximum number of allowd characters, which is 16.
  • A VERY experimental program for checking the waveform of a tape.
  • When you run it, you'll get a blank screen, and it will display flashing bars when reading the tape.
  • If the bars are white, it means the waveform is undetermined, this normally happens on leader sections.
  • If the bars are continuous green, then the waveform is normal (low-high).
  • If the bars are red, then the waveform is inverted (high-low), and will need a halfwave TAP.
  • If you get mixed white and coloured bars, or flashes of white bars, then it probably uses an asymmetric waveform (Darron Broad's turbo loader is an example of this). Again, this will need a halfwave TAP.
  • Also, try not to start playing the tape in the middle of a leader section, as it may show the wrong colour happy.

How it works

This software listens to a tape when you play it and visually displays the type of wave being read. This software works just aswell in an emulator with a TAP file.

  • The Start sceen
  • If it shows a blank screen when the tape is being played then not data is being read.
  • If the bars are white, it means it is an undetermined wave type, this normally happens on leader sections.
  • If the bars are continuous green, then the wave type is a: Normal (low-high) wave
  • If the bars are continuous red, then the wave type is an: Inverted (high-low) wave. This will need a halfwave TAP image.
  • If you get mixed white and coloured bars, or flashes of white bars, then the wave type is probably an: Asymmetric wave. This will need a halfwave TAP image.
     
  • Also, try not to start playing the tape in the middle of a leader section, as it may show the wrong colour.

Software

  • Tape Wave Checker (TAPE_WAVE_CHECKE.PRG) - See above
  • MTAP
  • TAPDancer - Play TAPs from PC
  • FinalTAP / TAPClean differences - Commodore 64 (C64) Forum
    • Hello, I am the author of Final TAP (SubChrist), I worked on it between 2001-2006 and released the source in 2006, the sources were picked up and worked on further by a team of TAP experts/enthusiasts and their project is TAPclean.
    • I honestly couldn't tell you exactly what the differences are as I havn't followed their developments too much but it's probably safe to assume they have added support for a few of the less common formats/loaders (this was their primary objective I think) + ironed out a few issues and ported it to other platforms.
    • The difference between TAP v0 and TAP v1 is all about the pauses (silence) in TAP files, TAP v1 uses a kind of run-length compression rather than storing long sequences of zeroes in the file (TAP v0).
    • ps. The preferred method of dumping cassette tapes to binaries is now through the use of the (Luigi di Fraia's) DC2N hardware which can create higher resolution dumps (not TAP files) than were previously possible with ie. the X1541 adaptor and ie. Marcus Brenner's MTAP software.
    • These high res dump files (from the DC2N) should never be edited/cleaned but TAP files can be generated from them.
    • Some other TAP information
  • [CSDb] - TapEx V1.7 by SLC (2021) - A tool for analysing, verifying and cleaning Commodore 64 tape images (TAP V0, TAP V1 and DC2N RAW files (16, 24 & 32 bit).
  • Commodore Software - Tape Utilities  
    • A great list of tools
    • Categories: Tape Tester / Alignment Tools, Tape Transfer Tools, Tape Turbo Loaders
    • Tree: Commodore 64 Software --> Utilities --> Tape Utilities
  • Crossplatform Transfer Utilities | Zimmers.net - A wealth of software here but might not be the most upto date software.

Basic Commands I might use

To load and show directory
LOAD"$",8
LIST

To load a PRG and execute it
LOAD"TTAPSERV.PRG",8
RUN
Published in Emulators
Read more...
Wednesday, 12 January 2022 20:16

Capture and convert DV tapes with chapters

DV tapes are now a legacy technology but because they are digital you can still get your videos back without data loss as long as the tapes and camera are in good working order. There are a couple of steps to go through to get these videos in a usable format.

Prerequisites

Setting Up Windows

  • Make sure you have a FireWire device installed in your Windows 7 PC, obviously.
  • Change to Legacy drivers
    • This is done because Windows 7 removed the DV functionality from the drivers that were present in Vista, however they are still accessible if you want them. The legacy driver has been completely removed in Windows 10, hence why we are using Windows 7. Vista is far too old to be using.
    • Go into the Device manager and locate your 1394 FireWire Drivers
    • Change this to the Legacy version
    • A reboot is probably not needed but it does not harm to perform one here.
  • Install all of the software above so it is ready when needed.

Notes

Capture the Videos (WinDV)

For the user of the camcorder (video camera) it was possible to hit a button on the camera that would put a marker on the video stream which was breakpoint (like a chapter). These breakpoints were also present when the user stopped and started the camcorder. WinDV captures the video stream but when it discovers one of these breakpoints it creates another file. Not all cameras add this marker. The files are labelled sequentially (by default) in time order.

  • Go into the Device manager and Change the FireWire drivers to the 'Legacy' version.
  • Connect your camera to the PC with the Firewire cable.
  • Turn your camcorder on.
    • Windows will find a new device and then new drivers for it are installed. It location indevice manage should be in `Imaging devices`:
    • The camera should be turned to the correct mode for data transfer if not already.
  • Put a cassette in the Video Camera
  • Now open WinDV and you are ready to go (i did use open as administrator, but i don't think it makes any difference)
  • Click Capture
    • when you click capture on WinDV it waits until it gets a signal from the camera to start recording, and when the tape is done will stop capturing.
  • Press play on the camcorder and the capture will begin

Notes

  • You are more than likely to get many files create but this is normal.
  • Windows 10 does not natively support DV Tapes anymore, it was removed. I think the last version that has it is windows 7
  • In windows 7 + vista you still need to change the drivers to legacy drivers
  • You can install the drivers in windows 10 by using some extracted (tutorials in my links)
  • Firewire cards don't always work because of incompatibilities
  • Using windows 7 or vista and WinDV is probably the easiest way to capture videos
  • Some software allows you to control the DV camera from the PC (possibly virtual dub)
  • WinDV - No audio on some scenes / Only 1 scene has no sound, the rest of the tape is fine.
    • There can be a mix of working and non working scenes, it does not have to be just 1, this description just makes it easier to explain.
    • Cause:
      • This is caused by recording over another video with a different sound format (12/16bit) and there being no defined scene definition, so WinDV never changes is sample rate/type hence the lost audio.
      • This is caused by the audio type changing while playback is occurring, and this is most likely because the user used the same tape in 2 different cameras and they had different settings. Changing the settings on a camera would also have the same effect.
        • WindDV will does not recognise the change in the audio type so audio is no longer recorded because the codec it is using no longer understands the audio stream.
        • This issue only occurs when you swap scenes and the 2 scenes have different audio encodings.
        • Video is unaffected because the video format is always constant and if the bit rate changes it does not matter. there are also key frames at the begining of every scene.
    • Solutions:
      • You have to record the scenes before and after the audio change separately and then later bring them back together but this time using the same codec and audio settings. This will involve transcoding.
      • Start capturing in the new section (to be sure, when audio of the new scene starts) so there is no failed format transition (Preferred)
      • Use audio cable to capture audio the old fashion way
      • Use another software (untested)
      • Try WinDV Type1 capture (untested)
    • Links
      • no sound in DV captures - VideoHelp Forum
        • Joining the discussion late, but starting the camera slightly before capturing is the solution I also found when I had a similar issue (no sound on some transferred DV). There's a known bug in WinDV that effects using the WinDV camera controls and not getting audio; confirmed by WinDV's developer.
      • WinDV - help in transferring Type-II AVI - VideoHelp Forum
      • No sound in winDV captures, DV-type 2 - VideoHelp Forum
        • It's probably a mismatch between audio sample rates. (edit: as I see you are now noticing.) DV can record 4 channels @ 32khz or 2 channels @ 48. Once you have "taught" windv what you are using it ought to be ok. If your tape switches back and forth you will need to restart the capture at each break. 44.1 is very unusual though technically it is in the specs.
      • [SOLVED] Strange problem with capturing MiniDV via firewire - missing audio - VideoHelp Forum
        • sometimes tapes were re-used and the audio settings had been changed. if the very beginning of the tape was used in 16 bit mode and later recorded over at the 12 bit setting windv or other transfer software will start in 16 bit mode and stay there even if the recording goes to 12 bit mode.
        • one way to get around it is to not start he capture on the computer until you hear audio from the recording you want. that way the software will start in the right mode.
      • WinDV, Audio capture issue - VideoHelp Forum - WinDV captures and separates all the dv video as desired. However, 2 of the 12 files do not have any sound (2 video segments). The 2 files are at random places in the dv tape. The 10 other files are perfectly fine.
  • No Sound at all
    • Causes:
      • Is the relevant codec on your PC missing?
      • Does your TV support the relevant codec?
    • Solutions:
      • Try VLC player
      • You could recapture the tape but it is unlikely to be this if other tapes sample fine
      • Install missing codecs
      • Try playing on another device

Correct DV Video Bitrates (AviDemux)

The video and audio formats must match otherwise errors will occur when we try and merge the videos with MKVToolNix. For the most part you will find your videos are all the same format but occasionally they might not be.

Checking the videos have the same format

Below are some options for checking if the video files are all in the same format:

  • Examine the files with MediaInfo (preferred)
    • Gives you complete readout of each files specifications and codecs it uses.
    • The information provided here will be helpful if you need to converts a video file to a matching format, as you will need to know the new format.
  • Use Windows explorer
    • Open the folder with the files in
    • Change to details view
    • Enable the bitrate column
    • Examine the results to see if all the bitrates match
  • You can try merging the videos and see if you get an error. It will do this before proceeding to do any real work.

Changing the Audio Bitrate (When required)

These instructions are just for changing the audio because this is most likely going to be the only issue you have

  • Open Avidemux
  • Open the file with the incorrect bitrate (File --> Open)
  • Under Audio Output
    • Change 'Copy' to 'PCM'
    • Click 'Filters'
    • Click 'Resampling (Hz) and change to the relevant bitrate (i.e. 48000)
    • Click 'OK'
  • Under Output Format
    • Select 'AVI Muxer'
  • Click 'Save'
  • You will be prompted to save the new video, select a name and location and click 'OK'
  • Repeat this for each video segment you need to re-sample.

These new files can now be used along with your others in the next section because they will all match.

Notes

Merging the video files while creating chapters (MKVToolNix)

The videos (and happy memories) have now been recovered into a digital format (corrected when needed) on your PC but are in a weird format which only VLC Player will play and they are still split in to multiple files, 1 for each chapter. If you only have 1 file then not all of this section will apply to you.

The files created by WinDV are labelled (by default) in time order so we can still use them to create chapters and this is one of the reasons for the multistep approach.

We will now use MKVToolNix to combine the video parts into one MKV file with chapters:

  • In the input window add the first video fragment/file.
  • Select the file you have just added in the input window and click 'Append Files'
  • Select all of the other files/chapters you want to add
    1. Set the view to details to sort by date/time, so do they start oldest first
    2. Now select them all (excluding the first file which is already added)
    3. They can now be added in one go, instead of one by one (unless you want too !!)
  • Set (Output Tab --> 'Generating chapters') = 'One chapter for each appended file'
  • Set the output location and file.
  • Now to run click on 'Start multiplexing'
  • This software will now create a single MKV file with chapters using the original uncompressed codecs, hence the large size.

Transcode to a Usable Codec (HandBrake)

The file we created in the previous section will play and have chapters in VLC Player but the codecs used are very old and not many Televisions or consumer kit will play the DV codec so we must now convert them into one they will use, however you just want to archive the files or play them on a PC you do not need to do this step as these files can be played with VLC player.

 

  • Prevent crashing tips
    • Do not control your transcoding over Remote Dsektop (RDP)
    • Set windows to the High Performance profile
    • Disable screen being turned off
    • Disable the screen saver
    • Set Windows to never sleep
    • Once encoding is started, leave your PC alone
    • limit the numbe rof threzs encding can use. This is set in `Advanced Options` on the `Video Tab`.
  • Otehr noteds
    • H.265 is not present on all new devices, but the number that support this improved codec is growing.
  • Target CODEC: I used HandBrake to transcode (convert) my video files (MKV) with great success. I use the following codecs:
    • Video: H264 - MPEG-4-AVC(part 10) (avc1) :: H.264 (x264)
    • Audio: MPEG AAC Audio (mp4a) :: AAC (avcodec)

 

 

 

Instructions

In the panels below make sure your settings match what is shown and that you read each step's notes.

The instructions are for PAL so if you are working with NTSC you will need to change frame rates and resolutions to the relevant settings.

We will also be converting the source from an interlaced format to a progressive format (i.e. 576i --> 576p, 720i --> 720p) while retaining the original Storage Aspect Ratio (SAR) and setting the Display Asprect Rartion (DAR) to 4:3.

First Field Selection (Top field / Bottom Field) - this section might not be keps because field detection is ustomati - the information here is still valid adn to be kept

An interlaced source has 2 fields per frame so when transcoding a video with interlacing, you need to know the field order. On a digital format you can check this with MediaInfo but they should be as follows:

  • VHS:  Top Field first
  • DVD:  Top Field First
  • DV:    Bottom Field First

I have not actually done this in my handbrake, but it is a concerns. Is this automatically detected? You can manually set the options int he advanced options on the `Video Tab`. To override you can select custom options

Source Selection

  • Open your file that you want to transcode.
  • Once you get the hang of things you can do batch conversions.

Summary Tab

  • Select the H.264 MKV 720pp30 preset
  • Set you output location

Dimensions Tab

  • The cropping is disabled because PAL and NTSC always rely on the frames being stored as one resolution, and then stretched to match the screen it is being displayed on which is why we have the `Anamorphic` option which controls the display resolution.
  • Leave `Anamorphic`on automatic as it always works as expected as you can see by the 4:3 Aspect Ratio.
  • We will not be using anamorphic as it is pointless nowadays and these files are not for archival purposes, we already have those generated in the previous section of this tutorial.
  • Target resolutions
    • PAL = 720 x 576
    • NTSC = 720 x 540
  • If you see black bars, these would of been cut of when played back as the CRT screens never showed the whole frame/image, but the image will always have a 4:3 ratio.

Filters Tab

  • Interlace Detection
    • All DV and VHS videos are interlaced, so detecting them is pointless.
  • Deinterlace
    • Yadif
      • I use this in OBS and the results are great.
    • EEDI2
      • This is a more powerful deinterlacer but takes a lot more time to process
    • Bob
      • When you select this option, the deinterlacer will look at the previous and next frame to help it build the current frame.
      • eg: If you select Yadif + Bob this is the same as Yadif 2x on OBS studio.
      • The name Bob comes from the fact that the scanner bobs between the odd and even lines.
    • Field order detection (Parity) - for Yadif and Bwdif and I think also for DeComb.
      • The default value is auto. If the interlacing is unknown or the decoder does not export this information, top field first will be assumed.
      • When dealing with digital sources (i.e. DV) the file will have a value in it's heading saying what field is first and this is read and used by HandBrake.
      • You should never need to set a custom Deinterlace option for this.
  • Colourspace:
    • BT 709 is the modern colour space and it makes sense to conver these at the same time. When dealing with digital you could elave them as is?
    • BT 709 is the recommended colour space for H.264

Video Tab

  • Video Encoder
    • 50 fps for PAL
    • 59.94 fps for NTSC
    • Because DV, VHS and TV all use constant frame rate, this is what you should use.
  • Quality
    • Constant Quality = CQP
    • RF 23 = is what I use in OBS for VHS capture and is a good level for video sources where there can be some graininess.
    • The RF level could possibly be increased for DV tapes, but you will need to experiment, I am happy with results I get with CQP 23.
  • Encoder Options
    • Encoder Preset
      • VerySlow is the best you should use. Placebo generates such large files for such little gain it is not worth using.
      • This set the preset flag for the H.264 encoder.
      • Each of the encoder's presets enable different preconfigured settings like those you can add in `Advanced Options`(such as look ahead and B frames) and inmost cases only selecting the best preset for you is all that is needed.
      • The presets do not directly control compression but because as you go up the presets, more features are enabled which in turn increases compression of the video.
    • Encoder Tune
      • Maybe set this to `Film` to combat some graininess (see h.265 link for better explanation)
      • Leave `Fast Decode` off, this is for old and low powered hardware that will struggle to play highly compressed files.
    • Encoder Profile
      • Auto = HandBrake will pick the best profile it thinks you setup will handle.
      • Baseline = The most compatible with the weakest compression.
      • High = Most CPU to compile, most CPU to decompress.
      • I don't know what `high422` and `high444` are.
    • Encoder Level
      • This controls the maximum compression but also what devices can play this file as they need to be able to play this level of file.
      • A “best settings” guide for Handbrake 0.9.9 and 0.10 | mattgadient.com
        • Higher profiles & levels tend to get you better compression (so better quality in a given filesize). However, you’re going to be limited by the profile support of the hardware devices you’re planning to play your videos on.
        • Currently, High Profile, Level 4.1 is the most popular profile on recent / cutting edge devices. Such a device will also play Baseline/Main, and any level between 1.0-4.0. The industry’s stagnated at Level 4.1 for a couple years, probably because it’s at the point where it’s “good enough” until H265 starts taking over.
      • Levels - Advanced Video Coding - Wikipedia
        • As the term is used in the standard, a "level" is a specified set of constraints that indicate a degree of required decoder performance for a profile. For example, a level of support within a profile specifies the maximum picture resolution, frame rate, and bit rate that a decoder may use. A decoder that conforms to a given level must be able to decode all bitstreams encoded for that level and all lower levels.
    • Advanced Options
      • nothing to add here because the `Encoder Preset` has done everything for us.

Audio Tab

  • These audio settings are the highest quality the DV format can go and are far better than VHS, so use these.
    • Bitrate / 384kbps / Stereo / 48kHz

Subtitles Tab

  • Nothing to do here.

Chapters Tab

  • Nothing to do be done here unless you want to name all of your chapters. The file/chapters can always be edited later without re-encoding.
  • Leave chapter markers on.

Start Encode

  • HandBrake will now create a new MKV file, maintaining the chapters.
  • This MKV will have a massively reduced filesize with the additional benefit that this file will play on most modern TVs and devices.
  • For me a 1 Hour tape (13.3GB) was about 1GB in size when converted.

Save your Preset (optional)

Now that you have entered your settings you might as well save them for future use.

  • Click on the `Save New Preset` button
  • Fill in your presets basic information

  • Audio is not stored in the main part of the preset so for some reason has to be set independently here other wise the Audio will be left as default when you load your preset.
  • Subtitles does not need anything changing.

Batch Convert

Make sure you have followed the instructions above at least once and you have saved the settings as a preset (e.g. 'DV Tapes').

  • Optionally set your default output folder before starting.
    • Open and HandBrake
    • Set folder here: Tools --> Preferences --> Output Files --> Default Path
      • The default folder is the `Videos` folder in your profile.
    • Close HandBrake
  • Open HandBrake
  • Select all of your files either by dragging or selecting a folder.
  • All of the files will be opened as titles.
  • Select your 'DV Tapes' preset which will apply it's settings to all of the titles.
    • However the output folder is not included and as such it will use the default output folder.
    • The input filename is used as the output file name but with the extensions changed to MKV (in our case).
  • Add to Queue --> Add all
  • Start Encode
  • Done, wait for the files to be transcoded.

 

 

 

Notes

My Camera Notes

Sony Digital8 CCD-TRV228E

JVC GR-DVL100E

  • JVC GR-DVL100 INSTRUCTIONS MANUAL Pdf Download | ManualsLib
    • View and Download JVC GR-DVL100 instructions manual online. JVC Digital Video Camera Instructions.
    • Page 34: 'Connection To A Personal Computer'
  • E04 - Safeguard Mode
    • How to Get a JVC Camcorder out of Safeguard Mode | It Still Works - JVC's digital video camcorders are sensitive and complicated electronic devices. They contain both powerful digital video processing chips and intricate parts that work together to focus and zoom the lens and handle the DV cassette that the camera uses for recording and playback. These cameras contain sensors that can detect issues in their operation. When they detect an issue, they can put the camera into a special "Safeguard Mode" that shuts it down until it can be reset.
    • JVC GR-DV800U Error Message Fix/Bypass - YouTube - Possible solution to JVC GR-DV800U error messages (E01,E02,etc.). It may possibly work for other JVC camcorders with similar error messages. This solution allowed me to retrieve several hours of DV tape footage from an old camera that is plagued by error messages. Now it is functioning normally with an occasional error which I suspect is caused by the tape tension sensor.
    • E04 Unit in safeguard mode | Pechorins
      • Q: I am using a GR-DVL-100 now for three years. I had no problems untill (I suspect) that I attached a Hahnel Lithium Ion battery (HL-408). Since I used this battery (as second battery device) problems started. Maybe it is coinsedence I do not know. It is good to know that more people have problems. I'll ask my lawyer if they come up with repair charges!Best regards,Perry
      • A: Here's what i found. I was able to clear the message by opening and closing and ejecting the cassette. I also had to play with the battery at the same time. Keeping the cassette door open and then pulling out the battery. After is did this a couple of times i got the message to clear. I too haven't used my camcorder for some time and now that the error message has cleared the camcorder tells me that i need to clean the heads. Hope this helps. Below is the message i found that pointed me toward ejecting the cassette. Good Luck, i'm off to buy a tape head cleaning cassette. An error indication (E03 or E04) appears. - (E03 or E04 UNIT IN SAFEGUARD MODE REMOVE AND REATTACH BATTERY) This massage a malfunction of some kind has occurred. In this case the camcorder's functions become unusable. - Eject the cassette once and re-insert it, then check if the indication clears. When it does, you can resume using the camcorder. If the indication remains, consult your nearest JVC dealer or service center  .
    • Re[2]: E04 Unit in safeguard mode | Pechorins
      • Q: Yeah, I have seen this problem 2 weeks ago after not using my GR-DVL100 for several months. The unit is about 5 years old. After I read comments on the topic, I went to buy the head cleaner. The problems seems to come on and off, but haven't given up trying. At least, the picture now show but with no sound, and at a bit faster speed. Look forward to see the end results.
      • A: Remove and reattach battery.
    • Re: E04 Unit in safeguard mode | Pechorins
      • Here's what i found. I was able to clear the message by opening and closing and ejecting the cassette.
      • I also had to play with the battery at the same time, keeping the cassette door open and then pulling out the battery. After is did this a couple of times i got the message to clear.
      • I too haven't used my camcorder for some time and now that the error message has cleared the camcorder tells me that i need to clean the heads. Hope this helps.
      • Below is the message i found that pointed me toward ejecting the cassette.
      • Good Luck, i'm off to buy a tape head cleaning cassette.
      • "An error indication (E03 or E04) appears. - (E03 or E04 UNIT IN SAFEGUARD MODE REMOVE AND REATTACH BATTERY) This massage a malfunction of some kind has occurred. In this case the camcorder's functions become unusable. - Eject the cassette once and re-insert it, then check if the indication clears. When it does, you can resume using the camcorder. If the indication remains, consult your nearest JVC dealer or service center."
    • JVC camcorder stuck in 'safeguard' mode? - UK Vintage Radio Repair and Restoration Discussion Forum
      • Q: hey gave me a JVC GR-DVL100E (European spec?) digital camcorder. I don't think it's been used in quite a while. When I power it up I get a blue screen with: "E04 - unit in safeguard mode", then it tells me to "remove and reattach battery". But doing that does not reset the camera and the messages persist. Any ideas chaps?
      • A: I've got a couple of JVC's camcorders, and I've seen this fault with them before. It's something in the tape drive getting stuck, on one of mine it was the eject mech jamming because of one of the gears was missing a tooth, to get around that I just have to give the tape some help in and out! On another one it was a damaged ribbon cable causing it, also in the tape drive. Try ejecting the tape mech and see if it's opening and closing correctly. Might just be because it's not been used for a while.
    • SOLVED: E04 unit in safeguard mode - Fixya
      • Q: When i turn the camera on i get in the screen that the camera is in ( E04 safeguard mode remove and reattach battery) i got like two years with this problem, in the beginning i just remove the battery and put in back and start and some times i just put the cable from the charger and work just fine, but lately that dont work i check the manual but dont said nothing about it other than remove the battery any suggestion in what can a do?
      • A:
        • I tried every method I could find from people posting on the web, but the method that worked for me is this.
          1. Eject the tape and close the tape holder.
          2. Turn the camera off and remove the battery.
          3. Put the battery back in. Leave the camera off.
          4. Open the tape holder.
          5. Slide the tape in, but as you press the holder in to accept the tape, keep your finger pressing in on the holder, being careful not to slow the tape holder from moving downward.
          6. Turn the camera on.
          7. If it doesn't work the first time, press more firmly to help the tape holder accept the tape.
        • That's it. This method seems to work for me and lets me use the camera for a long while. If the camera is sitting unused for more than a few hours, I have to do it again. And each time I insert a tape, I make sure that I keep pressing as the tape holder lowers the tape.
      • The thread has a few good ideas.
    • SOLVED: I am recieving error E04 "safeguard mode" - Fixya
      • Q: The camcorder is in safeguard mode and even when I remove the battery the camcorder remains in safeguard mode. How can I get the camcorder out of safeguard mode so I can use it again? 
      • A: The thread has a few good ideas.

Samsung VP-D361 - Digital Video Camcorder

  • Manual (PDF) - Not an official site.
  • SAMSUNG VP-D36 SERIES USER MANUAL Pdf Download | ManualsLib - View and Download Samsung VP-D36 Series user manual online.
  • VIA IEEE 1394 Card not Recognising Samsung VP-D361i - VideoHelp Forum
    • Q:
      • I have just purchased a IEEE 1394 Card - brand Ritmo. Windows detected it and installed and it is in device manger as VIA OHCI Compliant IEEE 1394 Host Controller and working properly.
      • My Samsung VP-D361i camera came with Ulead Video Studio V9 which I have installed but when I go into capture it will only find my ATI Radeon card and no other source. I have looked in device manager under sound and video game controllers and there is no reference to my camera. I also loaded the Samsung DV Media Pro Driver 10 from the Samsung site and still no good.
      • Can someone please let me know if I have a driver issue with the IEEE card or if it is my camera.
    • A:
      • When you check, have you connect the Cam to the Ritmo 1394, and turn on the cam and put it in playback mode?
      • You can also use WinDV ( from tool section ) to detect you DV cam. ( After connect the Cam to the Ritmo 1394, and turn on the cam and put it in playback mode. )
      • There are lots of compatibility problems with firewire cards. Try another card with a different chipset. 

 

--------------------------------------to sort and move to appropriate sections maybe------------------------------

  • Official Sites (HandBrake)
  • Official Documentation (HandBrake)
    • When looking at the documentation be aware there is documentation for each version number and Google/Bing just pick one. So my advise is too make sure the version in the URL is `latest` and does not specify a number like 1.7.0 which you can just replace with the word `latest`.
    • HandBrake Documentation - English documentation is currently available for the following HandBrake versions.
    • HandBrake Documentation — CLI Guide - The following details all the available options in the command line interface.
    • HandBrake Documentation — Anamorphic Guide
    • HandBrake Documentation — Summary of Filters
      • Colorspace + tonemap filter. Can change/tonemap the colorspace of the video into one of the following:
      • If this set to “Off” then HandBrake will keep the colorspace of the video.
    • HandBrake Documentation — Custom Presets
      • HandBrake comes with many built-in presets which cover the most common use case scenarios. However there are many scenarios and devices that many not be covered.
      • Audio and Subtitle settings are special. The preset does not store selected audio and subtitle tracks from the main window. It uses rules to automatically select them for each new source or title that you selected. You can modify these rules by editing the settings on the relevant “selection behaviour” screen.
  • Field Selection (HandBrake)
    • Using Yadif or Bwdif, if you want to specify which field is first you need to specify the mode and then the field to be first as follows: 
      ## Yadif/Bwdif format
mode=m;parity=p

## Yadif/Bwdif Bob with automatic selection
mode=7;parity=-1

## Yadif/Bwdif Bob with TFF
mode=7;parity=0

## Yadif/Bwdif Bob with BFF
mode=7;parity=1
      • Decomb has different options to (Yadif and Bwdif)
      • If You hover over the Custom input field it will show you the format.
    • HandBrake/libhb/param.c - L173-L216 · HandBrake/HandBrake · GitHub - Shows the modes and presets in the code.
    • Deinterlace looks great, but the frames are out of order | Reddit
      • If you hover over a field, there is a tooltip (most of the time) that explains the options.
      • For Yadif, if you select Custom as the preset, and hover over the box that appears, it tells you that Yadif has mode=m:parity=p options, so I then went into the activity log to see what option number I'd been using and plugged that in, then used 0 for one test and 1 for another test to see which would work on my video. From that one test I'm guessing that 1 is default, so try using parity=0 on the videos that weren't working for you.
    • add frame order setting to filters when using deinterlace or decomp · Issue #1928 · HandBrake/HandBrake · GitHub
      • This has a link to the presets values for the deinterlacers
      • It would seem you want to set mode=23:parity=p for bob, with p set to -1 for auto, 0 for top field first, or 1 for bottom field first.
    • Handbrake settings for DVD Decomb/Deinterlace? | AVS Forum - Custom Deinterlace options. These might be an old version.
  • Handbrake guides
  • HandBrake vs FFMpeg
    • HandBrake Documentation — Supported source formats
      • One of HandBrake’s strengths is its ability to open a wide variety of video formats. HandBrake uses FFmpeg libavformat and libavcodec libraries under the hood and generally can open whatever FFmpeg will, in addition to disc-based formats like DVD and Blu-ray.
    • What the difference between ffmpeg and software like Handbrake? | Reddit
      • Handbrake will do things to try and save the user from themselves. For example, if you resize the video to a size that is not within the original video’s aspect ratio, it will automatically correct and resize to a size that maintains aspect ratio. It may have updated to include a option to choose to not do this, but things like this are what make handbrake different. With Ffmpeg, you can see the automatic choices being made if you turn up its logging verbosity.
      • Handbrake uses ffmpeg's libraries under the hood. It's essentially a GUI frontend for ffmpeg. As such, it simplifies a lot of operations for the user, meaning that it's both much easier to use for newcomers but also much more restricted in its functionality.
      • Handbrake does have at least one feature advantage over FFmpeg. They have their own NLMeans filter which is multithreaded and therefore massively faster than FFmpeg's single-threaded one. They also have presets for the various filters they have, which I'd say counts as a feature advantage, I certainly could use such presets in FFmpeg.
    • [FFmpeg vs HandBrake] What’s the Difference and Which One to Choose? - Aiming to help users understand what transcoding program they want, this article reviews two well-known open-source software FFmpeg and HandBrake, and makes a detailed comparison between them in an FFmpeg vs HandBrake guide.
    • Handbrake vs ffmpeg | Reddit
      • Handbrake (in most cases) uses the same encoders on back-end that ffmpeg uses. H264 is encoded using a software library called x264, and HEVC uses a software library called x265. Handbrake just gives you a nice GUI and hides some advanced options available in ffmpeg.
  • Misc
    • Test output resolution
      • You can test the output resolution by
        • playing the video file in VLC Player
        • Taking a Snapshot: (Video --> Take Snapshot) which will go into your "My Pictures" folder.
        • Check the resolution of the Snapshot.
  • General
  • Maybe for VHS article
  • Digitising DV Tapes tutorials
  • Colour Space
  • Handbrake Deinterlace general
  • Handbrake - Which field first

 

Published in Media
Read more...
Tuesday, 11 January 2022 09:18

My Radiator Notes

Ecostrad, Beldray, Warm Home Heating are all made in the same factory because they all have the same specs, the advertising liturature is all the same and they all use the same APP to control them. This is my opinion.

 

Published in House
Read more...
Friday, 31 December 2021 16:42

My pfSense Notes

A pfSense setup guide by QuantumWarp covering most areas people will need to start using the router software on a regular PC. The notes also can be used for setting up pfSense on Netgate hardware.

  • I have disabled IPv6 and not used it in things such as the OpenVPN servers
  • This Tutorial has only been possible with the help of:
  • HeadingsMap Firefox Add-On
    • This plugin shows the tree structure of the headings in a side bar.
    • It will make using this article as a reference document much easier.

Preface

This tutorial will show you how to setup your pfSense router from scratch and only a small amount of networking knowledge is needed. All settings and research is provided and no stone has been left unturned.

This article can also be used as a reference document (along with Ctrl+F) when you cannot remember that one thing that you need to do.

As I discover new things I will add them here for my reference and so other people can see them

This tutorial covers the following:

  • Installing pfSense on your own hardware
  • Configure Internet Routing
  • pfBlocker - A Detailed setup including Blocklists.
  • DNS Hijacking - Force the use of specified DNS servers using encryption for privacy with DNS Hijacking in place to prevent unauthorised use of DNS/DoT/DoH on your network.
  • Port Forwarding and NAT Reflection
  • OpenVPN Server (TUN Routed network and TAP Bridged network)
  • OpenVPN Client Gateway (PrivacyVPN) - Using Policy Routing
  • Many other settings for pfSense are covered

My Setup Overview

The Software

Just for clarity, the version of pfSense I did these notes with is: 2.6.0-RELEASE (amd64) CE and pfSense configuration mostly the same as this article.

The Hardware

You can easily virtualise pfSense with PCI Passthrough for the quad NIC on TrueNAS using KVM which is what I do.

  • Lenovo FCKT99AUS
    • Low profile PC
    • Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
    • 4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
    • AES-NI CPU Crypto: Yes (inactive)
    • QAT Crypto: No
    • A single Realtek 1000MB onboard NIC
    • 8GB RAM
    • 120GB Kingston SSD
  • Dell 0YT674 0H092P Intel PRO/1000 VT Quad Port Gigabit Ethernet Adapter Card
    • Intel Low-Profile Bracket for E1G44ET, E1G44ET2, I340-T4, I350-T4 Quad Port
    • This is an old card
  • Cisco branded - Intel i350T4V2 with iSCSI NIC (UCSC-PCIE-IRJ45) Quad Port 1Gbps NIC
  • Openreach/Huawei EchoLife HG612 FTTC VDSL NTE modem

Using only the dedicated quad network card is better because you can utilise the onboard processing power of the card. These cards are always better at handling traffic because they are not a budget item bolted onto the motherboard.

Hardware Settings

  • PC BIOS
    • Make sure the power state is set to ON after a power cut.
      • This makes the pfSense function just like a normal router and maintains your network's connectivity by bringing the router backup as soon as there is power

Installation

  • I saw the the installer noticed the Virtualization technologies were disabled. I don't think it needs them, but it does not harm to turn them on.
  • Download the ISO
  • Partitioning

    • Choose Auto (ZFS) : Guided Root-on-ZFS
  • ZFS Configuration - Configure Options
    • Partition Scheme: GPT (BIOS+UEFI)
      • This creates an EFISYS partition on GPT hard disk (on my UEFI)
      • I had to turn on legacy booting (CSM) in my PC's BIOS to get this to work. Probably because the PC is old.
    • Swap Size
      • This should be twice the systems RAM. So if your PC has 8GB then the Swap Size should be 16GB.
      • This cannot be changed once pfSense is installed, you would need to re-install pfSense to reset this as it is a Swap partition and not a Swap file.
  • Pool Type/Disks



    • 1 disk striped (No Redundancy)
    • Notes
      • My selection is becasue I can easily swap the SSD and restore my backed up router config quickly. My settings do not change all the time and there is not data I am that bothered about and also a bit of downtime is not the end of the world.
      • Companies that cannot afford any downtime (mission critical) or loss of data such as logging data on the router should always run a mirror using at least 2 SSDs.
      • ZFS Partitioning | pfSense Documentation
  • When you start the installation
  • Complete
    • Click Reboot
    • Shell is for advanced users
  • pfSense will now load to the console (WebConfigurator)


At the Console (WebConfigurator)

  • Assign Interfaces
    • No VLANs
    • WAN, Autodetect, Plug the WAN (PPPoE modem) cable into the onboard network socket, Press enter
    • LAN, Autodetect, Plug the LAN cable into the top socket (port 0) on the Intel low profile NIC, press enter
    • Add no more
    • Accept settings
  • Set Interface(s) IP address
    • Set LAN
    • IPv4 = 10.0.0.1
    • Subnet: 24 (255.255.255.0)
    • No LAN IPv6 (enable at a later date)
    • Enable DHCP on LAN
    • Client address range = 10.0.0.100 --> 10.0.0.199
    • Do you want to revert to HTTP as the webConfigurator protocol?
      • Currently the web-server is using HTTPS and this is asking if you want to downgrade to HTTP
      • Select No

Setup Wizard via GUI (WebGUI)

  • If password does not work just after you set it using the setup wizard (common issue) then the solution is easy.
  • Login into pfsense GUI
  • Follow the pfSense Setup Wizard
    • Step 1 - Netgate® Global Support is available 24/7
      • Just click next.
    • Step 2 - General Information
      • Hostname: pfsense
      • Domain: mydomain.com
        gives: pfsense.mydomain.com
      • Primary DNS Server: 9.9.9.9
      • Secondary DNS Server: n/a
      • Override DNS: unticked
    • Step 3 - Time Server Information
      • Time server hostname: 2.pfsense.pool.ntp.org (default)
      • Timezone: GB (or what ever you require)
      • Seths these:
        • System --> General Setup --> Timezone
        • System --> General Setup --> Timeservers
    • Step 4 – Configure WAN Interface
      • Fill in details as required
      • DNS Server Override: Unticked
      • Block RFC1918 Private Networks: leave ticked
      • Block bogon networks: leave ticked
    • Step 5 – Configure LAN interface
      • LAN IP Address: 10.0.0.1
      • Subnet Mask: 24
    • Step 6 – Set Admin WebGUI Password
      • Make sure you use a complex one as shortly your router will be online
    • Step 7 – Reload configuration
      • Click ‘Reload’
    • Step 8 - Reload in progress
      • Just wait
    • Step 9 – Wizard Complete
      • Check for updates (optional)
      • Click Finish

Advanced/Further settings via GUI (WebGUI/webConfigurator)

Take your time going through these instructions as a small mistake can stop things working and can be potentially hard to find where you went wrong.

GUI Appearance

These settings are for the general use and appearance of the GUI.

  • Set Dashboard to 3 Columns
    • System --> General Setup --> webConfigurator --> Dashboard Columns: 3
  • Dark Theme
    • System --> General Setup --> WebConfigurator --> Theme: pfsense-dark
    • This is a much easier theme to use than the default brite white theme.
  • Disable the constant pfSense version checker on the Dashboard
    • System --> Update --> Update Settings --> Disable the Dashboard auto-update check: ticked
    • Releases are not that frequent
  • Show log entries in reverse order (newest entries on top)
    • Status --> System Logs --> Settings (tab) --> Forward/Reverse Display
    • This will make logs a lot easier to read.
  • Display page name first in browser tab (Optional)
    • System --> Advanced --> Admin Access --> webConfigurator --> Browser tab text = ticked
    • When this is unchecked, the browser tab shows the host name followed by the current page. Check this box to display the current page followed by the host name.
    • If you only have one router you might prefer this. I have left it as default for now.

Hostname and Domain

  • System --> General Setup
    • Hostname: pfsense (already done in wizard)
    • Domain: mydomain.com (already done in wizard)

Disable IPv6

This is done because I want to make sure I control all of my traffic and I don't fully understand IPv6.

The following instructions are partially based on this article (with pictures)

  • Disable IPv6 on PFSense | by Teri Radichel | Cloud Security | Medium
    • If you don€'t need IPv6 you can disable it to simplify network management
    • Whenever I post something about disabling IPv6 I get slammed by a bunch of IPv6 fans so bracing for it with this post.
    • I’ve already written that IPv6 can be implemented securely and if you need it, you can use it. Do you need IPv6? I wrote about that.
    • I also have written about how disabling it can simplify network management on a home network here.
    • A continuation of my posts on network security.
  • System --> Advanced --> Networking
    • Allow IPv6: unticked
    • Prefer IPv4 over IPv6: leave ticked
  • Services --> DHCPv6 Relay = unticked
  • Disable IPv6 on each of the interfaces (this will need to be re-run at the end of the tutorial for additional interfaces......)
    • Interfaces --> WAN (pppoe0) --> General Configuration --> IPv6 Configuration Type: None
    • Interfaces --> LAN (igb0) --> General Configuration --> IPv6 Configuration Type: None (already done in wizard)
    • Repeat for all other interfaces that this applies to.
  • System --> Routing --> Gateways --> Default gateway IPv6 = None
    • This might already be set to none.
  • Firewall rules to block IPv6
    • You probably don’t need this as well but I also create firewall rules to block IPv6 so I can tell if something isn’t working or gets misconfigured.
    • Firewall --> Rules --> LAN --> Rules --> Default allow LAN IPv6 to any rule: Disable (but don't delete as you might want it for experimenting with later)

I am still getting some IPv6 traffic which is getting blocked, however I think it is coming from my locally connected devices such as my Windows PC.

Manually set the default gateway

This might already be done and thus no changes required.

  • System --> Routing --> Gateways --> Default gateway
    • Default gateway IPv4: WAN_PPPOE
    • Default gateway IPv6: none

Monitoring IP for WAN

  • System --> Routing --> Gateways --> WAN_PPPOE --> Edit --> Monitor IP: 9.9.9.9
  • This just allows pfSense to make sure the WAN_PPPOE connection is up.
  • Not all Gateways respons to ICMP request so this gets around that issue.
  • This is needed to control what happens when your connection goes down.
  • You can use any reliable IP you want.
  • Make sure you check the IP responds to a PING manually so you know it will work for the monitor.
  • pfSense create a static route to 9.9.9.9 (I think this is hidden)

Custom WAN DNS Servers (Secure) (Forced)

  • System --> General Setup --> DNS Server Settings
    • DNS Servers:
      • (DNS Server: 9.9.9.9 / DNS Hostname: dns.quad9.net) (Primary DNS) (partly already done in wizard)
      • (DNS Server: 149.112.112.112 / DNS Hostname: dns.quad9.net) (Secondary DNS) (optional)
        • Dont add a Secondary DNS, this will make it harder to diagnose DNS issues and security later. The performance increase is negligible.
    • DNS Server Override: unticked (already done in wizard??)
    • DNS Resolution Behavior: Use local DNS (127.0.0.1), ignore remote DNS Servers
  • Services --> DNS Resolver --> General Settings
    • Network Interfaces: All
      • ALL is easier but on a high load system you might want to specify these.
    • Outgoing Network Interfaces: WAN
    • Strict Outgoing Network Interface Binding: ticked
    • DNSSEC: unticked
      • This option enables DNSSEC validation.
      • Quad9 does all of this upstream so we don't need too and waste CPU cycles. Quad9 will only ever return valid domains.
      • DNSSEC needs to be turned off because it just causes extra traffic. Since you're forwarding the request and the remote resolver already is using this. I verified this by running a test with DNSSEC turned off in pfSense.
      • This is option is not required to allow devices on my network to use Dig, Delv and other apps to make DNS requests to get responses with DNSSEC information.
      • Should I enable DNSSEC in pfSense when using Quad9 and full DNS HiJacking? | Netgate Forum
        • no, if you forward DNS, enabling DNSSEC on pfSense makes zero sense - either where you forward is doing it or they are not. You setting it in unbound is going to cause issue. Even Quad9 themselves tell you that.
      • Disasble DNSSEC - DNS Forwarder Best Practices - Quad9 Documentation
        • Disable DNSSEC Validation: Since Quad9 already performs DNSSEC validation, DNSSEC being enabled in the forwarder will cause a duplication of the DNSSEC process, significantly reducing performance and potentially causing false BOGUS responses.
    • Enable Forwarding Mode: ticked
      • DNS Resolver uses unbound and the old way of doing things was with DNS Forwarder powered by dnsmasq which could only forward DNS requests.
      • Controls whether unbound uses resolver mode (unchecked) or forwarding mode (checked). See DNS Resolver Mode for an explanation of the modes.
      • I need to put DNS Resolver into forwarder mode to utilise Quad9 blocking capabilities.
    • Use SSL/TLS for outgoing DNS Queries to Forwarding Servers: ticked
  • Services --> DNS Resolver --> General Settings
    • Query Name Minimization: unticked
    • Strict Query Name Minimization: unticked
      • This is unticked by default, see above.

Custom LAN DNS Servers

  • Services --> DHCP Server --> LAN --> Servers --> DNS Servers: 10.0.0.1
    • This makes sure the DNS servers given out over DHCP are not those configure in General settings but the one(s) we specify.

Enable DNS over TLS (DoT)

Using DoT on my local network is good because it can prevent MITM attacks on local DNS traffic which can happen if your network is compromised. Your DNS server will still be able to read the requests when it receives them as the encryption is only between pfSense and the client.

Enabling this allows pfSense to serve DNS over TLS (Port 853). It is not required for standard DNS to work as this is on port 53

  • Services --> DNS Resolver --> General Settings
    • Enable SSL/TLS Service: ticked
      • Configures the DNS Resolver to act as a DNS over SSL/TLS server which can answer queries from clients which also support DNS over TLS.
      • Activating this option disables automatic interface response routing behaviour, thus it works best with specific interface bindings.
        • This warning is about unbound's behavior when bound to multiple interfaces, especially bound to all. With that box active, it responds back to the client from the closest interface routing-wise for UDP. With it inactive, unbound is smart enough to always reply back to the client from the address to which the original query was sent. That behavior doesn't matter for most. Primarily things like DNS over IPsec tunnels are affected.
        • DNS Resolver Enable SSL/TLS Service - automatic interface response routing behavior | Netgate Forum
          • Some one was asking that question a while ago: Quad9 DNS-over-TLS setup with Unbound & forwarding in 2.4.4-RC | Netgate Forum
          • The answer was: The warning there is about unbound's behavior when bound to multiple interfaces, especially bound to all. With that box active, it responds back to the client from the closest interface routing-wise for UDP. With it inactive, unbound is smart enough to always reply back to the client from the address to which the original query was sent. That behavior doesn't matter for most. Primarily things like DNS over IPsec tunnels are affected.
    • SSL/TLS Certificate: WebConfigurator default
      • Clients may reject this certificate if it is self-signed, consider using a certificate from ACME.
      • Client on your local network performing a DoT request can usually be configured to not be strict with certificate validation and this will not affect operations.
      • It is always best to have a valid SSL/TLS certificate.
      • I have a separate Web Server that is taking all of the DNS and WWW requests and I dont want the hassle of setting up a fully verified cert on my router, i just want to enforce TLS/SSL. It is this Webserver that would be required to generate the SSL certificates or at least install the SCME wildcard TXT record.
    • SSL/TLS Listen Port: empty/853

Notes

Set DHCP Pool

  • Services --> DHCP Server --> LAN --> General Options --> Range: 10.0.0.100 – 10.0.0.199 (already done in wizard)
  • It did not seem to get setup correctly in the GUI, however pfSense was respecting this range. So perhaps a small GUI bug fixed by just re-saving the range here.

Automatic Hostnames

  • Services --> DNS Resolver --> General Settings --> (DHCP Registration) Register DHCP leases in the DNS Resolver: ticked
    • Note that this will cause the Resolver to reload and flush its resolution cache whenever a DHCP lease is issued.
    • Potential could put a high load on the DNS server, fine for small outfits (this is my interpretation)
  • Services --> DNS Resolver --> General Settings --> (Static DHCP) Register DHCP static mappings: ticked
    • This is so you statically mapped DHCP clients are registered (this will not break pfBlockerNG - Unbound Python Mode)

 

pfBlockerNG

This package make pfSense even more powerful and can block traffic based on IP, DNSBL and other metrics. It is highly configurable.

  • Packages — pfBlockerNG Package | pfSense Documentation
  • pfBlockerNG-devel is the Next Generation of pfBlockerNG.
  • Manage IPv4/v6 List Sources into 'Deny, Permit or Match' formats.
  • GeoIP database by MaxMind Inc. (GeoLite2 Free version).
  • De-Duplication, Suppression, and Reputation enhancements.
  • Provision to download from diverse List formats.
  • Advanced Integration for Proofpoint ET IQRisk IP Reputation Threat Sources.
  • Domain Name (DNSBL) blocking via Unbound DNS Resolver.

Install the package

  • System --> Package Manager --> Available Packages --> Search: pfBlockerNG 3 (3.1.0_4 at time of writing) --> Install

Run the Wizard (can be re-run)

  • Firewall --> pfBlockerNG
  • Step 1 (pfBlockerNG Components)
    • Click next
  • Step 2 (pfBlockerNG IP Component Configuration)
    • Select Inbound Firewall Interface: WAN
    • Select Outbound Firewall Interface: LAN
  • Step 3 (pfBlockerNG DNSBL Component Configuration)
    • VIP Address: 10.10.10.1
    • Port: 8081
    • SSL Port: 8443
    • IPv6 DNSBL: unticked
      • Lawrence Systems does not mention about this so leave as is
      • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Webserver Configuration --> IPv6 DNSBL
    • DNSBL Whitelist: ticked
  • Step 4 (pfBlockerNG Finalize)
    • Click Finish
    • pfBlockerNG has been successfully configured and updated. This installation will now block IPs based on some recommended Feed source providers. It will also block most ADverts based on Feed sources including EasyList/EasyPrivacy. Some additional Feed source providers include some malicious domain blocking.

Force Options

Firewall --> pfBlockerNG --> Update Settings

  • Running a 'Force' option
    • Select your 'Force' option (Update|Cron|Reload)
    • if you choose 'Reload'
      • Select 'Reload' option: All
    • Click 'Run' to start
  • View will show you the log
  • 'Force' Options explained
    • Update
      • This will process new changes and download new Alias/Lists.
      • This downloads and updates lists from the internet. This will trigger a reload if there are changes to the files.
    • Cron
      • This will download any Alias/Lists that are within the Frequency Setting (due for Update).
    • Reload
      • This will reload all Lists using the existing Downloaded files.
      • This is useful when Lists are out of "sync", Whitelisting, Blacklisting, Suppression, TLD or Reputation changes were made.
      • This rebuilds pfBlockerNG internal lists it uses for blocking, it does not download files from the internet.

Misc Settings

  • Remove the default DNSBL Whitelist entries
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Whitelist (at bottom of page)
    • Swap the default entries for 
      .pfsense.org
.netgate.com
      • If this causes any issues you can add exceptions backin. The default entries asre in the notes pfBlockerNG notes section towards the end of this article.
      • pfBlockerNG Default Whitelist - More info needed | Reddit
        • Could you delete them, sure. But they are like Google, YouTube, Dropbox, PBS, github and AWS; they were most likely put there as a failsafe though. If you don't like the whitelist, copy/save them and then titrate to what you want.
  • Firewall --> pfBlockerNG --> Reports --> Alerts --> Alert Settings --> Select the DNS server for the DNSBL Whitelist CNAME lookup
    • This is set by default to 8.8.8.8
    • Change this to 9.9.9.9
    • You are not able to select 127.0.0.1 here or your own custom DNS servers which is not secure.
    • I have reported this issue here: https://redmine.pfsense.org/issues/13200
  • Firewall --> pfBlockerNG --> General settings --> Download Failure Threshold: 2
  • Firewall --> pfBlockerNG --> DNSBL --> DNSBL IPs --> List Action: Disabled
    • This is disabled by default
    • When IPs are found in any Domain based Feed, these IPs will be added to the pfB_DNSBL_IP IP Aliastable and a firewall rule will be added to block those IPs.
    • I will leave this off for now because I don't know if it will benefit me. Most DNSBL lists do not have IPs in them.

IP handling

  • Firewall --> pfBlockerNG --> IP --> IP Configuration
    • CIDR Aggregation: ticked
      • This is CPU intensive when it builds the lists but you get better performance afterwards.
      • pfBlockerNG: What are De-Duplication and CIDR Aggregation for? : PFSENSE
        • BBCan177: That CIDR aggregation is for IP blocking. It has nothing to do with routing. There is no harm in enabling these options and is meant to reduce the number of entries in the firewall. They can however uses some CPU power to process and it's why it's an optional tunable setting.
    • ASN Reporting: Enabled - ASN entries cached for 24 hours
  • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration
    • Inbound Firewall Rules: WAN
    • Outbound Firewall Rules: LAN
      • As you add inbound interfaces, you might need to add them here e.g. (LAN/ROUTEDVPN/BRIDGEDVPN/OpenVPN etc..)
    • Floating Rules: ticked
      • Apply rules not specific to any interface
      • All rules appear in one place
      • Appear in the floating tab instead of separate (WAN/LAN/LAN2) tabs (Firewall --> Rules)
    • Kill States: ticked
      • If an IP appears in a Blocklist which you have live connections too, drop them
  • You need to Force Reload of settings for the floating rules to appear in floating tab instead of WAN and LAN tabs in the firewall rules (Firewall --> Rules)

Configure IP Reputation

  • Firewall --> pfBlockerNG --> IP --> Reputation
    • Individual List Reputation
      • Max: ticked
      • Max Setting: 5
    • Collective List Reputation
      • pMAX: ticked
      • pMax Setting: 50
      • dMAX: ticked
      • dMax Setting: 5
    • Country Code Settings (max/dMax)
      • ccwhite Action: Ignore
      • ccblack Action: Block
      • IPv4 Country Exclusion:  select countries you are interested in excluding, if any.

This might use a lot of CPU when compiling the modified lists but after that it is no extra performance cost

Maxmind GeoIP API Setup

This is required to be able to use the Maxmind API for our GEOIP lookups. There is no issue with signing up for a free account

  • Firewall --> pfBlockerNG --> IP --> MaxMind GeoIP configuration
  • Register and get a license key
    • GeoLite2 Sign Up | MaxMind
    • Account --> Manage License Keys --> Generate new license key
      • License key description: pfSense
      • Old versions of our GeoIP Update program use a different license key format. Will this key be used for GeoIP Update?: Yes
      • Select "Generate a license key and config file for use with geoipupdate version 3.1.1 or newer."
      • Click Confirm
      • Store the Key somewhere safe
  • Enter the settings
    • MaxMind License Key: xxxxx
    • MaxMind Localized Language: Your language
    • Check to disable MaxMind CSV updates: unticked
    • Click 'Save IP Settings'
  • All settings usually require a reload to apply them, so do this now.
  • Blocking outbound and Inbound are different
  • pfBlockerNG MaxMind Registration required to continue to use the GeoIP functionality! | Lawrence Systems

GeoIP Configuration (MaxMind)

  • Firewall --> pfBlockerNG --> IP --> GeoIP
  • By Default all lists are disabled.
  • You need to edit each list as required including selecting enable and then save
  • Action: Deny Inbound
    • stops spammers coming in but allow your traffic out to wherever your want.
    • You can stop outbound traffic if you want.

Apply Block Rules to Inbound and Outbound IPv4 traffic

These rules are probably created by pfBlockerNG. I dont know if you can set the direction in pfBlockerNG, eitherway make sure the IP lists act in both directions.

  • Firewall --> pfBlockerNG --> IP --> IPv4 --> [PRI1] --> Action: Deny Both
  • Do for each group

As you add groups in the future, always select 'Deny Both'

View/Delete Blocklists

This is for reference only.

  • They are in different locations for the different type of list
  • Firewall --> pfBlockerNG --> IP --> IPv4 --> [PRI1] --> Edit
  • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups [Name] --> Trash Can

Add a Blocklist Feed (IPv4/IPv6/DNSBL)

This is for reference only.

  • Firewall --> pfBlockerNG --> Feeds
  • Click on a desired list
  • Change the state to ON
  • Save Settings
  • (Optional) Firewall --> pfBlockerNG --> IP --> IPv4
    • Enable the relevant group and click save
    • Might already be on
    • NB: The added list will be assigned to it's relevant group
  • A file pole is required because we need to get the newly specified file
    • Firewall --> pfBlockerNG --> Update --> Select 'Force' option: Upload
    • Click 'Run' for the changes to apply

Add Alienvault IPv4 Blocklist

I have added this here as an example and not necessarily as a recommendation.

  • Lawrences add this one and it is by AT&T
  • Firewall --> pfBlockerNG --> Feeds
  • Client the 'Add' icon on the 'Alienvault' line
  • Set 'State' to On
  • Leave everything else as is
  • Click 'Save IPv4 Settings'
  • Goto Firewall --> pfBlockerNG --> IP --> IPv4
  • PRI2 Collection --> Action --> Deny Both
  • Click Save

Add Easylist DNSBL

I have added this here as an example and not necessarily as a recommendation.

  • The EasyList filter lists are sets of rules originally designed for Adblock (adblock.mozdev.org on WayBackMachine (archive.org)) that automatically remove unwanted content from the internet, including annoying adverts, bothersome banners and troublesome tracking.
    EasyList is the primary filter list that removes most adverts from international webpages, including unwanted frames, images and objects. It is the most popular list used by many ad blockers and forms the basis of over a dozen combination and supplementary filter lists.
  • Firewall --> pfBlockerNG --> Feeds
  • Client the 'Add' icon on the 'EasyList' line
  • Set
    • DNSBL Source Definitions --> State: On
    • Settings --> Action: Unbound
  • Leave everything else as is
  • Click 'Save IPv4 Settings'
  • Goto Firewall --> pfBlockerNG --> IP --> IPv4
  • PRI2 Collection --> Action --> Deny Both
  • Click Save

Add a Custom DNSBL Blocklists (Eg DoH)

This is for reference only.

  • Firewall --> pfBlockerNG --> DNSBL --> Add
  • See 'Add a Custom DoH DNSBL Blocklist into pfBlockerNG' below

DNSBL Category (Web Category Blocking)

Firewall --> pfBlockerNG --> DNSBL --> DNSBL Category

Not everyone will want to set this feature. This will allow you to block websites of a certain category and this data is currently supplied by 2 feeds:

  • Shallalist
    • Dead but still in pfBlockerNG.
  • UT1
    • The University of Toulouse Capitole has been broadcasting a blacklist of URLs, managed by Fabrice Prigent in order to allow better control of the use of the Internet. This database, which is widely used by schools, can be integrated into a large number of free or commercial tools, in addition to other lists.

The settings are straight forward on this page and do not need further instructions.

 

DNS Hijacking

Clients can make their own connections to direct connections to DNS servers, so block them on TCP/UDP ports 53 and 853 to ensure the clients only query the pfSense DNS Resolver. We also have to block DNS requests sent over HTTPS (DoH) which is harder to do.

This block of rules need to be at the top of your Floating rules and below pfBlockerNG rules if they are present. To make sure your BNS is actually Hijacked.

DNS and DoT

  • Add the following floating rules in order - These rules will work on every interface you select and only need to write them once
    • Blocking External Client DNS Queries | pfSense Documentation
    • Firewall --> Rules --> Floating
    • Allow Web Server DNS Queries (Port 53) (only needed when you are running your own DNS server for hosted websites)
      • Action: Pass
      • Quick: ticked
      • Interface: LAN (this is the network your Web server is on)
      • Direction: any
      • Address Family: IpV4
      • Protocol: TCP/UDP
      • Source: any
      • Destination: LAN net (You can specify the actual Web Server IP for more security here i.e. 10.0.0.13)
      • Port: DNS (53)
      • Description: Allow Web Server DNS
    • Allow Web Server DNS over TLS (DoT) (Port 853) (only needed when you are running your own DNS server for hosted websites)
      • Action: Pass
      • Quick: ticked
      • Interface: LAN (this is the network your Web server is on)
      • Direction: any
      • Address Family: IpV4
      • Protocol: TCP
      • Source: any
      • Destination: LAN net (You can specify the actual Web Server IP for more security here i.e. 10.0.0.13)
      • Port: DNS over TLS (853)
      • Description: Allow Web Server DoT
    • Allow Local DNS Queries (Port 53)
      • Action: Pass
      • Quick: ticked
      • Interface: LAN
      • Direction: any
      • Address Family: IpV4
      • Protocol: TCP/UDP
      • Source: any
      • Destination: This Firewall (self)
      • Port: DNS (53)
      • Description: Allow Local DNS
    • Allow Local DNS over TLS (DoT) (Port 853)
      • Action: Pass
      • Quick: ticked
      • Interface: LAN
      • Direction: any
      • Address Family: IPv4+IPv6
      • Protocol: TCP
      • Source: any
      • Destination: This Firewall (self)
      • Port: DNS over TLS (853)
      • Description: Allow Local DoT
    • Block DNS Queries (Port 53)
      • Action: Reject
      • Quick: ticked
      • Interface: LAN
      • Direction: any
      • Address Family: IPv4+IPv6
      • Protocol: TCP/UDP
      • Source: any
      • Destination: any
      • Port: DNS (53)
      • Description: Deny DNS
    • Block DNS over TLS (DoT) and DNS over QUIC (DoH) (Port 853)
      • Action: Reject
      • Quick: ticked
      • Interface: LAN
      • Direction: any
      • Address Family: IPv4+IPv6
      • Protocol: TCP/UDP
      • Source: any
      • Destination: any
      • Port: DNS over TLS (853)
      • Description: Deny DoT / DoQ

Notes

  • Redirect DNS Traffic on port 53
    • Rather than redirecting, I prefer to block unwanted traffic.
    • Redirecting Client DNS Requests | pfSense Documentation
    • Instead of dropping all of the DNS requests we can re-route them through our secure DNS chain.
    • This has the benefit of your can see all of the requests.
    • Redirecting DoT (853) requests would have issues with the certificates not matching up. So this only worked for standard DHCP
    • You need one rule for Ipv4 (127.0.0.1) and one for IPv6 (::1)
    • This option might be better for some IoT devices by not hard blocking DNS requests but just sending to my router. IoT might not resend a DNS request, however do you want these sorts of devices sending dodgy DNS requests.
    • This will grab all traffic on port 53/853.
  • Floating Rules
    • Floating rules are run on the WAN, then then port forward rules are applied (NAT), then the floating rules are re-proccesed for the LAN. This is why there is a rule for the Web Server DNS in the floating rules, it needs to run before the other floating rules otherwise your Web Server will not resolve DNS requests.
    • Firewall — Rule Methodology | pfSense Documentation - Floating rules are run first but only applied first if the 'Quick' option is ticked

DoH Blocking

  • Firewall --> pfBlockerNG --> DNSBL DNSBL SafeSearch --> DNS over HTTPS/TLS Blocking --> DoH/DoT Blocking: Enable
    • Select all domains

Canary Domain (FireFox Only)

Add Inbuilt DoH Feeds

  • Firewall --> pfBlockerNG --> Feeds
    • Scroll down to find the DoH group
    • Add the DoH group
      • Click on the + icon on the right handside of the DoH group line (in grey). This will add all of the feeds in the group.
      • Enable all feeds by changing their State to On
      • Change the Action to Unbound
      • Leave all the other settings the same
      • Click 'Save DNSBL Settings'
      • It should look like this when you are done:
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups
      • Move DoH group to the top of the list and save

These lists will not be added until you Update and reload your Feeds.

Add Inbuilt DoH IP Feeds

Generally DoH is handled by domain names but there are some exceptions such as 9.9.9.9 and 1.1.1.1 so this list is a good additional for those edges cases and possibly softwarte getting around traditional DoH blocking by not using domain names. Ignore the group DOH_6 as this is for IPv6, unless you need to block IPv6 addresses.

  • Firewall --> pfBlockerNG --> Feeds
    • Scroll down to find the DoH group
    • Add the DoH_IP group (currently only has 'TheGreatWall_DoH_IP' feed)
      • Click on the + icon on the right handside of the DoH_IP group line (in grey). This will add all of the feeds in the group.
      • Enable all feeds by changing their State to On
      • Change the Action to Deny Both
      • Leave all the other settings the same
      • Click 'Save IPv4 Settings'
  • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups
    • Move DoH_IP group to the top of the list and save

These lists will not be added until you Update and reload your Feeds.

Add a Custom DNSBL DoH feed into pfBlockerNG (dibdot)

  • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups -->Add
    • Info
      • Name: Custom_DoH
      • Description: Custom DoH Blocklist
    • DNSBL Source Defintions
    • Settings
      • Action: Unbound
      • Update Frequency: Weekly (daily for more active lists)
      • Weekly (day of Week): Monday
      • Auto-Sort Header field: Enable auto-sort
      • Group Order: Default
      • Logging / Blocking Mode: DNSBL WebServer/VIP
      • TOP1M Whitelist: unticked
    • Advanced Tuneables
      • Leave as is
    • DNSBL Custom_List
      • Leave as is
    • Click 'Save DNSBL Settings'
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups
      • Move to the top of the list and save

These lists will not be added until you Update and reload your Feeds.

Add a Custom DoH IP feed into pfBlockerNG (dibdot)

  • Firewall --> pfBlockerNG --> IP --> IPv4 -->Add
    • Info
      • Name: Custom_DoH_IP
      • Description: Custom DoH IP Blocklist
    • IPv4 Source Definitions
    • Settings
      • Action: Deny Both
      • Update Frequency: Weekly (daily for more active lists)
      • Weekly (day of Week): Monday
      • Auto-Sort Header field: Enable auto-sort
      • Enable Loggin: Enabled
      • States Removal: Enabled
      • TOP1M Whitelist: unticked
    • Advanced Inbound Firewall Rule Settings
      • Leave as is
    • Advanced Outbound Firewall Rule Settings
      • Leave as is
    • Advanced Tuneables
      • Leave as is
    • IPv4 Custom_List
      • Leave as is
    • Click 'Save IPv4 Settings'
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Groups
      • Move to the top of the list and save

These lists will not be added until you Update and reload your Feeds.

Test DNS Hijacking

DNS and DoT

The test will be similiar for DNS and DoT but the ports will be different.

  • Check if you are using Quad9
  • DNS Leak Test
    • I think a DNS leak test resends DNS requests mant times and analyses the result to see the different reponding IP addresses. These addresses hide behind lets say the 9.9.9.9 which is an entry point to an AnyCast network where a group of servers respond as a swarm.
    • You should only see servers from your provider and if running a VPN those from your VPN provider only. Otherwise this shows that the DNS traffic is not all going to the same place, a DNS leak.
    • eg: My laptop has Wifi and ethernet, this test can see if the DNS is going out of the wrong interface or even if it is being hijacked by malware
    • DNSLeakTest.com - DNSleaktest.com offers a simple test to determine if you DNS requests are being leaked which may represent a critical privacy threat. The test takes only a few seconds and we show you how you can simply fix the problem.
    • DNS Leaks Tests | Strong Technology - DNS leak is a security flaw, which can be used by your ISP or DNS server provider to log your activity, collect statistics, block access to some domains, or other purposes. Even though you are using encrypted VPN service with DNS leak your privacy is at risk.
    • Quad9 is on an AnyCast network so 9.9.9.9 traffic is subrouted to many servers throughout the world so you will see many IPs and servers but only from your choosen provider, In my case Quad9 provided by WoodyNet or PCH and others.
  • DNS Spoofing Test (GRC.com)
  • DNS Benchmark (GRC.com)
    • GRC's | DNS Nameserver Performance Benchmark - DNS Nameserver Performance Benchmark
    • All dns servers should be blocked except 10.0.0.1
    • To verify the internet connection, DNS Benchmark tries to access the internets Root DNS Server. It should fail this and say the internet is not working because we have denied access to these servers.
    • Continue by ignoring the error.
    • Nameserver resolution will now occur using your computers Primary DNS server (10.0.0.1) to resolve the nameservers hostnames, it will fail to resolve any (or most) of the DNS servers because we have DoH protection in place.
    • Run the Benchmark (optional)
    • So, the DNS Benchmark should fail and only have 1 server listed as online.
  • Telnet
    • This is a simple one. If you run the test below on windows you should get a response (black screen) which indicates there is a server present and that you can access it. If there is no response then you can assume the request is blocked.
    • telnet 8.8.8.8 53
    • Do this test on a few DNS servers jsut for an overview.
  • NSLookup
    • This test on windows is a good indicator but I dont know how to change the port so only standard DNS on port 53 is available for test.
    • nslookup bing.com
nslookup bing.com 8.8.8.8
nslookup bing.com 10.0.0.1
  • Packet Capture and WireShark
    • Diagnostics --> Packet Capture
    • Using Packet capture and WireShark you can analyse the DNS traffic to see if correct blocking is happening.
    • Study the WAN and LAN interfaces, for posts 53 and 853.
    • There should only be traffic out of the WAN to 9.9.9.9 on port 853
    • Traffic to any address other than 10.0.0.1 on port 53 should be getting blocked and you can tell this by the request not having any response and this traffic not going out of the LAN.
DoH

Do DoH testing before adding DoH_IP blocklists

Blocklist Research

  • Don't add a load of lists in one go. Add in blocks or one by one so you can make sure you don't get any adverse effects by blocking too much.
  • Make sure the lists you use are not too big otherwise they will break pfBlockerNG and pfSense will probably stop routing. If this happens there will be an error generated and you can figure out which of your lists is too large.
  • Everyone's list will probably be different

Blocklists

Notes

  • NSFW = Not safe for work
  • pfBlockerNG Guide - zenarmor.com (was sunnyvalley.io)
    • IPv4 Category feeds are divided into five groups(PRI1-5). These PRI groups are Known Ransomware, malware, botnets, Command & Control (C&C) servers, bots, web scripts, phishing & compromised servers, malicious IP's found attacking SSH, SMTP, IMAP, TELNET, FTP endpoints and other known originators of malicious behavior. In general, the lower the number, the more pfBlockerNG tries to avoid false positives.
    • You may enable different DNSBL feeds as you wish on your pfBlockerNG by following the next steps. Here, we will enable EasyList group feeds on our pgBlockerNG as an example. We also recommend you add the Steven Black feed is one of the best-maintained blacklist databases on the internet.
    • EasyList is the primary filter list that removes the majority of advertisements from international webpages, as well as unwanted frames, images, and objects. It is the most commonly used list by many ad blockers and serves as the foundation for over a dozen combination and supplementary filter lists.
    • Open your favorite browser and enter the domain name that you added to the Custom DNSBL list. It is dnsbltest.com for our example.
  • Blocking... or trying to... DNS over HTTPS (aka DoH) - Thiago Crepaldi
    • This post is complementary to a previous POST protecting your network from malicIous DNS. Here we are going to leverage a recent addition to pfBlockerNG: a brand new DoH feed! What is the big deal in allowing DNS over HTTPS (aka DoH) on your network?! Well, users can bypass the DNS over TLS of your...
  • Block DNS over HTTPS (DoH), using pfsense | jpgpi250 | GitHub - PDF covers a lot of stuff and has some useful information.
    Block DNS over HTTPS (DoH), using pfsense | jpgpi250 | PDF
  • User-defined script (sh or python) on a downloaded Feed
    • Add new Feature to execute a user-defined script (sh or python) on a downloaded Feed.
    • Useful to process Amazon Web Services IP Range feed for specific Regions.
    • When you edit a feed you can run a 'Pre-process Script' which currently has scripts in it to download Amazon AWS server IPs in particular regeions. Not exaclty sure how to use these.
    • AWS IP address ranges - AWS General Reference - Lists the IP address ranges for AWS.

 

Enable Auto Config Backup (Free service)

Auto Configuration Backup automatically encrypts configuration backup content using the Encryption Password below and then securely uploads the encrypted backup over HTTPS to Netgate servers.

  • Get your 'Device key' and store it somewhere safe
    • Services --> Auto Configuration Backup --> Backup Now --> Device key
  • Services --> Auto Configuration Backup --> Settings
    • Enable automatic configuration backups: ticked
    • Backup Frequency: Automatically backup on a regular schedule
      • If you edit your config a lot on one day, and knowing that 'Auto Config Backup' has a limit of the last 100 configs, you should set a daily backup schedule as I have.
    • Schedule:
      • These settingsfollow the Cron format and will perform a daily backup at 00:20 every morning.
      • Minute (0-59): 20
      • Hours (0-23): 0
      • Day (1-31): *
      • Month (1-12): *
      • Day of week (0-6): *
    • Encryption Password: Make this very complex and then back it up somewhere safe
    • Hint/Identifier: Something human readable and unique
    • Manual backups to keep: 20
  • Run a backup now
    • Services --> Auto Configuration Backup --> Backup Now --> Backup
  • [pfSense] Making automatic backups with AutoConfigBackup – Provya
  • This Stores the last 100 configs at Netgate hashed by your encryption Key
  • This only backs up the basic configurations, not the extended information of pfSense.

Enable SSH

  • Instructions
    • System --> Advanced --> Admin Access --> Secure Shell --> Enable Secure Shell: ticked
    • Username: root
    • Password: same as you admin password
    • Protocol: SFTP over SSH
    • NB: This will not be available on the WAN unless you setup firewall rules (i am guessing)
  • Guides

Port Forwarding

This is for reference only.

Allow WAN Ping (Optional, but useful)

  • Firewall --> Rules --> WAN --> Add
    • Action: Pass
    • Protocol: ICMP
    • ICMP subtypes: Echo request
    • Source: any
    • Destination: This Firewall (self)
    • Description: Allow Ping
  • Notes

Allow WAN IGMP (Optional)

NAT Reflection (Enable Globally)

  • Accessing Port Forwards from Local Networks | pfSense Documentation
    • System --> Advanced --> Firewall & NAT --> Network Address Translation
      • NAT Reflection mode for port forwards: Pure NAT
      • Enable NAT Reflection for 1:1 NAT: unticked
        • This is not needed for normal NAT Loopback.
        • I am not sure what this is for really except I don't currently needed it.
        • This might solve a problem that I am not having.
      • Enable automatic outbound NAT for Reflection: ticked

Notes

 

VLANs

These are easy to setup and dont really need an explanation

Notifications (Email)

  • System --> Advanced --> Notifications --> Email
  • These are important so you are notified about events. I have not found where you can configure specific events.
  • Fill in as below when using a CWP server on your local network (with NAT Reflection).

    • FQDN (server.mydomain.com ) does not work so you must use its local IP address (192.168.1.11).
    • You cannot Validate the SSL/TLS certificate.
  • pfSense email notification when fallback WAN connection goes down | cyberciti.biz - Explains how to configure pfSense email notification with cloud based smtp server or old good Google gmail smtp server to send notification e-mails.

Add additional Widget(s) to the Dashboard

  • Click on the + at the top right and select a widget
  • I have on my dashboard
    • Column 1
      • Gateways
      • System Information
    • Column 2
      • Services Status
      • Thermal Sensors
      • Disks
      • S.M.A.R.T. Status
      • pfBlockerNG
    • Column 3
      • Interfaces
      • Traffic Graphs

OpenVPN Server

  • Virtual Private Networks — OpenVPN | pfSense Documentation
  • You can install both types of OpenVPN server at the same time. This lets you play with both types, but can just do one if you want. Both types have their merits.
  • You cannot use 192.168.70.x as your IP range or even 192.168.x.x because of Android Hotspots. See notes below for more details
  • We will use UDP for the servers but you can easily change this to TCP when you need to. TCP will ensure data integrity but will come at the cost of speed.
  • TUN vs TAP
    • TUN
      • Is the modern and more secure method for VPN and should be your first choice.
      • This is more secure because all traffic is routed and thus can be controlled by the pfSense router.
    • TAP
      • Is a legacy method of running a VPN.
      • It will connect you into your network as if you were there, including having a an IP address from the local network, and sending/receiving broadcast traffic which is useful for legacy apps and network discovery.
      • Routing of UDP broadcasts can now be done my a pfSense package UDP Broadcast Relay, so this is one less reason to use this OpenVPN method.
    • OpenVPN TAP works, but cant access any services on the router | Netgate Forum
      • jimp (netgate) - TAP is not the most efficient connection. It is pointless to broadcast traffic over a vpn unless there is specific actual requirement for such a connection and also it adds overhead to every packet.
    • BridgingAndRouting – OpenVPN Community
      • For a brief introduction on bridging and routing
      • This discussion needs to start with TAP vs TUN devices.
      • Benefits and drawbacks of TAP and TUN networks.

OpenVPN TUN Server (RoutedVPN) - Using the wizard

  • Tutorial: pfsense OpenVPN Configuration For Remote Users 2020 | Lawrence Systems
  • This is the preferred method for clients accessing your network.
  • You can re-run the wizard to make changes
  • Yes, it does skip some steps (some configurations don't need all of the steps)
  • This is the more secure method of VPN because you can take advantage of network routing.
  • The client will get an IP from 10.200.1.0/24 range (not 10.200.1.254 - This is for the gateway)
  • Your VPN clients are not on the same subnet as your LAN clients so you might need to do some additional steps (outlined below) to get clients to talk to each other but this is normal.
  • RoutedVPN will still allow access to local network assets but with no broadcast traffic passing between networks and is better for security.
  • This connection can be tied down because it's traffic will always be routed between 2 or more network segments on the router.

Run the Wizard

  • VPN --> OpenVPN --> Wizards
  • Step 1 - Authentication Type
    • Select an Authentication Backend Type --> Type of Server: Local User Access
  • Step 6 - Add Certificate Authority (CA)
    • Descriptive name: OpenVPN
    • Key length: 2048 bit
    • Lifetime: 3650
    • Country Code: your details
      • This is capital sensitive, so do in ALL Caps i.e US, GB
    • State or Province: your details
    • City: your details
    • Organization: your details
    • Randomize Serial: ticked
      • This option might not be present as it depends on your version of pfSense.
      • If this option is not present perform the workaround at the end of these steps.
  • Step 8 - Add a Server Certificate
    • Descriptive name: OpenVPN
    • Key length: 2048 bit
    • Lifetime: 398
    • Country Code: your details
    • State or Province: your details
    • City: your details
    • Organization: your details
  • Step 9 - Server Setup (Only change settings mentioned)
    • General OpenVPN Server Information
      • Protocol: UDP on IPv4 only
      • Local Port: 2000 (don't use the default port for security)
      • Description: RoutedVPN
    • Cryptographic Settings
      • Hardware Crypto: Intel RDRAND engine - RAND (you might have different options or none)
    • Tunnel Settings
      • Tunnel Network: 10.200.1.0/24
      • Redirect Gateway: ticked
        • Force all client generated traffic through the tunnel.
        • Should all traffic go into your network and then back out or only what traffic is required to go in your network?
        • See video for further info
        • If unsure, ticked is the best option
      • Local Network: 10.0.0.0/24
    • Client Settings
      • DNS Default Domain: mydomain.com
      • DNS Server: 10.200.1.1 (you can use 10.0.0.1)
  • Step 10 - Firewall Rule Configuration
    • Traffic from clients to server
      • Firewall Rule: ticked
      • Firewall --> Rules --> WAN --> OpenVPN RoutedVPN Wizard
    • Traffic from clients through VPN
      • OpenVPN rule: ticked
      • Firewall --> Rules --> RoutedVPN --> OpenVPN RoutedVPN Wizard
  • Step 11 - Finished!
    • Click 'Finish'
    • You have now been redirected to
      VPN --> OpenVPN --> Servers

Randomize Serial not present in the wizard - Workaround

This is only needed if the option was not available above. These instructions will enable the missing option in the OpenVPN CA and then regenerate the OpenVPN Certificate but with a random serial.

  • System --> Cert. Manager --> CAs --> Certificate Authorities --> 'OpenVPN' --> Edit
    • Trust Store: unticked
      • I have not used this option. It implies that it will install the CA into an OS when it is downloaded, maybe this is a flag to ask the OS to install it.
      • The default for this is off, I thinnk.
    • Randomize Serial: ticked
      • This is to ensure the certificate's serial is random.
      • 'Next Certificate Serial' value will be ignored when using randomized serials
    • Click save
  • System --> Cert. Manager --> Certificates --> Certificates --> 'OpenVPN' --> 'Reissue/Renew
    • When this certificate was setup using the wizard, serial randomisation was not enabled because that option was not present in the wizard. Running this will generate a new certificate with a random serial.
    • Don't worry about revoking as this certificate has never be used or sent out in to the wild.
    • Leave everything as is
    • Click 'Renew/Reissue'

Fine Tuning

  • Edit your server you have just created
  • Endpoint Configuration
    • Local port: 2000
      • check this is correct as sometimes the wizard makes mistakes.
      • If it is wrong, check the WAN Firewall rule (OpenVPN RoutedVPN wizard)
  • Cryptographic Settings
    • Peer Certificate Revocation list: pfSense Certificate Revocation List (CA: OpenVPN)
      • I have set a revocation list, dont know how to use it but this setting needs to be on to be able to revoke certificates.
    • Hardware Crypto: Intel RDRAND engine - RAND (or your supported engine)
    • Strict User-CN Matching: ticked
    • Client Certificate Key Usage Validation: ticked
    • OCSP Check: unticked
      • Leave this off as you have to specifiy an external server.
  • Tunnel Settings
    • Redirect IPv4 Gateway: ticked
    • Redirect IPv6 Gateway: unticked
    • Concurrent connections: 1 (Only I am connecting)
    • Inter-client communication: ticked
      • This option controls whether or not connected clients are able to communicate with one another. To allow this behavior, check the option. When unchecked, clients can only send traffic to the server or destinations beyond the server such as routed networks or the Internet.
      • Typically in remote access style deployments it is unnecessary for clients to reach each other, but there are use cases when it can be helpful. One example is remote web developers working together and running test servers on their local workstations. With this option activated, the developers can reach the other self-hosted test servers for collaborative development.
  • Advanced Client Settings
    • DNS Default Domain: ticked + mydomain.com
    • DNS Server enable: ticked
    • DNS Server 1: 10.200.1.1 (you can use 10.0.0.1)
    • Block Outside DNS: ticked
    • Force DNS cache update: ticked
    • NetBIOS enable: unticked
  • Advanced Configuration
  • Click Save
  • Services --> DNS Resolver --> General Settings --> OpenVPN Clients: ticked
    • This will register the certificate name and not the PCs name (i think) i.e. user.mydomain.com - In which case make the cert match the PC name and there will be no issue.

Assign OpenVPN interface

  • Assigning OpenVPN Interfaces | pfSense Documentation
  • It is needed to allow the assigning of DHCP addresses and for routing.
  • Assigning this Interface allows you to monitor traffic coming in on that connection on the dashboard which is very useful
  • The procedure for assigning an OpenVPN interface is covered in Assigning OpenVPN Interfaces.
  • Manually set the default gateway
    • This should already be done earlier in this tutorial
    • System --> Routing --> Gateways
      • Default gateway IPv4: WAN_PPPOE
      • Default gateway IPv6: none
  • Navigate to Interfaces --> Assignments
    • Available network ports: ovpn1 (RoutedVPN)
    • Click Add
  • Navigate to Interfaces --> OPT1 (ovpns1)
    • Enable: ticked
    • Description: RoutedVPN
    • Click 'Save'
    • Click 'Apply Changes'
  • VPN --> OpenVPN --> Servers --> Edit --> RoutedVPN
  • Click Save to refresh the VPN configuration and restart its process

Firewall rules

  • Firewall --> Rules --> WAN
    • OpenVPN RoutedVPN
      • This rule is for the OpenVPN server and allows the traffic through the WAN
      • There should already be a rule created by the OpenVPN wizard called 'OpenVPN BridgedVPN Wizard'
      • Edit it as follows (default is 1194 but I have changed mine after I ran the wizard for security)
        • Edit Firewall Rule
          • Action: Pass
          • Interface: WAN
          • Address Family: IPv4
          • Protocol: UDP
        • Source
          • Invert match: unticked
          • any
        • Destination
          • WAN address
          • From - OpenVPN (2001)
          • To - OpenVPN (2001)
        • Description: OpenVPN BridgedVPN
  • Firewall --> Rules --> BRIDGEDVPN
    • These rules allow the flow of traffic over the ROUTEDVPN interface.
    • Add/Edit them in order show here
    • Default allow ROUTEDVPN to any rule
      • There should already be a rule created by the OpenVPN wizard called 'OpenVPN RoutedVPN Wizard'
      • Edit it as follows
        • Edit Firewall Rule
          • Action: Pass
          • Interface: ROUTEDVPN
          • Address Family: IPv4
          • Protocol: Any
        • Source
          • Invert match: unticked
          • ROUTEDVPN net
        • Destination
          • any
        • Extra Options
          • Description: Default allow ROUTEDVPN to any rule

pfBlockerNG

  • Assign ROUTEDVPN as an 'Outbound Interface'
    • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration

Convert RoutedVPN into an Internet Only Secure Connection (with Kill Switch) (Optional)

  • You can still access the pfsense box through its IPs such as 10.0.0.1 and probably it's domain name.
  • Deal with the access problem by restricting the admin interface. Dealt elsewhere in this article.
  • By manually assigning a gatway, the internal routing table of pfSense is ignored/not used so routing to other assets is not possible.
  • This useful for mobile phones and devices out in the real mode that just need a secure and private connection to the internet and not access to any of your local assets.
  • You can setup another VPN specifically for this type of network profile, but here we are going to repurpose the current Routed VPN network as it is easy to turn this feature on or off.
  • Obviously if you do this you will not be able to access any pfSense network assets.
  • This still allows the assignment of clients on the Routed VPN network to the Privacy network because the 'Clients to be routed over PIA' is triggered first if the clients IP is defined in the alias.
  • You will be able to see your locally hosted webservers if
    • They are specified in a port forward rule on the WAN interface. This allows NAT reflection to take place.
    • You specify the WAN as per these instructions.

This conversion will force all RoutedVPN traffic through the WAN and prevent communication with the pfSense local network which will give the following benefits:

  • For the Remote Device
    • Secure DNS (pfSense's DNS Server only)
    • A clean network to access the internet
    • Private connection - Traffic is hidden from the device's local environment
  • pfSense Network
    • Local clients cannot be compromised by malware on the remote device

Interface Rule (Add the Tag)

This rule selects the WAN Gateway and adds a tag to the OpenVPN packets where the source is from the 10.200.1.0/24 network (ROUTEDVPN net).

This rule needs adding to every interface where you want to control which clients are routed through the Privacy VPN or the default gateway (WAN) by the 'Route_Out_Over_PIA' alias. Obviously just change the interface.

  • Firewall --> Rules --> RoutedVPN --> Add
    • Action: Pass
    • Interface: OpenVPN
    • Address Family: IPv4
    • Protocol: Any
    • Source:
      • Invert match: unticked
      • ROUTEDVPN net
    • Destination: any
    • Description: Route the Routed VPN Network over WAN (Internet Only)
    • Click 'Display Advanced'
    • Advanced Options
      • Tag: WAN_ONLY
      • Gateway: WAN_PPPOE
    • Click Save
    • Click 'Apply Changes'
    • Move the rule below 'IP's to be routed over PIA'
      • This allows the privacy policy to still work

Floating Rule (Checks for the Tag and blocks the traffic)

On the WAN interface, block all Privacy VPN traffic that attempts to go into the WAN by matching any packets on the WAN that have a 'Private_VPN_ONLY' Tag.

  • Firewall --> Rules --> Floating --> Add
    Firewall — Floating Rules | pfSense Documentation
    • Action: Block
    • Quick: unticked
    • Interface: OpenVPN
    • Direction: any
    • Address Family: IPv4
    • Protocol: Any
    • Source:
      • Invert match: unticked
      • any
    • Destination:
      • Invert match: ticked
      • WAN address
    • Log: ticked
    • Description: Only Allow WAN Gateway (Kill Switch)
      • This rule can be re-used for other interfaces by just including them.
    • Click 'Display Advanced'
      • Tagged:
        • Invert: unticked
        • Tagged: WAN_ONLY
    • Put this rule below your 'Privacy VPN' rules.
  • You can turn this kill switch mechanism of by just disabling the interface rule 'Route the Routed VPN Network over WAN (Internet Only)'. You don't need to delete it.
  • The floating rule 'Only Allow WAN Gateway (Kill Switch)' currently allows for access to your web server and other local assets via NAT Relection. If you do not want this routing, just change the 'Destination' settings to:
    • Invert match: unticked
    • any
  • This method does not stop access to your router i.e. 10.0.0.1

OpenVPN TAP Server (BridgedVPN) - Using the wizard

  • Bridging OpenVPN Connections to Local Networks | pfSense Documentation
  • This method is better for site-to-site connections because it allows broadcasts and uses the same IP range.
  • The examples in most other OpenVPN recipes are routed using tun interfaces which operate at layer 3 and are generally the best practice. OpenVPN also offers the option of using tap interfaces, which operate at layer 2 and support bridging clients directly onto the LAN or other internal network. This can make the remote clients appear to be on the local LAN. This is a tap VPN.
  • This method is what most newbies want, a remote connection to their network sharing the same subnet.
  • Either Run the wizard again with the changes below or just copy the 'OpenVPN server' from above and make changes, not forgetting you need a relevant firewall rule.
  • The client will get an IP from 10.0.0.0/24 range which is the same as your LAN, infact the IP address will come from your DHCP pool.
  • BridgedVPN is a raw connection to your network which allows broadcasts to flow between the LAN and the VPN clients.
  • The TAP VPN method is an old way of doing things but has its uses.
  • It is probably a less secure method than RoutedVPN because of the open pipe to your network.
  • This connection can be tied down, but not as much as RoutedVPN.

Run the Wizard

Follow the setup wizard settings above (RoutedVPN), but with the following changes below

  • Step 9 - Server Setup (Only change settings mentioned)
    • General OpenVPN Server Information
      • Local Port: 2001 (don't use the default port for security)
      • Description: BridgedVPN
    • Do the fine tuning as set out above

Re-Configure The OpenVPN Server for Bridging mode

  • VPN --> OpenVPN --> Servers --> Edit: Your OpenVPN Server
  • Mode Configuration
    • tap - Layer 2 Tap Mode (this will change the options later in the page)
  • Tunnel Settings
    • IPv4 Tunnel Network: empty (i.e. remove 10.200.1.0/24)
    • Bridge DHCP: ticked
    • Bridge Interface: LAN
      • Your LAN interface (or whatever interface you want remote clients to exit onto)
      • Do not select a bridge interface here. You should not have one configured anyway.
      • It is correct to use 'LAN net' here because the clients on BridgedVPN will be sharing the same IP range and DHCP pool.
    • Bridge Route Gateway: unticked
    • Server Bridge DHCP Start/End:
      • Optional
      • Allow you to set what IP addresses are used for DHCP on the OpenVPN connections
      • These will not show up in your DHCP leases so the normal DHCP Server is bypassed.
      • Start: 10.0.0.190
      • End: 10.0.0.199
    • Advanced Client Settings
      • DNS Server 1: 10.0.0.1
      • NetBIOS enable: ticked
      • Node Type: b-node
      • Scope ID: leave blank

NetBIOS

  • I have enabled NetBIOS on the BridgedVPN because I have (and use) NetBIOS on my network.
  • Enabling this makes for a more complete bridge.
  • Modern Windows systesm use WSD for device discovery and MAC use mDNS, so this might not be needed anymore for the more modern network.
  • OpenVPN - NetBIOS Options | pfSense Documentation
  • Windows Settings - Configuration | OpenVPN.net - You can configure Windows-specific settings to enable/disable NetBios for connected clients. These settings don’t affect clients installed on other operating systems.

Assign OpenVPN Server to an Interface

  • Assigning OpenVPN Interfaces | pfSense Documentation
  • The VPN interface must be assigned before it can become a bridge member. The procedure for assigning an OpenVPN interface is covered in Assigning OpenVPN Interfaces.
  • Manually set the default gateway
    • This should already be done earlier in this tutorial
    • System --> Routing --> Gateways
      • Default gateway IPv4: WAN_PPPOE
      • Default gateway IPv6: none
  • Navigate to Interfaces --> Assignments
    • Available network ports: ovpns2 (BridgedVPN)
    • Click Add
  • Navigate to Interfaces --> OPT2 (ovpns2)
    • Enable: ticked
    • Description: BridgedVPN
    • Click 'Save'
    • Click 'Apply Changes'
  • VPN --> OpenVPN --> Servers --> Edit --> BridgedVPN
  • Click Save to refresh the VPN configuration and restart its process

Create Bridge between LAN and BRIDGEVPN

  • Bridged OpenVPN clients also receive broadcast and multicast traffic which can greatly increase the amount of traffic passing over the VPN.

You should not create an interface on this bridge. It is not needed and might cause you issues down the line and you do not need to assign it an IP.

Firewall Aliases

Before we make our firewall rules we need to create some alias that will be used by them.

  • Firewall --> Rules --> Aliases --> Add
  • Broadcast Sources
    • Name: Broadcast_Sources
    • Description: Broadcast packet sources
    • Type: Network(s)
    • Network(s)
      • 0.0.0.0/8       -    self-identification (RFC 6890)
      • 10.0.0.0/24    -    LAN / BridgeVPN Network
  • Broadcast Destinations
    • Name: Broadcast_Destinations
    • Description: Broadcast packet destinations
    • Type: Network(s)
    • Network(s)
      • 224.0.0.0/24               -    Non-Routable Multicast address range
      • 239.255.255.250/32    -    Simple Service Discovery Protocol address
      • 239.255.255.253/32    -    Service Location Protocol version 2 address
      • 255.255.255.255/32    -    IPv4 Broadcast address
  • Make sure you 'Apply Changes'
  • These aliases are a collection of single IPs and subnets, but both have to be declared as networks for the aliases to allow both types.
  • The 0.0.0.0/8 is a special case, even thought it is treated as a single IP it is infact is the IPv4 complete network space.
  • Where I have used /32, this means a single IP address
  • Later on if you use these in pfBlockerNG you will find it's advanced rules will only accept aliases of Network(s) type or, Port(s) where appropriate.

Firewall Rules

  • Firewall --> Rules --> WAN
    • OpenVPN BridgedVPN
      • This rule is for the OpenVPN server and allows the traffic through the WAN
      • There should already be a rule created by the OpenVPN wizard called 'OpenVPN BridgedVPN Wizard'
      • Edit it as follows
        • Edit Firewall Rule
          • Action: Pass
          • Interface: WAN
          • Address Family: IPv4
          • Protocol: UDP
        • Source
          • Invert match: unticked
          • any
        • Destination
          • WAN address
          • From - OpenVPN (2001)
          • To - OpenVPN (2001)
        • Extra Options
          • Description: OpenVPN BridgedVPN
  • Firewall --> Rules --> BRIDGEDVPN
    • These rules allow the flow of traffic over the BRIDGEDVPN interface.
    • Add/Edit them in order show here
    • Allow DHCP Broadcasts from LAN
      • Edit Firewall Rule
        • Action: Pass
        • Interface: BRIDGEDVPN
        • Address Family: IPv4
        • Protocol: UDP
      • Source
        • Source
          • Invert match: unticked
          • Single host or alias
          • 0.0.0.0
        • Source Port Range
          • From: (other)
          • Custom: 68
          • To: (other)
          • Custom: 68
        • Destination
          • Destination
            • Invert match: unticked
            • Single host or alias
            • 255.255.255.255
          • Destination Port Range
            • From: (other)
            • Custom: 67
            • To: (other)
            • Custom: 67
        • Extra Options
          • Description: Allow DHCP Broadcasts from LAN
    • Allow Broadcasts from LAN
      • Edit Firewall Rule
        • Action: Pass
        • Interface: BRIDGEDVPN
        • Address Family: IPv4
        • Protocol: Any
      • Source
        • Source
          • Invert match: unticked
          • Single host or alias
          • Broadcast_Sources
        • Destination
          • Destination
            • Invert match: unticked
            • Single host or alias
            • Broadcast_Destinations
        • Extra Options
          • Description: Allow Broadcasts from LAN
        • Advanced Options
          • Allow IP options: ticked
    • Default allow BRIDGEDVPN to any rule
      • This allows traffic to work normally as if were on the LAN interface directly.
      • LAN net
        • has been used as the source because the clients will be on the same network range as the LAN.
        • is an alias for 10.0.0.0/24
      • There should already be a rule created by the OpenVPN wizard called 'OpenVPN BridgedVPN Wizard'
      • Edit it as follows (default is 1194 but I have changed mine after I ran the wizard for security)
        • Edit Firewall Rule
          • Action: Pass
          • Interface: BRIDGEDVPN
          • Address Family: IPv4
          • Protocol: Any
        • Source
          • Invert match: unticked
          • LAN net
        • Destination
          • any
        • Extra Options
          • Description: Default allow BRIDGEDVPN to any rule

Notes

  • The broadcasts are restricted to LAN/Bridge/BridgedVPN because:
    • broadcasts are not routable, hence why we need the bridge.
    • the firewall rules above have source and destination defined.
    • firewall rules are applied on an inbound basis.
    • LAN has its own hidden rules for handling broadcasts
  • Why use 'LAN Net' on the BRIDEGEDVPN interface
    • BRIDGEDVPN does not have a network assigned to each so you would not be able to match LAN clients with 'BRIDGEDVPN net'.
    • 'LAN net' is an alias for the network 10.0.0.0/24
  • Why is 'IP options' enabled on 'Default allow BRIDGEDVPN to any rule'
    • IP Options | pfSense Documentation
      • Checking this box will allow packets with defined IP options to pass. By default, pf blocks all packets that have IP options set in order to deter OS fingerprinting, among other reasons. Check this box to pass IGMP or other multicast traffic containing IP options.
    • Allow packets with IP options to pass. Otherwise they are blocked by default. This is usually only seen with multicast traffic.
    • We are bridging 2 networks so there should be no blocks on the same subnet.
  • DHCP 0.0.0.0 is a special case so needs it's own firewall rule
  • The LAN interface has it's own hidden rules to handle broadcasts which you cannot see.

pfBlockerNG

  • Assign BRIDGEDVPN as an 'Outbound Interface'
    • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration

Reboot the Router

  • I had many issues with OpenVPN (TAP) and pfBlockerNG, the solution seems to be a proper reboot.
  • If you have issues with the OpenVPN (TAP) after this, check below for more advanced troubleshooting.

Restrict Access to OpenVPN Server(s) (optional)

It is not advisable to leave your OpenVPN servers open to everyone. You should restict access where possible and these are the instructions to do this task.

  • Create an alias with the following details
    • Firewall --> Aliases --> Add
    • Name: Allowed_Remote_Clients
    • Description: Allowed remote clients
    • Type: Host(s) / Network(s)
    • Hosts --> IP or FQDN
      • Add your IPs here that you want to allow access to your OpenVPN servers.
  • Edit the OpenVPN server NAT rules
    • Firewall --> Rules --> WAN -->
      • OpenVPN RoutedVPN
      • OpenVPN BridgedVPN
    • Source
      • Invert match: unticked
      • Single host or alias
      • Allowed_Remote_Clients
  • Save
  • Apply Changes

Move all Rules from the OpenVPN Interface to Individual Interfaces

If in the instructions above you have placed any rules on the OpenVPN interface they should all be moved as follows to their individual interfaces (RoutedVPN|BridgedVPN) for better security and control.

Change Interface Firewall Rule Assignments

These are how the rules should be arranged. Either move the ones in the OpenVPN interface or create new ones and delete where necessary.

The rules are shown below in the order they should appear.

  • Firewall --> Rules
  • OpenVPN
    • OpenVPN (Common) - Disabled
  • ROUTEDVPN
    • Clients to be routed over PIA
    • Route the Routed VPN Network over WAN (Internet Only)
    • Default allow ROUTEDVPN to any rule
  • BRIDGEDVPN
    • Clients to be routed over PIA
    • Allow DHCP Broadcasts from LAN
    • Default allow LAN to any rule

Change Firewall Floating Rule Assignments (Optional)

  • Firewall --> Rules --> Floating
    • Currently in Floating Rules the rules listed below are all attached to the OpenVPN interface (and possibly others). For consistency you should keep the assignments as there are, but instead of using OpenVPN you should use ROUTEDVPN and BRIDGEDVPN.
    • If you left the rules as they are, they will still work in exactly the same way because both VPNs traffic runs through the OpenVPN interface first but is better for control to have the separated.

Change pfBlockerNG Interface Assignments (Optional)

  • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration --> Outbound Firewall Rules
    • Just like the Floating Rules, pfBlocker has the OpenVPN interface assigned because it is a common ancester of ROUTEDVPN and BRIDGEDVPN. So all you have to do is remove OpenVPN and add ROUTEDVPN and BRIDGEDVPN in 'Outbound Firewall Rules', pfBlocker will do the rest.
    • As mentioned above, if you did not change this, blocking would still work on all of your VPN traffic as normal.

Additional OpenVPN Security Settings

  • Update pfBlockerNG rules
    • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration --> Outbound Firewall Rules: LAN/OpenVPN
    • This is required otherwise blocking by pfBlockerNG would not occur.
    • Add the OpenVPN interface to pfBlockerNG. OpenVPN is common for both ROUTEDVPN and BRIDGEDVPN so only this should be needed.

Configure internal Certificate Revocation List (CRL)

This feature allows you to revoke certificates you have issued for OpenVPN.

  • Step 1
    • System --> Certificate Manager --> Certificate Revocation --> Create or Import a New Certificate Revocation List: OpenVPN
    • Click Add
  • Step 2
    • Certificate Authority: OpenVPN
    • Method: Create an internal Certificate Revocation List
    • Descriptive name: OpenVPN
    • Lifetime (Days): 730
    • Serial: 0
    • Click Save

Privacy Network (with OpenVPN Client Gateway / Privacy VPN)

Common

Sign up for a Privacy VPN

  • Download the relevant .ovpn file from your provider for the country or location you want traffic to come from
    • This will have the certificate and usually OpenVPN settings that you will need.
    • Some times separate files such as .crt/.key are downloadable individually.
    • The certificate is the same in each .ovpn file from your provider

Add your Privacy VPN Certificate Authority (CA)

  • Step 1
    • System --> Certificate Manager --> CAs --> Add
    • Click Add
  • Step 2
    • Descriptive name: PIA
    • Method: Import an existing Certificate Authority
    • Trust Store: unticked
    • Randomize Serial: ticked
      • This will not be needed unless you use this CA for signing Certificates, but you should turn it on anyway.
    • Certificate data:
      • Get this from any of the following sources:
        1. Within the .ovpn files and is the block starting with -----BEGIN CERTIFICATE-----
        2. from the file ca.rsa.4096.crt
    • Certificate Private Key: leave blank
      • We do not know the private key because this is only known by PIA.
    • Next Certificate Serial: leave blank
    • Click Save

Add your Privacy VPN Certificate Revocation List (CRL)

This feature allows your VPN provider revoke certificates and therefore yours will be revoke if there is a security breach.

Not all providers will have this, in which case just sckip this section.

  • Step 1
    • System --> Certificate Manager --> Certificate Revocation --> Create or Import a New Certificate Revocation List: PIA
    • Click Add
  • Step 2
    • Certificate Authority: PIA
    • Method: Import an existing Certificate Revocation List
    • Descriptive name: PIA
    • CRL data:
      • Get this from any of the following sources:
        1. Within the .ovpn files and is the block starting with -----BEGIN X509 CRL-----
        2. from the file crl.rsa.4096.pem
    • Click Save

Create an OpenVPN Client

This gets pfSense to act as a OpenVPN client. This will not affect the OpenVPN servers you setup earlier, both can be run at the same time.

  • VPN --> OpenVPN --> Clients --> Add
  • You will get all of these connection settings from your .ovpn file
  • Fill the following in on the form
    • General Information
      • Description: PIA Switzerland
    • Mode Configuration
      • Server mode: Peer to Peer ( SSL / TLS )
      • Device mode: tun - Layer 3 Tunnel Mode
    • Endpoint Configuration
      • Protocol: UDP on IPv4 only (We are only doing IPv4)
      • Interface: WAN
      • Local port: leave blank
      • Server host or address: de-berlin.privacy.network
      • Server port: 1197
    • User Authentication Settings
      • Username: your-privacyvpn-username
      • Password: your-privacyvpn-password
      • Authentication Retry / Do not retry connection when authentication fails: unticked
    • Cryptographic Settings
      • TLS Configuration: unticked
      • TLS keydir direction: Use default direction
      • Peer Certificate Authority: PIA
      • Peer Certificate Authority: PIA (CA: PIA)
      • Client Certificate: None (Username and/or Password required)
      • Data Encryption Negotiation: ticked
      • Data Encryption Algorithms: leave default settings
      • Fallback Data Encryption Algorithm: leave as default
      • Auth digest algorithm: leave as default
      • Hardware Crypto: Intel RDRAND engine - RAND
        • Your options might vary or their are none because your system does not support any.
      • Server Certificate Key Usage Validation: ticked
    • Tunnel Settings
      • Allow Compression: Compress packets (WARNING: Potentially dangerous!)
      • Compression: Disable Compression, retain compression packet framing [compress]
        • It is best not to have compression nowadays but not all Privacy VPN support this yet.
        • If in your .ovpn you have the option comp-lzo then you will need to:
          • Set Allow Compression: Compress packets (WARNING Potentially dangerous!)
          • Set Compression: LZO Compression [compress lzo, equivalent to comp-lzo yes for compatibility]
      • Don't pull routes: ticked
      • Don't add/remove routes: ticked
      • Pull DNS: unticked
    • Ping Settings
      • leave untouched
    • Advanced Configuration
      • Custom Options:
        • Examine your .ovpn file to get any additional settings you need for custom options or get them from your provider.
        • Paste your new settings into Custom Options as recommended by your VPN provider.
        • You can see the 'Custom Options' notes at the bottom of this article for more information.
        • These are the PIA settings
          persist-key;
persist-tun;
remote-cert-tls server;
reneg-sec 0;
auth-retry interact;
      • Gateway creation: Ipv4 only (we are not using IPv6 on this setup)
    • Click Save
  • Check the connection works
    • Status --> OpenVPN --> Client Instance Statistics --> PIA Switzerland
      • Look at the entry and you should see some bytes sent and received
    • Also check the log files
      • Status --> System Logs --> OpenVPN

Add VPN Client as Gateway/Interface

By attaching a OpenVPN Client connection to an interface you can use it as a gateway.

  • Interfaces --> Interface Assignments
  • Available network ports: ovpnc1 (PIA Switzerland)
  • Click Add
  • Edit The interface (Might be called OPT1/2/3/4 depending and what you have setup already)
    • General Configuration
      • Enable: ticked
      • Description: PIA_Switzerland
    • Reserved Networks
      • Block private networks and loopback addresses: ticked
      • Block bogon networks: ticked
      • NB: This interface is a gateway so these settings are valid.
    • Click Save
    • click 'Apply Changes'
  • Restart the OpenVPN Client to change state from pending and fixes some IP issues (see vid @ 588)
    • Status --> OpenVPN --> Client Instance Statistics --> PIA Switzerland --> Restart openvpn Service
  • Manually set the default gateway
    • This should already be done earlier in this tutorial
    • System --> Routing --> Gateways
      • Default gateway IPv4: WAN_PPPOE
      • Default gateway IPv6: none
  • Set a monitor IP for checking the connection is up - on my pfSense 2.6 this breaks monitoring and prevents the connection coming up
    • System --> Routing --> Gateways --> PIA_SWITZERLAND_VPN4 Gateway --> Edit --> Monitor IP: 149.112.112.112
    • I have already used 9.9.9.9 for my WAN.
    • This IP is the secondary DNS server from Quad9
  • Update pfBlockerNG to process this Interface
    • Firewall --> pfBlockerNG --> IP --> Inbound Firewall Rules: Add 'PIA_SWITZERLAND'
      • This is a gateway and should be treated as such even though there should be no un-solicited traffic because of the nature of a VPN.

Outbound NAT Rules

  • Currently this needs to be done for LAN (10.0.0.0/24), PRIVACYLAN (10.100.1.0/24), ROUTEDVPN (10.200.1.0/24)
  • PRIVACYLAN is optionally added later in this tutorial.
  • If you don't do this, traffic will not be routed to the internet.
  • Firewall --> NAT --> Outbound --> Outbound NAT Mode: Hybrid Outbound NAT rule generation. (might of been set to this earlier in the tutorial)
  • Click Save
  • Click 'Apply Changes'
  • Every network that wants to use the PIA_SWITZERLAND needs a manually mapping rule.
    • Option 1 (the hard Way)
      • Do a rule for each (Subnet/LAN) you want to be able to access the interface PIA_SWITZERLAND with the following settings
        • Firewall --> NAT --> Outbound --> Mappings --> Add
          • Interface: PIA_SWITZERLAND
          • Address Family: IPv4
          • Protocol: any
          • Source Type: Network
          • Source network: (10.0.0.0/24 | 10.100.10/24 | 10.200.1.0/24)
          • Destination: Any
          • Description: LAN Outbound NAT
    • Option 2 - Alias (the easy way)
      • Create an alias
        • Firewall --> Aliases --> IP --> Add
        • With the following details
          • Name: My_Local_Networks
          • Description: My local networks
          • Type: Networks(s)
          • Networks
            • 10.0.0.1-24 LAN
            • 10.100.1.0/24 PRIVACYLAN
            • 10.200.1.0/24 ROUTEDVPN
        • Now follow 'Option 1' with the single change:
          • Source network: My_Local_Networks /32
            • it will revert to /32 anyway
  • Click 'Apply Changes'
  • Now all the mappings for your networks should be done. You can add or remove networks from the alias.

Privacy Policy Client (Policy Routing)

Policy routing is where we set rules on how to handle certain traffic. In this case we are using an alias to tell pfSense the identifed devices should be routed out over the PrivacyVPN. This method has the benefit of an admin being able to add and remove devices at will from the Privacy Network routing (but obviously not when you are on the PRIVACYLAN network dealt with in the next section)

I summary the mechanism as follows:

  1. We create a firewall alias which we add our devices to as required.
  2. A firewall rule identifies packets from the devices specified in the alias and then adds a tag to the packets, but also tells the packets to go out of the PrivacyVPN.
  3. If the packet's destination is not the PrivacyVPN, a `Kill Switch`firewall rules drops the packet.
    • This prevents the packet going out of any other WAN that is available.
    • This is a known issue with pfSense. I would not call it a bug otherwise it would of been fixed years ago.

This does not prevent network segment cross communication on your local router.

Specify devices with a Firewall Alias

An alias is a list that contains specified Hosts(s), Network(s) or IP(s) which can be dynamically changed without having to update your rules manually because you specify the alias in your rules and not the individual devices.

In this alias you will add the devices that you want to be routed out over your VPN. Devices can be added and removed very easily in one place.

  • Firewall --> Aliases --> IP --> Add
    • Name: Route_Out_Over_PIA
    • Description: Devices that route over VPN
    • Type: Host(s)
      • You can change this to other options such as 'network(s)'
    • Hosts:
      • Add all the required devices here from single devices (they should ideally have static IPs) to network ranges.
      • e.g. 10.0.0.182, 10.0.015-10.0.0.20
      • The Host(s) description is optional but always better to have a one. If one is not entered I think it is fill with the creation date.

Gateway Selection with a Firewall Kill Switch Rule

These rules will route any traffic for a device specifed in the 'Route_Out_Over_PIA' alias down the Privacy VPN. So I can add/remove any device on my network easily in to a Privacy Network in a single operation. Traffic is also prevented escaping over the WAN when they have been tagged to go over the Privacy VPN. Gateway selection is where the term 'Policy Routing' comes from becasue we have determined the route by a policy we have installed.

These rules combined:

  • Any packets identified as a their source being present in the alias are
    • Tagged with 'Private_VPN_ONLY' tag.
    • 'PIA_SWITZERLAND_VPNV4' is set as the gateway
    • Each interface will require this rule.
  • Floating rules are then processed which is where we have our rule 'Block Alias for VPN Going Out over WAN' which will prevent any Privacy traffic going out the WAN whent he Privacy VPN gateway fails (goes offline)
    • This rule sits on the WAN interface only.
    • All traffic from a Privacy client is tagged with 'Private_VPN_ONLY'
    • This rule checks for the 'Private_VPN_ONLY' tag
    • If any traffic has the tag 'Private_VPN_ONLY' it is dropped.

Notes

Interface Rule

This rule selects the Privacy VPN Gateway and adds a tag to the LAN packets where the source is specified in the alias 'Route_Out_Over_PIA'.

This rule needs adding to every interface where you want to control which clients are routed through the Privacy VPN or the default gateway (WAN) by using the 'Route_Out_Over_PIA' alias. Obviously just change the interface.

  • Firewall --> Rules --> LAN --> Add
    • Action: Pass
    • Interface: LAN
    • Address Family: IPv4
    • Protocol: Any
    • Source:
      • Invert match: unticked
      • Single host or alias
      • Route_Out_Over_PIA
    • Destination: any
    • Description: IP's to be routed over PIA
    • Click 'Display Advanced'
    • Advanced Options
      • Tag: Private_VPN_ONLY
      • Gateway: PIA_SWITZERLAND_VPNV4
    • Click Save
    • Click 'Apply Changes'
    • Move the rule to the top of your block Rules, if not already, then click Save

Floating Rule

On the WAN interface, block all Privacy VPN traffic that attempts to go into the WAN by matching any packets on the WAN that have a 'Private_VPN_ONLY' Tag.

  • Firewall --> Rules --> Floating --> Add (top)
    Firewall — Floating Rules | pfSense Documentation
    • Action: Block
    • Quick: unticked
    • Interface: WAN
    • Direction: any
    • Address Family: IPv4
    • Protocol: Any
    • Source: any
    • Destination: any
    • Log: ticked
    • Description: Block Alias for VPN Going Out over WAN
    • Click 'Display Advanced'
      • Tagged:
        • Invert: unticked
        • Tagged: Private_VPN_ONLY
    • Put this rule above your Custom DNS rules.

Notes

  • This rule should be assigned to the WAN interface because it is the default gateway.
  • If you have anything else set as your default gateway then this rule should be applied to that.
  • If you have any other gateways your traffic could fall over to, just to be on the safe side, add this rule to them. This includes Gateway Groups.
  • I only have 1 WAN so this is enough for me.
  • This rule will be performed on traffic sent from all Interfaces to the WAN with the tag 'Private_VPN_ONLY'.
    • This means you only need one rule to block Privacy traffic going down the WAN.
    • If you have several privacy networks setup, use the same tag.

PRIVACYLAN - A Physically separated Privacy Network (Optional)

Create an additional PRIVACYLAN Interface with a different subnet for your own Physically separated Privacy Network (Optional) where all traffic is routed through your Privacy VPN.

  • This section will create another LAN interface on your pfSense router with a subnet and functioning internet.
  • It is not currently private or routed through the Privacy VPN.
  • The will be a physically segmented network with a different subnet.
  • This requires unused ethernet sockets on your router
  • If you do not have any spare sockets, skip to the next section.

The reason you would want to do this is:

  • To have a physically different network for devices that you want to go through the Privacy VPN.
  • To have a network seperated by VLAN for such things as IoT devices.
    • VLANs can also be used when utilising different ethernet sockets, but again this is optional
  • You know that all devices connected through the related network socket would be protected.
  • You want this network to be on a different subnet
  • All traffic will be forced over the Privacy VPN and no local routing will occur.
  • The DNS will be piped through the secure DNS channel we setup earlier (ie DNS Hijacking)

Create the interface

  • Interfaces --> Interface Assignments --> Available network ports: igb3
    • I have an Intel Quad Port network card. This is the last port, but you can choose any one you want.
  • Click Add
  • Edit The interface (Might be called OPT1/2/3/4 depending and what you have setup already)
    • General Configuration
      • Enable: ticked
      • Description: PRIVACYLAN
      • IPv4 Configuration Type: Static IPv4
      • IPv6 Configuration Type:None
    • Static IPv4 Configuration
      • Static IPv4 Address: 10.100.1.1/24
      • IPv4 Upstream gateway: None
    • Click Save
    • Click 'Apply Changes'

Configure the DHCP Server

  • Services --> DHCP Server --> PRIVACYLAN
    • General Options
      • Enable: ticked
      • Range: 10.100.1.1 - 10.100.1.254
    • Servers
      • DNS Servers: 10.100.1.1 (you can use 10.0.0.1)
    • Other Options
      • Domain name: privacy.lan
        • It makes sense here to have a diffferent domain for easy identification of devices. You can use mydomain.com if you really need to.
    • Click Save
  • Restart the DHCP Service

Firewall Rules

  • Update pfBlockerNG to process this network
    • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration --> Outbound Firewall Rules: Add 'PRIVACYLAN'
  • Update the DNS Floating Rules to include PRIVACYLAN
    • Firewall --> Rules --> Floating
      • Deny DNS (Except Local)
      • Deny DoT / DoQ
  • Add an allow all rule for the network
    • Firewall --> Rules --> PRIVACYLAN
      • Action: Pass
      • Interface: PRIVACYLAN
      • Address Family: IPv4
      • Protocol: Any
      • Source:
        • Invert match: unticked
        • PRIVACYLAN net
      • Destination:
        • Invert match: unticked
        • any
      • Description: Route network over PIA
      • Click 'Display Advanced'
      • Tag: PRIVACY_VPN_ONLY
      • Gateway: PIA_SWITZERLAND_VPNV4
    • Click 'Save'

Outbound NAT Rules

Make sure you create an Outbound NAT rule if you have not already done this as per the instructions in the 'Outbound NAT Rules' section above.

Add the network 10.100.1.0/24 to the My_Local_Networks alias.

Allow Privacy Network Clients to see Web Server (on LAN)

  • When opening up assets on your LAN you should be very specific and only give access to specific services and specific ports (e.g. your public webserver on ports 80 and 443).
  • You should not blanket open your LAN network to your Privacy Network/Privacy Policy Clients as this will give them full access to your LAN and therefore the Privacy Network is no longer isolated defeating the object of this security step.

Currently everyone who is in the Privacy Network (either a client of the PRIVACYLAN or a Privacy Policy Client) will not be able to connect to any local LAN Clients/Servers which can be awkward.

How does this work

  • All privacy traffic packets are tagged with the Private_VPN_ONLY tag when they enter the router.
  • If the destination IP of the traffic is determined to be our local webserver.
    • NAT Reflection will change the destination IP from the Web Server's Public IP (123.123.123.123) to the Web Server's local network IP (10.0.0.13).
    • You can access your webserver directly with it's local IP address.
  • This packet is then processed through the floating rules
    • Floating rules are processed first, however the Private_VPN_ONLY tag has been added prior to these allowing this mechanism to work (modified Kill Switch).
  • If a rule condition is met, the traffic will be sanctioned and will be forwarded to the web server.
    • pfSense uses it's System Routing Table rather than the Privacy VPN Gateway, allowing the Servers/Assets to be potentially accessible while on the Privacy Network.
  • If non of the rules are matched because the traffic is not destined for the local network it will be allowed to carry on to the Privacy Gateway.
    • When a gateway is specified in a rule this is called Policy Based Routing and the System Routing Table is ignored meaning pfSense cannot use predefined rules to route traffic to where it is needed such a locally based web servers on the LAN.

The Rules

These rules will not break privacy, your DNS will still be routed via your secure channel (DNS Hijacking) and it is only specif LAN traffic for your webserver that is redirected.

Assuming the following has been correctly set (As above)

  • WAN is normal gateway with your public IP
  • LAN is your primary network with your Web server on it
  • You have setup port forwarding rules on the WAN interface to point to a LAN address
  • Correct 'Outbound NAT Mode' is set.

If have been following this tutorial in totality, you need both rules. If you have not employed the PRIVACYLAN option, then obviously you do not need the second rule.

  • The PRIVACYLAN clients and the Privacy Policy Clients both get the tag Route_Out_Over_PIA
  • You could merge these 2 rules into one but I like to keep the Privacy Network and the Privacy Policy Clients separate for easy logic.
  • To merge, delete the PRIVACYLAN Clients rule and add the PRIVACYLAN network into the Privacy Policy Rule. Rename it if you want.
Create a Web Server Allowed Ports Alias

We need to restrict access to the web server ports from the privacy network

  • Firewall --> Aliases --> Ports --> Add
    • Name: Web_Server_Allowed_Ports
    • Description: My Web Server's allowed ports list 
    • Type: Port(s)
    • Port:
      • 80 / HTTP
      • 443 / HTTPS
  • Apply Changes
Allow Privacy Policy Clients to see Web Server

Floating Rule to allow any device defined in 'Route_Out_Over_PIA' alias (Privacy Policy) to see LAN based servers and assets while maintaining privacy via the Privacy VPN for all other traffic.

  • Firewall --> Rules --> Floating --> Add
    • Action: Pass
    • Disabled: unticked
    • Quick: ticked
    • Interface: LAN, OpenVPN
    • Direction: any
    • Address Family: IPv4
    • Protocol: TCP/UDP
    • Source:
      • Invert match: unticked
      • Address or Alias
      • Route_Out_Over_PIA
    • Destination:
      • Invert match: unticked
      • Address or Alias
      • Web_Server_Local_IP
    • Destination Port Range
      • From: (other)
      • Custom: Web_Server_Allowed_Ports
      • To: (other)
      • Custom: Web_Server_Allowed_Ports
    • Description: Allow Privacy Policy Clients to see Web Server
  • This rule needs to be above the 'Block Alias for VPN Going Out over WAN' rule.

No tag checking this is done in this rule as it is not needed, it is in the Kill Switch preventing gateway escape.

Allow PRIVACYLAN Clients to see Web Server

Floating Rule to allow any device on the PRIVACYLAN interface to see LAN based servers and assets while maintaining privacy via the Privacy VPN for all other traffic.

  • Firewall --> Rules --> Floating --> Add
    • Action: Pass
    • Disabled: unticked
    • Quick: ticked
    • Interface: PRIVACYLAN
    • Direction: any
    • Address Family: IPv4
    • Protocol: TCP/UDP
    • Source:
      • any
    • Destination:
      • Invert match: unticked
      • Address or Alias
      • Web_Server_Local_IP
    • Destination Port Range
      • From: (other)
      • Custom: Web_Server_Allowed_Ports
      • To: (other)
      • Custom: Web_Server_Allowed_Ports
    • Description: Allow PRIVACYLAN Clients to see Web Server
  • This rule needs to be below the 'Block Alias for VPN Going Out over WAN' rule.

No tag checking this is done in this rule as it is not needed, it is in the Kill Switch preventing gateway escape.

The Floating rules should look like this

The separator blocks are optional. If you have not added them you should take a moment and do this now as it will makes things easier. Jusy click on the 'Separator' button at the bottom right. Don't forget to click save when you have added them.

Additional Step for RoutedVPN

For the Routed VPN, we to add another 'Outbound Mapping' so the traffic will route from ROUTEDVPN to the PIA_SWITZERLAND gateway, otherwise pfSense will not know how to route it and the packets will get dropped.

  • Firewall --> NAT --> Outbound --> Mappings --> Add
    • Interface: PIA_SWITZERLAND
    • Address Family: IPv4
    • Protocol: any
    • Source Type: Network
    • Source network: 10.200.1.0/24
    • Destination: Any
    • Description: ROUTEDVPN Outbound NAT
  • NB: The Bridged VPN terminates on the LAN so does not need a mapping.

Prevent routing between the Privacy Network and local Network Segments (RFC 1918)

Now we have created our Privacy Network we need to make sure it stays isolated from the rest of our network and to do that we need to add a few more rules.

  • These rules only handle IPv4.
  • These rules will block access to the WebGUI becasue the router's IP is a RFC 1918 address.
  • Clients of the LAN will always be able to access the WebGUI because fo the Anti-Lockout Rule and this is ok.

Before continuing, take a full backup of your pfSense config, just incase.

Create an Alias

This alias will contain the private IP address ranges in IPv4 which ahppen to be called RFC 1918.

  • Create a firewall Alias as follows (Firewall --> Aliases --> IP --> Add):

Create a PRIVACYLAN rule

This rules applies to traffic in and out of the PRIVACYLAN network.

  • Firewall --> Rules --> Floating --> Add (at bottom of all your rules)
    • Action: Block
    • Disabled: unticked
    • Quick: ticked
    • Interface: PRIVACYLAN
    • Direction: any
    • Address Family: IPv4
    • Protocol: Any
    • Source: Any
    • Destination: Address or Alias / RFC_1918_Networks
    • Description: Block PRIVACYLAN Clients to WebGUI and other Network Segments

Create a Privacy Policy Clients rule

  • This rule applies to traffic in and out of the network segment the client is on.
  • Add all of the required networks except for PRIVACYLAN as this would be pointless.
  • Firewall --> Rules --> Floating --> Add (at bottom of all your rules)
    • Action: Block
    • Disabled: unticked
    • Quick: ticked
    • Interface: LAN, OpenVPN
    • Direction: any
    • Address Family: IPv4
    • Protocol: Any
    • Source: Address or Alias / Route_Out_Over_PIA
    • Destination: Address or Alias / RFC_1918_Networks
    • Description: Block Privacy Policy Clients to WebGUI and other Network Segments

Notes

  • Block traffic/No routing between LANs on pfSense | Netgate Forum
    • I think this will also block access to the WebGUI except on the LAN where there is an anti-lockout rule.
    • viragomann
      • Add an alias of type networks in Firewall > Aliases and add all RFC 1918 networks to it: 
        10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
      • Set a name for the alias like RFC1918
      • If you only want to enable upstream traffic on all interfaces, you can do this with one rule: 
        Action: Pass
Protocol: TCP/UDP (or any if you want)
Source: any or only the respective network to have it more restrictive
Destination: "Invert match." checked, "Single host or alias" and enter the RFC1918 alias.
      • You may also enable upstream access with a single floating rule applied on multiple interfaces.
    • johnpoz
      • While The inverted rule can work, and I have used it for long time.. It has been known to have some issues if your using any sort of vips and stuff. So if your going to use the inverted rule on your allow, make sure test that everything is actually working how you think it should be working.
      • The other method, which is preferred method is to actually do an explicit block above your allow. You can use the same rfc1918 alias created.. Also don't forget to allow the stuff you need/want to allow to get pfsense IP on that network or else where on your nework.. Maybe you run ntp on opt2 network or something and want lan and opt1 to be able to use that for net.. DNS is forgotten a lot, set to tcp (default) vs tcp/udp - both are need for dns. I like to be able to ping my gateway as well for connectivity checks... Here is example for allow internet but not anything else on your network.
      • You can use just normal block vs reject if you want, but since its local - sometimes it better to actually tell the client, sorry you can not do that. Vs the client having to wait for timeout to figure out that not going to work.
      • Users normally have issues with rules, is that they forget that rules are evaluated top down, first rule to trigger wins, no other rules are evaluated.
      • You always place your rules on the interface where traffic will enter pfsense.. Not on the interface of the dest, etc.

Wierd Behaviour and Fixes

This section tries to address what happens to clients on the PRIVACYLAN.

When the router is rebooted, the Privacy VPN does not come back up automatically

  • Workaround: You have to manually restart the OpenVPN Client which will bring the VPN back online.
  • Fix: Later on in this tutorial we will install 'Service_Watchdog' which you can use to monitor the following services (and a lot of others) and will auto-restart them if they fail or in this case after a restart:
    • openvpn: OpenVPN server: RoutedVPN
    • openvpn: OpenVPN server: BridgedVPN
    • openvpn: OpenVPN Client Gateway (Privacy VPN) - Will be called something different on your system

What happens when the Privacy VPN gateway is down (Offline)

  • The Privacy VPN gateway is replaced in all firewall rules with the default gateway (usually WAN). This seems stupid and can cause unwanted routing.
  • When the WAN is the assigned gateway, NAT reflection works and you can see your webserver that you have set up to be visible from the internet with Port Forwarding (WAN --> LAN).
  • The Kill Switch (Tag) prevents the 'Privacy Traffic' going out of the WAN gateway by using the following firewall rules:
    • LAN or PRIVACYLAN Rule: Adding a tag 'Private_VPN_ONLY' irrespective of what gateway is assigned. This rule also specifies to use the Privacy VPN.
    • Floating Rule: This checks all packets going in/out of the WAN for the presense of a 'Private_VPN_ONLY' tag and if it finds it, the firewall will block the packet/traffic.

System --> Advanced --> Miscellaneous --> Skip rules when gateway is down

  • This option is not ideal and could potential have issues with fail-over lines
  • By default, when a rule has a gateway specified and this gateway is down, the rule is created omitting the gateway. This option overrides that behavior by omitting the entire rule instead and in which case the 'Private_VPN_ONLY' tag will not get applied allowing Privacy traffic to escape out of the WAN.
  • OpenVPN Client needs restart after pfsense reboot : PFSENSE | Reddit
    • If it is not checked and pfsense detects that your gateway is down, it recreates the rule with gateway set as default gateway instead of your vpn gateway. That will create a leak, and then you will have traffic continuing that route due to states being set up.
    • If you want to be extra sure that the traffic doesn't escape onto your WAN, follow each rule going out the VPN gateway with a duplicate rule except that it is blocked instead of passed, and the default gateway is used instead of the VPN gateway. The rules with the "Skip rules when gateway is down" option selected will prevent that leak.

Gateway Notes

  • Multiple WAN Connections — Policy Routing Configuration | pfSense Documentation
    • Setting a Gateway on a firewall rule will cause traffic matching the rule to use the chosen gateway or group, following the configured behavior of the group.
    • This can be generalized by making an alias for any RFC1918 traffic which would cover all private networks, and then using that in a rule. The alias contains 192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8.
  • All traffic will use default gateway unless otherwise specified (because the default gateway is set on every rule)

Final Things

This is not always needed but if you are having problems with the Privacy VPN not routing traffic where it should or not at all, then run these.

  • These are required to make sure the pfSense routing information is correct
    • Status --> Filter Reload --> Reload Filter
    • Diagnostics --> States --> Reset States
    • NB: you can reboot the router instead (Diagnostics --> Reboot)

Test your Privacy Network

Test your devices to see if they are on the Privacy VPN.

While testing, remember that if a connection is made then that route's state is sometimes maintained, so when enabling/disabling rules you need to close the browser or command prompt to use another state which will then follow the new rules.

  • PRIVACYLAN Clients should:
    • not access the WebGUI
    • not talk to other network segments
    • go out of the Privacy Gateway
  • Privacy Policy Clients should:
    • not access the WebGUI (unless on LAN)
    • not talk to other network segments
    • go out of the Privacy Gateway
  • What is my IP address? — ifconfig.co
    • Use the following website to check your VPN works as expect
    • The best tool to find your own IP address, and information about it.

OpenVPN Clients

How to setup and configure the OpenVPN Client for various platforms.

Setup Android, Laptops and PCs to connect into your network

Never use the 'admin' account or other privileged user for VPN connections because if the VPN user's device or credentials are comprimised, so is your router and the VPN have a big flag telling the hacker where it is.

You can manually setup an OpenVPN client or you can use the 'Client Export' package to get the correct files for various applications and devices.

  • Create a User with a strong password
    • System --> User Manager --> Users --> Add
      • Username: xxxxx
      • Password: xxxxx
      • Click to create a user certificate: ticked
      • Descriptive name: My OpenVPN
    • Click 'Save'
  • Install the 'OpenVPN client export' tool
    • System --> Package Manager --> Available Packages: Install openvpn-client-export
  • Client Export
    • VPN --> OpenVPN --> Client Export
    • Settings
      • OpenVPN Server
        • Remote Access Server: Select the appropriate 'Remote Access Server' e.g. RoutedVPN/BridgedVPN
      • Client Connection Behaviour
        • Host Name Resolution: Interface IP Address
        • Verify Server CN: Automatic - Use verify-x509-name where possible
        • Block Outside DNS: ticked
        • Legacy Client: unticked
        • Silent Installer: unticked
        • Bind Mode: Do not bind to the local port
      • Certificate Export Options
        • PKCS#11 Certificate Storage: unticked
        • Microsoft Certificate Storage: unticked
          • Use Microsoft Certificate Storage instead of local files.
          • This option is more secure when using Windows because the relevant files are not stored in plain text on the harddrive.
          • Don't use this option if you are testing because if you need to remove a certificate it will take a few more steps.
        • Password Protect Certificate: unticked
          • Use a password to protect the pkcs12 file contents or key in Viscosity bundle.
          • Don't use this option if you are testing.
          • I have not tested all the packages with this option.
          • My advice is to start with the password on if working in a live enviroment for the extra security and if you find it does not work as expected or does not suit your needs you can disable this option. Security first!!!
          • This is not your user's password.
      • Proxy Options
        • Use A Proxy: unticked
      • Advanced
        • Additional configuration options: empty
    • Click 'Save as default' (optional)
      • This step just stops you having to keep ticking the right options everytime.
    • Scroll down to the OpenVPN Clients section
    • Click on the Export button for the relevant config you want for the user that you just created. The configurations should include all information as required except for the username and password.

Notes

  • Adding OpenVPN Remote Access Users | pfSense Documentation
  • On Windows the OpenVPN Keys, Config and Certs are located here when using plain text mode as in these instructions otherwise they are stored in the 'Microsoft Certificate Storage'
    C:\program Files\OpenVPN\config
  • The exported creates a user specific package with all the required configurations present. It is very handy but does not always give you the latest OpenVPN client. After install you can get the latest client from the link below and just install over the top:
  • When using TAP (BridgedVPN) the OpenVPN Client does not show the IP for the adapter (10.200.1.x) which is normal as the adapter does not get an IP from the OpenVPN server but the LAN DHCP server.
  • 'Block Outside DNS' option
    • This option is present on both the server and the client exporter and is intentional.
    • Todo #14155: 'Block Outside DNS' option is present in the server and on the client - pfSense Packages - pfSense bugtracker
      • The two options cover different scenarios: The option in the base pushes to all clients, the option in the client export activates it only on specific clients. It gives the user control over who gets the option and who doesn't. The same is true of any options in the same context.
      • It's in the package for convenience and to make it more obvious that it's possible. Yes, anything could be put in advanced options (or made GUI options) but we try to strike a balance of convenience. Removing the option would make it less convenient and it's been there for years and very few others appear be confused by it vs those that find it useful.
  • OpenVPN Client export has private key in it. | Netgate Forum
    • Q (me):
      • When exporting Android or 'OpenVPN Connect' inline configurations with 'Client Export' for OpenVPN they include the private key.
      • -----BEGIN PRIVATE KEY-----
      • I was under the assumption that a private key should stay private on the server. Am I correct or is there a use of the private key I do not know
    • A (viragomann):
      • No, it's the client's private key.
      • The servers private key stays stored on pfSense. But you can export it from the cert manager if you want.
      • The private key is needed on the device, which has to provide its certificate to the remote device.
    • A (johnpoz Moderator):
      • It has your cert and then you have your private key for that cert..
      • Now I might mess up the steps here, its been forever since have looked into the specific of openvpn auth method. Or for that matter just ssl/tls in general, keep in might your also prob using the tls key which is also encrypting or signing or both depending on method of auth or auth and encryption of the control channel info - in general openvpn uses the static tls key to throw away bad traffic, etc. Like said its been a while..
      • But in a nutshell this should be somewhat close to the process.. And why you need the private key to your cert..
      • You might want to look up how the session key is exchanged in ssl, the server cert and client cert are not actually used for encryption of the data that will be exchanged they are used for auth and exchange of the symmetric key..
      • So you have the CA of the server cert, so you can validate that a cert the server sends is signed by the CA.. Just like how you validate that somewhere.domain.tld cert they present to you is signed by the CA.. And with the cert they send you and them signing it with their private key you can validate.. Look up how signing works.
      • You then use the cert they hand you that you know is signed by the CA, You then send your cert to them via this cert encryption, they know your cert has been signed by the same CA.. And they have their key to decrypt that traffic that you sent them.
      • Your signature on what you sent them is done with the private key you have. They can validate this with just the public, and anything they send to your public cert you would need the private key to decrypt, but it might only be used for you to sign what your sending. Like I said it has been a long while ;)
      • Short version is you validate that your talking to the correct server, and the server validates you are a valid client. And you exchange a session or symmetrical key that is used for the actual encryption and decryption of actual data you will send over the vpn.
      • You do need that private key, but I don't recall if they ever send you traffic that you need to decrypt with it - I believe its only used for the signing of the session key the client sends.. The server can validate your signature via your cert.

Rename Firewall Assets (Optional)

I have left asset names as close as possible to the Lawrence Sytems Youtube video but I would recommend some improvements to make them more relevant to this setup. These are optional but if you change them, you must make sure you change them everywhere they occur.

You can of course use your own. I would just recommend that you make sure everything works and you backup before changing these assets.

  • Alias
    • Route_Out_Over_PIA --> Privacy_Policy_Clients
  • Tag
    • Private_VPN_ONLY --> PRIVACY_VPN_ONLY
  • Interface Rule Description
    • IP's to be routed over PIA --> Clients to be routed over PIA
  • Floating Rule Description
    • Block Alias for VPN Going Out over WAN --> Block Privacy Clients going through WAN (Kill Switch)

Install and configure these Additional Packages

These are the packages I would install and use from the begining.

System_Patches

This package allows you to install recommended patches (from Netgate) or custom ones as required.

Recommended patches are usually made available for large issues and major security issues preventing the need to update the whole pfSense installation or for Netgate to make another point release. They are sometimes available in the forum and you can apply them by creating a custom patch but you should only apply patches like this if you are experienced at using pfSense.

You are able to apply and revert patches as required but this should be done with some caution.

The recommended System Patches are for the specific running version of pfSense software. These patches are curated by Netgate and may include security fixes, bug fixes, and other beneficial changes which come up between releases. This list is only updated when the package is updated, so check the package manager for updates. The controls in this section are limited as there is no need to edit the entries or the list. The patch list is updated only when you update the package so Netgate could utilise the update mechanisim already present without having to write another system.

From the 'System Patches' page:

"After upgrading, do not revert a patch if the changes from the patch were included in the upgrade. This will remove the changes, which is unlikely to be helpful."

  • Install Package
    • System --> Package Manager --> Available Packages
    • Search for 'System_Patches'
    • Click Install
    • Click Confirm
  • A new menu item will now be present
    • System --> Patches

Notes

  • Apply all system patches | Netgate Forum
    • Q: Do I apply all patches or only ones to fix my specific problems? What is recommended?
    • A:
      • SteveITS
        • Only install packages for your version, or risk breaking it. If yours is older, select it in System/Update/Update Settings.
        • When upgrading, let it finish; do not reboot early. Allow 10-15 minutes, or more depending on packages and device speed.
      • jimp (Netgate)
        • All of the patches are "Recommended" or they wouldn't be in the "Recommended" list :-)
        • They are not all lumped together or automatic because users like control over what they apply and some environments have rules/regulations about what they can apply and when.
        • Nobody but you can tell you if you want all of them. Most likely you would want all of them, but every environment is different.
        • Most likely you would want all of them, but every environment is different.
        • By the time a patch makes it into the "Recommended" list it's usually either already included in a newer release or it's been well tested internally and confirmed to solve the problem in question.
      • stephenw10 (Netgate)
        • Personally I only apply the patches for things I know I might hit.
      • SteveITS
        • My personal view on that is, patches listed as Recommended are solid enough for Netgate to want to push them out. Fixes suggested in forum posts (often by @jimp!) are still pretty solid but haven't yet made the Recommended list. Of course one can just view the patch contents and (if familiar with PHP/coding) see what is being changed.
        • Basically all of them are generated due to Redmine bug reports and once the issue is closed they will be in the next version of pfSense anyway. So maybe the answer is to track down the Redmine entry and see what it says.
  • What happens to patched during an upgrade
    • The patched get delisted from Patches because they dont not exist for the new version as the code is part of the update.
    • So if you apply aload of patches, you dont need to rever them before an upgrade.
    • If the patch is still shown in the new versioin, do not revert it. I would only expect this to ever be an inssue in point releases.
    • Ugrade Example with patches applied. (2.7.1 --> 2.7.2)
      • Before (open in a new window for larger version)
      • After (open in a new window for larger version)

Service_Watchdog

This package allows you to add services that if they fail should be brought back up automatically by this software. There is an additional option to be notfied of these events on a per service basis.

You need to have a notification mechanism setup in (System --> Advanced --> Notifications)

  • Install Package
    • System --> Package Manager --> Available Packages
    • Search for 'Service_Watchdog'
    • Click Install
    • Click Confirm
  • Configure the Service(s) you want to monitor and automatically restore
    • Services --> Service Watchdog --> Add New Service
    • Click 'Add the service'
    • Service to Add: your-choosen-service
    • Click Add
  • Configure your Notifications
    • Services --> Service Watchdog
    • Check the 'Notify' box for each service you want event notifications about
    • Click 'Save Notification Settings'

I would add all available services with notifications on.

If you stop any service manually you must make sure they are not in the list otherwise they will start again. If you disabled a service I dont think this will restart it.

  • pfb_dnsbl: pfBlockerNG DNSBL service
  • pfb_filter: pfBlockerNG firewall filter service
  • unbound: DNS Resolver
  • syslogd: System Logger Daemon
  • dhcpd: DHCP Service
  • dpinger: Gateway Monitoring Daemon
  • sshd: Secure Shell Daemo
  • openvpn: OpenVPN server: RoutedVPN
  • openvpn: OpenVPN server: BridgedVPN
  • openvpn: OpenVPN Client Gateway (Privacy VPN) - Will be called something different on your system

Status_Traffic_Totals

This will allow you to see a nice overview of your traffic. this does install a daemon but I dont think it is a high resource one.

  • Install 'Status_Traffic_Totals' via the package manager
  • Goto (Status --> Traffic Totals)
  • Select the interfaces you want to see a graph for
  • Click 'Display Advanced'
  • Click 'Enable Graphing'
  • Enable monitoring and notifications
    • Services --> Service Watchdog

It will take a couple of minutes before any graphs will show because to needs to collect some data first.

Blocklist Research (IP and DNSBL)

This is important and is probably very specific to you setup. If I can come up with a basline setup I will add it here.

Final Things

DNS Leak Test

We need to make sure you DNS is private. Go to this website to do a test DNS leak test

  • DNSleaktest.com offers a simple test to determine if you DNS requests are being leaked which may represent a critical privacy threat. The test takes only a few seconds and we show you how you can simply fix the problem.
  • Quad9 Network Providers: WoodyNet, PCH.net, i3D, GSL Networks - When you run a leak test you might not see 9.9.9.9, see this page for different options.
  • For me the leak test showed the ISP as WoodyNet which is Quad9.

Hardware Settings

  • System --> Advanced --> Networking --> Network Interfaces
    • Configuration — Advanced Configuration Options — Networking Tab | pfSense Documentation
    • Leave these settings alone, I have put them here to make you aware of them so you dont touch them
    • Disable hardware checksum offload: unticked (Default: unticked)
    • Disable hardware TCP segmentation offload: ticked (default ticked)
      • Do not uncheck this option unless directed to do so by a support representative.
    • Disable hardware large receive offload: ticked (default ticked)
      • Do not uncheck this option unless directed to do so by a support representative.
  • System --> Advanced --> Miscellaneous --> Cryptographic & Thermal Hardware
    • Configuration — Advanced Configuration Options — Miscellaneous Tab | pfSense Documentation
    • Cryptographic Hardware: AES-NI and BSD Crypto Device (aesni, cryptodev)
      • This will speed routing up by using the Crypto hardware on the CPU.
      • If you look in the 'System Information' widget on the Dashboard, it tells you if your CPU supports AES-NI
      • Load both the AES-NI and BSD Crypto Device modules together, which is the optimal configuration in most cases. Choose this unless a specific environment or configuration is found to work better without it.
      • If your CPU does not support AES-NI try 'BSD Crypto Device' on its own.
    • Thermal Sensors: Intel Core* CPU on-die thermal sensor
      • More sensors = better
      • Because I have an Intel i3 CPU
      • 'None/ACPI' works fine but has limited sensors
      • The Intel driver displays the ACPI sensors and individual CPU Core temperatures.

After changing hardware settings, it is recommended to reboot the router to make sure the correct kernel modules are loaded/unloaded.

Manual Config Backup

You can back up all required aspects of pfSense with this feature

  • Instructions
    • Diagnostics --> Backup & Restore --> Backup & Restore
      • Backup area: All
      • Skip packages: unticked
      • Skip RRD data: ticked
      • Include extra data: ticked (this is important otherwise things like pfSense Custom Block lists are not backed up)
      • Backup SSH keys: ticked
      • Encryption: optional
    • Click on 'Download configuration as XML'
    • Save the file somewhere safe
  • Notes

Disable or remove what you will never use to reduce your attack surface (optional)

In this tutorial I have enabled several types of OpenVPN, you might not want all of these, if not your should disabled or remove them and then do another backup.

Update and reboot

  • Update Blocklists
    • Firewall --> pfBlockerNG --> Update --> Select 'Force' option: Update
  • Reboot the router
    • Diagnostics --> Halt System

Security (Additional)

  • I have not exposed my pfSense GUI to the internet. I will use an OpenVPN into my network and then access it.
  • Missing administration features when using not using the 'admin' account? Some features might only be available under the main admin account.

I have done this section last because incorrect settings here could leave you locked out of your pfSense router, easy enough though for someone experienced to get around. If you get locked out, see Troubleshooting — Troubleshooting Access when Locked Out of the Firewall | pfSense Documentation.

Even if your pfSense router (GUI/SSH) is never exposes to the internet, you should consider side attacks from within your network. You might not want or need all of this security but I will just mention them and let you make your own mind up. Some of these settings are already on their default values which is fine for most people.

Admin Access | pfSense Documentation - A lot of these security items are mentioned here, so this is worth a read.

WebGUI

  • WebGUI Login Autocomplete
    • System --> Advanced --> Admin Access --> webConfigurator --> WebGUI Login Autocomplete: unticked
  • Max number of webConfigurator processes to run
    • System --> Advanced --> Admin Access --> webConfigurator --> Max Processes
    • Enter the number of webConfigurator processes to run. This defaults to 2. Increasing this will allow more users/browsers to access the GUI concurrently.
    • You can reduce this to 1.
  • Brute force protection
    • System --> Advanced --> Admin Access --> Login Protection
    • The sshguard daemon is used by the firewall to protect against brute force logins for both the GUI and SSH connections. The options in this section fine-tune the behavior of this protection.
    • The default settings are probably good for most people.
  • Console Password
    • System --> Advanced --> Admin Access --> Console Options --> Password protect the console menu (Console menu) (optional)
    • Normally the firewall always presents the menu on the console, and the menu will be available as long as someone has physical access to the console. In high-security environments this is not desirable.
    • This option adds password protection to the console.
    • The console accepts the same usernames and passwords used to access the GUI.
    • After setting this option, the firewall must be rebooted before it takes effect.
    • While this will stop accidental key presses and keep out casual users, this is by no means a perfect security method.
    • A knowledgeable person with physical access can still reset the passwords (see Forgotten Password with a Locked Console).
    • Consider other physical security methods if console security is a requirement.

Admin Account

The Anti-Lockout Rule should never be removed unless you know what you are doing

  • Rename the default admin account
    • The admin user cannot be deleted and its username may not be changed, but you can disable it.
  • Disable the admin account
  • Apply All Recommend Patches
    • System --> Patches --> Apply All Recommend
    • Patches are there for a reason. You might not need all of them, but it should not harm.
  • Other
  • Restrict access to the WebGUI / Restricting WebGUI Access To one Interface
    • As mentioned previously unless you remove the Anti-Lockout Rule you will always be able to access the WebGUI from the LAN.
    • The rule below will block access to the WebGUI on whichever network you specify and we have also specified the ports to allow other services from the firewall to run correctly.
    • This is not needed on network segements where you have blocked the RFC 1918 addresses such as in the section `Prevent routing between the Privacy Network and local Network Segments (RFC 1918)`.
    • Create a Port Alias
      • Firewall --> Aliases --> Ports --> Add
        • Name: pfSense_WebGUI_Ports
        • Description: pfSense WebGUI Ports
        • Type: Port(s)
        • Port:
          • 22 / SSH
          • 80 / HTTP
          • 443 / HTTPS
        • This cannot currenyl be used because tehre are 2 different protocols being used.
    • Create the rule
      • Firewall --> Rules --> Floating --> Add (at bottom of all your rules)
        • Action: Block
        • Disabled: unticked
        • Quick: ticked
        • Interface: Your Choosen Network
        • Direction: any
        • Address Family: IPv4
        • Protocol: Any
        • Source: Any
        • Destination: This Firewall (self)
        • Description: Block access to the pfSense WebGUI
      • As long as this is at the very bottom of your floating rules eveything should work as expected.
    • Allowing Remote Access to the GUI | pfSense Documentation
      • The default configuration of pfSense software allows management access from any machine on the LAN and denies it to anything outside of the local network.
      • There is also an anti-lockout rule enabled by default that prevents firewall rules from being configured in a way that will lock the user out of the web interface.

What Now!!!

  • Now go through all of the settings and change any that you need.
  • If you are not sure what a setting does, then don't change it on a live system.
  • Setup a test pfSense box and play with all of the settings first.

Additional Features

pfBlockerNG - Unbound Python Mode

  • What is this? This is a Python Script (active process rather than passive) for handling DNSL requests and has more features than the standard 'Unbound Mode'.
  • To use pfBlockerNG You must be using pfSense as a Resolver i.e. not forwarding DNS requests / Forwarding mode turned off).
  • BBCan117: You can easily flip back and forth between the 2 modes. (Pytnong and non-Python)
  • Unbound Mode mode will utilize Unbound local-zone/local-data entries for DNSBL (requires more memory).

The only downside of using 'Unbound Python Mode' is that machines that specify their hostname when requesting an IPv4 DHCP lease CANNOT be registered in the DNS Resolver so that their name can be resolved. See mitigations below.

  • Pros (features)
    • This mode will utilise the python integration of Unbound for DNSBL.
    • This mode will allow logging of DNS Replies, and more advanced DNSBL Blocking features.
    • You can see all DNS requests (DNS Replies)
      • Blocked requests either do not get to the DNS server or are returned as NXDOMAIN
    • Regex Blocking
    • You can do some TLD domain filtering
    • You can block IDN domains
      • All IDN domains start with xn--
      • IDN domains are domains that use non-ascii characters and can be used to trick people into thinkinf they are a different website (i.e. spoofing)
      • These should be blocked.
    • Wildcard domain blocking
    • Has better performance.
    • Uses significantly less memory than Unbound Mode. This is especially helpful if you have large lists or multiple ones.
    • Enabling Python mode also enables more DNSBL filtering options (including Unified tab)
    • Includes a policy section that allows you to add IP's that are allowed to circumvent pfblocker.
    • With this mode you can now whitelist a sub domain even when it's wildcard blocked via TLD.
    • Reports has some more tabs
      • DNS Reply - Individual information about the replay and where it comes from etc..
      • DNS Reply Stats - Compiled statistics for DNS replies
    • The unified feed is eaiser to read and has additional columns.
    • There are additional options in the 'General Settings' settings
      • Firewall --> pfBlockerNG --> General Settings
    • It is possible to exclude IP (or even networks ?) using the pfBlockerNG GUI.
  • Cons
    • none

Enable Unbound Python mode

  • The Unbound mode is toggled here
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL --> DNSBL Mode: (Unbound mode | Unbound python mode)
  • Disable DHCP
    • Services --> DNS Resolver --> General Resolver Options --> DHCP Registration = Disabled
    • Apply the change
    • This needed to be enabled in versions earlier than pfSense 2.7.0 but is no longer an issue. I have kept it here for reference so people know what option to reverse.
  • Change pfBlockerNG to use 'Unbound Python Mode'
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL --> DNSBL Mode = 'Unbound python mode'
    • Apply changes
    • Firewall --> pfBlockerNG --> Update --> Update Settings --> Select 'Forece' option = Reload
    • Click Run
  • Done

Configuration (and brief explanation)

Do these settings

  • Wildcard Blocking (TLD) = On
    • TLD is fully automated to wildcard block all root domains that are listed in the Feeds and not wildcard block any sub-domains that are listed in the Feeds.
    • You can manually Blacklist or Whitelist TLD
      • Examples - Doesnt have to be a root domain (e.g. .com)
        com
quantumwarp.com
mysubdomain.quantumwarp.com
anothersubdomain.mysubdomain.quantumwarp.com
      • Wildcard Blocking (TLD) is also in 'Unbound mode'
    • BBCan177 - How to block subdomains with pfBlockNG? : pfBlockerNG
      • This is compatible with the YOYO feeds and many other feeds.
      • There are hundreds of feeds that can be used with pfBlockerNG. The pkg has a parser for most formats of feeds, however, the feed/format you linked is not supported. That format is also non-standard.
      • The feed that is recommended is this one (as listed in the pfBlockerNG Feeds Tab): - https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
      • YOYO is one of the few maintainers of feeds that support a wildcard blocklist. Almost 99.9% of the other feeds, including ones that list malicious domains are in a single domain per line.
      • Almost all other DNS based pkgs will only block the single root domain and not wildcard block these domains. Since these feeds are not static, it is not practical to try and manually manage which domains in these feeds should be wildcard blocked or not.
      • The TLD Wildcard feature will parse the YOYO feed and all other feeds, and provide the exact same experience that you are expecting and its all automatic. On top of that, it provides increased security to wildcard block those malicious domains.
      • For the single purpose of AD Blocking, this is pretty much irrelevant as wildcard blocking is not needed for just that purpose alone.
      • You can also opt to use the YOYO pre-defined Unbound Feed and add it to the pfSense Unbound custom config without using any package. https://pgl.yoyo.org/as/serverlist.php?hostformat=unbound
      • The other features for TLD are "TLD Blacklist/Whitelist", that is used to wildcard block a whole TLD like "CN" or "RU". The benefit of that is to reduce the number of domains that need to be added to Unbound, since a single local-zone entry will be used to block the whole TLD's, and in effect consume less memory and overhead.
    • BBCan177 - Best way to block all subdomains? : PFSENSE
      • TLD is an automated process that goes thru all of the feeds and determines if each domain should be individually blocked, or wildcard blocked automagically.
      • For AD blocking TLD/wildcard blocking is not as necessary, but for malicious domains, you would want to block the full domain and subdomain.
      • There are many users that add millions of domains and most of those feeds all post the root domain, so other blockers are only blocking the root domain only. Keep in mind that malware is hardly ever on the root domain, but in the subdomains.
      • A simple test is to ping a domain that is in the blacklists, and then try to ping a random subdomain. 
      • TLD relies on Unbound zone entries. Each domain is one zone. Each zone is a pointer in memory. So the more domains that are blocked via TLD, the more memory required. The Unbound Resolver is a different animal than DNSMasq.
      • This is present in 'Unbound mode' aswell
      • You can manually add TLD to be blocked in a box that appears below
      • When this feature is enabled. The TLD Whitelist is not utilized and instead uses the DNSBL Whitelist
    • BBCan177 - Creating a wildcard list (fruitless effort) : pfBlockerNG
      • You need to enable the TLD wildcard feature. So when it processes any feeds, it will wildcard block any root domains that are listed and otherwise block sub-domains.
      • So if the feed had "example.com", it would block that domain and sub-domain.
      • If the feed had " ads.example.com" it only will block that sub-domain only.
      • TLD utilizes the Mozilla TLD list to help determine which domains should be wildcard blocked.
      • It's all automatic, and no need to append any asterixes which are not valid anyways.
      • If there is an odd ball domain, you could also add that to the python regex feature.
      • Also if you are blocking the whole "ru" TLD for example, TLD will also remove all the ru domains as they are all blocked anyways freeing some memory requirements.
      • In pfB, only the Whitelist allows a prefixed "." For wildcard.
      • The more TLDs that are blocked, the more domains that could be added before hitting memory limits since TLD will trim the domains that are blocking whole TLDs.
    • BBCan177 - Benefit of TLD blocking (ELInoob?) : pfBlockerNG
      • Well ADs are on most webpages, so you see all that activity... Hitting malicious domains is hopefully less prevalent in your network :)
      • Everyone focuses on how many ADs are getting blocked, but pay zero attention to when devices on you LAN are hitting malicious domains.
    • PfBlockerNG v2.1 w/TLD | Netgate Forum - List memory requirements for TLD Domain blocking
    • pfBlockerNG-devel TLD | Netgate Forum
      • When TLD is enabled, there is a significant benefit in wildcard blocking of malicious domains. Most DNSBL feed post the root domain for a malicious site. ie: example.com. So with TLD disabled, DNSBL will only block DNS resolution to example.com. So for ADverts this works fine since you typically just need to block the single sub-domain that is serving the ADverts. But typically, malicious sites host malware etc on sub-domains. So with TLD enabled, it will wildcard block the root domain and all sub-domains and thus protect your network from these domains fully!
      • TLD is fully automated to wildcard block all root domains that are listed in the Feeds and not wildcard block any sub-domains that are listed in the Feeds.
  • Python Control = Off
    • Allow sending python_control commands (via DNS TXT) to the Python integration
    • Unless you know what this is, leave this off.
  • DNS Reply Logging = On
    • Enable the logging of all DNS Replies that were not blocked via DNSBL.
    • Blocked DNS replies/request are logged by default, but allowed replies, this fixes that.
  • DNSBL Blocking = On
    • This option must be on for the DNSBL blocking to work.
    • I assume you can turn this off so you can do other things with this section if you know what you are doing.
  • HSTS mode = On
    • I am not sure waht this is for but it is on by default.
  • TLD Allow = Off
    • This will block all TLDs that are not specifically selected.
  • IDN Blocking = On
    • IDN domains are domains that use non-ascii characters and can be used to trick people into thinkinf they are a different website (i.e. spoofing)
    • All IDN domains start with xn--
  • Regex Blocking = Off
    • This is only required if you need to do some specific domain filtering by using REGEX.
    • Just keep in mind that regex is expensive in processing, so it can slow DNS resolution depending on how well the rules are drafted and how many regex entries are added.
  • CNAME Validation = On
    • All CNAMES will be evaluated against DNSBL database and blocked.
    • This option must be enabled to make sure that an Ad domain cannot bypass DNSBL by using a different DNS name.
  • no AAAA = Off
    • This can be used to force IPv6 DNS requests to be downgraded to IPv4 for certain domains.
    • If and IPv6 request is blocked most systems will re-request with IPv4.
    • If no re-request is done, the clietns request will just fail.
    • Most people will not need this as it is probably an edge case usage scenario.
    • When on, another box appears where you can enter domains.
  • Python Group Policy = Off
    • Allow certain Local LAN IPs to bypass DNSBL
    • When on, another box appears where you can enter IP addresses.
    • Only use this when you need it.

You should now do a 'Force Reload' to make sure everything is correct.

Notes

Network intrusion prevention and detection system (IDS/IPS)

An intrusion detection system (IDS; also intrusion prevention system or IPS) is a device or software application that monitors a network or systems for malicious activity or policy violations.

The following software are available as pfSense packages. These packages are memory heavy and if you do a lot of logging will need a lot of disk space. The packages can use up to 2GB for high load networks.

This type of system is different to pfBlockerNG because it analyses the traffic.packets for dodgy things rather than the source and desitnation of a packet. Can detect applications active on your network by analysing layer 7 data.

Snort (preferred package)

Packages — IDS / IPS — Configuring the Snort Package | pfSense Documentation

What is Snort?
  • Snort is an intrusion detection and prevention system. It can be configured to simply log detected network events to both log and block them. Thanks to OpenAppID detectors and rules, Snort package enables application detection and filtering.
  • Snort is the foremost Open Source Intrusion Prevention System (IPS) in the world. Snort IPS uses a series of rules that help define malicious network activity and uses those rules to find packets that match against them and generates alerts for users.
  • Snort can be deployed inline to stop these packets, as well. Snort has three primary uses: As a packet sniffer like tcpdump, as a packet logger — which is useful for network traffic debugging, or it can be used as a full-blown network intrusion prevention system. Snort can be downloaded and configured for personal and business use alike.
  • Snort is an open source network intrusion prevention and detection system (IDS/IPS). Combining the benefits of signature, protocol, and anomaly-based inspection.
Instructions
  • These instructions mainly follow this instructional video, pfSense + snort is AWESOME, quick look at IPS/IDS (For Free) | YouTube | The Network Berg.
  • You should always put Snort on your LANs, WANs are no longer recommended.
  • I have left the instructions to follow the video for easy of learning.
  • For this article you should setup Snort on LAN, PRIVACYLAN, ROUTEDVPN, BRIDGEDVPN.
  • See the notes below for the WAN/LAN debate
  • For interfaces with a real hardware interface such as LAN, PRIVACYLAN use INLINE IPS, for the rest where this mode is not possible you should use LEGACY MODE.
  • Backup your pfSense
  • Create an account with Snort and get an Oinkmaster Code
  • System --> Package manager --> Available Packages --> Search --> snort --> Install
  • Services --> Snort
    • Snort Interfaces
      • Nothing Yet
    • Global Settings
      • Enable Snort VRT
      • Snort Oinkmaster Code = You got this from your Snort account.
      • Enable Snort GPLv2
      • Enable ET Open
      • Enable OpenAppID
      • Enable AppID Open Text Rules
      • Enable FEODO Tracker Botnet C2 IP Rules
      • Update Interval = 12 Hours
      • Update Start Time: 00:22
      • Hide Deprecated Rules Categories
      • Remove Blocked Hosts Interval = 1 Hour
        • If something goes wrong, at most your IP will only be blocked for 1 hour.
      • Click Save
    • Updates
      • Update Your rule Set --> Update Rules
    • Alerts
      • Nothing to do here
    • Blocked
      • Nothing to do here
    • Pass Lists (Whitelists)
      • Nothing to do here
    • Supress
      • Nothing to do here
    • IP Lists
      • Nothing to do here
    • SID Mgmt
      • Nothing to do here
    • Log Mgmt
      • Nothing to do here
    • Sync
      • Nothing to do here
  •  Services --> Snort --> Snort Interfaces
    • Snort Interfaces -->Add
      • LAN Settings
      • This will just be for the LAN (this is all most setups will need)
      • Add with the following settings
        • General Settings
          • Enable: ticked
          • Interface: LAN
          • Description: LAN
          • Snap Length: 1518
        • Alert Settings
          • Leave all options off
          • Send Alerts to System Log: ticked (optional, but you probably don't need this)
            • The only reason you would need to do this is so you can send the Snort data is sent to pfSense system log which is then pushed to a remote Syslog server for deeper analysis.
            • Enable Remote Logging needs to be enabled to be able to send data to the remote server
              • Status --> System Logs --> Settings (tab) --> Remote Logging Options: Fill in as required
        • Block Settings
          • Just use the default IDS (detection-only) mode for at least two weeks and potentially a month so you can see what alerts happen on your network. This lets you investigate and weed out false positives without getting frustrated because things get blocked. Come back to this later
          • Block Offenders: ticked
            • NB: if you are not sure if you might end up blocking access to the router or other devices, dont enable 'Block Settings' until you have run pfSense with Snort on it for a while, maybe a week or so.
          • IPS Mode:
            • Select blocking mode operation. Legacy Mode inspects copies of packets while Inline Mode inserts the Snort inspection engine into the network stack between the NIC and the OS. Default is Legacy Mode.
            • Legacy Mode uses the PCAP engine to generate copies of packets for inspection as they traverse the interface. Some "leakage" of packets will occur before Snort can determine if the traffic matches a rule and should be blocked. Inline mode instead intercepts and inspects packets before they are handed off to the host network stack for further processing. Packets matching DROP rules are simply discarded (dropped) and not passed to the host network stack. No leakage of packets occurs with Inline Mode. WARNING: Inline Mode only works with NIC drivers which properly support Netmap! Supported drivers: bnxt, cc, cxgbe, cxl, em, em, ena, ice, igb, igc, ix, ixgbe, ixl, lem, re, vmx, vtnet. If problems are experienced with Inline Mode, switch to Legacy Mode instead.
            • Inline Mode (preferred)
              Important Information About IPS Inline Mode Blocking
              • When using Inline IPS Mode blocking, you must manually change the rule action from ALERT to DROP for every rule which you wish to block traffic when triggered.
              • The default action for rules is ALERT. This will produce alerts but will not block traffic when using Inline IPS Mode for blocking.
              • Use the "dropsid.conf" feature on the SID MGMT tab to select rules whose action should be changed from ALERT to DROP.
              • If you run the Snort Subscriber Rules and have an IPS policy selected on the CATEGORIES tab, then rules defined as DROP by the selected IPS policy will have their action automatically changed to DROP when the "IPS Policy Mode" selector is configured for "Policy". This will not affect other rule sets present in Snort.
              • When you choose 'Inline Mode' you will get this error on unsupported interfaces:
                The following input errors were detected:

                • The 'lan' interface do not support Inline Mode.
              • pfSense hardware must be configured correctly, amend as needed and do not change any other settings in this section.
                WARNING! IPS inline mode requires that Hardware Checksum Offloading, Hardware TCP Segmentation Offloading and Hardware Large Receive Offloading all be disabled for proper operation. This firewall currently has one or more of these Offloading settings NOT disabled. Visit the System > Advanced > Networking tab and ensure all three of these Offloading settings are disabled.
            • Legacy Mode

              When you choose 'Legacy Mode' there are a few things you should know.

              • You can only alert or block offenders.
              • Legacy mode cannot perform 'DROP' on packets at the driver level so it must use the firewall to block IP. (I think it uses the firewall, not 100%)
              • Some packets will leak through because of the nature of the parallel inspection method used via the PCAP package.
              • Kill States: ticked
              • Which IP to Block: BOTH
                • By default your local networks are whitelisted so BOTH is fine. See the section 'Choose the Networks Snort Should Inspect and Whitelist' for verifiction.
      • Detection Performance Settings
        • Leave as default
      • Choose the Networks Snort Should Inspect and Whitelist
        • Leave as default
      • Choose a Suppression or Filtering List (Optional)
        • Leave as default
      • Custom Configuration Options
        • Leave as default/empty
    • Save
  • Edit the LAN interface
    • LAN Categories (Select which types of rules will protect the network)
      The categories on this page contain the rules, and those that are ticked will be applied.
      • Packages — IDS / IPS — Configuring the Snort Package | pfSense Documentation
      • Resolve Flowbits: ticked
      • Use IPS Policy: ticked
      • IPS Policy Selection: Security
        • The IPS policies are only available when the Snort VRT rules are enabled.
        • If Snort is unfamiliar to you, then using the less restrictive Connectivity policy in non-blocking mode (the default setting) is recommended as a starting point to identify and whitelist false positives. Once experience with Snort has been gained in this network environment, blocking mode may be enabled (via the Block Offenders option in the Snort Interface Settings tab) and a more restrictive IPS policy may be chosen.
      • IPS Policy Mode: Policy (Inline Mode Only)
        • When Policy is selected, this will automatically change the action for rules in the selected IPS Policy from their default action of alert to the action specified in the policy metadata (typically drop, but may be alert for some policy rules).
        • This enables blocking on as per the rules set out in the IPS policy.
        • You still need to enable 'Block Offenders' to block offenders.
        • If this is left on 'Alert' and you are on
          • 'Inline Mode': then no dropping of packets will occur, only alerts when a rule is triggered, irrespective of it's policy rule.
      • Snort GPLv2 Community Rules (Talos certified): ticked
      • Feodo Tracker Botnet C2 IP Rules: Ticked
      • Click save
    • LAN Rules (Select which types of signatures will protect the network)
      • This tab is for manually editing rules within enabled rule categories. It does not enable or disable rulesets.
      • Packages — IDS / IPS — Configuring the Snort Package | pfSense Documentation
        • Click the Rules tab for the interface to configure individual rules in the enabled categories. Generally this page is only used to disable particular rules that may be generating too many false positives in a particular network environment. Be sure they are in fact truly false positives before taking the step of disabling a Snort rule!
      • Legacy Interface rules always have the action of 'Alert' because that is how they are loaded.
      • Edit a rule (not required at this time)
        • Available Rule Categories --> Category Selection: 'IPS Policy - Security'
        • Rule Signature ID (SID) Enable/Disable Overrides --> SID Actions --> Apply
      • Rule Signature ID (SID) Enable/Disable Overrides
        • These are just action butons for manually applying changes to the rules.
    • LAN Variables (Define servers to protect and improve performance)
    • LAN Preprocs (Preprocessor Settings)
      • Preprocessing: An Introduction - Snort Cookbook [Book] | O'Reilly
        • Snort has several components other than the rules engine. For example, some packets and applications have to be decoded into plain text for Snort rules to trigger. The component that handles the packets before they get to the rules engine is called the preprocessor.
        • Lists what the individual Snort preprocessors do.
      • Auto Rule Disable: ticked
      • Leave the rest as default
      • Click save
    • LAN IP Rep (IP Reputation Preprocessor Configuration)
    • LAN Logs (View the Logs)
      • Nothing to do here
  • Services --> Snort --> Snort Interfaces --> LAN
    • Click on the 'Play' button to start Snort on this interface.
  • Service Watchdog
    • Irrelevant of Snort's service status, when you reboot the router the service comes back online.
    • You might consider adding Snort to the 'Service Watchdog'

Snort is now all setup.

Notes
  • Official Sites
  • General
    • Version 3 is multi-threaded.
    • This also has a dashboard widget.
    • This was the original IDS
    • SNORT happens before the firewall so blocks will not be shown in firewall logs ?
    • Block Offenders
      • Services --> Snort --> Snort Interfaces --> interface --> Edit --> interface Settings --> Block Settings --> Block Offenders
      • If enabled then Snort will be in IPS (intrusion prevention system) mode where it will block and take action against bad traffic.
      • If disabled then Snort will be in IDS (intrusion detection system) mode where it will only log and alert you of infractions.
  • Rules
    • Why are rules commented out by default?
      • There are five states that we place rules in when created, four of the states are assigned to policies.
      • This lists and explains the uses of the different policies.
      • The Max-Detect policy is meant to be used in testing environments and as such is not optimized for performance. 
    • How Automatic SID Management and User Rule Overrides Work in Snort and Suricata | Netgate Forum | bmeeks
      • The explains the flow of packets through the rules in detail.
      • Both Snort and Suricata offer two similar ways to customize the rules utilized for inspecting traffic. You can use the CATEGORIES tab or the SID MGMT tab. Before diving off into the details, let's first review a few basic points.
      • There are three ways to enable rules and rule categories in the pfSense Snort and Suricata package
        1. The first is to use the CATEGORIES tab to select (by checking) the rule categories you want to use from the list extracted from the gzip rule archives you have enabled for download (Snort, Emerging Threats, etc.).
        2. Another way to select rules or rule categories is by using the features on the SID MGMT tab of each IDS package.
        3. The third way to select individual rules is by using the User-Forced Enable/Disable icons on the RULES tab.
      • GID:SID
      • SID MGMT
      • What Does "SID State Order" Mean on the SID MGMT Tab?
    • Snort Package 4.0 -- Inline IPS Mode Introduction and Configuration Instructions | Netgate Forum | bmeeks
      • There are up to three methods to choose from for the method which you want to use for changing rule actions to DROP for those rules you wish to block traffic. You can only pick one.
        1. Use the SID MGMT tab
        2. Manually force rule action changes on the RULES tab
        3. Use the Snort Subscriber Rules and choose an IPS Policy and set the policy action to "Policy" (CATEGORIES tab).
      • Using IPS Policy to Automatically Change Rule Actions
      • Using SID MGMT to Automatically Change Rule Actions
      • Force Rule Actions Using Icons on the RULES Tab
      • What is 'Enable Automatic SID State Management'?
        • Enable automatic management of rule state and content using configuration lists. Default is Not Checked.
        • Snort will automatically enable/disable/modify text rules upon each update using criteria specified in SID Management Configuration lists. The supported configuration format is the same as that used in the PulledPork and Oinkmaster enablesid.conf, disablesid.conf and modifysid.conf files. You can either upload existing configurations to the firewall or create new ones using ADD below.
        • The default lists are empty.
    • SID mgmt enable/disable question | Netgate Forum | bmeeks
      • I'll try to explain how SID MGMT works in the package without getting too deep into the woods.
      • Excellent explanation  of the logic flow.
    • SNORT SID Mgmt Disable not working | Netgate Forum
      • A worked example of using SID MGMT setup.
      • Hello! I have never used SID management before. I read through @bmeeks "How Automatic SID Management and User Rule Overrides Work in Snort and Suricata" sticky and I think I set this up correctly.
    • Snort Rules Examples and Usage: A Beginner's Guide - Sapphire - A snort effectively identifies suspicious patterns and behaviors, providing early alerts. What are the Snort rules, examples, and usage?
  • Preprocessors
    • Services --> Snort --> Snort Interfaces --> LAN(igb1) --> LAN Preprocs  (change the interface as required)
    • Dont touch unless you know what you are doing.
    • Preprocessing: An Introduction - Snort Cookbook [Book] | O'Reilly
      • Snort has several components other than the rules engine. For example, some packets and applications have to be decoded into plain text for Snort rules to trigger. The component that handles the packets before they get to the rules engine is called the preprocessor.
      • Lists what the individual Snort preprocessors do.
  • Wan vs LAN
    • Always monitor the LAN so you can see the original source and destination of a packet. If you monitor the WAN you will only see the NAT'ed address on one side.
    • It is pointless to monitor the WAN and LAN.
  • Tutorials
  • Information
    • Snort (software) - Wikipedia
    • Capacity Planning for Snort IDS | Bulbous, Not Tapered - Snort is a very capable network intrusion detection system, but planning a first-time hardware purchase can be difficult. It requires fairly deep knowledge of x86 server performance, network usage patterns at your site, along with some snort-specific knowledge. Documentation is poor, current planning guides tend to focus on one or two factors in depth without addressing other broad issues that can cause serious performance problems. This post aims to be a comprehensive but high-level overview of the issues that must be considered when sizing a medium to large snort deployment.
    • Snort IDS/IPS Explained. What - Why you need - How it works - sunnyvalley.io
      • Snort IDS/IPS Explained
      • What - Why you need
      • How it works
      • in-depth
      • What are the Differences Between Snort and Suricata?
      • Software's history
    • PFsense IP and DNS filter with PFBLOCKERNG / Application Filter with Snort and OpenAppID - HackMD
      • Today I will show you guys how to filter website with PFBLOCKERNG on pfsense
    • What is Snort Blocking Right Now? | Netgate Forum
      • The IP that Snort blocked will be shown in two places. One is on the ALERTS tab and the other is on the BLOCKS tab. They are as plain as day to see there if you go look. Both tabs show you the blocked address, and the ALERTS tab shows you both the source and destination IP addresses (if you are running Snort on the LAN interface as I recommend). If you are running Snort on the WAN, then the only local IP address you will see is your WAN IP.
      • Snort and Suricata are not like an anti-virus client. You can't just install the package, download all the rules and call it done. Both packages are designed for security admins with training on IDS/IPS operation, rule selection and tuning. If you don't want to take the time to do all the research to learn how to do these things, running either of those packages is not going to be fun for you. You are going to get lots of blocks from false positives. These packages are really not intended for use on home networks unless your day job is an IDS/IPS admin.
    • Understanding and Configuring Snort Rules | Rapid7 Blog - In this article, we will learn the makeup of Snort rules and how we can we configure them on Windows to get alerts for any attacks performed.
    • Snort doesn't know about SRC-DST pairs thus unable to whitelist anything | Netgate Forum
      • bmeeks
        • On the INTERFACE SETTINGS page for your LAN configure Snort to block "BOTH" source and destination IP addresses. This is because bad traffic can be coming in either direction (sourced from some Internet site, or destined from your LAN host to an Internet-based CnC server). The hint text on the page suggests you use the BOTH setting for "which IP to block".
        • By default, your LAN hosts will not be blocked if you leave the Pass List setting on the INTERFACE SETTINGS tab set to "default".
        • Now things will work as follows:
          1. A source on the Internet attempts to send a malicious payload to your LAN host. The flow is from the Internet (SRC) to your LAN host (DST). Snort detects the payload incoming and will attempt to block both IP addresses in the packet (source and destination) because the setting for which IP to block is BOTH; however, because the default Pass List says to never block LAN IP addresses, only the Internet source IP of the malicious traffic will actually get blocked. The flow of bad packets is now stopped, and that's what you wanted.
          2. Conversely, if a LAN host is already infected via some other method (say from a USB drive), and that host attempts to talk out to some CnC bot server to download more malicious code, Snort will detect the attempt and again act to block both IP addresses in the packet. This time, though, the flow is from your LAN host (SRC) to the remote host (DST) so SRC IP is your LAN host and DST IP is the remote Internet CnC server. Snort will once again wind up only blocking the remote DST IP of the CnC server because the default Pass List says to never block LAN host IPs. The flow of bad packets is now stopped -- again what you wanted.
        • It really does no good to leave such blocks in place forever, so I recommend users configure the setting on the GLOBAL SETTINGS tab to remove blocked hosts on an interval. One hour, or even a shorter time, is a good setting for clearing out blocked hosts that have not seen any traffic within the time interval selected.
      • bmeeks
        • If you want to primarily block LAN users from visiting corporately-banned content such as porn or perhaps social media sites, then you would really need to head towards another tool. Snort and Suricata are funamentally designed to detect and block malicious content based on data signatures. OpenAppID within Snort can do some DPI (deep packet inspection) and alert based on application ID (Facebook, YouTube, Messenger, etc.), but that gets more and more dicey as all web content moves to SSL. Encrypted packets can't be inspected. Only decrypted packets can be inspected. OpenAppID works by looking for some really basic header stuff that is outside the SSL encryption wrapper. Truly guarding against offensive content requires some type of proxy with MITM (man-in-the-middle) certificates. That is not necessarily hard to accomplish in a corporate network where you rigidly control the software on machines. It is darn near impossible to administer on BYOD (bring-your-own-device) networks.
      • bmeeks
        • Ah-ha! I understand now. As you have correctly surmised, Snort works by placing IP addresses in a pre-existing pf table called snort2c. That table is created by the pfSense code at bootup and is present even when the Snort package is not installed. There are also a handful of other special tables created at bootup by pfSense. These types of firewall tables can only take IP addresses as their input parameters. They are not set up to accept port numbers.
        • Snort on pfSense actually blocks by use of a custom output plugin created for the Snort binary by the original package author. I inherited maintenance of the package several years ago. I have made some modifications and enhancements to that custom plugin, but I did not create it. The plugin makes FreeBSD system calls to place IP addresses into the snort2c table. The plugin does not actually create a firewall rule, though. The rule is pre-existing (created at firewall bootup) and simply uses the table IP addresses as its SRC/DST addresses. The blocking plugin simply feeds the offending IP addresses to the table.
  • Inline Mode
  • Output Data
  • Errors
    • (spp_sip) Maximum dialogs within a session reached
      • This is caused by my VOIP phone.
      • Solution is to change the phone from UDP to TCP in it's settings.
  • Troubleshooting
    • Snort Keeps Stopping | Netgate Forum - I recently upgraded my pfSense system to version 2.3.3-RELEASE-p1 and now Snort keeps stopping on its own about once every 2 days.  I have to keep manually restarting the service.  Anyone have any ideas how I can troubleshoot this?  It appears to have failed during a rules update.
    • Enabling blocking offenders results in net down and lost access to the GUI | Netgate Forum
      • I had to change the following for INLINE mode to work here, there was a popup somewhere that told me to change these settings in (System --> Advanced --> Networking --> Network Interfaces):
        • CHECK (ticked) - Hardware Checksum Offloading/Disable hardware checksum offload
        • CHECK (ticked) - Hardware TCP Segmentation Offloading/Disable hardware TCP segmentation offload
        • CHECK (ticked) - Hardware Large Receive Offloading/Disable hardware large receive offload
      • bmeeks
        • All three offloading settings should always be disabled for all modes of the IDS/IPS packages. That's because the NIC will create packets that are too large when these offloading options are not disabled. With those offloading options enabled, the NIC itself is reassembling packets instead of letting the pfSense kernel do so. The IDS/IPS packages expect standard size packets (typically 1500 bytes or so). Having the offloading options enabled will result in packets that are too large for the kernel-configured netmap buffer size. That will lead to netmap errors.
        • sers need to understand that Inline Mode for both packages (Snort and Suricata) uses a special kernel-provided networking device called the netmap adapter. That adapter, especially on FreeBSD-11 and earlier, requires the NIC hardware driver be cognizant of and operate with the kernel's netmap device. If not, problems will occur. FreeBSD-12 is a little better as the netmap interoperability was moved into the new iflib wrapper for network drivers. But that wrapper code is new and still getting the occasional bug fix on the FreeBSD side.

Suricata

I have not used this.

what is Suricata?

  • This is similiar to SNORT and uses SNORT's rules.
  • Suricata is a high performance, open source network analysis and threat detection software used by most private and public organizations, and embedded by major vendors to protect their assets.

Instructions

  • System --> Package manager --> Available Packages --> Search --> Suricata --> Install

Suricata Notes

IDS/IPS Notes

  • General
    • Unless you're port forwarding and/or host a website from your home there's little need for an IDS. This not always the case.
    • IDS = Detection only
    • IPS = Detection and blocking
    • IDS Vs IPS - Check Point Software - IDS vs IPS, which is the better solution? Here we discuss their advantages and disadvantages, and how both contribute to protecting an organization.
  • Snort or Suricata
    • Open source IDS: Snort or Suricata? [updated 2021] | Infosec Resources - Although early types of Network Intrusion Detection Systems go back all the way to the early 1980s, the concept of IDS took off when Martin Roesch created his free and open source IDS system SNORT.
    • Snort or Suricata which one is better? | Netgate Forum
      • I'm a novice as well and still going through the learning pains, however I can offer this bit of insight/experience.
        • They both essentially do the same thing, however they just do it differently.
        • That being said, i started out with Snort, i now been running suricata. Suricata has it's own rules and can run the snort rules as well, so you get the best of both worlds. Also, Suricata is more modern and built to take advantage of modern multi-core cpus, whereas Snort in the beginning could only run on a single thread. The latest version of Snort was re-written to take advantage of multiple cores, but how well it does it in its current incarnation, i don't know.
      • Suricata also is capable of inline scanning, I don't know if Snort is at the moment.
        • Yes it is if you use 'Inline Mode'
        • The current Snort version on pfSense does indeed offer an Inline IPS Mode, the same as Suricata.
      • Suricata can use most Snort rules, but not all. If you were to enable all of the Snort rule categories in Suricata, you would see up to a couple hundred or more fail to load and generate errors in the suricata.log file for the interface. It won't stop Suricata from starting, but it will discard those Snort rules that contain syntax Suricata does not understand.
      • Snort3 is the latest multithreaded version of Snort from upstream, but it does not yet exist as a pfSense package. So multithreaded Snort is not possible for now on pfSense.
  • WAN or LAN? What interface(s) should i put my IDS on? (WAN/LAN/WAN+LAN)
    • You should put Snort/IDS on your LAN interface(s) because:
      • Allows you to see the internal/local IP of the device the traffic is destined or coming from. If you install on the WAN you will always see the destination/source of the traffic as your public IP and not the actual local IP address of the device.
      • pfBlockerNG is active on the WAN. This blocks by using blocklists and can perform country blocking. There is no point in analysing traffic that will be dropped by pfBlockerNG.
      • A properly configured firewall would drop a lot of traffic anyway before it goes to the LAN
      • If you install on your LAN all traffic is inspected on that network before it goes up any gateway so only 1 instances of Snort is needed. If you choose to install Snort on the WAN(s) you will have to install it on every Gateway you have including OpenVPN connections.
      • Traffic from the LAN which is routed over a OpenVPN connection will be inspected before going down that pipe.
    • You don't need to run snort on both your WAN and LAN interface, it's basically just doing everything twice
    • Snort on LAN, WAN or DMZ? | Netgate Forum
      • The LAN is the place I recommend, mainly for the reason you listed -- local hosts show up with their actual IP addresses instead of everything having the WAN's public IP due to NAT.
      • If you have Internet-facing hosts in the DMZ, then an IDS/IPS instance there can be useful running rules specific to any exposed risks in the DMZ. For example, if you run web hosts, then you would run Snort's web server and similar rules on the DMZ instance. If you have public-facing DNS or mail servers, then you would run Snort's DNS and mail server rules, and so forth.
      • There is usually never a good reason to put Snort on your WAN. First, a properly configured firewall is going to drop a lot of unsolicited inbound traffic anyway. So why waste CPU cycles analyzing traffic that your firewall rules are going to drop? If a packet is destined for your LAN or DMZ, then the Snort rules running there will catch and inspect it. So again, having it inspected on the WAN does next to nothing. pfSense itself is pretty well secured. Having Snort on the WAN does nothing for pfSense itself. If you think the firewall is insecure enough that you need an IDS/IPS on your WAN to protect the firewall, then you need a new firewal
    • Best rules to best protection in WAN and LAN Interface | Netgate Forum
      • bmeeks - For someone new to an IDS/IPS, here is my recommendation.
        1. Configure Snort on your LAN interface only. There is generally no extra security obtained by putting an instance on your WAN as the WAN, by default in pfSense, drops all unsolicited inbound traffic anyway.
        2. Do NOT configure blocking at first. Just use the default IDS (detection-only) mode for at least two weeks and potentially a month so you can see what alerts happen on your network. This lets you investigate and weed out false positives without getting frustrated because things get blocked.
        3. Register for either a free or paid ($29.99/year for paid) Snort Subscriber Rules Oinkcode. There is link for that on the GLOBAL SETTINGS tab when you click the checkbox to enable the Snort Subscriber Rules. For convenience, here is another copy of the link: https://www.snort.org/products#rule_subscriptions. Once you have done this, go to the UPDATES tab and force a rules update so your Snort Subscriber Rules will download.
        4. Edit the LAN interface in Snort and go to the CATEGORIES tab. Check the box to use an IPS Policy and then choose IPS-Connectivity in the drop-down selector. This is an excellent starter policy that offers very good protection with hardly any false positives. Save the change then start Snort on the LAN interface (or restart it if it was already running).
        5. Sit back and study the alerts you receive by periodically reviewing the ALERTS tab. It is likely you will get some false positive alerts from the HTTP_INSPECT preprocessor rules. Here is a link to an older thread about Suppression Lists and using the SID MGMT tab to control false positives: https://forum.netgate.com/topic/50708/suricata-snort-master-sid-disablesid-conf. Remember that with Snort, once blocking is enabled, every alert you see could have resulted in a block of host traffic. This is why you examine the alerts and suppress or disable those rules which are firing on benign traffic in your environment.
        6. After you get the rule set tuned up, you can go back and enable blocking mode. If things are smooth, then you can bump up your IPS Policy to IPS-Balanced and see how that works for you. I do not recommend folks use the IPS-Security policy as that one enables a bunch of extra rules that are highly prone to false positives (especially in home networks). You can also choose to start using some of the free Emerging Threats rule categories by going back to the GLOBAL SETTINGS tab and enabling the Emerging Threats Open rules. You would then add those rule categories to your ruleset back on the CATEGORIES tab for your LAN interface.
      • Snort at home - WAN or LAN? | Netgate Forum
        • LAN is where you want it with NAT.  Otherwise, all the IP addresses you see in alerts will either be your WAN IP or some far-end Internet host.  You would never see any LAN IP addresses if you run Snort on the WAN interface. Without the LAN addresses, identifying an infected host on your LAN becomes quite hard. This is because Snort on the WAN only sees the traffic after NAT rules have been applied.

NTP Server (NTPD)

This feature allows pfSense to run as a NTP server and can be polled by other computers on your network to sync their times. This can be very useful if running time sensitive machines on your local network as they will all be on the same times.

  • The NTP server is enabled by default.
    • You may need to disable NTP if pfSense is running in a virtual machine and the host is responsible for the clock.
      • This is not an issue for TrueNAS Scale becasue it uses KVM which avoids these issues by providing guest virtual machines with a paravirtualized clock (kvm-clock). 
      • When you start a KVM, the time (UTC/Local) from your Host is used as the start time for the emulated RTC of the Guest, then it is soley maintained in the VM.
  • NTPD | pfSense Documentation
    • The NTP service is a Network Time Protocol (NTP) daemon which will listen for requests from clients and allow them to synchronize their clock with that of a firewall running pfSense® software.
    • By running a local NTP server and using it for local clients, it reduces the load on the lower-stratum servers and can ensure that local systems can always reach a time server.
  • NTP Server Configuration | pfSense Documentation
    • The NTP server is located in the GUI at:
      • Services --> NTP
    • This pages lists all of the configurable options.

Transparent Proxy (Squid Proxy)

Article Notes

pfSense

General

Networking

Settings

  • Kernel Page Table Isolation (PTI)
    • Leave as is, it is set automatically as required. 
    • Kernel Page Table Isolation (PTI) - Miscellaneous | pfSense Documentation
      • Kernel PTI is a method for working around CPU vulnerabilities such as Meltdown. By exploiting that vulnerability without Kernel PTI, kernel memory could be accessed by unprivileged users on affected CPUs.
      • Kernel PTI is active by default only on CPUs affected by the vulnerability.
      • This option forces the workaround off, and requires a reboot to change.
      • If a vulnerable CPU is not detected, PTI is disabled by default and this option will have no effect.
      • The current state of Kernel PTI is printed below the option.
    • Kernel PTI disabled | Netgate Forum
      • Enabling it in pfSense prevents users/processes access the memory regions of other users/processes by exploiting the Meltdown vulnerability.
      • As I understand it that only affects users/processes running in pfSense not pfSense as a VM. You need to be looking for a fix in the hypervisor for that.
      • In general Meltdown/Spectre has minimal impact for most pfSense use cases where there are not multiple users with different privilege levels running on the firewall. IMO ;)
      • Still better to have it available than not though.
  • Microarchitectural Data Sampling (MDS) Mitigation
    • Leave off
    • Microarchitectural Data Sampling (MDS) Mitigation - Miscellaneous | pfSense Documentation
      • Microarchitectural Data Sampling (MDS) mitigation is a method for working around weaknesses in Intel CPUs which support hyperthreading. By exploiting MDS without mitigation in place, kernel memory could be accessed by unprivileged users on affected CPUs.
      • MDS mitigation is a feature that protects against Microarchitectural Data Sampling (MDS) attacks that exploit CPU vulnerabilities to leak sensitive data.
      • The mitigation clears the affected CPU buffers when switching between user and kernel space or between host and guest.
    • MDS - Microarchitectural Data Sampling — The Linux Kernel documentation - Microarchitectural Data Sampling is a hardware vulnerability which allows unprivileged speculative access to data which is available in various CPU internal buffers.
    • MDS Mitigation: any reason that's not enabled automatically? | Netgate Forum
      • A possible 10-15% performance hit
      • johnpoz
        • It could be a problem if were say running 3rd party code on your firewall that is untrusted. Or allowing users to access said firewall that could exe code.
        • But in the vast majority of your typical firewall deployment this would not be a concern, in its present form.
        • You also have those people that would scream and complain that why isn't xyz implemented... Even though not actually a concern.
        • So you make it available, and those that "want" to implement it can - but with it being a performance hit.. I have to think that it would be the rare oddball use of pfsense that this could ever come into play as a concern.
      • jimp
        • It's not on by default because it doesn't impact most users an appliance role.
        • You could turn it on if you want if:
          • You have other users who login to the firewall who can run arbitrary code (e.g. from shell or Diag > Command), but they already probably have access to read anything this exploit would get them
          • You run something on the firewall from an untrusted third party repository or package source
          • You have enabled some other situation we didn't cover that has a way to run untrusted code on the firewall.
        • It's there if you need it, it's there if you want it, but for most people using pfSense in its typical roles, it doesn't come into play.
    • Feature #9532: GUI indication and options for MDS mitigation - pfSense - pfSense bugtracker
  • File Manager
    • there is not a fully fledged file manager becasue it is not needed but you can browse save, load and edit files using:
      • Diagnostics --> Edit File
    • Editing Files on the Firewall | pfSense Documentation - Diagnostics --> Edit File contains a file editor that allows editing and creating files on the filesystem of a device running pfSense® software.

Interfaces

  • General
    • Interface Types and Configuration — Interface Configuration | pfSense Documentation
      • Explains Bogon networks
      • Bogons on external gatweays should always be enabbled on all gateways = yes
    • Interface Aliases
    • Changing the WAN interface PPPoE 'Network port'
      • Interfaces --> Assignments --> PPPs --> Edit your 'PPP Interface'
      • Change the 'Link interface' to the new port as required.
      • Save
      • Move the modem from the old ethernet socket to the new one and it will auto dial
      • Should be working now, but a reboot for me did not harm.
    • Changing the LAN interface 'Network port'
      • Notes
        • You cannot change an interface's 'Network port' when it is active.
        • Do NOT use the console to change the interface assignments, it will most likely cause all of your interfaces, Gateways and Gateway Groups to be deleted and will probably mess up all of your firewall rules.
        • Cannot change LAN interface Network port | Netgate Forum - My thread
          • The info helped, try this:
            • Login to the pfSense GUI via any other route that the LAN interface. If you do not have another method, set one up (OpenVPN, LAN2)
            • Unplug the LAN ethernet (might not be needed but you need to do it any way)
            • Interfaces --> LAN --> Edit --> Disable
            • Save and apply changes
            • Interfaces --> Assignments --> LAN --> Change the Network port
            • Save changes
            • Interfaces --> LAN --> Edit --> Enable
            • Save and apply changes
            • Plug the ethernet cable into the new socket
            • should be working now, but a reboot for me did not harm
          • And obviously reconnect via the LAN to test
      • Login to the pfSense GUI via any other route that the LAN interface. If you do not have another method, set one up (OpenVPN, LAN2)
      • Unplug the LAN ethernet (might not be needed but you need to do it any way)
      • Interfaces --> LAN --> Edit --> Disable
      • Save and apply changes
      • Interfaces --> Assignments --> LAN --> Change the Network port
      • Save changes
      • Interfaces --> LAN --> Edit --> Enable
      • Save and apply changes
      • Plug the ethernet cable into the new socket
      • Should be working now, but a reboot for me did not harm.
    • Blocking Access Between Subnets/Interfaces
      • You can only block between interfaces.
      • You should not have more than one IP range on one interface.
      • Put each network on a separate pyshical ethernet port.
      • Firewall rules are evaluated on interface traffic enters pfsense top down first rule wins and no other rules are evaluated.
  • Interface Groups

Routing

DNS

  • Logging DNS Queries
    • Services --> DNS Resolver --> General Settings --> Custom options: Add
      server:
log-queries: yes
log-replies: yes
#log-tag-queryreply: yes
    • Troubleshooting DNS Queries | pfSense Documentation
    • Access the static logs via the GUI
      • Status --> System Logs --> System --> DNS Resolver
      • Only up to a Maximum 500 records.
    • Access the log via SSH
      • /var/log/resolver.log
    • Is there a means within pfSense to log accessed web traffic in real time? | Reddit
      • full instructions on watch live DNS requests via the command line.
      • Squid and squidguard can do this and will get the direct urls accessed. You can also do this with pfblocker but your only getting the dns domain and not the full url or objects accessed.
  • DNS / DoH / DoT / DoQ - DNS Hijacking
    • Quad9 Recommended Settings - Recommended: Malware Blocking, DNSSEC Validation (this is the most typical configuration)
    • Configuring Quad9 on pfSense - Linux Included
    • Configuring DNS over TLS | pfSense Documentation
    • DNS Over TLS On pfSense 2.4.5 | Lawrence Systems
      • Covers a little on PF Blocker
      • Mentions DoH and related issues with blocking it.
    • DNS over TLS with pfSense | Netgate
      • Cloudflare’s new DNS service has a lot of industry attention, so we wanted to offer a quick guide that covers setting up your DNS servers in pfSense®, including configuring DNS over TLS. In addition to Cloudflare DNS servers, the following guide also applies to Quad9 DNS service.
      • These instructions are old, but have settings you can put in the 'Custom Options' box which will still be valid.
    • Request Info - DoH BlockList | Reddit - some useful information.
    • DNS over QUIC (DoQ)
      • New DNS over QUIC protocol makes encrypted DNS traffic faster and more efficient | Cybersecurity | SIDN - This spring, DNS-over-QUIC (DoQ) has been standardized as RFC 9250. This makes the benefits of native QUIC – namely faster response times and higher efficiency – also immediately available for encrypted DNS transport.
      • What is DNS over TLS (DoT), DNS over Quic (DoQ) and DNS over HTTPS (DoH & DoH3)? - Getting Started - NextDNS Help Center
      • Where is pfSense support for HTTP/3 and QUIC protocol support? | Netgate Forum
        • Most of the QUIC payload is encrypted.
        • The push for QUIC is that the connection setup is done in one handshake instead of the 3 needed for current TCP+TLS+HTTP/2
        • Currently browsers which support QUIC also send a TCP 443 connection as a fallback in case the QUIC (UDP 443) packet is blocked.
        • QUIC is intended to replace ANY TCP connection not just HTTP.
        • It uses UDP simply to raise the possibility of transit through intermediary routers instead of trying to introduce a new IP protocol number.
        • The video I posted is an interview with one of the RFC authors David Marx and goes into the protocol level of how it works -- it's a pretty easy video to follow.
        • The new challenge is that QUIC is already past the 25% mark of total traffic since it's used by the big content providers (Google, FB) and supported by the major browsers. Chrome will flip the setting bit soon to enable it by default.
      • What is QUIC - This new Google Protocol makes Firewalls Blind
        • QUIC (Quick UDP Internet Connections) is a new generation Internet protocol that speeds online web applications that are susceptible to delay, such as searching, video streaming etc., by reducing the round-trip time (RTT) needed to connect to a server.
        • UDP Transport to avoid TCP head-of-line blocking
        • Cisco and Palo Alto for example recommend administrators to block UDP port 443 on the firewalls in order to force Chrome browsers to fall-back to regular TCP 443 connections instead of QUIC. Connectivity of the users will not be lost since the browser will silently fall-back to TLS (TCP443).
        • Therefore, if you want to block some Google applications on your next generation firewall (such as Youtube, Gmail etc) you will need to block UDP443 in order to block QUIC. Otherwise, the proprietary encryption used with QUIC will not allow the firewall to correctly identify Google applications and restrict them if needed.
  • 'Split Horizon' DNS (Split DNS)
    • This just means having 2 (or more) sources of DNS, one for the outside internet and one for your internal network. This setup can also be implemented whilst using NAT Reflection.
      • Options on pfSense
        • Host Overrides
          • Services --> DNS Resolver --> General Settings --> Host Overrides
          • An host override in pfSense required for each hostname in use behind the firewall.
        • Domain Overrides
          • Services --> DNS Resolver --> General Settings --> Domain Overrides
          • Domain Overrides are used to configure specific DNS Servers for particular domains.
    • My Split DNS Setup
      • My 2 sources of DNS are:
        1. Webserver
        2. Host Overrides
      • For my webserver and its hosted websites I did the following Host Overrides
        • I setup an intial host with my primary webserver domain:
          • quantumwarp.com
            • Host:
            • Domain: quantumwarp.com
            • IP: 123.123.123.123
        • Then I set all of the other hosted websites as aliases of this record. Do not add separate records for each domain as they will get wiped out. You can only have 1 record per IP.
    • Network Address Translation — Split DNS | pfSense Documentation
      • A preferable alternative to NAT reflection is deploying a split DNS infrastructure. Split DNS refers to a DNS configuration where, for a given hostname, public Internet DNS resolves to public IP address, and DNS on the internal network resolves to the internal, private IP address.
    • Split-horizon DNS - Wikipedia
      • In computer networking, split-horizon DNS (also known as split-view DNS, split-brain DNS, or split DNS) is the facility of a Domain Name System (DNS) implementation to provide different sets of DNS information, usually selected by the source address of the DNS request.
    • Configuring Windows to Support Split-Horizon DNS | Jamf Learning Hub
      • Split-horizon DNS (also known as split-view DNS, split-brain DNS, or split DNS) is a term used when two zones for the same domain are created—one zone is used by the internal network and the other by the external network (usually the internet). This means that a domain can resolve to different IP addresses depending on which DNS server you are using, which depends on whether you're "inside" or "outside" the network.
    • Split DNS - Advanced Configurations - A split DNS allows you to rewrite DNS requests from *.domain.com directly to your server instead of having to go through the router, it has several benefits.
  • Host Overrides / Static Hostnames
  • Domain Overrides
    • Services --> DNS Resolver --> General Settings --> Host Overrides
    • Domain Overrides are used to configure specific DNS Servers for particular domains. This is useful for split DNS.
    • pfSense Domain Overrides explained - Learn everything you need to know about pfSense Domain Overrides as well as pfSense Host Overrides. Easy & Beginner-Friendly.
  • Slow DNS
    • Services — DNS Resolver — DNS Resolver Advanced Options | pfSense Documentation
      • 'Prefetch Support'
        • Services --> DNS Resolver --> Advanced --> Prefetch DNS Key Support
        • Controls whether or not Unbound prefetches message cache elements before they expire to help keep the cache up to date.
        • This option can cause an increase of around 10% more DNS traffic and load on the server, but frequently requested items will not expire from the cache.
        • Probably only of any use for a larger network.
      • DNS local Cache and slow DNS Queries in pfSense | Reddit
        • I recently enabled DNS over TLS and used DNS Query Forwarding to quad 9 servers and I found that my initial queries were very slow and was making my internet a mess since I have a lot of smart devices.
        • A Quad9 staff member did some techical explantions of how Quad9 works.
        • Some settings mentioned
          • Prefetch Support = Enable
          • Prefetch DNS Key Support = Enabled (only matters if you have DNSSEC enabled)
          • Serve Expired = Enabled
  • Clear DNS Cache
    • Troubleshooting the DNS Cache | pfSense Documentation
      • This has all the different methods to clear the pfSense/unbound DNS cache.
      • Restarting pfSense
        • I found this not to clear the cache.
      • Restarting the DNS Resolver
        • I found this not to clear the cache.
      • From the shell 
        unbound-control -c /var/unbound/unbound.conf flush <name>
unbound-control -c /var/unbound/unbound.conf flush example.com
unbound-control -c /var/unbound/unbound.conf flush www.example.com
    • How to flush/clear the pfsense DNS forwarder cache - nixCraft - I am using the pfSense based firewall. How do I clear the unbound DNS forwarder cache when using the pfSense firewall?

NAT

Internet IP --> WAN IP --> NAT --> 10.0.0.x
LAN IP --> WAN IP --> NAT Reflection --> 10.0.0.x
OpenVPN --> LAN IP --> WAN IP --> NAT Reflection --> 10.0.0.x
  • The webserver never sees the real IP address when using nat reflection
  • NAT re-writes the traffic so it appears to be coming from it.
  • 1:1
    • Network Address Translation — 1:1 NAT | pfSense Documentation
      • 1:1 NAT (pronounced “one-to-one NAT”) maps one external IP address (usually public) to one internal IP address (usually private).
      • All traffic originating from that private IP address going to the Internet through the interface selected on the 1:1 NAT entry will be mapped by 1:1 NAT to the public IP address defined in the entry, overriding the Outbound NAT configuration.
  • NAT Reflection
    • Network Address Translation — NAT Reflection | pfSense Documentation - NAT reflection refers to the ability to access external services from the internal network using the external (usually public) IP address, the same as if the client were on the Internet.
    • Can someone explain to me what is NAT Reflection Mode in simple terms like level 1 (i'm old and newbie to pfSense) | Reddit - A simple and concise description of this feature.
    • I've lack of knowleadge about NAT reflection in pfSense - Networking & Firewalls - Lawrence Systems Forums - A more indepth scenario of how NAT Reflection works.
    • vpn - Route specific HTTP requests through pfSense OpenVPN - Server Fault
      • Dual WAN / 2 IP solution
      • Q:
        • Problem: We have an external website which we recently firewalled so it only accepts traffic from our office IP addresses. This works well at the office, but doesn't work for remote access through VPN as we don't route all traffic through OpenVPN. I would rather avoid forcing everyone to route all traffic through just to accommodate this one site.
      • A:
        • Got it! A coworker and I stumbled through onto success (for now).
        • XXX.XXX.XXX.XXX = IP of the specific site we wanted routed over the VPN
        • Solution:
          1. VPN --> OpenVPN --> Server --> Edit --> Custom Options: push "route XXX.XXX.XXX.XXX";
          2. Firewall --> NAT --> Outbound -->
            • Select Manual Outbound NAT (AON)
            • Add 2 routes below:
              • Interface Source SrcPort Destination DestPort NATAddr NATPort StaticPort
              • WAN 10.23.23.0/24 * * * * * NO
              • WAN 10.0.8.0/24 * XXX.XXX.XXX.XXX/32 * * * NO
          3. Profit ;-)
        • Step 3 is optional
      • A:
        • Additionally: In my case I have a dual wan setup, so I had to add a rule in the OpenVPN section to redirect traffic to a specific wan and restart OpenVPN
    • OpenVPN to IP Alias, NAT reflection not working | Netgate Forum
      • Dual IP solution
      • Q:
        • I have a problem where Open VPN clients cannot connect to the public IP of a server behind the pfSense firewall.
        • From the Internet, connecting to the public IP is no problem - tells me that 1:1 NAT and IP Alias is working. From inside the protected network, DHCP clients can connect via the public IP - tells me that NAT reflection is working for those clients. But for clients connected via OpenVPN, there's no such luck.
        • OpenVPN is running on the pfSense firewall itself. The public IP is not the same as the firewall public IP (provider assigned multiple addresses).
        • I have two 1:1 Nat entries for this host, using each of the WAN and OpenVPN interfaces. I also have checked the option to "Automatically create outbound NAT rules which assist inbound NAT rules that direct traffic back out to the same subnet it originated from."
      • A:
        • I actually figured out a workaround … I created another 1:1 NAT rule with OpenVPN as the interface. Otherwise the rule is the same for the 1:1 NAT rule that sends public traffic to the private IP.
        • NB: for OpenVPN clients who do not use the "send all traffic over the VPN" option, accessing the public IP is no problem, but for clients who DO send all their traffic over the VPN, this is necessary to connect to public IPs. In a few critical scripts which we share with our customers the public hostname/IP is configured, so staff who might use those scripts from a hotel/airport/conference while tunneling all traffic to the firewall make this configuration requisite.
  • NAT Troubleshooting

Firewall

  • Moving multiple firewall rules
  • Firewall — Firewalling Fundamentals | pfSense Documentation
    • This includes arguments between Reject and Block
    • WAN use Block, LAN use Reject
  • Logging
    • Rule Tracker ID is the number in the brackets on each line in the firewall log i.e. (45645678)
    • pfSense enable Logging for Firewall Rules to log internal Client IPs - .matrixpost.net
      • By default for outbound internet traffic with NAT you won’t see the internal client ip addresses in the firewall logs of pfSense
      • pfSense by default only will log the NAT address and destination address.
      • So if you want to check and see which internal client was connected to a specific internet destination, you have to enable Log packets that are handled by this rule at the specific rule which allows the client to access this destination.
    • pfSense view allowed/permitted log - Server Fault
      • I've just installed pfSense and noticed that it only logs "blocked" traffic". How do I log the default allowed traffic.
      • Status --> System Logs --> Settings (tab) --> Log packets matched from the default pass rules put in the ruleset
      • Unchecked by default. When set, logging will occur for packets matching the default pass out rules on interfaces. Setting this option will generate a large amount of log data for connections outbound from the firewall. The best practice is to only enable this for brief periods of time while performing troubleshooting or diagnostics.
    • System Monitoring — Log Settings | pfSense Documentation
  • As soon as a firewall rule is matched, the packet is processed with that rule and then further firewall rules are ignored.
  • Rules are not working
    • Double check your rules are correct.
    • Some or all of these steps might be required to make sure a rule is working or not working as expected, try them in order.
      • Reload Filters (Status --> Filter Reload)
      • Check and clear state connections for an individual IP/Device
        • Diagnostics --> States --> States
        • Filter by IP ie 10.0.0.189
        • You should see just connections to and from 10.0.0.189
        • Kill all states for the filtered results and this should get rid of any old states causing issues. (This function seems broken at the minute pfSense 2.6.0CE)
      • Clear All States ( Diagnostics --> States --> Reset States)
      • Restart the Browser
      • Disconnect / Reconnect the Network Connection on the Windows PC
      • Reboot the Windows PC
      • Reboot the Router
  • 'IP Options'
    • Firewall - IP Options | pfSense Documentation
      • Checking this box will allow packets with defined IP options to pass. By default, pf blocks all packets that have IP options set in order to deter OS fingerprinting, among other reasons. Check this box to pass IGMP or other multicast traffic containing IP options.
    • Firewall Rule --> Advanced Options --> 'Allow IP options'
      • Allow packets with IP options to pass. Otherwise they are blocked by default. This is usually only seen with multicast traffic.
    • IP Options - Internet Core Protocols: The Definitive Guide [Book] | O'Riley
    • Internet Protocol Options - Wikipedia
    • IP Options Chapter (pdf) | net.academy.lv - Technical document on the IP Options header.
    • The TCP/IP Guide - IP Datagram Options and Option Format - A technical article on this subject.
    • [SOLVED] Routing a bridged LAN connection. | Netgate Forum
      • nimrod
        • I plugged in my iptv box into port 4 and iptv box is working for a few seconds and then it stops. Picture just freezes. Then it starts working again, and then it stops again. I know that bridging two ports is not really ideal solution because of performance issues.
        • Here is the question. As i already mentioned above, my isp router is set into bridge mode by default. Is it possible to plug it into port 3 on my pfsense appliance and route that bridge traffic to port 4?
      • stephenw10 (Netgate)
        • You probably need some additional firewall rules on the bridge (or bridge members) to pass mutlicast traffic. That requires IP options be enabled.
          I would expect to see some blocked traffic in the firewall logs.
      • nimrod
        • pfSense working as a router, firewall, vpn, adblocker, and parental control device.
    • IPTV and IGMP Proxy problems | Netgate Forum
      • I'm wondering if anyone could help me setup IPTV, or give a points to what I'm doing wrong?
      • After a lot of head scratching I finally managed to get it to work. I'll post the settings I used, in case someone reads this post later and possibly find them useful.
      • Modify the "Default allow LAN to any rule" and enable the option "This allows packets with IP options to pass
  • If you leave a ping going, this will not change when you change the rules because the state has already been defined/cached.
  • While learing and configuring, it is best to kill all states between changes
  • When making firewall chnges, sometimes they are not instant
  • The firewall will normally leave the state tables intact when changing rules.
  • This Firewall (Self)
    • This references all IPs that are assigned to pfSense Interfaces, internally or externally.
    • Get all addresses on the firewall
      Go to:
https://hostname.domain:port/status.php#NetworkInterfaces
or
https://your_firewall's_IP-addr:port/status.php#NetworkInterfaces

Ctrl+F "inet"
      This status.php page is not exposed in the menus that I have found so only direct URL access is available.
    • This includes the exposed WAN address eg 8.8.8.8
  • Tag and Tagged
  • Rules Processing Order
    • Firewall — Rule Methodology | pfSense Documentation
      • In pfSense® software, rules on interface tabs are applied on a per-interface basis, always in the inbound direction on that interface
      • There are three main classes of rules: Regular interface rules, Floating rules, and Interface Group rules (including VPN tab rules). The order of processing of these types is significant, and it works like so:
        1. Floating Rules
        2. Interface Group Rules
        3. Interface Rules
      • The rules are ordered in that way in the actual ruleset, keep that in mind when crafting rules. For example, if an interface group contains a rule to block traffic, that rule cannot be overridden with an interface tab rule because the traffic has already been acted upon by the group rule, which was matched first in the ruleset.
      • The rules are processed until a match is found, however, so if a packet is not matched in the group rules, it can still be matched by an interface rule.
    • Network Address Translation — Ordering of NAT and Firewall Processing | pfSense Documentation
      • Understanding the order in which firewalling and NAT occurs is important when configuring NAT and firewall rules.
      • The page lists the order in which the rules are processed when dealing with NAT.
      • Floating rules without quick set process as “last match wins” instead of “first match wins”. Therefore, if a floating rule is set without quick and a packet matches that rule, then it also matches a later rule, the later rule will be used. This is the opposite of the other tab rules (groups, interfaces) and rules with quick set which stop processing as soon as a match is made. See Floating Rules for more details on how floating rules operate.
  • Aliases
    • Firewall — Aliases | pfSense Documentation
    • An alias is a list that contains specified Hosts(s), Network(s) or IP(s) which can be dynamically changed without having to update your rules manually because you specify the alias in your rules and not the individual devices.
    • Alias IPs in a single list can be in different subnets
    • Create an Alias and a Firewall Rule with pfSense - Today I show you how to create an Alias and a Firewall Rule with pfSense and explain to you, why it is useful using Aliases in pfSense!
    • How to find where any Alias is used
      • Locating unused aliases | Reddit
        • I would create pfSense Backup and use an XML parser or a text editor to exmaine the file and search for the Alias.
        • Search for the actual name of the Alias and not the description.
        • To automate, it shouldn't be too difficult to write a python script to loop through all the <aliases> and see if they appear in a <rule><source> or <rule><destination>.

Bridge / Transparent Bridge / LAN Bridge / Bridging / Network Switch

  • Bridging | pfSense Documentation
  • Non-Routable packets cannot be routed by the router, so can only be passed by a bridge.
  • A bridge only allows broadcasts to cross over (non-routable packets), it does not pass routable traffic.
  • The traffic flows as follows (Interface 1 --> Bridge --> Interface 2) and when the traffic is passed inot Interface 2 it is subject to firewall rules just as the routed traffic is.
  • What Are The Fundamental Differences Between Bridging And Routing In Terms Of Configuration? | OpenVPN - Confused as to what the fundamental differences between bridging and routing in terms of configuration are? Find your answer here.
  • How to Enable LAN Bridge with pfSense® - Protectli - A guide to enable LAN Bridge with pfSense®: Assigning the LAN interface to a bridge utilizing the additional ports, OPT1 and OPT2, on the Vault.
  • How To Setup A Transparent Bridge & Firewall With pfsense and Suricata - YouTube | Lawrence Systems - How To Setup A Transparent Bridge & Firewall With pfsense and Suricata
  • 2- Bridging on pfSense - YouTube
    • In this LAB, I will show you how you can profit from Bridging on pfSense
    • The brdige will be configured as an interface and the DHCP will sit on this interface.
  • Configure pfSense bridge over multiple NICs as LAN | Irregular Motif - pfSense doesn’t seem to have a simple “bridge-all-NICs” option. Really?
  • How To PFSense Configure Network Interface As A Bridge / Network Switch - nixCraft
    • Warning: Only one interface on a bridge should have an IP address. In this example, I’ve assigned IP address to lan interface (192.168.1.254). Do not add multiple IP addresses in the same subnet on different bridge member interfaces. Other interfaces on the bridge should remain with an IP type of None.
    • You need to add a firewall rule to allow traffic between each interface of the bridge
  • Bridging | Learn pfSense 2.4 - packt - Bridging chapter
    • A bridge only allows broadcast traffic between the interfaces? allows traddic onto the lan if that is where its target is without procssing more rules?
    • NB: you can use an interface group to apply common rules to a group of interfaces. After these rules are process the individual interface rules would be processed.
  • Setting up pfSense as a Stateful Bridging Firewall | Diggory Gray (PDF)
    • The goal of this page is help you setup a pfSense firewall, with the following features:
      • Bridging firewall, not a NAT firewall
      • QoS/Packet shapping to avoid saturation of your Frodo link with low priority traffic
      • Intrusion prevention using SNORT (optional, see further documentation)
      • Firewall rules to block undesirable traffic.
      • Integration with Oxford services, such as NTP and DNS (hum drum stuff)
  • Running on a Transparent pfSense Bridge - KB & Manual - ADAMnetworks - This article will show you how to setup pfSense as a transparent bridge, and installing adam:ONE (DNSthingy) to filter all traffic.
  • How to Configure a pfSense Network Bridge on a Dedicated Server | OVHcloud - Bridged networking can be used to configure your pfSense virtual machine as a NAT firewall for other virtual machines on the same host. It could even be used as an extra filter for a web server. Specific steps and configurations are needed to allow the pfSense router to work on the OVHcloud network and this article will show you how a basic pfSense NAT configuration is done.
  • pfSense with a bridge as a LAN interface : traffic blocked between interfaces - Server Fault
  • Bridging | pfSense Documentation
    • A bridge interface (e.g. bridge0) itself may be assigned as interface. This allows the bridge to act as a normal interface and have an IP address placed upon it rather than a member interface
    • Bridging and Layer 2 Loops
      • Managed switches employ Spanning Tree Protocol (STP) to handle situations like this, because it is often desirable to have multiple links between switches, and the network shouldn’t be exposed to complete meltdown by someone plugging one network port into another network port.
      • pfSense enables STP on bridge interfaces to help with loops, but it can still lead to unexpected situations. For instance, one of the bridge ports would shut itself down to stop the loop, which could cause traffic to stop flowing unexpectedly or bypass the firewall entirely.
  • Bridging — Bridging Two Internal Networks | pfSense Documentation
    • When bridging one internal network to another, two things need to be done. First, ensure that DHCP is only running on the interface containing the IP address and not the bridge members without an address.
    • Second, an additional firewall rule may be necessary at the top of the rules on the member interfaces to allow DHCP traffic.
  • Bridging — Bridging interoperability | pfSense Documentation
    • Mixing Bridged and NAT Segments
      • For hosts behind the NAT/routed segment, NAT must occur as traffic exits toward the bridged systems so that the return traffic will come back to the firewall.
      • For hosts on the bridged segment to reach hosts behind the NAT segment directly, a static route could be used on the bridged hosts or upstream gateway to send the “private” subnet traffic to the IP address of the firewall in the bridged network.

Other Services of Note

Additional Software Packages (Official)

There are some great software packages that can enhance pfSense just pick the ones you need. Have a look here Package List | pfSense Documentation.

Install packages via (System --> Package Manager)

The Common pfsense Packages / Plugins We Use and Why | Lawrence Systems

  • Suricata
  • arpwatch - Useful to monitor secure networks for device changes
  • darkstat - Basic network statistics gather.
    • A network statistics gatherer that offers bandwidth graphs for an interface, as well as traffic to/from specific IP addresses. Once installed, it appears under Diagnostics > darkstat.
    • It's a packet sniffer that runs as a background process on a cable/DSL router, gathers all sorts of statistics about network usage, and serves them over HTTP.
    • This allows you to see traffic per device on your network. Basic but does the job if you want to see who is using all the bandwidth.
    • Adds the menus items
      • Diagnostices --> darkstat
      • Diagnostics --> darkstat Settings
    • Goto settings and do the following:
      • Enable darkstat
      • Capture interface LAN/WAN (or your preference)
      • Web Interface Binding: LAN
      • Web Interface port: 666
      • Web Interface Hostname or IP Address (Optional): leave empty
    • You can access the http panel by the following:
      • Diagnostics --> darkstat
      • Diagnostics --> darkstat Settings --> Access darkstat (tab)
      • http://Web Interface Binding:666/
      • http://[Web Interface Hostname]:666/
      • if HSTS is not enabled
        • http://[pfSense Hostname]:666/
        • System --> Advanced --> Admin Access --> webConfigurator --> HTST
  • iperf - Great for speed testing your network
  • nmap
    • Nmap is a utility for network exploration or security auditing.
    • Only useful for professionals from the command line.
  • Status_Traffic_totals
    • pfSense has always needed this, an overview of trafic usage for the various networks
    • Make sure you click 'Enable Graphing' otherwise it wont do anything.
  • Zabbix - This is for corporate device monitor amongst other stuff
  • Snort - An open source network intrusion prevention and detection system (IDS/IPS). Combining the benefits of signature, protocol, and anomaly-based inspection.
  • Suricata - Similiar package to Snort

Additional Software Packages (Not-Official)

I ahve come across these packages but have not used them.

Upgrading pfSense

  • Read the release nots of the new version.
  • Backup your settings (including extra data)
  • Generate a list of installed packages before
    • openvpn-client-export
    • pfBlockerNG-devel
    • Service_Watchdog
    • Shellcmd
    • snort
    • Status_Traffic_Totals
    • System_Patches
  • Uninstall all packages
    • Due to major changes in PHP and base OS versions, there is a higher than usual chance that packages will interfere with the upgrade process.
    • To give an upgrade the best possible chance of going smoothly, uninstall all packages before starting the upgrade.
  • Revert 'System Patches'
    • Not sure if this is needed
  • Now upgrade.
  • Re-Install all packages
  • Reboot
  • Done

DHCP

General
  • DHCP leases question (deleting a dhcp lease manually) | Netgate Forum
    • Q: Is there a way (in the GUI) to manually delete DHCP lease? The DHCP server seems to "remember" even the expired leases, but I would prefer if they were removed, or there was a way to delete them.
    • A:
      • In the GUI, go to Diagnostics --> Edit File and load /var/dhcpd/var/db/dhcpd.leases~
      • Delete the leases you want removed, then hit Save.
      • Do the same for /var/dhcpd/var/db/dhcpd.leases
      • Then pressing the Save button in the DHCP Server page so it updates the current DHCP lease listing in the GUI? - this step might not be neeed
      • NB: I do not know what there is 2 lease files and which is the actual one you need to manipulate.
    • A:
      • Deleting all the leases shown in the two files did not work for me, a reboot does however, although clearly this is a less preferable fix!
  • How to release/renew DHCP | Netgate Forum - In Linux & Windows, it's easy to release the DHCP lease. In FreeBSD, the command dhclient -r is supposed to release the lease, but in pfSense, the -r option doesn't seem to exist. How is the lease released in the pfSense version of FreeBSD?
  • List leases by interface
    • I was told this was not possible, but at the bottom it lists counts by interface.
    • You can sort by IP which will approimate listing by interface as you should know what interface has the IP range on it.
    • You can also filter by IP.
DHCP Static Mappings / Static DHCP Leases (Static Leases)

This is a combination of DHCP and static IP addresses. You specify an IP address to be given out to a device when it requests a DHCP address ands this is done by using the MAC address.

There are many advantages to this but should not be used for all devices and circumstances.

The section in pfSense is located here:

  • Services --> DHCP Server --> [pick you interface] --> DHCP Static Mappings for this Interface
  • Services --> DNS Resolver --> General Settings --> (Static DHCP) Register DHCP static mappings: ticked
    • This is so your statically mapped DHCP clients are registered (this will not break pfBlockerNG - Unbound Python Mode)

Notes

  • Real static IP for servers, static DHCP mappings for portables devises such as laptops.
  • Static IP vs DHCP Reservation - The Tech Journal
    • In the many years I’ve been providing IT Services, I’ve noticed that whenever taking over a customer from a competitor, or providing consulting services for a company that has IT staff, that I don’t see DHCP reservations being used all that frequently. I wanted to write a post and create a video to discuss the comparison, when each should be used and the various case scenarios. I’m hoping my readers may provide their own input in the comments.
    • Excellent explanation with YouTube video
    • Read the comment for more insights.
    • you dont need to look further, this guy speaks from exporience and knowledge explaining everything.
  • It's mainly handy for something like a laptop. You would set a static assignment on the server and DHCP on the laptop. When local you can access the laptop via hostname, but when attempting to connect to another network you wouldn't have to switch back to DHCP on the laptop.
  • DHCP reservations - #2 by Acestes - Lawrence Systems Forums
    • This is what Netgate says:
      • Static Mappings Inside DHCP Pools While ISC dhcpd will allow a static mapping to be defined inside the DHCP range/pool, it can result in unexpected behavior....
    • Just assign your static IPs to one range and your DHCP to another range.
    • Issuing a static IP from your DHCP pool can cause issues.
  • What is Static Mapping? - Management, Networking, Logging and Reporting - UTM Firewall - Sophos Community
    • Static Mapping = Always assign an IP Address to specific MAC addresses and Prevent the DHCP Server from assigning it to other devices even if it is in the DHCP Range?
    • That is correct. A statically mapped IP in the DHCP server can be assigned to another client. Best practice is to assign the IP outside of the regular DHCP range and it will map correctly.
    • Bit different from the Windows Server implementation I normally work with where a 'reservation' is assigned to an IP/MAC pair and the IP is removed from the rest of the DHCP pool.
  • Static ARP
    • Why create a static entry in the Arp table? | Netgate Forum
      • When setting a static IP address for hosts on the network, there is an option for "Create an ARP Table Static Entry for this MAC & IP Address pair", sometimes I have checked it, sometimes I haven't. But I have never known whether it's best to check or not check it.
      • johnpoz
        • Can be used as a security measure. Can prevent spoofing and or poisoning, can allow for WOL (wake on lan), can shave a ms or so off from having to arp for the IP every 20 minutes or so - whatever your cache is set for. If your whole network is setup with static arps - would lower the amount of arp traffic on that L2 network.
        • Generally speaking the typical user would have no reason to set static arp entries up..
        • Can be used to limit who can talk to pfsense, via only allowing to talk to IPs that have static arp entries.
        • Con's would be that IP is locked to that mac - another device would not be able to use that IP, or that device would not be able to use a different IP.. Arp spoofing can be used for legitimate reasons - so if you have static arp set, you would not be able to do that.
        • But again most uses are outside the scope of day to day operation for a typical home network to be honest.. Its pretty safe to say if you don't understand its use case, you wouldn't have use of it ;)
    • Static ARP Entries on Ethernet Interfaces | Junos OS | Juniper Networks
      • You can provision static Address Resolution Protocol (ARP) entries for a device instead of dynamically resolving an IP address to a MAC address. Note that dynamic resolution of an IP address is the default behavior. These static ARP entries enable the device to respond to ARP requests even if the destination address of the request is not local to the Ethernet interface that receives the incoming traffic.
    • What Is an ARP Table? Address Resolution Protocol 101 | Auvik - Let's peer under the hood of how ARP tables help us see data similar to the "show arp" or "arp-a" command without having to open a terminal connection.
  • Client ID
    • When setting up a static DHCP lease, what's the difference between the Client ID field and the Hostname field? : PFSENSE
      • Client ID is used to identify the client. It’s an alternative to MAC address.
    • Why DHCP protocol needs Client ID? - Network Engineering Stack Exchange
      • A DHCP request from multiple hosts can be differentiated using Src Mac Address. So, why do need DHCP client ID to differentiate the DHCP request. I understand that they need some IP address because DHCP works on Application layer. So, that IP address can be 255.255.255.255 for the destination. For the DHCP client the src IP will be 0.0.0.0. However, the request can be differentiated easily with the mac address. Thus, they really don't need DHCP client ID?
      • DHCP defines a new 'client identifier' option that is used to pass an explicit client identifier to a DHCP server. This change eliminates the overloading of the 'chaddr' field in BOOTP messages, where 'chaddr' is used both as a hardware address for transmission of BOOTP reply messages and as a client identifier. The 'client identifier' is an opaque key, not to be interpreted by the server; for example, the 'client identifier' may contain a hardware address, identical to the contents of the 'chaddr' field, or it may contain another type of identifier, such as a DNS name. The 'client identifier' chosen by a DHCP client MUST be unique to that client within the subnet to which the client is attached. If the client uses a 'client identifier' in one message, it MUST use that same identifier in all subsequent messages, to ensure that all servers correctly identify the client.
      • In my opinion, this is absolutely mandatory to support DHCP relaying. In DHCP relaying, the src IP and destination IP's are changed. Also, the source and destination mac address. So, on receiving the dhcp messages from the dhcp server, the relay interface can't keep a track of the mac address. In fact, it will just forward the packet to broadcast IP with no accurate destination mac -address. With the help of client ID, the DHCP client can understand whether the packet is meant for them or not.
  • DHCP static mappings that don'€™t specify hostnames | Netgate Forum
    • Q: I wanted to use static mappings and let the client decide its hostname
    • A:
      • In short, it appears it is not possible to create the behavior I'm looking for. It's not terribly important to us, it just would've been nice.
      • The scripting based solution works very nicely, it just would've been nice if we had the option to assign an IP address via DHCP static reservation without having to assign a hostname in the static reservation. That creates a condition where, when I create static reservations, I have to remember to leave the hostname field blank for Windows machines (which will register their DNS hostnames directly with AD DNS) but specify hostnames in the reservations for non-Windows systems.
  • Static Mappings Inside DHCP Pools | pfSense Documentation
    • While the ISC DHCP daemon will allow a static mapping to be defined inside the DHCP range/pool in its configuration, doing so can result in unexpected behavior.
    • A static mapping entry in the ISC DHCP daemon is not a reservation and it does not remove that IP address from the pool.
    • The daemon only checks via ICMP ping to ensure that an IP address is not actively in use when making assignments.

Gateway Groups and Load Balancing (Optional)

Up until now we have dealt with single gateways as the single terminus/endpoint for your traffic exiting your network which is ok, but by using 'Gateway Groups' we can do the following:

  • Load Balance
    • We can spread traffic out over several gateways
    • See the official documentation about the specifics of this.
    • Quick answer = set all the Gateways you want to use to 'Tier 1'
  • Failover
    • If a specified gateway goes down the traffic can be invisible routed over a redundant gateway to keep the network flowing.
    • See the official documentation about the specifics of this.
  • Scraping
    • Because gateway groups allow you to use multiple IP endpoints (because of the multiple gateways) you can have a battery of IP address available for doing such things as webscraping without your IP being over used and then getting flagged.
    • Set a load of OpenVPN clients (with different locations if you want), set them all to 'Tier 1' and now every connection will probably use a different gateway and get a different IP allowing for better scraping and not getting flagged so quickly.
    • The more gateways the better for scraping.
    • If you need to scrape GEO-sensitive sites, make sure all of your OpenVPN endpoints are in the same country.
  • Extreme Privacy
    • By using multiple OpenVPN endpoints, maybe in different countries, your traffic will be almost impossible to track.
  • Geo-Location:
    • You create a 'Gateway Group' with many different OpenVPN locations and then select one county as an endpoint.
    • This is done by setting all to 'Never', then set the country you to 'Tier 1'
    • You can change your the country endpoint at any time easily withou having to change a load of firewall rules.
  • Gateway Alias
    • Instead of going through all of your firewall rules that have policy routing and then manually changing their gateway, you can use a 'Gateway Group' instead which works like an alias (but it isn't).
    • With this configuration you can change the gateway(s) within the 'Gateway Group' instead of having to manually change all of the rules where the gateway was specified/
Create a 'Gateway Group'

You can do this with one or more gateways that you have set up. If you want to have 5 Privacy VPNs which all point to different countries for instance you would need to create each of them as an interface outlined elsewhere in this article, however at this point you should have at least one setup (e.g. PIA_Switzerland).

  • System --> Routing --> Gateway Groups --> Add
  • Use these example details to create the group
    • Group Name: Privacy_Group
    • Gateway Priority:
      • Set all to 'Never'
      • Set 'PIA_SWITZERLAND' to 'Tier 1' and any other gateways you want to use in this group, perhaps other PIA VPNs
      • If you use this gateway group for 'Privacy Clients' make sure all the interfaces you enable are 'Privacy VPNs' otherwise your users will not have a private connection.
    • Link Priority: n/a
    • Trigger Level: Member Down
    • Description: Privacy Gateway Group
  • If you are converting a rule already there such as our example, you need to
    • Change all references to the PIA_SWITZERLAND gateway in the firewall rules to 'Privacy_Group' gateway

You can now maniplulate 'Privacy_Group' gateways as required such as adding further Gateways, changing their Tier settings or changing the group's purpose with having to visit each individual firewall rule.

Add additional OpenVPN Gateways into the Gateway Group

This has been dealt with in detail elsewhere in this article, but for reference this is a quick guide how how to get a OpenVPN connection present in 'Gateway Group' as a Gateway.

We will start at the beggining and add one new gateway into the 'Privacy_Gateways', should you want more modify the instructions as required.

  • Create a new OpenVPN client
    • VPN --> OpenVPN --> Clients --> Add
  • Assign the OpenVPN client to an interface
    • Interfaces --> Assignments --> Add
    • Enable
      • Block private networks and loopback addresses
      • Block bogon networks
    • By default none of these new interfaces will have any firewall rules which is ok, therefore all unsolicited incoming traffic is blocked which is ok.
  • Convert interfaces to Gateways
    • This is done automatically when you assign the OpenVPN client to the interface
    • Once you have setup one gateway, utilise the copy button  it will save you a lot time
    • Edit and save each gateway as this applies hidden settings (gateway goes from '' to 'dynamic')
  • NAT Mappings
  • Add the interfaces to 'Privacy_Group' gateways
    • System --> Routing --> Gateway Groups --> 'Privacy_Group' --> edit
  • Kill Switch
  • Allow Privacy Policy Clients to see LAN servers Floating Rule (LAN interface assigned)
  • pfBlockerNG
    • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration --> Inbound Firewall Rules
      • Treat the new OpenVPN Gateways as just that, Gateways. They should be selected in the 'Inbound Firewall rules'
      • Add the individual interface (with an individual interface we can specify the type of connection it is.)
      • Save
      • Force Reload
  • Update Custom DNS Firewall rules
    • Firewall --> Rules --> Floating
    • Add the individual interface to each Custom DNS rule as appropriate, should be all but 'Allow Web Server DNS'
    • In this case just making sure the 'OpenVPN' interface is selected in the Custom DNS rules will do, as we do want all OpenVPN connections to follow these rules (Clients and Servers).
  • Reload Filter Rules
    • Status --> Filter Reload
  • Update 'Service Watchdog'
    • If you want to make sure a particular interface (OpenVPN client) is brought backup after failing, then you need to set it in this section.
    • This assumes you have installed 'Service watchdog'
How to use a 'Gateway Group'
  • Just configure the 'Gateway Group' into the particular mode you want. These modes are explained at the beginning of this section.
  • They can be used in 2 places, maybe more, but these are the obvious ones.
    • Assign 'Gateway Groups' to Firewall rules instead of using an individual gateway as required.
    • These groups can be assigned as the default gateway
      • System --> Routing --> Gateways --> Default gateway -->
        • Default gateway IPv4:
        • Default gateway IPv6:
To change Gateway Group name (not official)

I have not tested these

  • Option 1
    • Backup your config.
    • Duplicate the gateway group and set the new name.
    • Then go through all of the firewall rules, and where the the old gateway group is present, swap out for the new gateway group.
    • Check the Default WANs do not use this gateway group.
      • System --> Routing --> Gateways --> Default gateway -->
        • Default gateway IPv4:
        • Default gateway IPv6:
    • Do another backup and then scan it for the old rule name to make sure you have got all references (dont forget there should be one reference of the old name which is the actual rule definition).
    • Once all rules changes, delete the old rule
  • Option 2
    • Backup your config.
    • Text replace the old interface group name with the new one.
    • Restore the modfied config back to your router.
  • Option 3
    • You cannot directly rename a 'Gateway' or 'Gateway Group' you have to copy them, everything else such as interfaces you can just rename them.
    • Example - rename PIA_SWITZERLAND gateway to WIZARD_UK_VPN4
      • System --> Routing --> Gateways --> PIA_SWITZERLAND_VPNV4 --> Copy gateway
        • Set the new name and description
        • Save
      • System --> Routing --> Gateway Groups
        • This section is only required if the Gateway is part of a 'Gateway Group'
        • Edit the 'Privacy_Gateways'
          • set PIA_SWITZERLAND_VPNV4 to 'Never'
          • set WIZARD_UK_VPN4 to 'Tier 1' PIA_SWITZERLAND_VPNV4
      • System --> Routing --> Gateways
        • Delete the PIA_SWITZERLAND_VPNV4 gateway
Notes

Create an 'Interface Group'

This feature allows you to groups interfaces together and use the new group in firewall rules juts like aliases but for interfaces instead. A good example of why you would use this is if you have a lot VPN gateways which can be grouped togther to make referencing them easier, such as Privacy_Group and Scraping_Group.

When making these groups it might be worth keeping partity of the names with any releated gateway groups you have created (System --> Routing --> Gateway Groups), such as the Privacy_Group gateways, but you are free to use whatever names you want, just remeber the names cannot be changed at a later date, officially, but there might be workarounds.

You should not group WANs together using this method as it can cause issues. In this instance I believe a WAN is a physical connection/route from your network to another not such things as VPNs that go out over the WAN. Read the offical documentation for more information.

Instructions

We will create a privacy interface group to keep with our gateway group we created earlier.

  • Interfaces --> Assignments --> Interface Groups --> Add
  • Configure the 'Interface Group Configuration' as follows:
    • Group Name: Privacy_Group
    • Group Description: Privacy Interface Group
    • Group Members: PIA_SWITZERLAND...
      • And all of the others you specified in the Privacy_Group gateway.
      • Parity of members is not required, but for this tutorial it is.
  • Save

Now when you create or edit firewall rules you can use this interface group instead of specifying each interface manually. You can alos add and remove interfaces with ease.

To change Interface Group name (not official)

I have not tested these

  • Option 1
    • Backup your config.
    • Duplicate the interface group and set the new name.
    • Then go through all of the firewall rules, and where the the old interface group is present, swap out for the new interface group.
    • Do another backup and then scan it for the old rule name to make sure you have got all references (dont forget there should be one reference of the old name which is the actual rule definition).
    • Once all rules changes, delete the old rule
  • Option 2
    • Backup your config.
    • Text replace the old interface group name with the new one.
    • Restore the modfied config back to your router.
Notes
  • Interface Groups | pfSense Documentation
    • An Interface Group is not a type of interface that can be assigned.
    • Interface groups are used to apply firewall or NAT rules to a set of interfaces on a common tab.
    • The best practice is to not use interface groups with multiple WANs. Doing so may appear to be convenient, but the group rules do not receive the same treatment as actual WAN tab rules.
    • For example, rules on a tab for a WAN-type interface will receive reply-to which allows pf to return traffic back via the interface from which it entered. Group tab rules do not receive reply-to which effectively means that the group rules only function as expected on the WAN with the default gateway.
  • [pfSense] How network interfaces work – Provya
  • Interface Group turns out to be ^Rule Group^! Which is awesome !! | Netgate Forum - And guess what, a couple of days back I discovered that ^Interface Groups^ are not at all ^Interface Groups^, but it are in fact ^rule-groups^.
  • pfSense - Isolate vLANs with an Interface Group - Rocky Mountain Tech Team - The Use CaseOne pfSense with multiple vLANs that need to be locked down or isolated from each other. Also, need to block many vLANs from being able to access the pfSense web interface.

Disable Services on Boot (workaround)

This section is using OpenVPN as an example but the Shellcmd can be used for most services. This workaround stops the services after they have started which will do for most and prevents editing the system files manually.

  • Install Shellcmd package
    • Package Manager --> Available Packages --> Install
    • The shellcmd utility is used to manage commands on system startup.
  • Add a boot command to disable the OpenVPN services
    • See the next section where you get the Client/Server and 'OpenVPN ID'
    • Services --> Shellcmd --> Add
      • Command: 
        pfSsh.php playback svc stop openvpn server 1
or
pfSsh.php playback svc stop openvpn client 3
      • ShellcmdType: shellcmd
      • Description: Disable my OpenVPN on boot
  • Repeat for each OpenVPN service you want to disable

Notes

  • Disable OpenVPN clients on reboot | Netgate Forum
    • My Thread
    • This is my post and might have some other ideas when you read it.
  • The Shellcmd do not seem to work until you have rebooted once (it fails), then it works everytime after. This is the case for the OpenVPN commands above, but not tested on any other command.

Shell / Command Prompt / CLi

There are 2 types of shell

  • Shell
    • This is the standard Linux command line
    • You can run PHP-Shell stuff by invoking pfSsh.php before the commands
  • PHP-Shell
    • This is a pfSense shell environment for PHP commands.
    • This is for accessing the pfSense software rather than the hardware and underlying Linux system.

Notes

Hardware

Troubleshooting (pfSense)

Locked out of pfSense
  • Official
  • WebGUI / webConfigurator
    • An administrator can (very temporarily) disable firewall rules by using the physical console or SSH.
    • To disable the firewall, connect to the physical console or ssh and use option 8 to start a shell, and then type: 
      pfctl -d
    • That command will disable the firewall, including all NAT functions. Access to the GUI is now possible from anywhere, at least for a few minutes or until a process on the firewall causes the ruleset to be reloaded (which is almost every page save or Apply Changes action)
    • Once the administrator has adjusted the rules and regained the necessary access, turn the firewall back on by typing: 
      pfctl -e
  • Disabled Admin Account (other methods)
    • Accidentally disabled Admin account, what are my options? | Netgate Forum - Common scenario and perfect fix
      • Q:
        • Hello, in making a few security tweaks to my pfSense box I decided one of the things I would do is use a non-standard admin account to help protect myself from a brute force attack. I created a new user, gave it a password, added it to the admins group, clicked save, then disabled the admin account via the GUI. I then attempted to log in as my new user, and no luck it tells me no pages have been assigned to this user. I try logging into SSH with my new user and it closes the connection as soon as I log in. Logging in as admin tells me my user/pass is incorrect at the GUI but lets me into the shell. I attempted to reset the admin account via the shell (option 3 in the menu) and it tells me it was successful, however I am still unable to log in.
        • Unfortunately I don't have a backup of my config so I was hoping to avoid restoring to factory defaults, what can I do to get back into the web interface?
      • A:
        • From the physical console, reset the password.
    • Disabled Admin Account & Locked Out | Netgate Forum
      • Q:
        • I created a new user, assigned privileges, then disabled the admin account and now the user I created is not working.
        • I tried to login several times with the wrong password and Web Admin blocked me at the firewall.
        • I have SSH access for some reason: How can I fix the admin account and delete the firewall rule locking me out?
      • A (Gertjan):
        • What about SSH in and look (make a save copy and edit) /cf/conf/config.xml
        • Find the key <auth_method>
        • I have <auth_method>local</auth_method>
        • Change, save, reboot.
      • A (tacfit):
        • I learned also that to do it easier, you can just type "viconfig", which will load the config, and upon closing it will kill the cached config file, causing PFsense to reload the new config. Means you don't have to reboot.
pfSense will not boot
  • If you set the Partition Scheme: GPT (UEFI) and the pfSense does not boot then it could be the PC BIOS is not able to handle the EFI partition even though it says it can.
  • Clean install of 21.02 on ZFS filesystem not booting | Netgate Forum
    • This is an old bug and might not be a thing anymore.
    • At the end of the install open the terminal and run this command 
      echo 'zfs_load="YES"' >> /boot/loader.conf
  • UEFI can be flaky on older PCs so the only choice that might work is BIOS/Legacy/CSM mode
  • Partition Scheme: GPT (UEFI) did not work on my PC : created a GPT disk with a EFI partition etc..
  • Partition Scheme: GPT (BIOS+UEFI) + I had to turn on legacy booting : created a GPT disk with a EFI partition etc..
Block IPv4 link-local (1000000102) and Block IPv4 link-local (1000000101)
  • You can disable logging of all default pfsense firewall rules (Not recommended)
    • Status --> System Logs --> Settings (tab) --> Log firewall default blocks
  • Noob question: Block IPv4 link-local (1000000102) and Block IPv4 link-local (1000000101) | Netgate Forum
    • Q: I was inspecting my logs to see if my basic setup was working and I keep seeing Block IPv4 link-local (1000000102) and Block IPv4 link-local (1000000101) in the system log. 
    • A: jimp (Netgate)
      • Blocking ipv4 link local, so you mean 169.254 - yeah those are APIPA address when dhcp client didn't get an IP.. Some iot devices will use them as well even when they have a IP... Not a fan myself.. My dvr bridge from directv does that.
      • Yes a firewall is going to block some noise that many users are not really aware was there when they used their soho that doesn't show them any logs of any kind, etc. Or just scares them with stopped attack without any details, etc.
  • Block IPv4 link-local | Netgate Forum
    • Q: Is there any way to disable just the "Block IPv4 link-local" from reporting to the firewall log!
    • A: A real dirty hack and not recommended. This might not work on newer version of pfSense
      $config['system']['no_apipa_block'] = true; 
write_config();
pfsense is blocking Autoconfiguration IPv4 Address
  • pfsense is blocking Autoconfiguration IPv4 Address why ? | Netgate Forum
    • A: jimp (Netgate)
      • They are blocked because they have no business hitting the firewall. They can still communicate locally, but it's link-local/L2 traffic that can never route outside of its segment.
      • You'd only receive one of those IP addresses if you had no DHCP service running. You can't use those addresses as your local network the same way you would use an RFC1918 network.
How to mount a ZFS drive in windows
SWAP usage keep going to 100% (not all my RAM is used)

The image below is self explanatory.

Solution

  • Since I changed the swap file from 1GB (default) to 16G (recommended / twice the RAM) the swap file has not filled up.
  • I also upgraded to 2.7.2 at the same time, but I am not sure if it is this that fixed things, you never know though.
  • Make sure your hypervisor is upto date and it's settings are correct
  • Rebooting all of your devices might help.
  • Updating all related spftware is also a must as issues do get patched.
  • Turn off your hypervisor dynamic memory option (if present and enabled).
  • snort and squid settings are causing them to consume massive amounts of memory.

Diagnostics

  • You can also check what is using all of the RAM. Run this command from SSH not the GUI to to get the biggest memory eater at the top:
    top -n -o res

or try

htop
  • System Activity (Top) | pfSense Documentation
    • The Diagnostics --> System Activity page displays several aspects of system activity as reported by top which are updated every few seconds.
    • This is equivalent to running the command top -aSH at a shell prompt.
  • Linux Find Out What Process Are Using Swap Space - nixCraft - Linux Find Out What Process Are Using Swap Space - How to Get swap space usage and process that are using it in Linux?
  • How to install htop on pfSense firewall - TrySitePrice - Here is how to install top on pfSense: Top is part of pfSense now.
    1. Log in to the pfSense web interface
    2. Go to System -> Package Manager -> Available Packages
    3. Search for top and select it
    4. Click the Install button
    5. Wait for the installation process to complete
  • How to install htop on pfSense firewall - nixCraft
    • This tutorial explains how to install htop process viewer on a pfSense firewall using pkg command line option over the ssh based session.
    • I have done this and it is safe.

Notes

  • PfSense uses 100% of swap but only 10% of memory. | Netgate Forum
    • Ok I managed to solve the issue. The problem was that in the VM Hypervisor (Proxmox) I accidentally enabled „ballooning“ memory (which means it will remove memory on the fly if it detects that the VM is not using it).
    • I disabled it and now it all works like a charm. I guess the GUI (and myself) was confused that the RAM was removed from the VM while running, therefore showing memory usage „of 11GB“ while in reality it only had way less memory at this point.
  • PfSense uses SWAP even though memory is not even used 10%. Is htop wrong? Is the GUI wrong? Is Pfsense broken? | Reddit
    • I'm sorry i don't know much about proxmox, does that hypervisor have a dynamic memory option that may be tightening the virtual memory artificially? If so try turning that option off and see if it helps. I've seen some situations when this causes issues on Hyper-V also. Not specifically with pfSense but other apps.
    • OMG I did indeed enable a „ballooning“ memory option for this VM (for whatever reason, I forgot). Disabled it and now it works!! It now only consumes 4GB and no service crashed! Thank you very much for pointing me in the right direction!
  • Not All Swap Usage is Bad - Memory Management | pfSense Documentation
    • Given that knowledge, one might assume that any usage of swap is bad and should be avoided at all costs, however, that is not universally true.
    • Swap consumption generally only becomes a concern when there is significant usage, such as near 50% or higher. At that point it becomes necessary to inspect everything consuming memory to see what can be reduced or tuned.
  • PFSense High Swap Usage | Netgate Forum
    • More than likely your snort and squid settings are causing them to consume massive amounts of memory. There is not enough information in what you have shown to speculate about a cause with any accuracy. Post a full "ps uxawwd" output for starters.
  • Snort and Memory Usage / Swap Usage Concerns | Netgate Forum
    • Snort eats memory like nothing else especially if you don't take steps to prevent it from doing so. In that respect your increased memory usage is completely normal. However you do not, ever, want to be swapping. It will bog down the performance dramatically. If you're not seeing that then it's likely the swap usage was not a continuous thing but it shows you are the edge of your ram requirement. You should probably either take steps to reduce Snorts footprint or add more ram.
  • Swap ON or OFF in PfSense | Netgate Forum
    • Unless you have specific concerns about space or drive writes etc just accepting the defaults is fine.
  • In general pfSense should never use SWAP and of you see it swapping it's usually because something is misconfigured.
  • I still have some test systems that run from CF and on those I always disable SWAP because of the limited write cycles there.
Why do I only have 1gb of swap? / Increase Swap Size
  • The swap file size is set during the installation. IT is a swap file partition and defaultts 1GB.
  • It is recommended as a rule of thumb to have twice your RAM as SWAP.
  • swap size: The amount of disk space dedicated to swap space (virtual memory). This is optional. Commonly set to 2x the available RAM in the firewall, but with smaller disks that may be too large.
  • The SWAP is also used for kernel dumps in emergencies. and this is why you need at least more SWAP than RAM to allow for a complete dump.
  • Increase swap size | Netgate Forum
    • Q: On to the real question. During the fresh install, I forgot to adjust the swap size. Last time I used the recommend twice the ram size for my swap. I have plenty of space on my SSD for it. Since I forgot to do it during the install, is there another way to adjust it? I couldn't find anything in the book or on this forum about that. It's not important, but just wondering more than anything. I'm going to leave it alone otherwise.
    • A: You'll have to reinstall to change the size of the disk partitions. Even if you could adjust it with a bunch of different commands, the process would be long and prone to error. You can take a backup, reinstall with the right options, and be back up in minutes.
    • Generally speaking you should not be using SWAP so the size is not that important. If you are using SWAP space it's probably because you don't have enough RAM for what you're trying to do or something is misconfigured.
Slow Network
  • This could be caused by a faillin switch, powerline adapter or network card and so on. You need to power cycle through all of your network infrastructure to find out what device(s) is causing the issue. Sometimes a reboot is all that is needed.
  • For me it was one of my Powerline adapters, even though it showed it was working fine, it obviously wasn't. As soon as I rebooted the offending powerline adapter the rest of the network went back to normal.
Twitch will not work
  • This will be because of my strong blocking configuration.
  • This is how I figured it out:
    • The DNS request is successful.
    • A packet is sent to an IP
    • and that IP is blocked.
    • This means that only an IP block will be found with no coressponding DNS block
  • 146.75.74.167 / twitch.tv | Shodan - Ports open: 80, 443
pfSense not routing but console is working / PPPoE won't reconnect after link loss

Background

My pfSense router is connected to an ADTRAN SDX611B/D ONT from Openreach and pfsense uses PPPoE to control it.

The issue

Every couple of weeks (the length varies) and seems to be in the early hours of the night, I wake up to find my pfSense router is:

  • No longer routing to internal and external networks
  • Cannot ping pfsense (10.0.0.1)
  • WebConfigurator (GUI) is not available
  • Console via TruenAS (pfSense is virtualised)
    • This is available
    • WAN is down
    • LAN shows an IP
  • VPN gateways
    • Cannot ping them
    • Not all gateways have lost their IP but I still cannot ping those

Diagnostics

  • Get the exact time the failure happened.
    • I use Uptimerobot so this is easy.
  • Check the ONT status
  • Check pfSense GUI/TrueNAS available
  • Check pfSense Console
    • If you look at the console you can see some useful messages such as the one blow. I believe each line below corresponds to a PPPoE dial up packet.
  • Check TrueNAS is available via IP
  • Check other TrueNAS VMs are avaiable from their IP such as your webserver.
  • Look at the logs for
    • TrueNAS Virtual Machine (optional)
    • pfSense System Logs (when pfSense is running is easier) 
      /var/log/system.log
/var/log/system.log.0
/var/log/system.log.xxx

I also tried some fo the options from the console and they did not change the status of the router:

  • Restart PHP-FPM
  • Restart webConfigurator
  • Ping Host

NB: Restarting the router form the console does work.

Explantion

  • If the NIC has failed/locked for whatever reason, this explains why there is no routing.
  • The console is available becasue it does not rely on the network and is directly accessed at the hardware level (i.e. keyboard and monitor).

Cause (not confirmed)

This issue does not seem to be an issue with the TrueNAS Virtual Machine, it is purley a pfsense issue.

Bear in mind you might have one or more of these issues, for instance if your firmware update fails your internet could be off for more than an hour.

  • ONT
    • Firmware update
      • This causes the pfSense to become disconnected from the internet.
      • pfSense will keeps retrying to reconnect but will always fail, so this could be an issue with the ONT.
    • IPv6 Support
      • The ONT might not support IPv6 correctly.
    • Is the ONT crashed out by the constant dial up requests after the initial drop?
  • pfSense
    • Long internet outage bug
      • The WAN goes down and then if the WAN is not re-connected within an hour or so this can cause the router to crapout (as described above).
      • Example reconnection log lines 
        Oct  4 00:36:28 pfs ppp[87560]: [wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Oct  4 00:36:28 pfs ppp[87560]: [wan] IFACE: Down event
Oct  4 00:36:28 pfs ppp[87560]: [wan] IFACE: Rename interface pppoe0 to pppoe0
Oct  4 00:36:28 pfs ppp[87560]: [wan] IFACE: Set description "WAN"
Oct  4 00:36:28 pfs ppp[87560]: [wan] IPCP: Down event
Oct  4 00:36:28 pfs ppp[87560]: [wan] IPCP: LayerFinish
Oct  4 00:36:28 pfs ppp[87560]: [wan] Bundle: No NCPs left. Closing links...
Oct  4 00:36:28 pfs ppp[87560]: [wan] IPCP: state change Closing --> Initial
Oct  4 00:36:28 pfs ppp[87560]: [wan] Bundle: Last link has gone, no links for bw-manage defined
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] LCP: SendTerminateAck #25
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] LCP: LayerDown
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] PPPoE: connection closed
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] Link: DOWN event
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] LCP: Down event
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] LCP: state change Stopping --> Starting
Oct  4 00:36:28 pfs ppp[87560]: [wan_link0] Link: reconnection attempt 1 in 4 seconds
Oct  4 00:36:32 pfs ppp[87560]: [wan_link0] Link: reconnection attempt 1
Oct  4 00:36:32 pfs ppp[87560]: [wan_link0] PPPoE: Connecting to ''
Oct  4 00:36:41 pfs ppp[87560]: [wan_link0] PPPoE connection timeout after 9 seconds
Oct  4 00:36:41 pfs ppp[87560]: [wan_link0] Link: DOWN event
Oct  4 00:36:41 pfs ppp[87560]: [wan_link0] LCP: Down event
Oct  4 00:36:41 pfs ppp[87560]: [wan_link0] Link: reconnection attempt 2 in 2 seconds
Oct  4 00:36:43 pfs ppp[87560]: [wan_link0] Link: reconnection attempt 2
Oct  4 00:36:43 pfs ppp[87560]: [wan_link0] PPPoE: Connecting to ''
Oct  4 00:36:52 pfs ppp[87560]: [wan_link0] PPPoE connection timeout after 9 seconds
Oct  4 00:36:52 pfs ppp[87560]: [wan_link0] Link: DOWN event
Oct  4 00:36:52 pfs ppp[87560]: [wan_link0] LCP: Down event
      • This is an old bug but might still be an issue.
    • Network Card (NIC) can lock up, this explains the network is down but the console can still be accessed.
  • pfBlockerNG
    • The updates can apparently bring the WAN down, but this might be an old bug.
  • FreeBSD
    • Might have issue with PPPoE

Solutions

  • Remove network cable from the ONT and put back in.
  • Reboot/power cycle the ONT.
  • Reboot pfSense
    • This might have the same effect as removing the ethernet cable from the ONT.
  • Update pfSense to the latest version
    • this can resolve bugs and driver issues.
  • Swap the network card
    • Card might be faulty
    • Realtek NICs have a known fault where then can lock up
    • Make sure the swerver is kept cool to prevent the NIC overheating.
  • Setup a CRON to automatically reboot the pfSense router
    • This is not recommend because you should never have to reboot pfSense and so you are just hiding the issue you are trying to fix rather than fixing it.
    • Useful if you are going on holiday and do not have time to fix a particular issue

Notes

  • Bug #13092: PPPoE WANs fail to reconnect after parameter negotiation failure - pfSense - pfSense bugtracker
    • After a six hour ISP outage, the service was restored but pfSense didn't automatically re-establish the PPPoE connection and a reboot of the router (Netgate 1100) was required to re-establish. Shorter outages don't seem to affect automatically reauthenticating.
  • PPPoE WAN fails to reconnect after link loss | Netgate Forum
    • I'm using pfSense 2.4.3 with em Intel network interfaces and my WAN is PPPoE based. When the connection drops for any reason, pfSense does not reconnect until a full system reboot is done.
    • The same WAN in a domestic router (e.g. TP-Link) reconnects just fine. I found a similar issue on Redmine but it is from six years ago.
    • I have to reboot the whole pfsense box. Then it gets connected automatically. But I have tried restarting the nic it does not help.
    • The threas has some great diagnostic ideas.
  • Bug #1943: PPPoE won't reconnect after link loss when using vr(4) NICs on certain ISPs only - pfSense - pfSense bugtracker
    • We got a wan interruption (they cut the cable while doing road works), and after everything was reconnected the pppoe interface doesn't come up automatically (wait many minutes).
    • Disable-Enable the WAN interface doesn't help, and I restored the pppoe connection with this command: /usr/local/sbin/pfSctl -c 'interface reload wan'
  • PPPOE randomly disconnect on PfSense, have to reboot to reconnect | Reddit
    • I just moved to PfSense as main router. Now I suspect some issue that WAN (PPPoE) randomly disconnect without any symptoms. There's no way to "reconnect", only the reboot helps.
    • I had a similar issue when I first setup pfsense 2 weeks ago. pfsense pings the ISP gateway address every second or so to determine whether the connection is still working or not. Turns out the gateway of my ISP didn't respond to pings so after several attempts, pfsense initiates a reconnect. After doing this several times it would just fail to connect at all until rebooted. Finally found the solution that worked for me in the forum's for my ISP. I went to to System -> Routing, clicked the "Edit" icon next to the WAN PPPoE gateway and changed the "Monitor IP" to something like Google DNS 8.8.8.8 (or any other reliably online IP address that accepts pings). I haven't had any trouble since.
    • I just had this exact same issue except on internal LAN. Like clockwork, once an hour i would completely loose internet connection and then it would come right back up. It was a hard drop. It ended up being pfblockerng pulling down updates and refreshing. I changed the schedule from once an hour to 3am. I didn't need the update to be that aggressive anyways. Perhaps this is what your issues is as well. GL!
  • Pfsense on Hyper-V PPPoe Reconnect Issue | Reddit
    • Q:
      • My dedicated pfsense pc died today so I decided to finally virtualize it in Hyper-V on my Server 2019 machine. I've got it up and running no problem but I've encountered an issue when the host restarts. The machine has two NICs, set as separate virtual switches in Hyper-V. WAN switch has host OS access disabled and all hardware offloading is disabled on both switches.
      • Issue is: when the host restarts and PFsense comes back up, the PPPoE connection is not re-established. Going into the PFsense GUI and disabling then re-enabling the WAN interface brings it back up again.
    • A:
      • Do your PPPoE logs show an empty PADO response?
      • I've seen this randomly for *years* when using PPPoE. (edit, it's not specific to HyperV) AFAIK it's an issue with FreeBSD upstream, and not something which has ever been fixed. For whatever reason the netgate devs ignore any thread on the forum discussing it too... Indeed, disabling / enabling the interface, or rebooting, fixes it.
      • It's an horrific bug. I spent some time with my ISP attempting to diagnose it, indeed, sniffing the interface shows a
      • I've totally given up on PPPoE in pfsense, YMMV.
  • PPPoE reconenction fix - 2023 | Netgate Forum
    • PPPoE via a Openreach ONT has been a headache for me with dropped connections, crashing on interface changes and multiple attempts to achieve a PPPoE handshake.
    • In my case this was linked to IPv6 being enabled (as it should be) and a race condition with v23.01. The patch applied in the v23.05 update appears to have stopped all the symptoms.
    • The only thing I could test changing interface details and bringing the PPPoE down and up again. Early days for sure but to date it has been rock-solid and the ppp log file looks nice and healthy.
  • PPPoe reconnect issue | Netgate Forum
    • I have an issue when the router looses WAN connection on the pppoe link, the link does not automatically reconnect.
    • I have to unplug the cable to the fibre ONT several times before it reconnects or I have to run this command several time to re-establish the connection: 
      /usr/local/sbin/pfSctl -c 'interface reload wan'
    • Is there any missing config for auto reconnect?
  • Every couple of weeks pfSense completly stops responding? | Netgate Forum
    • Q: I remember that pfSense was not responding to pings nor I could get GUI to respond.
    • A:
      • Both those things would happen if the LAN NIC locked up as Realtek NICs sometimes do.
      • But that would not stop the console responding and that's an important test. In the console is still responsive then you know you have a NIC issue. It is isn't you probably have some other hardware issue, bad ram overheating etc.
  • iflib_netmap_config | Netgate Forum
    • Q: I have been getting this message on reboot: 586.587322 [857] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
    • A:
      • Those are harmless information messages from the netmap kernel device during its initialization. It is printing out the current configuration of netmap based on the NIC driver being used. It is detailing the number of rings, TX/RX descriptors, and the buffer size that will be used.
      • There is no way to stop the message. It comes from the kernel device itself. Nothing pfSense nor any packages can do about it.
Crash debugging / Using the logs

pfBlockerNG

General

  • Official Websites
  • Guides
  • pfBlockerNG vs PiHole
  • Interfaces
    • Packages — pfBlockerNG Package | pfSense Documentation
      • Set the interfaces to be monitored by pfBlockerNG (both inbound and outbound), where the inbound is the Internet connection.
      • To prevent devices or users from accessing sites in the selected countries/IP addresses, select local interfaces under outbound.
    • pfSenseNG does not support Interface Groups as you need to configure the individual network interfaces as In or Out.
    • pfBlockerNG on interface groups | Reddit
      • my current pfSense rule setup uses interface groups in order to define rules for a number of subnets and VPNs that belong together (e.g. an "Office LAN" interface group that allows 443/80 to the internet and denies anything else).
      • The problem here is that I can't apply pfBlockerNG rules to an interface group. Since the rule processing order is "Floating" -> "Group Rules" -> "Interface Rules", traffic from the Office LAN group to the Internet (Port 80/443) hits the "allow" rule in the Group rules first, and the pfBlockerNG rules are never evaluated.
      • The way I understand it, I'm currently forced to use Floating Rules, which is too generic for me. Is there another way around it? Is pfBlockerNG considering supporting Interface Groups as targets for auto rule creation?
  • Blocks Lists / Feeds
    • IPv4 Blocklists focus on attacks (Inbound) and security where as DNSBL focus on privacy and advertising (Outbound). They are not mutually exclusive though.
    • Look at selecting your preference of IPv4 and DNSBL blocklists but do not select them all.
    • To see the what IPs a pfBlockerNG rules is actually  blocking, goto your firewall rules and hover over the alias on a firewall rule.
      • Firewall --> Rules --> LAN --> Destination: pfB_PRI_v4
      • This modal window is limited in the amount of records it can show.
    • The default PRI1 - Collection is pretty good for stopping ADs
  • Reports and Stats
    • Firewall --> pfBlockerNG --> Reports
      • Reports Homepage
    • Firewall --> pfBlockerNG --> Reports --> Alerts
      • See blocks happening here
      • This will show actual blocks and what Blocklist stopped it
  • Logs
    • /var/unbound/var/log/pfblockerng/
  • Upgrading pfBlockerNG
  • How to change pfBlockerNG Branch
  • Using Interface Groups
    • Currently pfBlockerNG cannot utilise 'Interface Groups'
    • pfBlockerNG on interface groups | Reddit
      • The way I understand it, I'm currently forced to use Floating Rules, which is too generic for me. Is there another way around it? Is pfBlockerNG considering supporting Interface Groups as targets for auto rule creation?
    • Feature #14100: Use interface groups as an Alias for IP Interface/Rules Configuration - pfSense Packages - pfSense bugtracker
      • I have asked here to see if the 'Interface Groups' can be used as an alias for the 'IP Interface/Rules Configuration'
    • I would call my groups
      • Name: Privacy_GWIG
      • Description: Privacy Gateway Interface Group
    • What I did so it would fit in with my instructions (if it worked)
      • Create interface
      • Add in the interface group
      • Add in the gateway group
      • update the pfBlockerNG 'IP Interface/Rules Configuration' assignments
        • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration
      • Interface groups were not present and rebooting did not work
    • I wanted to move all of my OPENVPN Gateway Interfaces into a group and just alter the group rather than changing assignments all the time, just like alias.
  • Ability to use pfBlockerNG Python mode and Unbound (FIXED in 2.7.0)
    • 2.7.0 New Features and Changes | pfSense Documentation
      • A long-standing difficult-to-reproduce crash in Unbound during reloading has been addressed. Christian McDonald tracked down the source of the Unbound SIGHUP crashes to a reference counting bug within the MaxMindDB Python module. Both a patch to MaxMind and a port revision to FreeBSD ports were submitted and accepted, and the fix is included in the 2.7.0 release.
      • It is now safe again to enable DHCP registration alongside Unbound Python mode in pfBlockerNG.
      • In addition to the Unbound crash, Christian also identified a memory leak with DHCP registration and Unbound Python mode (#10624). This is largely mitigated by updates to Python and related libraries, but there is additional ongoing work to resolve it further for future release.
    • Releases — 23.01 New Features and Changes | pfSense Documentation
      • A long-standing difficult-to-reproduce crash in Unbound during reloading has been addressed. Christian McDonald tracked it down the source of the Unbound SIGHUP crashes to a reference counting bug within the MaxMindDB Python module. Both a patch to MaxMind and a port revision to FreeBSD ports were submitted and accepted, and the fix will be included in the 23.01 release. It is now safe again to enable DHCP registration alongside Unbound Python mode in pfBlockerNG.
      • In addition to the Unbound crash, Christian also identified a memory leak with DHCP registration and Unbound Python mode (#10624). This is largely mitigated by updates to Python and related libraries, but there is additional ongoing work to resolve it further for future release.
    • sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak | Netgate Forum
      • This has been addressed in the latest snapshots.
      • We are testing the changes and will include them in 23.01 which is due soon.
    • Register DHCP leases in the DNS Resolver will work again in pfSense+ 23.01 | Reddit.
      • BBCan177
        • With the work that Christian has committed so far, it should allow for the Unbound Python mode and the DNS Resolver Register DHCP leases to be used together. With these changes, it will still reload Unbound. It will cause a short DNS blackout until it's reloaded, but python mode is quicker than Unbound Mode. It is best to add as many Static DNS entries as possible and set long lease times.
        • The second Redmine, as Christian stated, will take some more development, but that will use Unbound-control to add/remove dhcp lease changes without needing to Reload Unbound and having any blips in DNS coverage.
    • BBcan177 | creating pfBlockerNG: A Firewall - IP and Domain Management pack | Patreon
      • For pfSense 2.6, there is still a restriction for DHCP Registration and DNSBL Python mode. To overcome this restriction, you will need to migrate to pfSense CE or pfSense + due to python compatibility issues
      • There is a typo above. This is not fixed on the CE version of pfSense yet.
    • Is it Possible Now? pfBLockerNG-devel in Unbound python mode with DHCP Registration using pfSense 2.6.0 : pfBlockerNG
      • Still unresolved
    • Python DNSBL mode is not compatible with the DNS Resolver DHCP Registration option (Unbound will Crash)!
      • From the code: DNSBL Python mode is not compatible with dhcpleases binary code, as it attempts to HUP the Unbound PID and will cause DNSBL Python mode to crash Unbound.
      • BBCan177
        • The DNS Resolver (Unbound) DHCP Registration option is not compatible with DNSBL Python mode. The pfSense devs are aware and changes are required to be made to the dhcpleases binary to stop/start Unbound instead of sending a SIGHUP. The use of this option and the Unbound Python mode will cause an Unbound crash.
        • If DHCP Registration is enabled in Unbound Python mode, or DHCP Registration enabled after Unbound Python mode is enabled, Unbound Python mode will be downgraded to Unbound mode to prevent Unbound from crashing.
        • I am not sure when the issue with DHCP Registration will be fixed. The pfSense devs have to modify the dhcpleases binary to stop/start Unbound instead of sending a HUP command which causes a "reload" and leads to a crash in Unbound with the Python mode enabled.
      • This option: Services --> DNS Resolver --> General Resolver Options --> DHCP Registration
      • If this option is set, then machines that specify their hostname when requesting an IPv4 DHCP lease will be registered in the DNS Resolver so that their name can be resolved. Note that this will cause the Resolver to reload and flush its resolution cache whenever a DHCP lease is issued.
      • With 'DHCP Registration' disabled, machines that specify their hostname when requesting an IPv4 DHCP lease CANNOT be registered in the DNS Resolver so that their name can be resolved.
        • See mitigations below.
    • Unbound Python Mode - DHCP Registration Mitigations
      1. Use 'Static DHCP Leases'  (Static Leases)
        • Define the device's DHCP IP mapping: Services --> DHCP Server --> YOUR-INTERFACE (probably LAN) --> DHCP Static Mappings for this Interface
        • Register the devices name manually in the DNS: Services --> DNS Resolver --> General Settings --> Host Overrides
        • Services --> DNS Resolver --> General Settings --> (Static DHCP) Register DHCP static mappings: ticked
      2. Use Static IP addresses and then register the names manually in the DNS
        • Set the IP: manually enter the settings on the device
        • Register the names manually in the DNS: Services --> DNS Resolver --> General Settings --> Host Overrides
      3. devicename.local
        • This will not register the devices name, so it will not change anything in pfSense
        • Windows does not explicitly need a PC to register with DNS for it to contact it.
        • If you are in a Windows only environment (and some others), if windows cannot resolve via DNS then it will do a NETBIOS broadcast for the device's name in the .local domain.
  • Sync / Push Configs to remote pfBlockerNG instance
    • Firewall --> pfBlockerNG --> Sync
    • This feature allows you to push one pfBlockersNG configuration to a remote pfBlockerNG instance.
    • A push event will will occur on a Force Update or Cron event.
    • The documentation needs some improve ment which I have reported here: Todo #14221: Sync settings and inline documentation needs improving - pfSense Packages - pfSense bugtracker
    • XMLRPC Replication Targets
      • You can use the 'admin' account of the remote pfSense router but creating a second user in a restricted groug just for the pfBlockerNG syncs is preferable so you do not expose your primary admin account.
      • Use the same protocol/ip/hostname/port of the remote pfSense router.
      • HTTP is insecure because your password will be transmitted in plain text so always use HTTPS

IPv4 Whitelists

  • There are 3 ways to Whitelist IPs
    • Firewall --> pfBlockerNG --> IP --> IPv4 Suppression
    • Firewall --> pfBlockerNG --> IP --> IPv4 --> Add --> Permit
    • Firewall --> pfBlockerNG --> IP --> IPv4 --> Add --> (Alias | Alias Native)
      • Now make a Firewall rule manually and add use this alias
  • Warning: traffic that is allowed by these pfBlockerNG/Firewall rules will causes further processing of rules to be stopped and the traffic will pass to the appropriate interface

Sometimes when you import a feed it will include certain IPs or ranges that will prevent your local devices or external devices accessing assets on your network. This issue can easily be overcome with Whilisting without having to disable a whole feed which other than these exceptions is exactly what you want.

We will use the inbuilt pfBlockerNG mechanism so the rules are always placed where they should be. If you use the manual alias method you might have to keep moving your rule to the top of the firewall floating rules list everytime you refresh your rules.

When using Whitelisiting on 'outbound interfaces' you are not forced to use 'Advanced Firewall Rules' which means you can leave the protocol setting on 'Any' which allows such protocols as ICMP. It is only on 'inbound interfaces/rules' you are forced to use 'Advanced Inbound Firewall Rule Settings' which then forces you to select a protocol, and the protocol list is very limited.

Whitelist - IPv4 Suppression customlist (preferred method)
  • This removes the entries from pfBlockerNG's Deny aliastable, pfBlockerNG then acts as if the IPs were never added in the first place.
  • This method will still required your traffic to be allowed on your interfaces with firewall rules.
  • You can be more specific with the rules on your interfaces when using this method.
  • These entries might get lost in a large list if you use the IPv4 Suppression feature a lot, but they will still work.
  • Suppression will only handle IPs in a CIDR /32 or /24 format. For larger ranges you need to use another method.
  • Firewall --> pfBlockerNG --> IP --> IPv4 Suppression
    • Add the following lines to the top of the list: 
      191.101.64.81/32 # BBC.com
151.101.192.81/32 # BBC.co.uk
    • Firewall --> pfBlockerNG --> Update --> Force Reload (IP)
  • This is a simple way of removing IPs from the Deny aliastable with a few limitations.
  • This suppression list is for [ /32 or /24 ] IPv4 addresses only!
  • When manually adding an IPv4 address [ /32 or /24 only! ] to this Suppression List, you must run a "Force Reload - IP" for the changes to take effect.
  • Add an IP supression from the Alerts tab (this is for an IP record, the options change depending on the record type)
    • Firewall --> pfBlockerNG --> Alerts --> go to relevant record row --> click the + icon
    • Read the information
    • Do you want to Supress or Add to a Permit Whitelist Alias?: Suppress
    • Select Suppression Mask: Supress /32 (/32 = single ip, /24 = Class C network i.e. 10.0.0.x)
    • Do you want to add a description (optional)
      ss
      • Please enter a Supression description: Test suppression
    • The entry is now added
    • The list is here (Firewall --> pfBlockerNG --> IP --> IPv4 Suppression)
    • You do not need to do "Force Reload - IP" for the changes to take effect when using this method.
Whitelist - IPv4 Whitelist using Custom_List (from Alerts tab)

You can add IPv4 entries from the Alerts tab and here is how you do it.

  • Firewall --> pfBlockerNG --> Alerts --> go to relevant record row --> click the + icon
  • Read the information
  • Do you want to Supress or Add to a Permit Whitelist Alias? Whitelist
  • Option 1 - No whitelist
    • Select Whitelist: Create new pfB_Whitelist_v4

      • Because pfBlockerNG knows there is no whitelist it is asking us to create on.
    • Do you want to add a description?: No
      • If you select Yes, the IP address and description will not get transfered, this is a bug.
    • Confirm the whitelist by just click save, you do not need to do anything else.
    • The whitelist has been created and the IP entry added.
      • The whitelist is here: Firewall --> pfBlockerNG --> IP --> IPv4 --> IPv4 Summary
      • it is called Whitelist but its pfSense identification is pfB_Whitelist_v4
  • Option 2 - White list is already created
    • Select Whitelist: pfB_Whitelist_v4
      • I am not sure how to add addition whitelists here
    • Do you want to add a description?: (optional)
      • Enter the description
    • The entry is now added with an optional description
  • The IP  has been added to the Whitelist.
    • The whitelist is here: Firewall --> pfBlockerNG --> IP --> IPv4 --> IPv4 Summary
    • it is called Whitelist but its pfSense identification is pfB_Whitelist_v4
Whitelist - IPv4 Custom_List (Manual)
  • Your LAN and BRIDGEDVPN must assigned as Outbound Firewall Rules in pfblocker
  • This method works by adding a firewall allow rule before the pfBlockerNG Deny rule are processed.
  • With this method you do not need rules on the interfaces to handle the traffic as they have already been allowed with this rule.
  • This is for an outbound rule.
  • Firewall --> pfBlockerNG --> IP --> IPv4 Summary --> Add
    • Info
      • Name: Whitelist_BBC
      • Description: Allow BBC websites
    • IPv4 Source Definitions
      • none
    • Settings
      • Action: Permit Outbound
      • Update Frequency: Never
    • IPv4 Custom_List
      • Enable Domain/AS: unticked
      • 151.101.64.81
151.101.192.81
    • Save
    • Move the entry to the top of the 'IPv4 Summary' and Save. (This is not required but makes things easier to understand having your Whitelists at the top.)
    • Reload the lists (Firewall --> pfBlockerNG --> Reload)
  • In pfBlockerNG you will find it's advanced rules will only accept aliases of Network(s) type.
  • If you wanted to further restrict by source you can do, but this will not allow IGMP packets because of a limitation in 'FreeBSD packet fence'. This might get fixed/improved in later version of pfSense.
    • Create an alias with the IPs you whant to Whitelist
      • Firewall --> Aliases --> Add
    • Advanced Outbound Firewall Rule Settings
      • Custom Source
        • Enable: ticked
        • Invert: unticked
        • Source: Alias created above
Whitelist - IPv4 Feed

As above but instead of using a IPv4 Custom_List, but you use a feed (or multiple feeds). In fact you can probably do combination of both Feeds and IPv4 Custom_List.

This is possible, but there are restrictions. You can only define the protocol to be (UDP/TCP/TCP and UDP) and because of this I could only get http(s) monitors from uptimerobot.com working. Where I wanted to have Ping (ICMP) monitors running I could not because that protocol (ICMP) is not currently available. Feature request here: https://redmine.pfsense.org/issues/13202

This is a worked example to allow access to my Web Server from the UptimeRobot servers.

  • Create a firwall alias for your web server
    • Firewall --> Aliases --> IP --> Add
      • Name: Web_Server_Local_IP
      • Description: My Web Server's local IP address
      • Hosts:
        • IP of FQDN: 10.0.0.58
        • Description: Local IP
  • Create a Whitelist rule
    • Firewall --> pfBlockerNG --> IP --> IPv4 --> Add
      • Info
        • Name: Whitelist_Web_Server
        • Description: Allow access to Web Server from these IPs
      • IPv4 Source Definitions
      • Settings
        • Action: Permit Both
        • Update Frequency: Weekly
        • Weekly (Day of Week): Monday
      • Advanced Inbound Firewall Rule Settings
        • Custom Destination:
          • Enabled: ticked
          • Invert: unticked
          • Alias: Web_Server_Local_IP
          • Custom Protocol
      • Advanced Outbound Firewall Rule Settings
        • Ignore
      • Advanced Tuneables
        • Ignore
      • IPv4 Custom_List
        • Ignore
    • Click 'Save'
Whitelist - Alias (Deny|Permit|Match|Native)

Firewall --> pfBlockerNG --> IP --> IPv4 --> Add

  • Create an IP list but in select one of the following 4 Action types to generate a pfBlockerNG alias which all have their own feature set
    • Alias Deny:
    • Alias Permit:
    • Alias Match:
    • Alias Native: This create a native pfSense Alias that can be used as normal Alias through pfSense.
    • There are a couple of different flavours of the Alias method.
  • This type of list only generates a pfSense alias that you can use them in your own manual rules.
  • This method allows for more complex rules but the downside is you have to manually create them.
  • This will create the aliastable with the IPs, but will not create any rule You can create your rules as required and associate the aliastables in those rules.
  • Alias Prefix
    • When manually creating 'Alias' type firewall rules; Prefix the Firewall rule Description with pfb_ This will ensure that that Dashboard widget reports those statistics correctly. Do not prefix with (pfB_) as those Rules will be auto-removed by package when 'Auto' rules are defined. (This is from the Blue infoblock icon for the (Settings--> Action) in pfBlockerNG)
    • BBcan177
      • For Alias type rules, you need to prefix the Firewall Rule description with pfb_ for it to show in the Dashboard widget.
IPv4 Custom Blocklist
  • pfBlockerNG Customize Blocklist » Simplificando Redes
    • We present two ways to use pfBlockerNG to customize blocklist. Using customize blocklists allows to block specific domains.
      1. Using the Blacklist/Whitelist TLD​
      2. Second way: Creating your static blocklist​ - uses the 'IPv4 Custom_List'
Notes

IPv4 Custom Blocklist

  • If you set action to 'Permit Both' or 'Deny Both',  pfBlockerNG will create a rule for the 'Inbound/Outbound Firewall Rules' groups as defined in pfblockerng (Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration)
  • Enable Domain/AS
    • Firewall --> pfBlockerNG --> IP --> IPv4 --> Add/Edit --> IPv4 Custom_List --> Enable Domain/AS
    • This enabled Domain/AS resolution, if it is not ticked domains and ASN that are added will not work. If disabled, the list requires lesss processing.
    • Domain names or AS numbers, will be converted into their respective IPv4 addresses.
    • Autonomous system (Internet) - Wikipedia - this explains what the AS number (ASN) is.
  • Direction of IpV4 Custom_List
    • IN OUTBOUND interfaces/rules : IPv4 Custom_List = the IP destination list
    • IN INBOUND interfaces/rules rule : IPv4 Custom_List = the IP source list
  • How to bypass a client IP in IP4? | Reddit
    • BBcan177
      • Best to use the Alerts tab to review the blocked events. Also recommend to use pfBlockerNG-devel which has an improved Alerts tab to add IPs to a whitelist, or suppress the IP if required. The firewall rules (stateful firewall) are processed top to bottom so you would add the IP to and permit rule so that it is not blocked by a block rule. Just be careful about permitting IPs inbound as you typically just need to allow the IP outbound.
    • BBcan177
      • Create a new alias and add these IPs that you want whitelist to the custom list at the bottom of the page, then in the Adv. Outbound Firewall rule settings, add an alias for the Lan IPs that this applies to. Then set the Action setting to "Permit Outbound".
      • Check your firewall rule order setting to ensure permit rules are before the block rules (or opt for Alias type rules and manually create the firewall rules as required)
      • Then you can use the Alerts tab (pfBlockerNG-devel) to add any new IPs to this new permit Alias as required.
        • Firewall --> pfBlockerNG --> Reports --> Alerts
  • 'Advanced Inbound/Outbound Firewall Rule Settings' cannot use 'Any'
    • Advanced Inbound Firewall Rule Settings - confusing description on Custom Protoc | Netgate Forum
      • BBcan177
        • Note: In general, Auto-Rules are created as follows: Inbound 'any' port, 'any' protocol, 'any' destination and 'any' gateway
          • When you do not use Adv. In/Out settings, you need to leave the default as any…
          • Once you add any SRC/DST/Ports settings etc in Adv. In/Out settings, you can't use any, and must use one of the other Protocol settings…. This is a limitation of FreeBSD packet fence...
          • If, for example, you added Adv. In/Out settings, and left the Protocol as any, the additional SRC/DST/Ports settings etc are not utilized by packet fence and you will still see noise in the log for other blocked alerts which are already being blocked by the Default Block implicit rule.
          • If you only have one open WAN port, then utilizing the Adv. Inbound Settings will reduce the noise hitting the WAN and will only log blocked attempts to the open WAN port (and other settings configured in the rule)
        • There are settings for both Adv In/Out settings. So if you define the Adv Inbound but leave the Adv Outbound as is (default), then with Deny Both, the inbound rules are created with the Adv dst and port settings while the Outbound is set as default blocking anything outbound to those IPs.
  • Whitelist an IPv4
    • IP Whitelisting in pfBlockerNG | Netgate Forum
      • BBcan177
        • Create a new 'Permit Outbound' alias in pfBlockerNG. Then add any IPs that you want to allow outbound in the custom list at the bottom of the permit alias.
        • If none of the defined auto-rule options apply to your setup, then you will need to use 'alias type' settings and define the rules manually.
    • Custom IPv4 List
      • Firewall --> pfBlockerNG --> IP --> IPv4 --> Add
      • Name - Give this list a name such as 'Custom IP Whitelist'
      • IPv4 Source definitions should be empty
      • Action: (Permit Inbound|Permit Outbound|Permit Both)
        • Pick what sort of whitelisting you want
      • Update Frequency: Never
      • Adjust the Advanced Inbound/Outbound firewall rule settings as required.
        • If you are wanting to unblock outbound rules you could use the 'My_Local_Networks' alias.
      • Put the IP(s) in the 'IPv4 Custom_List' box at the bottom
      • Save
      • This rule should go at the top of the list (for ease)
        • Permit' rules create high priority 'pass' rules on the stated interfaces. They are the opposite of Deny rules, and don't create any 'blocking' effect anywhere. They have priority over all Deny rules.
  • Action Types explained

    • Packages — pfBlockerNG Package | pfSense Documentation
    • Disabled
      • Will just keep selection and do nothing to selected Lists.
    • Deny Inbound
      • Will deny access from selected lists to the local network.
    • Deny Outbound
      • Will deny access from local users to IP address lists selected to block.
    • Deny Both
      • Will deny access on Both directions.
    • Permit Inbound
      • Will allow access from selected lists to the local network.
      • This requires you to use the 'Advanced Inbound Firewall Rule Settings'
    • Permit Outbound
      • Will allow access from local users to IP address lists selected to block.
    • Permit Both
      • This requires you to use the 'Advanced Inbound Firewall Rule Settings'
    • Match (Inbound|Outbound|Both)
      • When you select "Match" it will only log the packets and nothing else.
    • Alias (Permit|Deny|Match|Native)
      • Alias Native would have the same net effect as the other options,but using it would involve less processing when updating the list but more processing while using the list because of how it makes the list.
      • These options create an alias that can be used throughout pfSense starting with the prefix pfB_ and then the rule name added, but they will not appear in (Firewall --> Alias)
      • When manually creating 'Alias' type firewall rules; Prefix the Firewall rule Description with pfb_ This will ensure that that Dashboard widget reports those statistics correctly. Do not prefix with (pfB_) as those Rules will be auto-removed by package when 'Auto' rules are defined.
      • All the alias options do the same and create a single alias. The action type is just used as a visual indicator as to what YOU are using it for. The only exception is 'Alias Native' which still creates a single alias like the other alias options but without any Suppression or Deduplication, the Feeds are downloaded and used in its native format.
      • PfBlockerNG Alias | Netgate Forum
        • BBcan177 Explantion of Alias Action Type
          • There are "Auto" generated rules from normal use of pfBlockerNG, and then there are "Alias" type rules.
          • With "Alias" type rules, the pfBlockerNB package makes an Alias table for the feed with the IPs, and then you have to manually create the Firewall rules according to your network needs using the newly created Alias.
          • When you select any of the Alias types [ Deny, Permit, Match or Native ], they do not create any Firewall rules, so in that sense there is no difference between any of those options. However, If you are going to use this Alias for a "Permit" rule, then select "Alias Permit".
          • Alias Match, would be used for a rule whereby you just want to log packets that match the IPs in the list, but do not block or permit them... But selecting "Alias Match" and configuring the rule to be a "Permit" action is in essence the same. I would recommend to use Alias Permit for permit rules, and Alias Match for Match type rules.
          • Alias Native is typically used instead of Alias Deny, where its used for a Block Type action, but the IPs do not go through the Suppression or Deduplication processes... IE: they remain native as per the source of the Feed.
        • PfBlockerNG Alias | Netgate Forum
          • BBcan177
            • When you select "Permit" it will create rules to allow traffic.
            • When you select "Match" it will only log the packets and nothing else.
            • When you select "Native" its the same as "Deny" except that there is no Suppression or Deduplication, the Feeds are downloaded and used in its native format.
            • There are "Auto" generated rules, and then there are "Alias" type rules. With "Alias" type rules, the pkg makes the Aliastable with the IPs, and then you have to manually create the Firewall rules according to your network needs.
          • BBcan177
            • When you select any of the Alias types [ Deny, Permit, Match or Native ], they do not create any Firewall rules… So in that sense there is no difference between any of those options... However, If you are going to use this Alias for a "Permit" rule, then select "Alias Permit"...
            • Alias Match, would be used for a rule whereby you just want to log packets that match the IPs in the list, but do not block or permit them... But selecting "Alias Match" and configuring the rule to be a "Permit" action is in essence the same.... I would recommend to use Alias Permit for permit rules, and Alias Match for Match type rules.
            • Alias Native is typically used instead of Alias Deny, where its used for a Block Type action, but the IPs do not go thru the Suppression or Deduplication processes... IE: they remain native as per the source of the Feed.
          • aborsic
            • To recap the discussion, would it be correct to state the following:
              • "Alias Permit", "Alias Deny", "Alias Match", and "Alias Native" do not create any rule, but they just create lists of IPs (aliases)
              • There is not difference in the IP lists created by "Alias Permit", "Alias Deny", "Alias Match", and "Alias Native"
              • The "Permit", "Deny", "Match", and "Native" indicates only the intended purpose of the created alias, but actually selecting one alias type versus an other would not make any difference.
            • This what I understood from the discussion, and would be very thankful if you kindly confirm whether this is correct.
          • SteveITS
            • So while using Alias Native would have the same net effect, using it would involve less processing when updating the list but more processing while using the list.
            • Alias Native does not look for duplicates.
            • However you should all probably read this thread (IP not covered in generated deny alias) which seems to have found that Alias Deny will remove IPs found in other lists which may not be the result you want, if rules for both lists are not denying the same port.
          • Translation
            • All alias action types create a pfSense Alias
            • Deny/Permit/Match are functionally all the same with the following exceptions
              • Deny has de-duplication performed on it (but has a wiered behaviour)
              • When you use the Alerts tab to add something to a Whitelist or Blocklist, then these action types might have a function by the Alerts Tab filtering available pfBlocker Aliases by action type depending on whether the user wants to Block or allow an IP..
            • Native
              • creates an alias from the raw sources without any CIDR processing or de-duplications.
            • There is a potential issue with the de-duplication process, read this article (IP not covered in generated deny alias) 
      • IP not covered in generated deny alias | Netgate Forum
        • Alias Deny's are subject to being processed, Alias Native's are not.
        • If I'm reading this right, it would appear that all Alias feeds (regardless of which Alias group they are in) except Native feeds, are evaluated and deduped as a whole. If this is correct it certainly isn't the way I thought it worked.
        • Thanks for this. That sheds some light for me too. So if I understand right, when you have both permit and deny alias groups and/or your rules target disparate port sets, it's probably safest to have pfblocker generate native aliases and define independent firewall rules to utilise them. That, or disable deduplication. Does a CIDR aggregation also operate across multiple alias groups?
        • So it seems to me that the deduping is only happening across lists that are in an Alias Deny group.
        • In the Deny Alias vs Native Alias above, CIDR aggregation has no effect (on the results of the regex grep at any rate). I checked with it on and off.
      • PfBlockerNG Alias types (Deny, Permit, Match, and Native) | Netgate Forum
        • BBcan177 introduces Advanced Inbound Firewall Rules (old post)
          • With v1.09 I have added "Adv. Inbound Firewall settings" where you can fine-tune the Inbound Port/Destination instead of needing "Alias" type rules… But for more complicated Rules, you can still use Alias types...
          • When using De-duplication, all of the Aliases/Lists are acting as a whole. So one list can have a blocked range instead of many lists having the reference to a blocked range. So if you create alias type rules, you need to add rules for all of the aliases to get full coverage...
          • With "Native", any lists that are used are not de-duplicated so that you can create a rule using that Alias for a certain configuration.... So its all about choice and what you are trying to achieve. Native is also good if you want to block say "Facebook" using the Hurricane Electric list. This way all of the IPs that the list has are used in the Alias without any chance of being affected by other Aliases.
          • Match and Permit also do not use any De-duplication.

DNSBL

  • Virtual IP Address
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Webserver Configuration --> Virtual IP Address
    • When you do a DNS lookup it's DNS IP, if it is blocked, the DNS reply comes back as 10.10.10.1
    • This is normal and means the domain is getting blocked and sent to the VIP.
    • 10.10.10.1 is the default Virtual IP Address (VIP) pfBlockerNG uses to send rejected DNS requests too.
  • Here is are screenshots of the DNSBL Blocked Pages.
    • This is only displayed when a full Domain is blocked and not for an ADvert on a page! (e.g. https://click.redditmail.com/)
    • This is only displayed when it is an ADvert that is being blocked, not a full domain. As you can see it is a single pixel so it can replace ADs without destroying the page. (e.g. https://click.redditmail.com/chicken/)
  • You can also create your own block page to display any customizations.
    • Custom block web pages can be added to: /usr/local/www/pfblockerng/www/ folder.
    • and then select the template here: Firewall --> pfBlockerNG --> DNSBL -->  DNSBL Configuration --> Blocked Webpage
    • This will only replace full domain blocks.
    • It might be backup up when you backup your routers settings but this is untested by me.
  • Permit Firewall Rules
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Configuration --> Permit Firewall Rules
    • LAN Segment - Network Encyclopedia - Lan Segment is a physical portion of a local area network (LAN) that is separated from other portions by bridges or routers.
    • Should "Permit Firewall Rules" be enabled? | Reddit
      • You only need this if you're using the dnsbl web server..ie the page you're shown when you visit a blocked site (http only no HTTPS) and you operate multiple LAN segments/vlans. 
      • The DNSBL webserver is used to show a block page on http sites that you have a blocked domain for. ie it will show you a webpage that says "domain X is blocked and found on dnsbl list Y". This DNSBL Webserver server normally is only accessible to the LAN interface and so only your LAN clients would be shown this block page. If you want this DNSBL webserver available to show the block page your IOT or OPT Vlans, you need to check this box and select the interfaces you want it on.
      • Do both conditions need to be true? yes, in order for the checkbox to have any practical use. If you are not using vlans that need to be shown the block page then this box will have no effect. Further, if your users aren't using HTTP to browse a website, the block page doesn't work at all. That said, there will not be any material harm to leaving it enabled.
    • Feature #14196: permitted firewall rules - additional text - pfSense Packages - pfSense bugtracker - Can you add some additional information here for the end user to explain lan segment and some possible scenarios when you would use this option.
DNSBL Null Block
  • DNSBL webserver https | Reddit
    • Is it possible to assign a cert to the DNSBL Webserver so you don't get a 'cert invalid' error or to rederict the https request to the http version (guess not)?
    • BBCan177
      • No, that will not fix that issue... The browser will see the the cert doesn't match the blocked domain. Otherwise you would have to MITM the connection which the pkg doesn't do.
      • You can create a new DNSBL Feed in DNSBL, and add those specific domains to the Custom list at the bottom of the page, and set the Logging to Disabled, and the Group Order to Primary. Follow that will a Force reload. This will null block (0.0.0.0) instead of using the DNSBL VIP address and avoid those cert errors.
      • Its not needed with the upcoming python integration. You will be able to null block (0.0.0.0) and still get all the other DNSBL features and yes, it will allow null blocking logged events. - This is an old post.
DNSBL Whitelist
  • Whitelisted Items in my feed update/reload
    • Firewall --> pfBlockerNG --> Update
    • When you do a Update/Reload you will see log messages showing that some of your DNSBL items are white listed (shown below) and this is normal but needs some explanation.
      [ EasyList ]			 Reload [ 05/4/22 19:11:57 ] . completed ..
  Whitelist: adsafeprotected.com|amazon-adsystem.com|
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  17237    17237      821        2          0          16414                
  ----------------------------------------------------------------------
  • I don't have a whitelist?
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL Whitelist (at bottom of page)
    • Yes you do, and it has some default entries:
      s3.amazonaws.com
s3-1.amazonaws.com # CNAME for (s3.amazonaws.com)
.github.com
.githubusercontent.com 
github.map.fastly.net # CNAME for (raw.githubusercontent.com)
.gitlab.com
.sourceforge.net
.fls-na.amazon.com # alexa
.control.kochava.com # alexa 2
.device-metrics-us-2.amazon.com # alexa 3
.amazon-adsystem.com # amazon app ads
.px.moatads.com # amazon app 2
.wildcard.moatads.com.edgekey.net # CNAME for (px.moatads.com)
.e13136.g.akamaiedge.net # CNAME for (px.moatads.com)
.secure-gl.imrworldwide.com # amazon app 3
.pixel.adsafeprotected.com # amazon app 4
.anycast.pixel.adsafeprotected.com # CNAME for (pixel.adsafeprotected.com)
.bs.serving-sys.com # amazon app 5
.bs.eyeblaster.akadns.net # CNAME for (bs.serving-sys.com)
.bsla.eyeblaster.akadns.net # CNAME for (bs.serving-sys.com)
.adsafeprotected.com # amazon app 6
.anycast.static.adsafeprotected.com # CNAME for (static.adsafeprotected.com)
google.com
www.google.com
youtube.com
www.youtube.com
youtube-ui.l.google.com # CNAME for (youtube.com)
stackoverflow.com
www.stackoverflow.com
dropbox.com
www.dropbox.com
www.dropbox-dns.com # CNAME for (dropbox.com)
.adsafeprotected.com
control.kochava.com
secure-gl.imrworldwide.com
pbs.twimg.com # twitter images
www.pbs.twimg.com # twitter images
cs196.wac.edgecastcdn.net # CNAME for (pbs.twimg.com)
cs2-wac.apr-8315.edgecastdns.net # CNAME for (pbs.twimg.com)
cs2-wac-us.8315.ecdns.net # CNAME for (pbs.twimg.com)
cs45.wac.edgecastcdn.net # CNAME for (pbs.twimg.com)
cs2-wac.apr-8315.edgecastdns.net # CNAME for (pbs.twimg.com)
cs2-wac-us.8315.ecdns.net # CNAME for (pbs.twimg.com)
cs45.wac.edgecastcdn.net # CNAME for (pbs.twimg.com)
.pfsense.org
.netgate.com
  • If you enable DNSBL SafeSearch or DNS over HTTPS/TLS Blocking, these will be added to the Whitelist to prevent them being re-added from the standard DNSBL feeds but will still be blocked as normal.
    • Firewall --> pfBlockerNG --> DNSBL --> DNSBL SafeSearch
    • This explains why you have Whitelist entries that appear to be domains you want blocked or think are
  • Should I remove the entries in the whitelist because I never added them?
    • Looking at the entries I cannot see any harm in removing them. I will ask on the forum.
    • I would definitly leave the following domains in for obvious reasons so no-one can pollute a feed and prevent my firewall updating. 
      .pfsense.org
.netgate.com
  • DNSBL deny all except whitelisted | Netgate Forum
  • How to Enforce pfBlockerNG DNSBL filtering for Specific Network Clients - How to use pfBlockerNG-devel to filter content for clients while allowing specific IPs or networks to bypass DNSBL and visit the web normally.
  • Whitelisting : pfBlockerNG
    • I loaded in a bunch of feeds, and all is working well. However, there are 3 domains that I need to whitelist for some users on our network.
    • With Unbound mode, when a domain is block by TLD, you need to whitelist the root domain which will also whitelist any subdomains that are blocked. Other options include using the TLD Exclusion list to stop that Domain from being wildcard blocked and then whitelist the subdomain as required.
    • With the new Unbound Python mode, you can now whitelist a sub domain even when it's wildcard blocked via TLD.
    • BBCan177
      • You don't need to add to the TLD Whitelist. Best to remove those entries and use only the DNSBL Whitelist (and best to whitelist from the Reports tab).
      • That whitelist is only used when you are blocking whole TLD like "cn". So the TLD Whitelist, would allow you to add a "cn" domain to bypass the TLD cn blocking.
Exclude a local IP from DNSBL protection / Bypassing DNSBL for specific local IPs

There are a couple of different ways of bypassing the pfBlockerNG DNSBL.

Python Group Policy

This is a native pfBlockerNG feature where Python will make sure that DNS requests from the specified IPs will not be subject to it's DNSBL.

  • Python Mode
    • This requires pfBlocker to be in 'Unbound python mode'
      • Firewall --> pfBlocker --> DNSBL --> DNSBL Mode = Unbound python mode
    • Enable 'Python Group Policy'
      • Firewall --> pfBlocker --> DNSBL --> Python Group Policy = ticked
    • A new expandable section called 'Python Group Policy' will appear below. Expand this section.
    • Enter the required IPs in the 'Bypass IP section'
    • Click Save
    • Do a Forced Reload
      • Firewall --> pfBlocker -->Update
  • Unbound Custom DNS options
    • Bypassing DNSBL for specific IPs | Netgate Forum
      • Allow local LAN clients to bypass pfBlockerNG
      • Python mode is very iffy. I have found that using unbound custom dns options works way better to exclude hosts or networks from DNSBL where as python mode only allows /32 (Single IP) exclusions which isn’t very scalable at all. I would highly recommend that over python mode.
      • The latest pfSense, and the latest pfBlockerNG makes it possible to exclude IP (or even networks ?) using the pfBlockerNG GUI.
        • Select the (DNSBL) 'Python Group Policy' option and fill in the IP's.
        • pfBlockerNG --> DNSBL --> Python Group Policy

Unbound Views

Views are possible in Unbound 1.6+ and can be used for bypassing DNSBL zones for specific IPs/ranges. I think the unbound process happens after the Python code has been run (if enabled) and is the base package for DNS on pfSense (DNS Resolver).

The solution is from Ns8h posted here Bypassing DNSBL for specific IPs | Netgate Forum along with some troubleshooting and more advanced examples.

  • Add some stuff to the custom unbound options (tailored for needs)
    • Services --> DNS Resolver --> General Settings --> Display Custom Options --> Custom Options
    • server:
    access-control-view: 192.168.0.2/32 bypass
    access-control-view: 192.168.0.0/24 dnsbl
view:
    name: "bypass"
    view-first: yes
view:
    name: "dnsbl"
    view-first: yes
include: /var/unbound/pfb_dnsbl.*conf
    • Host 192.168.0.2 is able to bypass all pfBlockerNG inserted DNSBL zones but is able to resolve other local zones e.g. DHCP added zones. Everything else on the 192.168.0.0/24 subnet gets blocked as normal through DNSBL.
  • Forec Reload pfBlocker
    • Firewall --> pfBlockerNG --> Update --> Force Update

What pfBlockerNG rule was triggered?

  • Quick easy way to determine if an IP is on a pfBlockerNG list? | Netgate Forum
    • Every once in a while I'll see an IP has been blocked and want to determine if it is on a blocklist from pfBlockerNG and if so which one, is there an easy/quick way to do so? Right now I am just hovering over the rule and manually looking at the IPs, but this is tedious.
    • You should be able to see this in the pfBlockerNG Alerts tab
    • You can also grep from the shell: 
      cd /var/db/pfblockerng/deny/
grep "^1.2.3.4" *

Other examples:

grep "^1.2.3" *
grep "^1.2." *
grep "^1." *

add    | grep '/'  to only report CIDRs.
  • How to find which pfBlockerNG rule is triggered by an IP/Domain
    • This is also the same for normal firewall rules
    • Status --> System Logs --> Firewall
      • Do a DNS/IP request and then look at the logs
      • Filter by local IP of your computer or of the remote domain/computer
      • The rule name will be against the lookup
    • NB: Rules need to have logging enabled on them. All of the pfBlocker rules have logging enabled by default

Alerts

  • Alert Lists - Default Sort order
    • Firewall --> pfBlockerNG --> Alerts
    • When you click on column header to re-sort the records as you want, pfSense/pfBlockerNG remember this choice. This is also why there is no default sort order option.
    • This setting must be saved locally on your pfSense box under your user preferences.
    • This solution might be valid for other lists in pfSense
  • Alerts - Unknown or Not listed error
    • Firewall --> pfBlockerNG --> Alerts
    • I get this quite often but it is normal.
    • pfblockerng-devel error: Unknown Not listed! | Netgate Forum - This explains what is going on here.
    • pfblockerng error: Unknown Not listed! | Netgate Forum
      • Alerts will display Unknown Not listed when the IP is no longer in any IPV4 lists or during a Update as the database is being rebuilt.
    • 'Unknown'
      • This could be related to this IP being in a CIDR or an aggregated CIDR.
      • What is "Unknown" feed? : pfBlockerNG
        • BBCan117 - It's not listed anymore in DNSBL, so that it why it shows as "unknown". Maybe your OS/browser is caching the blocked domain. Are you getting any new events for that domain?
        • This post is 4 years old and might be out of date.
    • 'Not Listed
      • I think this means the IP has been found in a CIDR or an aggregated CIDR. This should be changed to something like 'In CIDR'
      • The IP has been flagged because it is in the list.
    • There are 2 record shown on each row.
      • Current rule that flagged this IP
      • Previous rule that flagged this IP

pfBlockerNG DNSBL and IP Parser

This section is to educate people on what sort of formats pfBlockerNG can handle for it's DNSBL and IP feeds.

The file responsible for this code is pfblockerng.inc and is reference by line numbers as you go through the steps below.

Look at the in-built feeds in pfBlockerNG as they have quite a range but at least you know all of them work and are in an acceptable format.

Overview
  • IP Block Lists
    • There are no defined supported formats. All files are treated as text files and then an IPv4 and IPv6 regex is applied and all of the IPs are extracted allowing all formats and more (json, csv, xml, text).
  • DNSBL
    • CSV can only be used if they match one of the internally coded formats which are feed specific.
    • Each line is processed as a single record
    • If a valid (non 0.0.0.0 / 127.0.0.1) address is found on the same as a domain, then the domain is ignored.
    • Domains are extracted (via various clean ups) from the record and then processed as you expect.
DNSBL Feeds

There is no Regex for grabbing domains from their feeds but there is a defined process that uses standard PHP functions and pfSense custom functions. This code also grabs any IP addresses it finds in the feeds.

Relevant Variables in the script

The variables below are useful in seeing what is going on in the parser:

  • GitHub Lines 8342 - 8703 
    $header		= "{$row['header']}";
$liteparser	= FALSE;	// Minimal DNSBL Parser
$rev_format	= FALSE;	// Host style format is reversed
$domain_data_ip	= array();	// Array of IPs found in feed
$domain_data	= '';		// List of Domains found in feed
  • The 'CSV Parser' processes one line of the feed per loop, and adds the domain to $domain_data_ip
  • The parser is one giant loop that relies on a continue statement when matches on lines are made.
  • Does the parser accept zips? The parser does not directly handle zips so if you can add a ZIP feed then this file type is handled upstream and the parser only ever sees the uncompressed file.

Parse downloaded file for Domain names - The Loop

Below is a quick overview of the most relevant parts of the DNSBL feeds parser. Notes are at the bottom

GitHub Lines 8342 - 8703

Start of Loop

  • Initial Processing
    • GitHub Lines 8349 - 8443
    • // Validate EasyList/AdBlock/uBlock/ADGuard Feeds
      • Checks for special sources and flags as needed
    • // Remove any '^M' characters
    • // Remove invalid characters
      • Removes all unwanted characters from the beginning and end of the line
      • If easylist skip the first line
    • // If 'tab' character found, replace with whitespace
    • // If '%20' found, remove.
      • Remove whitespace
    • // Remove comment lines and special format considerations
      • Remove lines that start with #
      • Remove special case lines that match these rules
    • // Remove slash comment lines
      • Remove lines that start with // to make comments
    • // Remove any 'End of line' comments (Some contains commas)
      • Comments that are allowed at the end of a line using #
    • // Convert CSV line into array
      • If source is a csv, convert line into an array and add to $csvline
    • // Remove blank lines
  • // CSV parser
    • GitHub Lines 8446 - 8562
    • Only run this section if a CSV is the source
    • All CSV files are special case, there are no rules to handle unknown csv feed types. This probably because a CSV by definition is customized.
    • If you want to use a CSV file, select a format of one of these special cases and the parser will not know any difference.
    • This will exract one line from the csv and return it as a single line for the script as $line
  • // EasyList Parser
    • GitHub Lines 8565 - 8589
    • Specific parser for easylost
    • // Typical Host Feed format - Remove characters before space
    • // Remove characters after space
    • // Determine if line contains only an alpha-numeric Domain name
  • // Lite Parser
    • GitHub Lines 8591 - 8627
    • This cleans up the line and extract the domain name (host name)
    • This is the main code that extracts the domain and uses $host = parse_url($line)
    • // If 'http|https|telnet|ftp://' found, remove
    • // If '/' character found, remove characters after '/'
    • // If '#' character found, remove characters after '#'
    • // If '?' character found, remove characters after '?'
    • // If special characters found, parse line for host
    • // Remove any Port numbers at end of line
  • // Collect any IPs found in domain feed
    • GitHub Lines 8631 - 8641
    • This will check the record for an IP address and store them in $domain_data_ip
    • If an IP address is found, it is stored in $domain_data_ip and the script continues to the next record. Domain processing will be skipped for this line.
    • This routine will extract and validate a single IP address. It expects only a single IP address.
    • The function is_ipaddrv4() is located in the file /etc/inc/util.inc
  • // Convert IDN (Unicode domains) to ASCII (punycode)
  • // Domain Validation
  • Write the domain to the relevant file

End Of Loop

  • // Remove duplicates and save any IPs found in domain feed

The rest of the does what it does, DNS lookup for all of the Domain names and then combines the 2 lists that have just been generated perhaps.

Notes

  • A single line is processed as a single record
  • All whitespace and tabs are removed
  • All comment lines are removed (// and #) 
  • All 'End of line' comments are removed (#)
  • All protocols are removed (http|https|telnet|ftp://)
  • All port numbers are removed (:8080)
  • All parts of the slug are removed (/snailsarebetter/thanslugs/)
  • All queries are removed (?somevariable=here)
  • Unwanted special characters are removed.
  • IDN (Unicode domains) are converted to ASCII (punycode)
  • CSV are handled as specific cases and must match to one of the specific rules otherwise they are failed.
  • If an IP and a domain are on the same line in a feed, then domain is ignored unless (I guess) the IP address is an invalid one such as 0.0.0.0 or 127.0.0.1
  • From BBCan117 - Block lists - Supported formats : pfBlockerNG
    • One domain per line is the simplest format to maintain. It has parsers for many different feeds but those are all specific use cases.
    • You can add ”#" comments which will be ignored. So have a # at the start of the line will ignore the whole line, have a # after the domain will collect the domain and skip the remaining comment.
    • Do not use * (asterisks) or any other special characters unless it's in Puny code or ISDN (Unicode) format which is also acceptable
    • For IP, one entry per line. Best as a single IP with or without CIDR. Using a range format is also acceptable but just takes more effort to convert. No special characters allowed., and the same syntax for comments as above.
IP Feeds REGEX (IPv4 / IPv6)

This is straight forward, the IP parser handles all sources as a text file and performs a regex on the file to extract all of the IPs. This means that you can supply almost any sort of feel such as CSV, XML, JSON, HTML or even PHP.

The relevant parser section is shown below with the regex that is used

Download and Collect IPv4/IPv6 lists

GitHub Lines 9233 - 9790

// IPv4 REGEX Definitions
$pfb['range']	= '/((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))-((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))/';
$pfb['ipv4']	= '/(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)((\/(3[012]|[12]?[0-9]))?(?![-0-9a-zA-Z]))/';

// IPv6 REGEX Definitions - Reference: http://labs.spritelink.net/regex
$pfb['ipv6'] = '/((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?(\/[0-9][0-9]?|1([01][0-9]|2[0-8]))?/';
Asset Locations

OpenVPN

General

Security

  • Data Encryption Algorithms
    • The default list of encryption algorithms and their order is the best so no need to change.
      • AES-256-GCM
      • AES-128-GCM
      • CHACHA20-POLY1305
    • Virtual Private Networks — OpenVPN — OpenVPN Configuration Options — Server Configuration Options — Cryptographic Settings | pfSense Documentation
    • 128 or 256 bit Encryption: Which Should I Use? - Ubiq
      • Picking Between AES-128 and AES-256
        • 128-bit and 256-bit AES both have their pros and cons. AES-128 is faster and more efficient and less likely to have a full attack developed against it (due to a stronger key schedule). AES-256 is more resistant to brute force attacks and is only weak against related key attacks (which should never happen anyway).
        • Since both algorithms are secure against modern and anticipated future threats, the choice between them doesn’t really matter from a security perspective. Our best guidance is that AES-128 provides more than adequate security while being faster and more resource-efficient but readers who want that extra security provided by greater key sizes and more rounds in the algorithm should choose AES-256.
    • What's the difference between AES-CBC and AES-GCM? - Knowledgebase / Technical / Application Settings and Features / Encryption - PIA Support Portal - A good description of the difference and why AES-128-GCM is good enough in modern day.
    • Change Encryption Cipher In Access Server | OpenVPN - Before you begin changing the encryption cipher, it’s important to note that all OpenVPN Access Servers work with a single encryption scheme.
    • Pfsense / OpenVPN advise - Traffic through tunnel seems very slow
      • Q:
        • Looking for input on tuning OpenVPN on pfSense. Traffic through tunnel seems very slow. Currently downloading less than 1MB/s via SMB through OpenVPN. RTSP streams over 800kbps don't seem to make it through the tunnel.
      • A:
        • Use AES-GCM. The AEAD packet format has a smaller crypto overhead than the CBC packet format, (e.g. 20 bytes per packet for AES-128-GCM instead of 36 bytes per packet for AES-128-CBC + HMAC-SHA1), and SHA1 isn't accelerated on most systems (it is on C3000).
        • Since SHA isn't accelerated, it becomes the dominant part of the operation. AES-GCM doesn't need the authentication algorithm second pass (because it is an AEAD), and therefore, you end up with all AES-NI accelerated crypto.
        • AES-GCM is also faster than AES-CBC (even without an HMAC) when AES-NI is used.
        • Also: OpenVPN will prefer AES-GCM-256 unless you set AES-GCM-128 and disable 'ncp-enable'.
        • Use AES-GCM instead of AES-CBC. Chain Block Cipher is single threaded because it needs to wait on the result of one block before it can be used to encrypt the next block. Galois Counter Method is able to encrypt each block independent of each other, making it parellalizable. Although it does provide slightly better security, it's main benefit is that it provides vastly better performance, which is perfect for older hardware.
        • Other detailed information on some types of encryption.
  • Don't Use Compression
    • OpenVPN compression | Netgate Forum
      • I want to setup OpenVPN, Compression has multiple options. In the past I have choosen Adaptive LZO Compression, recently I have read that Adaptive LZO Compression is depreciated in 2.4 version and has been removed in 2.5. Which option is recommended to choose? "Omit Preference (Use OpenVPN Default) has been choosen by default.
      • Jim Pingle: The current best practice is to disable compression for OpenVPN, due to attacks such as VORACLE which are possible when VPN traffic is compressed.
      • "Disable Compression, retain compression packet framing (compress)" - This page also discussed how to use this method (similiar to PIA pfSense 2.6.0)
    • HEADS UP: OpenVPN deprecating shared key mode, requires TLS, deprecating cipher selection | Netgate Forum
      • Disable compression on all OpenVPN tunnels, which you should do anyhow because compression is insecure and doesn't help much these days.
    • Deprecated Options in OpenVPN - Option: --comp-lzo
      • OpenVPN is a software VPN product which has been around since ​May 2001. And it has mostly been backwards compatible on the most important features through all these years. But the world moves forward, security issues are discovered, and expectations of how a secure VPN should be configured have changed over the years.
      • Compression is not recommended and is a feature users should avoid using. See --compress for more details.
    • VORACLE attack and OpenVPN - Security researcher Ahamed Nafeez has ​presented a new attack vector which targets VPN tunnels which utilizes compression, named VORACLE. The attack vector bears similarities to the CRIME and BREACH attacks, which hit especially HTTPS based connections.
  • Revoke Certificate
    • How to revocate user certificate on pFSense (OpenVPN) | IT Blog - You have pFSense OpenVPN configured with local CA and user certificates, and now – somebody is leaving the company, or certificate is compromised, what should you do? Simply deleting user account or certificate is not a good practice, and it probably won`t work.

OpenVPN Settings

  • route-nopull
    • Just add "route-nopull" to the client openvpn config, then all pushed commands from the server are ignored.
    • To get access to the local net, you must now add e.g. "route 192.168.5.0 255.255.255.0" to the client openvpn config, if the local net you want to connect to is 192.168.5.0/24
  • redirect-gateway / redirect-gateway def1 (Redirect IPv4 Gateway)
    • Force all client-generated IPv4 traffic through the tunnel.
    • This will add a static route to the VPN service you use, remove your current default route and add a default route towards the VPN tunnel. Keep in mind, however, that this could lead to undesirable consequences, such as inability to access resources of your corporate network or your ISP DNS servers when the VPN connection is active.
    • routing - How does openvpn's redirect-gateway option work? - Server Fault
    • OpenVPN equivalent: --redirect-gateway flags
    • Redirect IPv4 Gateway and Redirect IPv6 Gateway do NOT prohibit communication with other devices on the same subnet, those options ONLY effect routing of traffic outside of the local subnet.
    • This does not block local traffic, for that you need --block-local
    • Flags
      • local -- Add the local flag if both OpenVPN servers are directly connected via a common subnet, such as with wireless. The local flag will cause step 1 above to be omitted.
      • autolocal -- Try to automatically determine whether to enable local flag above.
      • def1 -- Use this flag to override the default gateway by using 0.0.0.0/1 and 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of overriding but not wiping out the original default gateway.
      • bypass-dhcp -- Add a direct route to the DHCP server (if it is non-local) which bypasses the tunnel (Available on Windows clients, may not be available on non-Windows clients).
      • bypass-dns -- Add a direct route to the DNS server(s) (if they are non-local) which bypasses the tunnel (Available on Windows clients, may not be available on non-Windows clients).
      • block-local
        • Block access to local LAN when the tunnel is active, except for the LAN gateway itself. This is accomplished by routing the local LAN (except for the LAN gateway address) into the tunnel.
        • redirect-gateway's block-local parameter isn't working - OpenVPN Support Forum
          • block-local is a redirect-gateway flag and not a command
          • Example commands 
            server-side:   push "redirect-gateway def1 block-local"
client-side:   redirect-gateway def1 block-local
      • ipv6 -- Redirect IPv6 routing into the tunnel. This works similar to the def1 flag, that is, more specific IPv6 routes are added (2000::/4, 3000::/4), covering the whole IPv6 unicast space.
      • !ipv4 -- Do not redirect IPv4 traffic - typically used in the flag pair ipv6 !ipv4 to redirect IPv6-only.
      • IgnoreRedirectGateway – OpenVPN Community
        • If you are running OpenVPN as a client, and the server you use is using push "redirect-gateway" then your client redirects all internet traffic over the VPN. Sometimes clients do not want this, but they can not change the server's configuration. This page explains how to override redirect-gateway so the client does not need to redirect internet even though the server says to.
        • def1 -- Use this flag to override the default gateway by using 0.0.0.0/1 and 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of overriding but not wiping out the original default gateway.
      • Firewall Traffic Needs "redirect-gateway def1" to Route Thru VPN? | Netgate Forum
        • To manually use VPN as default gateway, use "redirect-gateway def1;" in advanced options, don't set under System > Routing
        • OpenVPN will put 0.0.0.0/1 and 128.0.0.0/1 in routing table so it does not stomp system default gateway, otherwise VPN traffic couldn't exit.
  • OpenVPN Configuration Options — Server Configuration Options — Tunnel Settings | pfSense Documentation
  • OpenVPN Configuration Options — Custom Configuration Options | pfSense Documentation
    • Redirect Gateway(s)
      • 'Redirect IPv4 Gateway' option adds to the server config: push "redirect-gateway def1";
      • 'Redirect IPv6 Gateway' option adds to the server config: push "redirect-gateway ipv6";
      • Both of these options will force all traffic down the VPN (opposite of a Split Tunnel)
      • Forces all traffic through the VPN by making it the default gateway. This is what most people want 99.999% of the time. You would not want this if for whatever reason, you only wanted some traffic going over the VPN, and other traffic going out your real gateway (Split Tunnel).
  • 'Bridge Route Gateway' (pfSense)
    • Push the Bridge Interface IPv4 address to connecting clients as a route gateway.
    • 'Bridge Route Gateway' when enabled removes a lot of explicit route that are pushed to the client. I think if this is on 'Redirect IPv4 traffic' might not be fully respected
    • I have recently found the .OVPN config files in /var/etc/openvpn
    • The equivalent OpenVPN command is --route-gateway
      • The way I found this setting is
        • Create an pfSense XML Backup
        • Search the XML for the relvant key (you might need to change some wording to find your choosen key as they will not be named exactly the same)
        • What I found 
          <openvpn-server> 
    ......
    <serverbridge_routegateway>yes</serverbridge_routegateway>
        • I now made an educated guess that is was --route-gateway
      • This setting is present on all OpenVPNs but can only be active on TAP servers.
  • client-side Scripting
    • Explanation Of Client-side Scripting With Simple Examples | OpenVPN
      • What is client-side scripting and what can it do? Learn about this scripting process here.
      • OpenVPN Access Server supports pushing scripts to VPN clients. By default OpenVPN Connect for Windows and Mac prompts once for the user to confirm if scripts should be executed or not. If the user approves then the script runs from that point on whenever the user connects or disconnects.
      • For security reasons, client-side script commands are limited to the user context. For system-wide configuration, you must call a program that asks the user for privilege escalation.
  • Force a host resolution to a given IP on Windows.
  • Disable IPv6
    • Only enable IPv4 (UDP|TCP) on the OpenVPN server in pfSense
      • VPN --> OpenVPN --> [Server] --> edit --> Endpoint Configuration --> Protocol
    • disable IPv6 for a client - OpenVPN Support Forum
      • In client specific overrides, add these lines to disable IPv6:
        push-remove ifconfig-ipv6;push-remove route-ipv6
    • --block-ipv6
      • From Reference Manual For OpenVPN 2.5 | OpenVPN
      • On the client, instead of sending IPv6 packets over the VPN tunnel, all IPv6 packets are answered with an ICMPv6 no route host message.
      • On the server, all IPv6 packets from clients are answered with an ICMPv6 no route to host message.
      • This options is intended for cases when IPv6 should be blocked and other options are not available. --block-ipv6 will use the remote IPv6 as source address of the ICMPv6 packets if set, otherwise will use fe80::7 as source address.
      • For this option to make sense you actually have to route traffic to the tun interface. The following example config block would send all IPv6 traffic to OpenVPN and answer all requests with no route to host, effectively blocking IPv6 (to avoid IPv6 connections from dual-stacked clients leaking around IPv4-only VPN services). 
        --ifconfig-ipv6 fd15:53b6:dead::2/64 fd15:53b6:dead::1
    • Blocking ipv6 traffic through tunnel, LAN still accessible - OpenVPN Support Forum
      • So if I use block-ipv6 then there is no need for routing a specific ipv6 subnet like 2000::/3?
      • In that case would it be best recommended to use block-ipv6 server-side, and then push block-ipv6 to the client, just so I've covered both sides?
      • Like explained in the manual I've currently put this in my server config, along with a push for block-ipv6 incase it isn't in any of my client configs:
        push "ifconfig-ipv6 fd15:53b6:dead::2/64 fd15:53b6:dead::1"
push "redirect-gateway def1 ipv6 bypass-dhcp"
push "block-ipv6"
block-ipv6
  • OpenVPN Configuration files on pfSense
    • The .OVPN config files for the OpenVPN servers and clients are saved in the path 
      /var/etc/openvpn/
    • These files are useful to see what OpenVPN settings have actually been configured by pfSense server side.
    • You can see what settings are pushed to the clients.
    • Do not edit these files directly.
  • Verobiosty Levels
  • remote-cert-tls server

Custom Options

Reference Manual For OpenVPN 2.5 (Options) | OpenVPN

  • Examine your .ovpn file to get any additional settings you need for custom options. Here is a worked example
    client                         - Is this a client or server (pf)
dev tun                        - type of adapter eg tun/tap (pf)
proto udp                      - protocol to use (pf)
remote 103.103.103.103 1194    - the remote server and port (pf)
resolv-retry infinite          - Resolve server hostname retries, infinite is the default setting.
nobind                         - Do not bind to local address and port (pf: Interface and Local Port)
tun-mtu 1500                   - Maximum Transmission unit (OpenVPN default is 1500)
tun-mtu-extra 32               - This parameter only controls internal OpenVPN buffer sizing, so there is no transmission overhead associated with using a larger value.
mssfix 1450                    - Maximum MTU before encapsulation?
persist-key                    - When restarting with non-root, keep Keys
persist-tun                    - Don't close and reopen TUN/TAP device
auth-user-pass                 - keep the username and password in memory (pf: probably)
comp-lzo                       - Compression enabled and type (pf)
reneg-sec 0                    - Renegotiate data channel key after n seconds. This controls how often OpenVPN renegotiates authentication with clients. (default=3600)
verb 3                         - Logging Level (pf)
    • Add additional settings 
      auth-retry interact       - If authentication fails OpenVPN will automatically try to re-authenticate (no 100% about this setting)
remote-random-hostname    - Prepend a random string (6 bytes, 12 hex characters) to hostname to prevent DNS caching.
                            For example, "foo.bar.gov" would be modified to "<random-chars>.foo.bar.gov". 
                            (optional) No good if you want to control clients by their hostname
block-local               - Block access to local LAN when the tunnel is active, except for the LAN gateway itself.
                            This is accomplished by routing the local LAN (except for the LAN gateway address) into the tunnel.
    • Add Semi-Colons on the end of each line. However I do think that as long as each command is on a new line the semi-colon is optional.
    • The new Custom Options should look like
      tun-mtu 1500;
tun-mtu-extra 32;
mssfix 1450;
persist-key;
persist-tun;
reneg-sec 0;
auth-retry interact;
      • I am not sure if the packet size specifications (tun-mtu, tun-mtu-extra) are needed in pfsense but there is no options for them and the options above look like default values anyway.
    • These PIA settings will assist your VPN reconnect quicker if the connection goes down and are recommended
      persist-key;
persist-tun;
remote-cert-tls server;
reneg-sec 0;
auth-retry interact;

Pushing Options

  • You cannot push all options. If you try and push an invalid option you will get a context error.
  • Pushing DHCP Options To Clients | OpenVPN - The OpenVPN server can push DHCP options such as DNS and WINS server addresses to clients. However there are some caveats to be aware of.
  • Win 10 connection made but no access to LAN; Android works fine - OpenVPN Support Forum - no description
  • How to push DNS servers to the Windows client with net30 on the server? - OpenVPN Support Forum - no description
  • DNS and Domain Suffix not pushing - OpenVPN Support Forum - no description
  • Push Examples
    • VPN --> OpenVPN --> Client Export Utility --> [Your OpenVPN Client] --> Advanced --> Additional configuration options
      dhcp-option DNS 10.0.0.1
dhcp-option DOMAIN mydomain.com
    • VPN --> OpenVPN --> Servers --> [Your OpenVPN Server] --> Edit --> Advanced --> Additional configuration options
      push "route 192.168.100.0 255.255.255.0"
push "dhcp-option WINS 10.0.0.1"
push "dhcp-option DNS 10.0.0.1 "
push "dhcp-option DOMAIN mydomain.com"
  • What options can be pushed
    • Reference Manual For OpenVPN 2.5 | OpenVPN
      • search for "--push option"
      • Push a config file option back to the client for remote execution. Note that option must be enclosed in double quotes (""). The client must specify --pull in its config file. The set of options which can be pushed is limited by both feasibility and security. Some options such as those which would execute scripts are banned, since they would effectively allow a compromised server to execute arbitrary code on the client. Other options such as TLS or MTU parameters cannot be pushed because the client needs to know them before the connection to the server can be initiated.
      • This is a partial list of options which can currently be pushed: --route, --route-gateway, --route-delay, --redirect-gateway, --ip-win32, --dhcp-option, --inactive, --ping, --ping-exit, --ping-restart, --setenv, --auth-token, --persist-key, --persist-tun, --echo, --comp-lzo, --socket-flags, --sndbuf, --rcvbuf
    • openvpn - How to get a list of options, that can pushed to the clients? - Server Fault
      • This give a technical method of going through the source code to find what options can be pushed.

Routing

  • Expanding The VPN To Include Additional Machines | OpenVPN
    • Once the VPN is operational, it may be desirable to expand the scope of the VPN so that clients can reach multiple machines on the server network.
    • Cpvers both tun and tap connections.
  • windows - OpenVPN: Only route a specific IP addresses through VPN? - Super User
    • By default, OpenVPN routes all network packets destined for the remote network on which the VPN server resides, through the VPN. Unfortunately, accessing the file server through the VPN is extremely slow!
      • How can I configure the OpenVPN client to ONLY route traffic through the VPN that is destined for a single, specific IP address -- namely the database server?
  • Solved: LIVEcommunity - Access to Internal Web Site Through pfSense VPN - LIVEcommunity - 414918
    • Q: The problem I am running into when i connect to the pfSense VPN i cannot browse to a web server that sits on server 192.168.130.221. I can ping the host just appears that no TCP communications is allowed. I have also checked my policies and nothing in my findings is blocking it.
    • A:
      • Looks like asymmetric routing issue.
      • Pfsense will see server local and go direct
      • Add a static route on the server to 10.31.253.0/24 via Gatway 192.168.130.249
  • Routing specific websites through your VPN gateway using pfSense - Geek is the Way!
    • The idea behind this post is creating a Firewall alias on your pfSense and then modify your LAN firewall rules to switch to a different gateway (aka the VPN gateway) when the selected websites are detected.
    • Has a kill switch option.
  • vpn - pfSense OpenVPN clients routing - Server Fault
    • Q:
      • Local subnet is 192.168.1.0/24, VPN clients are 10.0.1.0/24
      • We also have another subnet 192.168.0.0/24 which is connected through pfSense IPSec as a Site-To-Site VPN.
      • OpenVPN clients are able to ping hosts on 192.168.1.0/24 subnet without any problems, but are unable to ping hosts on 192.168.0.0/24.
      • pfSense itself is able to ping 192.168.0.0/24 clients from "Diagnostics > Ping" menu, but unable to do so from shell.
      • We need to enable OpenVPN clients to access hosts on 192.168.0.0./24 subnet.
    • A:
      • You need two things. One, a push route on the OpenVPN server to send the clients a route. push "route 192.168.0.0 255.255.255.0" Then a second phase 2 with the local being 10.1.0.0/24 and remote 192.168.0.0/24, and vice versa on the remote end.
  • Setting Up Routing | OpenVPN
    • If you set up a routed VPN, you need to set up routing between the subnets so that packets will transit the VPN. Here is a possible network configuration.
    • Instructions for Linux
  • PFsense OpenVPN NAT - #11 by Token - Networking - Level1Techs Forums
    • PPPoE is Point to Point Protocol over Ethernet and is used for dialing ADSL modems from the router.
    • L2TP is Layer 2 Tunnelling Protocol, although technically relevant, its not here.
    • OpenVPN gets its routes for what local networks it can connect to via the server as upon connecting a user the server will push the routes to them. (see the openvpn server config -> IPv4 Local Networks)

Pushing Routes

  • networking - What is "push route" used for in OpenVPN? - Super User
    • route is used to allow a client remote access to a subnet (i.e. LAN) behind the router. push is specified in the server config to push the route directive to the client upon the client connecting to the server, negating the need to have the route directive in the client's *.ovpn config.
  • Difference "route" & "push route" commands - OpenVPN Support Forum
    • route 10.0.1.0 255.255.255.0 is used to add to local OpenVPN server's routing table only. And it may be used as on OpenVPN server as on client too.
    • push "route 10.0 .2.0 255.255.255.0" - is used only in OpenVPN server's config to push the routes to client's. Insteed of using "route" command on all client's config, you can use one "push route" on server config to do the same on all clients.
  • routing - OpenVPN: Push a route to client with a different gateway - Unix & Linux Stack Exchange
    • push "route 10.10.10.0 255.255.255.0 10.0.0.2 1"
    • --route network/IP [netmask] [gateway] [metric]
    • This tells the server config to "push" to the client, the route command which sets a networking route of the 10.10.10.0/24 subnet via the gateway 10.0.0.2 with a metric of 1. Metrics are used to give "preference" if multiple routes exist (such that the lowest cost wins).
  • ip routing - How to push a gateway and route to an OpenVPN client? - Server Fault
    • I just noticed that push "route-gateway 10.10.0.1" and push "route 10.10.0.0 255.255.0.0 10.10.0.1" pushes the gateway/route to the "static server clients" but not to the clients that receive their VPN IP via DHCP. Why are they not also applied to the DHCP clients?
    • push "route 10.10.0.0 255.255.0.0 net_gateway 1" - to use the client's gateway
  • '--route args' (from OpenVPN 2.5 Manual)
    • push"route {target_ip_or_network} {NET_MASK} {GATEWAY} {METRIC(PRIORITY)}"

push "route 123.123.123.123 255.255.255.255 10.0.0.1 1"     this does push the route but seems to be put after the tunnel
push "route 192.168.5.0 255.255.255.0 vpn_gateway 5"        vpn_gateway is just an alias
push "route 123.123.123.123 255.255.255.255 10.0.0.1 1"     push "route vpn_gateway 255.255.255.255 net_gateway 1"
    • Metric does not seem to match when using the commands above to those on my Windows PC.
  • Exclude IP from OpenVPN route - Stack Overflow
    • I have an OpenVPN server, and would like to make clients route an address range, like 1.2.3.0/24 through VPN. However, I do NOT want to make clients use VPN for a specific IP address on that same range.
    • You can add a more specific route for the IP address that you don't want to go through the VPN and use the net_gateway and vpn_gateway options to specify the gateway for the route.
    • This shows how 'net_gateway and vpn_gateway' should be used.

Network Topology

  • OpenVPN vs IPSec vs L2TP Vs WireGuard (Different types of VPN)
    • IPsec is good for site-to-site connections (Router to Router)
    • Layer 2 Tunnel Protocol (L2TP) is a VPN protocol that doesn’t offer any encryption. That’s why it’s usually implemented along with IPsec encryption. As it’s built into modern desktop operating systems and mobile devices, it’s fairly easy to implement. But it uses UDP port 500 — that means it can’t be disguised on another port, like OpenVPN can. It’s thus much easier to block and harder to get around firewalls with.
    • IPsec can be used on its own for point to point connectsion. If you are connecting 2 sites, use this.
    • You can connect clients in with IPSec and L2TP/IPSec such as windows and phones but it is not recommended.
    • L2TP on its own is unencrypted, that is why you find the term L2TP/IPSec because L2TP provides the tunneling and IPSec provides the security.
    • Which Is the Best VPN Protocol? PPTP vs. OpenVPN vs. L2TP/IPsec vs. SSTP - Want to use a VPN? If you’re looking for a VPN provider or setting up your own VPN, you’ll need to choose a protocol. Some VPN providers may even provide you with a choice of protocols.
    • PPTP vs IPSec IKEv2 vs OpenVPN vs WireGuard - Compare PPTP, IPSec IKEv2, OpenVPN and WireGuard to determine which VPN protocol offers the best combination of security, speed and ease of use for your needs.
    • VPN Protocols: OpenVPN vs IPSec, WireGuard, L2TP, & IKEv2 - This guide examines the different VPN protocols, including OpenVPN, IPSec, WireGuard, L2TP, and IKEv2 to see which performed the best.
    • VPN Protocols: From PPTP, L2TP, and OpenVPN to Wireguard - Don't get overwhelmed by endless acronyms like PPTP, L2TP, or IKEv2; learn what they really mean and how they can and can't protect you.
    • Which is better L2TP vs OpenVPN? | VPN Tutorials - Which is better L2TP vs OpenVPN?
    • IPSec Vs OpenVPN | 5 Differences between IPSec and OpenVPN - With all the threats surrounding the internet, the VPNs have become the popular option among users. This not only applies to organizati...
    • PPTP vs IPSec IKEv2 vs OpenVPN vs WireGuard
    • Types of VPNs | Mastering OpenVPN | packt
      • There are many VPN products available on the market, both commercial and open source. Almost all of these VPN products can be separated into the following four.
      • Excellent description of Openvpn technoclgies and the other types of vpn.
      • The IPSec standard is the official IEEE/IETF standard for IP security. It is officially registered as RFC2411 (see https://www.ietf.org/rfc/rfc2411.txt for the full standard). IPSec is also built into the IPv6 standard. = The way to go for for bridging two networks.
    • WireGuard vs OpenVPN: Is WireGuard Better Than OpenVPN? - OpenVPN and WireGuard are the two best VPN protocols available, but which is better and which should you use? Find out in this direct comparison guide.
    • Virtual Private Networks — WireGuard | pfSense Documentation
      • WireGuard is a new VPN Layer 3 protocol designed for speed and simplicity. It performs nearly as fast as hardware-accelerated IPsec and has only a small number of options in its configuration.
      • This implies that IPsec is harware accelerated.
    • WireGuard vs OpenVPN: Is WireGuard Better Than OpenVPN? - OpenVPN and WireGuard are the two best VPN protocols available, but which is better and which should you use? Find out in this direct comparison guide.
    • VPN Protocols Explained & Compared | Which Protocol Is Best? - Find out which VPN protocols will secure your data, which will put you at risk, and which will slow you down in this complete guide to VPN protocols.
  • IPsec
  • L2TP
    • A protocol to establish connections.
    • Is not required for OpenVPN to work, but probably can be used for something.
    • What is Layer 2 Tunneling Protocol (L2TP)? | NordVPN
      • The online world has traffic, and there are tunnels to protect it. What is Layer 2 Tunneling Protocol (L2TP) and how does it works with a VPN?
      • What is L2TP protocol? Layer 2 Tunneling Protocol (or L2TP) is a tunneling protocol used by both VPNs and internet service providers (ISPs). It doesn’t encrypt your content but simply creates a connection between you and a VPN server.
      • L2TP is applied as a configuration protocol for VPN services;
    • Layer 2 Tunneling Protocol - Wikipedia
      • In computer networking, Layer 2 Tunneling Protocol (L2TP) is a tunneling protocol used to support virtual private networks (VPNs) or as part of the delivery of services by ISPs. It uses encryption ('hiding') only for its own control messages (using an optional pre-shared secret), and does not provide any encryption or confidentiality of content by itself. Rather, it provides a tunnel for Layer 2 (which may be encrypted), and the tunnel itself may be passed over a Layer 3 encryption protocol such as IPsec.
    • L2TP/IPsec Remote Access VPN Configuration Example | pfSense Documentation
    • MikroTik Site to Site VPN with L2TP/IPsec - System Zone
      • Connecting remote workstation/client: In this method, a L2TP client supported operating system such as Windows can communicate with MikroTik L2TP server through L2TP tunnel whenever required and can access remote private network as if it was directly connected to the remote private network.
      • Site to Site L2TP: This method is also known as VPN between routers. In this method, a L2TP client supported router always establishes a L2TP tunnel with MikroTik L2TP Server. So, private networks of these routers can communicate to each other as if they were directly connected to the same router.
    • PfSence Firewall:How to Configure L2TP VPN Part03.2020 - YouTube no sound
  • TCP vs UDP?
    • UDP vs. TCP: What's the Difference & Which Should You Use?
      • UDP and TCP are communication protocols used to send data. Discover the differences between the two and learn which is better for your VPN connection.
      • This answers all of my questions and is an excellent article.
      • Applications will continue to use TCP for their connection inside your UDP VPN tunnel, which means any services that require TCP’s guaranteed packet delivery can still have it.
      • It is a common misconception that you should use a TCP-based VPN tunnel for applications that need extra reliability. This is not necessary, and often not a good idea. The inner connection still has TCP if it needs it – the VPN tunnel doesn’t need to add another layer.
      • If you wrap a TCP VPN tunnel around a TCP connection, you run the risk of “TCP meltdown”. This happens when both layers of TCP attempt to compensate for lost packets. It occurs because the inner and outer TCP layers have independent timers used to decide when a packet is lost.
    • TCP vs. UDP: Protocol & Port Differences - What’s the difference between TCP and UDP protocols and ports? Find everything you need to know about TCP and UDP, plus when and how to use them.
    • Why Does OpenVPN Use UDP And TCP? | OpenVPN
      • The OpenVPN protocol itself functions best over just the UDP protocol. Learn more about why we use both UDP and TCP here.
      • For example on a public network it can be quite normal to see that only traffic for protocols such as HTTP, FTP, SMTP, POP3, and IMAP, are allowed, with usually some additional ports for SSL secured versions of those protocols, like HTTPS. Those protocols are almost all TCP-only and not UDP. That is why TCP is an option to allow the traffic to pass on these restricted networks.
    • TCP vs UDP VPN: Learn the differences | NordVPN
      • Internet protocols can be confusing. How do they transfer data? What are their strengths and weaknesses? Read this post to learn more about TCP and UDP.
      • NordVPN wants to provide the best browsing experience without compromising on speed, so we use the UDP protocol by default. We recommend trying the UDP protocol first and only switching to TCP if you experience any issues
    • Is TCP or UDP better for VPN ? :: SG FAQ
      • Generally, TCP is more reliable than UDP, however, UDP performs better than TCP.
      • Generally, VPN UDP is better for VoIP, media streaming and gaming traffic, as well as high capacity broadband connections where packet loss is unlikely. TCP is better for overall reliability of transmitted data, and slower/unreliable connections (3G/4G/Wireless, Satellite, etc.)
      • OpenVPN has a TCP mode for highly unreliable connections but this mode sacrifices significant performance due to the inefficiency of encapsulating TCP within TCP.

OpenVPN Client Software

Where do I find the 'OpenVPN ID'

  • Whether the connecion is a server or a client is fairly straight forward and is found along with the ID when you know where to look.
  • You will probably be looking for this if you get the message in the CLi. This can be cause by an incorrect statement of the fact the number is missing. 
    Invalid OpenVPN ID, must be numeric
  • Edit the OpenVPN (Server|Client) service configuration
    • VPN --> OpenVPN --> (Servers|Clients) --> edit --> Unique VPN ID
  • Service Status Widget
    • Service name format: openvpn, openvpn_2, openvpn_3
    • It has client/server in the description
  • Interface Assignments / Network port
    • Service name format: ovpns1, ovpns1, ovpnc1, ovpnc2
    • In the service names: c = client, s = server
  • Status / Services
    • Service name format: They are all labelled 'openvpn'
    • It has client/server in the description
  • Notes

Bridging

  • OpenVPN Bridging Tutorials (pfsense)
    • Configure Layer 2 'tap' firewall VPN with Open Source PfSense & OpenVPN | by Sarathi Balakrishnan | Medium
      • Seems to be incomplete
    • How to setup pfSense to act as OpenVPN server for Ewon devices (PDF) - This document lists the different steps to configure pfSense to work as OpenVPN server in
      Bridged mode and how to connect Ewon devices to the pfSense
    • Howto: Bridged L2 VPN without "tunnel network" / Control over "server"directive | Netgate Forum - jimp (Netgate) says how to set it up in 4 paragraphs.
      • First, read all of the text descriptions on the new fields that show up when you switch to TAP. The notes are important.
      • Add a new VPN instance, select tap, fill in all your other info as you want, If you want to provide DHCP to clients, check the box to bridge DHCP, select the interface you will be bridging to, and (optionally) fill in the DHCP server pool. If you fill in the DHCP server start/end it should be a range of IPs outside of your existing DHCP pool. If you leave the IPs blank, it will pass DHCP through to your LAN DHCP server.
      • After you save the VPN settings, go to Interfaces --> (assign), assign the new VPN interface. Go to Interfaces --> OPTx, enable, leave IP type as "none", save. Go to Interfaces --> (assign), bridges tab, bridge the VPN interface and your LAN or whatever internal interface(s) you want.
      • Go to Firewall --> Rules, on the VPN interface be sure to add rules there that will pass DHCP and whatever other traffic you want (or just pass any/all).
    • Bridge confusion | Netgate Forum
      • Q:
        • I am trying to follow the pfSense book to configure my OpenVPN server using the 'tap' device mode. I am confused as to whether or not a bridge is actually needed.
        • from the Docs "If Bridge DHCP is selected, DHCP will be passed through to the bridged interface that will be setup later. In the most common scenario, this would be LAN. Using this method, connecting clients would receive IPs from the same DHCP pool used by directly wired LAN clients."
        • Does this mean that I should declare the LAN as the "Bridge Interface"? Or, does it simply mean that the LAN interface is typically the DHCP server, BUT you must still create a Bridge Interface to use the DHCP server running from the LAN interface?
      • A:
        • jimp (Netgate)
        • If you want the VPN to be connected to LAN you must do both. Selecting LAN for the bridge in OpenVPN does not create a bridge, it only tells it where your LAN network is. You must create the LAN/OpenVPN bridge yourself separate from that setting.

Troubleshooting (OpenVPN)

In this section I will deal with OpenVPN troubleshooting for both client and server.

General
  • Troubleshooting pfSense | pfSense Documentation - All aspects of pfSense are covered by troubleshooting topics here.
  • Troubleshooting OpenVPN | pfSense Documentation - This section describes several troubleshooting techniques for OpenVPN, as well as common issues users encounter with OpenVPN along with their solutions.
  • Troubleshooting — Troubleshooting OpenVPN Internal Routing (iroute) | pfSense Documentation - For a site-to-site PKI (SSL) OpenVPN setup with a tunnel network larger than /30, OpenVPN must have an internal route for the client subnet. Without the internal route, the firewall will forward traffic into OpenVPN but OpenVPN will drop the traffic as it has no way to determine the proper destination. There are a couple common scenarios where this may have difficulties.
  • OpenVPN Bridge on pfsense: once LAN pings clients, connectivity breaks - Server Fault
    • System -> Advanced -> firewall/NAT -> disable scrubbing
  • If your OpenVPN do not show as online after intial setup, reboot your router and it will probably be fine.
  • Troubleshooting Reaching Systems Over The VPN Tunnel | OpenVPN
    • Having trouble reaching systems over the VPN tunnel. Get your answers and solutions here.
    • What we mean by connection path problems is the path between the OpenVPN client and the target server you're trying to reach. We are specifically not talking here about problems with establish the OpenVPN tunnel itself. That is handled on a separate page: troubleshooting client VPN tunnel connectivity problems.
    • This page deals with doing tests that eliminate possibilities until a conclusion emerges that you can use to effectively resolve the issue.
  • DNS
    • Troubleshooting DNS Resolution Problems | OpenVPN
      • The guide provides a way of checking to see if the DNS query you are doing from your OpenVPN client device, is actually making it through the VPN tunnel.
      • Split-DNS is the principle of resolving only certain zones (domains) through a DNS server pushed by the VPN server, and the rest through your already present local DNS servers.
  • Site to Site
pfSense
  • OpenVPN client Gateway showing 100% packetloss
    • Majority of VPN provider ignore ping which explain why you getting 100% Packetlost
      • Try my solution below and if that does not work, then you will need to diable gateway monitoring.
    • My solution is easy, don't specify a monitor IP address.
      • System --> Routing --> [Your VPN Gateway] --> edit
        • Configure as follows
        • Disable Gateway Monitoring = unticked
        • Disable Gateway Monitoring Action = unticked
        • Monitor IP = empty
      • What happens is that if not monitor IP is set pfSense uses the 'Virtual IP' presented by the OpenVPN client to PING and this only becomes available after the OpenVPN client has successfully connected.
      • I suspect that PINGs to the outside world are happening before the client/route is initialised or that the PING is not being routed and this is why if you use a real IP the monitoring is not working and because of this failure, pFsense flags the Gateway as offline.
    • OpenVPN client showing 100% packetloss following 2.5.0 upgrade | Netgate Forum - This thread has the common ideas and workarounds.
  • Unable to check for updates
    • The error
      • System --> Update --> System Update --> "Unable to check for updates"
    • Causes
      • This can be caused if you have not kept up to date with updates and pfSense gets confused.
      • This might be caused if there are no updates.
      • The Netgate servers are overloaded.
      • Your configuration is blocking the update servers.
    • Solution 1
      • Refresh the page: System --> Update --> System Update
    • Solution 2
      • System --> Update --> System Update Settings
      • Select "Current Stable Release (x.x.x)"
      • Click Save
      • System --> Update --> System Update
      • You should now see any updates and not the error message.
    • A Working System
      • An update is available
      • The system is up to date
OpenVPN Server
  • Errors: 
    • AEAD Decrypt error: bad packet ID (may be a replay)
OpenVPN Client
  • Troubleshooting — Troubleshooting Windows OpenVPN Client Connectivity | pfSense Documentation - Historically, OpenVPN client software on Windows had issues with routing due to a lack of privileges. Current versions of the OpenVPN client software for Windows run as a service which only requires administrative privileges during the installation process and not when the client software runs afterward.
  • Errors
    • WARNING: You have specified redirect-gateway and redirect-private at the same time (or the same option multiple times). This is not well supported and may lead to unexpected results
      • Error Explained
        • In pfSense the redirect-private is never actually used from what I can see which means that the redirect-gateway is being called multiple times and this is not recommended.
      • Causes
        1. Redirect IPv4 Gateway is enabled there is an entry in the hidden field IPv4 Local network(s).
        2. Redirect IPv6 Gateway is enabled there is an entry in the hidden field IPv6 Local network(s). I have not verified this one but should be the same as the IPv4.
        3. Redirect IPv6 Gateway is enabled but you do not have IPv6 enabled. This might just causes a second entry in the log rather than actually causing the error which is just replicated when present by having 2 redirect gateway commands issued.
        4. redirect-gateway is overridden in either the Server or Client Custom Options.
      • Solutions
        1. Remove the check mark at 'Redirect IPv4 Gateway', are there any entries in 'IPv4 Local network(s)'? If so remove them and re-check redirect gateway.
        2. Remove the check mark at 'Redirect IPv6 Gateway', are there any entries in 'IPv6 Local network(s)'? If so remove them and re-check redirect gateway.
        3. If not using IPv6, disable 'Redirect IPv6 Gateway'.
        4. Remove all custom 'redirect-gateway' from 'Custom Options' for both server and client.
      • What Now
        • You should now cured this error and know what rule caused it.
        • Leave or fix the rule that is causing the issue.
        • This error can be ignored as everything still works. Ignoring it though, is not ideal.
      • Notes
        • redirect-private
          • This option behaves very similar to the redirect-gateway directive, especially when the new parameters are used, but it does not alter the default gateway.
          • I cannot find where the option redirect-private is being added (possibly having the redirect IPv6 when the protocol is not enabled)
            • Not in the Client config
            • Not in the server config: /var/etc/openvpn/
          • In the OpenVPN reference manual it gives the following information
            --redirect-private [flags]
    Like --redirect-gateway, but omit actually changing the default gateway. Useful when pushing private subnets.
      • Links 
    • WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
      • Error Explained
        • Don't cache --askpass or --auth-user-pass username/passwords in virtual memory.
        • The OpenVPN client is by default is allowed to store the username and password in memory and/or on the disk and this can be a security risk so a warning is issued.
      • Solutions
        • Disabled the ability to cache the username and password by adding auth-nocache into the client's Custom Options which in pfSense adds it to the client's .OVPN file.
          • This will require the username and password to be re-entered on every connection and potentially every hour if the connection timeout is still set to the default.
        • You can put your login and password to the text file login.txt and add option 'auth-user-pass login.txt' to your config. This suppress login window and you can use auth-nocache option without reasking password.
          • The risks with this is that your password is stored in plain text on your computers harddrive.
      • What Now
        • This warning is not causing any issue with your connection so depending on your situation you can probably just ignore this.
      • Notes
        • By default OpenVPN will want to reconnect every hour.
        • This command cannot be pushed, if you try you will get an error as follows: "Options error: option 'auth-nocache' cannot be used in this context ([PUSH-OPTIONS])"
      • Links 
    • WARNING: OpenVPN was configured to add an IPv6 route. However, no IPv6 has been configured for OpenVPN TAP-Windows6, therefore the route installation may fail or may not work as expected.
      • Error Explained
        • This one is exactly what it is says.
      • Causes
        • You have enabled Redirect IPv6 Gateway in your OpenVPN server but do not have the IPv6 protocol enabled.
      • Solutions
        1. Enable the IPv6 protocol on your OpenVPN server.
        2. Disable 'Redirect IPv6 Gateway' on your OpenVPN server.
    • WARNING: OpenVPN was configured to add an IPv4 route. However, no IPv4 has been configured for OpenVPN TAP-Windows6, therefore the route installation may fail or may not work as expected.
      • Error Explained
        • This one is exactly what it is says.
      • Causes
        • You have enabled Redirect IPv4 Gateway in your OpenVPN server but do not have the IPv4 protocol enabled.
      • Solutions
        1. Enable the IPv4 protocol on your OpenVPN server.
        2. Disable 'Redirect IPv4 Gateway' on your OpenVPN server.
    • Warning: route gateway is not reachable on any active network adapters: 10.0.0.1
      • Error Explained
        • It is saying that it cannot reach my pfSense gateway of 10.0.0.1
      • Causes
        • Possibly the correct routes have not been pushed
      • Solutions
        • This error went away when I rebooted my pfSense router.
      • Notes
        • When I got this error I was testing my OpenVPN enviroment a lot and changing settings often.
  • After one hour of use, OpenVPN asks me to sign in again
    • After one hour of use, OpenVPN request me again to sign in for unknow reason. | Netgate Forum
      • Cause
        • OpenVPN renegotiates every hour by default.
        • Caching authorization on the client means you generally do not notice.
        • People tend to see problems when they employ multi-factor authentication.
      • Solution
        • Adding this to the client disables negotiation from the client side: reneg-sec 0;
        • That can be added in the client exporter or usually directly on the client.
        • You can then control renegotiations on the server with something like: reneg-sec 43200;
  • Client IP not showing up in system tray or OpenVPN client GUI
  • No Gateway assigned to client but everything still works
    • OpenVPN no Gateway assigned to client | Netgate Forum
      • OpenVPN does not assign a gateway.
      • It installs two routes: 0.0.0.0/1 and 128.0.0.0/1
      • These cover the entire IPv4 internet, are more specific than 0.0.0.0/0 so they are preferred, and relieves OpenVPN of the duty to save and put back the existing default gateway when it closes.
      • This is governed by the Redirect IPv4 Gateway checkbox in the server configuration.
      • Redirect IPv4 Gateway = redirect-gateway def1 option.
      • The client can also choose to ignore the gateway routes pushed by the server.
      • In the pfSense client configuration, that is the Don't pull routes checkbox.
      • No need for a gateway when this route is installed.
  • OpenVPN clients can still see local LAN even with redirect IPv4/6 Gateway Options selected? : PFSENSE | Reddit
    • Redirect IPv4 Gateway and Redirect IPv6 Gateway do NOT prohibit communication with other devices on the same subnet, those options ONLY effect routing of traffic outside of the local subnet.
    • It is working as intended.
    • You would need to add block-local gateway flag (push this via the server or add to the client .OVPN). The server push command would look: 
      push "redirect-gateway def1 block-local"
      • pfSense currently does not have the ability to configure this flag in the GUI
  • OpenVPN TAP works, but cant access any services on the router | Netgate Forum
    • A few people had issues with OpenVPN and pfBlockerNG
    • The problem seemed to be pfblockerng.
    • Solution: Stop and start pfBlockerNG
      • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration --> Outbound Firewall Rules: LAN, BRIDGEDVPN, OpenVPN
      • Click 'Save'
      • Firewall --> pfBlockerNG --> General Settings --> pfBlockerNG: unticked
      • Click 'Save'
      • Firewall --> pfBlockerNG --> General Settings --> pfBlockerNG: ticked
      • Click 'Save'
      • Diagnotics --> Reboot --> Normal reboot
      • Click 'Submit'
      • Diagnostics --> Reboot
      • NB: it is more than likely that just rebooting the router that did the trick, but just in-case it wasn't.
  • How to Fix OpenVPN block-outside-dns Problems in Windows 10 - Arador - If you’re using an OpenVPN client on Windows 10 and you use the block-outside-dns option to prevent DNS leaks then you may experience some serious connection delays. The easiest way to fix the block-outside-dns connection issue is through the Network and Sharing Center, here’s how to do it
  • Set PC routing table entry metrics
    • Windows 10 DNS resolution via VPN connection not working - Super User
      • I have fixed this problem permanently by manually setting the metric of my LAN connection to a higher value than the metric of the VPN connection.
      • This can be done in two ways
        • Through the GUI: Network connections > Properties > double click IPv4 > Advanced > Uncheck Automatic Metric > Enter 15 for interface metric > OK > OK.
        • Command line: netsh int ip set interface interface="LAN CONNECTION NAME" metric=15
    • Default gateway route doesn't appear in Routing Table - Windows Client | Microsoft Learn - Provides a solution to an issue where default gateway route doesn't appear in the Routing Table.
    • Viewing and Changing Your Computer's Routing Table
      • Here, you use the route command to view and change your computers internal routing table. Even though your computer is not a router, it maintains an internet routing table with entries for the network interface network, the loopback network, and details of other internal networks.
      • Metric - The metric assigned to the route. If there are two entries for the network destination, the lower metric is the route chosen.
  • Find out the IP route of a packet (windows)
    • Use 'pathping' as it is the easiest command to see which gateway your traffic is going.
    • 'route print' will show the route table in windows
    • networking - On Windows, how to determine route for IP destination? - Super User
      • pathping - The pathping command is similar to tracert but includes the outgoing interface.
      • tracert - Find the path to the target in hops.
      • Find-NetRoute - This is probably the closest you're going to get to ip route get on Windows.
  • How do I changes the automatic connection timer when I start the OpenVPN client
    • OpenVPN-GUI-New - Entering Username and Password | OpenVPN Community
      • The way it works is like this: if username/password is available, the dialog window is prefilled and displayed for 6 seconds. If during that time the user clicks on the window, the timeout is cancelled and the dialog stays on the screen until manually submitted. Otherwise it's auto-ubmitted after the 6 seconds timeout.
      • The 6 seconds value is not configurable.
      • If silent-connection is enabled and the username/password are saved then the dialog is not shown.
  • How to prevent OpenVPN connect from connecting at startup? - Super User
    • Some workarouns listed here such as chaninging the service's start type.
Cannot use 192.168.70.x for Routed VPN because of Android Hotspots

You cannot use 192.168.70.x as shown in the Lawrence Systems YouTube video. We must change IP Subnet to prevent issues with Android Hotspots.

This might apply to more than Android Hotspots, but I have been bitten by this one while making this tutorial. I was using a laptop connected to my Android Hotspot to test the Routed VPN and one day when I connected, things were not working (Internet/Local Web server/DNS) so I checked and re-checked all of my settings and firewall rules but could not find any issue. It was only by chance I discovered that the Android Hotspot was using the IP range 192.168.70.0/24 and then the penny dropped.

The Android Hotspot is hardcoded to use 192.168 for the first 2 octects and then a randomly choosen 3rd Octect (192.168.X) for it's network on each connection giving a network in the range (192.168.1.0 - 192.168.254.0). The gateway and IP addresses are also random on each connection.

The only way to avoid this issue is to use a network subnet for the routed VPN that will never be assigned by the Android Hotspot, so I chose 10.200.1.0/24 and used it above.

BridgedVPN (TAP) client connects, but does not get a DHCP address

This is a common and frustration situation to be in. The following are common reasons for DHCP not working:

  • OpenVPN Server Settings
    • 'Bridge Interface' should be 'LAN'
      • Most people make the mistake of setting a bridge interface here which is wrong. You should set the traget networks interface which in most people;s case is 'LAN'
      • "The interface to which this TAP instance will be bridged. This is not done automatically. This interface must be assigned and the bridge created separately."
  • pfBlockerNG
    • one of its rules is blocking DHCP requests/responses
      • Solution 1 - Disable pfBlockerNG (good for checking it is pfBlockerNG but not a permanent solution)
      • Solution 2 - Find the offending group and feed, then disable it
        • I found the firewall/pfBlocker rule that was triggered, opened the relevant group, took each feed and opened them in my browser and looked for entries that would block DHCP
        • In my case my firehol_v4 rule and in particular the Level 1 feed had  0.0.0.0/8 (blocks DHCP) and 10.0.0./8 (blocks local address range)
      • Solution 3 - Whitelist in broadcast traffic pfBlockerNG
      • Solution 4 - If your are not getting any traffic even on the normal network, Stop and Start pfBlockerNG as outlined above (Solution: Stop and start pfBlockerNG), not just a reboot.
  • Snort
    • Are any Snort rules blocking DHCP. Check the logs.
  • Bridge
    • Have you created the bridge
    • Have you created the bridge interface. If you did you should delete it as it might have unwanted firewall rules on it, or the lack of them.
  • Firewall Rules
    • Are your firewall rules allowing the DHCP packets to pass or even blocking the packets.
    • You should have a DHCP allow rule on the BRIDGEVPN interface.
  • Client
    • Dont rely on the client to show you your IP. When using TAP you will not see an IP in the client, your need to use ipconfig at the command prompt.
      • [SOLVED] Client has no ip addressed assigned - OpenVPN Support Forum - This is "by design" : server-bridge + remote DHCP has the side effect that the openvpn client does not bring up the tap0 interface by itself - this is left to the OS; on windows this is done automagically, on linux you need to run something like 'dhclient tap0' ; this can be added to the client config using 
OpenVPN Clients cannot PING or connect to LAN clients and Vice Versa

This is mostly an issue for RoutedVPN because the OpenVPN network and the LAN network are on different subnets however some fixes will be valid for BridgedVPN.

At this point you have to understand your pfSense router is behaving as instructed and if you are having issues, some further configuration is needed.

You more than likely can use the internet through your VPN and connect to the router aswell as ping it, but not much else.

Causes

  • A device has Ping disabled or is not capable of a ping response.
  • Device Firewall(s)
    • Windows Defender Firewall
      • Windows Defender Firewall by default only allows traffic from local subnet for private and public profiles (domain is not used unless you are logged intoa domain and in which case there are no restrictions)
      • Does not have PING enabled (on the correct profile Domain/Private/Public)
      • PING is restricted to the same subnet
      • File sharing is not enabled (on the correct profile Domain/Private/Public)
      • File sharing is restricted to the same subnet
    • Comodo Firewall
      • Windows Defender Firewall is not automatically turned off when CIS is installed (because of M$)
      • No allow rules added for the remote subnet/network
    • Other Firewalls
  • pfSense Firewall Rules (or lack off)
  • pfBlockerNG
    • One of its rules generated from a feed is blocking access to the network
      • In my case my firehol_v4 rule and in particular the Level 1 feed was blocking the ranges 10.0.0.0/8, 0.0.0.0/8, 255.255.255.255 and others.
      • This only causes issue if you are connecting to your VPN server from your local network which you usually do when testing.
NAT Reflection is not working

Can NAT Reflection work over OpenVPN?

The answer is yes and no.

  • No: If you only have 1 public IP address because your OpenVPN will be on the same Public IP as your assets such as a webserver
  • Yes: If you have 2 Public IPs and the assets you are trying to access are not on the same public IP as your OpenVPN server.

Explanation of why NAT Reflection will not work with 1 public IP

It is not an issue of whether you can do NAT Reflection over OpenVPN but more a matter of IP routes.

For the purpose of the example we will assume the following:

  • Setup
    • You have 1 public IP which is running your OpenVPN server and your Webserver
    • OpenVPN TAP server (called BRIDGEDVPN) which is bridged to my LAN via a bridge (bridge0) which does not have an IP or an interface
    • 'Redirect IPv4 Gateway/Force all client-generated IPv4 traffic through the tunnel.' is on (as per this article)
    • I have added the 'allow DHCP rule' and 'Allow traffic on the brideged interface' rule on the BRIDGEDVPN interface.
    • I am connecting in remotely with a laptop over OpenVPN
  • What Works
    • My openvpn clients get an IP from DHCP etc...
    • My devices on the OpenVPN client can talk to devices on my LAN
    • I can get the internet from the LAN and the OpenVPN clients
    • I can talk to my router either by IP or by its FQDN
    • On my LAN, the devices can access my webserver via its FDQN because I have NAT Reflection on.
    • DNS on both segments seem correctly hijacked by my firewall DNS floating rules which are attached to the OpenVPN interface.
    • My webserver is fully available from the internet
  • NAT Reflection is not working
    • I cannot access my webserver via its FQDN on my LAN network segment
    • Tracert gets no response from the pings and therefore no route

From the example above the OpenVPN clients can access the internet and talk to local clients with no issue, they just cannot talk to the webserver ony uour LAN and the websites hosted on it, whereas the direct LAN clients can.

But I hear you say "is the OpenVPN connection not bridged to the LAN?", and you would be right about this. The two interfaces are bridged so you would think that the traffic would go onto the LAN and then through the WAN as traffic direct from the LAN, which is exactly how I thought it should.

After many hours on diagnostics, Packet Sniffing and monitoring of logs I came across the answer:

The following statements below happen to the OpenVPN client as it tries and connects to your local webserver:

  • Your OpenVPN and Webserver are on your public IP 123.123.123.123
  • On your remote OpenVPN client Windows PC once the VPN is connected:
    • There is a route to your public IP over the internet through your normal internet connection
    • There is route to your private network through the VPN tunnel (10.0.0.0/24)
    • You have an IP from your LANs DHCP server (10.0.0.171)
  • OpenVPN connects to your public IP over the internet using the route from the routing table. It can only ever go this route.
  • OpenVPN captures all IPv4 traffic (except the actual VPN tunnel traffic because you cannot send the VPN connection down itself)
  • You now try and connect to your website hosted on your local server quantumwarp.com
  • The DNS request is sent over the VPN tunnel and is received correctly
  • 1 of 2 things will now happen depending on the setting of 'Bridge Route Gateway' in your BridgedVPN OpenVPN server
    • Bridge Route Gateway = Off
      • You will not be able to access your website
    • Bridge Route Gateway = On
      • You can access your website

Answer

  • All traffic that needs to go your public IP (123.123.123.123) will always go down the first route it finds in the computers routing table, and because the first route it finds is the same one that your OpenVPN client is using to connect to your VPN server, the traffic will be sent down that one.
  • This means that any traffic that needs to go to 123.123.123.123 will never go down the VPN tunnel.
  • The open VPN client does not do any packet inspection to be able to say when traffic for 123.123.123.123 should go down the tunnel or the open internet.
  • When 'Bridge Route Gateway' is enabled you can access your websites because
    • Any traffic in the tunnel that is destined for 123.123.123.123 has a route over the open internet available to it and it will use it.
    • This option pushes a gateway/route to allow the traffic to go over the open internet by using the local adapters gateway.
    • This is less secure because traffic is going out over the open internet.
    • You will not have access to hidden/restricted services that are only available when access your webserver from the LAN.
    • RoutedVPN does not have this setting but behaves as if it were on.
  • If the 'Bridge Route Gateway' option is disabled then any traffic destined for 123.123.123.123 (except for the VPN tunnel traffic) will NOT flow out from the OpenVPN client either over the adapter gateway or the over the VPN tunnel to your pfsense router.
  • The OpenVPN client cannot distinguish between tunnel traffic and webserver traffic because they share the same destination IP, so traffic from the OpenVPN client destined your webserver never reaches your LAN because it is never routed over the VPN tunnel therefore NAT Reflection on your pfSense router for the webserver traffic never takes place so rather that it not working it is just never required. The webserver traffic is either routed over your open internet connection or it never leaves the OpenVPN client because it has not route which explains the 'Bridge Route Gateway' option.

Suggested Fixes

These are just ideas I had:

  • Use another OpenVPN client that is able to route the webserver traffic down the VPN tunnel (not found one).
  • OpenVPN could improve their client to allow for routing webserver traffic down the VPN tunnel.
  • Push DNS override via the OpenVPN server config

Workarounds

  • You can use your windows hosts file to do a DNS mapping to the local IP, when you are connected by OpenVPN
  • VPN in to your network and then connect in to a local computer with RDP/VNC

Solutions

  • 'Split Horizon' DNS (Split DNS)
    • Quick answer: So you add an entry in your DNS resolver to point your local domains to your webserver's local IP so you always access by the 10.0.0.x address
  • 2 Public IPs
    • Host your OpenVPN server on a different IP that that of your other webassets
    • Both IPs can be present on your pfSense router.
OpenVPN Clients not getting DHCP address

I will now outline some steps to run through to diagnose where the fault it.

Start by going down the checklist and they see if any of the sultions in this section will fix the problem.

Checklist
  • Reboot your pfSense Router
  • Check your ISP has not got a firewall on
    • I have recently upgraded to FTTP from FTTC and even though it says the firewall is still off, their systems might of turned it back on and not changed the status iin my ISPs cpanel. Turn it on and off again.
  • Is your remote IP allowed in the firewall rules, I have mine restricted.
  • If using a mobile phone as a hotspot, check it's internet connection has not swapped from the mobile connection to your local WiFi which is your local LAN (where your pfsense router is).
  • Setting python to normal 'unbound' mode
  • Enabled logging on your OpenVPN NAT rule
    • should be on: Firewall --> Rules --> WAN
  • Examine the Firewall log and see if a rule is blocking the OpenVPN connection
  • Run through the OpenVPN section to make sure your settings are correct such as:
    • VPN --> OpenVPN --> BridgedVPN --> Edit --> Bridge Interface = LAN
    • You have not assigned and interface to the BridgedVPN bridge
  • pfBlocker
    • I discovered IPs and ranges were being blocked by the firehol_v4 group and in particular the Level 1 feed:
      • 10.0.0.0/8 - This IP range block is only an issue if you are trying to connect to your OpenVPN service locally (usually for testing).
      • 0.0.0.0/8 - This is used as souce IP for broadcasts, the most common is DHCP.
      • 255.255.255.255 - This is used as the destination for broadcast traffic.
      • Some IGMP traffic.
    • I identified the individual feed by opeing the Feed group and opening each active feed in my browser and searching to see if my IP was listed.
    • I used my phone as a hotspot for testing instead of disabling a feed as there are many IPs blocked in this list and I wanted to keep them active.
    • Solutions
      • Disable any offending feed.
      • Whitelist broadcast traffic
  • Snort
    • Check Snort logs to make sure this IDS is not blocking the OpenVPN traffic.
  • Enable OpenVPN logging
    • VPN --> OpenVPN --> RoutedVPN/BridgedVPN --> Edit --> Advanced Configuration --> Verbosity level = 4
    • Level 4 is the best
  • Examine the OpenVPN Logs
    • Status --> System Logs --> OpenVPN
    • Have a look to see if there are any errors related to your connetion. You can filter by IP if required.
  • Check the certificates are valid
    • System --> Cert. Manager --> Certificates
  • Check to see if there are any patches for the issue
    • System --> Patches
  • Check to see if the client is the issue
    • Uninstall and re-install the OpenVPN client
    • Check the Passwords
pfBlocker - Allow Broadcast Traffic (BridgedVPN)
  • Firewall --> pfBlockerNG --> IP --> IP Interface/Rules Configuration

Sometimes when you import a feed it will include certain IPs or ranges that will prevent broadcasts being allowed over the bridge.

These solutions will allow DHCP, IGMP, NetBIOS and other broadcast traffic accross the OpenVPN Bridge so your connection into your network works as expected. You can just allow DHCP if you wanted but this is a more complete solution.

We will use the inbuilt pfBlockerNG mechanism so the rules are always placed where they should be.

IPv4 Suppression (preferred method)

  • This removes the entries from pfBlockerNG's Deny aliastable, pfBlockerNG then acts as if the IPs were never added in the first place.
  • This method will still require your traffic to be allowed on your interfaces with firewall rules.
  • You can be more specific with the rules on your interfaces when using this method.
  • These entries might get lost in a large list if you use the IPv4 Suppression feature a lot, but they will still work.
  • Firewall --> pfBlockerNG --> IP --> IPv4 Suppression
    • Add the following lines to the top of the list: 
      224.0.0.0/24 # Non-Routable Multicast address range
239.255.255.250/32 # Simple Service Discovery Protocol address
239.255.255.253/32 # Service Location Protocol version 2 address
255.255.255.255/32 # IPv4 Broadcast address
    • Firewall --> pfBlockerNG --> Update --> Force Reload (IP)

Whitelist - IPv4 Custom_List

  • Your LAN and BRIDGEDVPN must assigned as Outbound Firewall Rules in pfblocker
  • This method works by adding a firewall allow rule before the pfBlockerNG Deny rule are processed.
  • With this method you do not need rules on the interfaces to handle broadcasts as they are already allowed, but I would not delete them as you might disable pfBlockerNG at some point.
  • This will not let IGMP traffic with 'IP Options' pass. This is because you cannot enable the 'IP options' setting on the firewall rules that are created.
  • Firewall --> pfBlockerNG --> IP --> IPv4 Summary --> Add
    • Info
      • Name: Whitelist_Broadcasts
      • Description: Allow broadcast packets through the OpenVPN TAP Bridge
    • IPv4 Source Definitions
      • none
    • Settings
      • Action: Permit Outbound
      • Update Frequency: Never
    • IPv4 Custom_List
      • Enable Domain/AS: unticked
      • 224.0.0.0/24
239.255.255.250
239.255.255.253
255.255.255.255
    • Save
    • Move the entry to the top of the 'Pv4 Summary' and Save. (This is not requiered but makes things easier to understand having your Whitelists at the top.)
    • Reload the lists (Firewall --> pfBlockerNG --> Update)
  • In pfBlockerNG you will find it's advanced rules will only accept aliases of Network(s) type.
  • If you wanted to further restrict by source you can do, but this will not allow IGMP packets because of a limitation in 'FreeBSD packet fence'. This might get fixed/improved in later version of pfSense.
    • Advanced Outbound Firewall Rule Settings
      • Custom Source
        • Enable: ticked
        • Invert: unticked
        • Source: Broadcast_Sources (alias created earlier)
Certificate Issues (advanced)

I have seen this as a recommendation to regenerate the certificates fixed the issue even though they were valid. Only do this as a last resort because you have to revoke certificates and all the admin that goes with that procedure. I did not find this to work when I did it. My issue was caused by an issue from the checklist I missed.

  • Regenerate Certificates
  • Use Strict Security for certificates
  • Regenerate OpenVPN TLS Key
  • "CRL has expired" error
    • This is caused by a pfSense v2.6.0 bug and you get this or similiar error 
      VERIFY ERROR: depth=0, error=CRL has expired: C=GB, ST=Lancashire, L=London, O=QuantumWarp, CN=user, serial=4
    • Solution
      • Install 'System Patches' package
      • System --> Patches
      • Apply the patch: Fix for CRL expiration lifetime default and maximum values (Redmine #13424)
      • Reboot Firewall
      • NB: you might have to generate and delete a certificate top regenerate some things. See article below.
    • pfsense - CRL has expired in openvpn server
      • A few days ago we ran into an issue where pfsense appliances started to refuse openvpn connections by showing "CRL has expired" error messages. As it shows the reason is an overflow a date.
      • This is an indepth analysis of the issue and resolution.
    • How to fix "CRL has expired" openvpn error on pfSense - Pasquale 'sid' Fiorillo
      • Straight to the point and solution.
      • Mentions the 'System Patches' package
    • CRL = Certificate Revocation List
Cannot Ping other devices (with diagnostics)

Ping and Connect to other devices

  • Try PING'ing or connecting to various devices to see if it is just the one device you are having connectivity issues with.
  • This might seem simple, but Windows PCs do not respond to ICMP Ping by default from different subnets. The same goes for Windows file sharing.
  • If you can connect and ping other devices, especially no windows devices then it is an issue with the remote device.

Enable Logging on firewall rules

  • To see rules being flagged in the firewall logs, they first need to have the 'Log packets that are handled by this rule' enabled.
  • This is a very useful tool to see what is going on.

Disable the Firewall on the remote device you are trying to ping

  • On the remote device you are trying to ping, disable any 3rd party firewall and if a Windows PC also make sure the Windows Defender Firewall has been disabled.
  • If you find you can now ping the device then it is the device's firewall(s) on this device that need fixing.

pfSense Firewall Rules (or lack off)

  • There is no one size fit all solution here.
  • Just check you have ICMP allowed on the firewall for the relevant interfaces.
  • Putting 'Allow all' rules on various interfaces temporarily could help.

Diagnose pfSense Firewall rules using Packet Capture

These are some quick instructions on how to make sure your packets/traffic is getting where it should so you don't tear your hair how trying to guess all of the time.

  • Connect a PC to your RoutedVPN network
    • Verify you have internet and can ping the router. If using a Windows PC make sure you have allowed ping from different subnets.
    • We will call this computer VPNPC.
    • We will assume it's IP is 10.200.1.2
  • Get another PC and put it on your LAN network.
    • Make sure you can ping this PC from the LAN network.
    • We will call this PC LANPC.
    • We will assume it's IP is 10.0.0.189
  • On the VPNPC run the following command 
    ping 10.0.0.189 -t
  • Login to your pfSense router and got to
    • Goto: Diagnostics --> Packet Capture
    • Select the 'Interface': OpenVPN Server: RoutedVPN
    • Enter the 'Host Address': 10.200.1.2 (LANPC)
    • Click 'Start'
    • Wait for a couple of pings from VPNPC
    • Click 'Stop'
    • You should now see the ping results showing at the bottom of the page (you might have other traffic showing aswell)
      • Success - You can see a request and then a response.
        14:19:24.151898 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 114, length 40
14:19:24.151905 IP 10.0.0.189 > 10.200.1.2: ICMP echo reply, id 1, seq 114, length 40
14:19:25.153165 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 115, length 40
14:19:25.153170 IP 10.0.0.189 > 10.200.1.2: ICMP echo reply, id 1, seq 115, length 40
14:19:26.151414 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 116, length 40
14:19:26.151420 IP 10.0.0.189 > 10.200.1.2: ICMP echo reply, id 1, seq 116, length 40
14:19:27.192176 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 117, length 40
14:19:27.192181 IP 10.0.0.189 > 10.200.1.2: ICMP echo reply, id 1, seq 117, length 40
      • Failure - There are no responses, just requests. 
        14:20:55.191734 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 118, length 40
14:21:00.152081 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 119, length 40
14:21:05.152389 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 120, length 40
14:21:10.194476 IP 10.200.1.2 > 10.0.0.189: ICMP echo request, id 1, seq 121, length 40
    • You should test for traffic on each Interface in the following order until you find where the break is.
      • OpenVPN Server: RoutedVPN (done above)
      • OpenVPN
      • ROUTEDVPN
      • LAN
    • When you get to the LAN Interface and you can still see the traffic, you know that the packets are getting routed correctly to the LAN.
      • You dont need to see a ICMP response, just the fact the packets are arriving at the LAN is enough.
      • This means it is likely the device you are pinging is not responding rather than a routing issue.
      • Check Firewall rules on the device (i.e. Windows and subnets)
      • Check firewall there are no firewall rules blocking traffic in the opposite direction.
    • Still not working, go and have a coffee and double check everything.
Solutions

You should now know what device/firewall is causing the issue and I have outlined a solution below. If I find anymore causes I will also add them here.

  • These solutions are for PCs that are present on the LAN where RoutedVPN clients are connecting in.
  • To allow LAN clients to connect to RoutedVPN clients just change the IP ranges in the solutions below from (10.200.1.1 - 10.200.1.254) to (10.0.0.1 - 10.0.0.254)

Hardware checksum offload

  • networking - OpenVPN on pfSense, can ping but nothing else - Server Fault
    • Xenserver has a problem with TX checksum offloading and it is not handled properly. You will have to disable it in the pfSense interface and for all the xenserver interface that pfsense use.
    • System --> Advanced --> Networking -->Disable hardware checksum offload = unchecked,  then you'll have to reboot pfSense manually, even if not prompted.

Solution: Window 10 PC running Windows Defender Firewall with Comodo Firewall installed and enabled will not Ping

This assumes your pfSense is correctly configured and you have run through the steps above to confirm this.

These rules will fix the ping issue, but will also fix File and Printer sharing.

  1. Turn off Comodo Firewall and then turn off the Windows Defender Firewall
    • This should only be temporary measure.
  2. Leave both the Comodo and Windows Defender Firewalls on and:
    • Comodo Firewall Settings:
      • Add 'Allow' rules for (ICMP Ping/File Sharing/Print Sharing)
      • See below for options
    • Windows Defender Firewall Settings:
      • Remove the Local subnet restrictions for (ICMP Ping/File Sharing/Print Sharing)
      • See below for options
  3. (Comodo Recommended) Disable the Windows Defender Firewall and keep the Comodo Firewall on and:
    • Comodo Firewall Settings:
      • Add 'Allow' rules for (ICMP Ping/File Sharing/Print Sharing)
      • See below for options

Remove Windows Defender Local Subnet restrictions

Option 1

Add a single rule to allow an IP range through your firewall

Follow one of the tutorials below:

Option 2

Manually remove the restrictions on each of the rules. This is a lot of mess and is not recommended.

  • Remove the restriction for ICMP Ping to 'Local subnet' only
    • Control Panel --> All Control Panel Items --> Windows Defender Firewall --> Advanced Settings --> Inbound Rules: File and Printer Sharing (Echo Request - ICMPv4-In) (Public|Private) --> Scope --> Remote IP address: Any IP Address
  • Remove the restriction for File Sharing to the 'Local subnet' only
    • Control Panel --> All Control Panel Items --> Windows Defender Firewall --> Advanced Settings --> Inbound Rules: File and Printer Sharing (Echo Request - ICMPv4-In) (Public|Private) --> Scope --> Remote IP address: Any IP Address
  • Remove the restriction for Print Sharing to the 'Local subnet' only
    • There are a lot more serices. I did not try this but they are in the same place.
  • You might only need to do the 'Public' profiles

Comodo Firewall - Allow Local services from the RoutedVPN Subnet/Network

In the rules below you can see I have used a Zone called 'pfSense Network' and this is just a zone created in (Settings --> Firewall --> Network Zones) with the IP range 10.200.1.1 - 10.200.1.254 added. This Zone can be used like an alias so I could even have a couple of network ranges in the same zone or specify the subnet with a different type. I would also point out that the network zone that you are on now with your pfSense box is probably the best Zone to add the subnet's IP range into, this is because it is this zone that you will be on when connections come in from the 10.200.1.x subnet. You could just set the IP range in the firewall rule instead of using the alias.

Add an 'Allow RoutedVPN Network' rule to allow connections from the network 10.200.1.x

  • Settings --> Firewall --> Global Rules --> Add:
    • Action: Allow
    • Log as firewall event if this rule is fired: unticked
    • Protocol: IP
    • Direction: In or Out
    • Description: Allow RoutedVPN Network
    • Source Address:
      • Type: Network Zone
      • Zone: pfSense Network
    • Destination:
      • Type: Network Zone
        Zone: pfSense Network
    • IP Details
      • IP Protocol: Any
  • Click 'OK'
  • Click 'OK' again

Add an 'Allow Ping' rule to allow pings from the network 10.200.1.x (This is not needed if you have used the rule above)

  • Settings --> Firewall --> Global Rules --> Add:
    • Action: Allow
    • Log as firewall event if this rule is fired: unticked
    • Protocol: ICMP
    • Direction: In or Out
    • Description: Allow ICMP Ping
    • Source Address:
      • Type: Network Zone
      • Zone: pfSense Network
    • Destination:
      • Type: Network Zone
        Zone: pfSense Network
    • ICMP Details
      • Type: ICMPv4
      • Message: ICMP Echo Request
  • Click 'OK'
  • Click 'OK' again
Notes

This is not just related to OpenVPN but the worked example is.

  • pfSense
    • Can route traffic between subnets connected on local interfaces by default
    • The hostname of the connected OpenVPN client is that of it's certificate, not the device name as defined in Windows. (not sure if this can be changed)
    • Firewall rules are required to allow traffic as there is a 'Deny All' default rule in place.
  • pfSense and OpenVPN
  • Network General
    • Broadcast packets are not passed between subnets
    • Bridging 2 interfaces allows broadcast traffic to be passed.
    • A static route is only needed when routing to another network which is behind a gateway and this tells pfSense the route to the target network.
    • Ubuntu/Linux/IoT Devices do not generally have a subnet restriction for traffic.
  • Windows PCs
    • Have PING disabled by default
    • Will only accept traffic from the same subnet by default
    • Firewall rules have 3 different 'Profiles'
      • Domain - for when on an Active Directory
      • Public - On a Public network
      • Private - On a private network such as Home or Work
  • Windows Firewall
  • Comodo Firewall (CIS)
    • General
    • The Windows Defender Firewall is still running after the Comodo Firewall (or CIS) is installed and this is normal. Official explanations and how you should setup.
      • Win10 firewall might be still on after Comodo Firewall install - Firewall Help - CIS
        • NEVER disable the Windows Firewall Service . . . . . it is required for several essential Windows functions
        • Turn off the Windows Firewall on your private network(s)
        • Above you said to turn it off on private networks (highlighting mine).
        • When I go to Control Panel --> Windows Defender Firewall --> Turn Windows Defender Firewall on or off,
          there are 2 options: one to turn it on/off for private network and a separate section for public network.
        • So to clarify, are you saying I should ONLY turn it off for private network but leave it enabled for public network?
        • You should turn it off for each network type that way only CFW handles connection filtering regardless on which type of network you are connected to.
      • Windows Firewall not disabled on W10 1511 [M1790] - CIS
        • From Comodo: CIS disabled Window FW only on Win 7 and and lower. According to requirement from Microsoft, CIS does not has the right to turn off their components beginning with Windows 8.
        • So users would need to manually turn off the windows firewall if they should choose to. Note this applies to the Windows Firewall control panel applet, do not disable the windows firewall service. Moving to resolved.
        • This is not a bug, but by design, a design dictated by Microsoft.
      • Should I disable Windows Firewall with CIS [Merged] - Firewall Help - CIS
        • Right click the Start button, select Control Panel. Select Windows Firewall, and when the firewall window opens, on the left side, select Turn off Windows Firewall.
        • Yes you should disable the windows firewall if you have comodo firewall installed and you shouldn't get warnings from windows if you disable the windows firewall as long as comodo's firewall is on.
        • Don't disable the service, but do turn off the firewall. This will prevent any unexpected conflicts to arise, giving unexpected results.
        • It is possible to set rules in one just to discover they don't work due to the other firewall letting whatever bypass the former's rules.
          Disabling one will help ensure you don't experience issues with either.
      • Should we turn off Windows Firewall after installing Comodo IS? - Firewall Help - CIS
    • When you disable the Comodo Firewall you still cannot ping.
      • When you disable the Comodo Firewall the Microsoft Defender Firewall enables itself or is still running.
      • Fix: disable Comodo Firewall and then disable the Defender Firewall.
    • Firewall Log
      • Diagnostics
        • Settings --> Firewall --> Global Rules --> your-rule --> edit: 'Log as firewall event if this rule is fired'
        • On any rule you create you can enable logging and then see if it is getting triggered by the firewall.
      • In firewall logs I see that there is an application call 'Windows Operating System'
        • Windows Operating System Blocked - Firewall Help - CIS
          • You are specifically telling the firewall to log blocked incoming connection attempts, which in turn will show 'Windows Operating System'(WOS) as no application is listening on the given port that was blocked.
        • Request for enhancement - CIS to block when the application is not running - CIS
          • Not true, in fact when you see a block event with Windows Operating System listed under application column it means the firewall blocked it from being sent to the OS. If the packet had made it the OS and there was no application listening on that given port, then Windows would send either a TCP packet with the RST (Reset) flag set to the sender or an ICMP Destination Unreachable - Port Unreachable (Type 3, Code 3) error message if the connection attempt was for a UDP port. This is all because the firewall filter driver resides in the kernel at the same layer where the OS kernel resides and hence filter packets at the kernel aka ring0.
    • Network Zones
      • Comodo Internet Security Network Zones, Firewall Protection | Internet Security
      • What Network Zone am I on?
        • Tasks Widget --> Firewall Tasks --> Manage Networks
      • Disable
        • Settings --> Firewall --> Network Zones --> Enable automatic detection of private networks: unticked
      • Any: Defaults to an IP range of 0.0.0.0- 255.255.255.255 to block connection from all IP addresses
      • Once created, a network zone can be: Quickly called as 'Zone' when creating or modifying a Firewall Ruleset (alias)
    • Stealth Ports

Windows

General

Windows PC Cannot Ping Local Devices by their Hostnames

This wil be a worked example of how I fixed my issue so might not exactly be the same as your issue if your are reading this. The issue for me was caused by OpenWRT routers that I had configured as AP points. These routers had their DHCP disable, but this only stops DHCP for IPv4, IPv6 has its own version of DHCP which I had not disabled.

This is an overview of the issue:

  • Ping is allowed by my firewall / Firewall is disabled (see above to troubledshoot pinging)
  • Can ping external domains
  • Can ping domains on my local hosted webserver (Port Forward / NAT Reflection)
  • Can ping local devices by their IP (10.0.0.x / 192.168.1.x)
  • Cannot ping local devices by their hostnames (mycomputer.lan / mycomputer.lan / mycomputer)
  • nslookup
    Microsoft Windows [Version 10.0.19044.1586]
(c) Microsoft Corporation. All rights reserved.

C:\Users\quantumwarp>nslookup
Default Server:  UnKnown
Address:  ffff:bb14:329f::1

> bbc.co.uk
Server:  UnKnown
Address:  ffff:bb14:329f::1

Non-authoritative answer:
Name:    bbc.co.uk
Addresses:  2a04:4e42::81
          2a04:4e42:200::81
          2a04:4e42:400::81
          2a04:4e42:600::81
          151.101.0.81
          151.101.64.81
          151.101.128.81
          151.101.192.81

> mytv.mydomain.com
Server:  UnKnown
Address:  ffff:bb14:329f::1

*** No internal type for both IPv4 and IPv6 Addresses (A+AAAA) records available for mytv.mydomain.com
>
    • Defaults to an IPv6 server
    • Resolves external domain names
    • Resolves domains on my locally hosted webserver
    • Does not resolve local hostnames
Diagnostics
  • ifconfig /all
  • Disable Firewalls
    • I disabled my Comodo CIS Firewall and the Windows Defender Firewall and this made no difference.
    • After confirming this, I turned the Firewall back on immediately.
  • Reset Windows Network
    • Settings --> Network & Internet --> Network reset
  • Disable 'Microsoft Wi-Fi Direct Virtual Adapter' adapters
  • Re-Install VirtualBox (because I use it)
    • Multiple 'VirtualBox Host-Only Network' adapters turned back up, so this is a setting in VirtualBox
      • File --> Host Network Manager
      • Make sure you only need one and delete the others
  • Disabled IPv6 in Network Adapter(s)
    • Control Panel --> Network and Internet --> Network Connections --> Ethernet/WiFi Adapter --> Properties --> Internet Protocol 6 (TCP/IPv6): unticked
    • I can now ping by hostnames
    • nslookup by default uses an IPv4 address (i.e. 10.0.0.1)
    • re-enabled IPv6 for futher testing
  • Ping the IPv6 DNS server
    • Do you get a response?
    • Leave a continuous ping going and uplug kit from the network
      ping -6 ffff:bb14:329f::1 -t
  • Open the IPv6 DNS server in a Web Browser
  • Log the DNS queries
    • See if this gives you any more infromation

At this point with IPv6 still enables, none of what we have done has fixed the issue.

Packet Capture all 'Port 53' traffic on the LAN interface

I have DNS Hijacking enabled on my pfSense box so how are DNS lookups getting made when IPv6 has also been turned off (mostly) on my router.

  • I followed the steps below:
  • Keep all traffic to a minimum
    • Close all browsers
    • Close all apps
  • Open nslookup by just typing nslookup which defaults to using the IPv6 server which is what we want.
  • Type in a domain name bbc.co.uk but do not press enter
  • Opened (pfSense --> Diagnostics --> Packet Capture) with the following settings:
    • Interface: LAN
    • Port: 53
    • leave the rest the same
  • Scroll to the bottom and click 'Start'
  • Press enter in nslookup
  • Wait a moment and then click 'Stop' on the Packet Capture page.
  • In the 'Packet Capture' field at the bottom you should see something like this:

    • What this shows is that the only address that did a DNS lookup as far as pfSense is concerned is 10.0.0.5
    • Where is my DNS lookup because my IP address is 10.0.0.183
    • Your capture log might not be as clean as the one above.
  • Further analysis is required with Wireshark
    • Download the capture by click 'Download Capture'
    • Open the file with Wireshark. Install the software if you dont have it.
    • You should now see something like this
    • We can see clearly that the address 10.0.0.5 did a DNS lookup for bbc.co.uk which must of been initiated by me or by chance the device with the address 10.0.0.5 browsed the BBC website at the exact same time, so lets assume it was me.
    • So how did my DNS request get to 10.0.0.5?
Cause

I had an OpenWRT router configured as an AP (Access Point) using the IP 10.0.0.5 which I had disabled DHCP on them as you would expect.

So to explain the packet capture above:

  • The OpenWRT AP was still running RA and SLAAC services (IPv6 equivalent of DHCP)
    • My Windows PC was being assigned IPv6 Temporary Addresses and IPv6 DNS servers because of this.
  • My Windows PC DNS request was:
    • Sent over IPv6 to the OpenWRT AP
      • because there was an IPv6 server configured on the Windows PC, and by default in Windows IPv6 is the prefered network protocol, so the IPv6 Server was used.
    • The OpenWRT device used its own internal DNS Daemon to forward the request to its upstream DNS resolver over IPv4 which was my pfSense router (10.0.0.1)
      • You cannot see it, but the pfSense router then forwarded it upstream to it's resolver, and processed the returned results.
    • pfSense sent the results to the OpenWRT AP over IPv4
    • The OpenWRT AP then returned the results to the Windows PC over IPv6

So the IPv6 traffic skipped the pfSense router and as far as it was concerned the OpenWRt AP was making the DNS request.

This OpenWRT AP had the IPv4 10.0.0.5 and the IPv6 address ffff:bb14:329f::1 that was assigned to my Windows PC as a DNS server.

Solutions

You might find a small speed improvement when either of these options are applied because now your DNS lookups are going direct to the pfSense router rather than goping through the OpenWRT AP.

Option 1 - On the OpenWRT AP - Disable IPv4 and IPv6 DHCP, DNS and related services on the LAN interface only

This assumes you are using the default configuration where LAN is your primary network and it is this that is connected to the rest of the network via ethernet.

  • OpenWRT --> Network --> Interfaces --> LAN --> Edit --> DHCP Server
    • General Setup
      • Ignore interface: ticked
    • IPv6 Settings
      • RA-Service: disabled
        • This includes SLAAC
      • DHCPv6-Service: disabled
        • This includes Local IPv6 DNS server
      • All the other settings will dissapear.
  • Click 'Save'
  • Click 'Save & Apply'
  • Reboot OpenWRT
    • OpenWRT -->System --> Perform Reboot
  • Reboot the Windows PC
    • You might be able to get away with stopping and starting your Network adapter.

Option 2 - On the OpenWRT AP - Disable all IPv4 and IPv6 DHCP, DNS and related services

  • OpenWRT --> System --> Startup --> Stop and Disable
    • dnsmasq
    • odhcpd
  • This configuration will not survive a system upgrade.
  • This is Ok for quick testing and a temporary fix.
  • It will disable these services across all interfaces.
Workarounds

These are not ideal and I have just put them here for reference or for when you cannot access the relvant nework kit to re-configure them as required.

  • Prefer IPv4 over IPv6 (on Windows PC)
    • This will force Windows ot use IPv4 first thus skipping the IPv6 issue.
    • Prefer IPv4 over IPv6 in Windows Networks - .matrixpost.net - By default Windows will prefer IPv6 over IPv4. Below you will see how you can change this behavior.
    • Prioritising IPv4 over IPv6 on Windows 10 and 11 - IPv6 is an integral part of Windows. It's tempting to disable IPv6 on Windows especially on home networks. This can have a variety of side effects that might cause applications and processes to misbehave. Microsoft explicitly does not recommend you disable IPv6. The best way to ensure your Windows computer uses IPv4 is to simply change the IPv6 prefix policy precedence.
    • networking - IPv4 vs IPv6 priority in Windows 7 - Super User - I have IPv6 connectivity through Hurricane Electric tunnel. Since IPv6 day this year, many services (google.com, facebook.com, etc.) enabled IPv6 on their main domains. On my Windows machine, IPv6 is preferred over IPv4. This means that whenever I visit Google, all traffic goes through my tunnel to Hurricane Electric, which raises the latency by more than 100%:
    • Prefer IPv4 over IPv6 on Windows – Dimitri's Wanderings - Prefer IPv4 over IPv6 on Windows using KB 929852 registry setting, GPO or alternatively netsh prefix policy without reboot needed.
    • Why does Windows 10 prefer IPv4 over IPv6? - Super User - I'm running the latest version of Windows 10 as of this writing (1903) and I have a working dual-stack connection. If I ping the hostname of a dual-stack device from the command line, Windows prefers the IPv4 address. If I turn off IPv4, it will use the IPv6 address. According to numerous sources (including this answer), Windows should prefer IPv6.
    • Prioritizing IPv4 Connections | SocketTools10
      • Windows will normally prefer to use IPv6 instead of IPv4 if the local system has been configured with IPv6 enabled and the system you want to connect with has an IPv6 address assigned in its DNS record. However, there may be situations where you want the system to prioritize IPv4 instead.
      • A good article and clearly explained.
    • Use IPv4 Instead of IPv6 | UNBLOG - How to use IPv4 prefers instead of IPv6 using Windows. The affine coexistence of IPv6 and IPv4 for resources in the transition period is not necessarily
  • Disable IPv6 (on Windows PC)
    • This is self explanatory
    • Control Panel --> Network and Internet --> Network Connections --> Ethernet/WiFi Adapter --> Properties --> Internet Protocol 6 (TCP/IPv6): unticked
  • Remove the OpenWRT AP device from the network
    • This is self explanatory
Conclusion
  • All the Temporary IPv6 addresses have now dissapeared.
  • It was the OpenWRT AP causing the problem by leaving the IPv6 DHCP Equivalent services running.
  • My traffic was getting split, IPv4 to pfSense and IPv6 traffic to OpenWRT AP.
  • This mis-configuration might of been causing issues with network connectivity beyound local hostname lookups.
  • nslookup now defaults to 10.0.0.1 for its DNS server which means there is no IPv6 DNS server on the network.

Notes

Networking

I will add non pfsense specific networking links and information.

Alternative Firewalls

The following is a list of pfSense alternatives just incase you wanbt to have a look.

  • The Firewalls
    • OPNsense® a true open source security platform and more - From Virtual Private Networking to Intrusion Detection, Best in class, FREE Open Source Project.
    • Sophos Home Edition Firewall - Our Free Home Use Firewall is a fully equipped software version of the Sophos Firewall, available at no cost for home users – no strings attached. Features full protection for your home network, including anti-malware, web security and URL filtering, application control, IPS, traffic shaping, VPN, reporting and monitoring, and much more.
    • IPFire - The Open Source Linux-based Firewall Operating System with a Comprehensive Feature Set.
  • Vs

 

 

 

 

Published in Networks
Read more...
Thursday, 23 December 2021 08:47

My InnoDB Notes

InnoDB is the better engine and is the one I will be using in all of my projects. InnoDB is faster and more resilient to errors.

These article brings together all of my InnoDB notes which were built while trying to get my head around the different SQL engines (InnoDB vs MyISAM) and which one to use.

my.cnf / my.ini Example with Annotations

This is an example my.cnf/my.ini (my-innodb-heavy-4G.ini) taken from an old version of Xampp running on Windows. This is useful because it has annotations against a lot of the settings.

#BEGIN CONFIG INFO
#DESCR: 4GB RAM, InnoDB only, ACID, few connections, heavy queries
#TYPE: SYSTEM
#END CONFIG INFO

#
# This is a MySQL example config file for systems with 4GB of memory
# running mostly MySQL using InnoDB only tables and performing complex
# queries with few connections.
# 
# MySQL programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, see:
# http://dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#

#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]
#password	= [your_password]
port		= 3306
socket		= /tmp/mysql.sock

# *** Application-specific options follow here ***

#
# The MySQL server
#
[mysqld]

# generic configuration options
port		= 3306
socket		= /tmp/mysql.sock

# back_log is the number of connections the operating system can keep in
# the listen queue, before the MySQL connection manager thread has
# processed them. If you have a very high connection rate and experience
# "connection refused" errors, you might need to increase this value.
# Check your OS documentation for the maximum value of this parameter.
# Attempting to set back_log higher than your operating system limit
# will have no effect.
back_log = 50

# Don't listen on a TCP/IP port at all. This can be a security
# enhancement, if all processes that need to connect to mysqld run
# on the same host.  All interaction with mysqld must be made via Unix
# sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#skip-networking

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections = 100

# Maximum amount of errors allowed per host. If this limit is reached,
# the host will be blocked from connecting to the MySQL server until
# "FLUSH HOSTS" has been run or the server was restarted. Invalid
# passwords and other errors during the connect phase result in
# increasing this value. See the "Aborted_connects" status variable for
# global counter.
max_connect_errors = 10

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_open_cache = 2048

# Enable external file level locking. Enabled file locking will have a
# negative impact on performance, so only use it in case you have
# multiple database instances running on the same files (note some
# restrictions still apply!) or if you use other software relying on
# locking MyISAM tables on file level.
#external-locking

# The maximum size of a query packet the server can handle as well as
# maximum query size server can process (Important when working with
# large BLOBs).  enlarged dynamically, for each connection.
max_allowed_packet = 16M

# The size of the cache to hold the SQL statements for the binary log
# during a transaction. If you often use big, multi-statement
# transactions you can increase this value to get more performance. All
# statements from transactions are buffered in the binary log cache and
# are being written to the binary log at once after the COMMIT.  If the
# transaction is larger than this value, temporary file on disk is used
# instead.  This buffer is allocated per connection on first update
# statement in transaction
binlog_cache_size = 1M

# Maximum allowed size for a single HEAP (in memory) table. This option
# is a protection against the accidential creation of a very large HEAP
# table which could otherwise use up all memory resources.
max_heap_table_size = 64M

# Size of the buffer used for doing full table scans.
# Allocated per thread, if a full scan is needed.
read_buffer_size = 2M

# When reading rows in sorted order after a sort, the rows are read
# through this buffer to avoid disk seeks. You can improve ORDER BY
# performance a lot, if set this to a high value.
# Allocated per thread, when needed.
read_rnd_buffer_size = 16M

# Sort buffer is used to perform sorts for some ORDER BY and GROUP BY
# queries. If sorted data does not fit into the sort buffer, a disk
# based merge sort is used instead - See the "Sort_merge_passes"
# status variable. Allocated per thread if sort is needed.
sort_buffer_size = 8M

# This buffer is used for the optimization of full JOINs (JOINs without
# indexes). Such JOINs are very bad for performance in most cases
# anyway, but setting this variable to a large value reduces the
# performance impact. See the "Select_full_join" status variable for a
# count of full JOINs. Allocated per thread if full join is found
join_buffer_size = 8M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size = 8

# This permits the application to give the threads system a hint for the
# desired number of threads that should be run at the same time.  This
# value only makes sense on systems that support the thread_concurrency()
# function call (Sun Solaris, for example).
# You should try [number of CPUs]*(2..4) for thread_concurrency
thread_concurrency = 8

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size = 64M

# Only cache result sets that are smaller than this limit. This is to
# protect the query cache of a very large result set overwriting all
# other query results.
query_cache_limit = 2M

# Minimum word length to be indexed by the full text search index.
# You might wish to decrease it if you need to search for shorter words.
# Note that you need to rebuild your FULLTEXT index, after you have
# modified this value.
ft_min_word_len = 4

# If your system supports the memlock() function call, you might want to
# enable this option while running MySQL to keep it locked in memory and
# to avoid potential swapping out in case of high memory pressure. Good
# for performance.
#memlock

# Table type which is used by default when creating new tables, if not
# specified differently during the CREATE TABLE statement.
default-storage-engine = MYISAM

# Thread stack size to use. This amount of memory is always reserved at
# connection time. MySQL itself usually needs no more than 64K of
# memory, while if you use your own stack hungry UDF functions or your
# OS requires more stack for some operations, you might need to set this
# to a higher value.
thread_stack = 192K

# Set the default transaction isolation level. Levels available are:
# READ-UNCOMMITTED, READ-COMMITTED, REPEATABLE-READ, SERIALIZABLE
transaction_isolation = REPEATABLE-READ

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size = 64M

# Enable binary logging. This is required for acting as a MASTER in a
# replication configuration. You also need the binary log if you need
# the ability to do point in time recovery from your latest backup.
log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

# If you're using replication with chained slaves (A->B->C), you need to
# enable this option on server B. It enables logging of updates done by
# the slave thread into the slave's binary log.
#log_slave_updates

# Enable the full query log. Every query (even ones with incorrect
# syntax) that the server receives will be logged. This is useful for
# debugging, it is usually disabled in production use.
#log

# Print warnings to the error log file.  If you have any problem with
# MySQL you should enable logging of warnings and examine the error log
# for possible explanations. 
#log_warnings

# Log slow queries. Slow queries are queries which take more than the
# amount of time defined in "long_query_time" or which do not use
# indexes well, if log_short_format is not enabled. It is normally good idea
# to have this turned on if you frequently add new queries to the
# system.
slow_query_log

# All queries taking more than this amount of time (in seconds) will be
# trated as slow. Do not use "1" as a value here, as this will result in
# even very fast queries being logged from time to time (as MySQL
# currently measures time with second accuracy only).
long_query_time = 2


# ***  Replication related settings 


# Unique server identification number between 1 and 2^32-1. This value
# is required for both master and slave hosts. It defaults to 1 if
# "master-host" is not set, but will MySQL will not function as a master
# if it is omitted.
server-id = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    changes in this file to the variable values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>

# Make the slave read-only. Only users with the SUPER privilege and the
# replication slave thread will be able to modify data on it. You can
# use this to ensure that no applications will accidently modify data on
# the slave instead of the master
#read_only


#*** MyISAM Specific options


# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size = 32M

# MyISAM uses special tree-like cache to make bulk inserts (that is,
# INSERT ... SELECT, INSERT ... VALUES (...), (...), ..., and LOAD DATA
# INFILE) faster. This variable limits the size of the cache tree in
# bytes per thread. Setting it to 0 will disable this optimisation.  Do
# not set it larger than "key_buffer_size" for optimal performance.
# This buffer is allocated when a bulk insert is detected.
bulk_insert_buffer_size = 64M

# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMIZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
myisam_sort_buffer_size = 128M

# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size = 10G

# If a table has more than one index, MyISAM can use more than one
# thread to repair them by sorting in parallel. This makes sense if you
# have multiple CPUs and plenty of memory.
myisam_repair_threads = 1

# Automatically check and repair not properly closed MyISAM tables.
myisam_recover

# *** INNODB Specific options ***

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size = 16M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size = 2G

# InnoDB stores data in one or more data files forming the tablespace.
# If you have a single logical drive for your data, a single
# autoextending file would be good enough. In other cases, a single file
# per device is often a good choice. You can configure InnoDB to use raw
# disk partitions as well - please refer to the manual for more info
# about this.
innodb_data_file_path = ibdata1:10M:autoextend

# Set this option if you would like the InnoDB tablespace files to be
# stored in another location. By default this is the MySQL datadir.
#innodb_data_home_dir = <directory>

# Number of IO threads to use for async IO operations. This value is
# hardcoded to 8 on Unix, but on Windows disk I/O may benefit from a
# larger number.
innodb_write_io_threads = 8
innodb_read_io_threads = 8

# If you run into InnoDB tablespace corruption, setting this to a nonzero
# value will likely help you to dump your tables. Start from value 1 and
# increase it until you're able to dump the table successfully.
#innodb_force_recovery=1

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency = 16

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit = 1

# Speed up InnoDB shutdown. This will disable InnoDB to do a full purge
# and insert buffer merge on shutdown. It may increase shutdown time a
# lot, but InnoDB will have to do it on the next startup instead.
#innodb_fast_shutdown

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions). 
innodb_log_buffer_size = 8M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size = 256M

# Total number of files in the log group. A value of 2-3 is usually good
# enough.
innodb_log_files_in_group = 3

# Location of the InnoDB log files. Default is the MySQL datadir. You
# may wish to point it to a dedicated hard drive or a RAID1 volume for
# improved performance
#innodb_log_group_home_dir

# Maximum allowed percentage of dirty pages in the InnoDB buffer pool.
# If it is reached, InnoDB will start flushing them out agressively to
# not run out of clean pages at all. This is a soft limit, not
# guaranteed to be held.
innodb_max_dirty_pages_pct = 90

# The flush method InnoDB will use for Log. The tablespace always uses
# doublewrite flush logic. The default value is "fdatasync", another
# option is "O_DSYNC".
#innodb_flush_method=O_DSYNC

# How long an InnoDB transaction should wait for a lock to be granted
# before being rolled back. InnoDB automatically detects transaction
# deadlocks in its own lock table and rolls back the transaction. If you
# use the LOCK TABLES command, or other transaction-safe storage engines
# than InnoDB in the same transaction, then a deadlock may arise which
# InnoDB cannot notice. In cases like this the timeout is useful to
# resolve the situation.
innodb_lock_wait_timeout = 120


[mysqldump]
# Do not buffer the whole result set in memory before writing it to
# file. Required for dumping very large tables
quick

max_allowed_packet = 16M

[mysql]
no-auto-rehash

# Only allow UPDATEs and DELETEs that use keys.
#safe-updates

[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
# Increase the amount of open files allowed per process. Warning: Make
# sure you have set the global system limit high enough! The high value
# is required for a large number of opened tables
open-files-limit = 8192

 

Published in MySQL
Read more...
Thursday, 16 December 2021 11:11

Create a Divi Theme Demo Site

There are a couple of ways that I know of to create a theme demo site and I will outline them below.

  • Single WordPress Website (Multiple Themes)
    • Pro
      • Probably only works for themes with builders
      • less files and WordPress installs
      • all theme templates are available in 1 dashboard
      • only 1 set of plugin to configure
      • don't have the hassle of setting up multisite
      • can add iframe-x header 1 and configure as needed
      • less server overheads
      • easy to manage
      • 1 set of credentials to log in with
      • uploaded files are easier to manage
    • Con
      • cant use divi global headers for each theme
      • cant use the inbuilt divi theme menu in div customizer
  • Multiple WordPress Websites (1 Theme per Website)
    • Pro
      • Each theme can be fully controlled
    • Con
      • A lot more inodes and server space is required
      • more of a security risk because of the greater attack surface
      • harder to manage with all of the updates needed to be applied to every site
      • More CPU power needed
      • Takes much longer to setup a theme
      • many different credentials and databases to manage
  • WordPress Multisite (same as Multiple Wordpress Sites, but one WordPress instance)
    • Pro
      • Should work for all WordPress themes
      • can use divi global headers for each theme
      • only the pages for the defined theme will be in the database
      • can configure plugins on a per-theme basis (i think)
      • can use the inbuilt Divi theme menu in the Divi Theme customizer
    • Con
      • extra configuration required to enable multi-site
      • all themes are separate and makes them harder to manage
      • loads of credentials
      • many theme will start to be cumbersome to manage


So after outlining the options above I figured out the best way of setting out my themes was to use the Single WordPress Website option. This allows for ease of management and a great base to keep your templates for export when you need them to build client sites.

I will not only use this as my demo site, but i will use this to store my templates which i directly export and use to build my client sites reducing the time it takes to make a site.

You can also add and make modifications to these pages as you go to fix errors, add extra features and general improvements for future project. I find when i use a template for a client, I sometimes design or use a layout I really like so i can then add this back into my theme for future use.

Build the Demo Site (Single WordPress Website)

If you follow these easy instructions below you can quickly build your demo site which allows for easy expansion and management.

Install WordPress

This is straight forward and does not need notes except

  • When creating the Database use:
    • Use utf8mb4_unicode_ci for database collation
    • InnoDB for DB engine
  • Extract WordPress into the public_html
  • Run the WordPress Wizard
  • Setup WordPress using the following details:
    • Site Title: WordPress Themes

Configure WordPress and Extensions

These should be straight forward. I am using Divi as my Theme engine, but if you use another it should be fine.

  • Themes (Install and Configure)
    • Divi
      • Manually add API key: xxxxxxxxxxxxxx
      • Import Theme options
      • Import 'Theme Customizer'
      • In options, enable the Divi Gallery (should be done as part of the options import)
      • Set Divi to auto update
    • Delete widgets
    • Delete unwanted themes (i.e. Theme 21)
  • Plugins (Install and Configure)
    • Delete unwanted plugins (i.e. Hello Dolly)
    • Wordfence
      • (no-reply@qwdemos.com) or what email address you use
      • configure/ import settings (via long hash)
      • Wordfence --> Login Security --> Settings --> Disable XML-RPC authentication: ticked
      • Change file hash scanning because of high I/O (change 'Basic Scan Type Options' to 'Limited Scan' , 'Standard Scan' is normal
      • Enable Automatic updates
    • Easy WP SMTP (Only needed if no mail() function)
      • configure/ import settings
    • Manage Notification E-mails
      • import settings or configure as:
        • Turn off - Automatic WordPress core update e-mail
        • Turn off - Automatic WordPress plugin update e-mail
        • Turn off - Automatic WordPress theme update e-mail
        • Leave the rest same
    • W3 Total Cache
      • Either the PHP configuration, web server configuration or a script in the WordPress installation has zlib.output_compression enabled.
      • configure/Import settings
        • Load options (General Settings --> Import / Export Settings)
      • Enable Page Cache (contact forms might fail to work but are not needed on the Demo Sites)
      • Save settings and purge
      • Now deactivate until you have finished building
    • Velvet Blues Update URLs
      • This is optional but if you are importing layouts from different URLs you might want this installed so you can change the links.
      • You can leave this deactivated when not in use.
    • Disable all auto updates
    • Set all plugins to auto update
  • WordPress Settings
    • WordPress --> Settings
      • General:
        • remove tagline
      • Reading:
        • Set home page
      • Discussion:
        • 'Attempt to notify any blogs linked to from the post' = off
        • 'Allow link notifications from other blogs (pingbacks and trackbacks) on new posts' = off
        • 'Allow people to submit comments on new posts' = off
        • 'Users must be registered and logged in to comment' = on
        • 'Comment must be manually approved' = on
    • delete test comments and posts (might not do this) ??
    • WordPress --> Updates
      • Updates:
        • 'Switch to automatic updates for maintenance and security releases only.'
      • Updates all Plugins and Themes
    • Delete all Comments, Posts and Pages. Make sure you empty the trash afterwards.
  • Set Favicon

Create a Primary Menu and Homepage

We need to create a default menu and homepage for WordPress for all requests that do not have a proper page then at least we have a proper page that is displayed.

  • Create a new menu call 'Primary'
    • This should have at least one 'Custom Link' pointing to the 'homepage' of the site https://themes.mydomain.com/ called 'Themes Home'
    • Display Location = Primary Menu
    • This is so that all default Divi pages have a menu that does not have every page listed.
  • Create a homepage
    • Goto (Pages --> All Pages)
    • Add 'New Page' called 'QWThemes - Home'
  • Set the static homepage
    • (Settings --> Reading --> Your homepage displays -> Homepage) = 'Themes - Home'
  • Fill in content as require. You can leave this blank or add in some company information for example.

Notes

  • You could create a menu structure pointing to the
    1. homepage of each theme (if you don't have a theme toolbar this could be a good option)
    2. to every themes page (not recommended)

Adding Themes (Pages)

When I refer to a Theme, I actual mean a group of pages that have the same styling such as found on Divi Layouts by Elegant Themes

Each theme will require you to do the following (I will use example of Theme1). You can expand this to have more pages if you require.

  • Create Theme Homepage
    • Title: Theme1 - Home
    • Slug: theme1
    • Parent: Main Page (no parent)
    • Template = Blank Template
    • Use Divi
  • Create Theme Content pages
    • Services
      • Title: Theme1 - Services
      • Slug: services
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • Gallery
      • Title: Theme1 - Gallery
      • Slug: gallery
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • About
      • Title: Theme1 - About
      • Slug: about
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • Contact
      • Title: Theme1 - Contact
      • Slug: contact
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • Alt1 (optional)
      • Title: Theme1 - Alt1
      • Slug: alt1
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • Alt2 (optional)
      • Title: Theme1 - Alt2
      • Slug: alt2
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
    • Alt3 (optional)
      • Title: Theme1 - Alt3
      • Slug: alt3
      • Parent: Main Page (Theme1 - Home)
      • Template = Blank Template
      • Use Divi
  • Change the internal links to now work on your website.
    • Run 'Velvet Blues Update URLs' as required.

Adding Menus

Each theme needs its own custom menu and the instructions below show you how this should be done. Again I will use Theme1 as an example.

  • Import my Theme Menu Module Layout into Divi Library
    • this only needs to be done once
    • You might not have a pre-built layout, but it is easy to make one. Just create a layout with a menu module in a section.
    • Import my pre-built Divi Menu module layout into the Divi library and call it Themes Header
    • Edit the layout and add a logo (optional)
    Create Theme's WordPress Menu
    • Create a menu called 'Theme1'
    • Select all of the pages for Theme1 and import into the menu
    • Put them in the correct order with drag and drop (Home, Services, Gallery, About, Contact, Alt1, Alt2, Alt3)
    • Edit menu item's navigation label to read (Home, Services, Gallery, About, Contact, Alt1, Alt2, Alt3)
  • Import the 'Themes Header' into each page of the theme
    • Edit the page
    • Click on the add section button and select Themes Header
    • Edit the 'Menu Module' and select the 'Theme1' menu as the menu source
    • Move the menu section it to the top as this will be needed there as this is the primary menu.
    • Now save the page

Repeat this for each theme you want to add. You can create the layouts on the fly in the demo site but will have to use a plugin (How To Clean Up Your WordPress Media Library | WP Engine / Media Cleaner – Clean & Optimize Space – WordPress plugin | WordPress.org ) to cleanup unwanted images after you finish or you have a separate development site.

Notes

Theme Content

Whether you have built you own layouts or used ones from Elegant themes I recommend reading the following about setting out the actual content.

The blog layouts in the Divi themes can be a good source of a standard clean page.

Use these Contact details

020 7123 456
07747 123456
no-reply@qwdemos.com

QuantumWarp House, London SW1A 1AA

QuantumWarp House
Easy Street
Westminster
London
SW1A 1AA
United Kingdom

Import a Divi Layout

  • Select the theme you are going to use
  • Select a layout for each page of your theme (usually the namesakes are good ones to use, About --> About, Contact --> Contact)
  • Import Layout
    • You can import via the (Load from Library --> Premade Layouts) or a layout file you have locally.
    • Either, 'Do Not Replace Content' as you want to keep the menu in place that you have just added.

Convert a Divi Theme to one you can use for your Clients

Not all clients send enough information and I have found that a lot of Divi Themes look nice but can be a bit impractical to use for my clients so I need to make alterations to them first before I can even use them.

Follow the steps below for each of the Pages/Layouts to make them ready to use for your clients:

  • Google Maps (if present)
    • The Google map is usually just on the contact page.
    • I prefer just to have the map present on the contact page
    • Hide the Google Map module for later use. This is just incase you client has specific needs.
    • Where the 'Google Map Module' add a 'Code Module' as we are going to add an iframe version which does not require a credit card.
    • Add the following Demo iframe code: 
      <iframe 
src="https://www.google.com/maps/embed?pb=!1m18!1m12!1m3!1d2483.64596804941!2d-0.1440786842302416!3d51.501363979634085!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!3m3!1m2!1s0x48760520cd5b5eb5%3A0xa26abf514d902a7!2sBuckingham%20Palace!5e0!3m2!1sen!2suk!4v1639323812962!5m2!1sen!2suk" 
width="100%" 
height="450" 
style="border:0;display:block;" 
allowfullscreen="" 
loading="lazy"
></iframe>
  • 'Call Us' / 'Call (451) 350-3922' Buttons
    • Rename these all to Contact Us and add the link https://themes.qwdemos.com/theme1/contact/#contact-form
    • If you have a button with a phone number on it and the client whats this changing it can take time.
    • Phone numbers should stay in the header and footers', and the contact page only
  • Contact Form(s)
    • Remove all contact forms except the one on the contact page
    • This allows you to change the contact email address if every needed
    • Reduces the attack surface
    • Can apply page caching to more pages.
    • On the Contact Page, on the contact form add the CSS ID contact-form to the contact form row. The row is used to allow it to appear on screen better.
    • Enable Standard Captcha
      • Contact Form --> Advanced --> Custom CSS --> Captcha Field --> margin-left: 5px;
      • Contact Form --> Advanced --> Custom CSS --> Captcha Text --> color: #ffffff
        • Only if styling is required.
        • You don't have to use white
    • Make all 'Contact Us' buttons point to https://themes.qwdemos.com/theme1/contact/#contact-form
    • Add a success message
      • This is the default Divi one: 'Thanks for contacting us'
      • No need to add one unless you want to change the default message.
    • Add the email address no-reply@qwdemos.com otherwise emails will get sent to the default email account and you will get a lot of SPAM
  • Gallery page
    • If there is not already a Gallery page on your chosen theme, create one using the following basic layout:
      • Grid
      • 6-8 images
      • No captions or pagination
  • Upload logo for the menu (if needed)
  • Make sure all of the text is in Lorem Ipsum.
  • Remove all non-free assets
    • If you have imported layouts from client websites it is important to remove all of their images and asets
  • Examine each page and remove/hide sections that cannot be altered for a generic website. i.e.
    • Highly specific graphics
    • Cartoons
    • Replace images where needed with generic ones (not for copyright reasons though)

Control who can Embed your Demo Site using an iframe

Now you have done all of this hard work you don't want other people stealing your bandwidth so you need to control who can put your site in an iframe and this can be done by several options, some old, some new, but extensively the hosting server tells the client if it can use the website in an iframe. This behaviour all works on the browser or software at the other end respecting these directives.

The options I outline below can usually be managed by decent software such as W3C Total Cache, but there is no harm in doing it manually.

Content Security Policy (CSP)

CSP allows you to control how the browser behaves with your code remotely, and locally. You can tell the browser not to execute scripts on a page or not to embed the content in and iframe. Not allowing your content to be embedded in an iframe is quite obvious, but why would you want to stop your own scripts from running? This feature prevents scripts that have been maliciously implanted from running by only allowing scripts that you want to be run. The rules can be specified a lot more precisely, but this is just a simple example.

Enable restrictions via .htaccess

These are some examples of the .htaccess rules. You only need to specify one.

<IfModule mod_headers.c>
    ## Allow embedding from sitea.com or siteb.com including sub-domains using any protocol
    Header set Content-Security-Policy "frame-ancestors 'self' *.sitea.com *.siteb.com"
    
    ## Allow embedding from sitea.com or siteb.com including sub-domains using only HTTPS protocol
    Header set Content-Security-Policy "frame-ancestors 'self' https://*.sitea.com https://*.siteb.com"
    
    ## Allow embedding from example.org, example.com, store.example.com using only HTTPS protocol
    Header set Content-Security-Policy "frame-ancestors 'self' https://example.org https://example.com https://store.example.com;"    
</IfModule>
  • You can use wildcards
  • Using frame-ancestors 'self' is similar to using X-Frame-Options: sameorigin
    • If you do not need this option, you can just remove 'self'
  • When you add multiple Policies, they can be on separate lines to make it easier to read.
  • You do not need to add any wrapping conditions like you do when using X-Frame-Options because the conditions are in the statement.

Enable restrictions via W3 Total Cache

  • You can add these in WordPress using the W3C Total Cache plugin:
    • WP Admin --> Performance --> Browser Cache --> Security Headers --> Content Security Policy: ticked
    • WP Admin --> Performance --> Browser Cache --> Security Headers --> frame-ancestors: 'self' *.sitea.com siteb.com;
    • WP Admin --> Performance --> Browser Cache --> Security Headers --> X-Frame-Options: unticked

Notes

X-Frame-Options Header (rules based without using ALLOW-FROM)

You can control the X-Frame-Options headers in either .htaccess or PHP but I will use .htaccess code here because it is easier to implement and is not script dependent.

The Code

The .htaccess code below uses the X-Frame-Options and gives the same effect as using ALLOW-FROM but without using this obsolete command.

# Conditional X-Frame-Options for iframe Embedding Control
<If "%{HTTP_REFERER} == 'https://www.content-site.com/' || %{HTTP_REFERER} == 'https://www.sitea.com/' || %{HTTP_REFERER} == 'https://www.siteb.com/'">
    <IfModule mod_headers.c>
        Header always unset X-Frame-Options
    </IfModule>
</If>
<Else>
    <IfModule mod_headers.c>
        Header always append X-Frame-Options SAMEORIGIN
    </IfModule>
</Else>

Notes

Combined .htaccess Example

No matter if you use a plugin or manually create the rules in your .htaccess they should look something like this:

<IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=31536000"
    Header set X-XSS-Protection "1; mode=block"
    Header set X-Content-Type-Options "nosniff"
    Header set Referrer-Policy "no-referrer-when-downgrade"
    Header set Content-Security-Policy "frame-ancestors 'self' *.sitea.com *.siteb.com"
    #Header set Content-Security-Policy "frame-ancestors 'self' https://*.sitea.com https://*.siteb.com"
    #Header set Content-Security-Policy "frame-ancestors 'self' https://example.org https://example.com https://store.example.com;"
</IfModule>

Notes

  • There are some commented out Content-Security-Policy examples for reference.
  • HSTS is on.
  • The live Content-Security-Policy does not define the protocol of connections allowed. This means that both HTTP and HTTPS are allowed but because HSTS is on, all incoming HTTP connections must be upgraded to HTTPS.

Final Tasks

These are somes task that might need doing after you have built all of your Theme pages.

  • Remove all un-needed images and assets
  • Activate W3C Total Cache
    • It might aswell run quick

Your Theme Demo Site is now finished

You have built all of your Theme Demos, the last thing you have to do is:

  • Update QWDemobar (only if you have this plugin on your main business site)
    • You need to add the new Theme Pages to be able to show you clients

 

Published in Wordpress
Read more...
Friday, 10 December 2021 16:16

My APC SMT1500IC UPS Notes

These are my notes on using and configuring my APC Smart-UPS SMT1500IC 1500VA with SmartConnect.

General Notes

  • Websites
  • Different software versions
    • There are two versions of PowerChute. The Personal Edition and the Business Edition. The personal Edition does not recognize the professional UPS.
  • Maintenance and longer life
  • Minimum Load
    • The minimum load is 10% on my SMT1500IC which translates to 100W, my server is using 70W which is below the minimum load. 100W usage will give me 2h52min runtime at current levels.
      • Rated power in W: 1000 W
      • Rated power in VA: 1500 VA
      • This means I probably went overkill on my purchase, but I was originally running 2 PCs of the UPS until I decided virtualisation was the way to go.
      • I need to test the minimum load theory
    • APC UPS units and minimum load... | Reddit
      • Q:
        • I need a new UPS and for various reasons have settled on the APC SMT*** series. I'm in a 240v country. The SMT750i would fit the bill, my load is very low maybe 100-200w, however I have the chance to get a used SMT3000i for less than the cost of a new SMT750i.
        • However one thing confuses me... on the APC site it indicates for each UPS the minimum load is 10% of the maximum, so in the case of SMT3000i it says 270w. I messaged APC support and they said that was the case. I asked what would happen if you had a load of less than that, or a load of 400w then switched off one device bringing the load to 200w, and they said the UPS would beep and then shut down the output completely.
      • A:
        • This would only make sense to me if they had an “auto-off” feature that waited for load to drop below 10%, then taking this as a sign that you’ve shut everything down, switch off the output to save wear on the batteries. Otherwise theoretically the thing would just keep burning batteries until they ran to zero.
        • 18 months ago I ran a couple of LED christmas tree lights off a pair of SRT3000XLI 3kVA UPSes as no mains power was available - total load of probably 5 watts per device, estimated battery duration in excess of 48 hours. No issues whatsoever - but note this was battery only, no mains power to battery transfers as would happen with a mains outage.
        • Line interactive UPS's don't really care about the load if it's less than maximum. Larger UPS would use more power for housekeeping, that's why they define a minimum load. It's a minimum reasonable load in fact. Below that threshold you'd be in the low efficiency range due to UPS self-consumption.
        • This is exactly what happened to us. We verified that our UPS was working correctly, battery had recently been replaced, 100 % charged ... then we had a power outage and the UPS switched off immediately. APC support just replied "your load was too low, it must be at least 10 % or the UPS will shut down."
  • Buying Guides
    • What are the various generations of Smart-UPS Products? | APC Canada - What are the various generations of single phase Smart-UPS Products? There are pictures to help identification as well.
    • Help with APC UPS - SMT vs SMC? - General Support - Unraid
      • SMT will give you longer runtime.  It's more of an enterprise grade model than the SMC.
      • Also with the SMT you can add that network management card into the back of it.
    • Is this the older version of the smc1000i? what's the difference between smc and smt? | Amazon.co.uk: Customer Questions & Answers
      • The main differences between SMC and SMT is:
        • SMT has a higher output power capacity at 1000 watts compared to only 900 watts for the SMC.
        • SMT has a Network Management Card slot for use with network communication.
        • SMT units have a 3 years factory warranty while the SMC only have 2 years factory warranty.
    • SMC vs SMT (pdf) | APC
    • Which Smart ups? | MacRumors Forums
      • I have been using a APC SmartUPS 1500 for several years with no problems.
      • In simple terms, a true "Sine Wave" output mimics the power coming from your electrical outlets and provides more total power (energy) to your computer than does a "Step Wave" output which has many gaps. This causes the power supply in your computer to work harder to deliver the required current needs when running from a stepped input waveform.
      • There is an alternative supplier remember - CyberPower. The forum search will show you plenty of threads.
      • As long as you buy a Pure Sinewave model rather than a Stepped Sinewave you will be fine. For APC that means the SmartUPS (pure) not the BackUPS (stepped) range.
      • To reduce risk of mission critical parts, I always advice a preventative maintenance plan. Rule of thumb, replace your PS every 6 - 8 years.
      • I've used Tripp-Lite UPSs in the past and they worked fine. But, as everyone has stated, APC seems to be the gold standard now. However, you should have no issues with the Tripp-Lite and it should work fine for you.
  • Some Shops (note used or recommending)
    • UPS, New Batteries - UPS Trader
      • Supplier of Refurbished UPS, New Batteries and UPS associated hardware. 12 Month RTB warranty and new batteries as standard.
      • We have been retailing UPS and associated hardware since Aug 2001. We have thousands of units in stock and have extended experience of working with UPS. We can guide customers to the the best option for their specific use, and all of our units are priced at a fraction of their original RRP. Unless stated otherwise, our UPS will have new cells fitted and 12 month RTB warranty with us. They will all be fully functional.
      • We also offer the battery packs that UPS are dependent upon.
      • Has a support forum.
    • Scan
    • eBuyer
    • Amazon
    • Box
    • eBay
    • Comms Express
  • How to turn off
    • how to turn off apc ups? - YouTube
      • Press and hold the power button on the front and as soon as you hear the beep, release the button.
      • This is not in the manual
    • SMT1500IC - just press and hold the power button until unit powers off.

PowerChute

  • Last time I checked, you cannot read the serial number of the UPS via PowerChute
  • PowerChute does not do days. (I think this refers to setting the installation date.)
  • How do I request a new feature in future versions of APC by Schneider Electric products? - APC USA
    • For DCIM products such as StruxureWare Data Center Expert, StruxureWare Data Center Operation, StruxureWare Portal and NetBotz, you can enter your request here.
    • For all other APC products, such as Uninterruptible Power Supplies, Power Distributions Units, Automatic Transfer Switches, Network Management Cards, Acccessories and PowerChute software, you can enter your request on the contact form here.

Shutting down a PC with your UPS

Battery Installation

When you initially setup the UPS and you come to the install date, use only the arrow keys to configure the date as pressing enter accepts the date and sets it. This can be awkward to change. This does not affect any of the performance of the UPS but does change the predicted fail date of the battery which is about 4.5+ years.
  • When you connect the UPS to the mains power (Utility) the control unit is on i.e. the menu display and configuration options. The output power to which you will connect your protected devices is not on.
  • Power Button
    • The power button controls the power to the outlets only, not the menu and config control circuitry.
    • If the unit is 'on' and you press the power button, a menu with different options is presented rather that the power outputs just being terminated.
  • The battery tab at the back
    • This is the blade connector you had to connect when setting the unit up
    • It can be safely removed while the unit is on as I have to do this while on the phone with an APC support tech. I have done this several times and caused me no issue.
    • If possible I would make sure you kit is turned off and unplugged from the UPS when doing this.
    • I believe this is to allow hot swapping of batteries.
    • The tab at the back just connects the battery circuit to allow charging and to be able to use them as the power supply.
  • The UPS will charge the batteries whether the unit is on or off as long as it has a good mains connection (and of course the battery tab is installed)
  • SmartConnect - Smart-UPS™ - Welcome
    • This is great for basic monitoring for people at home or small installs.
    • You cannot configure the device remotely with this platform.
    • It allows you to do remote firmware updates.
    • The platform emails you when there are issues with your UPS.
    • Limited for professionals but useful for the small guy.
    • Registering your UPS gives you +1year warranty on the battery. This alone is worth registration.
  • The SMT range uses PowerChute Business Edition software. Available for Windows and Linux.

Changing the Battery Install date

Do not have any kit connected as I am not sure if this will cause any issue.

Method 1 (unverified)

How to Update Battery Date for Smart UPS during Battery Insertion - YouTube

This example shows the procedure using a rack mount UPS, the only difference is that the battery connection for my UPS is at the back, the 'Battery Tab'

  • Pull Battery Tab out
  • Wait for the UPS to give a battery warning.
  • Plug the Battery Tab bag in
  • Follow the menu and set the new battery install date.

Method 2 (didn't work)

  • Pull Battery Tab out
  • Factory reset
  • Power Off via menu
  • Disconnect mains supply (Utility)
  • Wait 30 seconds
  • Re-Install battery Tab
  • Reconnect the mains supply (Utility)
  • Access menu by pressing the enter Key
  • Follow the wizard

Method 3 (didn't work)

  • Pulled tab out at the back
  • Reset to Factory

    to
  • (Optional) Instead of doing a factory reset you can just run the setup wizard from the menu and should have the same effect as a factory reset without loosing your settings.
    • Enter Setup
        Wizard:

      to
  • Follow the setup wizard
    • Setup Wizard
        Press any key
    • Language:
        English
    • Output Voltage:
        230v
    • Local Power:
        Quality
    • Menu Type:
        Standard/Advanced

      to
    • The unit will reboot
    • New Battery
        Installed

      to

        (now when you select Yes, this just goes to the 'Setup Complete',
        it should go to the 'Battery Install
        Battery Install
        Date: 05-may-2021
        just use the arrow keys for setting the date and then enter when finished
    • Setup Complete

Method 4 (doesn't work)

These are the official guides from APC.

The reason they are no good is that they tell you to go to the  "Install New Battery" from the configuration menu, but this does not exist.

Method 5 - Using PowerChute

I have not used this to change my 'Battery Installation Date' but I cant see why it will not work.

  • Install PowerChute
  • Connect your UPS
  • Change your date and apply.

NB: This method does not have an option to change the day, only the month and year.

Troubleshooting

  • Why is my new battery only charging up to 99%
  • PowerChute - UPS Communication Lost
    • Unplug and re-plug USB cable. It does not matter if you have restarted or turned the PC on or off. This could be because of an always on power supply to the USB bus of your PC, not all PCs have this feature.
    • Solved: UPS Communication Lost - Schneider Electric Community
      • Q: I have my SMT1500C connected to my computer via USB and it only retains the USB connection for 15-20 minutes after login before disconnecting; the only way to get the connection back is to unplug and plug in the USB cable again. When the connection is lost, Device Manager shows the USB connection is working under Human Interface Devices and shows the unit under Batteries. Also, the APC PBE Agent service is running and restarting the services doesn't resolve the problem. I've had this unit for less than a week and this has been an issue from day 1. 
      • A: The next time this happens reset comm of the SMC. On the display interface hold the MUTE and MENU buttons simultaneously for 5 seconds. The display flashes to indicate that comm has restarted. 

 

Published in Hardware
Read more...
Page 7 of 96