• Home

Items filtered by date: December 2015

Thursday, 07 December 2017 17:44

Firefox printing output is always reduced or smaller than it should be

This issue plagued me for ages.

When i printed from Firefox either directly or by creating a PDF (via cutePDF), the output was scaled to about 25% of normal size making the printouts unusable.

The percetage reduction is just a guidline as there can be some variation, but the outcome is always a smaller version of what you want.

Solution

In Firefox page setup:

  • remove margins
  • remove shrink to fit + reset scale to 100%
  • change paper type under printer options: Letter -> A4, and set to automatically select
Published in Firefox
Read more...
Thursday, 07 December 2017 17:36

Phronesis - How do I change to separate home screen and apps?

No need to re-flash the rom. You can change it from 'Home screen settings'.

  • Find a blank area on the home page and then press and hold. This will bring up a menu.
  • Select 'Home screen settings'.
  • Select 'Home screen layout'
  • Select one of the following:
    • Home and Apps screens
    • Home screen only

 

Published in Android
Read more...
Thursday, 07 December 2017 17:19

cPanel Security

This article will outline some basic steps to increase the security of cPanel and reduce SPAM but only as an end user. This is not targetted at people who have access to the server, but they can use this information.

To Reduce SPAM

Enable the following services:

  • DMARC
  • SPF
  • DKIM
  • Forced SMTP Authentication - I believe this can help with delivery.
  • Forced SSL/TLS SMTP - Disable all plain text email services.
  • SpamAssassin - Use the following settings
    • score BAYES_95 - 4.5
    • score BAYES_99 - 6.0
    • score BAYES_999 - 7.0
    • score SPF_FAIL - 10.0
    • score SPF_SOFTFAIL - 10.0
    • score URIBL_BLACK - 6.0
    • score URIBL_DBL_SPAM - 6.0
    • score URIBL_JP_SURBL - 4.0
    • score URIBL_WS_SURBL - 4.0

Services

Enable the following services:

  • DNSSEC
  • Anti-Virus
    • ClamAV
    • Ummunify

Things to do

  • Go through the cPanel Security Center
  • Run Health Check
  • Run Security Advisor
  • Check for any account over-quota, this can be a sign of a virus.
  • Only use secure DNS providers such as 9.9.9.9 and avoid using 8.8.8.8 etc..

 

Published in cPanel
Read more...
Tuesday, 28 November 2017 11:00

Run OpenWrt as a Virtualbox virtual machine

These are my instructions to install LEDE on a Virtualbox virtual machine so i can play around with various setups and add-ons without breaking my router. I am a windows user, so these instructions are windows based but a lot should be the same. Through this guide I will use 'VB' which is instead of typing virtualbox virtual machine lots of times.

Download and install VirtualBox before anything else.

These instructions are a companion to LEDE Project: Run LEDE as a Virtualbox virtual machine which has some errors in it.

Which Image File

This is can be confusing so I will outline what the different types are below. The image files are also specific to certain chips or technology, so in this case these images are for x86/x64 archeitectures. The Stable Releases are also sorted by their release number, so make aure you always get teh latest version.

  • Stable Release - These are the current stable production releases and also include LuCI (the GUI for LEDE)
  • Snapshots - These are the bleeding edge release and are for development and testing. They do NOT contain LuCI.
  • squashfs
    • Images with this in title store all the primary system files in a compressed read only partition. Now I am not 100% how this works but the 2 scenarios but there are
      • All changes to the system files are stored on the eread/write partition and override those on the read only partition
      • The system files on first boot are copied into the read/write partion and it is those files that are used.
    • These images allow the use of a factory reset because they can wipe the read/write partition and then restore data from the read only partition
  • ext4 - Images with this in the title are = not sure yet, perhaps there is just 1 single ext4 partition

The 2 files to choose from will have filenames similiar to these which are from the 17.01.4 stable release folder.

I have chosen to use lede-17.01.4-x86-64-combined-ext4.img.gz because I can easily expand the ext4 storage partition.

Notes

Prepare the virtual hard drive Image file (lede-17.01.4-combined-ext4.img)

  • This is the preferred image to use for Full systems, PCs and Virtual Machines.
  • The ext4 partition can be easily extended for those who need more storage.
  • You will also notice that this image file is full size when you extract it.

Make the VirtualBox VDI file

Run the following command in the directory you have your LEDE image file.

"C:\Program Files\VirtualBox\VBoxManage" convertfromraw --format VDI lede-17.01.4-combined-ext4.img lede-17.01.4-combined-ext4.vdi

Now skip to the section Make the Virtualbox Virtual Machine.

Prepare the virtual hard drive Image file (lede-17.01.4-combined-squashfs.img)

  • This image is designed for embedded systems or systems with a small amount of storage.
  • You can usually image this directly to a real machine but image file will not work out of the box because it needs to be prepared to use in a virtual machine.
  • I have not been able to resize the storage partion for this image type and from what I read you are suppose to do this when you compile the image file. I am sure there is a work aroun but i cannot find it.

Make the VirtualBox VDI file

Run the following command in the directory you have your LEDE image file.

"C:\Program Files\VirtualBox\VBoxManage" convertfromraw --format VDI lede-17.01.4-x86-64-combined-squashfs.img lede-17.01.4-x86-64-combined-squashfs.vdi

Converting the img file to a VDI file fails with an error

When you try and convert an untouched  lede-17.01.4-x86-64-combined-squashfs.img to a virtualbox VDI file using VboxManage you will always get an error similiar to the one below:

D:\>"C:\Program Files\VirtualBox\VBoxManage" convertfromraw --format VDI lede-17.01.4-x86-64-combined-squashfs.img lede-17.01.4-x86-64-combined-squashfs.vdi
Converting from raw image file="lede-17.01.4-x86-64-combined-squashfs.img" to file="lede-17.01.4-x86-64-combined-squashfs.vdi"...
Creating dynamic image with size 19852146 bytes (19MB)...
VBoxManage.exe: error: VD: The given disk size 19852146 is not aligned on a sector boundary (512 bytes)
VBoxManage.exe: error: Error code VERR_VD_INVALID_SIZE at F:\tinderbox\win-5.1\src\VBox\Storage\VD.cpp(7002) in function int __cdecl VDCreateBase(struct VBOXHDD *,const char *,const char *,unsigned __int64,unsigned int,const char *,const struct VDGEOMETRY *,const struct VDGEOMETRY *,const union RTUUID *,unsigned int,struct VDINTERFACE *,struct VDINTERFACE *)
VBoxManage.exe: error: Cannot create the disk image "lede-17.01.4-x86-64-combined-squashfs.vdi": VERR_VD_INVALID_SIZE

D:\>

Explanation

This simple to explain. The image file is only 19MB in size but represents a drive that is nearly 300mb in size (16mb + 256mb). VboxManage parses the img file and sees a 300mb file system so it expects a file that is 300mb long, but because the file is not 300mb long as expected VboxMan returns an error. This is basically an unexpected end of file error. VboxMan actually interprets the files system in the file which is stored at the beginning so this error is expected.

NB: the img file does not need to be 300mb in normal circumstances because if you image the file 'as-is' onto a large enough drive then you would not have this issue because the OS would just see the extra space dues tot he partition table being correct. This keeps the file size to the absolute minimum. We only change the image because we want to create a virtual drive from the img and this process like validate partions and files systems so it can convert them to its own format.

Research / Examples

Look at these 2 examples where I have padded the files with 100mb and 300mb respectively, converted them to a VDI image, loaded them in the LEDE virtual machine and then booted with Gparted to see what was going on with the disks.

The 100mb example has an error that come up before you even get the disk map.

100m partition error

This error actually appears twice, I am guessing that is because there are 2 partitions.

100mb disk map

Now as you know is not possible to fit the sda2 256mb partition onto a hard disk that is only 100mb in size. . The 100mb drive is likely to be corrupted or cause many issues.

So I now move on to where I padded the image file with 300mb.

300mb disk map

You can see the whole partition sda2 is within the drive boundries so will be healthy. VboxManage considers the whole file size when creating a VDI file. Both the 100mb and the 300mb VDI files were 9,216kb exactly so there must only be a slight difference in a setting controlling the virtual hard drive size.

NB: I do not know why the file system is unknown.

Padding the file

Now we know what the issue is we can fix it. We do this by making the file 300mb in size using the linux command dd. This allows us to extend the length of the file without affecting the contents that are already there.

You need a Linux system to be able to run the following commands. If you are on windows there is section below briefly telling you how to setup an Ubuntu virtual machine.

This command is wrong (from LEDE Project: Run LEDE as a Virtualbox virtual machine )

dd if=lede-17.01.4-x86-64-combined-squashfs.img of=lede.img bs=128000 conv=sync
  • bs=128000 - this sets the block size that dd will work with and this is to small to wrk with a 300mb file so the secondary partion will get truncate but the primary system partition is still thre. This is why people will get a VB machine to boot but have no spare disk space to work with.
  • The outputed file is xxxxxxxxMB in size.

Use this command

dd if=lede-17.01.4-x86-64-combined-squashfs.img of=padded-lede-17.01.4-x86-64-combined-squashfs.img bs=300M conv=sync
  • This command makes an image that is 300mb which is more than enough to hold the whole file system.
  • The names are slightly different but that is just to make managing the files easier.
  • The outputed file is 300MB in size.

We now have a file that will convert

Run the following command in the directory you have your padded img file to create your virtual hard drive VDI file.

"C:\Program Files\VirtualBox\VBoxManage" convertfromraw --format VDI padded-lede-17.01.4-x86-64-combined-squashfs.img lede-17.01.4-x86-64-combined-squashfs.vdi

Notes

Create an Ubuntu Virtual Machine

This is a workaround for windows users to be able to use the dd command. There must be a windows equavalent but I don't know of one at the minute so here are my instructions to setup an Ubuntu virtual machine.

I used an Ubuntu 17.10 Artful (Final) Virtualbo prebuilt image from http://www.osboxes.org/ubuntu/

Credentials

  • Username: osboxes
  • Password:org

Create the VB Machine

  • Extract the VDI file to your Vitual Machine images folder
  • Create a VB machine in expert mode using the settings below
    • Name: Ubuntu 64-bit
    • Type: Linux
    • Version: Ubuntun (64-bit)
    • RAM: 2048mb
    • Use an existing virtual hard disk file: point it to the VDI file you just extracted
    • Click create
    • Now run the machine

Configure the VB

Ubuntu is now running but we need to get the image files into the installation so there are a few settings that need to be configured to amke this an easy process.

  • Open Settings
    • Top right, click the drop down arrow
    • Click on the screwdriver and spanner
    • All settings can be managed here
  • Set regional settings (optional)
    • The keyboards is set to American , so needs changing to your locale.
    • Check Time region aswell
  • Update ubuntu
    • this is required to be able to enable file sharing (for some reason, see 12.10 - Package 'samba' is virtual - Error when installing Samba - Ask Ubuntu) other wise you will get errors about missing services (sharing service is not install / package 'samba' is virtual / Could not find package 'libpam-smbpass' / Sharing service installation failed)
    • verify you are connectedd to the internet
    • Open up a terminal console and run sudo apt-get update
    • Install any updates you are prompted for
    • Reboot
  • Enable file sharing
    • Make sure the Ethernet is enabled in ubuntu (should be)
    • Follow the instructions from Share Folders On Local Network Between Ubuntu And Windows
      • share out desktop folder with ‘Allow others to create and delete files in this folder’ and ‘Guest access’ enabled
      • allow any permissions that you are prompted for

Pad the file

You can now access your Ubuntu install by using \\osboxes

  • Start the Ubuntu machine (if not already)
  • on your windows PC run \\osboxes and you should now see the Ubuntu desktop
  • copy the lede-17.01.4-x86-64-combined-squashfs.img to the Ubuntu desktop
  • Open up a terminal console and type the following commands
  • navigate to theDesktop folder by type the command cd Desktop
  • now run the dd command shown above and this will pad the file as required.
  • on your windows PC open \\osboxes and copy of the padded file
  • shutdown your Ubuntu machine as it is not needed now. Dont delete the machine you might need a linux machine in the future.
  • Carry on with the next section below.

Notes

Make the Virtualbox Virtual Machine

Follow the instructions below

  • Open the page LEDE Project: Run LEDE as a Virtualbox virtual machine
  • Goto VM Setup in VirtualBox section
  • Follow the VM Creation instructions
    • I set my RAM to be 512MB (optional)
  • Follow the VM setup/Virtualbox settings instructions
    • ignore that fist paragraph of text
    • the first network section deals with general Virtualbox network preferences and should all be setup.
    • You might see VirtualBox Host-Only Network or VirtualBox Host-Only Network #2 and these can be used instead of vboxnet0 as they are just different names for the same thing.

Make the data partition larger (squashfs) (research only/not working)

I never got this to work (resizing sda2) but I did a lot of looking on the internet and here are the links I collected.

Extroot

Make the data partition larger (ext4 only) (optional)

The virtual hard disk is split into 2 partitions, 16mb system and a 256mb data partition. For most people this should be enough however if you want more space follow these instructions which are best done before you start playing with your LEDE install bu you can do this later if you want.

Increase the capacity of the virtual hard drive

  • Unmount the VDI file from your virtual machine if attached
  • Backup the VDI file (optional)
  • Run the following command in the same folder as the image file to resize the virtual hard drive to 10GB
  • "C:\Program Files\VirtualBox\VBoxManage" modifymedium lede-17.01.4-combined-ext4.vdi --resize 10000

NB: For compatibility with earlier versions of VirtualBox, the "modifyvdi" and "modifyhd" commands are also supported and mapped internally to the "modifymedium" command. see Chapter 8. VBoxManage

Increase the partition size

Just by increasing the size of the hard drive will not increase the size of the Partitions so now we have to make the partition(s) bigger

  • remount the new larger VDI file to your LEDE virtual machine
  • Download Gparted
  • Load the ISO into the DVD-ROM drive of your virtual machine
  • Power on the  LEDE Virtual machine and boot of the DVD Drive
  • Gparted will now load after you have selected the appropriate settings
  • Use Gparted to expand the required partition. (Dont forget to apply the changes)
  • Power of the virtual machine and remove the Gparted ISO
  • Power on and boot LEDE

Your sda2 partition will not have 10GB of free space.

Notes

Notes on partitions, their layouts and the process of making more space available to LEDE in a virtual machine.

Configure LEDE

Quick Setup

LEDE will now boot but we will not be able to access it because the IP address is wrong. To fix that:

  • Start the new Virtual machine
  • Goto the virtual machine as we will be working in it now
  • Press enter when br-lan message is display and LEDE console will appear
  • Type the following commands in, each followed by pressing enter
    • uci set network.lan.ipaddr='192.168.56.2'
    • uci commit
    • reboot

Because we used the stable release when LEDE is booted up you can access LuCI via you browser by using the address http://192.168.56.2/ .This means you can do a lot of editing in the GUI rather than the command line. If you used a snapshot you will need follow the Full Setup shown below and install LuCI manually.

Full Setup

There are many more settings that 'might' need to be done but I have not got that far yet.

You are all done now, enjoy!!!

Misc

These are some other things that do not fit into the guide above but might be useful for some people.

Upgrading

I do not know how to do this. I am guessing you just update the root/system/sda1 partition with either of these packages as appropriate:

I/O Errors

If you get I/O errors when using the uci command it is probably caused by lack of free space on the file system. Instructions below. Using VI will not fix these errors but will let you change the IP address in /etc/config/network so you can access LuCI for easier diagnosis.

Editing with VI editor

Just incase you cannot make any changes with the uci commands then here is a crash course on VI editor

  • Insert = starts editing
  • Escape = takes you to the VI command line
    • :w = writes the changes
    • :q = quits VI editor

Notes

Moving a VDI image

You really should disconnect a VDI image from Virtualbox via the 'Virtual Media Manger' before moving the VDI file but if you dont here is the quick way of fixing that issue.

  • Goto the 'Virtual Media Manager'
  • Release teh affect VDI image
  • Delete the VDI image, but keep the file
  • Goto the Virtual machine
  • re-add the VDI while it is in it's new location

Notes

Questions

Published in DSL / Broadband
Read more...
Saturday, 25 November 2017 16:22

Install OpenWrt on a BT Home Hub 5 / Plusnet One Router

Updated to [OpenWrt Wiki] OpenWrt 21.02.1 - Service Release - 25 October 2021

Overview / Purpose of this guide

These instructions are for aimed at users of Windows but a lot of the information will work for other OS users.

I wrote these instructions just to clear few things up so next time I flashed a BT Home Hub 5 Model A (HH5A) router, it would be easier. The instructions done by other people are not necessarily faulty but not as clear as I needed at points and this mini guide will address those. It must be said I am also a bit of a newbie at electronics so not everything is obvious.

These instructions are complementary to the guides and information that has already been done by the community and in particular thanks goes out to:

  • Bill at ebilan for the excellent instructions for installing OpenWrt on a HH5A.
  • OpenWrt groups for the hard work with the open source router software.
  • mkresin - for the BT HH5A install image
  • Anyone else I have not mentioned who helped with this setup.

My guide covers all aspects of this process:

  • Opening the routers case
  • Soldering the serial connection
  • Flashing / Installing OpenWrt and backup of the original router firmware
  • Configuring OpenWrt on the router

Things you need

Step 1 – Opening the routers case

If you do not open the case carefully you will break the 'fixing tabs' inside. I broke them on my first go and then repaired them with a soldering iron. Look at these instructions in order on how to open the case.

Step 2 – Soldering the serial connection

Before you start soldering read this article:

Simple no soldering flashing jig made from a cat5e keystone jack - openwrt.ebilan.co.uk

It is perfect for those who cannot solder. You will need to buy a RJ45 Network Keystone from eBay.

You can use your own method for soldering the connections as long as you have access to the required connections on the circuit board and can attach them to a USB to TTL RS232 Converter.

People have done this many different ways but I am going to replicate the arrangement on the OpenWrt BT HomeHub 5.0 Type A as this is the page most people will land and the method is quite clean.

You do not have to use the ground points that I have used based on the OpenWrt BT HomeHub 5.0 Type A page.

This method described on this page mounts a 5 pin header block designed for reuse.

Connections of the 5 pin header

Before soldering, you need to know what you are soldering and why.

The images and instructions could be a little clearer about what is actually soldered to where, so I will just clear that up now.

I have taken the image off the OpenWrt HH5A page that shows the wires connected and then labelled the connections with an accompanying table below showing the connections. These are now labelled in a logical order and are easier to follow but do not have the same numbering as the OpenWrt HH5A page.

serial console connector

Image from the OpenWrt HH5A page

Pin No. USB Serial to TTL Adapter Wire Colour Home Hub Connection Point
1 n/a Blue boot_sel2 / R45 Pad
2 n/a n/a GND / Use the USB socket ground/case pin
3 TX Blue RX / R78 Pad
4 RX Purple TX / R77 Pad
5 GND n/a GND / Use the ground plane connection of the capacitor
  • As you can see there are 2 groups. One is for the ‘Boot Selection’ and the other is for the 'Data Connection'
  • There are 2 grounds because each group needs its own ground.
  • The ground on the left is a case pin of the USB socket and the one on the right is the ground plane of a capacitor. The 5 pin header is soldered directly to these points (pins 2 and 5) which gives it stability as well as a ground connection on each pin.
  • If left as is, the case would need to have some holes drilled into it to allow the 5 Pin header to be exposed outside of the case for permanent access.
  • After the process is complete the serial interface is not needed unless router becomes bricked.
  • Some people say to remove the serial interface (wires etc.) as it can affect the wireless calibration. Eliban left his in

This is a close up of the pads

pad positions close up

Image from the OpenWrt HH5A page

Fitting the 5 pin header and soldering the wires to the pads

These are my newbie notes

  • Do not use to much heat because you will damage stuff. If it gets a bit warm wait a few minutes before doing the next pad or pin.
  • if you are not a good solderer, get some practice (or someone to do it for you)
  • Mask of the area around the pads with some kapton tape to prevent accidental damage to components

Soldering instructions

  1. Get some wires and make sure you get the approximate length to match how it looks in the picture
  2. Strip the ends of the wire, but not too much.
  3. Solder the wires to the required pins of the 5 pin header.
  4. Solder the 5 pin header on to the 2 grounding points.
  5. Make sure you have not melted the 5 Pin header and the pins are still tight so do not move.
  6. Lay the wires in place on the board so they run from the header to the pads
  7. Tape the down the wires with some Kapton tape. This makes it much easier to solder them to the pads and removes load of the solder pads when the wires are soldered to the pads
  8. Solder the wires to the pads carefully. The solder mask will be quite forgiving.

Example of neat soldering

Step 3 - Flashing / Installing OpenWrt and backup of the orginal router firmware

Follow the instructions from 1-OpenWrt-LEDE Installation Guide for HH5A.pdf (ebilan) and use this as a companion guide.

Download Firmware / Install Files for the HH5A

You should of downloaded these from the prerequisites section.

Install the required software

  • PuTTY
  • Notepad++
  • TFTPD32 - Use the Standard edition installer
  • WinSCP - This is only needed for messing with the router later. FlashFXP will also work.

Connect the adapter

  • Connect the USB serial adapter to the 5 pin connector as the connection table above.
  • Connect the USB serial adapter to windows.
  • Get the COM number of the adapter by going to windows device manager and find the COM assignment of the new device.

Configure Putty

  • As per the ebilan instructions page 3

Access the router firmware via the console

With putty running and the adapter connected:

  • Short the boot_sel2 pin to Ground (GND) and power on the HH5A. Now immediately disconnect the boot_sel2 pin from Ground. The UART ‘CFG 04’ prompt will appear.
    Do NOT leave it connected any longer than necessary because it may damage the hub, causing it to remain permanently stuck in ‘CFG 04’ mode!
  • CFG04 mode should now appear in the console. This means it is in a read/write mode.
    • If you get CFG06 either you soldering is wrong for the boot select, you did not make a good short for long enough or you did not short the pins properly.

Transfer ‘HH5A LEDE/OpenWrt install image’ to the router

  • Transfer the lede-lantiq-bthomehubv5a_ram-u-boot.asc (u-boot) to the router
    • Open the .asc file with Notepad++, select all and copy
    • Goto the Putty console window and right click (should still be open and connected to the router). The asc file will transfer and execute.
    • I believe the .asc is file contains loads of pre-written scripts that get executed
    • After about 90 seconds, the custom U-boot will start and the BTHOMEHUBV5A U-boot prompt will appear.
  • On you windows PC set IP on the Ethernet to be 192.168.1.2/255.255.255.0
  • Open TFTPD32
  • Go back to putty
    • Type the following command into the console
      tftpboot 0x81000000 lede-lantiq-xrx200-BTHOMEHUBV5A-installimage.bin; bootm 0x81000000
    • The file should automatically transfer via TFTP
  • Wait 5+ mins for ‘br-lan’ to appear and then press enter

Now move on to the 1-OpenWrt-LEDE Installation Guide for HH5A.pdf (ebilan) as the rest of the instructions should be the same as the linux OS.

Backup BT Firmware (ebilan 3.0)

Before doing anything else you need to back your routers original firmware.

When you remove a pendrive from windows you should eject it properly so the volume is not marked as dirty

  • Plug a pendrive into the USB socket (if a FSCK notice is shown, the volume was not unmounted properly, just a warning but it is better to use a clean pendrive). It will auto mount, usually as USB-A1
  • Get the USB mount name by typing 
     ls /tmp/mounts
  • Check the pendrive works by doing a directory listing 
     ls /tmp/mounts/USB-A1
  • Run the backup command to send the nanddump to the pendrive took (12 minutes to complete on mine) 
     nanddump -f /tmp/mounts/USB-A1/hh5a.nanddump /dev/mtd4
  • Unmount the pendrive and transfer the nandbackup backup to a PC. You can run the backup process again as above and then binary compare them if you want. 
     umount /tmp/mounts/USB-A1
  • Whilst the USB is in the PC, put the file lede-17.01.4-lantiq-xrx200-BTHOMEHUBV5A-squashfs-sysupgrade.bin in the root of the USB
  • Plug the USB back into the router

Replace the stock BT firmware (ebilan 4.0)

  • Eun the prepare script by entering the command below. This will unlock the internal bootloader. Only ever run this once. 
    prepare
  • Enter the confirmation
  • The bootloader is now decrypted, unlocked and rewritten back to the firmware
  • A LEDE/OpenWrt console will now appear root@lede:/#
  • Enter the command 
     sysupgrade /tmp/mounts/USB-A1/lede-lantiq-xrx200-BTHOMEHUBV5A-squashfs-sysupgrade.bin
  • This only takes a couple of minutes at most before the CFG04 message is on screen and this means the flash has finished successfully
  • Powercycle the router

Backup settings - (ebilan 4.3)

Always backup your current OpenWrt configuration before making any significant changes.

  • (System-->Backup / Flash Firmware-->Backup / Restore-->Download Backup)
  • Get the backup of your configuration settings and store safely on another device.
  • This seems to include all of the /etc/ folder and in particular the /etc/config/ folder

We have now installed OpenWrt successfully installed on our HH5A. You should now follow my instructions but I still use references to the 1-OpenWrt-LEDE Installation Guide for HH5A.pdf (ebilan) tutorial.

Step 4 - Configuring OpenWrt on the router

The following are my instructions on how to configure OpenWrt to give the same functionality (and more) of the orginal HH5A.

Router password

(System-->Administration-->Router Password)

Notes

  • This must be done ASAP for security.
  • When the password is not set you can just click on the message at the top for easy access
  • I use the admin password off the router itself because it is easier to remember

Change hostname and local time

  • Goto (System --> System --> General Settings)
    •  Hostname: primaryrouter
      • You can pick a name of your choosing but it is easier to keep it like this for now while doing the tutorial.
      • This prevents confusion if using other OpenWRT routers
    • Timezone: Europe/London (this is correct for me)
  • Click 'Save & Apply'

SSH / SCP

This is optional and probably does not need changing for most peoples setup.

(System --> Administration --> SSH Access --> Dropbear Instance --> Interface)

  • By default dropbear is set to listen to port 22 on all interfaces (unspecified), but the default firewall blocks it on the WAN zone.
  • Perhaps set this to ‘LAN’ for better security?

Wireless

(Network --> Interfaces --> Wireless)

Only change things if they are noted below, else just leave as is. Some items are left in for reference if you are wondering.

Set the WiFi SSID and encryption with a suitable password

  • 5GHz / wlan0
    • Device Configuration --> General Setup
      • Operating frequency: (AC / 36(5180 MHz) / 80 MHz) (default)
      • Maximum transmit power: driver default (default)
    • Device Configuration --> Advanced Settings
      • You dont need to change anything here.
    • Interface Configuration --> General Setup
      • Mode: Access Point (default)
      • ESSID (SSID): quantumwarp_5G
    • Interface Configuration --> Wireless Security
      • Encryption: WPA2-PSK/WPA3-SAE Mixed Mode (strong security)
      • Cipher: auto (default)
      • Key: {your password}
    • Interface Configuration --> MAC-Filter
      • You dont need to change anything here.
    • Interface Configuration --> Advanced Settings
      • Country Code: GB United Kingdom (or your country)

 

  • 2.4GHz / wlan1
    • Device Configuration --> General Setup
      • Operating frequency: (N/ 11(2462 MHz) / 20 MHz) (default)
      • Allow legacy 802.11b rates: unticked (default) - HH5A would have this enabled, but no-one uses this slow speed any more.
      • Maximum transmit power: driver default (default)
    • Device Configuration --> Advanced Settings
      • You dont need to change anything here.
    • Interface Configuration --> General Setup
      • Mode: Access Point (default)
      • ESSID (SSID): quantumwarp
    • Interface Configuration --> Wireless Security
      • Encryption: WPA2-PSK/WPA3-SAE Mixed Mode (strong security)
      • Cipher: auto (default)
      • Key: {your password}
    • Interface Configuration --> MAC-Filter
      • You dont need to change anything here.
    • Interface Configuration --> Advanced Settings
      • Country Code: GB United Kingdom (or your country)

Notes

  • WiFi is off by default
  • You can also use the same SSID for both networks if you want. This is the way BT Home Hub normally does it.

BT/Plusnet fibre using the DSL socket

  • Read How do I connect the HH5A to my UK ISP ? (eliban 7.5) – this will give you the settings for your VDSL (fibre) or ADSL line but more importantly gove you the correction section to read next which depends on your required setup.
  • Select the correct section to goto, In my case (eliban 9.1)
  • Read Quick PPPoE setup using DSL port for VDSL Connection (eliban 9.1)
  • Follow the instructions below to configure your BT/Plusnet fibre.

(Network-->Interfaces-->WAN-->Edit)

  • Protocol =PPPoE
  • PAP/CHAP username = your broadband username
  • PAP/CHAP password = your password if you have one
  • If you need to set a VLAN tag (i.e. for BT with VLAN tag of 101)
    • (Network-->Interfaces-->WAN-->Edit-->Physical Settings-->Interface-->Custom Interface:) = dls0.101
    • This setting will not stay on Custom Interface: but will create a new setting called Software VLAN: "dls0.101" (wan) which will be selected.
  • Leave the rest

(Network-->Interfaces-->DSL)

  • Settings for the UK
    • Annex: B (all)
    • Tone: A (A43c + J43 + A43)
    • Encapsulation mode: PTM/EFM (Packet Transfer Mode) (default)
    • DSL line mode: VDSL - slightly faster to connect on VDSL if you specify, however the HH5A would have this on auto.
    • Ignore the rest

(Network-->Interfaces-->ATM Bridge)

  • Delete the ATM Bridge
    • This is only needed for ADSL

Now Reboot Router (System-->Reboot)

DNS

OpenDNS / Quad9 / Custom DNS servers

This still sends 192.168.1.1 to the clients but sets the DNS servers used by the router to OpenDNS

  • (Network-->Interfaces-->WAN-->Advanced Settings-->Use DNS servers advertised by peer) = unticked
  • (Network-->Interfaces-->WAN-->Advanced Settings-->Use custom DNS servers) = OpenDNS servers (208.67.222.222 / 208.67.220.220)
  • Check out Quad9 and use their security based Public DNS servers (9.9.9.9 / 149.112.112.112) as an alternative.

Notes

  • In (Networks-->Interfaces-->LAN-->Edit-->Advanced Settings-->Use custom DNS servers) you can set what DNS servers are issued to the LAN clients.
    • This becomes available after you unticked/disable:
      (networks-->Interfaces-->LAN-->Edit-->Advanced Settings-->Use DNS servers advertised by peer)
  • In (Network-->DHCP and DNS-->General settings-->DNS forwardings) you can change where DNS queries are forwarded to.
  • Edit the config file manually /etc/config/dhcp
  • Configuring OpenWrt and OpenDNS to log all DNS lookups - might has some relevance.
  • You can use the Dynamic DNS plugin to update the client IP at OpenDNS servers directly (or use DNS-O-Matic) so you can perform any required filtering you want such as a familiy filter etc...

DHCP

Notes

  • DHCP settings are configured on the interface (i.e. network-->interfaces-->LAN-->DHCP Server)
  • You can edit the DHCP settings manually by editing the Config file = /etc/config/dhcp
  • (network-->DHCP and DNS-->) has the non-interface specific settings for DHCP and DNS.

Set Red WAN port as a LAN port (optional)

  • 1-OpenWrt-LEDE Installation Guide for HH5A.pdf (ebilan) - part 9.7 (page 72)
  • (network-->switch-->VLAN ID 1) set the WAN to be untagged
  • (network-->switch-->VLAN ID 2) set the WAN to be off
  • You need to do both settings above otherwise you get this error
    “WAN is untagged in multiple VLANs!”
    You could delete VLAN 2, but I don’t know what that does.

Notes

  • By default: VLAN1 WAN = off, VLAN 2 WAN = untagged

Firewall

I currently do not have any specific configurations for the firewall because it has all been done.

Notes

  • OpenWRT - Firewall - Port Forwarding and Traffic Rules - YouTube ( Van Tech Corner) In video video, we use OpenWRT Firewall to configure Port Forwarding and Traffic Rules. We will discuss the basic concept of Firerwall, such as zones, actions and network interfaces.
  • Port Forwarding: Just that, it forwards ports based on patterns.
  • Traffic Rules: Thi can perform none routing actions on packets based on patterns and the actions are:
    • drop, accept
    • reject
    • don't track
    • assign conntrack helper
    • apply firewall mark
    • XOR firewall mark
    • DSCP classification

LED Lights

You are able to configure the router's 3 compound LEDs to provide feedback from the router's various functions with a selection of colours which can also be controlled independently.

There are some really nice features you can add such as the heartbeat effect where you can translate the CPU load in to a heartbeat pulse via the select LED. I use the WiFi LED configured as red.

Colours available

  • Red
  • Green
  • Blue
  • Orange - The router hardware is capable, but OpenWrt is not because of a limitation in the driver (See below)

Notes

Links

LED Configurations

This section contains various configuration for your HH5A LEDs.

Default OpenWrt Configuration

This is the default configuration for the LEDs when you first install OpenWrt.

config led 'led_wifi'
	option name 'wifi'
	option sysfs 'blue:wireless'
	option trigger 'phy0tpt'

config led 'led_dsl'
	option name 'dsl'
	option sysfs 'blue:broadband'
	option trigger 'netdev'
	option mode 'link tx rx'
	option dev 'dsl0'

config led 'led_dimmed'
	option name 'dimmed'
	option sysfs 'dimmed'
	option default '0'

Configure LEDs to be like the HH5A

This is not exactly the same as the default BT HH5A LED configuration because the LED configuration in OpenWrt lacks the features I need to program this.

This configuration will do the following:

  • The WIFI LED will be disabled.
  • The Internet LED will light up blue when the DSL connection is up
  • All LEDs will be dimmed.
  • The load up procedure and it's LED configuration is untouched because it is controlled by the bootloader. This also means the Power LED will stay on coloured as blue.
  • the rules are processed in order from the top down and can be re-ordered.

In LuCi

  • Goto LED Configuration
  • Edit wifi and set the following:
    • Name: wifi
    • LED Name: blue:wireless
    • Trigger: Always off (kernel: none)
  • Edit dsl and set the following:
    • Name: dsl
    • LED Name: blue:broadband
    • Trigger: Network device activity (kernel: netdev)
    • Device: dsl0
    • Trigger Mode: Link On
  • Edit dimmed and set the following:
    • Name: dimmed
    • LED Name: dimmed
    • Trigger: Always On (kernel: default-on)
    • Default state: ticked

This is the code the rules above create in the /etc/config/system file.

config led 'led_wifi'
	option name 'wifi'
	option sysfs 'blue:wireless'
	option trigger 'none'

config led 'led_dsl'
	option name 'dsl'
	option sysfs 'blue:broadband'
	option trigger 'netdev'
	option dev 'dsl0'
	list mode 'link'

config led 'led_dimmed'
	option name 'dimmed'
	option sysfs 'dimmed'
	option trigger 'default-on'
	option default '1'

Disable Power LED / Other LED

It is possible to disable the Power LED (or other LED) by adding a new rule via LuCI or directly in the /etc/config/system file. The Power LED is turned on by the bootloader and this is why you need to add an extra rule whereas other LEDs you could just delete the rule that turns them on.

In LuCi

  • Goto LED Configration
  • Click on the 'Add LED action' button
  • Add a new LED action with the following:
    • Name: power
    • LED Name: blue:power
    • Trigger: Always off (kernel: none)

Or you can add the following code directly into the /etc/config/system file.

config led
	option name 'power'
	option sysfs 'blue:power'
	option trigger 'none'

Unmodified HH5A LED behaviour

This is the default LED behaviour of an unmodified HH5A and I can use this to program up the LED behaviour in OpenWrt to match this default behaviour when the required functionality is added in a future version of OpenWrt.

Booting a configured router with DSL plugged in

Router bootloader

  • (0:00) Start:
    • Power: Green, Solid
  • (0:25) Power: Green, Flashing

Router has booted

  • (0:40) Power: Blue, Solid
  • (1:00) Power: Orange, Flashing
  • (1:50) Power: Orange, Solid
  • (2:00) Internet: Orange, Solid
  • (2:33) Internet: Orange, Off
  • (2:35) Power: Blue, Solid

NB: occasionally at 1:15 the orange light goes solid while the internet led flashes about 4 times before resuming flashing orange.

Remove DSL wire from a connected and configured router

  • (0:00) Start:
    • Power: Blue, Solid
  • (0:07) Power: Orange
  • (0:07) Internet: Orange, Solid
  • (0:13) Internet: Red, Flashing (0.5 second on/off)

Plug in DSL wire to a configured router

  • (0:00) Start:
    • Internet: Orange, Solid
    • Internet: Red, Flashing (0.5 second on/off)
  • (0:06) Power: Orange, Flashing (0.5 second on/off)
  • (0:40) Power Orange, Solid
  • (0:55) Internet: Orange, Solid
  • (1:24) Internet: Red, Off
  • (1:25) Power: Blue, Solid

Hold the WPS Button

  • Wifi: Orange, Flashing (2 second on/1 second off)

Step 5 - Additional OpenWrt functionality with Add-ons (Packages)

(System-->Software)

Your Router must be on the internet to be able to install software

These extra features do not come pre-installed in OpenWrt so need to be installed. The GUI (web interface) is called LuCI and all of the GUI add-ons are prefixed with ‘luci’ so you can search for just GUI add-ons by using the search term 'luci-'. I have found that if I install a ‘LUCI’ add-on then the other dependencies such as the actual service it will controls gets installed as well. This is why using LuCI is much better for most people than using the command line.

The plugins I have listed below are the main ones to get your router doing the original HH5A features and the little extras to make life easier.

There is currently no place online you can browse the software repository. The best place for descriptions is to type something in to the search box on the software page of your router and look at the packages you want to install and you will see a brief description of them but only on the 'Available Packages' tab.

Please bear in mind this is my setup to get a good baseline as close to the HH5A as possible. All those add-ons that have (not used) means I have installed them to make sure they work but have uninstalled them because I did not want the features they offered. Some people might want those features so I have left the install notes here for reference.

The optional add-ons should only be installed if you want those features. All the rest of the add-ons should be installed unless you absolutely do not need those features.

OpenWrt will run with none of these add-ons installed.

Add-on Notes

Editing Files (SSH / SCP / SFTP)

You need to use this to edit config files with windows using WinSCP or FlashFXP which is easier than the command line but this requires a little configuration. You can use SSH and edit via the command line using the VI editor.

  • Install openssh-sftp-server - OpenSSH SFTP server.
  • Installs nothing else

Install Notes

  • [OpenWrt Wiki] SSH access for newcomers
  • this sub system is require because FlashFXP give the error {see sftp flashfxp error.txt}
  • You can now edit files like you can in ftp
  • this might already be installed. I did a reset and uploaded my old config and openssh-sftp-server was in the installed packages. This could be a bug or an incomplete reset.
  • If you try and use SFTP over SSH without openssh-sftp-server installed you will get the following error 
    [15:11:23] [R] Connection failed (Unable to access SFTP sub-system, operation failed.)
[15:11:23] [R] Delaying for 10 seconds before reconnect attempt #1

FlashFXP/WinSCP

  • Install WinSCP in windows. FlashFXP will do the same (SFTP over SSH)
  • connect using the 'root' account and password
  • once connected, click up folder once to see all of the folders
  • Config settings are at /etc/config/
  • Edit them like a normal text file
  • Read WinSCP (ebilan 10.2)

Notes

UPnP (optional)

UPnP is not installed by default.

Once installed UPnP will need to be turned on.

WPS

WPS doesn't work out of the box on OpenWrt, you need to:

  • remove wpad-mini
    • This package contains a minimal IEEE 802.1x/WPA Authenticator and Supplicant (WPA-PSK only).
  • install wpad
    • This package contains a full featured IEEE 802.1x/WPA/EAP/RADIUS Authenticator and Supplicant
    • does not install anything else
  • install hostapd-utils
    • This package contains a command line utility to control the IEEE 802.1x/WPA/EAP/RADIUS Authenticator.
    • does not install anything else
  • remove wpad-basic-wolfssl
    • This package contains a basic IEEE 802.1x/WPA Authenticator and Supplicant with WPA-PSK, SAE (WPA3-Personal), 802.11r and 802.11w support.
  • install wpad-wolfssl
    • This package contains a full featured IEEE 802.1x/WPA/EAP/RADIUS Authenticator and Supplicant
    • does not install anything else
  • install hostapd-utils
    • This package contains a command line utility to control the IEEE 802.1x/WPA/EAP/RADIUS Authenticator.
    • does not install anything else
  • Reboot
  • Log back into LuCI
  • You will now have the option to enable WPS in LuCI
    • (Network-->Wireless-->(radio0/radio1) -->Edit-->Interface Configuration-->Wireless Security--> Enable WPS pushbutton, requires WPA(2)-PSK/WPA3-SAE
    • I use radio1/wlan1/2.4GHz but you might be able to choose both.
  • Tick the box
  • Click 'Save & Apply'
  • Done, WPS is enabled.

Notes

  • WPS works my HH5A with LEDE 17.0.0.4, and just to confirm, yes the button does work with no extra configuration. But i have read that WPS working, can be hit or miss varying from router to router.
  • [OpenWrt Wiki] Wi-Fi /etc/config/wireless - Contains information on configuring WPS, this is really for reference or advanced users.

WIFI On/Off Button (optional)

This is cannot be used use WPS as they use the same button, otherwise you can use this to configure the WPS button for something else such as turning the WIFI on or off.

  • install wifitoggle
    • Very versatile script to toggle Wi-Fi with a button. Allows to set timeouts, persist changes after boot, and set LEDs according to the state.
    • I dont think this installs anything else as I have not tested it.

Notes

uHTTPd GUI for the web server daemon (optional)

You only need this if you want to play around with the web server settings. Most people will not need this.

  • Install luci-app-uhttpd
    • uHTTPd Webserver Configuration
    • This installs nothing else
    • Creates a menu under services called ‘uHTTPd’ (you might need to refresh the page)
  • Change the Server Hostname (Services-->uHTTPd-->uHTTPd Self-signed Certificate Parameters-->Server Hostname) to match your FQDN which will be Hostname + Local domain
    • Hostname (System-->system-->General Settings--> Hostname) (primaryrouter)
    • Local domain (Network-->DHCP and DNS-->Server Settings-->General Settings-->Local domain (quantumwarp.com)
    • Server Name = primaryrouter.quantumwarp.com
    • NB: I am not sure if this needs to be changed, it just seems the right thing to do.
  • The other certificate settings at (Services-->uHTTPd-->uHTTPd Self-signed Certificate Parameters) will probably need to be changed if you were doing things right.

Notes

Force HTTPS

This is a must nowadays, and there are 2 ways of doing it.

  • uHTTPd
    • If you have installed uHTTPd above then goto
      (Services-->uHTTPd-->General Settings-->Redirect all HTTP to HTTPS)
    • Tick the box and apply.
    • NB: Make sure you are logged in via HTTPS otherwise it will fail when using LuCi.
    • NB: if you are changing details then once you change and save the details you must click the Remove configuration for certificate and key button to update the certificate.
  • Command line
    • Run the following commands 
      uci set uhttpd.main.redirect_https=1     # 1 to enable redirect, 0 to disable redirect
uci commit uhttpd
service uhttpd reload

Secure Connection Failed

If you get the following message after you have forced HTTPS then is it most likely a certificate mismatch error and all you have to do is closes and restart your browser.

Notes

  • [OpenWrt Wiki] How to get rid of LuCI HTTPS certificate warnings - Do you like the security of using LuCI-SSL (or Luci-SSL-OpenSSL), but sick of the security warnings your browser gives you because of an invalid certificate? With these instructions, you can generate your own self-signed certificate, which your browser will accept as valid.
  • FireFox does not handle you messing with HTTPS/SSL in LuCi so you might find using chrome easier while setting everything up reguarding this.
    • In FF, becasue the cert is unsafe, FF might not redirect to it. I cannot figure out the pattern, sometimes it works and sometimes it does not. (FF 94.0.2 64-bit).

HTTPS for LuCI (optional)

This is now enabled by default by default OpenWrt 21.02 I will leave this section here as a reference.

This installs https for LuCi, the required SSL libraries and disables http:// for LuCI,

  • Install luci-ssl-openssl
    • LuCI with OpenSSL as the SSL backend (libustream-openssl). OpenSSL cmd tools (openssl-util) are used by uhttpd for SSL key generation instead of the default px5g. (If px5g is installed, uhttpd will prefer that.)
    • Also installs
      • zlib
      • libopenssl
      • openssl-util
      • libustream-openssl

Notes

SSL

SSL is not installed by default OpenWrt 21.02 so you do not need to install SSL or CA-Bundles. I will leave this section here as a reference.

There seems to be a few SSL packages but most people seem to use one which is the package that is used by LuCI for https/SSL (see below).

Basic Open SSL

This will install the minimum to use SSL

  • Install libustream-openssl
    • ustream SSL Library (openssl)
    • Also installs
      • Zlib
      • libopenssl

CA Certificates (optional)

This installs all root certificates as one bundle file and seems to be updated on a regular basis to keep upto date with certificate authority changes. Root certificates are required to validate HTTPS certificates.

  • Install ca-bundle
    • System CA certificates as a bundle
    • This installs nothing else

ca-bundle vs ca-certificates

I do not know why there is ca-bundle and ca-certificates but I believe that ca-bundle is all of the root certificates in one file where as ca-certificates the root certificates are all separate and that some programs require them to be separate. Which ones these are is beyond me and unless otherwise told I will use ca-bundle.

LetsEncrypt (not used)

Control the ACME LetsEncrypt certificate interface. For you that dont know, LetsEncrypt gives out free SSL/TLS certificates.

  • Install luci-app-acme
    • Control the ACME Letsencrypt certificate interface
    • Also installs
      • libmbedtls
      • libcurl
      • luci-app-uhttpd
      • curl
      • netcat
      • ca-bundle
      • acme

Notes

  • Amongst other things, this installs a LuCI add-on for the uHTTPd webserver which is already installed and running for OpenWrt LuCi by default.
  • [OpenWrt Wiki] package: acme
  • How to run on OpenWrt · acmesh-official/acme.sh Wiki · GitHub
    • the instructions from the developer.
    • I dont think yu need to install luci-ssl-openssl anymore since OpenWrt 20.02
    • The developer tells you to force LuCi to be HTTPS, this might be optional and/or you have already done this above.

Adblock

LuCI support for Adblock

  • Install luci-app-adblock
    • LuCI support for Adblock
    • Also installs
      • adblock
    • Creates a menu under services called ‘Adblock’

Notes

  • Blocks advertising at the router by intercepting DNS requests and checking them against a list.
  • This adds a menu item under services called 'Adblock'.
  • Adblock Documentation | GitHub
  • Adblock support thread - Community Builds, Projects & Packages - OpenWrt Forum
  • Adblock [Turris wiki] _ this is a comprehensive tutorial on Adblock.
  • Adblock downloads the lists local and uses these for its lookups rather then doing an online lookup for each domain.
  • the block lists are stored locally after downloading them
  • Updating Block Lists
    • The block lists are updated when the adblock process is started or stopped but you can configure a cron job to do this on a regular basis.
    • To setup a cron job to update the lists
    • Goto (System-->Scheduled Tasks)
    • add the following line to the 'Scheduled Tasks' and click submit 
      # Adblock - Block list update
0 06 * * *    /etc/init.d/adblock reload
      This updates the block list every day @ 06:00
    • Reboot the router. This might not be needed but it is easier just to do it.
    • [OpenWrt Wiki] Scheduling tasks with cron
    • Adblock - Weekdays only - Installing and Using OpenWrt - OpenWrt Forum - This is very useful.
    • See the official documentation for more stuff you can do by the command line and CRON.
  • OpenDNS might do a similar job, but Adblock will check against its own lists aswell before forwarding the DNS request to your chosesn DNS server.
  • You can configure Adblock locally for your own whitelists and blacklists
  • Test your browsers ads blocker - Load this page and see if AdBlocker is working.
  • Adblock lists are all adblock formatted lists.
  • The firewall rules get added to the bottom of your rules, you might need to move them up depending on your setup.
  • DNS based adblock using OpenWRT, OpenDNS, and dnsmasq – Paul Bryan Vreeland - This is a alternative script that will do something similiar.

Dynamic DNS

LuCI Support for Dynamic DNS Client (ddns-scripts). The HH5A has this, but is optional to install.

  • Install luci-app-ddns
    • LuCI Support for Dynamic DNS Client (ddns-scripts)
    • Also installs
      • ddns-scripts
    • This adds a menu item under services ‘Dynamic DNS’

Configuring no-ip.com as a custom DDNS provider

For some reason no-ip.com is not in the list of DDNS providers and by the looks of it people have been using a custom script for no-ip.com. I dont know wether that is still the prefered method by most people.

no-ip.com now offer the standard http/https API method of updating the DNS records with them, so that is my prefered method of updating no-ip.com DNS records.

In (Services-->Dynamic DNS-->myddns_ipv4) configure these settings

These settings work fine and are using normal http.

You need to install SSL to use HTTPS

To use SSL you needs to install the basic SSL package or install https for LuCI will also install the basic SSL package and make the web interface use https at the same time.

If you try and use https for DDNS without installing a suitable SSL package you will get the following error and The DDNS service will also fail to start:

 223214  WARN : uclient-fetch: no HTTPS support! Additional install one of ustream-ssl packages - TERMINATE
 223214  WARN : PID '4597' exit WITH ERROR '1' at 2017-11-23 22:32

The resolution is simple, Install SSL functionality by following the instructions in the sections below.

Configure DDNS to use https

Making DDNS use https is really simple

  • install a suitable SSL package
  • in (Services-->Dynamic DNS-->myddns_ipv4)
    • enable Use HTTP Secure
    • a new box will apear called Path to CA-Certificate
    • You now have to options to enter in
      1. type in IGNORE and SSL will work but will not verify the SSL certificate of the provider. this does add a slight security risk because you do not verify the certificates. It is still better than not using https.
      2. type in /etc/ssl/certs/ and install the ca-bundle package - This installs the root certificates (CA Bundles) OpenWrt can use to verify the SSL certificates that are issued by the DDNS prover. I have not verified this works.

Notes

OpenVPN (not used)

This section is not complete. check out the links to get things working.

Ebilan now has a document on how to do this in his DropBox Repo

LuCI Support for OpenVPN

  • install luci-app-openvpn
    • LuCI Support for OpenVPN
    • installs nothing else
    • creates a menu item unders services called 'OpenVPN
  • install openvpn-openssl
    • Open source VPN solution using OpenSSL
    • Also installs
      • kmod-tun
      • liblzo
  • install openssl-util (optional?)
    • The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) protocols as well as a full-strength general purpose cryptography library. This package contains the OpenSSL command-line utility.
    • Also installs
      • openvpn-easy-rsa
  • install openvpn-easy-rsa – (optional?)
    • Simple shell scripts to manage a Certificate Authority
    • This might not be needed for some configurations.
    • i do
  • Creates a menu under service called ‘OpenVPN’

Notes

  • This might need something else installing to get working
  • OpenVPN on your OpenWrt router has 3 potential modes
    1. Force all local traffic through the VPN
    2. Configured as a VPN server so clients can connect to your network remotely.
    3. Bridge mode so 2 routers bridge their networks. (might be the same as 1 and 2)

Links

ClamAV (not used)

ClamAV is an anti-virus package you can run on the router independant of the clients but it is CPU intensive. This is better suited for higher spec setups such as virtual machines.

  • install luci-app-clamav
    • This package will install ClamAV Web UI.
    • This also installs
      • libmbedtls
      • libcurl
      • uclibcxx
      • clamav
    • This creates a menu item under services called 'ClamAV'
  • install freshclam (optional)
    • Database updater for ClamAV
    • this does not install anything else

Notes

  • This will put a high cpu load on the router.
  • freshclam is a service to keep the definition files upto date

Statistics

Out of the box, OpenWRT can show you real-time statistics, however it will not store the data for historical view. Install luci-app-statistics to have a nice historical data graphs for router, eg: ping, interfaces bandwidth utilization, cpu load, ram, disk, uptime, etc…

 

Step 6 - Secure DNS (optional - but recommended)

This needs its own section because there are several ways of encrpting your DNS request and also there is DNSSEC.

See the different methods below and pick whichs ones are right for you, but my recommendation is:

  • DNS Hijacking (via the Firewall)
    • Requires dnsmasq-full to filter by IPSets.
  • dnsmasq-full
    • Supports DNSSEC
    • Easy to use with Stubby for DoT
    • Closet to dnsmasq in terms of config and current LuCi setup
    • I dont need a full authoriative recursive DNS server (you need Unbound for this)
    • You need dnsmasq-full to do DNSSEC validation locally and you only need to do this if your upstream DNS provider does not do it for you.
    • DNS Hijacking needs dnsmasq-full to filter by IPSets.
  • DoT (dnsmasq and Stubby)
    • IT manages can still control traffic.
    • Doesn't require a full HTTPS stack like DoH
    • more compatible with devices
    • DoT runs on a different port so is easier for traffic management
    • Less resources needed
    • Your local ca-bundles must be upto date (there might an override but this would defeat the object of DoT)
    • dnsmasq supports IPv6
  • Quad9
    • It Supports DNSCrypt, DoH, DoT, DNSSEC
    • I think it validates DNSSEC on it's own server so i dont need to install dnsmasq-full to do this locally.
    • Its purpose is to send NXDOMAIN responses for domains that are flagged as malware/dirty.
  • DNSSEC (dnsmasq)
    • This validated cryptographocaly the domain you are talking it legitimate and not spoofed.
    • The DNSSEC validation is done at Quad9 servers
    • I dont believe I dnsmasq-full for this to work as I was getting verification just using dnsmasq
    • With just dnsmasq I went to intentionally misconfigured domain and Quad9 blocked me as expected.

DNS Hijacking (via the Firewall)

This script allows OpenWrt to intercept all DNS traffic from your local network and send it to your preferred DNS provider. This is useful nowadays because of all of the encrypted DNS traffic from various apps and devices on your network.

Standard DNS and DoT are easy to identify becasue of the port they run on, however for DoH you need to match the domain the packet is being sent to from a known list DoH which is updated regulary via the hotplug.d, you can see in the code where the list is if you want to have a look.

To install DNS Hijacking run the following code blocks from the official documentation [OpenWrt Wiki] DNS hijacking via a SSH command line

  1. Upgrade dnsmasq to dnsmasq-full if you have not already done so
    # Upgrade to dnsmasq full
opkg update
opkg remove dnsmasq
opkg install dnsmasq-full
  2. Command-line instructions - Configure firewall to intercept DNS traffic (The basic rule).
  3. Setup IP set extras
  4. Setup Hotplug extras
  5. NAT6 - Enable NAT6 to process IPv6 traffic when using dual-stack mode.
  6. DNS over HTTPS (DoH) - Filter DoH traffic with firewall and IP sets forcing LAN clients to switch to plain DNS. Set up IP set extras and Hotplug extras to automatically populate IP sets.
    • The script has paced a file in /etc/hotplug.d/online/ so when the router comes on line this script will run.
    • Setup .
    • Now run the code block
  7. DNS over TLS (DoT) - Configure firewall to filter DoT traffic forcing LAN clients to switch to plain DNS.
  8. Reboot your router

Notes

  • Script Notes
    • DoT port is unique matching both IPv4 and IPv6 traffic, so filtering by port works well. DoH uses the same port as HTTPS, so we need to filter by the destination IP address.
    • There are many public DoH servers, and filtering them all efficiently relies on IP sets. Each IP set contains only IPv4 or IPv6 entries, so a couple of sets/rules is necessary.
    • IP set extras to resolve the DoH domains, and Hotplug extras to trigger IP set update at startup.
    • The script downloads a domain list, converts it to ip addresses, create IPv4 and IPv6 block list and then the firewall uses this to block requests so the clients will downgrade to unsecure DNS requests on port 53 which can then be routed as required. But it does required IP set extras and Hotplug extras to be installed/run first.
    • The block list will only update when the router comes on line so basically when you reboot the router. This could potential be altered if you change the location of the 60-ipset-doh script file to another hotplug folder.
    • These scripts will need to be re-run after a firmware upgrade
    • DNS Hijacking Doc - 2 rules for DNS over HTTPS - Talk about Documentation - OpenWrt Forum
    • This script seems to need dnsmasq-full even thought it does not explicity mention it becasue all fo the examples of using those other scripts seem to say upgrade to dnsmasq-full
  • IP set extras
    • [OpenWrt Wiki] IP set extras
    • This instruction extends the functionality of IP sets.
    • Create and populate IP sets with domains, CIDRs and ASNs.
    • Populate IP sets automatically at startup.
    • The automatic script is the same as pasting in the code from the window
  • Hotplug
  • Hotplug extras
  • These are slightly different to the ones from Adblock but seem to be kept up to date.

Tutorials

DNS Hijacking (via Adblock)

Using this feature and OpenDNS (or other DNS provider) adds extra security to your network and prevents rogue uncontrolled DNS requests.

This option adds a rule into the OpenWrt firewall so all DNS requests are forced to the router and if you have set up OpenDNS as a custom DNS server then all DNS requests will be sent to OpenDNS via the router.

  • Goto (Adblock-->Settings-->General Settings-->Force Local DNS)
  • Enable Force Local DNS
  • Forced Zones: lan
  • Forced Ports: 83, 853, 5353
  • Save & Apply

Firewall rules have now been added to accomplish this however the firewall rules are removed when the Adblock service is disabled or you uninstall the adblock package.

From looking at this, these rules will only work for DoT and not DoH traffic.

Keep firewall rules for DNS requests but disable the Adblock service

When you disble the adblock service under (Adblock-->Settings-->General Settings-->Enabled) the firewall which force DNS routing are removed. Sometimes you might not want the Adblock service but really want the DNS funnelling rules. So this is how to keep the rules and have the adblock service disabled. If you really want you can manually add these rules to the firewall but who wants to do that.

  • Install Adblock (if not already)
  • Enable Force Local DNS
  • Goto (System-->Startup)
  • Find the line with adblock
  • Click Stop
  • Click Disabled

Now the Firewall rules are still in place and the adblock service is not running. It will also not restart the service when you reboot your router.

config redirect 'adblock_lan53'
	option name 'Adblock DNS (lan, 53)'
	option src 'lan'
	option proto 'tcp udp'
	option src_dport '53'
	option dest_port '53'
	option target 'DNAT'

config redirect 'adblock_lan853'
	option name 'Adblock DNS (lan, 853)'
	option src 'lan'
	option proto 'tcp udp'
	option src_dport '853'
	option dest_port '853'
	option target 'DNAT'

config redirect 'adblock_lan5353'
	option name 'Adblock DNS (lan, 5353)'
	option src 'lan'
	option proto 'tcp udp'
	option src_dport '5353'
	option dest_port '5353'
	option target 'DNAT'

DNS over TLS (DoT) (dnsmasq and Stubby)

This is very easy to set up and the 2 options I recommend are (both have the same end result:

  • Pick a method
  • Set your own DoT DNS provider to use (eg: OpenDns/Quad9)
    • edit the resolvers in /etc/config/stubby
    • If your ISP does not provide IPv6 then you must comment out or remove the IPv6 resovlers otherwise this will just generate errors.
    • I removed the CloudFlare resolvers as I only wanted to use Quad9.
  • Further Stubby configuration (from README.md)
    • Set dnsmasq to send DNS requests to stubby
      • The sets up the proxy arrangement but is already done by the script we ran earlier.
    • Disable sending DNS requests to ISP provided DNS servers
      • This changes your WAN connections to route DNS locally.
      • If using a DoT/Stubby setup: When you do a router upgrade you will temporarily change your WAN/WAN6 DNS settings to 9.9.9.9/2620:fe::fe while you install the required packages because without this change your router will not be able to talk to the internet for this process.
    • Enabling DNSSEC (Optional)
      • This is a bit misleading.
      • This should say enable local DNSSEC validation.
      • Use DNSSEC by dnsmasq
      • This is only required if your upstream resolver does not perfomr DNSSEC validation.
      • Quad9 does DNSSEC validation upstream and if it faills will return a NXDOMAIN result, however if there is no DNSSEC crypto for that domain, DNS lookup will occur as normal.
      • The setting (Network --> DHCP and DNS --> Advanced Settings -->DNSSEC) should be left unticked.
      • The setting (Network --> DHCP and DNS --> Advanced Settings -->DNSSEC check unsigned) is a related setting and is on by default.
        • Requires upstream supports DNSSEC; verify unsigned domain responses really come from unsigned domains
        • See DNSSEC support in OpenWrt 15.05 Chaos Calmer | falstaff - yet another tech blog
          There is one problem thought: If an attacker forges an answer and removes its DNSSEC records, dnsmasq silently accepts them. The dnsseccheckunsigned option avoids that by asking the upstream servers whether the domain in question really does not support DNSSEC. Hence this leads to additional DNS requests for all non DNSSEC domains… Well, security comes not for free
        • Should this be on or off?
  • Stubby Config File - Additional configurations you should set after going through the config file
    • set option triggerdelay '5' to prevent some stubby errors in your syslog.
      • If you dont have a delay on stubby getting triggered you get many of the following errrors 
        Sat Nov 27 17:24:00 2021 daemon.err stubby[3738]: Could not schedule query: None of the configured upstreams could be used to send queries on the specified transports 
      • This is caused purely becasue whent he WAN comes up it does not instantly have an internet connection availbe but stubby thinks it does and starts doing DNS requests. To stop  this we just add a delay between WAN coming up and Stubby doing DNS requests.
      • Depending on your router you might need to increase thjis a little but for mine this setting is good. 
        WAN Up --> Stubby Up --> No internet --> Errors
WAN Up --> Pause 4 seconds --> Stubby Up --> Internet --> No Errors
  • Complete the TLS1.3 setup
    These options can be overriden on a per-resolver basis but I dont have a need for that.
    • Force the newer TLS1.3 cipher standards by uncommenting the following line. Please note 
      option tls_cipher_list 'EECDH+AESGCM:EECDH+CHACHA20'
option tls_ciphersuites 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256'
      Please note that the following line is for TLS1.2 only and should be left commented out aws it is not needed.
      # option tls_cipher_list 'EECDH+AESGCM:EECDH+CHACHA20'
    • Set option tls_min_version '1.3'
      • This ensures no downgrade to TLS1.2
  • Remove CloudFlare Resolver sections
    • You can have as many resolvers as you want but I only want to use Quad9
  • Backup stubby YAML config files (optional)
    • This does not seem to be required becasue Stubby is using the UCI system
    • It says in the official doc to "You might want to add /etc/stubby/ to the list of config files that should be preserved on upgrade / backup! ", unless you use the .yaml files in that folder dont bother as all settings used are in /etc/config/stubby.
    • This is done in (System --> Backup / Flash firmware --> Configuration)
  • Reboot your router
  • Test your new DNS system (see links below)

DNS over TLS (DoT) (Unbound and odhcpd)

DNS over HTTPS (DoH) (dnsmasq and https-dns-proxy)

DNSSEC (dnsmasq-full)

OpentWrt comes with the small version of dnsmasq which is fine for most basic operations but DNSSec requires the full package to be installed.

It should be possible to run DNSSec and DNSCrypt if the target DNS server supports both of these technologies.

  • uninstall dnsmasq
    • It is intended to provide coupled DNS and DHCP service to a LAN.
  • install dnsmasq-full
    • It is intended to provide coupled DNS and DHCP service to a LAN. This is a fully configurable variant with DHCPv6, DNSSEC, Authoritative DNS and IPset, Conntrack support & NO_ID enabled by default.
    • Also installs
      • kmod-nfnetlink
      • libmnl
      • libgmp
      • libnettle
      • libnfnetlink
      • kmod-nf-conntrack-netlink
      • libnetfilter-conntrack
      • kmod-ipt-ipset

You probably will receive the follow error:

Collected errors:
 * resolve_conffiles: Existing conffile /etc/config/dhcp is different from the conffile in the new package. The new conffile will be placed at /etc/config/dhcp-opkg.

Compare the config files

You should note there might be more that one service using the dhcp config file.

  • Text compare the 2 files for obvious settings that need to be copied. Always keep your orginal dhcp file and move what is missing from dhcp-opkg (optional)
  • or you can freshly load the page (Network-->DHCP and DNS) after upgrading to dnsmasq-full and then 'save and apply' the settings thus creating any missing setting that needs to be added (i am guessing here).
  • the dnsmasq-full dhcp config file is almost the same as dnsmasq dhcp config file. dnsmasq-full dhcp config file seems messy. and when you save 'DHCP and DNS' settings in LuCI with no changes it encapsulates all of the settings in apostrophes which mean the original dsnmasq dhcp file format is correct. This most likely means that you should use you orginal dsnmasq dhcp config file. When you enable additional settings then they will be recorded in the config file so any that were possibluy missing will appear as required, this is probably how things worked even before the upgrade.
  • Delete the dhcp-opkg (or save it somewhere but do not leave it on the router)

Enable DNSSec

DNSSec is not enabled by default so will need to be turned on.

  • (Network-->DHCP and DNS-->Server Settings-->Advanced Settings-->DNSSEC) = enabled
  • (Network-->DHCP and DNS-->Server Settings-->Advanced Settings-->DNSSEC check unsigned) = enabled (optional)
    • is slower but more secure, see notes.
    • Requires upstream supports DNSSEC; verify unsigned domain responses really come from unsigned domains

DNSCrypt (Dnsmasq and dnscrypt-proxy) (not used)

dnscrypt-proxy is an application that acts as a local DNS stub resolver using DNSCrypt. It encrypts your DNS traffic improving security and privacy. dnscrypt-proxy is the client-side version of dnscrypt-wrapper. Follow DNSCrypt with Dnsmasq and dnscrypt-proxy to properly setup DNSCrypt via dnscrypt-proxy on your router.

  • This is a new way to get your DNS quries. It is not a web standard yet nor is it DNSSEC
  • Install luci-app-dnscrypt-proxy
    • dnscrypt-proxy provides local service which can be used directly as your local resolver or as a DNS forwarder, encrypting and authenticating requests using the DNSCrypt protocol and passing them to an upstream server. The DNSCrypt protocol uses high-speed high-security elliptic-curve cryptography and is very similar to DNSCurve, but focuses on securing communications between a client and its first-level resolver.
    • Also installs
      • ibsodium
      • dnscrypt-proxy-resolvers
      • dnscrypt-proxy

Notes

Step 7 - Final Configuration

In this section we just make changes that are appropriate here.

My ISP is not providing IPv6 and I am getting odhcp6c errors in the syslog

This error appears in your system error log. It will not cause any real issues but explanantions and solutions are below.

Mon Nov 29 15:56:12 2021 daemon.err odhcp6c[1363]: Failed to send SOLICIT message to ff02::1:2 (Permission denied)
Mon Nov 29 15:56:13 2021 daemon.err odhcp6c[1363]: Failed to send RS (Permission denied)

Cause

  • odhcp6c is the DHCPv6 client daemon. It is likely trying to get dhcpv6 address for WAN from ISP dhcpv6 server. (for WAN interface, the outside network is the link-local context. And that fails as long as the WAN interface has no connectivity.
  • ff02::1:2 is an IPv6 address that is used to send all messages to DHCP servers and relay agents on the local network segment (defined in RFC 3315). IPv6 DHCP works differently to IPv4.
  • WAN6 has dhcpv6 as the address protocol, so that your WAN port can get an IPv6 address from the ISP's DHCPv6 server (if there is such).
  • My provider does not offer IPv6 so this DHCPv6 request will always fail and potential keep retrying.
  • Once the WAN6 interface is started when configured as a DHCPv6 client, even if you stop it, or it appears stopped, the odhcp6c daemon is still running.

Solutions

Pick one of the solutions below.

  1. Get an ISP that provides IPv6
  2. Prevent the WAN6 interface from ever starting. (Preferred)
    • (Network --> Interfaces --> WAN6 --> Edit --> Bring up on boot) = unticked
    • This method has the benefit of leaving the interface intact should you ever want to quickly enable IPv6.
  3. Delete the WAN6 interface (not tried this but a few people have said this works)
  4. Disable IPv6 on your whole router (not tried this but seems overkill)

Links

Configure 802.11r Fast Transition (Fast Roaming)

This is only useful if you have mulltiple routers running AP (Acccess Points) on the same SSID.

Notes

  • The full version of wpad is installed. It should of been done earlier.
  • All of your APs will need to use the same SSID, encryption, and passwords
  • Repeat for each AP on this router (Openwrt / Openwrt_5g) you want to offer fast roam.
  • Do the same for any other AP on your network where you want Fast Roaming.

Method 1 - Easy

 

  • Goto (Network --> Wireless --> 'Your AP' (Openwrt/Openwrt_5g) --> Edit --> Interface Configuration)
  • Click the check box for 802.11r Fast Transition
  • Click 'Save'
  • Click 'Save & Apply'

This is based on a post from @Mushoz on the OpenWrt forum here Fast Roaming 802.11r - Multiple SSIDs = different mobility domains? | OpenWrt Forum where he says the Mobility Domain is created automatically by hashing the SSID.

 

Method 2 - Manual (Van Tech Corner)

This is a very useful video from Van Tech Corner OpenWRT - Configure 802.11r Fast Transition - Fast Roaming Wifi - YouTube

  • Goto (Network --> Wireless --> 'Your AP' (Openwrt/Openwrt_5g) --> Edit --> Interface Configuration)
  • Click the check box for 802.11r Fast Transition ans some more options will appear
  • NAS ID
    • This needs to be unique for each Router/AP
    • I just use the router's hostname (i.e. officerouter)
  • Mobility Domain
    • 4-character hexadecimal ID  (0-9 and A-F)
    • This will be the same for all of your APs that share the same SSID
    • I will use this for both (Openwrt / Openwrt_5g) because they are on different frequencies. If the APs have the same SSID but are on diffferent frequencies this rule should also apply.
  • Leave the other settings as they are.
  • Click 'Save'
  • Click 'Save & Apply'

 

Step 8 - Backup settings

After configuring the router you will want to backup your settings

  • (System-->Backup / Flash Firmware-->Backup / Restore-->Download Backup)
  • Get the backup of your configuration settings and store safely on another device.
  • This seems to include all of the /etc/ folder and in particular the /etc/config/ folder

Notes

  • When you make a backup it basically makes a copy of the /etc/ which includes all settings.
  • A backup does not contain downloaded software packages
  • When you restore a backup you overwrite everything in the /etc/ folder with that in the backup archive, but no files are deleted, just replaced.

 

Upgrading OpenWrt

Do not upgrade over WIFI and read How to update OpenWrt (ebilan 4.3)

It is ok just to use the basic LuCi upgrade procedure for minor versions, but you will probably find with major verion updates there are syntax differences within the config files and these need to be manually resolved. Always use the new config files as the base and move your settings into them.

When merging move the settings you understand and for those that you are not sure about go and find them in the LuCi GUI and see what they match up to.

  • If using a DoT/Stubby setup: When you do a router upgrade you will temporarily change your WAN/WAN6 DNS settings to 9.9.9.9/2620:fe::fe while you install the required packages because without this change your router will not be able to talk to the internet for this process.
  • Custom scripts might need to be re-installed eg: IP set extras and Hotplug extras required for DNS Hijacking (via the Firewall) (i am not 100% sure if this is the case for these scripts).

Basic

It should be noted that this method (on it's own) will require you to re-install all of the 3rd party packages manually for the new firware to be the same.

  • Goto (System-->Backup / Flash Firmware-->Flash new firware image)
  • Select a compatible firmware image
  • Keep settings: ticked (unless you want to start fresh)
  • Click ‘Flash Image’
  • Reinstall user-installed packages. (optional)
    • After a successful upgrade, you will need to reinstall all previously installed packages according to your notes.
    • Package configuration files should have been preserved due to steps above, but not the actual packages themselves. If you used the scripts provided in the forum, this step might not be necessary.
    • The new package installations will have installed new, default versions of package configuration files.

sysupgrade (Manual) (Keeping 3rd Party Packages) (20.00+)

Manually removed/installed packages are not preserved by default so we must make note of them before upgrading.

Not Added To Archive

  • Removed packages are not recognised in the installed_packages.txt currently and will still be present after restoration.
  • Service startup status are not maintained either.
    • An example is that this will affect people who disable the dsl_control service to prevent the restarting bug.
    • Automatic statirng of a script is based on the presence of a symbolic link to it in the folder /etc/rc.d
    • The startup scripts are stored in /etc/ini.d
    • It would be nice if this was preserved in the backup archives.
    • [OpenWrt Wiki] Init Scripts

These are based on official instructions [OpenWrt Wiki] Backup and restore - Preserving packages and the Luiz Angelo Daros de Luca method shown below.

installed_packages.txt issue in recent OpenWrt21.02.1 release

There has been a change and for some reason I cannot write to the /etc/backup/ folder, however the installed_packages.txt if enabled in the config backup will still be present in the backup. It might be this folder is now getting purged automatically. I have made some adjusments to the code below but have left the old stuff there just incase this is just a bug in 21.02.1 - It should be noted I can still place files in this folder with SFTP and that if i place installed_packages.txt this does not get overwritten but the backup folder in the archive now has 2 installed_packages.txt indicating the file is created elsewhere. This might be an issue with the /overlay and /rom partitions.

OpenWrt Bug Report: FS#4048 : sysupgrade with "-k"

For the instructions to work below you must

  • make sure the installed_packages.txt is included in the backup configuration.
  • output the file to the config directory and dont delete it sysupgrade -k -u -b /etc/config/deleteme.tar.gz
  • FTP or otherwise transfer the backup archive from your router
  • The installed_packages.txt will be correctly present at /etc/backup/installed_packages.txt
  • The normal backup archive will NOT have the installed_packages.txt present.
  • SSH into your router and run the following commands. I have remmed out the delete file line for now as it is more useful to keep it.
    sysupgrade -k -u -b /etc/backup/deleteme.tar.gz
# rm /etc/backup/deleteme.tar.gz
    • These will create a file located at /etc/backup/installed_packages.txt
    • This file will NOT be included by default in a config backup archive, but during a system upgrade it will be preserved.
    • You can also use this command to get a copy of all of your installed packages. It create a single list but tags the packages as in the /rom(in the firmware) or /overlay(user installed).
    • This is a bit of a hack to use this script as you have to create a backup to generate the installed_packages.txt file so we obviously need to delete the backuparchive when our file has been created.
    • Lastly this will overwrite any installed_packages.txt present.
    • I run this before creating a config backup archive.
  • (Optional) Add installed_packages.txt into the config backup archive.
    • Go to (Backup / Flash Firmware --> Configuration)
    • In the list add this line at the end
      /etc/backup/installed_packages.txt
    • Click Save
    • This is useful if you use the same config archive on multiple routers of the same model (MAC addresses might need changing)
  • Now run the Basic instructions above to update your firmware.
  • SSH into your router
    • Rrun the following commands which will install only the packages from the overlay partition and ingnores the ones in the rom partition.
      You can just open the installed_packages.txt and manually install the files you find in the list using the software manager.
      opkg update
grep "\toverlay" /etc/backup/installed_packages.txt | cut -f1 | xargs -r opkg install
rm /etc/backup/installed_packages.txt
reboot
    • There also might be conflicts with packages already installed.
      Collected errors:
 * check_conflicts_for: The following packages conflict with wpad-wolfssl:
 * check_conflicts_for:         wpad-basic-wolfssl *
 * opkg_install_cmd: Cannot install package wpad-wolfssl.
xargs: opkg: exited with status 255; aborting
root@officerouter:~# rm /etc/backup/installed_packages.txt
root@officerouter:~#
      • The opkg command will not remove them so you have to remove them manually. I would recommend using the LuCI (the GUI) for this.
      • So in the example above, you need to remove wpad-basic-wolfssl and then install wpad-wolfssl
      • Once done, re-run the opkg script above command above to finish any missed updates.
      • Reboot the router
    • If when a package is installed and the new config file does not match with the one in your config files, opkg will create a second config file eg: 
      adblock
adblock-opkg

      These will need to be compared with a merging program and you will need to make manual changes where needed. Try and keep the layout of the new config file

  • Reboot your router
  • Done !!!

opkg method (Manual) (Keeping 3rd Party Packages) (18.06+)

  • SSH in to your router
  • run the following command
    opkg list-installed | cut -f 1 -d ' ' > /etc/config/packages.list
    This will create a file called packages.list that will survive a flash when you keep the settings.
  • Backup your firmware config to your PC
    • (System-->Backup-->Backup-->Generate archive)
    • This will include the packages.list file in the archive.
    • After upgrade to 20.00+ you should delete this file as it is no longer needed and you should use installed_packages.txt instead.
  • Flash the new firmware using the Basic method.
    • After reading any notes from OpenWrt (i.e. dont skip to many versions)
    • Make sure you keep the settings
  • SSH in to your router
  • run the following command
    cat /etc/config/packages.list | opkg install

    This will now install the missing 3rd party packages.

  • Package Configurations
    • As your existing configuration files were already in place, opkg would have displayed a warning about this and saved the new configuration file versions under *-opkg filenames.
    • The new package-provided configuration files should be compared with your older customized files to merge in any new options or changes of syntax in these files.

Notes

  • I got these instructions from What happens to installed packages on a sysupgrade in OpenWrt? | Newbedev
  • Assumptions (opkg and packages.list)
    • if in the list it will install them
    • if the package is already installed it will skip to the next one and not overwrite what is already present
    • any dependencies that are needed for a package will be installed
    • if not present does it remove them?
  • You can make a list of just the packages you know you have installed (or want to install) and just use that by manually adding the file /etc/config/packages.list afterwards.
  • Create a list of user installed packages only
    • Currenltly the packages.list includes all installed, user ones (/overlay) and those already present in the firmware (/rom). To get a list of just user installed apcakges, do the following:
    • Generate a packages.list on your current setup
    • Factory reset the router.
    • Generate a packages.list on this vigin router.
    • compare the two package.list and remove all of the packages that are duplicated and this will give you a list of user installed packages.

Luiz Angelo Daros de Luca Method (20.00+)

This is a another method that I have come across and is developed by the person responsible for sysupgrade (I think)

sysupgrade -o -k -u openwrt-new-version.img
<auto reboot>
opkg update
grep "\toverlay" /etc/backup/installed_packages.txt | cut -f1 | xargs -r opkg install
rm /etc/backup/installed_packages.txt
reboot

Notes

Install Config archive on another OpenWrt router

When you have anoter router OpenWrt router on which you did not create the config backup on but want that same setup on there are a few things you need to do. I will use my setups as an example which might only need a few tweaks for yours, but you will get the idea.

  1. Upload and apply the config backup (from your first router)
  2. (Optional) Change root password to match the one on the new router's case (System-->Administration)
  3. (Optional) Change router Hostname (System-->System-->General Settings)
  4. (Network-->DHCP and DNS-->Server Settings)
    • (Optional) Change the Local server (Default is /lan/)
    • (Optional) Change the Local domain (Default is openwrt)
  5. (Network-->Inerfaces-->Devices)
    • remove the overriden MAC addresses. They can easily be seen because they are in bold and when you hover over them OpenWrt tells you.
  6. Setup your internet connection, one of these methods will apply:
    • Setup your DSL connection to get internet / Change the username and password on the interface.
    • Change the wireless client network to match new parent SSID if required
  7. Check you have internet with diagnostics (Network-->Diagnostics)
  8. (System-->Software)
    1. Update the software lists
    2. Remove specific packages that will cause issues becasue sysupgrade does not currently remove packages for you and some packages need to be remove for others to be installed.
      • primaryrouter: wpad-basic-wolfssl, dnsmasq
      • officerouter: wpad-basic-wolfssl
    3. (Optional) Install openssh-sftp-server
  9. Check installed_packages.txt is in /etc/backup and if not put it there or in /etc/config
  10. Run the following command via SSH 
    grep "\toverlay" /etc/backup/installed_packages.txt | cut -f1 | xargs -r opkg install
  11. Remove and *-opkg config files in /etc/config because they are not needed.
  12. (Optional) Change Server Hostname (Services-->uHTTPd-->General Settings)
  13. Remove old certificate and key (Services-->uHTTPd-->General Settings)
  14. Regenerate Dropbear keys - Not sure how to do this.
  15. Remove any packages you do not want on this new router. My example is tcpdump becasue i want to monitor network traffic but my mate does not.
  16. Remove any additional Firewall rules you had that dont apply to the new router's role.
  17. Reboot
  18. Backup new config and put it on file

General Notes

Upgrade All Installed Packages

Upgrading all packages without updating the firmware of the router is something you possibly want, so here it is:

 

Other Configurations, Features and Things

Stuff that was not covered above will be here.

Resetting the Router

What Happens

  • Resetting the router removes all of the user settings and downloaded packages by putting the router back to how it was when you first flashed it.
  • It will keep all of the system information like mac addresses.
  • It basically wipes the /etc/ folder

Notes

Try a new Add-on

When you uninstall a piece of software, this will not remove the dependencies automatically (OpenWrt should add a dependencies register to fix this). If you want to try out a new add-on, do the following so if the add-on installs a load of stuff you do not want you can remove the dependencies and revert back to your old settings.

Installing

  • make a backup of your settings configuration
  • install the Add-on
  • make a list of the dependencies installed from the log notifcations (at the top of the page) that appears after the add-on has been installed. Only dependencies that are not installed but required will be downloaded and installed. Add-ons can share packages

Removing

  • uninstall the LuCI app (if you used one)
  • remove the dependencies that were installed
  • delete the relevant settings in /etc/config/ (optional). these can be left for re-installation.

Failsafe Mode

Failsafe mode is where you can access OpenWrt Console/Linux Kerel via SSH or a serial connection and is used for those times OpenWrt will not boot up.

To access the Failsafe mode by SSH

  • when the router is booting up (Flashing Green LED), Press any button on the router to interrupt.
  • The OpenWrt normal boot procedure is interupted and the cut down console over SSH will become available on 192.168.1.11
  • You can now SSH into the router. Use the username 'root' and no password is required.

Notes

Config file conflicts

Config file conflicts can happen at various times but in this instance it was when I had done the following

  • Factory reset my router
  • Uploaded my config back
  • installed a dynamic DNS script luci-app-ddns which also installs ddns-scripts

I then received the following error:

Collected errors:
* resolve_conffiles: Existing conffile /etc/config/ddns is different from the conffile in the new package. The new conffile will be placed at /etc/config/ddns-opkg.

What this error means is that when i downloaded luci-app-ddns, OpenWrt discovered I already had a config file called ddns so it just renamed the new/default config file that came with the package to ddns-opkg so my original config file that had been restored with my backup was unaffected and I still had the opportunity to examine the new config file to see if there were any changes.

You can delete the file ddns-opkg as it is not used. You might want to just look in it to see if there any changes you need to know about.

VLAN

Notes

My Security Options Overview

I will briefly outline the security settings I have added to my OpenWrt HH5A setup. This might change over time.

  • Strong password for root
  • adblock - Advert blocking. List updated automatically
  • DNS Hijacking
  • DoT (via Stubby)
  • Quad9 used for the DNS provider
  • SSH console only allowed to be accessed from the LAN (WIFI and Ethernet)

Notes

 

Questions

I have not verified the answers to these so any feedback is welcome.

  • Can I leave the wires in place to access the serial, or are they no longer required. = does not seem to make much difference.
  • Is the config file specific to a router i.e. because of MAC addresses = I thinks so.
  • Where does OpenWrt store MAC addresses = in /etc/config/network and then in the image which is used for a factory reset (squashfs only).
  • What happens when I press the reset button on my HH5A OpenWrt router

General Notes

  • LuCi is the GUI for OpenWrt
  • LAN referrers to LAN and WIFI clients

Additional Settings

These settings here are useful ones I have come across but do not form part of the intial configuration of OpenWrt for the HH5A.

Wireless Isolation

This is not available for configuration in LuCI and the correct format seems to be option isolate '1'

Prevent Windows adding a new network every time the OpenWRT router is rebooted

Notes

General

Flashing Guides / Tutorials

Documentation / Configuration

Tech Specs

Websites of note

Forums

BT HomeHub stuck in CFG04 (UART) mode after installation

Logging

The UCI system

  • [OpenWrt Wiki] The UCI system - The abbreviation UCI stands for Unified Configuration Interface, and is a system to centralize the configuration of OpenWrt services.
  • UCI command usage - Teltonika Networks Wiki - Unified Configuration Interface (UCI) is a small utility written in C (a shell script-wrapper is available as well) and is intended to centralize the whole configuration of a device running on OpenWrt. This is a full list of information.

Questions

  • When you save settings in LuCi, does this reformat the related config files so they are neater?

 

Published in DSL / Broadband
Read more...
Sunday, 02 July 2017 11:18

How to build a BGA profile from scratch

These instructions do not cover the basic usage of the scotle/BGA Rework machine and how to program it etc.. but shows you have to make the perfect profile that you can use. Profiles are very specific to your machine and environment but there is a basic method that can be used on how to make these profiles that I have worked out and I will now show you.

My scotle is a 3 zone BGA rework machine however these instructions will work for a normal 2 zone machine. All you have to do is skip the lower HR zone bit. Also if you have an upper HR machine these instructions will work the same.

So the 3 temperatures we will need to figure out are

  • Lower IR
  • Lower HR (optional)
  • Upper IR/HR

Splifire mods recommending flatlining your machine, which basically is figuring out what your preheat ranges and temperature settings should be (at the point your top heater is initiated)

  • At chip/top of board – 160c to 170c
  • Bottom of board – 170c to 180c (200c is absolute max and should be avoided)
  • Preheating should usually take about 7 – 8 minutes. Colder climates might take longer

Other settings

  • Dark IR (IR) – ramp rate should always be 1.00
  • Hot Air (HR) – ramp rate should always be 3.00
  • The IR head should be 2.0cm to 2.5cm away from the target chip (about the height of a standard capacitor)
  • The upper HR head should be almost touching the chip, but make sure there is a gap so the air can escape.
  • The lower HR head should be ¼ inchi or 6.25mm away from the boards

So firstly we will prep out our machine

  • Rack a spare board (I used an old xbox 360 board)
  • Build the copperwire probe mod with an external temp sensor (I used TM-902c)
  • Put the copper probe under the middle’ish of the board (as per video) – A good green bit to prevent thermal welling.
  • Put an temp probe (small type ie.e the scotle one) under the target chip

Notes for testing

  • Try and do all of these tests/trials at the same time so you have the same room temperature as this can affect things
  • After each run make sure that you allow you BGA machine to cool back to the ambient temperature (same as it was for the first run) this gives you the same starting point
  • The readings from the TM-902c are always changing. I think these devices are very accurate and do not average the result before it is put on screen like some temperature sensors so the temperature shown is always changing within a range. So the result I write with this device are going to be higher than the average at that time. This effect could also be caused by the pulse nature of the lower IR plates, I would have to check this.
    • You can use a K Type USB logger to draw a graph which will give you much better results and easier to interpret and should be less erratice
    • You can use a temperature sensor device that shows average results (multimeters usually do this or the one in the spitfiremod videos {add model here)

Lower IR

  • Run each the scotle at each of the defined Lower IR values below, not forgetting to cool down before each run so you have the same baseline for each run.
  • As you are running note down the time taken to reach each of the lower board temperatures.
  • I run these for 20 minutes so I can definitely see the max temperature reached. Most reflowing/reballing should not take 20 minutes.

NB: johnnyx method usesthe scotle main probe on the top of the motherboard for the temperature (of 180c) and sets the upper IR to 0c long dwell, he also sets the lower HR to match the lower IR and then gets the temperature needed. This puts at risk the board as you do not know what temperature the lower componenets are exposed to. This method is probably ok for naked boards like the xbox 360.

Instructions

  • Set the first temperateure (180c) onto the lower preheat panel.
  • Get your stop watch ready
  • Check you temperature sensors are on
  • Turn on the preheat and start the stop watch
  • Fill in your table as things progress

You could use the profiles for the Upper IR/HR and Lower HR and set them to 0c and a large dwell time so you could use the inbuilt temperature probe, but the reason I have not is that the Lower HR on mine will always kick out air and if it is not heated it is cold which could affect my profiling. By getting just the IR preheated temperatures first I can find out what is safe.

NB - sort

  • I would not go above this because there is a +-10c shift in the scotle preheater and anything over 200c is dangerous. However if you are careful you might go higher when needed.
  • Try probing at different areas to see how much difference there is, this is especially true ig you are using a lower HR and you can probably afford the temperature to be lower in the tareget temp ie.e. 170c because when you use the lower HR it will compensate for this.
  • The larger plates of the scolte might be a lot hotter than the middle little plates.
  • The splitfire guy said it should come to temp 7-8 minutes, so if it takes ages you might need to notch up the heat a little bit (5c steps)
  • Times taken to reach bottom board temperature just by IR plates on my 3 zone sctole
  • Probe is in the middle’ish and on an unsurrounded green area, right in the middle of the plates. Add picture
  • For those who have notice that the lower HR (if you have one) is not turned on this is on purpose. I want to find out the temperatures developed by the lower platesd first. Yes there will be a lower temperature above the lower HR section.
  • The TM-902c is very flicky, not the main probe. My results are when that temp is first hit, not average.
  • The logger would give a better profile because it is using a graph and probably averages it mnore.
  • These results are just for safety and part of the HR profile ? not sure about this statement
  • The lower IR heaters are only on or off and uses pulsing to get the correct temperature? So this means with the heater curve is similar until required temperature is reached as the plates are either on or off. – this might not be true
  • There is some natural variability in my results because I am not using 100% lab conditions

 

Now

A table showing when the lower board Temperature Reached was reached (mins) and what the Lower IR presheater were set at.

Room:     c
Board:     c

Lower IR Temp

180

185

190

195

200

210

220

225

Lower Board Temperature

                  

140

                  

145

                  

150

                  

155

                  

160

                  

165

                  

170

                  

175

                  

180

                  

185

                  

190

                  

195

                  

200

                  

205

                  

210

                  

215

                  

NB: make a note of the room temperature and lower board temp before you start.

My actual results

Room: 20c
Board: 23c

Lower IR Temp

180

185

190

195

200

210

220

220

2nd

225

225
2nd

Lower Board Temperature

                      

140

  

3:57

3:24

3:45

3:50

2:32

3:38

2:88

2:24

2:20

3:27

145

  

4:08

3:45

3:52

4:05

3:48

3:46

2:35

2:27

2:24

3:38

150

  

5:08

3:55

5:04

4:59

3:54

3:50

2:42

2:45

2:27

3:48

155

  

6:34

6:12

6:13

6:16

4:08

4:00

3:48

3:48

2:37

4:47

160

  

7:50

6:23

7:39

6:22

5:11

5:11

5:04

3:53

3:53

4:55

165

  

11:34

7:39

8:53

8:44

6:28

5:22

5:24

5:17

3:59

5:08

170

  

13:15

8:51

11:30

9:04

7:37

6:56

6:27

5:20

5:09

5:16

175

  

18:14

10:36

13:51

11:22

9:09

8:02

6:30

5:42

5:22

5:21

180

  

-

15:00

14:00

13:08

10:31

10:58

6:38

6:56

5:42

7:47

185

  

-

-

-

17:14

15:51

13:32

6:42

7:07

9:19

9:56

190

  

-

-

-

-

19:11

16:21

11:24

10:19

9:34

11:15

195

  

-

-

-

-

-

-

12:28

12:18

12:48

12:25

200

  

-

-

-

-

-

-

15:17

15:48

17:44

14:09

205

  

-

-

-

-

-

-

-

17:40

18:55

15.28

210

  

-

-

-

-

-

-

-

-

-

18:34

215

  

-

-

-

-

-

-

-

-

-

-

Notes -

  • If the temp mod is done, temp results would be more constent
  • The lower HR nozxle size can affect temp (when dealing with lower HR config)
  • Your preheat is what compensates for your environment amnd machine for the most part. Once your profile is configures olny the preheat section (time and temp) would need to be changed depending on the environment and possibly the board you are working on
  • Pick a run and then set the top IR to off
  • The board above larger plates might get hotter
  • These results would be better from a graph
  • Try an use the lowest IR temperature to get you to the desired temps


 

Choose you lower IR temperature

You results are now in. We are looking for a Lower board temperature of 170c/180c in about 7 – 8 minutes. Now this is a guideline so If yours are slightly out don’t worry too much. If you are using a 3 zone machine you can get a away with 170c because your 3 zone (lower HR) will more than compensate for this.

When looking at the table you can see that all of the runs have the same sort of profile, they heat up rapidly and the slow right down when they get to about 75% max temp of the run, they then eventually hit the max temp and stay there.

My results above are the times when the temperature on my TM-902C reached that value. This is most likely at the top of the range of the actual temperature. I on avergage so a range on about +-10c so I would know off 5c off the tempratures reached and then use those. This might not be the case for the temperature senor you use especially if the result on screen is steady.

This means I will use 210c as it reaches 170c @ 7:37 mins. It should be noted that the max temperature for this run is 210c which is 10 degrees below the dangerous 200c.

On a warm day I might use 200c. It should also be noted I am in the north of England and it is cold,and I am using the scolte unmodified (i.e. the spitfire lower IR temp mod)


 

Lower HR profile(preheat only configuration) – if present

Before I start this section I need to tell you what a 3rd zone or lower HR heater is for. If you don’t have one you can skip this section

  • With this 3rd zone so you can deliver high tempreatures to a single chip like the playstation CPU where there needs to be a lot of heat delivered to the CPU for a successful lift but you do not want this heat going into the other components on the board because they will become damaged. These chips usually will have their own Lower HR profile which is not covered here, but I suspect it would potentially match the Upper IR/HR profile
  • We can also use the 3rd zone HR as part of our preheat process where is just set at a single temperature like the lower plates do the correct temperature is reached on the top of the board above the HR unit. This is used for smaller chips were a specific profile is not needed, preheating is sufficient. (this is what we are going to do here)

In our tests in the above section (Lower IR) we did not have the lower HR on and if we had tested the top board temperature above the HR unit, i.e. under the target chip we would have found it to be a lot lower than the rest of the board. (Literally the cold HR unit was in the way and there are no lower IR plates delivering heat to that part of the board, or definitely a greatly reduced amount.)

With this section we will use our choosen lower IR plates temp (200c) and try to findout what lower HR temperature setting will deliver 170/180c to the top of the board under the chip at the 7 – 8 minutes mark.

Instuctions

The easyway is just to set your lower HR to the same temp as lower IR plates with a ramp rate of 1

  • Set the lower IR preheat to 200c
  • Configure the Upper IR/HR to the following settings (this basically disables the upper unit)
    • L1 – 0c
    • R1 – 1.00
    • D1 – 2000
  • Configure the Lower HR
    • L1 – 180c
    • R1 – 1.00
    • D1 – 2000
  • Make sure you temperature sensors are on and your stop watch is ready
  • Start the profile and stop watch
  • Fill in the table as you go.
  • Run the profile for 20min to see what happens and what tempreatures are reached.

Run these instructions for each Lower HR temperature

Notes

  • We are looking for the top temp. we will use the main probe for this
  • Configure a preheat sgment on the upper IR i.e. mega long at 0c
  • Configure the lower IR profile to the required temp with a long time
  • R1 or R3?

A table showing when the upper board Temperature Reached was reached @ (mins) and what the Lower HR was set at. I have also added actual lower board temperature

Room:            c
Board:            c
Lower IR        c

Lower HR Temp

180

185

190

195

200

 210 220 225

Upper Board Temperature

                  

140

                  

145

                  

150

                  

155

                  

160

                  

165

                  

170

                  

175

                  

180

                  

185

                  

190

                  

195

                  
  • NB: make a note of the room temperature, lower board temp and lower IR temperature setting, before you start.

My Results

Room:            c
Board:            c
Lower IR        c

Lower HR Temp

180

185

190

195

200

 210 220 225

Upper Board Temperature

                  

140

                  3:46

145

                  4:06

150

                  4:29

155

                  5:02

160

                  5:45

165

                  9:27

170

                  -

175

                  -

180

                  -

185

                  -

190

                  -

195

                  -
  • NB: make a note of the room temperature, lower board temp and lower IR temperature setting, before you start.

Some notes - sort

  • make sure probe is clean and touching
  • moving the HR head does not seem to make much of a difference
  • moving the unpper IR does make a lot of difference after 2-2.5cm
  • running the lower HR causes a drop in temperature probably because of the air exgust blowing over the board and probe
  • on the probe table lower IR this is for noobies + to get the max delivered temp by the plates for safety
  • while experimenting aus 225 lower would deliver 160 on top of board, so johnny was right
  • chip might get hotter when you run upper IR
  • when the lower HR is on it seem to slightly lower the plate temp. must be HR exhaust
  • over time the temperaturs will creep up, however this will  follow that standard plateau graph.

Next phase - Build profile preheat segment

This will include

so at 5:45 mins the top board temperature is 160 and bottom board temperature is 160c(+/-10c) so the settings for me should be as follows:

  • Upper IR
    • L1 - 0
    • R1 - 1.00
    • D1 - 360
  • Lower IR
    • 225
  • Lower HR (if present)
    • L1 - 225
    • R1 - 1.00
    • D1 - 2000

Now build your profile

  • If 160 is not reached by step 2, increases your preheat temp. As long as you dont go into the bottom board temperature danger zone, makign it slightly longer is ok.
  • for lower IR dont set time as soon 1 is left, give it a moment. I think i meant to say here is it ok making a step a little be longer than  required to add a buffer into your profiel so you do not have to redue your profile every time. However the very last moments of your profile need to be the the same as this is the point at which reflow occurs. so adding a couplke more seconds in at 190c segment is ok but not as safe at 225c.
Published in Electronics
Read more...
Monday, 12 June 2017 15:55

Getting Poedit to work with smarty tsmarty2c.php extractor and gettext

Background

I have designed a project using smarty and I have decided that the easiest way to add a transaltion system was to use gettext() because it was widely supported. I shortly discovered there were many issues with it which I have resolved bar one, scanning smarty .tpl files with Poedit. I am not going to do all my translations manually!!!

I managed to get quite far using poedit and messing with the various parsers but they were not 100% for me.

The goto plugin for smarty to add gettext support is smarty-gettext , a plugin that allows you to put text you want to translate inbetween some custom tags and then these will be translated into gettext() statements.

{t}....{/t}

Now Poedit will not find these tags, nor can you config custom scans for it in the software. You can however use external extrators to do this.

smarty-gettext has a command line parser that will scan your .tpl files and return correctly formatted .po and .mo files but this means you have to scan from the command line with this script, then you have to use Poedit to scan your PHP files which could get quite messy and then somehow merge the 2. So to that end I have manged to get the command line parser to function as a Poedit extractor. I will also go through the different parts of the process so you can perhaps build your own parser or utilise one that is already there. I have also found a few other Poedit parsers that might be of use and i will also list them later.

These instructions were written with gettext() in mind but because of caching issues with gettext() I decided to use motranslate. The instructions are exactly the same for both technologies except you need to add some Additional keywords for motranslate.

The Process

My advise is to follow these instructions carefully and then once you have the process working it is at that point you should alter it to your needs. These instructions are also for Windows and Xampp.

I will be using my file locations so everything matches up. I will also  assume 

Prerequisites

  • Poedit 1.83 - The newer version is not as easy to use and I would not recommend it.
  • Xampp v5.6.30 - I use the zip version because it is easier.
  • Windows
  • gettext 0.19.8.1 and iconv 1.14 - Binaries for Windows (static) - I used 64bit but 32bit should be ok aswell
  • smarty-gettext (block.t.php, function.php so smarty will translate and tsmarty2c.php for the parser) - just copy raw version into a text file with the correct name.
  • the gettext() translation system is setup in your project
  • Smarty plugins are installed (block.t.php, function.php).
  • You also will need a couple of {t}....{/t} strings to exisit so they can be found.

Edit tsmarty2c.php to search .js files

When specifying a directory the script will only search within .tpl files and I need it to search with some .js files I have in my project. So edit the file and change:

// extensions of smarty files, used when going through a directory
$extensions = array('tpl');

 to

// extensions of smarty files, used when going through a directory
$extensions = array('tpl', 'js');

Configure the enviroment

  • install xampp to D:\websites
  • install Poedit
  • create a folder D:\websites\php\gettext
  • Extract msgcat.exe and msgmerge.exe from gettext 0.19.8.1 and iconv 1.14 - Binaries for Windows (static) and put them in D:\websites\php\gettext
  • Put a copy of tsmarty2c.php into D:\websites\php\gettext
  • add the 2 following paths to you windows PATH variable
    • D:\websites\php
    • D:\websites\php\gettext.
  • restart windows so the new paths will be recognised
  • I added the 2 folders to the windows PATH so the files within can be found easily. You can probably get all of this working without adding stuff to the PATH but I would not reommend it. php.exe, msgcat.exe and msgmerge.exe all need to be found for this to work.
  • You could probably get away with just adding the D:\websites\php tot he PATH but I found it neater to have these other files in their own folder.
  • I choose the static versions for the gettext() package so I did not have to worry about if the .dll files were registered or could be found
  • xammp does not need to be running
  • make sure you do not have other version of gettext in the path otherwise the software can get confused

Configure Poedit

All the files and settings are inplace we now need to configure Poedit. Make sure you download the older v1.8.13 version because you can still see how all of the parsers are configured.

On the orginal Poedit (v1.8) you were able to select which parsers were enabled. These were all configured the same except for the language type and what files they scanned. In v2.0 this has been removed. Below is the PHP Extractor setupfrom v1.8.

I have been using Poedit v2 with success for QWcrm.

Poedit PHP Extractor setup

Instructions

  • Open Poedit v1.8
  • Open up your primary language .po file D:\websites\htdocs\develop\qwcrm\language\en_GB\LC_MESSAGES\site.mo
  • Goto Catalogue->Properties->Source paths
    • and configure your folders as required for your project.
    • Dont forget you probably will have PHP and TPL sections. You can scan the whole project if you want.
  • Goto File->Preferences->Extractors
  • Disable all extractors you dont need - This makes experimenting easier but is optional. I just kept PHP
  • Create a new extractor with these settings
    • Language:    Smarty
    • List of extensions:    ‪*.tpl;*.js
      • and any other files types that use smarty and need translating.
    • Command to extract translations: 
      Search all folders supplied by Poedit (recommended)
php "D:\websites\php\tsmarty2c.php" -o %o %F

Only searches in the folder 'themes'
‪php "D:\websites\php\tsmarty2c.php" -o %o themes
    • An item in keywords list:    ‪-k%k
    • An item in input files list:    ‪%f
    • Source code charset:    ‪nothing in this
    • Looks like this for my example
      Poedit Smarty Extractor setup
  • Edit the Catalogue Properties - Will not appear until you have created a new catalogue and I have used QWcrm as an example. These settings are stored in the .po file.
    • Translation properties
      • Project name and version - QWcrm v3.0.0
      • Language team - https://quantumwarp.com/
      • Language - English (United Kingdom) - set this to the language of the translation you are doing.
      • Plural forms - leave as 'Use default rules for this language' unbless you know why you want to change this.
      • Charset - UTF-8 (recommended)
      • Source code charset - UTF-8 (recommended)
    • Source Paths
      • Base path - This should be the root folder of your project. 
        D:\websites\htdocs\develop\qwcrm
      • Paths - .  (only a period)
      • Excluded paths - This is important to reduce the files that are scanned such as cached files which also have gettext() functions in them. These are my exclusions, yours will be different.
        cache\
nbproject\
libraries\
logs\
media\
themes\default\js\dhtmlcombo\
themes\default\js\tinymce\
themes\default\js\jscal2\jscal2.js
themes\default\js\jscal2\unicode-letter.js
themes\default\js\jscal2\css\
    • Sources keywords
      • Additional keywords - These are only required if you are using motranslate. 
        _gettext    - This is for the function _gettext()
__          - This is for the function __()

NB:

  • I could not get %f to work with this process and so I had to use the folder name themes. Not exactly sure if Poedit is adding the rest of the file location onto it.
  • You must use %F in your extractor string i.e. uppercase.
  • themes is my root folder that is holding all of my .tpl files
  • you will notcie that i have added *.js and this is because I have some files that are *.js with some smarty tags in that need parsing.
  • when using this extrator the files you configure in the catalogue (i.e. include excludes) will be ignores because you are not using the %f option. I would like to get this fixed so i can use it.
  • because we are not using %F for the file list and instead a hardcode value we will need to change the target directory for each project for the Smarty Extractor. You could create multiple extractors and just keep them disabled until you need them.
  • Also when you are not using the particular smarty extractor you will need to disable it as it will always search that directory.
  • If there are not translations found by the smarty extractor then Poedit will generate and error saying thas the .po file might be corrupt. This might also be the case of all extractors but I cannot be 100% because i am not going to spend time testing that hunch.
  • Poedit does not always save you preferences and settings so make sure you save and check them if things are not working.
  • in Poedit a folder  'Exclusion' will override a folder 'Inclusion'.

I will just explain a little here about this extraction process. All of the other Extractors are using xgettext() to scan files and return a correctly formatted .po file. xgettext() is dated and will not recognise a lot of languages and you can not add custom scans in to it.

Poedit extractor switches explained (This is useful so you can workout how to build you own parser.)

  • -o    - This is just a standard switch that is pass to the extractor. It is not specific to Poedit. However the xgettext() uses this to denote the output file and so does tsmarty2c.php
  • %o    - this is a placeholder for Poedit to swap out with the tempory .pot file location where changes are merged to.
  • %k    - this placeholder is to be able to pass custom function names to your script (or xgettext) to scan within for strings i.e. instead of gettext("translate me") you could scan for chicken("translate me") or football("translate me"). It is not mandatory to have this.
  • %f    - this is placeholder for the file list. I am not sure of the format but it is in a format that ngettext() will accept. All files are passed in 1 large list here. I can't work out if this is type, but to pass the list you need to use %F.
  • ‪--from-code=%c    - this is always added on to the end of the command line and is to specify what charset the files houls be open as. I do not need this so have removed it. I thing it is mainly for xgettext. My scripts have been running with it attached though.
  • NB: the instructions for Poedit for the extractors does not make sense. There is information left out.

So what happens with an extractor when you hit 'update from sources'

  • poedit scans the project's folders (as per your rules) for allowed files and builds a list
  • one of the following happens (not sure which)
    • The files are passed as a single command line to an extractor that matches the file extensions (possible goes to multiple extractors if an extension is registered in more than 1 extractor)
    • Each file is passed as a single command line to an extractor that matches the file extension (possible goes to multiple extractors if an extension is registered in more than 1 extractor)
  • the extractor then parses these files and then merges the strings it finds to a temporary pot.file (using msgmerge.exe)
  • the next extractor is run and those files are merged into the same temporary .pot file(using msgmerge.exe)
  • once finished Poedit loads this temporary .pot file in to memory and you are returned to the main screen
  • your changes have not been saved yet. For this you actually need to hit save to apply the changes to you .po file.
  • Poedit will now build the .po and .mo files

Extract the Translations

Comodo Sandbox which is part of the comodo Internet Security (CIS) if running will cause the extraction process to fail because it blocks the PHP script. You need to disable the Comodo Sandbox before running the extraction. This might also be the case for other sandboxes and Firewalls.

So now run Poedit and extract the translations as outlined above by hitting the Update from sources menu item of click the Update button.

Published in PHP
Read more...
Friday, 09 June 2017 09:07

Gettext translation system

gettext() is a translation system written by GNU people and is present on many systems natively. This makes it a good standard to use. Gettext does have some issues especially on PHP and I will outline what I have found here with possible solutions

I now use motranslator which is based on the same technology as gettext because of the following:

  • gettext translation file (.mo) is cached by the server and is never refreshed until you restart server, so changes you make will not be seen untill then. Not practical for shared webhosting.
  • On Windows gettext will not allow translations to other than default language - There are some untested workarounds but they are messy (Custom Code, Thread Safe?, PHP as FastCGI not as a module)
  • On Windows gettext will only read the directory that belongs to the default language meaning all translations need to go in the default local directory to allow translations.

General Gettext

Gettext Tutorials

Gettext Documentation

Gettext software

Misc documents

Issues I have found with gettext

It is not straight forward to getting gettext() to workout of the box, especially when using xammp on windows. HEre I will address the issues I came across while trying to get translation to work. These issues probably also appear on diffrent setups.

Apache Gettext Caching issue

When i was developing QWcrm on my windows xammp setup I would make changes to translation strings in th .PO file (via POedit) and these would not get reflected in my software. After a lot of research I discovered that when you run PHP as an Apache module the the getttext translation files upon first load are cached 'premanently' or for a time which I can not figure out. So to get around this I needed to find a way of refreshing the cache or turning it of for my development site.

Solutions I found

  • change the .MO filenames loaded when ever you make changes - this works but is a bit of a mess and I would not recommend it for production sites.
  • use PHP as Fast CGI. When PHP is loaded as a module the .MO files are cached 'permantely' or for an undisclosed time I can not figure out but this apparently is not the case when using PHP as Fast CGI
  • The only true way to clear the cache it to restart apache. I think if you are running PHP as an Apache module, you have to restart Apache for the catalog to be reloaded.
  • clearstatcache() - i could not get this to work

I will now outline my research below

Emptying of MO gettext cache without restarting apache links

Gettext not working on windows / xampp

These following issues can cause a real problem when devloping with gettext on windows. The main issue I came across is the localisation handling.

Issue

  • gettext will work but only with the default language/region
    • The reason is that the language/region is set for the whole computer and cannot be changed for individual processes. I believe that this is why there are 2 types of apache/php thread safe and non thread safe. I am not 100% how to apply the 2 variants and I dont know of any work aruonds to change the settings just for xampp.
    • Windows uses different region codes thant those used by gettext (gettext = de_DE whereas windows = deu)

Solutions

  • if you use the standard gettext file layout windows will always be able to access the default language folder. This gives to unique features
    • it allows translation to work normally by using the default language (at least you can test gettext is working)
    • you can add in the other language translation files into the default language folder and then they can also be used (this is a workaround)
  • not confirmed - is this because I do not have the other languages installed on my windows PC - see this thread

You can write code to utilise the workaround outline above by address the locale issue. When trying to fix this do not forget about the caching issue above.

Links

xgettext will not scan within double quotes / remmed lines

xgettext is very particular about what it scans for, it seems to be syntax aware rather than just searching for the strings. i.e. if a line is remmed out it will ignore it.

Scanning source files for translations strings

once you have your project done you need to grab all of the strings and translate them, doing this manually would be difficult and you would miss strings. I outline the options I have found below:

  • Use software to extract the gettext strings
  • Scan .tpl files with specialised scripts
  • Compile all smarty templates and then just scan them with POedit. It is not ideal but will work

Applications

  • POEdit- will scan your PHP files for standard gettext() strings. It will not scann for {t}..{/t} strings in the .tpl files
  • POedit + a custom extractor that will scan the .tpl files. I dont currently have one.
  • Eazy Po - Eazy Po is a translation editor and a catalog manager for Gettext translation files.

Smarty TPL Scanning software

Poedit Links

Smarty Poedit Parsers (software) / and other parsers than can potentially be used

Poedit notes

  • in version 2 all of the parsers have bve removed and put into 1 extractor which you cannot edit or turn off so you should use version v1.83
  • because there are no extractors to see how they are configure i will give you one below. All of the parsers (except TWIG) use xgettext. the only different between the settings are the language name
    • Language:    Perl
    • List of extensions:    ‪‪*.pl;*.PL;*.pm;*.perl;
    • Command to extract translations:    ‪‪xgettext --language=Perl --add-comments=TRANSLATORS: --force-po -o %o %C %K %F
    • An item in keywords list:    ‪-k%k
    • An item in input files list:    ‪%f
    • Source code charset:    ‪‪--from-code=%c
  • if you run xgettext --help on the command line you will see all of the switches listed
  • in ngettext -o is an alias of --output=FILE  and all other switches are similiar.

Compiling all smarty templates

Translating in Smarty

This section is involve in actually translating the strings in the smarty software.

Smarty translation links (various)

Smarty Software

tsmarty2c.php Command line examples

setup the software as per these instructions - configure the software, you dont need the poedit software for this to work.

  • php "D:\websites\php\gettext\tsmarty2c.php" D:\test.pot %f | "C:\Program Files (x86)\Poedit\GettextTools\bin\xgettext -LC --add-comments --no-location -"
    • this involves sending it to xgettext and is not needed here
    • i have just added to show you can send output to another program with a  pipe (|) character 
  • php "D:\websites\php\gettext\tsmarty2c.php" -o test.pot "D:\websites\htdocs\develop\qwcrm\themes"
    • this will scan the 'themes' directory recursively for all .tpl files and parse them for the {t}...{/t} strings
  • php -q tsmarty2c.php mytemplate.tpl > mydomain.c
    • this will output to a file
    • you could just use the -o switch

 

Published in PHP
Read more...
Friday, 12 May 2017 07:45

Compressing PHP output with gzip or deflate

In this article we will be discussing the compression of output from a php, not normal assets such as JS and css. Because the nature of PHP files is dynamic we have to use another method. 

After needing to compress my PHP output for QWcrm I started researching on the internet after thinking it would just be a case of a couple of lines of code. What I discovered is there are several ways to compress PHP output and each has their pros and cons. I also found that some people were incorrectly using the wrong or outdated methods. Below I will go through each method I found and then I will sum up my thoughts at the end so you can easily workoout what method you want to use.

One other thing to note is that everyone goes on about how they gzip their content, but in these modern times there are 2 compression methos GZIP and DEFLATE, deflate being the newer method and can offer better compression.

.htaccess - (for static assets and PHP sometimes)

I have discovered that on some server installs that the following code (in particular text/html) will compress PHP output. Obviously you have to have mod_deflate installed. Worth a try and add other file types as needed.

<IfModule mod_deflate.c>
    <IfModule mod_filter.c>
        AddOutputFilterByType DEFLATE text/html
    </IfModule>
</IfModule>

Below is the normal way of compressing with deflate (gzip replacement) and the legacy gzip. I have added these for references only as it keeps coming up but will not actually compress PHP output but as a combined effect can help reduce the download footprint of the webpage and its assets.

Example 1 - Enable compression via .htaccess (mod_deflate)

This seems to be the prefered .htaccess compression method now because it gets better ratios.

<IfModule mod_deflate.c>
	<IfModule mod_filter.c>
		AddOutputFilterByType DEFLATE text/html text/css text/javascript application/javascript application/x-javascript
	</IfModule>
</IfModule>

Example 2 - Enable compression via .htaccess (mod_gzip)
http://www.awesomeinfolab.com/enable-gzip-compression/

I have never come across this method before in htaccess and might not work on all apache installs. I am thing of PHP as Fast-CGI/'Apache Module'

<ifModule mod_gzip.c>
    mod_gzip_on Yes
    mod_gzip_dechunk Yes
    mod_gzip_item_include file .(html?|txt|css|js|php|pl)$
    mod_gzip_item_include handler ^cgi-script$
    mod_gzip_item_include mime ^text/.*
    mod_gzip_item_include mime ^application/x-javascript.*
    mod_gzip_item_exclude mime ^image/.*
    mod_gzip_item_exclude rspheader ^Content-Encoding:.*gzip.*
</ifModule>

Links

ob_start("ob_gzhandler") - (Simple 1 liner)

This takes advantage of Php's buffered output and buffers all output of the script until the Php engine has completed, and then runs all the output through a special function that gzip compresses it before sending it on to the browser.

Just place this at the very top of all your PHP pages and it will send gzip-compressed output to the browsers with the correct headers.

ob_start("ob_gzhandler");

The function basically says start buffering PHP content and tag it to says the outputted content should be gzipped. The procedure should also send the correct headers so the browser knows it is compressed. I also think that unless the browser has told the server that it supports compression that the content will be returned uncompressed. All modern browsers send the 'I support compression' headers.

Some say once the script is finished it will flush the buffer and output the content automatically and that is why you can get away with 1 line, however this article from magicmonster tells you to add the flush command at the end to flush the cache.

ob_end_flush();

The method mentioned above is quick and easy, but the downfalls are that it only works on Apache with mod_gzip and according to the Php manual this is not the preferred method for gzipping.

You can increase the compression ratio by altering the php.ini or add the following to the script before ob_start("ob_gzhandler")

ini_set('zlib.output_compression_level', 4);

Another example

<?php if (substr_count($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip')) ob_start("ob_gzhandler"); else ob_start(); ?>

This example adds a check to see if the browser send the “Accept-Encoding: gzip” or “deflate” header before enabling the compression, however the buffer is still enabled even if compression is not required.

(From the Php manual at www.php.net)
note that using zlib.output_compression is preferred over ob_gzhandler().

Links

Full PHP buffer

Create a buffer, fill the buffer, compress the buffer data and send the content to the client

This is similiar to ob_start("ob_gzhandler") but there is a little more to it.

Example 1
http://www.webcodingtech.com/php/gzip-compression.php

This example utilises the PHP buffer correctely and manual takes the buffered content, sends the compress data headers relevant to the compression method, then it compresses the blob before returning it to the buffer. The buffers is then dispaly or returns to the browser as compress content. I dont think this method is dependant on having Apache mod_gzip installed for the auto detection of the compression headers from  the browser or to send the correct output headers.

This example has some issues an seems dated to use for compression. I think PHP has moved on. I would not use print as echo is quicker and I have no idea what print("\x1f\x8b\x08\x00\x00\x00\x00\x00"); does.

// Include this function on your pages
function print_gzipped_page() {

    global $HTTP_ACCEPT_ENCODING;
    if( headers_sent() ){
        $encoding = false;
    }elseif( strpos($HTTP_ACCEPT_ENCODING, 'x-gzip') !== false ){
        $encoding = 'x-gzip';
    }elseif( strpos($HTTP_ACCEPT_ENCODING,'gzip') !== false ){
        $encoding = 'gzip';
    }else{
        $encoding = false;
    }

    if( $encoding ){
        $contents = ob_get_contents();
        ob_end_clean();
        header('Content-Encoding: '.$encoding);
        print("\x1f\x8b\x08\x00\x00\x00\x00\x00");
        $size = strlen($contents);
        $contents = gzcompress($contents, 9);
        $contents = substr($contents, 0, $size);
        print($contents);
        exit();
    }else{
        ob_end_flush();
        exit();
    }
}

// At the beginning of each page call these two functions
ob_start();
ob_implicit_flush(0);

// Then do everything you want to do on the page
echo 'Hello World';

// Call this function to output everything as gzipped content.
print_gzipped_page();

Links

zlib.output_compression = on - (automatic and invisible)

This is the prefered method for gzipping over ob_gzhandler()

The zlib extension can be used to transparently compress PHP pages on-the-fly if the browser sends an “Accept-Encoding: gzip” or “deflate” header. Compression with zlib.output_compression seems to be disabled on most hosts by default, but can be enabled with a custom php.ini file:

The zlib extension is the undelying technology and the zlib.output_compression = On is a switch that enables transparent/invisible compression on all PHP content. The zlib extension is also the library that is used for other compressions such as ob_gzhandler. I would need to check which compressions operations were covered by it. 

This method is installed on most servers but left of by default. It will automatically send the output back to the client's browser in a compressed form if the 'allow compressed content' header is sent with the page request. If enabled in the php.ini then no further coding is required in any php (or other) script, or for that matter any assets.

Enable via php.ini

Add or alter the following line in the php.ini 

zlib.output_compression = On

Enable via a PHP script 

If zlib.output_compresssion is disabled but installed, you can enable either by editing the php.ini (as above) or you can add the following in to your PHP script. This will only enable compression for the script it is included in. and you must add this code before any headers or output is sent from the script. You should also note that having this method of compression will cause errors if further compression methods are implemented in your scripts.

if (extension_loaded("zlib") && (ini_get("output_handler") != "ob_gzhandler"))
{
    @ob_end_clean();
    @ini_set("zlib.output_compression", 1);
}

As you can see this simple script makes a few checks, that the libary is present and that ob_gzhandler() has not been set. This is perhaps optional depending on your script, the ob_end_clean() just makes sure any buffers are emptied, I am not sure this is needed either. The @ symbol just surpresses errors and again these could be removed

Links

gzencode / gzcompress / gzdeflate - (for a single blob)

There is a difference between these 3 functions even thought they all compress a blob you supply to them. However the concesous is to use gzencode() as it outputs in the correct format including the required checksums. This function is also supports both gzip and deflate compression algorithyms.

Joomla Example

I have included this to see how these guys do it as they have more experience than me. This process happens in 2 distinct sections, the first check to see if the gzip function is enabled whilst check to see if the server supports zlib compression and that ob_gzhandler has not already been set.

/**
 * Execute the application.
 *
 * @return  void
 *
 * @since   3.2
 *
 * From {Joomla}libraries/cms/application/cms.php
 */
 
 /**
 * @package     Joomla.Libraries
 * @subpackage  Application
 *
 * @copyright   Copyright (C) 2005 - 2016 Open Source Matters, Inc. All rights reserved.
 * @license     GNU General Public License version 2 or later; see LICENSE.txt
 */
 
public function execute()
{
    // Perform application routines.
    $this->doExecute();

    // If we have an application document object, render it.
    if ($this->document instanceof JDocument)
    {
        // Render the application output.
        $this->render();
    }

    // If gzip compression is enabled in configuration and the server is compliant, compress the output.
    if ($this->get('gzip') && !ini_get('zlib.output_compression') && (ini_get('output_handler') != 'ob_gzhandler'))
    {
        $this->compress();

        // Trigger the onAfterCompress event.
        $this->triggerEvent('onAfterCompress');
    }

    // Send the application response.
    $this->respond();

    // Trigger the onAfterRespond event.
    $this->triggerEvent('onAfterRespond');
}

The second section actually does the compression and further checking, and yes there is some duplication of these checks. This fucntion supports the use of gzip or deflate compression algorithyms which is great for compatability. If you look at the code you can see that gzencode() is used and that if performs compression on a single variable (blob) rather than a page. gzencode() is able to use both algorithyms where as the other 2 functions this section covers cannot.

The code has been compress but because of the nature of the function you have to manually send the 'Content-Encoding' header so the browser knows the payload is compressed and how it has been compressed.

/**
 * Checks the accept encoding of the browser and compresses the data before
 * sending it to the client if possible.
 *
 * @return  void
 *
 * @since   11.3
 *
 * From {Joomla}libraries/joomla/application/web.php
 */
 
 /**
 * @package     Joomla.Platform
 * @subpackage  Application
 *
 * @copyright   Copyright (C) 2005 - 2016 Open Source Matters, Inc. All rights reserved.
 * @license     GNU General Public License version 2 or later; see LICENSE
 */

protected function compress()
{
    // Supported compression encodings.
    $supported = array(
        'x-gzip' => 'gz',
        'gzip' => 'gz',
        'deflate' => 'deflate'
    );

    // Get the supported encoding.
    $encodings = array_intersect($this->client->encodings, array_keys($supported));

    // If no supported encoding is detected do nothing and return.
    if (empty($encodings))
    {
        return;
    }

    // Verify that headers have not yet been sent, and that our connection is still alive.
    if ($this->checkHeadersSent() || !$this->checkConnectionAlive())
    {
        return;
    }

    // Iterate through the encodings and attempt to compress the data using any found supported encodings.
    foreach ($encodings as $encoding)
    {
        if (($supported[$encoding] == 'gz') || ($supported[$encoding] == 'deflate'))
        {
            // Verify that the server supports gzip compression before we attempt to gzip encode the data.
            // @codeCoverageIgnoreStart
            if (!extension_loaded('zlib') || ini_get('zlib.output_compression'))
            {
                continue;
            }
            // @codeCoverageIgnoreEnd

            // Attempt to gzip encode the data with an optimal level 4.
            $data = $this->getBody();
            $gzdata = gzencode($data, 4, ($supported[$encoding] == 'gz') ? FORCE_GZIP : FORCE_DEFLATE);

            // If there was a problem encoding the data just try the next encoding scheme.
            // @codeCoverageIgnoreStart
            if ($gzdata === false)
            {
                continue;
            }
            // @codeCoverageIgnoreEnd

            // Set the encoding headers.
            $this->setHeader('Content-Encoding', $encoding);

            // Header will be removed at 4.0
            if ($this->get('MetaVersion'))
            {
                $this->setHeader('X-Content-Encoded-By', 'Joomla');
            }

            // Replace the output with the encoded data.
            $this->setBody($gzdata);

            // Compression complete, let's break out of the loop.
            break;
        }
    }
}

So that is how Joomla does compression and it should be noted that this method is probably the most compatible way of doing compression.

Links

What I used in QWcrm

I wrote my own function bas heavily of the Joomla function to enable gzip. In QWcrm I was able to do this becaus emy page is stored in a single varible/blob that I can manipulate.

In the main index.php I have the following code

################################################
#         Page Compression                     #
################################################

// Compress page and send correct compression headers
if ($gzip == true && $VAR['theme'] !== 'print') {

    $BuildPage = compress_page_output($BuildPage);
    
}

This is the function code stored in include.php

###########################################
#  Compress page output and send headers  #
###########################################

/**
 * Checks the accept encoding of the browser and compresses the data before
 * sending it to the client if possible.
 *
 * @return  void
 *
 * @since   11.3
 *
 * From {Joomla}libraries/joomla/application/web.php
 */

/**
 * @package     Joomla.Platform
 * @subpackage  Application
 *
 * @copyright   Copyright (C) 2005 - 2016 Open Source Matters, Inc. All rights reserved.
 * @copyright   Copyright (C) 2017 - Jon Brown / Quantumwarp.com
 * @license     GNU General Public License version 2 or later; see LICENSE
 */

function compress_page_output($BuildPage)
{
    // Supported compression encodings.
    $supported = array(
        'x-gzip'    => 'gz',
        'gzip'      => 'gz',
        'deflate'   => 'deflate'
    );

    // Get the supported encoding.
    $encodings = array_intersect(browserSupportedCompressionEncodings(), array_keys($supported));

    // If no supported encoding is detected do nothing and return.
    if (empty($encodings))
    {
        return $BuildPage;
    }

    // Verify that headers have not yet been sent, and that our connection is still alive.
    if (headers_sent() || (connection_status() !== CONNECTION_NORMAL))
    {
        return $BuildPage;
    }

    // Iterate through the encodings and attempt to compress the data using any found supported encodings.
    foreach ($encodings as $encoding)
    {
        if (($supported[$encoding] == 'gz') || ($supported[$encoding] == 'deflate'))
        {
            // Verify that the server supports gzip compression before we attempt to gzip encode the data.            
            if (!extension_loaded('zlib') || ini_get('zlib.output_compression'))
            {
                continue;
            }           

            // Attempt to gzip encode the page with an optimal level 4.            
            $gzBuildPage = gzencode($BuildPage, 4, ($supported[$encoding] == 'gz') ? FORCE_GZIP : FORCE_DEFLATE);

            // If there was a problem encoding the data just try the next encoding scheme.            
            if ($gzBuildPage === false)
            {
                continue;
            }            

            // Set the encoding headers.
            header("Content-Encoding: $encoding");

            // Replace the output with the encoded data.            
            return $gzBuildPage;
            
        }
    }
}

####################################################################
#  Get the supported compression algorithms in the client browser  #
####################################################################

function browserSupportedCompressionEncodings() {
        
    return array_map('trim', (array) explode(',', $_SERVER['HTTP_ACCEPT_ENCODING']));

}

In future I will try and see if the mod_deflate option is working on other servers as this might be built into newer version of the module to treat PHP output as normal files and compress them.

Other Links

Published in PHP
Read more...
Saturday, 15 April 2017 14:31

My ADOdb Notes

This is just a simple article to references most of the simple ADOdb functions I use in QWcrm and what they do:

Smarty Call

$smarty->assign('currency_sym', get_company_details($db));

The Basic Function

I will use this function for the demo. This function just returns all of the company information or a single item.

function get_company_details($db, $item = null){
    
    global $smarty;

    $sql = 'SELECT * FROM '.PRFX.'TABLE_COMPANY';
    
    if(!$rs = $db->execute($sql)){        
        force_error_page($_GET['page'], 'database', __FILE__, __FUNCTION__, $db->ErrorMsg(), $sql, $smarty->get_template_vars('translate_system_include_error_message_function_'.__FUNCTION__.'_failed'));
        exit;
    } else {
        
        if($item === null){
            
            return $rs->GetArray(); 
            
        } else {
            
            return $rs->fields[$item];   
            
        } 
        
    }
    
}

The Results

I will be changing the line and listing the results

return $rs->GetArray();

$rs->GetArray();

Array
(
    [0] => Array
        (
            [0] => QuantumWarp
            [NAME] => QuantumWarp
            [1] => 123456
            [NUMBER] => 123456
            [2] => The Gurking
            [ADDRESS] => The Gurking
            [3] => London
            [CITY] => London
            [4] => Greater London
            [STATE] => Greater London
            [5] => LA1 W10
            [ZIP] => LA1 W10
            [6] => AF
            [COUNTRY] => GB
            [7] => 123456
            [PHONE] => 123456
            [8] => 123456
            [MOBILE] => 123456
            [9] => 123456
            [FAX] => 123456
            [10] => 
            [EMAIL] => 
            [11] => £
            [CURRENCY_SYMBOL] => £
            [12] => GPB
            [CURRENCY_CODE] => GPB
            [13] => %d/%m/%Y
            [DATE_FORMAT] => %d/%m/%Y
            [14] => media/logo.png
            [LOGO] => media/logo.png
            [15] => 
            [WWW] => 
            [16] => 10
            [OPENING_HOUR] => 10
            [17] => 0
            [OPENING_MINUTE] => 0
            [18] => 17
            [CLOSING_HOUR] => 17
            [19] => 0
            [CLOSING_MINUTE] => 0
            [20] => 3.50
            [TAX_RATE] => 3.50
            [21] => <p>welcome</p>
            [WELCOME_MSG] => <p>welcome</p>
            [22] => <p>thanks</p>
            [INVOICE_MSG] => <p>thanks</p>
        )

)
  • ADODB Manual - Link 1 - Generate a 2-dimensional array of records from the current cursor position, indexed from 0 to $number_of_rows - 1. If $number_of_rows is undefined, till EOF.
  • Returns all rows.

$rs->GetRows();

  • Generate a 2-dimensional array of records from the current cursor position. Synonym for GetArray() for compatibility with Microsoft ADO.
  • Returns all rows.

$rs->GetAssoc();

Array
(
    [QuantumWarp] => Array
        (
            [NAME] => QuantumWarp
            [0] => 123456
            [NUMBER] => 123456
            [1] => The Gurking
            [ADDRESS] => The Gurking
            [2] => London
            [CITY] => London
            [3] => Greater London
            [STATE] => Greater London
            [4] => LA1 W10
            [ZIP] => LA1 W10
            [5] => AF
            [COUNTRY] => GB
            [6] => 123456
            [PHONE] => 123456
            [7] => 123456
            [MOBILE] => 123456
            [8] => 123456
            [FAX] => 123456
            [9] => 
            [EMAIL] => 
            [10] => £
            [CURRENCY_SYMBOL] => £
            [11] => GPB
            [CURRENCY_CODE] => GPB
            [12] => %d/%m/%Y
            [DATE_FORMAT] => %d/%m/%Y
            [13] => media/logo.png
            [LOGO] => media/logo.png
            [14] => 
            [WWW] => 
            [15] => 10
            [OPENING_HOUR] => 10
            [16] => 0
            [OPENING_MINUTE] => 0
            [17] => 17
            [CLOSING_HOUR] => 17
            [18] => 0
            [CLOSING_MINUTE] => 0
            [19] => 3.50
            [TAX_RATE] => 3.50
            [20] => <p>welcome</p>
            [WELCOME_MSG] => <p>welcome</p>
            [21] => <p>thanks</p>
            [INVOICE_MSG] => <p>thanks</p>
        )

)
  • ADODB Manual - Link 1 - Generates an associative array from the recordset. Note that is this function is also available in the connection object. More details can be found there.
  • ADODB Manual - Link 2 - Returns an associative array for the given query $sql with optional bind parameters in $inputarr. If the number of columns returned is greater to two, a 2-dimensional array is returned, with the first column of the recordset becomes the keys to the rest of the rows. If the columns is equal to two, a 1-dimensional array is created, where the the keys directly map to the values (unless $force_array is set to true, when an array is created for each value).
  • Returns all rows.

$rs->FetchRow();

Array
(
    [0] => QuantumWarp
    [NAME] => QuantumWarp
    [1] => 123456
    [NUMBER] => 123456
    [2] => The Gurking
    [ADDRESS] => The Gurking
    [3] => London
    [CITY] => London
    [4] => Greater London
    [STATE] => Greater London
    [5] => LA1 W10
    [ZIP] => LA1 W10
    [6] => AF
    [COUNTRY] => GB
    [7] => 123456
    [PHONE] => 123456
    [8] => 123456
    [MOBILE] => 123456
    [9] => 123456
    [FAX] => 123456
    [10] => 
    [EMAIL] => 
    [11] => £
    [CURRENCY_SYMBOL] => £
    [12] => GPB
    [CURRENCY_CODE] => GPB
    [13] => %d/%m/%Y
    [DATE_FORMAT] => %d/%m/%Y
    [14] => media/logo.png
    [LOGO] => media/logo.png
    [15] => 
    [WWW] => 
    [16] => 10
    [OPENING_HOUR] => 10
    [17] => 0
    [OPENING_MINUTE] => 0
    [18] => 17
    [CLOSING_HOUR] => 17
    [19] => 0
    [CLOSING_MINUTE] => 0
    [20] => 3.50
    [TAX_RATE] => 3.50
    [21] => <p>welcome</p>
    [WELCOME_MSG] => <p>welcome</p>
    [22] => <p>thanks</p>
    [INVOICE_MSG] => <p>thanks</p>
)
  • ADODB Manual - Link 1 - Returns array containing current row, or false if EOF. FetchRow( ) internally moves to the next record after returning the current row.

$rs->GetRowAssoc();

Array
(
    [NAME] => QuantumWarp
    [NUMBER] => 123456
    [ADDRESS] => The Gurking
    [CITY] => London
    [STATE] => Greater London
    [ZIP] => LA1 W10
    [COUNTRY] => GB
    [PHONE] => 123456
    [MOBILE] => 123456
    [FAX] => 123456
    [EMAIL] => 
    [CURRENCY_SYMBOL] => £
    [CURRENCY_CODE] => GPB
    [DATE_FORMAT] => %d/%m/%Y
    [LOGO] => media/logo.png
    [WWW] => 
    [OPENING_HOUR] => 10
    [OPENING_MINUTE] => 0
    [CLOSING_HOUR] => 17
    [CLOSING_MINUTE] => 0
    [TAX_RATE] => 3.50
    [WELCOME_MSG] => <p>welcome</p>
    [INVOICE_MSG] => <p>thanks</p>
)
  • ADODB Manual - Link 1 - Returns an associative array containing the current row. The keys to the array are the column names. The column names are upper-cased for easy access. To get the next row, you will still need to call MoveNext().
  • Returns one row.
  • same as $rs->GetRowAssoc($toUpper=true)
  • you can use $rs->GetRowAssoc($toUpper=false) to give all lowercase

$rs->fields[$colname];

QuantumWarp
  • ADODB Manual - Link 1 - Returns the value of the associated column $colname for the current row. The column name is case-insensitive.
  • Returns one field.
  • The 'f' is supposed to be lower case, it does not work otherwise.

Conclusion

Muliple Records/Rows = $rs->GetArray(); This returns a 2 dimensional array which allows you to loop through the records which are also arrays.

Single Record/Row = $rs->GetRowAssoc(); - This returns a standard associative array.

Single Item from a column in a single Row = $rs->Fields[$colname];

Links

 

 

 

Published in PHP
Read more...
Page 14 of 96