Pi-hole for Raspberry Pi 3 Model B+
With WireGuard, Cloudflared, Monit, healthchecks.io, PADD and raspiBackup

In Asus RT-AC86U router go to: WAN, Internet Connection: Connect to DNS Server automatically: No

Edit: Above is not needed, and actually breaks WireGuard installation later on.

Go to: LAN - DHCP Server - DNS Server and set it to your rpi3 IP: 10.0.1.245

Browse to Group Management - Adlists to add blocklists.

Add well known dbl.oisd.nl blocklist, but consider using mirrors, e.g. https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl.txt

Another alternative is to use Wally3k’s list. I only subscribe to “ticked” lists.

I use both dbl.oisd.nl and Wally3k’s lists.

The default blocklists Pi-hole comes with are also OK to be used, and reverting to them is easy if you later decide to do so.

This is is done in order to be in control what is installed and not get unexpected breakages. Daily updates disabled as per this StackExchange post:

$ sudo systemctl mask apt-daily-upgrade

$ sudo systemctl mask apt-daily

$ sudo systemctl disable apt-daily-upgrade.timer

$ sudo systemctl disable apt-daily.timer

$ sudo aptitude update

$ sudo aptitude safe-upgrade

List all installed, and automatically installed aptitude packages:

$ sudo aptitude search '~i'

Show removed packages whose config files have not been removed:

$ dpkg -l | grep '^rc'

These config files can be removed with:

$ sudo aptitude purge ~c

Also, remember to clean the /var/cache/apt/archives periodically:

$ sudo aptitude clean

New releases of Raspberry Pi OS OS are periodically informed at Raspberry Pi blog.

$ sudo apt update

$ sudo apt full-upgrade

Get updated on new versions of pi-hole when they’re available by subscribing to rss feed of their Github release channel.

Then do:

$ sudo pihole -up

  [i] Checking for updates...
  [i] Pi-hole Core:     update available
  [i] Web Interface:    up to date
  [i] FTL:              update available

  [i] Pi-hole core files out of date, updating local repo.
  [✓] Check for existing repository in /etc/.pihole
  [✓] Update repo in /etc/.pihole

  [i] If you had made any changes in '/etc/.pihole/', they have been stashed using 'git stash'

  [i] FTL out of date, it will be updated by the installer.

  [✓] Root user check

        .;;,.
        .ccccc:,.
         :cccclll:.      ..,,
          :ccccclll.   ;ooodc
           'ccll:;ll .oooodc
             .;cll.;;looo:.
                 .. ','.
                .',,,,,,'.
              .',,,,,,,,,,.
            .',,,,,,,,,,,,....
          ....''',,,,,,,'.......
        .........  ....  .........
        ..........      ..........
        ..........      ..........
        .........  ....  .........
          ........,,,,,,,'......
            ....',,,,,,,,,,,,.
               .',,,,,,,,,'.
                .',,,,,,'.
                  ..'''.

  [i] Existing PHP installation detected : PHP version 7.3.14-1~deb10u1
  [i] Performing unattended setup, no whiptail dialogs will be displayed
  [✓] Disk space check
  [✓] Update local cache of available packages

  [✓] Checking apt-get for upgraded packages... 6 updates available
  [i] It is recommended to update your OS after installing the Pi-hole!

  [i] Installer Dependency checks...
  [✓] Checking for apt-utils
  [✓] Checking for dialog
  [✓] Checking for debconf
  [✓] Checking for dhcpcd5
  [✓] Checking for git
  [✓] Checking for iproute2
  [✓] Checking for whiptail

  [i] Performing reconfiguration, skipping download of local repos
  [✓] Resetting repository within /etc/.pihole...
  [✓] Resetting repository within /var/www/html/admin...
  [i] Main Dependency checks...
  [✓] Checking for cron
  [✓] Checking for curl
  [✓] Checking for dnsutils
  [✓] Checking for iputils-ping
  [✓] Checking for lsof
  [✓] Checking for netcat
  [✓] Checking for psmisc
  [✓] Checking for sudo
  [✓] Checking for unzip
  [✓] Checking for wget
  [✓] Checking for idn2
  [✓] Checking for sqlite3
  [✓] Checking for libcap2-bin
  [✓] Checking for dns-root-data
  [✓] Checking for resolvconf
  [✓] Checking for libcap2
  [✓] Checking for lighttpd
  [✓] Checking for php7.3-common
  [✓] Checking for php7.3-cgi
  [✓] Checking for php7.3-sqlite3

  [✓] Enabling lighttpd service to start on reboot...
  [✓] Checking for user 'pihole'

  [i] FTL Checks...

  [✓] Detected ARM-hf architecture (armv7+)
  [i] Checking for existing FTL binary...
  [i] Latest FTL Binary already installed (v4.3.1). Confirming Checksum...
  [i] Checksum correct. No need to download!
  [✓] Checking for user 'pihole'
  [✓] Installing scripts from /etc/.pihole

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf

  [i] Installing blocking page...
  [✓] Creating directory for blocking page, and copying files
  [✗] Backing up index.lighttpd.html
      No default index.lighttpd.html file found... not backing up

  [✓] Installing sudoer file

  [✓] Installing latest Cron script

  [✓] Installing latest logrotate script
  [i] Backing up /etc/dnsmasq.conf to /etc/dnsmasq.conf.old
  [✓] man pages installed and database updated
  [i] Testing if systemd-resolved is enabled
  [i] Systemd-resolved is not enabled
  [✓] Restarting lighttpd service...
  [✓] Enabling lighttpd service to start on reboot...
  [i] Restarting services...
  [✓] Enabling pihole-FTL service to start on reboot...
  [✓] Restarting pihole-FTL service...
  [✓] Deleting existing list cache
  [i] Pi-hole blocking is enabled
  [✗] DNS resolution is currently unavailable
  [✓] DNS resolution is now available

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: Retrieval successful

  [i] Target: sysctl.org (hosts)
  [✓] Status: Retrieval successful

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: Retrieval successful

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: Retrieval successful

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: Retrieval successful

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: Retrieval successful

  [i] Target: smokingwheels.github.io (allhosts)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: gist.githubusercontent.com (Test.txt)
  [✓] Status: Retrieval successful

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] Number of domains being pulled in by gravity: 149202
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 126670
  [i] Number of whitelisted domains: 199
  [i] Number of blacklisted domains: 4
  [i] Number of regex filters: 16
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

  [✓] Force-reloading DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

  [i] The install log is located at: /etc/pihole/install.log
Update Complete!

  Current Pi-hole version is v4.4
  Current AdminLTE version is v4.3.3
  Current FTL version is v4.3.1

After update completed, start monit (if you’ve installed it):

$ sudo service monit start

Messages can be found from: http://10.0.1.245/admin/messages.php (or http://pi.hole/admin/messages.php) and reveal ee possible issues.

To manually check memory card wear, do:

$ sudo badblocks -v /dev/mmcblk0p1

Checking blocks 0 to 43924

Checking for bad blocks (read-only test): done

Pass completed, 0 bad blocks found. (0/0/0 errors)

check program badblocks with path "/sbin/badblocks /dev/mmcblk0p1"
  every "30 19 * * 6"
  if status != 0 then alert

This is optional but heavily recommended.

# Size for the ram folder, it defines the size the log folder will reserve into the RAM.
# If it's not enough, log2ram will not be able to use ram. Check you /var/log size folder.
# The default is 40M and is basically enough for a lot of applications.
# You will need to increase it if you have a server and a lot of log for example.
SIZE=320M

pyyhttu@raspberrypi:~ $ service log2ram status
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2020-01-18 09:33:45 EET; 3h 3min ago
  Process: 261 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
  Process: 3583 ExecReload=/usr/local/bin/log2ram write (code=exited, status=0/SUCCESS)
 Main PID: 261 (code=exited, status=0/SUCCESS)

pyyhttu@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  2.1G   12G  15% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M  1.7M  462M   1% /dev/shm
tmpfs           464M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/sda1       7.1G  2.5G  4.3G  37% /backup
/dev/mmcblk0p1   43M   23M   20M  53% /boot
log2ram         320M   41M   60M  41% /var/log
tmpfs            93M     0   93M   0% /run/user/999
tmpfs            93M     0   93M   0% /run/user/1000

• Restore query logging and implement actions as here.

Privacy settings are in Pi-hole at Settings - Privacy. Set privacy level to 3 - Anonymous mode and consider disabling the query logging, as documented here. However, without query logging I wouldn’t be a chance to troubleshoot then. But as with every self-hosted solution that is shared with family members:

1. Respect the privacy of others.
2. With great power comes great responsibility.

More about on logging and collecting data here.

Use these repositories, but, it seems that it would resulting in disabling OS version check. Better wait for official pi-hole release.

Update [2021-09-12 Sun]: Version FTL v5.9 supports Debian 11. Update repositories as per this post, or this post, and followup threads on success rate, such as this. Also weigh in if you should try to first setup second pihole, and for update, try the “semi-official upgrade instructions. Before upgrading to bullseye, test the restore operation of latest raspiBackup. After upgrading to bullseye, check if Predictable Network Interface Names are edited.

Update [2021-09-26 Sun]: This was a hassle. The update did not fail but after a reboot, device wasn’t anymore discoverable in network. Reverted with backups.

Rasbian is not made for rolling release, and thus every major OS version upgrade, which are inevitable with point release operating systems, culminate to a risk scenario where I need to reinstall everything again. Which means, that I either,

1. establish infrastructure as a code and learn to configure everything with something like Ansible
2. start running pi-hole dockerized
3. investigate on rolling release distribution

At this point, I’ll go with option 3).

So far this error has occurred twice, and there can be multiple root causes for the error. Start by checking errors with:

$ journalctl -u pihole-FTL --full --no-pager

Normally this should just return “– No entries –”.

In the first error case there was an error entry:

dnsmasq: illegal repeated keyword at line 33
of /etc/dnsmasq.d/01-pihole.conf_bak

Error was resolved after deleting the above 01-pihole-conf_bak backup file I had created and then doing:

$ sudo service pihole-FTL restart

In the second case:

$ journalctl -u pihole-FTL --full --no-pager the error entry was:

dnsmasq: failed to create
listening socket for port 53: Address already in use

I also checked my cloudflared setup for encrypted DNS with:

$ sudo systemctl status cloudflared.service

There was an error msg:

failed to connect to an HTTPS backend
\"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request:
Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout
exceeded while awaiting headers)

So by doing:

$ sudo systemctl restart cloudflared.service and then again:

$ sudo systemctl status cloudflared.service

The error was no longer visible. Also:

$ pihole status should now give:

[✓] DNS service is running
[✓] Pi-hole blocking is Enabled

• Also ping to 10.0.1.245 does not respond
• In my case I needed to unplug the power cord, and reattach it to reboot.

For some reason, boot sequence did not complete on the first try.

This was the first time blocklist update caused 0 Domains on blocklist. It may be linked to the fact that either monit or healthcheck.io caused the update to fail. Gravity updates the blocklist on randomly selected time between 3 am and 5 am my local time (to equalize server loads): /var/log/syslog shows my update started 03:49 and logs shows no errors. Updating gravity manually $ pihole -g did not help. Finally, by disabling monit $ sudo service monit stop and commenting out healthcheck.io cron with $ crontab -e, and then doing repair with $ sudo pihole -r, the blocklist on dashboard showed the correct blocklist.

Starting iOS 14, Apple started to randomize its devices mac addresses. This randomizing is done by Apple to reduce tracking across different Wi-Fi networks. The feature is redundant in home wi-fi, and actually causes DHCP servers to hand out IP addresses different than those statically configured which can lead to clients not being assigned to the proper group.

• This is evident by going in pi.hole to network overview.
• Trying first to flush the network clears some multiple ip-addresses assigned, but not all.
• Deleting every entry with sqlite3 does clear everything, along the clients, but after a while client entries are repopulating back, some with multiple IP addresses assigned to them again.
• Doing now Settings, Flush network table does clear the Network overview table so that clients in the table are left with single IPV4-address, but again, temporarily. After a while, Samsung television e.g. gets multiple IPs, like 10.0.1.251, 10.0.1.252 and 10.0.1.253.
• Trying next that I assign these devices an IP manually.
• Assigning an IP manually, then adding the IP and a custome DNS entry at http://pi.hole/admin/dns_records.php helped somewhat: now network overview at least has the hostname visible. Let’s see if they only will get a single IPv4.

$ service log2ram status
● log2ram.service - Log2Ram
Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-12-12 03:21:39 EET; 5 days ago
Main PID: 17699 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Also, the service won’t restart:

$ sudo service log2ram restart
Job for log2ram.service failed because the control process exited with error code.
See "systemctl status log2ram.service" and "journalctl -xe" for details.
pi@raspberrypi:/var/log $ systemctl status log2ram.service
● log2ram.service - Log2Ram
Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-12-17 16:51:26 EET; 11s ago
Process: 7311 ExecStart=/usr/local/bin/log2ram start (code=exited, status=1/FAILURE)
Main PID: 7311 (code=exited, status=1/FAILURE)

Dec 17 16:51:26 raspberrypi systemd[1]: Starting Log2Ram...
Dec 17 16:51:26 raspberrypi log2ram[7311]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Dec 17 16:51:26 raspberrypi log2ram[7311]: /usr/local/bin/log2ram: 45: /usr/local/bin/log2ram: mail: not found
Dec 17 16:51:26 raspberrypi systemd[1]: log2ram.service: Main process exited, code=exited, status=1/FAILURE
Dec 17 16:51:26 raspberrypi systemd[1]: log2ram.service: Failed with result 'exit-code'.
Dec 17 16:51:26 raspberrypi systemd[1]: Failed to start Log2Ram.

Root cause seemed to be that RAM disk is too small so increasing that in /etc/log2ram.conf to 320M and doing $ sudo service log2ram restart fixed the issue.

On [2020-01-18 Sat] morning raspiBackup had failed. /var/log size had grown past 40M, and due to that, log2ram service didn’t start, failing raspiBackup. It seems that pihole logging takes most of the space:

pyyhttu@raspberrypi:/var/log $ du -sh -- * | sort -h
[...]
1.2M    auth.log
1.2M    daemon.log
1.3M    auth.log.1
1.7M    pihole.log
33M     pihole.log.1

Monitor how big /var/log will grow and whether 160M is enough (it is not). Consider disabling query logging (Settings > System > Disable query logging). This means that can’t tail the logs ($ pihole tail), but will still see the queries in the query log and dashboard.

For now, I’ve increased in /etc/log2ram.conf from the default 40M to the SIZE=320M which seems to be enough.

Error was: ignoring query from non-local network 100.115.92.133 (logged only once). Noticed this error on [2023-04-29 Sat 10:50] as DNSMASQ_WARN in pi-hole diagnosis. This may be due to fact I have WireGuard VPN setup on the same device. As per analysis here, I have now "Allow only local requests" changed to "Bind only to inferface eth0" at http://10.0.1.245/admin/settings.php?tab=dns. Observing if error returns.

Update: [2024-04-24 Wed]: The Warning has not returned, but I’ve kept setting "Allow only local requests" on.

Does cellular takes precedence over wifi, rendering the ad-blocking useless?

A: Yes, seems so. Either disable in iOS “Wifi Assist” under Settings - Cellular.

Or then, be attached to VPN all the time and observe battery consumption.

Update: Seems to be that having the WireGuard tunnel always on does not result in noticeable battery consumption. On Androids, where having WireGuard baked in as a module in custom kernels, the performance and battery consumption should be even more better.

Validate when leases expire, as the IP changes which complicates the setup of WireGuard.

Update: it seems that the lease has been gradually getting more time (interval unknown), observable by doing

$ ipconfig /all

See Raspberry Pi documentation and Debian documentation.

Also implement points from this post as actionable list.

Enable automatic, unattended security updates. Investigate if this is possible with Debian Sid.

I may want opt out certain production/work devices by manually configuring each device. For above to take effect, purging iOS DNS cache may require toggling the wifi on/off.

Update: Starting with pi-hole v5.0 onward, above per-client manual DNS settings are no longer needed. Pi-hole supports per-client blocking. Effectively we can then group everyone’s devices and define whether or not they’re part of pi-hole. To set this up, see Per-client blocking examples. For this to work reliably, remember to assign in router a manual IP for every such device.

tl;dr: If you want to exclude a device from pi-hole, assign in pi-hole a client to a group, then remove the client from the default group.

See relevant reddit regexp thread for more info. I added mmotti/regexp list from github.

Not done, as this would be futile, for many reasons.

Merlin is an alternative firmware to stock Asus firmware. However, Merlin is not without its problems. Merlin allows to me force DNS traffic to router, as referenced also in here. Remember these things when upgrading the Merlin firmware.

Do this after upgrade to Merlin. Some devices have hard coded DNS servers that can’t be changed via DHCP or manually. This is disturbingly an increasing trend among device manufacturers, so I must consider forcing all local network route traffic to pihole. Here’s the instructions for Asus router and the comment link here has screenshotted instructions among routing rules.

Having two Pi-holes, or alternatively having one Pi-hole setup and adding a virtual IP to it so the badly behaving devices see 2 DNS servers, seem to mitigate these devices from calling their own DNS server, and bypassing Pi-hole.

Hard coded DNS servers may also become an issue when using iOS 14.

This is also evident if messages like “This network is blocking DNS traffic” starts to pop up in iOS 14.

When forcing traffic through port 53, pay attention if tutorial guides disabling dhcp on router, and enabling it on Pi-hole. Don’t do that, but investigate after installing Merlin (“John’s fork”) for N66U, if rerouting can be done with this tutorial instead.

If after Merlin upgrade I experience high amount of traffic from router visible in Pi-hole statistics, apply these router configs.

See also General Merlin troubleshooting.

On defining DNS servers in Merlin (check the screenshot on that thread).

When testing the force routing of traffic through port 53, use your Phillips Hue lamps as guinea pigs as they seem to have hardcoded dns.

Also check out what is DNSFilter with Merlin mentioned in here. DNSFilter with Merlin mentioned also here and here.

Acquired a pair of new Asus RT-AC86U routers to create an AiMesh network. Pi-holes operating in backhauled mesh should be fine. Then redid the configurations for routers and observed if needed to apply this fix. Never had to resolve the fix.

Before installing Elk, run diagnostic on what is system load before and after installing the Elk stack. Memory usage, processor usage etc.

Edit: Explored ELK and considered that it is not needed, as it complicates the install, eat resources of Rpi3 and the fancy charts offer me no real value over the regular charts Pi-hole comes bundled with.

For Skynet investigate on this reddit thread. For Yazfi scripts, look here.

USB drive is required for Swap: https://www.snbforums.com/threads/skynet-installation-help.88080/

Skynet forum: https://www.snbforums.com/search/1105236/?q=skynet+usb&t=post&c[child_nodes]=1&c[nodes][0]=60&o=relevance

https://www.snbforums.com/threads/skynet-installation-help.88080/

Disable password ssh-authentication to router. enable only key based authentication. Save SSH key to KeepassXC & Strongbox.

In order to save power and resources, disable unnecessary services by doing:

$ sudo systemctl disable wpa_supplicant

$ sudo systemctl disable bluetooth

$ sudo systemctl disable hciuart

$ echo "dtoverlay=pi3-disable-bt" | sudo tee -a /boot/config.txt

$ echo "dtoverlay=pi3-disable-wifi" | sudo tee -a /boot/config.txt

$ sudo reboot

After a reboot, wlan and bluetooth radios should not be on anymore. You can check this by doing:

$ sudo systemctl status wpa_supplicant.service

$ sudo systemctl status bluetooth.service

$ sudo systemctl status hciuart.service

Failed to start Disable WiFi if country not set

● wifi-country.service - Disable WiFi if country not set
   Loaded: loaded (/lib/systemd/system/wifi-country.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2022-06-23 18:10:20 EEST; 2 weeks 6 days ago
 Main PID: 458 (code=exited, status=1/FAILURE)

As described in r/pihole. Installing this is expected to increase my DNS resolving time, but by how much, needs to be measured. Currently running:

$ dig @127.0.0.1 -p 5053 google.co.uk

gives 19 msec, same as here for a guy with cloudflare DNS. If after implementing and measurement it’s the resolution is of same ballpark, leave the setting in.

Update: It seems to be that DNSCrypt does not bring me any benefits.

• If ever buying a new router, I could equip it with a Raspberry Pi Zero W on router’s powered usb, as a backup DNS-server. This would also result in even smaller power footprint with no cords.
• However, that would be a concious risk as RasberryPi does not negotiate the power with host and I end up with a broken system.
• However, people have been running Rpi Zero W powered directly from routers usb without issues, so this is worth the risk.
• Also, it seems that Asus routers implement RNDIS on their USB lane which Pi Zero would use, I could then potentially use the same lane for its Ethernet and power, disabling wireless from Pi Zero completely.
• And if the raspiBackup system is implemented on Pi Zero with the same monitoring and backup restore settings I now have on Rpi4, the risk is minimized and worth a try.
• Since I then will have two Pi-Holes operating, I’d like to keep their blocklists synchronized.
• Tutorial for running two pi-holes in high availability mode could be achieved manually with this tutorial and in automated fashion with gravity-sync (preferred).
• New router could be Asus RT-AC86U, since it can support also WireGuard loaded in.
• However, people have been getting on 1 Gbit connection only 450 Mbit/s on it, while WireGuard on rpi4 can push ~600 Mbit/s.
• It needs to be tested/googled whether the router could handle faster traffic than 450 Mbit/s (although that is plenty for my connection now).
• There are also other open firmwares out there, e.g. OpenWRT, AdvancedTomato, etc.

Idea, and also experience so far is, that with Debian Sid and apt-listchanges and apt-listbugs would enable a rolling release for me with which I would not need to go through OS reinstall through whenever support ends. Instead, I would do every weekend manual sudo aptitude update & sudo aptitude safe-upgrade, inspect the changelogs, pin packages with bugs, and despite this if something goes wrong, revert to backups taken previous night.

Start by checking Debian RaspberryPi and Debian raspi faq.

Then download tested image (Family 1, 0W).

Continue with these instructions. Do:

Enable remote SSH root login (disable this later once non-root user is setup) # echo "PermitRootLogin yes" >> /etc/ssh/sshd_config

Images come with empty root password, so change that to something else: # passwd root

Set the hostname: # hostnamectl set-hostname

The images come with wireless support, so add to /etc/network/interfaces.d/wlan0:

BEGIN_SRC bash allow-hotplug wlan0 iface wlan0 inet dhcp wpa-ssid MYNETWORK wpa-psk MYPASSWORD END_SRC

Reboot, give enough time for DHCP to handout an IP, then ssh to it and do the rest remotely:

$ ssh root@10.0.1.246

• Set up user pyyhttu
• Set up groups
• apt update && upgrade
• set sources to sid (deb https://ftp.debian.org/debian/ sid contrib main non-free non-free-firmware)
• disable remote root login

For security updates only, if possible. See: https://wiki.debian.org/UnattendedUpgrades

Then turn its repositories to Debian Unstable (Sid). Remember to add non-free firmware. Remember to setup wifi with nmcli instead of wpa_supplicant.

• Conditional forwarding is enabled to display

client names instead of their IPs in Pi-hole admin panel. This would also save me the trouble of having to edit the reservations and hosts file manually when new clients are added to the network.

• In Asus RT-AC86U go to LAN - DHCP Server and define for field “RT-AC86U's Domain Name” a Domain Name (e.g. house).
• Log in to Pi-hole and go to: Settings - DNS and tick “=Use Conditional

Forwarding=“. Define for ”Local network in CIDR notation“: ”10.0.0.0/24“, ”IP address of your DHCP server (router)“: 10.0.1.1 and for ”Local domain name“: house.

• Make sure that “Never forward non-FQDNs” and “=Never forward reverse lookups

for private IP ranges=“ are also ticked; otherwise the clients’ names in Pi-hole dashboard are still displayed with their IPs only ⁷.

• Update [2020-12-06 Sun]: After setting up new router (Asus AC86U) hostnames are now again not displayed in pi-hole

under Tools, Network, but are displayed as IPs. It seems this behavior is router specific.

• So far, my only resolution would be to enable DHCP in pi-hole, instead of the

router, OR, flash both routers (Ai Mesh) with Merlin, as I may be able to disable Asus router’s DNS that way.

• As per this reddit post.

Once my ISP (Elisa) takes IPv6 into use, then there’s is a potential slow down scenario with pi-hole:

Some ISPs do not hand out static IPv6 addresses. So if you configured Pi-hole with an IPv6 address during installation and that address is later changed by your ISP, you now run into the problem the wrong (i.e. an invalid) IPv6 address in gravity.list. Because Pi-hole looks to this file to find out where to go, your computer cannot find that IP address and the requests time out, causing the long loading times.

Eventually I need to take IPv6 into use and set it up in my router’s settings.

Once done, I need to retest my IPv6 readiness at http://test-ipv6.com/.

None of the rpi3’s have a built-in hardware clock ⁸, so they rely on a network time server to get the correct time.

Unbound requires the system clock to be pretty close to real time or it cannot resolve DNS queries.

All of the default NTP servers are setup as fully qualified hostnames, so in in an event of downtime, the clock will be off. Thus, after restart unbound can not resolve the hostnames of the ntp servers.

To prevent this in the future, add several hard coded IP addresses to /etc/systemd/timesyncd.conf. But first, check that your time is correct:

$ timedatectl

         Local time: Sun 2020-03-08 16:54:30 EET
     Universal time: Sun 2020-03-08 14:54:30 UTC
           RTC time: n/a
          Time zone: Europe/Helsinki (EET, +0200)
System clock synchronized: yes
        NTP service: active
    RTC in local TZ: no

$ systemctl status systemd-timesyncd.service

● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
     └disable-with-time-daemon.conf
Active: active (running) since Sat 2020-02-29 13:27:19 EET; 1 weeks 1 days ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 335 (systemd-timesyn)
Status: "Synchronized to time server for the first time 95.216.24.230:123 (2.debian.pool.ntp.org)."
Tasks: 2 (limit: 2200)
Memory: 1.4M
CGroup: /system.slice/systemd-timesyncd.service
     └335 /lib/systemd/systemd-timesyncd

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Add the NTP server IPs:

$ sudo nano /etc/systemd/timesyncd.conf

[Time]
NTP=162.159.200.123
FallbackNTP=185.112.82.30 95.216.160.182 95.217.10.168
#RootDistanceMaxSec=5

Above IP-addresses are of fi.pool.ntp.org specific pool zone: https://www.ntppool.org/zone/fi

Ensure time synchronize is enabled:

$ sudo timedatectl set-ntp true

Restart the NTP Daemon:

$ sudo systemctl daemon-reload

Instructions as per this reddit thread.

1. In iOS, go to Settings, Shortcuts and enable Allow Untrusted Shortcuts.
2. With iOS browser, go to https://routinehub.co/shortcut/5005 and download the shortcut.
3. Edit it in Shortcuts and scroll down to section DICTIONARY, and add under settings values:

API Key can be found in Pi-hole Settings, API / Web interface and Show API token.

• As per instructios here.

Reasoning in this Reddit thread. tl;dr: In a recent update, the Raspberry Pi Foundation installed a Microsoft vscode apt repository on all machines running Raspberry Pi OS.

Another OS could be Debian (thought it has a performance penalty compared to Raspberry Pi OS) and non-free firmware.

Do this for the next version image: https://lobste.rs/s/gls60k/uefi_secure_boot_on_raspberry_pi

If ISP uses CGNAT (Carrier-Grade NAT), then I will not have a publicly-reachable IP address on my home network. This limits my ability to run a private VPN, since I don’t have a direct gateway to the public Internet.

To verify this, I need to check public IP, also known as WAN IP, either from my Asus RT-AC86U router dashboard at:

General - Network Map, or by running:

$ curl ifconfig.me

If either of the methods return an IP that is of range 100.xxx.xxx.xxx or 10.xxx.xxx.xxx or 192.xxx.xxx.xxx, then it is an indication ISP uses CGNAT.

For ultimate verification, run traceroute to inspect the amount of network hops:

$ curl ifconfig.me | xargs traceroute

I have a result of a single hop in the results, which means I’m not behind CGNAT:

$ curl ifconfig.me | xargs traceroute
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13  100    13    0     0     68      0 --:--:-- --:--:-- --:--:--    68
traceroute to 88.115.55.164 (88.115.55.164), 30 hops max, 60 byte packets
 1  88-115-55-102.elisa-laajakaista.fi (88.115.55.102)  0.447 ms  0.272 ms  0.215 ms

If there is more than single hop, I’m behind ISPs CGNAT and I need to request a publicly routable IP address, or instead of self-hosting the WireGuard VPN, consider using a service like Tailscale, Cloudflare Tunnel, or Twingate.

As per instructions here. Update: Rewrite these instructions (check the link) since September 2021, WireGuard package has been available in RasperryPI OS repository.

$ echo "deb http://deb.debian.org/debian/ unstable main" | sudo tee --append /etc/apt/sources.list.d/unstable.list

$ wget -O - https://ftp-master.debian.org/keys/archive-key-$(lsb_release -sr).asc | sudo apt-key add -

$ printf 'Package: *\nPin: release a=unstable\nPin-Priority: 150\n' | sudo tee --append /etc/apt/preferences.d/limit-unstable

$ sudo aptitude update

• After this, loads of new package updates are available from Sid, something

like:

Current status: 12756 (+12755) new.

• Now I have the latest version of WireGuard (0.0.20190601-1) available from debian sid repositories, install that with dependencies:

$ sudo aptitude install WireGuard

• Keys are generated for WireGuard server and a client:

$ sudo su

# cd /etc/WireGuard

# umask 077

# wg genkey | tee client1_privatekey | wg pubkey > client1_publickey

# wg genkey | tee server_privatekey | wg pubkey > server_publickey

• Four keys are now generated.

# nano /etc/WireGuard/wg0.conf

• wg0.conf should contain:

[Interface]
Address = 10.9.0.1/24
ListenPort = 51819
PrivateKey = {server_privateKey}
DNS = 10.0.1.245

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
#Client1
PublicKey = {client1_publicKey}
AllowedIPs = 10.9.0.2/32, 10.0.1.245/32
PersistentkeepAlive = 60

Where,

Address

Server’s internal IP address to bypass the traffic through WireGuard. We get to assign it. Let’s use 10.9.0.1/24. This should be assigned outside of router’s DHCP IP pool range.

ListenPort

In order to access home network from outside, configure a port forward. This is the external UDP port I’ve forwarded in router (WAN - Virtual Server/Port Forwarding). WireGuard’s default port is 51820, but we’ll opt on using 51819. Nondefault listenPort is handy, if e.g. using public wifi. Furthermore, choosing port below 1024 may help circumventing blocking established by various providers. E.g. using port 989 (FTPS).

PrivateKey

Value from $ cat server_privatekey

PostUp & PostDown

Change eth0 to wlan0 in both lines if raspberry pi with pi-hole is connected via Wi-Fi, otherwise eth0 (this is presumed).

PublicKey

Value from $ cat client1_publickey

AllowedIPs

Client-IP/32 (keep it as is), Pi-hole’s IP

PersistentkeepAlive

this line is uncommented (remove the #) to keep the connection alive as I’m behind a NAT.

• The following config (client1.conf) goes ultimately to my iOS WireGuard, but we configure the file on rpi3 and then export it using qrencode, as it is faster this way.
• Create the config file by doing:

# nano /etc/WireGuard/client1.conf

• client1.conf should contain:

 1: [Interface]
 2: Address = 10.9.0.2/32
 3: DNS = 10.0.1.245
 4: PrivateKey = {client1_privatekey}
 5: 
 6: [Peer]
 7: PublicKey = 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=
 8: Endpoint = 88.115.55.164:51819
 9: AllowedIPs = 0.0.0.0/0, ::/0
10: #PersistentkeepAlive = 60

Where,

• Line 2 = Assigned IP of client
• DNS = Pi-Hole’s IP
• Privatekey = value from $ cat client1_privatekey
• Publickey = value from $ cat server_publickey

• Endpoint = public-ip : WireGuard-port-forwarded-in-router. Current public-ip value can be checked by doing

  $ curl ifconfig.me

AllowedIPs = 0.0.0.0/0, ::/0 (where 0.0.0.0/0, ::/0 allows all traffic to route through WireGuard aka full access to my home network). Note: ::/0 or ::0/:) are the same thing; they’re there to prevent ipv6 leakage.

OR Instead of full tunnel, if I wanted a split tunnel:

AllowedIPs

88.115.55.164/32 10.9.0.1/24 (where 10.9.0.1/24 is the subnet of my tunnel, and 88.115.55.164/32 is my public IP. Thus, this split tunnel setup would allow traffic only from iOS WireGuard tunnel to my home computer).

# sudo su

$ sudo aptitude install qrencode

$ sudo qrencode -t ansiutf8 < /etc/WireGuard/client1.conf

• A QR code will be generated, you will need to scan this code and import it to the WireGuard app on the phone by tapping Add Tunnel and then Create from QR Code.
• After that, your iOS WireGuard tunnel is created with the following settings:

https://pyyhttu-siilo.kapsi.fi/nextcloud/index.php/s/Photo_2019-07-15_1371/preview

• Enable WireGuard to autostart on boot:

$ systemctl enable wg-quick@wg0

• Secure the contents of /etc/WireGuard as it contains your private keys:

$ sudo chown -R root:root /etc/WireGuard/

$ sudo chmod -R og-rwx /etc/WireGuard/*

• Go to Pi-Hole Admin console at Settings - DNS and tick mark the following then save:

Listen on all interfaces (Allows only queries from devices that are at most one hop away (local devices)

• Then start WireGuard by doing:

$ sudo wg-quick up wg0

1. Run a DNS leak test by browsing to dnsleaktest.com and executing there “Standard test”. Public IP should be visible.
2. In the iOS app, check that data is exchanging in lines “Data sent” and “Data received” while connected.

On WireGuard server side traffic can be seen by inspecting the WireGuard routing table and peer connections with:

$ sudo wg show

interface: wg0
  public key: 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=
  private key: (hidden)
  listening port: 51819

peer: /3pz2xGhL+jTs3h9+nQ6fTeokvIPduj22afN1LYDn20=
  endpoint: 85.76.86.141:31960
  allowed ips: 10.9.0.2/32, 10.0.1.245/32
  latest handshake: 1 minute, 4 seconds ago
  transfer: 693.34 KiB received, 23.40 MiB sent
  persistent keepalive: every 1 minute

The “endpoint” ip of 85.76... above indicates my phone is attached to cellular network while WireGuard tunnel is on. If I was in my home wlan with WireGuard, the endpoint ip would be 10.0.1.1.

1. Check the ‘Query Log’ page in Pi-hole’s Admin console. You’ll see queries

coming from the wg client IP (10.9.0.2 in this case).

1. Check status of WireGuard:

root@raspberrypi:/etc/WireGuard# systemctl status wg-quick@wg0.service
● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2019-07-01 18:31:53 BST; 2 weeks 0 days ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.WireGuard.com/
           https://www.WireGuard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
 Main PID: 604 (code=exited, status=0/SUCCESS)

Jul 01 18:31:51 raspberrypi systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
Jul 01 18:31:51 raspberrypi wg-quick[604]: [#] ip link add wg0 type WireGuard
Jul 01 18:31:51 raspberrypi wg-quick[604]: [#] wg setconf wg0 /dev/fd/63
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip -4 address add 10.9.0.1/24 dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip link set mtu 1420 up dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] resolvconf -a tun.wg0 -m 0 -x
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip -4 route add 10.0.1.245/32 dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j AC
Jul 01 18:31:53 raspberrypi systemd[1]: Started WireGuard via wg-quick(8) for wg0.

Traffic should be routed through pi-hole. Check that by doing:

$ ip r | grep default

default via 10.0.1.1 dev eth0 src 10.0.1.245 metric 202

$ ip address show wg0

4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.8.0.1/24 brd 10.8.0.255 scope global wg0
       valid_lft forever preferred_lft forever
    inet 10.9.0.1/24 scope global wg0
       valid_lft forever preferred_lft forever

“Sadly, due to overhead, you’ll get a little less bandwidth than if you were on the network directly. But bandwidth is adequate for my purposes, assuming I’m on a stable Internet connection.

You can test the overhead from the VPN connecting while you’re on your home LAN. Try this, and run a speed test on https://speed.cloudflare.com, once without the VPN connected, and once connected. Compare the two speeds and that’s the overhead of the VPN connection.

On my LAN, the download speed goes from around 600 Mbps to 237 Mbps, but upload remains close to the measly 35 Mbps Spectrum gives me.

And connected through my iPhone on AT&T over a pretty poor signal (2/4 bars), I can get 32 Mbps down and 4 up through the VPN—not too bad! That’s a little under line speed on the iPhone through AT&T directly, but it’s workable, and has saved me a few times when I needed to grab something from the LAN remotely!“

Source: https://www.jeffgeerling.com/blog/2023/build-your-own-private-WireGuard-vpn-pivpn

On server side 10.0.1.245 create new key pair for the client to be added:

$ sudo su

# cd /etc/WireGuard && wg genkey | tee client2_private_key | wg pubkey > client2_public_key

Then register them on the server:

# wg set wg0 peer <client2_public_key> allowed-ips <new_client_vpn_IP>/32

Where,

On client side, install WireGuard on the client OS. Depending on the OS, apply the installation instructions from official website. On Ubuntu based distro:

# su pi

$ sudo add-apt-repository ppa:WireGuard/WireGuard

$ sudo apt-get update

$ sudo apt-get install WireGuard-dkms WireGuard-tools resolvconf linux-headers-$(uname -r)

On client OS side, create wg0-client.conf with:

[Interface]
Address = 10.9.0.3/32
PrivateKey = <insert_client2_privatekey>
DNS = 10.0.1.245

[Peer]
PublicKey = <insert server_publickey>
Endpoint = <insert_vpn_server_address>:51819
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 60

On client side, bring up the VPN interface and form a tunnel by doing:

$ sudo wg-quick up wg0-client

Tunnel should now be active. Check that by doing:

$ sudo wg show

To disconnect do:

$ sudo wg-quick down wg0-client

Use WireGuard Config Generator or WireGuard-Tools Script.

• If needed to revoke the client from the server, remove the [Peer] block related to your revoked client from the server’s configuration wg0.conf, then restart your VPN server with:

$ wg-quick down wg0

$ wg-quick up wg0

• This is to be done when new kernel-headers get updated after doing $ sudo aptitude safe-upgrade
• After updating the headers, purge both WireGuard and WireGuard-dkms:

$ sudo aptitude purge WireGuard WireGuard-dkms

• Then reinstall them:

$ sudo aptitude install WireGuard WireGuard-dkms

• Lastly reboot with $ sudo reboot and check tunnel status with $ wg show

Updated Asus stock firmware to newest version (3.0.0.4.386_45956) but after this none of the clients using WireGuard VPN worked. This was evident from ever increasing time stamps of latest handshakes:

$ sudo wg show

pyyhttu@raspberrypi:~ $ sudo wg show
interface: wg0
  public key: 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=
  private key: (hidden)
  listening port: 51819

peer: /3pz2xGhL+jTs3h9+nQ6fTeokvIPduj22afN1LYDn20=
  endpoint: 10.0.1.1:1026
  allowed ips: 10.9.0.2/32
  latest handshake: 17 minutes, 50 seconds ago
  transfer: 8.79 GiB received, 45.61 GiB sent
  persistent keepalive: every 1 minute

peer: guoJge3walbBMS+PMWVKpBDPkM7JaIxIcrb4xlBfnzU=
  endpoint: 10.0.1.1:1025
  allowed ips: 10.9.0.4/32
  latest handshake: 18 minutes, 33 seconds ago
  transfer: 610.86 MiB received, 5.83 GiB sent

peer: P0eDIPcZPSuELfma6DScKSriYXn/N/JzmeHUkHfqrz8=
  endpoint: 10.0.1.1:1024
  allowed ips: 10.9.0.3/32, 10.0.1.245/32
  latest handshake: 24 minutes, 23 seconds ago
  transfer: 264.57 MiB received, 3.75 GiB sent
  persistent keepalive: every 1 minute
pyyhttu@raspberrypi:~ $

Resolved by restarting WireGuard by doing:

$ sudo wg-quick down wg0 && sudo wg-quick up wg0

Noticed a connection break on [2022-04-04 Mon]. This was also registered by Monit and healthchecks.io. Then noticed that my iOS with WireGuard enabled (on-demand activated), was not able to resolve anything.

iOS WireGuard logs showed an error around the connection break:

Removing all keys, since we haven't received a new one in 540 seconds

Restarting the wg0 service on server by doing $ sudo wg-quick down wg0 && sudo wg-quick up wg0 did not help.

Decided to recreate the iOS client keys so that first deleted the keys in iPhone WireGuard (delete tunnel) and then on server recreated the client keys at /etc/WireGuard/ by doing:

# qrencode -t ansiutf8 < /etc/WireGuard/client3.conf

After scanning the qr-code and receiving the keys, the iOS WireGuard started to work again.

• Then repeat and document the connection speed test as per the guide here.
• Results: It seems to be that the result are on par with what was presented in the link.
• Rpi3’s network adapter with my current connection speed is more than enough to act as a WireGuard server.

• In order to connect to my public home IP where rpi3 WireGuard VPN is running, I need to tie that IP to a dynamic name service (ddns).
• My public home IP changes all the time and thus forces me to update it to server side /etc/WireGuard/client1.conf as well as to WireGuard app on client side.
• Register to a dynamic DNS provider, e.g. to dynu.com.
• Choose Option 1: Use our Domain.
• Define wanted domain to be used (e.g. rpivpn.mywire.org) and click Add.
• Go to Control Panel - My Account and define your Time Zone.
• In rpi3 terminal, test your domain by doing

$ nslookup rpivpn.mywire.org

This should return the same IP as shown in Dynu Control Panel - DDNS Services.

• In rpi3 terminal, install ddclient, which will recognize when your public IP changes, and updates this to point to rpivpn.mywire.org:

$ sudo aptitude install ddclient libio-socket-ssl-perl

• Bypass the setup wizard in every step by pressing esc. Setup ddclient manually:

$ sudo nano /etc/ddclient.conf:

daemon=300
syslog=yes
ssl=yes
protocol=dyndns2
use=web, web=checkip.dynu.com/, web-skip='IP Address'
server=api.dynu.com
login=your-dyny-login
password='your-dynu-password'
rpivpn.mywire.org

• Test IP update works to dynu.com by forcing an update manually:

$ sudo ddclient -force -verbose

• This update should be registered in dynu control panel (observable by the updated time stamp).
• Automate this update to be every 5 minutes:

$ sudo nano /etc/default/ddclient

run_dhclient="false"
run_ipup="false"
run_daemon="true"
daemon_interval="300"

• Start ddclient as a service:

$ sudo service ddclient start

• Check the service is started correctly as active: (running) by doing:

$ sudo service ddclient status

• Test the automated update service works. Change IP to something to 1.2.3.4 in Dynu control panel at DDNS Services.
• Delete in rpi3 the ddclient.cache:

sudo rm /var/cache/ddclient/ddclient.cache

• Wait for 5 minutes after which ddclient should’ve updated the dummy IP 1.2.3.4 back to correct.
• Now client1.conf and client2.conf can be updated with dynamic dns rpivpn.mywire.org and distributed to clients with:

sudo qrencode -t ansiutf8 < /etc/WireGuard/client1.conf

• As per the article here.

• Use this python-package.

Document then measure with this.

• Check current version with:

$ sudo cloudflared -v

cloudflared version 2019.12.0 (built 2019-12-19-2256 UTC)

• Then run update:

$ sudo cloudflared update

INFO[0017] cloudflared has been updated to version 2020.2.0

• In case there isn’t an update, one gets:

2021-04-24T08:44:50Z INF cloudflared is up to date version=

• Otherwise the update is applied. Notification of new updates can be gotten by subscribing to Cloudflared release channel.

• After did $ sudo reboot, DNS resolution does not work. For example doing $ ping google.com gives:

ping: google.com: Temporary failure in name resolution

• Also doing $ sudo monit status gives “Connection failed” on remote host test.
• Both are because we use Cloudflared for DSN resolution but Pi-Hole automatically adds 127.0.0.1 to /etc/resolvconf/run/resolv.conf. You can see the file’s timestamp changed after reboot by doing

$ ls -al resolvc.conf

• So if using Cloudlflare for DNS resolution, then update /etc/resolv.conf back to 1.1.1.1
• If/when going to unbound, this should be no longer be an issue as then an entry of 127.0.0.1 in resolv.conf is actually correct.

Edit: [2021-08-28 Sat] Contents of resolv.conf is actually (in working setup):

nameserver 10.0.1.245
nameserver 127.0.0.1

• After setting up Cloudflared, noticed after a while by visiting their test site that it indicated Using DNS over HTTPS (DoH) as No.
• Reason turned out to be the DNSSEC setting in pi-hole I had turned on, as per this reddit-thread. After disabling DNSSEC, the test site indicated correctly I have DoH enabled.

When doing: $ sudo cloudflared update I get:

failed to update cloudflared: context deadline exceeded (Client.Timeout or
context cancellation while reading body)

Resolved this with instructions from github issue 253.

• When trying to update cloudflared with $ sudo cloudflared update I get:

pi@raspberrypi:~ $ sudo cloudflared update
2021-08-28T15:32:10Z INF cloudflared is up to date version=
pi@raspberrypi:~ $

• Also $ cloudflared -v does not give version info but:

pi@raspberrypi:~ $ cloudflared -v
cloudflared version DEV (built unknown)
pi@raspberrypi:~ $

• To resolve this, redownload the binary from the same link, and repeat the install procedure (same as manually updating binary.
• After this, doing $ cloudflared -v instead of giving “built unknown”, I now get correctly:

pi@raspberrypi:~ $ cloudflared -v
cloudflared version 2021.8.6 (built 2021-08-27-2250 UTC)

• DNSSEC provides validation that DNS responses are untampered and can be trusted. One such test can be found from Cloudflare’s Browsing Experience Security Check but the site apparently sometimes provides false positives, evident also from this post.
• This is evident when disabling "Use DNSSEC" at pi-hole DNS Settings and rechecking the results at Cloudflare’s site gives an OK for DNSSEC.
• A more reliable test would be to use a DNSSEC Resolver Test.

• I’ve noticed a similar symptoms sometimes as reported in this reddit thread.
• Fix: If/when reinstalling Cloudflared, Instead of using /etc/default/cloudflared, make yourself a Cloudflared yaml config file.
• This is to be done if not going with Unbound.

• Inspect the Unbound configurations to improve its performance.
• If problems with installation, consult the reddit thread.
• If no problems, remove this and above.

• Check that Cloudlflare test gives with Firefox everything checked, and with iOS + VPN the top three checked (Encrypted SNI unchecked as not supported by mobile Safari).

• Check if root-dns needs to be installed or will up-to-date root-dns package follow with Debian Unbound package.

• Double check after installation that unbound is not running as a forwarding resolver, but as recursive resolver.

By doing

$ dig google.com @127.0.0.1 -p 5353

For more information see this.

With cloudflare DNS doing $ dig google.com now returns:

$ dig google.com @127.0.0.1 -p 5353

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> google.com @127.0.0.1 -p 5353
;; global options: +cmd
;; connection timed out; no servers could be reached

As per the test in this Reddit thread. The recommendation is to have the DNSSec of when using Unbound.

• Acquire USB-dongle and Pi Zero W with SD-card and put the kit together.
• Install Pi Imager.
• With it, write minimal Rasbian OS (Raspberry Pi OS Lite) on the SD card.
• The Raspberry Pi OS images no longer have SSH enabled by default, but it’s easy to enable it; place a file named ssh (without any extension) onto the boot-partition of the SD card after you have written the disk image.
• Also enable the wireless for Raspberry Pi Zero W by creating to same boot partition a file called wpa_supplicant.conf with:

country=FI
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
        ssid=YOUR_WIFI_NAME
        psk=YOUR_WIFI_PASSWORD
        key_mgmt=WPA-PSK
}

• Attach RPi Zero W to your router’s usb port.
• Change Raspberry’s IP to static by logging in to your router (in Asus RT-AC86U go to: LAN - DHCP Server: Enable Manual Assignment: Yes).
• Select raspberrypi from the client’s list and bind it to: 10.0.1.246.
• Once done, ssh into raspberry:

$ ssh pi@10.0.1.246

Ensure that pi user can log in also in the future SSH sessions. Add user pi to the ssh user group:

$ sudo adduser pi ssh

Leaving the default user pi, with the default password is a security risk. Add your own user, passwordless login is established later:

$ sudo adduser pyyhttu

Adding user `pyyhttu' ...
Adding new group `pyyhttu' (1002) ...
Adding new user `pyyhttu' (1001) with group `pyyhttu' ...
Creating home directory `/home/pyyhttu' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for pyyhttu
Enter the new value, or press ENTER for the default
        Full Name []: Tuomas Pyyhtiä
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
chfn: name with non-ASCII characters: 'Tuomas Pyyhtiä'
Is the information correct? [Y/n] Y

Check groups pi users is associated to:

$ groups

pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh gpio i2c spi

Add user pyyhttu to all the same groups:

$ sudo usermod -a -G pi,adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,ssh,gpio,i2c,spi pyyhttu

pyyhttu : pyyhttu pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh spi i2c gpio

Change hostname. This is good practice to do in case we have multiple machines so we know which is which:

$ sudo nano /etc/hostname

Delete the old name and setup new name.

$ sudo nano /etc/hosts

Replace any occurrence of the existing computer name with your new one.

Reboot the system for changes to take effect:

$ sudo reboot

Wait couple of minutes for reboot to finish, then log in as the new user, and delete the old pi user:

$ ssh pyyhttu@10.0.1.246

$ sudo deluser pi

Removing user `pi' ...
Done.

Now we will allow our new user to execute sudo without providing a password, because password fatigue is a thing (at least for me). First, we delete the sudoers config for the user pi since we already deleted that user:

$ sudo rm /etc/sudoers.d/010_pi-nopasswd

Then we make a new file for the new user:

$ sudo nano /etc/sudoers.d/pyyhttu-nopasswd and with content:

pyyhttu ALL=(ALL) NOPASSWD: ALL

Change permission for that file:

$ sudo chmod 440 /etc/sudoers.d/pyyhttu-nopasswd

Before uninstalling Cloudflared and installing Unbound, test with a shell script the performance of the most popular DNS resolvers from your location with dnsperftest:

$ git clone --depth=1 https://github.com/cleanbrowsing/dnsperftest/

$ cd dnsperftest

$ bash ./dnstest.sh

$ bash ./dnstest.sh |sort -k 22 -n

                  test1   test2   test3   test4   test5   test6   test7   test8   test9   test10  Average
127.0.0.1         1 ms    18 ms   1 ms    1 ms    23 ms   1 ms    1 ms    1 ms    1 ms    1 ms      4.90
cloudflare        14 ms   14 ms   14 ms   31 ms   15 ms   14 ms   14 ms   14 ms   14 ms   15 ms     15.90
quad9             17 ms   18 ms   17 ms   18 ms   28 ms   43 ms   18 ms   18 ms   17 ms   18 ms     21.20
yandex            14 ms   14 ms   15 ms   15 ms   42 ms   73 ms   15 ms   14 ms   15 ms   40 ms     25.70
google            20 ms   19 ms   29 ms   29 ms   40 ms   65 ms   19 ms   29 ms   19 ms   29 ms     29.80
cleanbrowsing     35 ms   37 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms     35.20
adguard           36 ms   37 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   36 ms   35 ms     35.40
comodo            36 ms   46 ms   37 ms   38 ms   36 ms   36 ms   36 ms   36 ms   36 ms   39 ms     37.60
norton            20 ms   20 ms   20 ms   20 ms   21 ms   20 ms   20 ms   20 ms   20 ms   211 ms    39.20
opendns           29 ms   29 ms   29 ms   49 ms   30 ms   145 ms  28 ms   48 ms   29 ms   29 ms     44.50
neustar           51 ms   48 ms   48 ms   55 ms   38 ms   38 ms   38 ms   48 ms   38 ms   52 ms     45.40
level3            47 ms   48 ms   46 ms   47 ms   47 ms   47 ms   47 ms   46 ms   47 ms   47 ms     46.90
freenom           20 ms   83 ms   20 ms   261 ms  201 ms  167 ms  52 ms   243 ms  20 ms   86 ms     115.30

Run the test couple of times to see the actual results, note them down, then proceed to uninstall Cloudflared:

• Here: https://thesmashy.medium.com/building-a-pihole-for-privacy-and-performance-f762dbcb66e5

See linked screenshot in here.

• Instructions here.

• and installing more up-to-date Unbound. Setup with cron to update the root.hints every 6 months with something like $ wget https://www.internic.net/domain/named.root -O /var/lib/unbound/root.hints

• by doing $ netstat -tunlp that cloudflare is not showing activity on ports 44685 tcp and 54 tcp and udp, as per these instructions.

• Register a hobbyist account and setup a project to get an unique url to ping against.

• Nothing.

• Setup an automated ping:

$ crontab -e

• And then in cronjob define a ping every second minute by adding a line:

0/2 * * * * dig @127.0.0.1 pi.hole && curl -fsS --retry 3 "https://hc-ping.com/your-unique-url"

$ sudo aptitude install monit

• At https://healthchecks.io/checks/“your-unique-url”details schedule a period for two minutes and grace for 1 minute. Grace: If check is late, service waits 1 minute before sending me a notification.
• Define notification method as email and SMS.

$ sudo nano /etc/monit/monitrc

• Uncomment the following three lines first:

set httpd port 2812 and
use address localhost
allow localhost

Otherwise $ sudo monit status would give an error:

Cannot create socket to [localhost]:2812 -- Connection refused.

• To monitor pihole-FTL, append at the end of monitrc:

1: check process pihole-FTL with pidfile /var/run/pihole-FTL
2:   start program = "/etc/init.d/pihole-FTL start" with timeout 20 seconds
3:   if failed port 4711 type tcp with timeout 5 seconds for 2 cycles then restart
4:   if 4 restarts within 5 cycles then unmonitor

Where,

• process: Pi-Hole’s IP

Save changes. Check that monitrc is correct, do $ sudo monit -t. Then start monit with $ sudo monit.

Monit will detach from the terminal and run as a background process, i.e. as a daemon process. As a daemon, Monit runs in cycles. It monitors services, then goes to sleep for a configured period, then wakes up and start monitoring again in an endless loop. If pihole-FTL is found to be down, it will be restarted automatically and findings are logged into /var/log/monit.log.

• If any changes are needed to be done to /etc/monit/monitrc, then make necessary modification and do:

$ sudo monit -t && sudo service monit restart

If there are finding, you’d like to receive an email. We’ll use gmail for sending those notifications.

Visit first your Google accounts security page and enable their 2-step verification.

Then visit your Google account App passwords and generate a password for monit to use.

Lastly, do $ sudo nano /etc/monit/monitrc and define:

set alert where.alert-is-sent@domain.com not on { instance, action } # do not receive alerts when monit starts/stops

set mailserver smtp.gmail.com port 587
        username "your-gmail-address@gmail.com" password "your-generated-google-app-password-for-monit"
        using tls
        with timeout 30 seconds

• Rebooting or backing up rpi3 with raspiBackup, will result in alert emails triggered by monit daemon with topics such as: “monit alert – Connection failed pihole-FTL” or “monit alert – Connection failed google.com”. The same emails are received from healthchecks.io.
• For monit, this is because monit starts to check services immediately after reboot, while the given services may not in fact be running yet.
• This can be resolved by editing /etc/monit/monitrc, define there with start delay 120 to give 2 minute delay before first check is triggered after reboot.
• For healthchecks.io this is more complicated, since it is a “dead man’s switch” based service. Upon investigating on solutions, such as Tronitor, controlled pausing can be done, but it would involve combining Tronitor with HealthChecks-Linux, which has dependency to Organizr setup relying on a reverse proxy, which then in turn requires installing and configuring NGINX… This too many dependencies, so we’ll live with healtcheck false positives for now: they’re fully predictable, scheduled to happen inside 3 min window, once a week when raspiBackup runs.

Use dockerized internet monitoring using Prometheus and Grafana. Do this once migrated over to RPi4.

The Raspberry Pi 3 has an SPI (Serial Peripheral Interface) bus to which the display is connected. See Waveshare wiki for more detailed installation instructions.

SPI is disabled by default, as is autologging which is needed for PADD display to emerge automatically after reboot. To enable both:

$ sudo raspi-config

Then select the following from the menu:

3 Boot Options
    B1 Desktop / CLI
        B2 Console Autologin

5 Interfacing Options
    P4 SPI

$ git clone https://github.com/waveshare/LCD-show.git && cd LCD-show/

$ chmod +x LCD35B-show-V2

$ ./LCD35B-show-V2

Try next time with $ ./LCD35B-show-V2 lite as instructed.

$ cd ~ && wget -N https://raw.githubusercontent.com/pi-hole/PADD/master/padd.sh

$ chmod +x padd.sh

Set PADD to auto run by adding the following to the end of ~/.bashrc:

# Run PADD
# If we’re on the PiTFT screen (ssh is xterm)
if [ "$TERM" == "linux" ] ; then
  while :
    do
        ./padd.sh
        sleep 1
    done
fi

Then reboot:

$ sudo reboot

$ sudo dpkg-reconfigure console-setup

UTF-8 -> Guess optimal character set -> Terminus -> 8x14

Running $ ./padd.sh locally after a PADD update spawns a new PADD, but not on the PADD display. Display still shows “Updates available”. Also reloading the ~/.bashrc by doing $ source ~/.bashrc does not help either. The only way to get PADD running in the display is to issue $ sudo reboot and let the .bashrc to handle the update. Edit: I could try next to tweak term in .bashrc as per issue 47 or issue $ ./padd.sh > tty1 as in issue 134. Update [2023-06-18 Sun]: This issue has resolved itself. Noticed when updated pihole with $ pihole -up and the tft screen’s versions were refreshed themselves to latest without reboot.

• Backing up takes in total something like 30 minutes on a first run. Later, the incremental weekly runs will take approximately between 3-6 minutes.

As an option to USB-stick backups, I could backup directly to my home NAS. Another option is to transfer the backup via sshfs to an offsite location, but local backups for now are preferred. This is because sshfs brings these additional complications:

• Network connection would be required. However, offsite backups for this system are not a must and restoring a backup when DNS queries are not resolving (Unbound being down) forms an added complexity.
• rsyncing to sshfs does not support hard linking, resulting in a slightly increased backup size.
• Also Permission errors would needed to be worked around.
• Taking offsite backup with rsync to local NFS/SSHFS on my NAS would take considerably more time than to a local usb-stick –> We want to keep the downtime as low as possible while backups run.
• Though unlikely, but basically admins of ssh hosting provider, have access via backups to my browsing history through /etc/pihole/pihole-FTL.db. This is a privacy concern.
• So local USB-stick attached to rpi3 it is.

I have the USB thumb drive labeled as /dev/sda1. To mount it to /backup:
$ sudo mkdir /backup

$ sudo mount /dev/sda1 /backup

To make sure usb is mounted after rpi3 is restarted, add an entry to fstab:

$ sudo nano /etc/fstab

UUID=edcaf029-b0ab-421b-9d45-23ae1d7ea02a /backup auto nosuid,nodev,nofail 0 0

UUID above is result of: sudo blkid /dev/sda1 | awk -F'"' '{print $2}'

We’ll use gmail for this. First, install a Mail Transfer Agent, we’ll use msmtp as it is easy to set up:

$ sudo aptitude install msmtp msmtp-mta

Configure it:

$ sudo nano /etc/msmtprc

# Gmail specifics
defaults
auth on
tls on
tls_certcheck off
logfile ~/msmtp.log

account        gmail
host           smtp.gmail.com
port           587

from          pi@raspberrypi
user           yourname@gmail.com
password       your-generated-google-app-password-for-mstmtp

# Default
account default : gmail

If you haven’t already, remember to visit Google’s security page, enable 2-step verification, and generate an app password for /etc/msmtprc. Otherwise Google rejects the mails to be sent.

Finally, test that mails are sent with:

$ echo 'test' | msmtp alert.destination@.mail.com

Configure raspiBackup to use this mail:

$ sudo nano /usr/local/etc/raspiBackup.conf

# email to send completion status
DEFAULT_EMAIL="alert.destination@mail.com"

# Send eMail and/or Telegram notification when backup starts
DEFAULT_NOTIFY_START=1

# Send email only in case of errors. Use with care !
DEFAULT_MAIL_ON_ERROR_ONLY=1

Test backup and mail sending with:

$ sudo bash /usr/local/bin/raspiBackup.sh -F

• As per instruction here.

Attach to rpi3’s USB port an SD-card reader, with an SD-card in it. The SD-card doesn’t need to be emptied or formatted; the restore script handles that ¹². Leave the USB-stick in place.

Check that Raspberry Pi OS identifies the new card we’re about to restore with a backup from the USB-stick:

$ fdisk -l | egrep "^Disk /|^/dev"

 1: pi@raspberrypi:~ $ sudo fdisk -l | egrep "^Disk /|^/dev"
 2: Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
 3: Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors
 4: Disk /dev/ram2: 4 MiB, 4194304 bytes, 8192 sectors
 5: Disk /dev/ram3: 4 MiB, 4194304 bytes, 8192 sectors
 6: Disk /dev/ram4: 4 MiB, 4194304 bytes, 8192 sectors
 7: Disk /dev/ram5: 4 MiB, 4194304 bytes, 8192 sectors
 8: Disk /dev/ram6: 4 MiB, 4194304 bytes, 8192 sectors
 9: Disk /dev/ram7: 4 MiB, 4194304 bytes, 8192 sectors
10: Disk /dev/ram8: 4 MiB, 4194304 bytes, 8192 sectors
11: Disk /dev/ram9: 4 MiB, 4194304 bytes, 8192 sectors
12: Disk /dev/ram10: 4 MiB, 4194304 bytes, 8192 sectors
13: Disk /dev/ram11: 4 MiB, 4194304 bytes, 8192 sectors
14: Disk /dev/ram12: 4 MiB, 4194304 bytes, 8192 sectors
15: Disk /dev/ram13: 4 MiB, 4194304 bytes, 8192 sectors
16: Disk /dev/ram14: 4 MiB, 4194304 bytes, 8192 sectors
17: Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors
18: Disk /dev/mmcblk0: 14.9 GiB, 15962472448 bytes, 31176704 sectors
19: /dev/mmcblk0p1       8192    96042    87851 42.9M  c W95 FAT32 (LBA)
20: /dev/mmcblk0p2      98304 31176703 31078400 14.8G 83 Linux
21: Disk /dev/sda: 7.2 GiB, 7759462400 bytes, 15155200 sectors
22: /dev/sda1  *       56 15155199 15155144  7.2G  c W95 FAT32 (LBA)

Where,

• mmcblk0: Old, internal SD card
• mmcblkp1: Old /boot partition
• mmcblkp2: Old /root partition
• dev/sda: Old, external usb-stick, mounted to /backup, containing the backup for restore
• : New, external SD-card, soon to be written with restored backup

To restore:

$ sudo raspiBackup.sh -d /dev/sda/backup/raspberrypi/raspberrypi-rsync-backup-20191228-031201/

More information at raspiBackup’s site.

Check the latest version you have: $ sudo /usr/local/bin/raspiBackup.sh --version

Version: 0.6.5.1 CommitSHA: 9cc17fc CommitDate: 2020-10-31 CommitTime:
21:52:43

To update, do $ sudo bash /usr/local/bin/raspiBackup.sh -U

$ sudo bash /usr/local/bin/raspiBackup.sh -U
--- RBK0031I: Checking whether a new version of raspiBackup.sh is available.
--- RBK0190I: Upgrading raspiBackup.sh from version 0.6.5.1 to 0.6.6.
--- RBK0038I: Are you sure? y/N
y
--- RBK0057I: Downloading file raspiBackup.sh from https://www.linux-tips-and-tricks.de.
--- RBK0072I: /usr/local/bin/raspiBackup.sh updated from version 0.6.5.1 to version 0.6.6. Previous version saved as /usr/local/bin/raspiBackup.0.6.5.1.sh. Don't forget to test backup and restore with the new version now.
--- RBK0241I: Merging current configuration v0.1.3 with new configuration v0.1.4 into /usr/local/etc/raspiBackup.conf.merged.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE=0.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE_DRYRUN=1.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE_OPTIONS="7 4 12 1".
--- RBK0248I: Added option DEFAULT_TELEGRAM_TOKEN="".
--- RBK0248I: Added option DEFAULT_TELEGRAM_CHATID="".
--- RBK0248I: Added option DEFAULT_TELEGRAM_NOTIFICATIONS="F".
--- RBK0248I: Added option DEFAULT_NOTIFY_START=0.
--- RBK0248I: Added option DEFAULT_COLORING="CM".
--- RBK0248I: Added option DEFAULT_EMAIL_COLORING="SUBJECT".
--- RBK0248I: Added option DEFAULT_DYNAMIC_MOUNT="".
--- RBK0249I: Deleted option DEFAULT_APPEND_LOG=0.
--- RBK0249I: Deleted option DEFAULT_APPEND_LOG_OPTION="-a".
--- RBK0249I: Deleted option DEFAULT_RESIZE_ROOTFS=1.
--- RBK0243I: Configuration merge finished successfullly but not activated.
!!! RBK0245W: Backup current configuration in /usr/local/etc/raspiBackup.conf.bak and activate updated configuration? y/N y
--- RBK0240I: Saving current configuration /usr/local/etc/raspiBackup.conf to /usr/local/etc/raspiBackup.conf.bak.
--- RBK0244I: Merged configuration /usr/local/etc/raspiBackup.conf.merged copied to /usr/local/etc/raspiBackup.conf and activated.

To be notified of new updates, subscribe to raspiBackup releases rss-feed.

After running $ sudo raspiBackupInstallUI.sh and selecting P2 UpdateraspiBackupInstallUI.sh, script exits and update passes, but then invoking the script again with $ sudo raspiBackupInstallUI.sh gives an error:

/usr/local/bin/raspiBackupInstallUI.sh: 2:/usr/local/bin/raspiBackupInstallUI.sh: Syntax error: newline unexpected

In this case reinstall the update script with:

$ curl -sLO https://www.linux-tips-and-tricks.de/raspiBackupInstallUI.sh

$ sudo mv raspiBackupInstallUI.sh /usr/local/bin

$ sudo chmod +x /usr/local/bin/raspiBackupInstallUI.sh

$ sudo chown root.root /usr/local/bin/raspiBackupInstallUI.sh

Verify the script now functions with the newest version:

$ sudo /usr/local/bin/raspiBackupInstallUI.sh -h
raspiBackupInstallUI.sh 0.4.3.4, 2020-11-04/22:18:13 - 6c4a442
raspiBackupInstallUI.sh ( -i [-e]? | -u | -U ) [-d]?
-d: enable debug mode
-e: unattended (re)install of raspiBackup extensions
-i: unattended (re)install of raspiBackup
-U: unattended update of raspiBackupInstallUI.sh
-u: unattended uninstall of raspiBackup

• Seemed to be because of issue 443. Fixed in latest commits so updating my script to latest with:

$ sudo bash /usr/local/bin/raspiBackup.sh -U -S (-S flag retrieves the latest commits) fixed the issue.

As reported in issue 264.

Fixed with a workaround by disabling mail colors in /usr/local/etc/raspiBackup.conf:

# Colorize console output (C) and/or email (M)
DEFAULT_COLORING="C"