#+TITLE: Pi-hole for Raspberry Pi 3 Model B+
#+SUBTITLE: With WireGuard, Cloudflared, Monit, healthchecks.io, PADD and raspiBackup
#+DATE: <2020-01-11 Thu>
#+AUTHOR: Tuomas Pyyhtiä
#+EMAIL: pyyhttu+pihole@pm.me
#+DESCRIPTION: Tutorial on how to setup a Raspberry Pi 3 with Pi-hole
#+KEYWORDS: Raspberry Pi 3 rpi3 pi-hole WireGuard rasbian
#+LANGUAGE: en
# Export underscore as _ instead of subscript https://emacs.stackexchange.com/questions/10549/org-mode-how-to-export-underscore-as-underscore-instead-of-highlight-in-html
#+OPTIONS: ^:{}
# highlight in HTML (was: ^:nil), include priority cookies:
#+OPTIONS: toc:nil ':t pri:t
# Rest of options (as explained here: https://orgmode.org/manual/Export-Settings.html):
#+OPTIONS: toc:t num:nil p:nil stat:t tags:t tasks:t tex:t timestamp:t
#+BEGIN_COMMENT
#+PROPERTY: header-args :eval never-export :exports both :results replace
#+END_COMMENT
# Include usable macros from https://github.com/fniessen/org-macros:
#+INCLUDE: ../org-mode/macros/org-macros.setup
#+BEGIN_COMMENT
#+INFOJS_OPT: view:content ltoc:nil path:../org-mode/js/org-info.js
# for more info see: https://orgmode.org/worg/code/org-info-js
#+END_COMMENT
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="../org-mode/css/org.css"/>
#+HTML_LINK_HOME: https://pyyhttu.kapsi.fi
#+HTML_LINK_UP: https://pyyhttu.kapsi.fi/debian
* Pihole
** Philosophy

#+begin_comment
Add to subtitle OS (Debian Sid) you install this for.
#+end_comment

#+BEGIN_QUOTE
The Pi-hole® is a DNS sinkhole [fn:1] that protects your devices from unwanted
content, without installing any client-side software.
#+END_QUOTE
- Why? Because [[https://hbr.org/2020/01/advertising-makes-us-unhappy][Advertising Makes Us Unhappy]]. After that, one can ask him/herself: [[https://shouldiblockads.com/][Should I use an ad blocker?]]
- If the answer is yes, then one can expect [[https://www.reddit.com/r/pihole/comments/j6cshg/wow_it_really_speeds_up_browsing/][faster web browsing]], as advertisement and trackers are blocked on home network-level --> saves bandwidth.
- No need to run separate ad-block browser extensions -> saves resources.
- This makes the infrastructure simple, plus I only need to trust the block list maintainers [fn:3].
- As ads are blocked on a network level, blocking takes place also on various IoT-devices, which otherwise couldn't run adblockers (like smart TVs).
- Pi-hole achieves this essentially by being a /"DNS sinkhole"/. Since the DNS is vital piece of infrastructure, pi-hole setup needs to be reliable, low-latency and easy to troubleshoot. Thus, putting anything else on this same raspberry pi 3 system is a risk, especially when installing on SD-card, as they tend to wear out and become corrupted [fn:2].
- Alternative to pi-hole is [[https://www.reddit.com/r/pihole/comments/kw7y72/pihole_vs_adguard_home/][AdGuard Home]].

** Preparation

#+begin_note
This guide configures Pi-hole with interface being eth0 (ethernet cable). If you install with wlan, you can do that there's [[https://www.reddit.com/r/pihole/comments/lqjzmf/wireless_pihole_vs_wired/][no noticieable difference]], but make the decision now and stick to that decision so you remember to take this into account when defining interfaces (lan or wlan) in this guide. 
#+end_note

Change modem to "[[https://www.reddit.com/r/pihole/comments/hjiv50/need_help_to_find_the_right_modem/][bridged mode]]" (so that it only passes through the traffic) in order to make port forwarding in your actual router simpler, see [[https://elisa.fi/asiakaspalvelu/aihe/laajakaista/ohje/kotiboksi-vmg3925/][Elisa ISPs router instructions]]. Passing through the traffic (port forwarding) is needed if WireGuard setup is wanted.

Install [[https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/][Pi Imager]]. With it, write minimal ([[https://www.raspberrypi.org/software/operating-systems/#raspberry-pi-os-32-bit][Raspberry Pi OS Lite]]) on the SD card. See [[https://www.reddit.com/r/raspberry_pi/comments/kqor9z/my_first_video_of_2021_how_to_install_raspberry/][tutorial video]] of the procedure if needed.

Pi Imager is capable of setting your:

- Hostname, identify your Pi on a network
- SSH on boot, useful for headless and remote projects
- WiFi, setup your WiFi without editing a config file
- Locale, set your language and location

If you prefer to do those steps manually instead of Pi Imager, here are the details:

#+begin_details Optional: Setting hostname\, enabling ssh\, setting pi user password\, configure wifi and setting time zone data

Enable SSH:

Raspberry Pi OS images no longer have SSH enabled by default, but it’s easy to enable it. Open file explorer ({{{kbd(win)}}}-{{{kbd(e)}}}), ensure you have from under menu =View= =File name extensions= enabled (otherwise your file extensions will be with .txt-extensions). Then right click, New > Text Document, name it =ssh= (without any .txt-extension) onto the =boot=-partition of the SD card after you have written the disk image. Or,

  + if on Linux, do ~$ touch ssh~, or,
  + if on Windows with Powershell, do ~$ $Null | Out-File .\ssh~

Then [[https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up][setup raspberry]].

Set pi user password:

Note that ssh user and pass must be [[https://stackoverflow.com/questions/71820854/raspberry-pi-os-bullseye-64bit-ssh-password-not-raspberry-anymore][setup]] since April 2022. Create onto =boot= also a file called =userconf=. Define it with =user:password hash=, where the hash is generated by running:

~$ echo "password" | openssl passwd -6 -stdin~

#+begin_note
*Note:* Hashing password is needed only from Raspberry Pi OS Debian version
11 (Bullseye) onward.
#+end_note

Raspberry starts when power cord is attached. Let it boot, then check its IP from router.

SSH into RPi3 from your host:

~$ ssh user@rpi3.ip.from.router~

and type in the ssh password.

Once in, change the rpi3 hostname. This is good practice to do in case we have multiple machines so we know which is which:

~$ sudo nano /etc/hostname~

Delete the old name and setup new name:

~$ sudo nano /etc/hosts~

Replace any occurrence of the existing computer name with your new one.

Reboot the system to changes take effect:

~$ sudo reboot~

#+begin_note
(/Optional/): =ssh= is disabled by default on Raspberry Pi OS, so enabling
=sshd= as per
[[https://www.raspberrypi.org/documentation/remote-access/ssh/][Raspberry pi documentation]] is a good idea as it easens the administrational tasks considerably (no need to hook up monitor, keyboard etc.).

~$ sudo systemctl status sshd~\\
~$ sudo systemctl enable ssh~\\
~$ sudo systemctl start ssh~\\
~$ sudo systemctl status sshd~\\
#+end_note

#+begin_note
(/Optional/): Also, creating passwordless login for =ssh= as per
  [[https://linux.tips/tutorials/how-to-login-ssh-with-private-key][instructions
  here]] is a good idea, as it it one of those rare security features that
  improves security *and* usability (by reducing password fatigue).
#+end_note

Configure the correct time zone. Date/time needs to be correct; otherwise none of the DNSSEC lookups will work:

~$ sudo dpkg-reconfigure tzdata~
#+end_details

Check the LAN-cable is connected, and boot Raspberry by attaching the power cord. Let it boot, then check its IP from router.

Change Raspberry's IP to static by logging in to your router (in Asus RT-AC86U
router go to: =LAN - DHCP Server: Enable= =Manual Assignment: Yes=). Select
raspberrypi from the client's list and bind it to: =10.0.1.245= [fn:7].

SSH into RPi3:

~$ ssh user@10.0.1.245~

and type in the ssh password.

#+BEGIN_SRC bash
The authenticity of host '10.0.1.245 (10.0.1.245)' can't be established.
ECDSA key fingerprint is SHA256:l/LA0mZ8187cXSazV5b1nNvzRws6+5KfVAm5EJhrCgY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.0.1.245' (ECDSA) to the list of known hosts.
#+END_SRC

Ensure that your user can log in also in the future SSH sessions. Add user to
the ssh user group:

~$ sudo adduser user ssh~

Configure the system for a static IP:

~$ sudo nano /etc/dhcpcd.conf~

#+BEGIN_SRC bash
# Example static IP configuration:
interface eth0
static ip_address=10.0.1.245/24
#static ip6_address=fd51:42f8:caae:d92e::ff/64
static routers=10.0.1.1
static domain_name_servers=127.0.0.1
#+END_SRC

Uncomment the values for =Example static IP configuration= and input your own.
Use =127.0.0.1= for =static domain_name_servers= as it is [[https://discourse.pi-hole.net/t/static-ip-and-dns-name/12090/4][considered best practice.]]

#+begin_note
On Debian OS 12 (bookwork) and later, =dhcp= is no longer used to set up the static ip, instead =nmcli= [[https://www.jeffgeerling.com/blog/2024/set-static-ip-address-nmtui-on-raspberry-pi-os-12-bookworm][is used]].
#+end_note

Save and reboot for new settings to apply. Then ssh to new ip =10.0.1.245=. You can then also verify static IP and DNS by doing:

~$ ifconfig~\\
~$ dig kapsi.fi~

Leaving the default user pi, with the default password =raspberry= is a security risk. Add your own user:

~$ sudo adduser pyyhttu~

#+BEGIN_SRC bash
Adding user `pyyhttu' ...
Adding new group `pyyhttu' (1002) ...
Adding new user `pyyhttu' (1001) with group `pyyhttu' ...
Creating home directory `/home/pyyhttu' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for pyyhttu
Enter the new value, or press ENTER for the default
        Full Name []: Tuomas Pyyhtiä
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
chfn: name with non-ASCII characters: 'Tuomas Pyyhtiä'
Is the information correct? [Y/n] Y
#+END_SRC

Check groups user =pi= is associated to:

~$ groups~
#+BEGIN_SRC bash
pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh gpio i2c spi
#+END_SRC

Add user =pyyhttu= to the same groups:

~$ sudo usermod -a -G pi,adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,ssh,gpio,i2c,spi pyyhttu~

#+BEGIN_SRC bash
pyyhttu : pyyhttu pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh spi i2c gpio
#+END_SRC

Wait couple of minutes for reboot to finish, then log in as the new user, and delete the pi user:

~$ ssh pyyhttu@10.0.1.245~\\
~$ sudo deluser pi~

#+BEGIN_SRC bash
Removing user `pi' ...
Done.
#+END_SRC

Now we will allow our new user to run sudo without providing a password. First, we delete the sudoers config for the user pi since we deleted that user:

~$ sudo rm /etc/sudoers.d/010_pi-nopasswd~

Then we make a new file for the new user:

~$ sudo nano /etc/sudoers.d/pyyhttu-nopasswd~ and with content:

#+BEGIN_SRC bash
pyyhttu ALL~(ALL) NOPASSWD: ALL
#+END_SRC

Change permission for that file:

~$ sudo chmod 440 /etc/sudoers.d/pyyhttu-nopasswd~

** Pre-installation tasks on Raspberry Pi OS, before installing Pi-hole

The following is optional. I prefer to do these steps as they help me to administer the rpi3 later on. If you don't do them, that is fine, but note that you'll then need to use =apt= instead of =aptitude=:

#+begin_details Optional: Setup aptitude\, localepurge\, deborphan\, apt-listbugs and apt-listchanges for pinning

Get aptitude cfg file (including no-recommended installations):

~$ sudo wget -c http://iki.fi/~pyyhttu/debian/aptitude/apt.conf /etc/apt/apt.conf~

Install minimal version of =aptitude= (or continue using =apt=):

~$ sudo apt update~\\
~$ sudo apt install --no-install-recommends aptitude~

Install and run =localepurge= to get rid of locales that we do not use, as well as localized packages and man pages. Install and run =deborphan= that searches for orphaned packages, i.e., which are not required by any other package upon your system:

~$ sudo aptitude update~\\
~$ sudo aptitude install localepurge deborphan~

Install =apt-listbugs= and =apt-listchanges= packages:

~$ sudo aptitude install apt-listbugs apt-listchanges~

When installing new package versions or upgrading the packages, I'm made aware
of grave-serious bugs (=apt-listbugs=) or important changes
(=apt-listchanges=). I can then have =apt-listbugs= to pin the package so that
the new buggy version is not installed. Every midnight =apt-listbugs= queries
the Debian bug database to see if there are new bugs, and/or unpins the package
if the bug has been fixed [fn:10]. These two packages save me from ton of
unnecessary troubleshooting and downtime.

Short tutorial on pinning with =apt-listbugs=:

I've just executed

~$ sudo aptitude update && sudo aptitude safe-upgrade~

Upgrade prompts me there's a "grave-serious" bug ([[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032235][#1032235]]) reported at Debian bug tracker for package =libargon2-1=. Inspecting the bug, I decide I want to stay for a while in the latest working package, thus I "pin" the =libargon2-1= package by issuing ~p libargon2-1~:

#+BEGIN_SRC bash
[...]
Need to get 0 B/47.7 MB of archives. After unpacking 157 kB will be used.
Do you want to continue? [Y/n/?] Y
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
grave bugs of libargon2-1 (0~20171227-0.3 → 0~20190702-0.1) <Outstanding>
 b1 - #1032235 - cryptsetup: libgcc_s.so.1 must be installed for pthread_exit to work
Summary:
 libargon2-1(1 bug)
Are you sure you want to install/upgrade the above packages? [Y/n/?/...] p libargon2-1
The following 1 package will be pinned:
 libargon2-1
Are you sure? [Y/n] Y
libargon2-1 will be pinned. Restart APT session to enable
Are you sure you want to install/upgrade the above packages? [N/?/...] N
,**********************************************************************
,****** Exiting with an error in order to stop the installation. ******
,**********************************************************************
E: Sub-process /usr/bin/apt-listbugs apt returned an error code (10)
E: Failure running script /usr/bin/apt-listbugs apt

Current status: 25 (-1) upgradable.
pyyhttu@raspberrypi:~ $
#+END_SRC

I then restart APT session by redoing

~$ sudo aptitude update && sudo aptitude safe-upgrade~

and upgrade is this time completed, without the now pinned =libargon2-1= package.

Later, =libargon2-1= has been fixed, so I do:

~$ sudo nano /etc/apt/preferences.d/apt-listbugs~

And delete the =Package=, =Pin=, and =Pin-Priority= lines, along with all the
corresponding explanation lines, and relaunch APT session (update & upgrade).
This is handy when running cutting edge releases in rolling releases manner,
like Debian Sid. Benefits are, that I have always the newest package versions
with latest features, but with some last line of defences in place to avoid
breaking my system.

But systems do break.

Especially volatile, rolling development releases such as Debian sid, as not everything is captured by people reporting bugs.

What to do then?

Couple of options, usually in this order:

1) Report the bug, and wait it out for the maintainers to fix the issue.

2) Downgrade the broken package with apt to its previous version:

~$ sudo aptitude install <package-name>=<package-version-number>~ OR to its previous target release:

~$ sudo aptitude -t=<target release> install <package-name>~.

See [[https://askubuntu.com/questions/138284/how-to-downgrade-a-package-via-apt-get][this thread for more info]].

As an example consider this real life scenario that happened:

I updated with:

~$ sudo aptitude update && sudo aptitude safe upgrade~:

#+BEGIN_SRC bash
[...]
Fetched 632 kB in 2s (369 kB/s)

Current status: 64 (+6) upgradable, 73192 (-55) new.
Resolving dependencies...
The following NEW packages will be installed:
  libatk-bridge2.0-0{a} libatk1.0-0{a} libatspi2.0-0{a}
The following packages will be REMOVED:
  libatk-bridge2.0-0t64{u} libatk1.0-0t64{u} libatspi2.0-0t64{u}
The following packages will be upgraded:
  apt-transport-https libcairo-gobject2 libcairo2 python-apt-common readline-common
5 packages upgraded, 3 newly installed, 3 to remove and 59 not upgraded.
Need to get 820 kB/1,010 kB of archives. After unpacking 4,096 B will be freed.
Do you want to continue? [Y/n/?]
[...]
#+END_SRC

And everything was seemingly fine.

However, there wasn't yet a bug reported that replacing =libatk-bridge2.0-0t64{u}= would break =emacs= which I found out by a chance during the same evening:

#+BEGIN_SRC bash
pyyhttu@raspberrypi:~$ emacs --debug-init
emacs: error while loading shared libraries: libatk-bridge-2.0.so.0: cannot open shared object file: No such file or directory
pyyhttu@raspberrypi:~$
#+END_SRC

I decided to wait it out. So during the night, the packages were reverted so next morning I updated the repositories again with:

~sudo aptitude update && sudo aptitude safe upgrade~

#+BEGIN_SRC bash
[...]
Fetched 492 kB in 2s (238 kB/s)

Current status: 68 (+9) upgradable, 73195 (+3) new.
Resolving dependencies...
The following NEW packages will be installed:
  libatk-bridge2.0-0t64{a} libatk1.0-0t64{a} libatspi2.0-0t64{a}
The following packages will be REMOVED:
  libatk-bridge2.0-0{u} libatk1.0-0{u} libatspi2.0-0{u}
The following packages will be upgraded:
  cpp-13 cpp-13-x86-64-linux-gnu gcc-13 gcc-13-base gcc-13-x86-64-linux-gnu libdeflate0 libgcc-13-dev libunistring5 xclip
The following packages are RECOMMENDED but will NOT be installed:
  at-spi2-core xauth
9 packages upgraded, 3 newly installed, 3 to remove and 59 not upgraded.
Need to get 33.6 MB/33.8 MB of archives. After unpacking 53.2 kB will be used.
Do you want to continue? [Y/n/?]
[...]
#+END_SRC

After which =emacs= worked again.
#+end_details

Update/upgrade the whole system. Before doing that, check the firmware version:

~$ /opt/vc/bin/vcgencmd version~

#+BEGIN_SRC bash
Aug 15 2019 12:06:42
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start)
#+END_SRC

Then issue:

~$ sudo aptitude full-upgrade~

This will also update the latest *stable* firmware. *Non-stable* beta firmware is installed with ~rpi-update~.

#+begin_note
Starting of Rpi4, device actually has [[https://jamesachambers.com/raspberry-pi-4-bootloader-firmware-updating-recovery-guide/][onboard upgradable firmware]] stored on an EEPROM chip where the firmware upgrade is written, instead of an SD-card.
#+end_note

** Pi-hole installation on Raspberry Pi OS

As per tutorial from [[https://www.smarthomebeginner.com/pi-hole-tutorial-whole-home-ad-blocking/][smarthomebeginner.com]].

To install pi-hole:

~$ wget -O basic-install.sh https://install.pi-hole.net~\\
~$ sudo bash basic-install.sh~

Graphical installer runs:
#+BEGIN_SRC bash
For Interface: eth0
For upstream DNS provider: Cloudflare
For third party block lists: all (by default)
For Protocols: both IPv4 and IPv6
For Static IP Address: This should be the same 10.0.1.245 I setup in the router.
For Web admin interface: Yes
For Web Server: lighttpd
For log queries: Yes (disable after a while once everything runs smoothly to save SD card)
#+END_SRC

Once installer has finished with "Installation Complete!" ensure that
  pi-hole is up and running:

~$ pihole status~
#+BEGIN_SRC bash
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled
#+END_SRC

Ad-blocking can be tested on an ad-infested site, such as
[[https://www.speedtest.net/][speedtest.net]] or at
[[https://canyoublockit.com/][https://canyoublockit.com/]].

#+begin_note
*Note on canyoublockit.com*: Part of the advertisement in there are
[[https://www.reddit.com/r/pihole/comments/f65sge/place_to_test_ad_block/][are
served from the same domain]], which isn't pi-hole's use case. In that case
additional blocker, like
[[https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/][ublock
origin]] is needed.
#+end_note

** Post-installation configuration tasks

*** DONE Configure router to use pi-hole as a DNS sinkhole

In Asus RT-AC86U router go to: +=WAN=, =Internet Connection=: =Connect to DNS Server automatically:= =No=+

*Edit:* Above is not needed, and actually breaks WireGuard installation later on.

Go to: =LAN= - =DHCP Server= - =DNS Server= and set it to your rpi3 IP: =10.0.1.245=

*** DONE Add blocklists

Browse to [[http://10.0.1.245/admin/groups-adlists.php][Group Management -
Adlists]] to add blocklists.

Add well known dbl.oisd.nl blocklist, but consider
  [[https://www.reddit.com/r/pihole/comments/frqisj/httpsdbloisdnl_unreachable_for_a_few_daysweeks_now/][using
  mirrors]], e.g.
  =https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl.txt=

Another alternative is to use
  [[https://www.reddit.com/r/pihole/comments/lh4vk6/best_adlist_to_have/][Wally3k's
  list]]. I only subscribe to "ticked" lists.

I use both dbl.oisd.nl and Wally3k's lists.

The default blocklists Pi-hole comes with are also OK to be used, and [[https://www.reddit.com/r/pihole/comments/jyck17/back_to_stock_blocklists_and_it_feels_so_good/gd2iwj4/?context=3][reverting to them]] is easy if you later decide to do so.

#+begin_details Optional: Analyze blocklists
After having been running the pi-hole with the above blocklists for several weeks, analyze which lists you need based on your browsing behaviour. See
  [[https://github.com/yubiuser/pihole_adlist_tool][Pihole Adlist Tool]].

Get the script:

~$ wget -O pihole_adlist_tool https://raw.githubusercontent.com/yubiuser/pihole_adlist_tool/master/pihole_adlist_tool~

Run the script:

~$ bash pihole_adlist_tool~
#+end_details

** Maintenance

*** DONE Disable automatic daily updates
This is is done in order to be in control what is installed and not get unexpected breakages. Daily updates disabled as per [[https://raspberrypi.stackexchange.com/questions/102377/rp-automatic-updates][this StackExchange post]]:

~$ sudo systemctl mask apt-daily-upgrade~

~$ sudo systemctl mask apt-daily~

~$ sudo systemctl disable apt-daily-upgrade.timer~

~$ sudo systemctl disable apt-daily.timer~

*** DONE Install OS-updates

~$ sudo aptitude update~

~$ sudo aptitude safe-upgrade~

List all installed, and automatically installed aptitude packages:

~$ sudo aptitude search '~i'~

Show removed packages whose config files have not been removed:

~$ dpkg -l | grep '^rc'~

These config files can be removed with:

~$ sudo aptitude purge ~c~

Also, remember to clean the =/var/cache/apt/archives= periodically:

~$ sudo aptitude clean~

*** DONE Upgrade Raspberry Pi OS
New releases of Raspberry Pi OS OS are periodically informed at [[https://www.raspberrypi.org/blog][Raspberry Pi blog]].

~$ sudo apt update~

~$ sudo apt full-upgrade~

*** DONE Upgrade pi-hole

Get updated on new versions of pi-hole when they're available by subscribing to rss feed of their [[https://github.com/pi-hole/AdminLTE/releases][Github release channel]].

#+begin_note
Before upgrading pi-hole, disable monit service if you have it in use:

~$ sudo service monit stop~

This is done because /"daemon restart initiated by monit is not advisable during a Pi-hole update/upgrade"/. See pi-hole's [[https://discourse.pi-hole.net/t/stop-start-monit-during-pi-hole-update/8072][discourse page]] for more info.
#+end_note

Then do:

~$ sudo pihole -up~

#+BEGIN_SRC bash
  [i] Checking for updates...
  [i] Pi-hole Core:     update available
  [i] Web Interface:    up to date
  [i] FTL:              update available

  [i] Pi-hole core files out of date, updating local repo.
  [✓] Check for existing repository in /etc/.pihole
  [✓] Update repo in /etc/.pihole

  [i] If you had made any changes in '/etc/.pihole/', they have been stashed using 'git stash'

  [i] FTL out of date, it will be updated by the installer.

  [✓] Root user check

        .;;,.
        .ccccc:,.
         :cccclll:.      ..,,
          :ccccclll.   ;ooodc
           'ccll:;ll .oooodc
             .;cll.;;looo:.
                 .. ','.
                .',,,,,,'.
              .',,,,,,,,,,.
            .',,,,,,,,,,,,....
          ....''',,,,,,,'.......
        .........  ....  .........
        ..........      ..........
        ..........      ..........
        .........  ....  .........
          ........,,,,,,,'......
            ....',,,,,,,,,,,,.
               .',,,,,,,,,'.
                .',,,,,,'.
                  ..'''.

  [i] Existing PHP installation detected : PHP version 7.3.14-1~deb10u1
  [i] Performing unattended setup, no whiptail dialogs will be displayed
  [✓] Disk space check
  [✓] Update local cache of available packages

  [✓] Checking apt-get for upgraded packages... 6 updates available
  [i] It is recommended to update your OS after installing the Pi-hole!

  [i] Installer Dependency checks...
  [✓] Checking for apt-utils
  [✓] Checking for dialog
  [✓] Checking for debconf
  [✓] Checking for dhcpcd5
  [✓] Checking for git
  [✓] Checking for iproute2
  [✓] Checking for whiptail

  [i] Performing reconfiguration, skipping download of local repos
  [✓] Resetting repository within /etc/.pihole...
  [✓] Resetting repository within /var/www/html/admin...
  [i] Main Dependency checks...
  [✓] Checking for cron
  [✓] Checking for curl
  [✓] Checking for dnsutils
  [✓] Checking for iputils-ping
  [✓] Checking for lsof
  [✓] Checking for netcat
  [✓] Checking for psmisc
  [✓] Checking for sudo
  [✓] Checking for unzip
  [✓] Checking for wget
  [✓] Checking for idn2
  [✓] Checking for sqlite3
  [✓] Checking for libcap2-bin
  [✓] Checking for dns-root-data
  [✓] Checking for resolvconf
  [✓] Checking for libcap2
  [✓] Checking for lighttpd
  [✓] Checking for php7.3-common
  [✓] Checking for php7.3-cgi
  [✓] Checking for php7.3-sqlite3

  [✓] Enabling lighttpd service to start on reboot...
  [✓] Checking for user 'pihole'

  [i] FTL Checks...

  [✓] Detected ARM-hf architecture (armv7+)
  [i] Checking for existing FTL binary...
  [i] Latest FTL Binary already installed (v4.3.1). Confirming Checksum...
  [i] Checksum correct. No need to download!
  [✓] Checking for user 'pihole'
  [✓] Installing scripts from /etc/.pihole

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf

  [i] Installing blocking page...
  [✓] Creating directory for blocking page, and copying files
  [✗] Backing up index.lighttpd.html
      No default index.lighttpd.html file found... not backing up

  [✓] Installing sudoer file

  [✓] Installing latest Cron script

  [✓] Installing latest logrotate script
  [i] Backing up /etc/dnsmasq.conf to /etc/dnsmasq.conf.old
  [✓] man pages installed and database updated
  [i] Testing if systemd-resolved is enabled
  [i] Systemd-resolved is not enabled
  [✓] Restarting lighttpd service...
  [✓] Enabling lighttpd service to start on reboot...
  [i] Restarting services...
  [✓] Enabling pihole-FTL service to start on reboot...
  [✓] Restarting pihole-FTL service...
  [✓] Deleting existing list cache
  [i] Pi-hole blocking is enabled
  [✗] DNS resolution is currently unavailable
  [✓] DNS resolution is now available

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: Retrieval successful

  [i] Target: sysctl.org (hosts)
  [✓] Status: Retrieval successful

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: Retrieval successful

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: Retrieval successful

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: Retrieval successful

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: Retrieval successful

  [i] Target: smokingwheels.github.io (allhosts)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: gist.githubusercontent.com (Test.txt)
  [✓] Status: Retrieval successful

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] Number of domains being pulled in by gravity: 149202
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 126670
  [i] Number of whitelisted domains: 199
  [i] Number of blacklisted domains: 4
  [i] Number of regex filters: 16
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

  [✓] Force-reloading DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

  [i] The install log is located at: /etc/pihole/install.log
Update Complete!

  Current Pi-hole version is v4.4
  Current AdminLTE version is v4.3.3
  Current FTL version is v4.3.1
#+END_SRC

After update completed, start monit (if you've installed it):

~$ sudo service monit start~

*** DONE Inspect pi-hole messages

Messages can be found from: http://10.0.1.245/admin/messages.php (or http://pi.hole/admin/messages.php) and reveal ee possible issues.

*** DONE Check periodically memory card wear

To manually check memory card wear, do:

~$ sudo badblocks -v /dev/mmcblk0p1~

#+BEGIN_SRC bash
Checking blocks 0 to 43924

Checking for bad blocks (read-only test): done

Pass completed, 0 bad blocks found. (0/0/0 errors)
#+END_SRC

#+begin_details Optional: Automate checks with Monit
If you have installed =monit=, you can automatize above check, do:

~$ sudo nano /etc/monit/monitrc~

And append there:

#+BEGIN_SRC
check program badblocks with path "/sbin/badblocks /dev/mmcblk0p1"
  every "30 19 * * 6"
  if status != 0 then alert
#+END_SRC
#+end_details

*** DONE Install Log2ram

This is optional but heavily recommended.

#+begin_details Optional: Install Log2ram

As query logging is enabled and those logs are saved to SD-card, the card can wear out in few months. To prolong card's life span, save query logs in memory with log2ram, and write them only once per every hour [fn:4] to SD card. With log2ram, since logs will then reside much of their time in volatile RAM, the downside is that in case of system/device crash, I lose those logs, which complicates troubleshooting.

However, pi-hole with its accompanied services has been rock-solid, so prolonging the lifetime of a SD-card brings more value.

~$ echo "deb http://packages.azlux.fr/debian/ buster main" | sudo tee /etc/apt/sources.list.d/azlux.list~

~$ wget -qO - https://azlux.fr/repo.gpg.key | sudo apt-key add -~

~$ sudo aptitude update~

~$ sudo aptitude install log2ram~

Increase the =SIZE= parameter of the RAM directory from default =40M= to =320M= in order to avoid in the future the [[https://github.com/azlux/log2ram/issues/90][error message]]: =ERROR: RAM disk too small. Can't sync.= when stopping & starting services with raspiBackup. Do:

~$ sudo nano /etc/log2ram.conf~

#+BEGIN_SRC bash
# Size for the ram folder, it defines the size the log folder will reserve into the RAM.
# If it's not enough, log2ram will not be able to use ram. Check you /var/log size folder.
# The default is 40M and is basically enough for a lot of applications.
# You will need to increase it if you have a server and a lot of log for example.
SIZE=320M
#+END_SRC

For changes to take effect, restart the service:

~$ sudo service log2ram restart~

Or alternatively reboot:

~$ sudo reboot~

- After restart/reboot, test that log2ram is invoked:

#+BEGIN_SRC bash
pyyhttu@raspberrypi:~ $ service log2ram status
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2020-01-18 09:33:45 EET; 3h 3min ago
  Process: 261 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
  Process: 3583 ExecReload=/usr/local/bin/log2ram write (code=exited, status=0/SUCCESS)
 Main PID: 261 (code=exited, status=0/SUCCESS)
#+END_SRC

Log2ram will also be mounted at =/var/log=:

#+BEGIN_SRC bash
pyyhttu@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  2.1G   12G  15% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M  1.7M  462M   1% /dev/shm
tmpfs           464M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/sda1       7.1G  2.5G  4.3G  37% /backup
/dev/mmcblk0p1   43M   23M   20M  53% /boot
log2ram         320M   41M   60M  41% /var/log
tmpfs            93M     0   93M   0% /run/user/999
tmpfs            93M     0   93M   0% /run/user/1000
#+END_SRC

#+end_details
*** DONE Further manage storage and logs
- Restore query logging and implement [[https://www.reddit.com/r/pihole/comments/k5hza1/raspberry_pi_storage_is_full_of_logs/][actions as here]].

*** DONE Experiment with Privacy settings vs. data collected
Privacy settings are in Pi-hole at =Settings - Privacy=. Set privacy level to =3 - Anonymous mode= and consider disabling the query logging, as documented [[https://www.reddit.com/r/pihole/comments/qux2rw/is_there_a_way_to_prevent_spying/hksxulp/?context=3][here]]. However, without query logging I wouldn't be a chance to troubleshoot then. But as with every self-hosted solution that is shared with family members:
  1) Respect the privacy of others.
  2) With great power comes great responsibility.

More about on logging and collecting data [[https://www.reddit.com/r/pihole/comments/a161ce/make_pihole_logs_less_verbose/ean2f7x/?context=3][here]].

*** REJECTED Upgrade Raspberry Pi OS version from Debian Buster to Debian Bullseye
Use [[https://www.linuxquestions.org/questions/blog/craigevil-176422/raspberry-pi-os-debian-11-bullseye-apt-repos-38636/][these
  repositories]], but, it seems that it would resulting in
  [[https://www.reddit.com/r/pihole/comments/p7t1c4/pihole_on_raspbian_bullseye/][disabling
  OS version check]]. Better wait for official pi-hole release.

*Update [2021-09-12 Sun]:* Version FTL v5.9
[[https://www.reddit.com/r/pihole/comments/pmfxfo/pihole_ftl_v59_web_v56_and_core_v54_released/hchmt7z/?context=3][supports
Debian 11]]. Update repositories as per [[https://www.reddit.com/r/pihole/comments/pmfxfo/pihole_ftl_v59_web_v56_and_core_v54_released/hcifxs1/?context=3][this post]], or [[https://www.reddit.com/r/RASPBERRY_PI_PROJECTS/comments/s8nkm0/how_to_update_your_debian_10_buster_system_to/][this post]], and followup threads on success rate, such as [[https://www.reddit.com/r/pihole/comments/pr64gg/updated_to_bullseye_needed_to_change_my_dhcpcdconf/][this]]. Also weigh in if you should try to first setup second pihole, and for update, try the [[https://forums.raspberrypi.com/viewtopic.php?t=323279]["semi-official upgrade instructions]]. Before upgrading to bullseye, test the [[https://www.linux-tips-and-tricks.de/en/restore/][restore operation]] of latest raspiBackup. After upgrading to bullseye, check if [[https://www.reddit.com/r/pihole/comments/pn93d3/bullseye_and_predictable_network_interface_names/][Predictable Network Interface Names]] are edited.

*Update [2021-09-26 Sun]:* This was a hassle. The update did not fail but after a reboot, device wasn't anymore discoverable in network. Reverted with backups.

Rasbian is *not* made for rolling release, and thus every major OS version upgrade, which are inevitable with point release operating systems, culminate to a risk scenario where I need to reinstall everything again. Which means, that I either,

1) establish infrastructure as a code and learn to configure everything with something like Ansible
2) start running pi-hole dockerized
3) investigate on rolling release distribution

At this point, I'll go with option 3).

** Troubleshooting

Pi-hole error messages are normally visible from
=/var/log/pihole.log=, =/var/log/pihole-FTL.log= or
=/var/log/syslog=. If the need is to inspect boot time logs, then do ~$ dmesg | less~ or ~$ journalctl~.

*** DONE ~$ pihole status~ gives "DNS Service is NOT running"

So far this error has occurred twice, and there can be multiple root causes
for the error. Start by checking errors with:

~$ journalctl -u pihole-FTL --full --no-pager~

Normally this should just return "-- No entries --".

In the first error case there was an error entry:

#+BEGIN_SRC bash
dnsmasq: illegal repeated keyword at line 33
of /etc/dnsmasq.d/01-pihole.conf_bak
#+END_SRC

Error was resolved after deleting the above =01-pihole-conf_bak= backup file I had created and
then doing:

~$ sudo service pihole-FTL restart~

In the second case:

~$ journalctl -u pihole-FTL --full --no-pager~ the error entry was:

#+BEGIN_SRC bash
dnsmasq: failed to create
listening socket for port 53: Address already in use
#+END_SRC

I also checked my cloudflared setup for encrypted DNS with:

~$ sudo systemctl status cloudflared.service~

There was an error msg:

#+BEGIN_SRC bash
failed to connect to an HTTPS backend
\"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request:
Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout
exceeded while awaiting headers)
#+END_SRC

So by doing:

~$ sudo systemctl restart cloudflared.service~ and then again:

~$ sudo systemctl status cloudflared.service~

The error was no longer visible. Also:

~$ pihole status~ should now give:

#+BEGIN_SRC bash
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
#+END_SRC

*** DONE After rebooting pi, ~$ ssh pyyhttu@10.0.1.245~ errors out with: Resource temporarily unavailable

- Also ping to =10.0.1.245= does not respond

- In my case I needed to unplug the power cord, and reattach it to reboot.
For some reason, boot sequence did not complete on the first try.

*** DONE After Gravity updating the blocklists, the dashboard shows "Domains on Blocklist 0"

This was the first time blocklist update caused 0 Domains on blocklist. It may be linked to the fact that either monit or healthcheck.io caused the update to fail.
[[https://discourse.pi-hole.net/t/gravity-update-when-does-it-run-exactly/20098/2][Gravity updates the blocklist on randomly selected ]] time between 3 am and 5 am my local time (to equalize server loads):
=/var/log/syslog= shows my update started 03:49 and logs shows no errors. Updating gravity manually ~$ pihole -g~ did not help.
Finally, by disabling monit ~$ sudo service monit stop~ and commenting out healthcheck.io cron with ~$ crontab -e~, and then doing repair with ~$ sudo pihole -r~, the blocklist on dashboard showed the correct blocklist.

*** DONE Disable in iOS 14 "Private Address" for home wifi connection
Starting iOS 14, Apple
[[https://www.reddit.com/r/pihole/comments/iuolqh/psa_ios_14_mac_address_randomization/][started
to randomize]] its devices mac addresses. This randomizing is done by Apple to reduce tracking across different Wi-Fi networks. The feature is redundant in home wi-fi, and actually causes DHCP servers to hand out IP addresses different than those statically configured which can lead to clients not being assigned to the proper group.

*** FOLLOWUP Investigate why certain clients gradually obtain multiple IPv4-addresses
- This is evident by going in pi.hole to [[http://10.0.1.245/admin/network.php][network overview]].
- Trying first to [[https://discourse.pi-hole.net/t/multiple-ip-addresses-in-network-overview/33562/13][flush the network]] clears some multiple ip-addresses assigned, but not all.
- [[https://www.reddit.com/r/pihole/comments/d534by/how_do_i_clear_the_network_overview_entries/][Deleting every entry]] with =sqlite3= does clear everything, along the clients, but after a while client entries are repopulating back, some with multiple IP addresses assigned to them again.
- Doing now =Settings=, =Flush network table= does clear the =Network overview=
  table so that clients in the table are left with single IPV4-address, but
  again, temporarily. After a while, Samsung television e.g. gets multiple IPs,
  like =10.0.1.251=, =10.0.1.252= and =10.0.1.253=.
- Trying next that I assign these devices an IP manually.
- Assigning an IP manually, then adding the IP and a custome DNS entry at
  http://pi.hole/admin/dns_records.php helped somewhat: now network overview at least
  has the hostname visible. Let's see if they only will get a single IPv4. 

*** DONE Log2ram failed and exited with an error code

#+BEGIN_SRC bash
$ service log2ram status
● log2ram.service - Log2Ram
Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-12-12 03:21:39 EET; 5 days ago
Main PID: 17699 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
#+END_SRC

Also, the service won't restart:

#+BEGIN_SRC bash
$ sudo service log2ram restart
Job for log2ram.service failed because the control process exited with error code.
See "systemctl status log2ram.service" and "journalctl -xe" for details.
pi@raspberrypi:/var/log $ systemctl status log2ram.service
● log2ram.service - Log2Ram
Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-12-17 16:51:26 EET; 11s ago
Process: 7311 ExecStart=/usr/local/bin/log2ram start (code=exited, status=1/FAILURE)
Main PID: 7311 (code=exited, status=1/FAILURE)

Dec 17 16:51:26 raspberrypi systemd[1]: Starting Log2Ram...
Dec 17 16:51:26 raspberrypi log2ram[7311]: ERROR: RAM disk for "/var/hdd.log/" too small. Can't sync.
Dec 17 16:51:26 raspberrypi log2ram[7311]: /usr/local/bin/log2ram: 45: /usr/local/bin/log2ram: mail: not found
Dec 17 16:51:26 raspberrypi systemd[1]: log2ram.service: Main process exited, code=exited, status=1/FAILURE
Dec 17 16:51:26 raspberrypi systemd[1]: log2ram.service: Failed with result 'exit-code'.
Dec 17 16:51:26 raspberrypi systemd[1]: Failed to start Log2Ram.
#+END_SRC

Root cause seemed to be that [[https://github.com/azlux/log2ram/issues/90][RAM disk is too small]] so
increasing that in =/etc/log2ram.conf= to =320M= and doing ~$ sudo service log2ram restart~ fixed the issue.

*** DONE Ensure that logrotate is working, or disable pihole query logging

On [2020-01-18 Sat] morning raspiBackup had failed. =/var/log= size had grown past 40M,
and due to that, log2ram service didn't start, failing raspiBackup. It seems that pihole logging
takes most of the space:

#+BEGIN_SRC bash
pyyhttu@raspberrypi:/var/log $ du -sh -- * | sort -h
[...]
1.2M    auth.log
1.2M    daemon.log
1.3M    auth.log.1
1.7M    pihole.log
33M     pihole.log.1
#+END_SRC

Monitor how big =/var/log= will grow and whether =160M= is enough (it is not). Consider
[[https://www.reddit.com/r/pihole/comments/a161ce/make_pihole_logs_less_verbose/ean2f7x/][disabling
query logging]] (=Settings > System > Disable query logging=). This means
that can't tail the logs (~$ pihole tail~), but will still see the
queries in the query log and dashboard.

For now, I've increased in =/etc/log2ram.conf= from the default =40M= to the
=SIZE=320M= which seems to be enough.

*** DONE =Warning in dnsmasq core:=
    CLOSED: [2024-04-24 Wed 21:36]
Error was: =ignoring query from non-local network 100.115.92.133 (logged only once)=.
Noticed this error on [2023-04-29 10:50:41]	as DNSMASQ_WARN in [[http://10.0.1.245/admin/messages.php][pi-hole
diagnosis]]. This may be due to fact I have WireGuard VPN setup on the same
device. As per analysis 
[[https://www.reddit.com/r/pihole/comments/rvpl2g/comment/hy86vmi/?utm_source=share&utm_medium=web3x][here]],
I have now ="Allow only local requests"= changed to ="Bind only to inferface
eth0"= at http://10.0.1.245/admin/settings.php?tab=dns. Observing if error returns.

*Update:* [2024-04-24 Wed]: The Warning has not returned, but I've kept setting
="Allow only local requests"= on.

** Observations, questions and future direction

*** DONE Observe with iphone, when at home and attached to both cellular and wifi
Does cellular takes precedence over wifi, rendering the ad-blocking
useless?

A: Yes, seems so. Either
[[https://www.reddit.com/r/apple/comments/8unliv/does_wifi_assist_ever_work_for_you/][disable
in iOS "Wifi Assist"]] under =Settings= - =Cellular=.

Or then, be attached to VPN all the time and observe battery consumption.

*Update*: Seems to be that having the WireGuard tunnel always on does not result
in noticeable battery consumption. On Androids, where having WireGuard baked
in as a module in custom kernels, the performance and battery consumption
should be even more better.

*** DONE Observe DNS lease expirations

Validate when leases expire, as the IP changes which complicates the setup of
WireGuard.

*Update:* it seems that the lease has been gradually getting more time
(interval unknown), observable by doing

~$ ipconfig /all~

*** NEXT Harden security

See
[[https://www.raspberrypi.com/documentation/computers/configuration.html#secure-your-raspberry-pi][Raspberry
Pi documentation]] and [[https://wiki.debian.org/SSH#Securing][Debian
documentation]].

Also implement points from this [[https://lemmy.world/post/12727493][post]] as actionable list.

Enable automatic, unattended security updates.
Investigate if this is possible with Debian Sid.

*** DONE if someone wants out...

I may want opt out certain production/work devices by [[https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245][manually
configuring each device]]. For above to take effect, [[https://www.reddit.com/r/pihole/comments/e2qm3r/iphone_not_respecting_routers_dns_settings_for/][purging iOS DNS cache]] may require toggling the wifi on/off.

*Update:* Starting with pi-hole v5.0 onward, above per-client manual DNS settings are no longer needed. Pi-hole supports
[[https://pi-hole.net/2020/05/10/pi-hole-v5-0-is-here/#page-content][per-client
blocking]]. Effectively we can then group everyone's devices and define
whether or not they're part of pi-hole. To set this up, see
[[https://docs.pi-hole.net/database/gravity/example/][Per-client blocking
examples]]. For this to work reliably, remember to assign in router a manual IP
for every such device.

*tl;dr*: If you want to exclude a device from pi-hole, assign in pi-hole a client to a group, then remove the client from the =default= group.

*** DONE Harden pihole's blocking with regexp
See [[https://www.reddit.com/r/pihole/comments/b3fj60/regex_megathread/][relevant reddit regexp thread]] for more info. I added [[https://github.com/mmotti/pihole-regex/blob/master/regex.list][mmotti/regexp list]] from github.

*** REJECTED Investigate on setting SSL (https) for the admin panel at http://10.0.1.245/admin
Not done, as this would be futile, for many [[https://www.reddit.com/r/pihole/comments/83wvmn/any_plans_to_add_ssl_to_the_web_interface/][reasons]].

*** DONE flash router with Merlin

[[https://github.com/RMerl/asuswrt-merlin.ng/wiki/About-Asuswrt/#about-asuswrt-merlin][Merlin]]
is an alternative firmware to [[https://www.asus.com/Content/ASUSWRT/][stock
Asus firmware]]. However, Merlin is not without its [[https://www.reddit.com/r/pihole/comments/diy86t/my_asus_rtac66u_with_merlin_was_hacked_and_i_only/][problems]].
Merlin allows to me [[https://www.reddit.com/r/pihole/comments/dfm5j4/guide_for_asuswrtmerlin_users_with_screenshots/][force
  DNS traffic]] to router, as referenced also in
  [[https://www.reddit.com/r/pihole/comments/nihk35/pihole_asus_merlin_with_dnsfilter_question/][here]]. [[https://x3mtek.com/asuswrt-merlin-firmware-upgrade/][Remember these things]] when upgrading the Merlin firmware.

*** NEXT Set Asus router with Merlin firmware to force all port 53 traffic (DNS) to pi-hole
Do this after upgrade to Merlin. Some devices have
[[https://www.reddit.com/r/pihole/comments/d0jzuu/i_was_reading_that_not_all_queries_go_through_the/][hard
coded DNS servers]] that can’t be changed via DHCP or manually. This is disturbingly an increasing trend among device
manufacturers, so I must consider
[[https://www.reddit.com/r/pihole/comments/8jm50o/can_i_force_all_traffic_to_use_the_pihole_dns/][forcing
all local network route traffic]] to pihole. Here's the
[[https://support.opendns.com/hc/en-us/community/posts/220011927-Asus-RT-AC88U-and-port-53][instructions
for Asus router]] and the [[https://www.reddit.com/r/pihole/comments/n67e7k/comment/gx61045][comment link here]] has screenshotted instructions among [[https://www.reddit.com/r/pihole/comments/dheym1/for_those_with_an_edgerouter_this_is_my_dnat_rule/][routing rules]].

Having two Pi-holes, or alternatively having one Pi-hole setup and adding a virtual IP to it so the badly behaving devices see 2 DNS servers,
[[https://www.reddit.com/r/pihole/comments/kahc0m/netflix_is_now_trying_to_bypass_pihole/gfajk9b/?context=3][seem to mitigate]] these devices from calling their own DNS server, and bypassing Pi-hole.

Hard coded DNS servers may also become an issue when [[https://www.reddit.com/r/pihole/comments/hpaea7/anyone_using_ios14_looks_like_some_apps_are/][using iOS 14]].

This is also evident if messages like [[https://www.reddit.com/r/pihole/comments/hfpraa/ios_14_privacy_features_your_network_may_be/]["This network is blocking DNS traffic"]] starts to pop up in iOS 14.

When forcing traffic through port 53, pay attention if tutorial guides disabling dhcp on router, and enabling it on Pi-hole. Don't do that, but
investigate after installing Merlin ("John's fork") for N66U, if rerouting can be done with
[[https://www.reddit.com/r/pihole/comments/fgtb6v/comment/fk7gldo][this tutorial]] instead.

If after Merlin upgrade I experience high amount of traffic from router visible in Pi-hole statistics, apply these
[[https://www.reddit.com/r/pihole/comments/gmwkeo/problem_with_asus_rtn66u_and_pihole/][router configs]].

See also [[https://www.snbforums.com/threads/troubleshooting-101-asus-rmerlin-routers-2020.65438/][General Merlin troubleshooting]].

[[https://www.reddit.com/r/pihole/comments/mawsjz/asus_router_keeps_pushing_itself_as_dns_server/][On defining DNS servers]] in Merlin ([[https://www.reddit.com/r/pihole/comments/mawsjz/asus_router_keeps_pushing_itself_as_dns_server/grusnbn/?context=3][check
the screenshot]] on that thread).

When testing the force routing of traffic through port 53, use your Phillips
Hue lamps as guinea pigs as they
[[https://www.reddit.com/r/pihole/comments/qrgel5/hue_bridges_have_hardcoded_dns_servers/][seem
to have hardcoded dns]].

*** NEXT Setup DNSFilter on Merlin
Also check out what is DNSFilter with Merlin mentioned in [[https://www.reddit.com/r/pihole/comments/ikw739/kind_of_related_to_pihole_but_more_along_the/][here]]. DNSFilter with Merlin mentioned also 
  [[https://www.reddit.com/r/pihole/comments/omlh01/force_clients_trying_to_reach_outsidehardcoded/][here]] and [[https://www.reddit.com/r/pihole/comments/nihk35/pihole_asus_merlin_with_dnsfilter_question/][here]].

*** DONE Observe with new router in case of loop back
Acquired a pair of new Asus RT-AC86U routers to create an AiMesh network. Pi-holes operating in backhauled mesh should
[[https://www.reddit.com/r/pihole/comments/ko4l1q/anyone_using_asus_aimesh_and_a_pihole/ghow310/?context=3][be
fine]].
Then redid the configurations for routers and observed if needed to apply [[https://www.reddit.com/r/pihole/comments/iggm88/pihole_and_asus_rtax56u_over_3_mio_requests/g2vdhkl/?context=3][this
fix]]. Never had to resolve the fix.

*** REJECTED Consider logging querying and visualisation options, like [[https://www.reddit.com/r/pihole/comments/e7t0qe/integrating_pihole_logs_in_elk_with_logstash/][ELK]]

Before installing Elk, run diagnostic on what is system load before and after installing the Elk stack. Memory usage, processor usage etc.

*Edit*: Explored ELK and considered that it is not needed, as it complicates the install, eat resources of Rpi3 and the fancy charts offer me no real
value over the regular charts Pi-hole comes bundled with.

*** NEXT Investigate whether Skynet + Yazfi script is beneficial for IP banning and extra security in Asus router.
For Skynet [[https://www.reddit.com/r/pihole/comments/diy86t/my_asus_rtac66u_with_merlin_was_hacked_and_i_only/f41jxv4/][investigate]] on this reddit thread. For Yazfi scripts, [[https://www.reddit.com/r/pihole/comments/lc75ms/guest_networks_pihole_asus_merlin/][look here]].

USB drive is required for Swap: https://www.snbforums.com/threads/skynet-installation-help.88080/

Skynet forum: https://www.snbforums.com/search/1105236/?q=skynet+usb&t=post&c[child_nodes]=1&c[nodes][0]=60&o=relevance

https://www.snbforums.com/threads/skynet-installation-help.88080/

Disable password ssh-authentication to router. enable only key based authentication. Save SSH key to KeepassXC & Strongbox.

*** DONE [[https://raspberrytips.com/disable-wifi-raspberry-pi/][Disable wifi and bluetooth radios]]

In order to save power and resources, disable unnecessary services by doing:

~$ sudo systemctl disable wpa_supplicant~

~$ sudo systemctl disable bluetooth~

~$ sudo systemctl disable hciuart~

~$ echo "dtoverlay=pi3-disable-bt" | sudo tee -a /boot/config.txt~

~$ echo "dtoverlay=pi3-disable-wifi" | sudo tee -a /boot/config.txt~

~$ sudo reboot~

After a reboot, wlan and bluetooth radios should not be on anymore. You can
check this by doing:

~$ sudo systemctl status wpa_supplicant.service~

~$ sudo systemctl status bluetooth.service~

~$ sudo systemctl status hciuart.service~

#+begin_note
Update [2022-06-22 Wed]: Noticed from kernel boot up messages at
=/var/log/boot.msg= that
wifi may not be down properly, as there was an error:
: Failed to start Disable WiFi if country not set

Also running ~$ sudo systemctl status wifi-country.service~ gave:

#+BEGIN_SRC
● wifi-country.service - Disable WiFi if country not set
   Loaded: loaded (/lib/systemd/system/wifi-country.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2022-06-23 18:10:20 EEST; 2 weeks 6 days ago
 Main PID: 458 (code=exited, status=1/FAILURE)
#+END_SRC

Defined as per
[[https://rigacci.org/wiki/doku.php/doc/appunti/hardware/raspberrypi_wifi_country_not_set][this
article]] in =/etc/wpa_supplicant/wpa_supplicant.conf= a line:

=country=FI= needed to be added. After this the error was gone.
#+end_note

*** REJECTED Investigate DNSCrypt for anonymized DNS
As described in [[https://www.reddit.com/r/pihole/comments/dhqz7s/dnscrypt_users_anonymized_dns_is_in_beta/][r/pihole]]. Installing this is expected to increase my DNS resolving time, but by how
much, needs to be measured. Currently running:

~$ dig @127.0.0.1 -p 5053 google.co.uk~

gives =19 msec=, same as [[https://www.reddit.com/r/pihole/comments/dhqz7s/dnscrypt_users_anonymized_dns_is_in_beta/f3qkl1m/][here]] for a guy with cloudflare DNS. If after implementing and measurement it's the resolution is of same ballpark, leave the setting in.

*Update:* It [[https://www.reddit.com/r/pihole/comments/dz8pxs/dnscrypt/][seems]] to be that DNSCrypt does not bring me any benefits.

*** NEXT New router with Raspberry Pi Zero W as a backup DNS for high availability
- If ever buying a new router, I could equip it with a Raspberry Pi Zero W on
  router's powered usb, as a [[https://www.reddit.com/r/pihole/comments/e22lcd/how_can_i_ensure_my_network_wont_go_down_if_my_pi/][backup DNS-server]]. This would also result in even [[https://www.reddit.com/r/pihole/comments/dov73j/comment/f5qotzc][smaller power footprint]] with no cords.
- However, that would be a concious risk as RasberryPi does not [[https://www.raspberrypi.org/forums/viewtopic.php?t=11092#p122551][negotiate the power with host]] and I end up with a broken system.
- However, people have been running Rpi Zero W powered directly from routers usb [[https://www.reddit.com/r/pihole/comments/h8443w/pi_zero_w_or_something_better/][without issues]], so this is worth the risk.
- Also, it seems that Asus routers
  [[http://www.snbforums.com/threads/preview-early-asuswrt-merlin-384-6-test-builds-are-available.47385/post-413834][implement
  RNDIS]] on their USB lane which Pi Zero would use, I could then potentially
  [[https://www.reddit.com/r/pihole/comments/nhp3do/how_beefy_of_a_pi_would_i_need_for_pihole_server/gz0w3om/?context=3][use the same lane for its Ethernet
  and power]], disabling wireless from Pi Zero completely.
- And if the raspiBackup system is implemented on Pi Zero with the same
  monitoring and backup restore settings I now have on Rpi4, the risk is minimized and worth a try.
- Since I then will have two Pi-Holes operating, I'd like to keep their [[https://www.reddit.com/r/pihole/comments/e8wi0y/synchronize_multiple_piholes/][blocklists synchronized]].
- Tutorial for running two pi-holes in high availability mode could be achieved manually with [[https://www.reddit.com/r/pihole/comments/jtqotd/best_practices_for_2_piholes_in_same_network/][this tutorial]] and in automated fashion with [[https://github.com/vmstan/gravity-sync][gravity-sync]] (preferred).
- New router could be Asus RT-AC86U, since it can support also [[https://www.snbforums.com/threads/experimental-WireGuard-for-rt-ac86u-ax88u.46164/][WireGuard loaded in]].
- However, people have been getting on 1 Gbit connection only 450 Mbit/s on it, while WireGuard on [[https://www.reddit.com/r/WireGuard/comments/d8k605/gigabit_WireGuard_performance_on_sbcs/][rpi4 can push ~600 Mbit/s]].
- It needs to be tested/googled whether the router could handle faster traffic than 450 Mbit/s (although that is plenty for my connection now).
- There are also [[http://esr.ibiblio.org/?p=8330][other open firmwares]] out there, e.g. OpenWRT, [[https://advancedtomato.com/][AdvancedTomato]], etc.

*** NEXT Install Debian on Pi Zero W
Idea, and also experience so far is, that with Debian Sid and =apt-listchanges= and =apt-listbugs= would
enable a rolling release for me with which I would not need to go through OS
reinstall through whenever support ends. Instead, I would do every weekend manual ~sudo aptitude update & sudo aptitude safe-upgrade~, inspect the changelogs, pin packages with bugs, and despite this if something goes wrong, revert to backups taken previous night.

Start by checking [[https://wiki.debian.org/RaspberryPi][Debian RaspberryPi]]
and [[https://raspi.debian.net/faq/][Debian raspi faq.]]

Then download [[https://raspi.debian.net/tested-images/][tested image]] (Family 1, 0W).

Continue with
[[https://mattray.github.io/2021/08/30/installing-debian-11-0-on-a-raspberry-pi.html][these
instructions]]. Do:

Enable remote SSH root login (disable this later once non-root user is setup)
~# echo "PermitRootLogin yes" >> /etc/ssh/sshd_config~

Images come with empty root password, so change that to something else:
~# passwd root~

Set the hostname:
~# hostnamectl set-hostname~

The images come with wireless support, so add to =/etc/network/interfaces.d/wlan0=:

BEGIN_SRC bash
allow-hotplug wlan0
iface wlan0 inet dhcp
    wpa-ssid MYNETWORK
    wpa-psk MYPASSWORD
END_SRC

Reboot, give enough time for DHCP to handout an IP, then ssh to it and do the rest remotely:

~$ ssh root@10.0.1.246~

- Set up user pyyhttu
- Set up groups
- apt update && upgrade
- set sources to sid (deb https://ftp.debian.org/debian/ sid contrib main non-free non-free-firmware)

- disable remote root login

*** NEXT Configure Debian unattended upgrades
#+begin_details
link-here:solution
For security updates only, if possible. See: https://wiki.debian.org/UnattendedUpgrades
#+end_details


For security updates only, if possible. See: https://wiki.debian.org/UnattendedUpgrades

*** NEXT Install 64-bit Raspberry Pi OS Lite on Rpi3B+.

Then
[[https://forums.raspberrypi.com/viewtopic.php?p=2079524&hilit=unstable+sid#p2079391][turn
its repositories]] to Debian Unstable (Sid). Remember to add
[[https://forums.raspberrypi.com/viewtopic.php?p=2079524&hilit=unstable+sid#p2079524][non-free
firmware]]. Remember to
[[https://www.jeffgeerling.com/blog/2023/nmcli-wifi-on-raspberry-pi-os-12-bookworm][setup
wifi]] with =nmcli= instead of =wpa_supplicant=.

*** NEXT Enable in Pi-hole admin panel Conditional forwarding
- Conditional forwarding is enabled to display
[[https://www.reddit.com/r/pihole/comments/6z7cr8/how_can_i_see_client_name_instead_of_my_client_ip/][client
names]] instead of their IPs in Pi-hole admin panel. This would also save
me the trouble of having to edit the reservations and hosts file manually
when new clients are added to the network.
- In Asus RT-AC86U go to =LAN - DHCP Server= and define for field "=RT-AC86U's Domain Name=" a Domain Name (e.g. =house=).
- Log in to Pi-hole and go to: =Settings - DNS= and tick "=Use Conditional
Forwarding=". Define for "=Local network in CIDR notation=": "=10.0.0.0/24=",
"=IP address of your DHCP server (router)=": =10.0.1.1= and for "=Local domain
name=": =house=.
- Make sure that "=Never forward non-FQDNs=" and "=Never forward reverse lookups
for private IP ranges=" are also ticked; otherwise the clients' names in
Pi-hole dashboard are still displayed with their IPs only [fn:8].
- Update [2020-12-06 Sun]: After setting up new router (Asus AC86U) hostnames are now again not displayed in pi-hole
under =Tools=, =Network=, but are displayed as IPs. It seems this behavior
is
[[https://old.reddit.com/r/pihole/comments/kt6csy/unable_to_see_clients/gik7070/][router
specific]].
- So far, my only resolution would be to enable DHCP in pi-hole, instead of the
router, OR, flash both routers (Ai Mesh) with Merlin, as I may be able to
[[https://www.reddit.com/r/pihole/comments/ko4l1q/anyone_using_asus_aimesh_and_a_pihole/ghr84c2/?context=3][disable
Asus router's DNS]] that way.

*** NEXT Troubleshooting conditional forwarding on Asus routers
- As per this [[https://www.reddit.com/r/pihole/comments/ouiyp1/conditional_forwarding_not_working/][reddit post]].

*** FOLLOWUP ISPs IPv6 enablement

Once my ISP (Elisa) takes IPv6
[[https://elisa.fi/asiakaspalvelu/aihe/mobiililaajakaista/ohje/ipv6-osoitteet/][into
use]], then there's is a potential
[[https://pi-hole.net/2018/02/02/why-some-pages-load-slow-when-using-pi-hole-and-how-to-fix-it/#page-content][slow
down scenario]] with pi-hole:

#+BEGIN_QUOTE
Some ISPs do not hand out static IPv6 addresses. So if you configured Pi-hole with an IPv6 address during installation and that address is later changed by your ISP, you now run into the problem the wrong (i.e. an invalid) IPv6 address in =gravity.list=. Because Pi-hole looks to this file to find out where to go, your computer cannot find that IP address and the requests time out, causing the long loading times.
#+END_QUOTE

Eventually I need to take [[https://www.asus.com/support/FAQ/113990/][IPv6
into use]] and set it up in my router's settings.

Once done, I need to retest my IPv6 readiness at http://test-ipv6.com/.

*** DONE Add hard coded IP addresses to the list of NTP servers

None of the rpi3's have a built-in hardware clock [fn:9], so they rely on a network time server
to get the correct time.

Unbound requires the system clock to be pretty close to real time or it
cannot resolve DNS queries.

All of the default NTP servers are
setup as fully qualified hostnames, so in [[https://www.reddit.com/r/pihole/comments/fklqv1/forcing_pihole_to_use_router_as_ntp_server/][in an event of downtime]], the clock
will be off. Thus, after restart unbound can not resolve the hostnames of the ntp
servers.

To prevent this in the future, add several hard coded IP addresses to
=/etc/systemd/timesyncd.conf=. But first, check that your time is correct:

~$ timedatectl~

#+BEGIN_SRC bash
         Local time: Sun 2020-03-08 16:54:30 EET
     Universal time: Sun 2020-03-08 14:54:30 UTC
           RTC time: n/a
          Time zone: Europe/Helsinki (EET, +0200)
System clock synchronized: yes
        NTP service: active
    RTC in local TZ: no
#+END_SRC

~$ systemctl status systemd-timesyncd.service~

#+BEGIN_SRC bash
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
     └disable-with-time-daemon.conf
Active: active (running) since Sat 2020-02-29 13:27:19 EET; 1 weeks 1 days ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 335 (systemd-timesyn)
Status: "Synchronized to time server for the first time 95.216.24.230:123 (2.debian.pool.ntp.org)."
Tasks: 2 (limit: 2200)
Memory: 1.4M
CGroup: /system.slice/systemd-timesyncd.service
     └335 /lib/systemd/systemd-timesyncd

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
#+END_SRC

Add the NTP server IPs:

~$ sudo nano /etc/systemd/timesyncd.conf~

#+BEGIN_SRC bash
[Time]
NTP=162.159.200.123
FallbackNTP=185.112.82.30 95.216.160.182 95.217.10.168
#RootDistanceMaxSec=5
#+END_SRC

Above IP-addresses are of =fi.pool.ntp.org= specific pool zone: https://www.ntppool.org/zone/fi

Ensure time synchronize is enabled:

~$ sudo timedatectl set-ntp true~

Restart the NTP Daemon:

~$ sudo systemctl daemon-reload~

*** DONE iOS shortcut to quickly see status and disable/enable Pi-hole
Instructions as per this
[[https://www.reddit.com/r/pihole/comments/g52hna/how_to_control_pihole_from_your_iphone_home_screen/][reddit
thread]].
1. In iOS, go to =Settings, Shortcuts= and enable =Allow Untrusted Shortcuts=.
2. With iOS browser, go to https://routinehub.co/shortcut/5005 and download the shortcut.
3. Edit it in Shortcuts and scroll down to section =DICTIONARY=, and add under settings values:

#+ATTR_HTML: :width 100%
| Settings           | Value                |
|--------------------+----------------------|
| Pi-hole Server     | http://10.0.1.245    |
| API Key            | YOUR_API_KEY         |

API Key can be found in Pi-hole =Settings=, =API / Web interface= and =Show API token=.

*** NEXT Control panel for system statistics
- As per instructios
  [[https://www.reddit.com/r/raspberry_pi/comments/glooc2/control_panel_for_raspberry_pi_4/][here]].

*** DONE Review whether Raspberry PI OS is anymore trustworthy and start migrating to another OS
Reasoning in this
  [[https://www.reddit.com/r/linux/comments/lbu0t1/microsoft_repo_installed_on_all_raspberry_pis/][Reddit
  thread]]. tl;dr: In a recent update, the Raspberry Pi Foundation installed
  a Microsoft vscode apt repository on all machines running Raspberry Pi OS.

Another OS could be [[https://wiki.debian.org/RaspberryPi][Debian]]
  (thought it has a
  [[https://www.reddit.com/r/raspberry_pi/comments/tes9fp/performance_difference_between_debian_armel_and/][performance
  penalty compared to Raspberry Pi OS]]) and
  [[https://gwolf.org/2022/04/how-is-the-free-firmware-for-the-raspberry-progressing.html][non-free
  firmware]].

*** NEXT Enable UEFI secure boot
Do this for the next version image: https://lobste.rs/s/gls60k/uefi_secure_boot_on_raspberry_pi

* WireGuard
** Philosophy
When outside of my home network, I can:

1. route traffic through the pi-hole with WireGuard,

2. block advertisements everywhere[fn:11],

3. gain access to Finnish georestricted content when abroad,

4. and gain access to my home network.

[[https://www.wired.com/story/WireGuard-gives-linux-faster-secure-vpn/][What is WireGuard]]?

[[https://www.vpnranks.com/blog/WireGuard-vs-openvpn/][Why WireGuard, and not OpenVPN]]?

In summarum WireGuard comparison to OpenVPN: WireGuard is simple to setup on
server side, roaming when using mobile devices (connection loss is rare),
quicker to establish connection and significantly less battery drain on
mobile devices, and does not rely on AES-NI encryption making it ideal for ARM-based boards.

#+begin_note
*Note:* Those with faster than 100 Mbps connection with WireGuard, may want
to go for rpi4, as it has a true gigabit wired LAN adapter onboard (rpi3
adapter is hampered by the shared USB 2.0). My connection is Elisa 100 down /
10 up so rpi3 is enough.
#+end_note

** Preparation

- As per [[https://www.reddit.com/r/pihole/comments/bnihyz/guide_how_to_install_WireGuard_on_a_raspberry_pi/][guide]] at r/pihole [fn:5]. Alternative, easier, but in this document an untested method is [[https://www.reddit.com/r/WireGuard/comments/f7ppty/WireGuard_on_a_raspberry_pi/][scripted piVPN]] installation.

Check your connection speed [fn:12]:

~$ sudo aptitude install speedtest-cli~

#+BEGIN_SRC bash
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Elisa Oyj (88.115.55.164)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Netplaza Oy (Helsinki) [7.91 km]: 18.786 ms
Testing download speed................................................................................
Download: 88.16 Mbit/s
Testing upload speed......................................................................................................
Upload: 10.43 Mbit/s
#+END_SRC

- If download speed is greater than 100 Mbit/s, then connection speed bottleneck can be avoided with rpi4.

** Pre-installation tasks on Raspberry Pi OS

*** DONE Verifying I have publicly-reachable IP address provided by my ISP

If ISP uses CGNAT (Carrier-Grade NAT), then I will not have a publicly-reachable IP address on my home network. This limits my ability to run a private VPN, since I don't have a direct gateway to the public Internet.

To verify this, I need to check public IP, also known as WAN IP, either from my Asus RT-AC86U router dashboard at:

=General - Network Map=, or by running:

~$ curl ifconfig.me~

If either of the methods return an IP that is of range =100.xxx.xxx.xxx= or
=10.xxx.xxx.xxx= or =192.xxx.xxx.xxx=, then it is an indication ISP uses CGNAT.

For ultimate verification, run traceroute to inspect the amount of network hops:

~$ curl ifconfig.me | xargs traceroute~

I have a result of a *single hop* in the results, which means I'm *not behind
CGNAT*:

#+BEGIN_SRC bash
$ curl ifconfig.me | xargs traceroute
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13  100    13    0     0     68      0 --:--:-- --:--:-- --:--:--    68
traceroute to 88.115.55.164 (88.115.55.164), 30 hops max, 60 byte packets
 1  88-115-55-102.elisa-laajakaista.fi (88.115.55.102)  0.447 ms  0.272 ms  0.215 ms
#+END_SRC

If there is more than single hop, I'm behind ISPs CGNAT and I need to request a
publicly routable IP address, or instead of self-hosting the WireGuard VPN,
consider using a service like Tailscale, Cloudflare Tunnel, or Twingate.

** Installation on Raspberry Pi OS
#+begin_note
*Note:* Alternative method to Debian unstable repository below is to [[https://kaspars.net/blog/WireGuard-raspberry-pi][compile from source]].
#+end_note

~$ sudo aptitude install raspberrypi-kernel-headers~

*** NEXT Installation using Debian Unstable sources
As per [[https://www.sigmdel.ca/michel/ha/WireGuard/WireGuard_02_en.html#buster][instructions
here]]. *Update:* Rewrite these instructions (check the link) since September 2021, WireGuard
package has been available in RasperryPI OS repository.

~$ echo "deb http://deb.debian.org/debian/ unstable main" | sudo tee --append /etc/apt/sources.list.d/unstable.list~

~$ wget -O - https://ftp-master.debian.org/keys/archive-key-$(lsb_release -sr).asc | sudo apt-key add -~

~$ printf 'Package: *\nPin: release a=unstable\nPin-Priority: 150\n' | sudo tee --append /etc/apt/preferences.d/limit-unstable~

~$ sudo aptitude update~

- After this, loads of new package updates are available from Sid, something
like:

#+BEGIN_SRC bash
Current status: 12756 (+12755) new.
#+END_SRC

- Now I have the latest version of WireGuard (0.0.20190601-1)
  available from debian sid repositories, install that with dependencies:

~$ sudo aptitude install WireGuard~

** Post-installation configuration tasks

- Once WireGuard is installed, enable IP Forwarding in raspberry pi, then reboot the Pi:

~$ perl -pi -e 's/#{1,}?net.ipv4.ip_forward ?= ?(0|1)/net.ipv4.ip_forward = 1/g' /etc/sysctl.conf~

~$ reboot~

- IPv4 port forwarding is now enabled in Pi, ensure that output of this is 1:
~$ sysctl net.ipv4.ip_forward~

*** DONE Generate private and public keys

- Keys are generated for WireGuard server and a client:

~$ sudo su~

~# cd /etc/WireGuard~

~# umask 077~

~# wg genkey | tee client1_privatekey | wg pubkey > client1_publickey~

~# wg genkey | tee server_privatekey | wg pubkey > server_publickey~

- Four keys are now generated.

*** DONE Configure WireGuard server on Raspberry pi

~# nano /etc/WireGuard/wg0.conf~

- =wg0.conf= should contain:

#+BEGIN_SRC bash
[Interface]
Address = 10.9.0.1/24
ListenPort = 51819
PrivateKey = {server_privateKey}
DNS = 10.0.1.245

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
#Client1
PublicKey = {client1_publicKey}
AllowedIPs = 10.9.0.2/32, 10.0.1.245/32
PersistentkeepAlive = 60
#+END_SRC

Where,
- Address :: Server's internal IP address to bypass the traffic through
  WireGuard. We get to assign it. Let's use =10.9.0.1/24=. This should be assigned outside of router's DHCP IP pool range.
- ListenPort :: In order to access home network from outside, configure a
  port forward. This is the external UDP port I've forwarded in router (=WAN= - =Virtual
  Server/Port Forwarding=). WireGuard's default port is =51820=, but we'll opt
  on using =51819=. Nondefault listenPort is handy, if e.g. using
  [[https://www.reddit.com/r/WireGuard/comments/c2p8m6/WireGuard_on_port_51820_blocked_by_public_wifi/][public
  wifi]]. Furthermore, choosing port below 1024
  [[https://www.reddit.com/r/WireGuard/comments/lyzbof/WireGuard_blocked_by_isp_how_to_circumvent/gpx94bl/?context=3][may help circumventing
  blocking]] established by various providers. E.g. using port 989 (FTPS).
- PrivateKey :: Value from ~$ cat server_privatekey~
- PostUp & PostDown :: Change =eth0= to =wlan0= in both lines if raspberry pi
  with pi-hole is connected via Wi-Fi, otherwise =eth0= (this is presumed).
- PublicKey :: Value from ~$ cat client1_publickey~
- AllowedIPs :: Client-IP/32 (keep it as is), Pi-hole's IP
- PersistentkeepAlive :: this line is uncommented (remove the #) to keep the connection alive as I'm behind a NAT.

*** DONE Configure client config file on Raspberry pi

- The following config (=client1.conf=) goes ultimately to my iOS WireGuard,
  but we configure the file on rpi3 and then export it using qrencode, as it
  is faster this way.

- Create the config file by doing:

~# nano /etc/WireGuard/client1.conf~

- =client1.conf= should contain:

#+BEGIN_SRC bash -n -r
[Interface]
Address = 10.9.0.2/32			(ref:address)
DNS = 10.0.1.245				(ref:dns)
PrivateKey = {client1_privatekey}	(ref:private)

[Peer]
PublicKey = 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=	(ref:public)
Endpoint = 88.115.55.164:51819	(ref:endpoint)
AllowedIPs = 0.0.0.0/0, ::/0		(ref:allowedips)
#PersistentkeepAlive = 60
#+END_SRC

Where,

- Line [[(address)]] = Assigned IP of client
- [[(dns)][DNS]] = Pi-Hole’s IP
- [[(private)][Privatekey]] = value from ~$ cat client1_privatekey~
- [[(public)][Publickey]] = value from ~$ cat server_publickey~
- [[(endpoint)][Endpoint]] = =public-ip= : =WireGuard-port-forwarded-in-router=.
  Current public-ip value can be checked by doing

  ~$ curl ifconfig.me~

#+begin_note
  (*Note*, I may get another public IP from my ISP, at latest when I reboot
  the router. It needs to be then updated to iOS WireGuard config).
#+end_note

[[(allowedips)][AllowedIPs]] = =0.0.0.0/0, ::/0= (where =0.0.0.0/0, ::/0= allows all traffic to route through WireGuard
 aka full access to my home network). *Note*: =::/0 or ::0/:=) are the same thing; they're there to [[https://www.reddit.com/r/WireGuard/comments/b0m5g2/ipv6_leaks_psa_for_anyone_here_using_WireGuard_to/][prevent ipv6 leakage]].

OR Instead of full tunnel, if I wanted a split tunnel:

- AllowedIPs :: =88.115.55.164/32 10.9.0.1/24= (where =10.9.0.1/24= is the
  subnet of my tunnel, and =88.115.55.164/32= is my public IP. Thus, this
  split tunnel setup would allow traffic only from iOS WireGuard tunnel to my home computer).

*** DONE Export the client configuration to iOS WireGuard using QR Code

~# sudo su~

~$ sudo aptitude install qrencode~

~$ sudo qrencode -t ansiutf8 < /etc/WireGuard/client1.conf~

- A QR code will be generated, you will need to scan this code and import it
  to the WireGuard app on the phone by tapping =Add Tunnel= and then =Create from QR Code=.

- After that, your iOS WireGuard tunnel is created with the following
  settings:

#+CAPTION: client1 settings in WireGuard iOS
[[https://pyyhttu-siilo.kapsi.fi/nextcloud/index.php/s/Photo_2019-07-15_1371/preview]]

*** DONE Finalize installation

- Enable WireGuard to autostart on boot:

~$ systemctl enable wg-quick@wg0~

- Secure the contents of =/etc/WireGuard= as it contains your private keys:

~$ sudo chown -R root:root /etc/WireGuard/~

~$ sudo chmod -R og-rwx /etc/WireGuard/*~

- Go to Pi-Hole Admin console at =Settings - DNS= and tick mark the following then save:

#+BEGIN_SRC bash
Listen on all interfaces (Allows only queries from devices that are at most one hop away (local devices)
#+END_SRC

- Then start WireGuard by doing:

~$ sudo wg-quick up wg0~

*** DONE Test that WireGuard tunnel is working

1. Run a DNS leak test by browsing to dnsleaktest.com and executing there "Standard test". Public IP should be visible.

2. In the iOS app, check that data is exchanging in lines “Data sent” and “Data received” while connected.

On WireGuard server side traffic can be seen by inspecting the WireGuard routing table and peer connections with:

~$ sudo wg show~

#+BEGIN_SRC bash
interface: wg0
  public key: 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=
  private key: (hidden)
  listening port: 51819

peer: /3pz2xGhL+jTs3h9+nQ6fTeokvIPduj22afN1LYDn20=
  endpoint: 85.76.86.141:31960
  allowed ips: 10.9.0.2/32, 10.0.1.245/32
  latest handshake: 1 minute, 4 seconds ago
  transfer: 693.34 KiB received, 23.40 MiB sent
  persistent keepalive: every 1 minute
#+END_SRC

The "endpoint" ip of =85.76...= above indicates my phone is attached to
cellular network while WireGuard tunnel is on. If I was in my home wlan with
WireGuard, the endpoint ip would be =10.0.1.1=.

1. Check the ‘Query Log’ page in Pi-hole’s Admin console. You’ll see queries
coming from the wg client IP (=10.9.0.2= in this case).

2. Check status of WireGuard:

#+BEGIN_SRC bash
root@raspberrypi:/etc/WireGuard# systemctl status wg-quick@wg0.service
● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2019-07-01 18:31:53 BST; 2 weeks 0 days ago
     Docs: man:wg-quick(8)
           man:wg(8)
           https://www.WireGuard.com/
           https://www.WireGuard.com/quickstart/
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg-quick.8
           https://git.zx2c4.com/WireGuard/about/src/tools/man/wg.8
 Main PID: 604 (code=exited, status=0/SUCCESS)

Jul 01 18:31:51 raspberrypi systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
Jul 01 18:31:51 raspberrypi wg-quick[604]: [#] ip link add wg0 type WireGuard
Jul 01 18:31:51 raspberrypi wg-quick[604]: [#] wg setconf wg0 /dev/fd/63
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip -4 address add 10.9.0.1/24 dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip link set mtu 1420 up dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] resolvconf -a tun.wg0 -m 0 -x
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] ip -4 route add 10.0.1.245/32 dev wg0
Jul 01 18:31:52 raspberrypi wg-quick[604]: [#] iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j AC
Jul 01 18:31:53 raspberrypi systemd[1]: Started WireGuard via wg-quick(8) for wg0.
#+END_SRC

Traffic should be routed through pi-hole. Check that by doing:

~$ ip r | grep default~

#+BEGIN_SRC bash
default via 10.0.1.1 dev eth0 src 10.0.1.245 metric 202
#+END_SRC

~$ ip address show wg0~

#+BEGIN_SRC bash
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.8.0.1/24 brd 10.8.0.255 scope global wg0
       valid_lft forever preferred_lft forever
    inet 10.9.0.1/24 scope global wg0
       valid_lft forever preferred_lft forever
#+END_SRC

*** NEXT Measure overhead
"Sadly, due to overhead, you'll get a little less bandwidth than if you were on the network directly. But bandwidth is adequate for my purposes, assuming I'm on a stable Internet connection.

You can test the overhead from the VPN connecting while you're on your home LAN. Try this, and run a speed test on https://speed.cloudflare.com, once without the VPN connected, and once connected. Compare the two speeds and that's the overhead of the VPN connection.

On my LAN, the download speed goes from around 600 Mbps to 237 Mbps, but upload remains close to the measly 35 Mbps Spectrum gives me.

And connected through my iPhone on AT&T over a pretty poor signal (2/4 bars), I can get 32 Mbps down and 4 up through the VPN—not too bad! That's a little under line speed on the iPhone through AT&T directly, but it's workable, and has saved me a few times when I needed to grab something from the LAN remotely!"

Source: https://www.jeffgeerling.com/blog/2023/build-your-own-private-WireGuard-vpn-pivpn

** Maintenance

*** DONE Setting up new client accesses (manually)

On server side =10.0.1.245= create new key pair for the client to be added:

~$ sudo su~

~# cd /etc/WireGuard && wg genkey | tee client2_private_key | wg pubkey > client2_public_key~

Then register them on the server:

~# wg set wg0 peer <client2_public_key> allowed-ips <new_client_vpn_IP>/32~

Where,

| Parameter    | Description                                |
|--------------+--------------------------------------------|
| client2_publickey | Value from ~$ cat client2_public_key~|
| new_client_vpn_IP| 10.9.0.3/32|


On client side, install WireGuard on the client OS. Depending on the OS,
apply the installation instructions from [[https://www.WireGuard.com/install/][official website]].
On Ubuntu based distro:

~# su pi~

~$ sudo add-apt-repository ppa:WireGuard/WireGuard~

~$ sudo apt-get update~

~$ sudo apt-get install WireGuard-dkms WireGuard-tools resolvconf linux-headers-$(uname -r)~

On client OS side, create =wg0-client.conf= with:

#+BEGIN_SRC bash
[Interface]
Address = 10.9.0.3/32
PrivateKey = <insert_client2_privatekey>
DNS = 10.0.1.245

[Peer]
PublicKey = <insert server_publickey>
Endpoint = <insert_vpn_server_address>:51819
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 60
#+END_SRC

On client side, bring up the VPN interface and form a tunnel by doing:

~$ sudo wg-quick up wg0-client~

Tunnel should now be active. Check that by doing:

~$ sudo wg show~

To disconnect do:

~$ sudo wg-quick down wg0-client~

*** FOLLOWUP Setting up new client accesses (automatically)

Use
[[https://www.reddit.com/r/WireGuard/comments/g2kpio/WireGuard_config_generator_free_webbased_tool/][WireGuard
Config Generator]] or
[[https://www.reddit.com/r/WireGuard/comments/hda2ih/easy_yet_powerful_script_for_configuring/][WireGuard-Tools
Script]].

*** FOLLOWUP Client access revocation

- If needed to revoke the client from the server, remove the
  =[Peer]= block related to your revoked client from the server's
  configuration =wg0.conf=, then restart your VPN server with:

~$ wg-quick down wg0~

~$ wg-quick up wg0~

*** DONE WireGuard kernel module needs to be recompiled

- This is to be done when new kernel-headers get updated after doing ~$ sudo aptitude safe-upgrade~

- After updating the headers, purge both =WireGuard= and
  =WireGuard-dkms=:

~$ sudo aptitude purge WireGuard WireGuard-dkms~

- Then reinstall them:

~$ sudo aptitude install WireGuard WireGuard-dkms~

- Lastly reboot with ~$ sudo reboot~ and check tunnel status with ~$ wg show~

** Troubleshooting

*** DONE WireGuard tunnels collapsed on router firmware update
Updated Asus stock firmware to newest version (3.0.0.4.386_45956) but
after this none of the clients using WireGuard VPN worked. This was evident
from ever increasing time stamps of latest handshakes:

~$ sudo wg show~

#+BEGIN_SRC
pyyhttu@raspberrypi:~ $ sudo wg show
interface: wg0
  public key: 0zV8aMt5Yu9jyL9jTOhR574/zX2kJdr21PuhpcamdmE=
  private key: (hidden)
  listening port: 51819

peer: /3pz2xGhL+jTs3h9+nQ6fTeokvIPduj22afN1LYDn20=
  endpoint: 10.0.1.1:1026
  allowed ips: 10.9.0.2/32
  latest handshake: 17 minutes, 50 seconds ago
  transfer: 8.79 GiB received, 45.61 GiB sent
  persistent keepalive: every 1 minute

peer: guoJge3walbBMS+PMWVKpBDPkM7JaIxIcrb4xlBfnzU=
  endpoint: 10.0.1.1:1025
  allowed ips: 10.9.0.4/32
  latest handshake: 18 minutes, 33 seconds ago
  transfer: 610.86 MiB received, 5.83 GiB sent

peer: P0eDIPcZPSuELfma6DScKSriYXn/N/JzmeHUkHfqrz8=
  endpoint: 10.0.1.1:1024
  allowed ips: 10.9.0.3/32, 10.0.1.245/32
  latest handshake: 24 minutes, 23 seconds ago
  transfer: 264.57 MiB received, 3.75 GiB sent
  persistent keepalive: every 1 minute
pyyhttu@raspberrypi:~ $
#+END_SRC

Resolved by restarting WireGuard by doing:

~$ sudo wg-quick down wg0 && sudo wg-quick up wg0~

*** DONE WireGuard client keys were removed after connection loss to ISP

Noticed a connection break on [2022-04-04 Mon]. This was also
registered by Monit and healthchecks.io. Then noticed that my iOS with
WireGuard enabled (on-demand activated), was not able to resolve anything.

iOS WireGuard logs showed an error around the connection break:

#+BEGIN_SRC bash
Removing all keys, since we haven't received a new one in 540 seconds
#+END_SRC

Restarting the wg0 service on server by doing ~$ sudo wg-quick down wg0 &&
sudo wg-quick up wg0~ did not help.

Decided to recreate the iOS client keys so that first deleted the keys in
iPhone WireGuard (delete tunnel) and then on server recreated the client keys
at =/etc/WireGuard/= by doing:

~# qrencode -t ansiutf8 < /etc/WireGuard/client3.conf~

After scanning the qr-code and receiving the keys, the iOS WireGuard started
to work again.

** Observations and future direction

*** DONE Install WireGuard client and config on Linux laptop
- Then repeat and document the connection speed test as per the [[https://oct8l.gitlab.io/posts/2019/141/WireGuard-performance-with-a-pi-3-a-/][guide here]].
- Results: It seems to be that the result are on par with what was presented in the link.
- Rpi3's network adapter with my current connection speed is more than enough to act as a WireGuard server.

*** DONE Setup dynamic DNS =Dynu= and =ddclient= to maintain connection
- In order to connect to my public home IP where rpi3 WireGuard VPN is running, I need
  to tie that IP to a dynamic name service (ddns).
- My public home IP changes all the time and thus forces me to update
  it to server side =/etc/WireGuard/client1.conf= as well as to WireGuard app
  on client side.

- Register to a dynamic DNS provider, e.g. to [[https://www.dynu.com/en-US/][dynu.com]].

- Choose =Option 1: Use our Domain=.

- Define wanted domain to be used (e.g. =rpivpn.mywire.org=) and click =Add=.

- Go to =Control Panel= - =My Account= and define your =Time Zone=.

- In rpi3 terminal, test your domain by doing

~$ nslookup rpivpn.mywire.org~

This should return the same IP as shown in Dynu
  [[https://www.dynu.com/en-US/ControlPanel/DDNS][=Control Panel= - =DDNS
  Services=]].

- In rpi3 terminal, install =ddclient=, which will recognize when your public IP changes, and updates this to point to =rpivpn.mywire.org=:
~$ sudo aptitude install ddclient libio-socket-ssl-perl~

- Bypass the setup wizard in every step by pressing {{{kbd(esc)}}}. Setup =ddclient= manually:
~$ sudo nano /etc/ddclient.conf~:

#+BEGIN_SRC
daemon=300
syslog=yes
ssl=yes
protocol=dyndns2
use=web, web=checkip.dynu.com/, web-skip='IP Address'
server=api.dynu.com
login=your-dyny-login
password='your-dynu-password'
rpivpn.mywire.org
#+END_SRC

- Test IP update works to =dynu.com= by forcing an update manually:
~$ sudo ddclient -force -verbose~

- This update should be registered in dynu =control panel= (observable by the updated time stamp).

- Automate this update to be every 5 minutes:
~$ sudo nano /etc/default/ddclient~

#+BEGIN_SRC
run_dhclient="false"
run_ipup="false"
run_daemon="true"
daemon_interval="300"
#+END_SRC

- Start =ddclient= as a service:
~$ sudo service ddclient start~

- Check the service is started correctly as =active: (running)= by doing:
~$ sudo service ddclient status~

- Test the automated update service works. Change IP to something to =1.2.3.4= in Dynu control panel at =DDNS Services=.

- Delete in rpi3 the =ddclient.cache=:
~sudo rm /var/cache/ddclient/ddclient.cache~

- Wait for 5 minutes after which =ddclient= should've updated the dummy IP =1.2.3.4= back to correct.

- Now =client1.conf= and =client2.conf= can be updated with dynamic dns =rpivpn.mywire.org= and distributed to clients with:
~sudo qrencode -t ansiutf8 < /etc/WireGuard/client1.conf~

*** NEXT Install WireGuard Dashboard
- As per the article [[https://www.reddit.com/r/WireGuard/comments/ocg9gj/new_updates_on_WireGuard_dashboard_v21/][here.]]

*** NEXT Find optimal MTU
- Use this [[https://www.reddit.com/r/WireGuard/comments/po1w4m/optimal_wg_server_peer_mtu_finder_part_2/][python-package]].

* Cloudflare DNS over HTTPS
** Philosophy

As per [[https://privacytests.org/][privacytests.org]]:

#+begin_quote
The Domain Name System (DNS) is the method by which web browsers look up the IP address for each website you visit. In a DNS query, a web browser will ask a DNS resolver (somewhere on the internet) for the IP address corresponding to a domain name (such as nytimes.com) for a website you want to visit. Traditionally, most web browsers have sent their DNS queries unencrypted, which means your ISP or anyone else on the network between your computer and the DNS resolver can eavesdrop on the websites you visit. In recent years, web browsers and operating systems have begun to introduce encrypted DNS, including the DNS over HTTPS (DoH) protocol, to encrypt the DNS request from your browser and the response from the resolver to keep your browsing history from leaking. These tests check whether a browser is still protecting its DNS requests by sending them encrypted.
#+end_quote

With Cloudflared, I can utlize DNS over HTTPS (DoH) encryption at my home network wide, [[https://developers.cloudflare.com/1.1.1.1/dns-over-https/][end-to-end]].

The following setup is done with [[https://docs.pi-hole.net/guides/dns-over-https/][these instructions]].

#+begin_note
*Note:* Cloudflared is a single point of failure from privacy point of
  view, as Cloudflared would have complete record of every DNS request I've made, [[https://www.reddit.com/r/pihole/comments/dtasmu/dnsoverhttps_will_eventually_roll_out_in_all/f6wgmdg/][all in one place]]. So it is a matter of trust. Unbound would solve this, as then I would resolve DNS myself, with a price of little bit slower DNS resolution time (but by how much, needs to be measured).
#+end_note

*** NEXT Measure DNS resolution time. Implement unbound, then measure again.
Document then  measure with
  [[https://www.reddit.com/r/pihole/comments/kh97o5/how_should_i_test_dns_performance/ggjzd7k/?context=3][this]].

** Preparation
Disable wireless from iphone, connect to cellural and form a WireGuard VPN
  tunnel. Then head to [[https://www.cloudflare.com/ssl/encrypted-sni][Cloudflare's probing page]] and
  check results with iOS Safari. I get:
#+BEGIN_SRC bash
[?] on Secure DNS
[✓] on DNSSEC
[✓] on TLS 1.3
[x] on Encrypted SNI
#+END_SRC

- Failure [x] or non-determination [?] on Secure DNS means, that anybody listening on the traffic can potentially see
  the DNS queries I make (and since I trust Cloudflare, obv. they can see them as well).

- Alternatively, you can run a check also at: https://1.1.1.1/help. Results
  should he the same.

- Note that TLS 1.3 and Encrypted SNI has nothing to do with Cloudflare or
  Pi-hole: those are measured and supported by my browser.

** Pre-installation tasks on Raspberry Pi OS

- None

** Installation on Raspberry Pi OS

~$ wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-arm.tgz~

~$ tar -xvzf cloudflared-stable-linux-arm.tgz~

~$ sudo cp ./cloudflared /usr/local/bin~

~$ sudo chmod +x /usr/local/bin/cloudflared~

~$ cloudflared -v~

~$ sudo systemctl enable cloudflared~

- Create a cloudflared user to run the daemon:

~$ sudo useradd -s /usr/sbin/nologin -r -M cloudflared~

- Proceed to create a configuration file for cloudflared by copying the
  following in to =/etc/default/cloudflared=. This file contains the
  command-line options that get passed to cloudflared on startup:

#+BEGIN_SRC bash
# Commandline args for cloudflared
CLOUDFLARED_OPTS=--port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query
#+END_SRC

- Update the permissions for the configuration file and cloudflared binary to
  allow access for the cloudflared user:

~$ sudo chown cloudflared:cloudflared /etc/default/cloudflared~

~$ sudo chown cloudflared:cloudflared /usr/local/bin/cloudflared~

- Then create the systemd script by copying the following in to
  =/lib/systemd/system/cloudflared.service=. This will control the running of
  the service and allow it to run on startup.

#+BEGIN_SRC bash
[Unit]
Description=cloudflared DNS over HTTPS proxy
After=syslog.target network-online.target

[Service]
Type=simple
User=cloudflared
EnvironmentFile=/etc/default/cloudflared
ExecStart=/usr/local/bin/cloudflared proxy-dns $CLOUDFLARED_OPTS
Restart=on-failure
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target
#+END_SRC

- Enable the systemd service to run on startup, then start the service and
  check its status:

~$ sudo systemctl enable cloudflared~

~$ sudo systemctl start cloudflared~

~$ sudo systemctl status cloudflared~

- Now test that it is working. Run the following dig command, a response
  should be returned similar to the one below:

~$ dig @127.0.0.1 -p 5053 google.com~

#+BEGIN_SRC bash
; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> @127.0.0.1 -p 5053 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3863
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b1e011d8a18c455e (echoed)
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             116     IN      A       172.217.21.142

;; Query time: 1 msec
;; SERVER: 127.0.0.1#5053(127.0.0.1)
;; WHEN: Sat Nov 30 13:28:16 EET 2019
;; MSG SIZE  rcvd: 77
#+END_SRC

** Post-installation configuration tasks

- Finally, configure Pi-hole to use the local cloudflared service as the
  upstream DNS server. Go to pi-hole at =10.0.1.245=, =Settings - DNS= and to
  =Upstream DNS Servers= under =Custom 1 (IPv4)= add:

[✓] =127.0.0.1#5053=

Remember to remove under =Upstream DNS Servers= tick boxes for Cloudflare if
had one setup.

Then press Save.

** Maintenance

*** FOLLOWUP Upgrade cloudflared binary version
- Check current version with:

~$ sudo cloudflared -v~

#+BEGIN_SRC bash
cloudflared version 2019.12.0 (built 2019-12-19-2256 UTC)
#+END_SRC

- Then run update:

~$ sudo cloudflared update~

#+BEGIN_SRC bash
INFO[0017] cloudflared has been updated to version 2020.2.0
#+END_SRC

- In case there isn't an update, one gets:

#+BEGIN_SRC bash
2021-04-24T08:44:50Z INF cloudflared is up to date version=
#+END_SRC

- Otherwise the update is
  applied. Notification of new updates can be gotten by subscribing to
  [[https://github.com/cloudflare/cloudflared/releases][Cloudflared release
  channel]].

** Troubleshooting

*** DONE Temporary failure in name resolution

- After did ~$ sudo reboot~, DNS resolution does not work. For example doing ~$ ping google.com~ gives:

#+BEGIN_SRC bash
ping: google.com: Temporary failure in name resolution
#+END_SRC

- Also doing ~$ sudo monit status~ gives "Connection failed" on remote host test.

- Both are because we use Cloudflared for DSN resolution but Pi-Hole
  automatically adds =127.0.0.1= to
  =/etc/resolvconf/run/resolv.conf=. You can see the file's [[https://discourse.pi-hole.net/t/how-do-i-disable-pi-hole-automatic-add-127-0-0-1-in-resolv-conf/18248/7][timestamp changed]] after reboot by doing

~$ ls -al resolvc.conf~

- So if using Cloudlflare for DNS resolution, then update
  =/etc/resolv.conf= back to =1.1.1.1=

- If/when going to unbound, this should be no longer be an issue as then an
  entry of =127.0.0.1= in =resolv.conf= is actually correct.

*Edit: [2021-08-28 Sat]* Contents of =resolv.conf= is actually (in working setup):

#+BEGIN_SRC bash
nameserver 10.0.1.245
nameserver 127.0.0.1
#+END_SRC

*** DONE Cloudflared test site shows DNS-over-HTTPS (DoH) as disabled

- After setting up Cloudflared, noticed after a while by visiting their
  [[https://1.1.1.1/help][test site]] that it indicated =Using DNS over HTTPS
  (DoH)= as =No=.
- Reason turned out to be the DNSSEC setting in pi-hole I had turned on, as
  per
  [[https://old.reddit.com/r/pihole/comments/jlnq70/why_wont_doh_work_with_my_cloudflared_setup/][this
  reddit-thread]].
  After disabling DNSSEC, the test site indicated correctly I have DoH
  enabled.

*** DONE Fix the update failure

When doing: ~$ sudo cloudflared update~ I get:

#+BEGIN_SRC bash
failed to update cloudflared: context deadline exceeded (Client.Timeout or
context cancellation while reading body)
#+END_SRC

Resolved this with instructions from
[[https://github.com/cloudflare/cloudflared/issues/253][github issue 253]].

*** DONE Cloudflared binary does not update

- When trying to update cloudflared with ~$ sudo cloudflared update~ I get:

#+BEGIN_SRC bash
pi@raspberrypi:~ $ sudo cloudflared update
2021-08-28T15:32:10Z INF cloudflared is up to date version=
pi@raspberrypi:~ $
#+END_SRC

- Also ~$ cloudflared -v~ does not give version info but:

#+BEGIN_SRC bash
pi@raspberrypi:~ $ cloudflared -v
cloudflared version DEV (built unknown)
pi@raspberrypi:~ $
#+END_SRC

- To resolve this, redownload the binary from the same link, and repeat the
  install procedure (same as [[https://docs.pi-hole.net/guides/dns/cloudflared/#if-you-configured-cloudflared-with-your-own-service-files][manually updating binary]].

- After this, doing ~$ cloudflared -v~ instead of giving "built unknown", I
  now get correctly:

#+BEGIN_SRC bash
pi@raspberrypi:~ $ cloudflared -v
cloudflared version 2021.8.6 (built 2021-08-27-2250 UTC)
#+END_SRC

** Observations and future direction

*** DONE DNSSec not enabled per Cloudflare browser check site
- DNSSEC provides validation that DNS responses are untampered and can
  be trusted. One such test can be found from
  [[https://www.cloudflare.com/ssl/encrypted-sni/][Cloudflare's Browsing
  Experience Security Check]] but the site
  [[https://www.reddit.com/r/pihole/comments/j0482t/pihole_dnssec_interfering_with_cloudflare_dns/][apparently
  sometimes provides false positives]], evident also from
  [[https://www.reddit.com/r/pihole/comments/k7h65v/how_do_i_enable_dns_over_https_and_dnssec/ger5a9h/?context=3][this
  post]].
- This is evident when disabling ="Use DNSSEC"= at pi-hole DNS Settings and
  rechecking the results at Cloudflare's site gives an OK for DNSSEC.
- A more reliable test would be to use a
  [[https://dnssec.vs.uni-due.de/][DNSSEC Resolver Test]].

*** FOLLOWUP Cloudlflared binary sometimes falls over and stops querying DNS
- I've noticed a similar symptoms sometimes as reported in [[https://www.reddit.com/r/pihole/comments/dkq33h/doh_on_pihole/f4khsdq/][this reddit thread]].

- Fix: If/when reinstalling Cloudflared, Instead of using
  =/etc/default/cloudflared=, make yourself a [[https://www.reddit.com/r/pihole/comments/dkq33h/doh_on_pihole/f4jlc2b/][Cloudflared yaml config file]].

- This is to be done if not going with Unbound.

* Unbound
** Philosophy
- [[https://ostif.org/our-audit-of-unbound-dns-by-x41-d-sec-full-results/][What is Unbound]].
- [[https://www.reddit.com/r/pihole/comments/dukbcf/i_love_pihole_but_how_can_i_use_it_and_also/f7770v9/][Why Unbound]].
- [[https://www.reddit.com/r/pihole/comments/fq2znr/what_are_the_advantages_of_using_unbound/flomee8/][Advantages of unbound]].

- Instead of Cloudflared, allows me to run my own recursive DNS server (answering the
  problem: [[https://docs.pi-hole.net/guides/unbound/][whom can I trust]]. Not
  enabled for now as I've chosen to [[https://www.theregister.co.uk/2018/04/03/cloudflare_dns_privacy/][trust Cloudflare]].
  However, Unbound has [[https://www.reddit.com/r/pihole/comments/dpcqu9/unbound_vs_cloudflare_doh/f5u8j2a/][its merits]].
- Since I will with Unbound be my own DNS-provider, I need to build high
  availability/fail over with second Raspberry Pi.

- If I later go with unbound, [[https://www.reddit.com/r/pihole/comments/dezyvy/into_the_pihole_you_should_go_8_months_later/][check link]] in reddit by user ukhaare.

*** NEXT Improve Unbound's performance
- [[https://www.reddit.com/r/pihole/comments/dl1fxe/questions_about_more_advanced_unbound/][Inspect]] the Unbound configurations to improve its performance.

- If problems with installation, consult the [[https://www.reddit.com/r/pihole/comments/e23jj1/question_for_those_running_pihole_unbound/][reddit thread]].

- If no problems, remove this and above.

*** NEXT After installation check the Cloudflare's check site
- Check that
  [[https://www.reddit.com/r/pihole/comments/eylcbd/pihole_unbound_not_passing_cloudflare_test/][Cloudlflare
  test]] gives with Firefox everything checked, and with iOS + VPN the top
  three checked (Encrypted SNI unchecked as not supported by mobile Safari).

*** NEXT Root-DNS
- Check if root-dns needs to be installed or will up-to-date root-dns package
  follow with Debian
  [[https://www.reddit.com/r/pihole/comments/fwgtda/how_exactly_does_unbound_work/fmo9dl1/][Unbound
  package]].

*** NEXT Forwarding resolver vs. recursive resolver
- Double check after installation that unbound is not running as a forwarding
  resolver, but as
  [[https://www.reddit.com/r/pihole/comments/fq2znr/what_are_the_advantages_of_using_unbound/flokt5a/][recursive
  resolver]].

*** Test Unbound

By doing

~$ dig google.com @127.0.0.1 -p 5353~

For more information see
[[https://www.reddit.com/r/pihole/comments/f4iuqm/unbound_server/][this]].


With cloudflare DNS doing ~$ dig google.com~ now returns:

#+BEGIN_SRC bash
$ dig google.com @127.0.0.1 -p 5353

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> google.com @127.0.0.1 -p 5353
;; global options: +cmd
;; connection timed out; no servers could be reached
#+END_SRC

*** NEXT Test DNSSEC results before and after enabling Unbound

As per the test in this
[[https://www.reddit.com/r/pihole/comments/euel7a/pihole_unbound_failing_almost_all_online_dnssec/][Reddit
thread]].
[[https://www.reddit.com/r/pihole/comments/kp1vq7/dnssec_and_cloudflared_enable_or_not_in_pihole/][The
recommendation ]] is to have the DNSSec of when using Unbound.

** Preparation

*** NEXT Document the parts for Raspberry Pi Zero W and usb-kit, and how to put it together

- Acquire [[https://www.amazon.de/gp/product/B07KR5PM7J/][USB-dongle]] and
  [[https://www.amazon.de/gp/product/B07C7FHJDX/][Pi Zero W]] with SD-card
  and put the kit together.
- Install
  [[https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/][Pi
  Imager]].
- With it, write minimal Rasbian OS
  ([[https://www.raspberrypi.org/software/operating-systems/][Raspberry Pi OS
  Lite]]) on the SD card.
- The Raspberry Pi OS images no longer have SSH enabled by default, but it’s easy to
  enable it; place a file named =ssh= (without any extension) onto the
  =boot=-partition of the SD card after you have written the disk image.
- Also enable the wireless for Raspberry Pi Zero W by creating to same =boot=
  partition a file called =wpa_supplicant.conf= with:

#+BEGIN_SRC bash
country=FI
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
	ssid=YOUR_WIFI_NAME
	psk=YOUR_WIFI_PASSWORD
	key_mgmt=WPA-PSK
}
#+END_SRC

#+begin_note
Raspberry Pi Zero W does not support 5 GHz, make sure you have 2.4 GHz enabled on the SSID that you are connecting to.
#+end_note

#+begin_note
In =wpa_supplicant.conf= the =psk​=your-wifi-password= is in clear text, but
the config's password can be replaced by its hash by doing ~$ wpa_passphrase
YOUR_WIFI_NAME YOUR_WIFI_PASSWORD~. Then, in the config use the resulting hash instead,
e.g. =psk​=6a24edf1592aec4465271b7dcd204601b6e78df3186ce1a62a31f40ae9630702=.
#+end_note

- Attach RPi Zero W to your router's usb port.

- Change Raspberry's IP to static by logging in to your router (in Asus
  RT-AC86U go to: =LAN - DHCP Server: Enable=
  =Manual Assignment: Yes=).
- Select raspberrypi from the client's list and bind it to: =10.0.1.246=.
- Once done, ssh into raspberry:

~$ ssh pi@10.0.1.246~

Ensure that pi user can log in also in the future SSH sessions. Add user pi
to the ssh user group:

~$ sudo adduser pi ssh~

Leaving the default user pi, with the default password is a security risk.
Add your own user, passwordless login is established later:

~$ sudo adduser pyyhttu~
#+BEGIN_SRC bash
Adding user `pyyhttu' ...
Adding new group `pyyhttu' (1002) ...
Adding new user `pyyhttu' (1001) with group `pyyhttu' ...
Creating home directory `/home/pyyhttu' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for pyyhttu
Enter the new value, or press ENTER for the default
        Full Name []: Tuomas Pyyhtiä
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
chfn: name with non-ASCII characters: 'Tuomas Pyyhtiä'
Is the information correct? [Y/n] Y
#+END_SRC

Check groups pi users is associated to:

~$ groups~
#+BEGIN_SRC bash
pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh gpio i2c spi
#+END_SRC

Add user pyyhttu to all the same groups:

#+BEGIN_SRC bash
$ sudo usermod -a -G pi,adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,ssh,gpio,i2c,spi pyyhttu

pyyhttu : pyyhttu pi adm dialout cdrom sudo audio video plugdev games users input netdev ssh spi i2c gpio
#+END_SRC

Change hostname. This is good practice to do in case we have multiple
machines so we know which is which:

~$ sudo nano /etc/hostname~

Delete the old name and setup new name.

~$ sudo nano /etc/hosts~

Replace any occurrence of the existing computer name with your new one.

Reboot the system for changes to take effect:

~$ sudo reboot~

Wait couple of minutes for reboot to finish, then log in as the new user, and
delete the old pi user:

~$ ssh pyyhttu@10.0.1.246~

~$ sudo deluser pi~

#+BEGIN_SRC bash
Removing user `pi' ...
Done.
#+END_SRC

Now we will allow our new user to execute sudo without providing a password,
because password fatigue is a thing (at least for me).
First, we delete the sudoers config for the user pi since we already deleted
that user:

~$ sudo rm /etc/sudoers.d/010_pi-nopasswd~

Then we make a new file for the new user:

~$ sudo nano /etc/sudoers.d/pyyhttu-nopasswd~ and with content:

#+BEGIN_SRC bash
pyyhttu ALL=(ALL) NOPASSWD: ALL
#+END_SRC

Change permission for that file:

~$ sudo chmod 440 /etc/sudoers.d/pyyhttu-nopasswd~

** Pre-installation tasks on Raspberry Pi OS

*** NEXT Validate DNS latency and throughput performance

Before uninstalling Cloudflared and installing Unbound, test with a shell
script the performance of the most popular DNS resolvers from your location
with [[https://github.com/cleanbrowsing/dnsperftest][dnsperftest]]:

~$ git clone --depth=1 https://github.com/cleanbrowsing/dnsperftest/~

~$ cd dnsperftest~

~$ bash ./dnstest.sh~

~$ bash ./dnstest.sh |sort -k 22 -n~

#+BEGIN_SRC bash
                  test1   test2   test3   test4   test5   test6   test7   test8   test9   test10  Average
127.0.0.1         1 ms    18 ms   1 ms    1 ms    23 ms   1 ms    1 ms    1 ms    1 ms    1 ms      4.90
cloudflare        14 ms   14 ms   14 ms   31 ms   15 ms   14 ms   14 ms   14 ms   14 ms   15 ms     15.90
quad9             17 ms   18 ms   17 ms   18 ms   28 ms   43 ms   18 ms   18 ms   17 ms   18 ms     21.20
yandex            14 ms   14 ms   15 ms   15 ms   42 ms   73 ms   15 ms   14 ms   15 ms   40 ms     25.70
google            20 ms   19 ms   29 ms   29 ms   40 ms   65 ms   19 ms   29 ms   19 ms   29 ms     29.80
cleanbrowsing     35 ms   37 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms     35.20
adguard           36 ms   37 ms   35 ms   35 ms   35 ms   35 ms   35 ms   35 ms   36 ms   35 ms     35.40
comodo            36 ms   46 ms   37 ms   38 ms   36 ms   36 ms   36 ms   36 ms   36 ms   39 ms     37.60
norton            20 ms   20 ms   20 ms   20 ms   21 ms   20 ms   20 ms   20 ms   20 ms   211 ms    39.20
opendns           29 ms   29 ms   29 ms   49 ms   30 ms   145 ms  28 ms   48 ms   29 ms   29 ms     44.50
neustar           51 ms   48 ms   48 ms   55 ms   38 ms   38 ms   38 ms   48 ms   38 ms   52 ms     45.40
level3            47 ms   48 ms   46 ms   47 ms   47 ms   47 ms   47 ms   46 ms   47 ms   47 ms     46.90
freenom           20 ms   83 ms   20 ms   261 ms  201 ms  167 ms  52 ms   243 ms  20 ms   86 ms     115.30
#+END_SRC

Run the test couple of times to see the actual results, note them down, then
proceed to uninstall Cloudflared:

** Installation on Raspberry Pi OS

*** NEXT One guide to check
- Here: https://thesmashy.medium.com/building-a-pihole-for-privacy-and-performance-f762dbcb66e5

*** NEXT Settings for Merlin with unbound
See linked screenshot in [[https://www.reddit.com/r/pihole/comments/k9pbp8/2_x_pihole_unbound_and_asus_merlin_wrt_what_the/][here]].

** Post-installation configuration tasks

*** NEXT Test Unbound resolves correctly
- Instructions [[https://www.reddit.com/r/pihole/comments/o40quc/how_to_check_if_unbound_is_working_as_recursive/][here]].

** Maintenance

** Troubleshooting

** Observations and future direction

*** NEXT Update =root.hints= (think of cron)
- and installing more
  [[https://www.reddit.com/r/pihole/comments/ee9lah/updating_unbound/][up-to-date
  Unbound]].
  [[https://www.reddit.com/r/pihole/comments/kc1bv2/do_i_need_to_manually_download_the_roothints_in/gfmjah0/][Setup
  with cron]] to update the =root.hints= every 6 months with
  something like ~$ wget https://www.internic.net/domain/named.root -O /var/lib/unbound/root.hints~

*** NEXT [[https://www.reddit.com/r/pihole/comments/id9178/check_that_unbound_is_doing_the_caching/][Check that unbound is doing caching]]

*** NEXT Ensure after uninstalling Cloudflared that it is indeed uninstalled
- by doing ~$ netstat -tunlp~ that cloudflare is not showing activity on ports
  44685 tcp and 54 tcp and udp, as per
  [[https://www.reddit.com/r/pihole/comments/kjxa77/cloudflared_still_shows_after_switch_to_unbound/][these
  instructions]].

* Monit and healthchecks.io
** Philosophy
- [[https://www.reddit.com/r/pihole/comments/bgzvem/how_to_health_checks_for_your_pihole/][Both]] are for system monitoring and automatic error recovery
- [[https://healthchecks.io/about][healthchecks.io]] is an open source,
  external service. Since it is external, it's very light weight and we can
  rely our internet connectivity check with it without exposing service by
  doing firewall/port forwardings. The service is based on "Dead
  man's switch" technique: if the healthchecks.io doesn't receive a heartbeat
  from my rpi3, I know network is down, and I will receive both SMS and email
  notifications.
- [[https://mmonit.com/monit/][Monit]] is an open source, lightweight (500
  kb) system monitoring and automatic error recovery software that sits
  locally on rpi.

** Preparation

*** Healthchecks.io
- Register a hobbyist account and setup a project to get an unique url to ping against.

*** Monit
- Nothing.

** Pre-installation tasks on Raspberry Pi OS
- None.

** Installation on Raspberry Pi OS

*** Healthchecks.io

- Setup an automated ping:

~$ crontab -e~

- And then in cronjob define a ping every second minute by adding a line:
#+BEGIN_SRC bash
0/2 * * * * dig @127.0.0.1 pi.hole && curl -fsS --retry 3 "https://hc-ping.com/your-unique-url"
#+END_SRC

*** Monit

~$ sudo aptitude install monit~

** Post-installation configuration tasks

*** Healthchecks.io

- At
  [[https://healthchecks.io/checks/"your-unique-url"/details/][https://healthchecks.io/checks/"your-unique-url"/details/]]
  schedule a period for two minutes and grace for 1 minute. Grace: If check
  is late, service waits 1 minute before sending me a notification.

- Define notification method as email and SMS.

*** Monit

~$ sudo nano /etc/monit/monitrc~

- Uncomment the following three lines first:

#+BEGIN_SRC bash
set httpd port 2812 and
use address localhost
allow localhost
#+END_SRC

Otherwise ~$ sudo monit status~ would give an error:

#+BEGIN_SRC bash
Cannot create socket to [localhost]:2812 -- Connection refused.
#+END_SRC

- To monitor =pihole-FTL=, append at the end of =monitrc=:

#+BEGIN_SRC bash -n -r
check process pihole-FTL with pidfile /var/run/pihole-FTL					(ref:process)
  start program = "/etc/init.d/pihole-FTL start" with timeout 20 seconds
  if failed port 4711 type tcp with timeout 5 seconds for 2 cycles then restart
  if 4 restarts within 5 cycles then unmonitor
#+END_SRC

Where,

- [[(process)][process]]: Pi-Hole’s IP

Save changes. Check that =monitrc= is correct, do ~$ sudo monit -t~. Then start monit with ~$ sudo monit~.

Monit will detach from the terminal and run as a background process, i.e. as
a daemon process. As a daemon, Monit runs in cycles. It monitors services,
then goes to sleep for a configured period, then wakes up and start
monitoring again in an endless loop. If =pihole-FTL= is found to be down, it
will be restarted automatically and findings are logged into
=/var/log/monit.log=.

#+begin_note
*Note:* Monit is run as root, because in case of error recovery, all programs
executed by Monit will then be started with superuser privileges. This is
wanted in our case, as we want to start =/etc/init.d/pihole-FTL= as root.
#+end_note

- If any changes are needed to be done to =/etc/monit/monitrc=, then make necessary modification and do:

~$ sudo monit -t && sudo service monit restart~

*** Setup email alerts
If there are finding, you'd like to receive an email. We'll use gmail for
sending those notifications.

Visit first [[https://myaccount.google.com/security][your
Google accounts security page]] and enable their 2-step verification.

Then visit [[https://myaccount.google.com/apppasswords][your Google account
App passwords]] and generate a password for ~monit~ to use.

Lastly, do ~$ sudo nano /etc/monit/monitrc~ and define:
#+BEGIN_SRC bash
set alert where.alert-is-sent@domain.com not on { instance, action } # do not receive alerts when monit starts/stops

set mailserver smtp.gmail.com port 587
        username "your-gmail-address@gmail.com" password "your-generated-google-app-password-for-monit"
        using tls
        with timeout 30 seconds
#+END_SRC

#+begin_note
*Note:* Deprecated method of setting up mail alert with Google was to enable
[[https://support.google.com/accounts/answer/6010255?p=less-secure-apps&hl=en&visit_id=637561368780686462-1693592209&rd=1#zippy=%2Cif-less-secure-app-access-is-on-for-your-account]["less
secure apps"]]-setting, which was discontinued on May 30, 2022. Instead, now
a 2-step verification is required, which *also* requires you to hand over to
Google your phone number for SMS or call for added verification.
#+end_note

** Maintenance

** Troubleshooting

*** DONE Avoid false positives by Monit and healthchecks.io when rebooting or backing up rpi3
- Rebooting or backing up rpi3 with raspiBackup, will result in alert emails triggered by
  monit daemon with topics such as: /"monit alert -- Connection failed
  pihole-FTL"/ or /"monit alert -- Connection failed google.com"/. The same
  emails are received from healthchecks.io.
- For monit, this is because monit starts to check services immediately after
  reboot, while the given services may not in fact be running yet.
- This can be resolved by editing =/etc/monit/monitrc=, define
  there =with start delay 120= to give 2 minute delay before first check is triggered
  after reboot.
- For healthchecks.io this is more complicated, since it is a /"dead man's
  switch"/ based service. Upon investigating on solutions, such as
  [[https://github.com/christronyxyocum/tronitor][Tronitor]], controlled pausing can
  be done, but it would involve combining Tronitor with
  [[https://github.com/christronyxyocum/HealthChecks-Linux][HealthChecks-Linux]],
  which has dependency to [[https://github.com/causefx/Organizr][Organizr]]
  setup relying on
  [[https://thehomeofthefuture.com/how-to/run-pi-hole-5-beta-behind-an-nginx-reverse-proxy/][a
  reverse proxy]], which then in turn requires installing and
  [[https://docs.pi-hole.net/guides/nginx-configuration/][configuring
  NGINX]]... This too many dependencies, so we'll live with healtcheck false
  positives for now: they're fully predictable, scheduled to happen inside 3
  min window, once a week when raspiBackup runs.

** Observations and future direction

There are several alternative software for monitoring, alerting and recovery:
- smokeping :: [[https://www.reddit.com/r/pihole/comments/6d9y2g/network_quality_monitoring/][Smokeping reddit thread]].

- Netdata :: As described [[https://www.reddit.com/r/pihole/comments/c7cxi6/pihole_stats_via_netdata_available/][here]] and [[https://www.reddit.com/r/pihole/comments/bwtx9q/netdata_and_pihole_discussion/][here]].

- RPi-Monitor :: As mentioned [[https://thesmashy.medium.com/building-a-pihole-for-privacy-and-performance-f762dbcb66e5][here.]]

*** FOLLOWUP Network downtime stats

Use [[https://github.com/geerlingguy/internet-pi][dockerized internet
  monitoring]] using Prometheus and Grafana. Do this once migrated over to
  RPi4.

* PADD
** Philosophy
- Shows Pi-hole status and various graphs. Can be put on a
  [[https://www.reddit.com/r/pihole/comments/j4v6xj/ive_updated_my_pihole_setup_with_pimoroni/][display]]
  [[https://giuliomagnifico.medium.com/pi-hole-unbound-on-a-raspberry-pi-4b-with-a-pimoroni-hyperpixel-4-padd-stats-b424a758d828][and
  stand]] to router.
- Serves also as a change management and awareness instrument in household and for guests:
  /"What's this info screen?/ /"Didn't know that many ads are blocked"/ /"Wait, can you
  spy with this my browsing habits?"/
- [[https://github.com/pi-hole/PADD/][PADD project]] is nowadays supported by
  the Pi-hole team.
- In case PADD does not cut it, there's an alternative, [[https://github.com/bpennypacker/phad][PHAD]].

** Preparation
PADD was originally designed for [[https://www.adafruit.com/product/2441][3.5" Adafruit display]] but I opted to go with cheaper a
  [[https://www.waveshare.com/product/displays/lcd-oled/lcd-oled-2/3.5inch-rpi-lcd-b.htm][Waveshare
  3.5" IPS screen]]. Available e.g. at [[https://shop.vadelmapii.com/tuote/35-kosketusnaytto/][vadelmapii]]. This display, (like a lot of the cheap displays) doesn't have the circuits to control the backlight via software, so the screens stays [[https://github.com/waveshare/LCD-show/issues/3][always
  on]], which if fine as impact to electricity consumption is negligible and
  the glow of the screen is not disturbing at all. The specs are available at [[https://github.com/waveshare/LCD-show#35inch-rpi-lcd-b][Waveshare model B
  github pages]].

** Pre-installation tasks on Raspberry Pi OS

*** DONE Install the display

The Raspberry Pi 3 has an
  [[https://www.raspberrypi-spy.co.uk/2014/08/enabling-the-spi-interface-on-the-raspberry-pi/][SPI]]
  (Serial Peripheral Interface) bus to which the display is connected. See
  [[https://www.waveshare.com/wiki/3.5inch_RPi_LCD_(B)#Hardware_Connection][Waveshare
  wiki]] for more detailed installation instructions.

*** DONE Enable console autologging and SPI

SPI is disabled by default, as is autologging which is needed for PADD
  display to emerge automatically after reboot. To enable both:

~$ sudo raspi-config~

Then select the following from the menu:

#+BEGIN_SRC bash
3 Boot Options
    B1 Desktop / CLI
        B2 Console Autologin

5 Interfacing Options
    P4 SPI
#+END_SRC

** Installation on Raspberry Pi OS

As per
  [[https://www.reddit.com/r/pihole/comments/8uxbiq/how_i_setup_pihole_to_show_stats_on_my_waveshare/][this
  reddit thread]].

*** DONE Install the drivers for Waveshare 3.5"

~$ git clone https://github.com/waveshare/LCD-show.git && cd LCD-show/~

~$ chmod +x LCD35B-show-V2~

~$ ./LCD35B-show-V2~

*** FOLLOWUP Alternative installation on Raspberry Pi OS Lite

Try next time with ~$ ./LCD35B-show-V2 lite~
  [[https://www.waveshare.com/wiki/3.5inch_RPi_LCD_(B)][as instructed]].

*** DONE Install PADD

~$ cd ~ && wget -N https://raw.githubusercontent.com/pi-hole/PADD/master/padd.sh~

~$ chmod +x padd.sh~

Set PADD to auto run by adding the following to the end of =~/.bashrc=:

#+BEGIN_SRC bash
# Run PADD
# If we’re on the PiTFT screen (ssh is xterm)
if [ "$TERM" == "linux" ] ; then
  while :
    do
        ./padd.sh
        sleep 1
    done
fi
#+END_SRC

Then reboot:

~$ sudo reboot~

** Post-installation configuration tasks

*** DONE Enable full screen usage and fonts

~$ sudo dpkg-reconfigure console-setup~

#+BEGIN_SRC bash
UTF-8 -> Guess optimal character set -> Terminus -> 8x14
#+END_SRC

** Maintenance

- Subscribe to new releases with PADD [[https://github.com/pi-hole/PADD/releases.atom][versions RSS-feed]].

- Once a new release is out, upgrade with:

~$ cd ~ && wget -N https://raw.githubusercontent.com/pi-hole/PADD/master/padd.sh~

*** Investigate how to enforce a seamless update to be visible on PADD display without issuing a reboot

Running ~$ ./padd.sh~ locally after a PADD update spawns a new PADD, but
not on the PADD display. Display still shows "Updates available".
Also reloading the =~/.bashrc= by doing ~$ source
~/.bashrc~ does not help either. The only way to get PADD running in the
display is to issue ~$ sudo reboot~ and let the =.bashrc= to handle the
update. *Edit:* I could try next to tweak term in .bashrc as per
[[https://github.com/pi-hole/PADD/issues/47][issue 47]] or issue ~$ ./padd.sh
> tty1~ as in [[https://github.com/pi-hole/PADD/issues/134][issue 134]].
*Update [2023-06-18 Sun]:* This issue has resolved itself. Noticed when
updated pihole with ~$ pihole -up~ and the tft screen's versions were
refreshed themselves to latest without reboot.

** Troublehooting

** Observations and future direction

* raspiBackup
** Philosophy
- For backups we use something lightweight and proven, so bash scripted
  solution, like [[https://github.com/framps/raspiBackup][raspiBackup]] is ideal.
- raspiBackup ensures the backups are available for fast system restore when
  eventually rpi3's SD-card wears out, resulting the DNS queries not
  resolving --> no internet connectivity.
- Without raspiBackup, I'd need to redo all the installation steps,
  manually.
- Instead, I can speed up the disaster recovery by just doing an install of a
  new barebone raspbian lite on a second SD card, boot it, install raspiBackup and do:
~$ sudo raspiBackup.sh -d /dev/sda /location-of-my/backup~

*** NEXT Measure how long restoration from USB takes

- Backing up takes in total something like 30 minutes on a first run. Later,
  the incremental weekly runs will take approximately between 3-6 minutes.

*** DONE Investigate on backupping to NFS share on NAS, or offsite to Kapsi with sshfs
As an option to USB-stick backups, I could
[[https://www.hackviking.com/single-board-computers/raspberry-pi/automated-raspberry-pi-backup-complete-image/][backup
directly]] to my home NAS. Another option is to transfer the backup via sshfs
to an offsite location, but local backups for now are preferred. This is because sshfs brings these additional complications:

- Network connection would be required. However, offsite backups for this
  system are not a must and restoring a backup when DNS queries are not
  resolving (Unbound being down) forms an
  added complexity.
- rsyncing to sshfs does not support hard linking, resulting in a [[https://www.linux-tips-and-tricks.de/en/faq/#a17][slightly increased backup size]].
- Also [[https://www.linux-tips-and-tricks.de/en/faq/#a29][Permission
  errors]] would needed to be worked around.
- Taking offsite backup with rsync to local NFS/SSHFS on my NAS would take considerably more
  time than to a local usb-stick --> We want to keep the downtime as low as
  possible while backups run.
- Though unlikely, but basically admins of ssh hosting provider, have access
  via backups to my browsing history through =/etc/pihole/pihole-FTL.db=.
  This is a privacy concern.
- So local USB-stick attached to rpi3 it is.

** Preparation

- Attach an USB pen drive into rpi3.

** Preinstallation tasks on Rasbian

I chose the cheapest 8 Gb USB thumb drive because I know the rpi3 backups will be ~2 GB
in size and rotation of 3 versions means I need at least 7 GB of space.
After attaching the USB, check its assigned device name:

~$ sudo blkid -o list~

#+BEGIN_SRC bash
device                                 fs_type       label          mount point                                UUID
---------------------------------------------------------------------------------------------------------------------------------------------------
/dev/mmcblk0p1                         vfat          boot           /boot                                      16D2-035F
/dev/mmcblk0p2                         ext4          rootfs         /                                          d065e631-6b9d-48c0-a8fe-e663b42828e0
/dev/sda1                              vfat          ADATA UFD      (not mounted)                              2999e9de-1aa2-43bb-b915-5d08d3301b91
/dev/mmcblk0                                                        (in use)
#+END_SRC

Leave the USB thumb drive as "not mounted" so that it can be formatted as ext4:

~$ sudo mkfs.ext4 /dev/sda1~

#+BEGIN_SRC bash
mke2fs 1.44.5 (15-Dec-2018)
/dev/sda1 contains a vfat file system labelled 'ADATA UFD'
Proceed anyway? (y,N) y
Creating filesystem with 1894393 4k blocks and 474208 inodes
Filesystem UUID: edcaf029-b0ab-421b-9d45-23ae1d7ea02a
Superblock backups stored on blocks:
  32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
#+END_SRC

Ext4 is preferred as it supports hard linking (resulting in smaller copies
when backupping with =rsync=).

*** DONE Mount the USB disk/thumb drive

I have the USB thumb drive labeled as =/dev/sda1=. To mount it to =/backup=:\\
~$ sudo mkdir /backup~

~$ sudo mount /dev/sda1 /backup~

To make sure usb is mounted after rpi3 is restarted, add an entry to fstab:

~$ sudo nano /etc/fstab~

#+BEGIN_SRC bash
UUID=edcaf029-b0ab-421b-9d45-23ae1d7ea02a /backup auto nosuid,nodev,nofail 0 0
#+END_SRC

UUID above is result of: ~sudo blkid /dev/sda1 | awk -F'"' '{print $2}'~

** Installation on Raspberry Pi OS

Fetch the latest raspiBackup and execute its configuration UI:

~$ cd ~ && curl -sSLO https://www.linux-tips-and-tricks.de/raspiBackupInstallUI.sh && sudo bash ./raspiBackupInstallUI.sh~

Select:

#+BEGIN_SRC bash -n -r
M2 Install components
  I1 Install raspiBackup using a default configuration

M3 Configure major options
  C2 Backup path
    /backup

  C3 Backup versions
    (*) Keep a maximum number of backups
        3

  C4 Backup type
    (*) Backup with rsync and use hardlinks if possible

  C5 Backup mode
    (*) Backup the two standard partitions

  C6 Services to stop and start     (ref:services)
    [*] 1: log2ram
    [*] 2: monit
    [*] 3: cron
    [*] 4: lighttpd
    [*] 5: cloudflared
    [*] 6: pihole-FTL

  C7 Message verbosity
    (*) Display important messages only

  C9 Regular backup
    R1 Enable regular backup
      R2 Weekday of regular backup
        (*) Saturday
      R3 Time of regular backup
        03:12
#+END_SRC

*Additional note on above [[(services)][C6 Services to stop and start]]*: The order matters. The most
vital services, like =cloudflared= and =pihole-FTL=, must be stopped last, and
started first. This is because if in case there's a problem in
stopping/starting services, the script fails, resulting in situation where
most vital services (=cloudflared= and =pihole-FTL=) may not be started after
they've been stopped.

[[https://www.reddit.com/r/pihole/comments/6seb16/services_used_by_pihole/][Pi-hole
related services]]
are marked to be stopped/started. This way we ensure the backups are robust
and no volatile, transactional data can jeopardize the integrity of the backups.

At the end of configuration UI, save configuration and cron settings for raspiBackup. Then choose Finish.

The configuration file is saved to =/usr/local/etc/raspiBackup.conf= and
weekly backup by cron is saved to =/etc/cron.d/raspiBackup=. The actual
executable backup script is run by cron every Saturday night at 03:12, and
will be saved to =/usr/local/bin/raspiBackup.sh=. Once the raspiBackup has
run, the logs are saved to
=/backup/raspberrypi/raspberrypi-rsync-backup-{date}-{time}/raspiBackup.log=.

Run the backup for the first time manually to inspect everything is working:

~$ sudo bash /usr/local/bin/raspiBackup.sh~

#+BEGIN_SRC bash
--- RBK0009I: raspberrypi: raspiBackup.sh V0.6.4.3 (2d927a2) started at Sat 14 Dec 16:06:25 EET 2019.
--- RBK0128I: Using logfile /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspiBackup.log.
--- RBK0116I: Using config file /usr/local/etc/raspiBackup.conf.
--- RBK0151I: Using backuppath /backup.
--- RBK0008I: Stopping services: 'systemctl stop pihole-FTL && systemctl stop dbus'.
Warning: Stopping dbus.service, but it can still be activated by:
  dbus.socket
--- RBK0081I: Creating backup of type rsync in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625.
--- RBK0036I: Saving partition layout.
--- RBK0044I: Creating backup of boot partition in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspberrypi-backup.img.
42+1 records in
42+1 records out
44979712 bytes (45 MB, 43 MiB) copied, 1.90504 s, 23.6 MB/s
--- RBK0045I: Creating backup of partition layout in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspberrypi-backup.sfdisk.
--- RBK0046I: Creating backup of master boot record in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspberrypi-backup.mbr.
--- RBK0158I: Creating native rsync backup "/backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625".
--- RBK0085I: Backup of type rsync started. Please be patient.
--- RBK0078I: Backup time: 00:16:35.
--- RBK0159I: 3 backups kept for rsync backup type.
--- RBK0007I: Starting services: 'systemctl start dbus && systemctl start pihole-FTL'.
--- RBK0033I: Please wait until cleanup has finished.
--- RBK0049I: Messages saved in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspiBackup.msg.
--- RBK0026I: Debug logfile saved in /backup/raspberrypi/raspberrypi-rsync-backup-20191214-160625/raspiBackup.log.
--- RBK0017I: Backup finished successfully.
--- RBK0010I: raspberrypi: raspiBackup.sh V0.6.4.3 (2d927a2) stopped at Sat 14 Dec 16:23:44 EET 2019.
#+END_SRC

First backup is created at =/backup=.

Two more separate backups are created weekly every Saturday morning at 03:12
AM. The oldest one will be overwritten. A full snapshot of 3 separate backups:

#+BEGIN_SRC bash
pi@raspberrypi:~ $ ls -al /backup/raspberrypi/
total 20
drwxr-xr-x  5 root root 4096 Jan 13 03:12 .
drwxr-xr-x  4 root root 4096 Jan 13 21:13 ..
drwxr-xr-x 22 root root 4096 Dec 14 03:22 raspberrypi-rsync-backup-20191214-031201
drwxr-xr-x 22 root root 4096 Dec 21 03:21 raspberrypi-rsync-backup-20191221-031201
drwxr-xr-x 22 root root 4096 Dec 28 03:17 raspberrypi-rsync-backup-20191228-031201
pi@raspberrypi:~ $
#+END_SRC

Newest is =raspberrypi-rsync-backup-20191228-031201=. Let's restore that in order to test that backups are working.

** Post-installation tasks

*** DONE Setup notification email

We'll use gmail for this. First, install a Mail Transfer Agent, we'll use
~msmtp~ as it is easy to set up:

~$ sudo aptitude install msmtp msmtp-mta~

Configure it:

~$ sudo nano /etc/msmtprc~

#+BEGIN_SRC bash
# Gmail specifics
defaults
auth on
tls on
tls_certcheck off
logfile ~/msmtp.log

account        gmail
host           smtp.gmail.com
port           587

from          pi@raspberrypi
user           yourname@gmail.com
password       your-generated-google-app-password-for-mstmtp

# Default
account default : gmail
#+END_SRC

If you haven't already, remember to visit
  [[https://myaccount.google.com/security][Google's
  security page]], enable 2-step verification, and generate an app password for
  =/etc/msmtprc=. Otherwise Google rejects the mails to be sent.

Finally, test that mails are sent with:

~$ echo 'test' | msmtp alert.destination@.mail.com~

Configure raspiBackup to use this mail:

~$ sudo nano /usr/local/etc/raspiBackup.conf~

#+BEGIN_SRC bash
# email to send completion status
DEFAULT_EMAIL="alert.destination@mail.com"

# Send eMail and/or Telegram notification when backup starts
DEFAULT_NOTIFY_START=1

# Send email only in case of errors. Use with care !
DEFAULT_MAIL_ON_ERROR_ONLY=1
#+END_SRC

Test backup and mail sending with:

~$ sudo bash /usr/local/bin/raspiBackup.sh -F~

*** NEXT Encrypt =msmtp= gmail password with =gpg=
- As per instruction [[https://www.howtoraspberry.com/2021/06/how-to-send-mail-from-a-raspberry-pi/][here.]]

*** NEXT Testing the restoration of a backup on an SD-card

Attach to rpi3's USB port an SD-card reader, with an SD-card in it. The SD-card
doesn't need to be emptied or formatted; the restore script handles that [fn:6]. Leave the USB-stick in place.

Check that Raspberry Pi OS identifies the new card we're about to restore with a
backup from the USB-stick:

~$ fdisk -l | egrep "^Disk /|^/dev"~

#+BEGIN_SRC shell -n -r
pi@raspberrypi:~ $ sudo fdisk -l | egrep "^Disk /|^/dev"
Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram2: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram3: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram4: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram5: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram6: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram7: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram8: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram9: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram10: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram11: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram12: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram13: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram14: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/ram15: 4 MiB, 4194304 bytes, 8192 sectors
Disk /dev/mmcblk0: 14.9 GiB, 15962472448 bytes, 31176704 sectors		(ref:mmcblk0)
/dev/mmcblk0p1       8192    96042    87851 42.9M  c W95 FAT32 (LBA)	(ref:mmcblkp1)
/dev/mmcblk0p2      98304 31176703 31078400 14.8G 83 Linux			(ref:mmcblkp2)
Disk /dev/sda: 7.2 GiB, 7759462400 bytes, 15155200 sectors			(ref:devsda)
/dev/sda1  *       56 15155199 15155144  7.2G  c W95 FAT32 (LBA)
#+END_SRC

Where,

- [[(mmcblk0)][mmcblk0]]: Old, internal SD card
- [[(mmcblkp1)][mmcblkp1]]: Old =/boot= partition
- [[(mmcblkp2)][mmcblkp2]]: Old =/root= partition
- [[(devsda)][dev/sda]]: Old, external usb-stick, mounted to /backup, containing the backup for restore
- : New, external SD-card, soon to be written with restored backup

To restore:

~$ sudo raspiBackup.sh -d /dev/sda/backup/raspberrypi/raspberrypi-rsync-backup-20191228-031201/~

More information at raspiBackup's [[https://www.linux-tips-and-tricks.de/en/restore/#rsd][site]].

** Maintenance

*** Updating raspiBackup

Check the latest version you have: ~$ sudo /usr/local/bin/raspiBackup.sh --version~

#+BEGIN_SRC bash
Version: 0.6.5.1 CommitSHA: 9cc17fc CommitDate: 2020-10-31 CommitTime:
21:52:43
#+END_SRC

To update, do ~$ sudo bash /usr/local/bin/raspiBackup.sh -U~

#+BEGIN_SRC bash
$ sudo bash /usr/local/bin/raspiBackup.sh -U
--- RBK0031I: Checking whether a new version of raspiBackup.sh is available.
--- RBK0190I: Upgrading raspiBackup.sh from version 0.6.5.1 to 0.6.6.
--- RBK0038I: Are you sure? y/N
y
--- RBK0057I: Downloading file raspiBackup.sh from https://www.linux-tips-and-tricks.de.
--- RBK0072I: /usr/local/bin/raspiBackup.sh updated from version 0.6.5.1 to version 0.6.6. Previous version saved as /usr/local/bin/raspiBackup.0.6.5.1.sh. Don't forget to test backup and restore with the new version now.
--- RBK0241I: Merging current configuration v0.1.3 with new configuration v0.1.4 into /usr/local/etc/raspiBackup.conf.merged.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE=0.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE_DRYRUN=1.
--- RBK0248I: Added option DEFAULT_SMART_RECYCLE_OPTIONS="7 4 12 1".
--- RBK0248I: Added option DEFAULT_TELEGRAM_TOKEN="".
--- RBK0248I: Added option DEFAULT_TELEGRAM_CHATID="".
--- RBK0248I: Added option DEFAULT_TELEGRAM_NOTIFICATIONS="F".
--- RBK0248I: Added option DEFAULT_NOTIFY_START=0.
--- RBK0248I: Added option DEFAULT_COLORING="CM".
--- RBK0248I: Added option DEFAULT_EMAIL_COLORING="SUBJECT".
--- RBK0248I: Added option DEFAULT_DYNAMIC_MOUNT="".
--- RBK0249I: Deleted option DEFAULT_APPEND_LOG=0.
--- RBK0249I: Deleted option DEFAULT_APPEND_LOG_OPTION="-a".
--- RBK0249I: Deleted option DEFAULT_RESIZE_ROOTFS=1.
--- RBK0243I: Configuration merge finished successfullly but not activated.
!!! RBK0245W: Backup current configuration in /usr/local/etc/raspiBackup.conf.bak and activate updated configuration? y/N y
--- RBK0240I: Saving current configuration /usr/local/etc/raspiBackup.conf to /usr/local/etc/raspiBackup.conf.bak.
--- RBK0244I: Merged configuration /usr/local/etc/raspiBackup.conf.merged copied to /usr/local/etc/raspiBackup.conf and activated.
#+END_SRC

#+begin_note
*Note:* Graphical installer =/usr/local/bin/raspiBackupInstallUI.sh= needs to be updated
separately by running ~$ sudo raspiBackupInstallUI.sh~ and navigating to =M5
Update components= - =P2 Update raspiBackupInstallUI.sh=.
#+end_note

To be notified of new updates, subscribe to raspiBackup
[[https://github.com/framps/raspiBackup/releases.atom][releases rss-feed]].

** Troubleshooting

*** DONE Updating =raspiBackupInstallUI.sh= errors out with Syntax error: newline unexpected
After running ~$ sudo raspiBackupInstallUI.sh~ and selecting =P2 UpdateraspiBackupInstallUI.sh=, script exits and update passes, but then
invoking the script again with ~$ sudo raspiBackupInstallUI.sh~ gives an
error:

#+BEGIN_SRC bash
/usr/local/bin/raspiBackupInstallUI.sh: 2:/usr/local/bin/raspiBackupInstallUI.sh: Syntax error: newline unexpected
#+END_SRC

In this case reinstall the update script with:

~$ curl -sLO https://www.linux-tips-and-tricks.de/raspiBackupInstallUI.sh~

~$ sudo mv raspiBackupInstallUI.sh /usr/local/bin~

~$ sudo chmod +x /usr/local/bin/raspiBackupInstallUI.sh~

~$ sudo chown root.root /usr/local/bin/raspiBackupInstallUI.sh~

Verify the script now functions with the newest version:

#+BEGIN_SRC bash
$ sudo /usr/local/bin/raspiBackupInstallUI.sh -h
raspiBackupInstallUI.sh 0.4.3.4, 2020-11-04/22:18:13 - 6c4a442
raspiBackupInstallUI.sh ( -i [-e]? | -u | -U ) [-d]?
-d: enable debug mode
-e: unattended (re)install of raspiBackup extensions
-i: unattended (re)install of raspiBackup
-U: unattended update of raspiBackupInstallUI.sh
-u: unattended uninstall of raspiBackup
#+END_SRC

*** DONE raspiBackup does not clear old backup versions at =/backup/raspberrypi/=
- Seemed to be because of
  [[https://github.com/framps/raspiBackup/issues/433][issue 443]]. Fixed in
  latest commits so updating my script to latest with:

~$ sudo bash /usr/local/bin/raspiBackup.sh -U -S~ (~-S~ flag retrieves the latest commits) fixed the issue.

*** DONE raspiBackup subject line contains MIME-Version

As reported in [[https://github.com/framps/raspiBackup/issues/264][issue
264]].

Fixed with a workaround by disabling mail colors in
=/usr/local/etc/raspiBackup.conf=:

#+BEGIN_SRC bash
# Colorize console output (C) and/or email (M)
DEFAULT_COLORING="C"
#+END_SRC

** Observations and future direction

----------------------------------------------------------------------------------------------------------

* Footnotes
[fn:1] [[https://en.wikipedia.org/wiki/DNS_sinkhole][Wikipedia article]] that elaborates on this.
[fn:2] With Pi-Hole use scenario, best SD-cards have a high randIO rating. Meaning, a card that tolerates lots of random read and write.
[fn:3] There are
[[https://www.reddit.com/r/pihole/comments/dujvpq/besides_ads_can_pihole_be_beneficial_from_a/][several]]
[[https://www.reddit.com/r/pihole/comments/dz6fi9/best_block_list/][blocklists]]
to
[[https://www.reddit.com/r/pihole/comments/gxopny/recommend_me_some_blocklists/][choose]]
[[https://firebog.net/][from]].
[fn:4] The hourly saving of logs does not nullify the real time logging, because as said, the logs are still there, but stored into RAM memory, which is dumped hourly to SD-card.
[fn:5] Additional [[https://github.com/adrianmihalko/raspberrypiWireGuard/blob/master/README.md][WireGuard]] [[https://github.com/pirate/WireGuard-docs/blob/master/README.md][guides in github]].
[fn:6] Use the same make and model (rpi3), and the same operating system for restore which was used to create the backup.
[fn:7] Raspberry shows under "Network Map Clients List" with "Manual" IP, not
"Static", but that should be the same. [[https://forums.tomshardware.com/threads/static-vs-manual-ip.3077474/][Manual vs. static IP explained]].
[fn:8] Note that Conditional Forwarding
[[https://www.reddit.com/r/pihole/comments/dj7pns/conditional_forwarding_doesnt_seem_to_be_working/f44zgyx/][does
not work for all routers]] and may lead to slower performance due to
[[https://www.reddit.com/r/pihole/comments/eyp231/super_slow_dns_performance_issues/fgj8ydd/][DNS
looping]]. This manifests itself as memory and log file space consumption.
[fn:9] rpi *can* have a hardware based realtime clock installed on it RTC
shim / hat, see e.g.
[[https://thepihut.com/blogs/raspberry-pi-tutorials/17209332-adding-a-real-time-clock-to-your-raspberry-pi][this
article]].
[fn:10] For mode information on maintaining system with apt-listchanges and
apt-listbugs: [[https://wiki.debian.org/DebianUnstable#What_are_some_best_practices_for_testing.2Fsid_users.3F][What
are some best practices for testing/sid users]]\\
Also see: [[https://www.reddit.com/r/debian/comments/65cjms/how_to_avoid_breaking_debian_unstable/][How
to avoid breaking debian unstable]]
[fn:11] Instead of installing WireGuard and routing VPN-traffic through home
connection, one can also bypass selfhosted WireGuard completely, and go for
iOS client based
[[https://www.reddit.com/r/WireGuard/comments/gxtqsp/blokada_ad_blocker_with_builtin_WireGuard_vpn_now/][Blokada]].
[fn:12] Alternative speedtest service is
[[https://github.com/librespeed/speedtest][Librespeed speedtest]], which can
also be selfhosted.